glusterfs.git/geo-replication/syncdaemon/master.py, branch v3.7.8

geo-rep: Symlink Rename issue

2016-01-04T06:35:38+00:00

If ENTRY creation failed for symlink in Slave and symlink
renamed in Master. If Source not exists to Rename in Slave
Geo-rep interprets as Create of Target file. Geo-rep sends blob
of regular file to create symlink instead of sending blob of
symlink.

With this patch, Geo-rep identifies symlink and sends respective
blob.

BUG: 1292697
Change-Id: If9351974d1945141a1d3abb838b7d0de7591e48e
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/12917
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Milind Changire 
Tested-by: Milind Changire 
Tested-by: NetBSD Build System 
 (cherry picked from commit 3151194fad1aed3ab340b53aeeb4e8d131538d12)
Reviewed-on: http://review.gluster.org/12998
Reviewed-by: Venky Shankar

geo-rep: use cold tier bricks for namespace operations

2015-12-08T06:42:35+00:00

Problem:
symlinks are not getting synced to slave in a Tiering based volume.

Solution:
Now, symlinks are created directly in cold tier bricks( in the backend).

Earlier, cold tier was avoided for namespace operations and only
hot tier was used while processing changelogs.

Now, cold tier is HASH subvolume in a Tiering volume.
So, carry out namespace operation only in cold tier subvolume and
avoid hot tier subvolume to avoid any races.

Earlier, XSYNC was used(and changeloghistory avoided) during initial sync
in order to avoid race while processing historychangelog in Hot tier.
This is no longer required as there is no race from Hot tier.

Also, avoid both live and history changelog ENTRY operations from Hot tier
to avoid any race with cold tier.

Change-Id: Ia8fbb7ae037f5b6cb683f36c0df5c3fc2894636e
BUG: 1288027
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12844
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Venky Shankar 
(cherry picked from commit 93f31189ce8f6e2980a39b02568ed17088e0a667)
Reviewed-on: http://review.gluster.org/12891

geo-rep: fd close and fcntl issue

2015-12-07T13:31:33+00:00

When any of the open fd of a file is closed
on which fcntl lock is taken even though another
fd of the same file is open on which lock is taken,
all fcntl locks will be released. This causes both
replica workers to be ACTIVE sometimes. This patche
fixes that issue.

BUG: 1287456
Change-Id: I8f113230d16075f18020044898a7eda5e2a5a0ac
Original-Author: Aravinda VK 
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/12752
Tested-by: Gluster Build System 
Reviewed-by: Milind Changire 
Tested-by: NetBSD Build System 
Reviewed-by: Aravinda VK 
Reviewed-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/12842

geo-rep: Avoid cold tier bricks during ENTRY operation

2015-12-07T09:50:47+00:00

This is a series of patch which aims to fix geo-replication
in a Tiering Volume.

Problem:
Consider, a file is placed in volume initially and then hot tier is
attached. During any operation on the file, due to lookup a linkto
file is created in hot tier.

Now, any namespace operation carried out on the file is recorded in
both cold and hot tier.
There is a room for races when both changelogs are replayed.

Solution:
So, We are going to replay (namespace related)operations
only in the hot tier.

Why?
a. If the file is directly placed in Hot tier , all fops will be
recorded in HOT tier.
b. If  the file is already present in Cold tier, and if any fop is
carried out, it creates linkto file in Hot tier.
Now, operations like UNLINK, RENAME are captured in Hot
tier(by means of linkto file).

This way, we can get both tier's operation in HOT tier itself.

Now, once the file is demoted to COLD tier, any namespace operation
carried out on the cold tier can be avoided as we directly RECORD
the same in HOT tier.

How?
1. Check whether the brick is cold tier and skip ENTRY operation.
2. Also, if it is cold tier brick, use Xsync(which is used during initial run).
   This will help in getting all cold tier bricks changes using File System crawl
   and helps in avoiding races with hot tier brick(which can happen
   if historychangelog used in cold tier brick).

Dependent patches:
1. http://review.gluster.org/12239
2. http://review.gluster.org/12326

Change-Id: I7692b1dbb8813a7e253451bca02f8f09a5782dde
BUG: 1275173
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12355
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK 
(cherry picked from commit 6188b5fcebc56b3d8af1956beeec9988f3e8f268)
Reviewed-on: http://review.gluster.org/12429
Reviewed-by: Venky Shankar

geo-rep: Fix FD leak from Active Geo-rep worker

2015-12-02T07:13:05+00:00

Active worker tries to acquire lock in each iteration. On every successfull
lock acqusition it was not closing previously opened lock fd.

To see the leak, get the PID of worker,
    ps -ax | grep feedback-fd
    watch ls /proc/$pid/fd

BUG: 1225567
Change-Id: Ic476c24c306e7ab372c5560fbb80ef39f4fb31af
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/12332
Tested-by: NetBSD Build System 
Reviewed-by: Milind Changire 
Reviewed-by: Saravanakumar Arumugam 
Reviewed-by: Venky Shankar 
 (cherry picked from commit 42def948ac8e5d24278cb000cc8f8906b83a8592)
Reviewed-on: http://review.gluster.org/12650
Tested-by: Gluster Build System

geo-rep: Handle FXATTROP and XATTROP

2015-11-27T11:55:39+00:00

GEO-REP INTEROP WITH SHARD FEATURE

If it is FXATTROP or XATTROP in changelog,
add the gfid to rsync queue.

BUG: 1284453
Change-Id: If598da1543bbff7fe9228ba57c7e813ed2a3a7f2
Reviewed-on: http://review.gluster.org/12226
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Aravinda VK 
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/12721
Reviewed-by: Saravanakumar Arumugam 
Reviewed-by: Milind Changire 
Reviewed-by: Venky Shankar

geo-rep: Fix syncing chown in xsync crawl

2015-11-27T11:54:51+00:00

GEO-REP INTEROP WITH SHARD FEATURE

Problem:
    The sequence of entry creation and chown in master
 is recorded as creation of entry with resulted
 user:group in xsync changelog. During sync, entry
 creation is always split into two ops, MKNOD and
 SETATTR. Hence the issue is not being hit otherwise
 it would have failed with EPERM if parent is owned
 by different user. But with shard translator being
 enabled on slave, doing entry creation with MKNOD and
 SETATTR is not allowed, SETATTR fails as it accesses
 inode structure which is not linked.

Solution:
    The sequence of entry creation and chown in master
 should be recorded as MKNOD and SETATTR separately always
 and do entry creation with single op in gfid-access
 xlator. The gfid-access patch will be sent separately.

BUG: 1284453
Change-Id: Ia577aa5270cb96b86830885d6c4c01fb0133eeed
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/12205
Tested-by: NetBSD Build System 
Reviewed-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/12729
Tested-by: Gluster Build System 
Reviewed-by: Milind Changire 
Reviewed-by: Venky Shankar

geo-rep: Update last_synced_time in XSync

2015-11-09T14:40:25+00:00

During XSync crawl, last_synced time in status file was not updated.
This patch fixes the issue by updating status file when stime xattr
is updated after Xsync or Changelog Crawl.

Change-Id: I4dc3a2d4c3d8378a939da0868caf1aef4f789599
Signed-off-by: Aravinda VK 
BUG: 1279306
Reviewed-on: http://review.gluster.org/11771
Tested-by: NetBSD Build System 
Reviewed-by: Jeff Darcy 
(cherry picked from commit ee0b1a3bf11a4d05696212d91c932ddb7c7091ee)
Reviewed-on: http://review.gluster.org/12545
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar

geo-rep: Add data operation if mknod with tier attribute

2015-10-29T07:18:27+00:00

This is a series of patches which aims to fix geo-replication
in a Tiering Volume.

Problem:
Consider, a file is placed in volume initially and then hot tier is
attached. During any operation on the file, due to lookup a linkto
file is created in hot tier.

Now, any namespace operation carried out on the file is recorded in
both cold and hot tier.
There is a room for races when both changelogs are replayed.

Solution:
So, We are going to replay (namespace related)operations
only in the hot tier.

Why?
a. If the file is directly placed in Hot tier, all fops will be
recorded in HOT tier.

b. If  the file is already present in Cold tier, and if any fop is
carried out, it creates linkto file in Hot tier.
Now, operations like UNLINK, RENAME are captured in Hot tier(by means of linkto file).

This way, we can get both tier's operation in HOT tier itself.

But, We may miss initial Data sync immediately after creating the
file as it is only recording MKNOD. So, if MKNOD encountered
with sticky bit set, queue DATA operation for the corresponding gfid.
(This is addressed here in this patch)

So, If tier-gfid linkto is set, we need to record the corresponding
MKNOD. Earlier this was avoided as it was set as INTERNAL fop.
(This changelog related changes are addressed in the patch:
 - http://review.gluster.org/12417)

Change-Id: I2fa84cfa2b0f86506c3d15d484138ab9651e4f83
BUG: 1275173
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12326
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Aravinda VK 
(cherry picked from commit ffc39c9d8807464b5c78959bc43dc12b22f5a37b)
Reviewed-on: http://review.gluster.org/12427
Reviewed-by: Venky Shankar

geo-rep: Fix history failure

2015-08-06T05:49:52+00:00

Both ACTIVE and PASSIVE workers register to changelog
at almost same time. When PASSIVE worker becomes ACTIVE,
the start and end time would be current stime and register_time
repectively for history API. Hence register_time would be less
then stime for which history obviously fails. But it will
be successful for the next restart as new register_time > stime.

Fix is to pass current time as the end time to history call
instead of the register_time.

Also improvised the logging for ACTIVE/PASSIVE switching.

BUG: 1247882
Change-Id: I40c582cc32fe29a6c30340ec81a3b5d30e461e71
Reviewed-on: http://review.gluster.org/11524
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Aravinda VK 
Reviewed-by: Venky Shankar 
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/11784
Reviewed-by: Milind Changire