glusterfs.git/geo-replication/syncdaemon/resource.py, branch v3.7.8

geo-rep: Symlink Rename issue

2016-01-04T06:35:38+00:00

If ENTRY creation failed for symlink in Slave and symlink
renamed in Master. If Source not exists to Rename in Slave
Geo-rep interprets as Create of Target file. Geo-rep sends blob
of regular file to create symlink instead of sending blob of
symlink.

With this patch, Geo-rep identifies symlink and sends respective
blob.

BUG: 1292697
Change-Id: If9351974d1945141a1d3abb838b7d0de7591e48e
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/12917
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Milind Changire 
Tested-by: Milind Changire 
Tested-by: NetBSD Build System 
 (cherry picked from commit 3151194fad1aed3ab340b53aeeb4e8d131538d12)
Reviewed-on: http://review.gluster.org/12998
Reviewed-by: Venky Shankar

Add support for sparse files to tarssh method

2015-12-31T04:18:46+00:00

Without '--sparse' option tar will not properly archive sparse file
and geo-replication will result in non-sparse file on the remote end.

Here is more on how I arrived at this
http://markelov.org/wiki/index.php/GlusterFS_3.6.1_on_CentOS_6.5:_geo-replication_and_sparse_files_problem

Change-Id: I8d671964a1b48bbb916e4a064571221bf3631494
BUG: 1290719
Signed-off-by: Alex Markelov 
Reviewed-on: http://review.gluster.org/12476
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Aravinda VK 
 (cherry picked from commit eff14863fb6a6da0ccf8a40343160b09f29ea493)
Reviewed-on: http://review.gluster.org/12949
Reviewed-by: Venky Shankar

geo-rep: use cold tier bricks for namespace operations

2015-12-08T06:42:35+00:00

Problem:
symlinks are not getting synced to slave in a Tiering based volume.

Solution:
Now, symlinks are created directly in cold tier bricks( in the backend).

Earlier, cold tier was avoided for namespace operations and only
hot tier was used while processing changelogs.

Now, cold tier is HASH subvolume in a Tiering volume.
So, carry out namespace operation only in cold tier subvolume and
avoid hot tier subvolume to avoid any races.

Earlier, XSYNC was used(and changeloghistory avoided) during initial sync
in order to avoid race while processing historychangelog in Hot tier.
This is no longer required as there is no race from Hot tier.

Also, avoid both live and history changelog ENTRY operations from Hot tier
to avoid any race with cold tier.

Change-Id: Ia8fbb7ae037f5b6cb683f36c0df5c3fc2894636e
BUG: 1288027
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12844
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Venky Shankar 
(cherry picked from commit 93f31189ce8f6e2980a39b02568ed17088e0a667)
Reviewed-on: http://review.gluster.org/12891

geo-rep: Avoid cold tier bricks during ENTRY operation

2015-12-07T09:50:47+00:00

This is a series of patch which aims to fix geo-replication
in a Tiering Volume.

Problem:
Consider, a file is placed in volume initially and then hot tier is
attached. During any operation on the file, due to lookup a linkto
file is created in hot tier.

Now, any namespace operation carried out on the file is recorded in
both cold and hot tier.
There is a room for races when both changelogs are replayed.

Solution:
So, We are going to replay (namespace related)operations
only in the hot tier.

Why?
a. If the file is directly placed in Hot tier , all fops will be
recorded in HOT tier.
b. If  the file is already present in Cold tier, and if any fop is
carried out, it creates linkto file in Hot tier.
Now, operations like UNLINK, RENAME are captured in Hot
tier(by means of linkto file).

This way, we can get both tier's operation in HOT tier itself.

Now, once the file is demoted to COLD tier, any namespace operation
carried out on the cold tier can be avoided as we directly RECORD
the same in HOT tier.

How?
1. Check whether the brick is cold tier and skip ENTRY operation.
2. Also, if it is cold tier brick, use Xsync(which is used during initial run).
   This will help in getting all cold tier bricks changes using File System crawl
   and helps in avoiding races with hot tier brick(which can happen
   if historychangelog used in cold tier brick).

Dependent patches:
1. http://review.gluster.org/12239
2. http://review.gluster.org/12326

Change-Id: I7692b1dbb8813a7e253451bca02f8f09a5782dde
BUG: 1275173
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12355
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK 
(cherry picked from commit 6188b5fcebc56b3d8af1956beeec9988f3e8f268)
Reviewed-on: http://review.gluster.org/12429
Reviewed-by: Venky Shankar

geo-rep: geo-rep to handle CAPS based Hostname

2015-12-03T10:24:31+00:00

Problem:
geo-replication session creation fails when Hostname
is having CAPS in it.
Issue is with the regex pattern which handles only small lettered
Hostname.

Fix:
Fix the regex pattern to handle CAPS based hostname as well.

Change-Id: I5c99c102e9706acc2b1fab1e6bf158e68beed373
BUG: 1287960
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12216
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Milind Changire 
Reviewed-by: Aravinda VK 
Reviewed-by: Vijay Bellur 
(cherry picked from commit f6142e97c6219d2c1c482a414690020ed1275bdb)
Reviewed-on: http://review.gluster.org/12862
Reviewed-by: Venky Shankar

geo-rep: New Config option for ssh_port

2015-11-25T07:18:38+00:00

If different port used for SSH instead of 22, Geo-replication
was failing to establish SSH connection.

ssh_port option can be added using config:ssh_command and
config:ssh_command_tar, but user has to remember complete
ssh command used with parameter to add/modify ssh port.

This patch adds new config option for ssh_port,

gluster volume geo-replication   \
        config ssh_port 52022

Change-Id: I7753a09485f0b1f49d2b2a80b962c720817c96f4
Signed-off-by: Aravinda VK 
BUG: 1283060
Reviewed-on: http://review.gluster.org/12444
Tested-by: NetBSD Build System 
Reviewed-by: Saravanakumar Arumugam 
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar 
(cherry picked from commit 7d35eb5926869ed084295600502a85ce13be506f)
Reviewed-on: http://review.gluster.org/12607
Reviewed-by: Kotresh HR

geo-rep: Kill Geo-rep Worker when Agent process dies

2015-11-25T07:17:04+00:00

When Changelog agent process dies, Geo-replication fails to detect
and worker will run without respective Changelog agent. Status shows
Active/Passive without any progress.

With this patch, Worker process gets killed whenever Changelog
agent dies.

Change-Id: I30b4cc77f924f7e8174b8bfe415ac17f0b3851b4
Signed-off-by: Aravinda VK 
BUG: 1279362
Reviewed-on: http://review.gluster.org/12485
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar 
Reviewed-by: Kotresh HR 
(cherry picked from commit 5d1ff7efd6ab3bd29a29922a9ea1e1aaf02544ad)
Reviewed-on: http://review.gluster.org/12550

geo-rep: ignore ESTALE as ENOENT

2015-06-28T14:47:40+00:00

When DHT can't resolve a File it raises ESTALE, ignore ESTALE errors
same as ENOENT after retry.

Affected places:
    Xattr.lgetxattr
    os.listdir
    os.link
    Xattr.lsetxattr
    os.chmod
    os.chown
    os.utime
    os.readlink

BUG: 1236093
Change-Id: I53f8dfa47911da93e0dcc20213afcbb47a14ccd8
Reviewed-On: http://review.gluster.org/11296
Original-Author: Aravinda VK 
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/11430
Tested-by: NetBSD Build System 
Reviewed-by: Milind Changire 
Reviewed-by: Vijay Bellur

geo-rep: Ignore ESTALE during unlink/rmdir

2015-06-01T06:38:15+00:00

during unlink/rmdir of Parent_GFID/Basename, if parent
directory does not exists. Parent GFID will not get resolved
and DHT raises ESTALE instead of ENOENT.

Now ESTALE errors ignored during unlink/rmdir

BUG: 1223286
Change-Id: If275c89fb9fc7d16004550805a4cd65be818540d
Signed-off-by: Aravinda VK 
Reviewed On: http://review.gluster.org/#/c/10837/
Reviewed-on: http://review.gluster.org/10913
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Venky Shankar

geo-rep: Disable xattrs and acls support with tar_ssh

2015-05-29T09:02:32+00:00

Geo-rep can't sync xattrs and acls with tar over ssh
for following reasons.

Issue 1: xattrs doesn't sync with tar over ssh.

Reason: untar doesn't respect '--overwrite' option when used along
        with '--xattrs'. So it sends unlink if the file exists on
        destination and re-creates afresh. But all entry operations
        are banned in aux-gfid-mount as it may lead to gfid-mismatch.
        Hence fails with EPERM. This happens only when some xattr is
        set on a file in master volume.

Issue2: acls on directories does not sync with tar over ssh.

Reason: tar tries to opendir ".gfid/" and is not supported
        by gfid-access-translator as readirp can't be handled on
        virtual inodes and hence fails with ENOTSUP where as it syncs
        for files.

Since the issue is with tar commmand it self and nothing could be
done from gluster side, disabling xattr and acls support with tar
over ssh option.

Geo-rep can sync xattrs and acls with 'rsync' as the sync engine.

BUG: 1223644
Change-Id: Ieba9e749ddbc41683b58ed8b1cc2ca50517fe58b
Reviewed On: http://review.gluster.org/#/c/10873
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10899
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System
Reviewed-by: Aravinda VK