glusterfs.git/geo-replication/syncdaemon/resource.py, branch v3.8rc2

geo-rep: Fix Entry Creation issue with non root UID/GID

2016-03-09T11:02:01+00:00

During entry_ops RENAME Geo-rep sends stat info along with the
recorded info from Changelog. In Slave side if Source file exists
Geo-rep renames to Target file by calling os.rename. If source file
does not exists, it tries to create Target file directly using available
stat info from Master. If UID and GID are different in Master for that
file then stat info will have different UID/GID during Create. Geo-rep
gets EACCES when it tries to create a new entry using gfid-access with
different UID/GID.

With this patch, Entry creation with different UID/GID is split into two
operations. Create Entry with UID:0 and GID:0 and then set UID/GID.

Change-Id: I4987e3a205d8513c06fa66198cde145a87003a01
BUG: 1313303
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/13542
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep: hard-link rename issues on changelog replay

2016-03-09T08:25:15+00:00

Problem:
LINK + RENAME changelog when replayed after worker restart causes stale
hard-links to persist since VFS returns success for RENAME if hard-links
point to same inode.

Solution:
Worker detects RENAME being issued on hard-links to the same inode and
unlinks the source file-name. Conditionally rename by verifying that the
source gfid matches with the on-disk gfid on the slave.

Change-Id: I3ff1c30ef79e77503c8b246d46dab8ac3059ccf2
BUG: 1296174
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/13189
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

geo-rep: Fix CPU utilization 100% with Popen

2016-03-08T13:53:49+00:00

Errorstore is maintained by Geo-rep to collect errors from
the child processes opened using Popen. If Popen.communicate
is used then it closes stderr. When stderr is not available
errorstore.tailer() will enter into infinite loop without gap.

With this patch, sleep time added when stderr of Child process
is already closed.

Change-Id: Ic36aabd6de35b259467d0bab7952468432867a94
BUG: 1315601
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/13637
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kotresh HR 
CentOS-regression: Gluster Build System

georep: avoid creating multiple entries with same gfid

2016-03-01T07:58:05+00:00

Problem:
CREATE + RENAME changelogs replayed by geo-replication cause
stale old-name entries with same gfid on slave nodes.
A gfid is a unique key in the file-system and should not be
assigned to multiple entries.

Solution:
Create entry on slave only if lstat(gfid) at aux-mount fails.
This applies to files as well as directories.

Change-Id: Ice3340f4ae1251c2dcef024a2388c4d33b5d4919
BUG: 1296206
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/13316
Smoke: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Aravinda VK 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

geo-rep: Handling Rsync/Tar errors efficiently

2016-02-26T10:15:01+00:00

Geo-rep processes Changelogs in Batch, if one file in batch
fails with rsync error that Changelog file is reprocessed multiple times.
After MAX_RETRY, it logs all the GFIDs from that batch as Skipped.

This patch addresses following issues,
1. When Rsync/Tar fails do not parse Changelog again for retry
2. When Rsync/Tar fails do not replay Entry operations, only retry
   rsync/tar for those GFIDs
3. Log Error in Rsync/Tar only in the last Retry
4. Do not log Skipped GFIDs since Rsync/Tar errors are logged for
   only failed files.
5. Changed Entry failures as Error instead of Warning

BUG: 1287723
Change-Id: Ie134ce2572693056ab9b9008cd8aa5b5d87f7975
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/12856
Reviewed-by: Kotresh HR 
Reviewed-by: Saravanakumar Arumugam 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Venky Shankar

geo-rep: Symlink Rename issue

2015-12-18T05:46:35+00:00

If ENTRY creation failed for symlink in Slave and symlink
renamed in Master. If Source not exists to Rename in Slave
Geo-rep interprets as Create of Target file. Geo-rep sends blob
of regular file to create symlink instead of sending blob of
symlink.

With this patch, Geo-rep identifies symlink and sends respective
blob.

BUG: 1289859
Change-Id: If9351974d1945141a1d3abb838b7d0de7591e48e
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/12917
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Milind Changire 
Tested-by: Milind Changire 
Tested-by: NetBSD Build System

Add support for sparse files to tarssh method

2015-12-11T08:27:49+00:00

Without '--sparse' option tar will not properly archive sparse file
and geo-replication will result in non-sparse file on the remote end.

Here is more on how I arrived at this
http://markelov.org/wiki/index.php/GlusterFS_3.6.1_on_CentOS_6.5:_geo-replication_and_sparse_files_problem

Change-Id: I8d671964a1b48bbb916e4a064571221bf3631494
BUG: 1276839
Signed-off-by: Alex Markelov 
Reviewed-on: http://review.gluster.org/12476
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Aravinda VK

geo-rep: use cold tier bricks for namespace operations

2015-12-03T10:34:53+00:00

Problem:
symlinks are not getting synced to slave in a Tiering based volume.

Solution:
Now, symlinks are created directly in cold tier bricks( in the backend).

Earlier, cold tier was avoided for namespace operations and only
hot tier was used while processing changelogs.

Now, cold tier is HASH subvolume in a Tiering volume.
So, carry out namespace operation only in cold tier subvolume and
avoid hot tier subvolume to avoid any races.

Earlier, XSYNC was used(and changeloghistory avoided) during initial sync
in order to avoid race while processing historychangelog in Hot tier.
This is no longer required as there is no race from Hot tier.

Also, avoid both live and history changelog ENTRY operations from Hot tier to avoid any race with cold tier.

Change-Id: Ia8fbb7ae037f5b6cb683f36c0df5c3fc2894636e
BUG: 1287519
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12844
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Venky Shankar

geo-rep: geo-rep to handle CAPS based Hostname

2015-12-02T15:35:18+00:00

Problem:
geo-replication session creation fails when Hostname
is having CAPS in it.
Issue is with the regex pattern which handles only small lettered
Hostname.

Fix:
Fix the regex pattern to handle CAPS based hostname as well.

Change-Id: I5c99c102e9706acc2b1fab1e6bf158e68beed373
BUG: 1265522
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12216
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Milind Changire 
Reviewed-by: Aravinda VK 
Reviewed-by: Vijay Bellur

geo-rep: New Config option for ssh_port

2015-11-17T15:00:06+00:00

If different port used for SSH instead of 22, Geo-replication
was failing to establish SSH connection.

ssh_port option can be added using config:ssh_command and
config:ssh_command_tar, but user has to remember complete
ssh command used with parameter to add/modify ssh port.

This patch adds new config option for ssh_port,

gluster volume geo-replication   \
        config ssh_port 52022

Change-Id: I7753a09485f0b1f49d2b2a80b962c720817c96f4
Signed-off-by: Aravinda VK 
BUG: 1276028
Reviewed-on: http://review.gluster.org/12444
Tested-by: NetBSD Build System 
Reviewed-by: Saravanakumar Arumugam 
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar