glusterfs.git/geo-replication/syncdaemon/syncdutils.py, branch v3.5.6

gsyncd / geo-rep: geo-replication fixes

2014-01-27T17:43:19+00:00

-> "threaded" hybrid crawl.
-> Enabling metatadata synchronization.
-> Handling EINVAL/ESTALE gracefully while syncing metadata.
-> Improvments to changelog crawl code.
-> Initial crawl changelog generation format.
-> No gsyncd restart when checkpoint updated.
-> Fix symlink handling in hybrid crawl.
-> Slave's xtime key is 'stime'.
-> tar+ssh as data synchronization.
-> Instead of 'raise', just log in warning level for xtime missing cases.
-> Fix for JSON object load failure
-> Get new config value after config value reset.
-> Skip already processed changelogs.
-> Saving status of each individual worker thread.
-> GFID fetch on slave for purges.
-> Add tar ssh keys and config options.
-> Fix nlink count when using backend.
-> Include "data" operation for hardlink.
-> Use changelog time prefix as slave's time.
-> Process changelogs in parallel.

Change-Id: I09fcbb2e2e418149a6d8435abd2ac6b2f015bb06
BUG: 1036539
Signed-off-by: Ajeet Jha 
Reviewed-on: http://review.gluster.org/6404
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati 
Reviewed-on: http://review.gluster.org/6809
Reviewed-by: Vijay Bellur

geo-rep: retry in case of ENOENT errors in entry creations

2013-09-17T10:42:19+00:00

Change-Id: I8961633a7371c941a3feee44c949d5c934eca998
Original-Author: Venky Shankar 
Signed-off-by: Amar Tumballi 
BUG: 847839
Reviewed-on: http://review.gluster.org/5933
Reviewed-by: Venky Shankar 
Tested-by: Venky Shankar

gsyncd / geo-rep: Introduce basic crawl instrumentation

2013-09-05T02:50:09+00:00

This patch extends the persistent instrumentation work done by
Aravinda (@avishwa), by introducing a handfull of instrumentation
variables for crawl. These variables are "pulled up" by glusterd
in the event of a geo-replication status cli command and looks
something like below:

"Uptime=00:21:10;FilesSyned=2982;FilesPending=0;BytesPending=0;DeletesPending=0;"

"FilesPending", "BytesPending" and "DeletesPending" are short-lived
variables that are non-zero when a changelog is being processes (ie.
when an active sync in ongoing). After a successfull changelog process
"FilesPending" is summed up into "FilesSynced". The three short-lived
variabled are then reset to zero and the data is persisted

Additionally this patch also reverts some of the changes made for
BZ #986929 (those were not needed).

Change-Id: I948f1a0884ca71bc5e5bcfdc017d16c8c54fc30b
BUG: 990420
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/5441
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

geo-replication: Use a md5 based unique control path

2013-09-04T19:29:04+00:00

A hostname fqdn can be of length 255 according to RFC1123
------------------------->
/usr/include/bits/posix1_lim.h:#define _POSIX_HOST_NAME_MAX  255
<-------------------------
On linux this length is 64
------------------------->
/usr/include/bits/local_lim.h:#define HOST_NAME_MAX 64
<-------------------------

When a given hostname is > 45 (characters) - SSH fails with

-------------------------->
"ControlPath too long for Unix domain socket".
<--------------------------

Indicating that the total length of ControlPath which is
on linux should be 108

------------------------->
/usr/include/linux/un.h:#define UNIX_PATH_MAX   108
<-------------------------

This leads to "faulty" geo-replication status.

This patch brings in a new file called manifest which carries
given a geo-rep session some unique information - with which
a unique `md5` is generated in a 32length digest, this ensures
that we don't exceed UNIX_PATH_MAX limitations instead we use
a conservative approach and still be able to provide a unique
socket path.

Change-Id: I3a6a27d605d751a86e7c82eace4561d9b0134fe1
BUG: 990330
Signed-off-by: Harshavardhana 
Reviewed-on: http://review.gluster.org/5681
Tested-by: Gluster Build System 
Reviewed-by: Csaba Henk

gsyncd: distribute the crawling load

2013-07-26T20:18:57+00:00

* also consume changelog for change detection.
* Status fixes
* Use new libgfchangelog done API
* process (and sync) one changelog at a time

Change-Id: I24891615bb762e0741b1819ddfdef8802326cb16
BUG: 847839
Original Author: Csaba Henk 
Original Author: Aravinda VK 
Original Author: Venky Shankar 
Original Author: Amar Tumballi 
Original Author: Avra Sengupta 
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/5131
Reviewed-by: Vijay Bellur 
Tested-by: Vijay Bellur

move 'xlators/marker/utils/' to 'geo-replication/' directory

2013-07-22T08:53:03+00:00

Change-Id: Ibd0faefecc15b6713eda28bc96794ae58aff45aa
BUG: 847839
Original Author: Amar Tumballi 
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/5133
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur