glusterfs.git/geo-replication/syncdaemon/master.py, branch v3.6.0beta2

geo-rep: Fixing issue with xsync upper limit

2014-09-08T19:04:23+00:00

While identifying the file/dir to sync, xtime of the file was compared
with xsync_upper_limit as `xtime < xsync_upper_limit` After the sync,
xtime of parent directory is updated as stime. With the upper limit
condition, stime is updated as MIN(xtime_parent, xsync_upper_limit)

With this files will get missed if `xtime_of_file == xsync_upper_limit`

With this patch xtime_of_file is compared as
xtime_of_file <= xsync_upper_limit

BUG: 1138952
Change-Id: I469e8638ab6923e518022a539a19e2d040b60eb0
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/8439
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Vijay Bellur 
Reviewed-on: http://review.gluster.org/8644

geo-rep: minimize xsync crawl usage and set upper limit to xsync crawl

2014-09-08T17:41:59+00:00

For effective handling of deletes and renames use history crawl
as much as possible. History crawl will run in loop till it syncs
all data before live changelog time.

When it uses xsync crawl(fallback when changelog not available, or
very first crawl) it sets upper limit to crawl.

After completing History crawl, it checks actual end time returned
by history api to compare with register time, if actual end is less
than register time then run history crawl one more time.

If first turn history processing time is less than the CHANGELOG
ROLLOVER TIME then sleep for the difference, After sleep if it is
guaranteed that rollover will happen and switches to live changelog
consumption without switching to xsync. This sleep is only when
history processing completed < CHANGELOG_ROLLOVER_TIME and sleep
only after the first turn, So will not affect the performance.

BUG: 1138952
Change-Id: Ida024211d312f60f0e8190805e7469b2165f00e1
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/8151
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar 
Tested-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/8639
Reviewed-by: Vijay Bellur

geo-rep: Avoid duplicate stat in xsync changelog processing

2014-07-09T08:52:24+00:00

When A file/dir is identified for metadata sync, it was doing
duplicate stat to get the metadata to sync.

With this patch it avoids doing one additional stat call. Xsync
performance will improve.

rsync will copy files metadata, so no need to include for
processing.

BUG: 1111490

Change-Id: I79dad6375fa4742d9aaca7d9856993c184a744dc
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/8124
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar 
Tested-by: Venky Shankar

geo-rep: History Change detector method select issue

2014-07-01T10:59:01+00:00

Geo-rep does history crawl even if change-detector is set
to xsync.

This patch fixes by taking priority to user configured
change detector.

BUG: 1113525
Change-Id: Ic6c34e187c9cb6608c9ef8a010ea07015ba60a80
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/8183
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Venky Shankar 
Tested-by: Venky Shankar

geo-rep: Making replica failover check interval configurable

2014-06-12T06:55:31+00:00

Replica failover check interval is hardcoded to 60 sec
by default. Now this option is made configurable and
defaulted to 1 sec.

To change the default value
gluster volume geo-replication  \
	:: config replica_failover_interval 15

Change-Id: Iada1b80d510452dcfedebd8a21bebd62394b0597
BUG: 1066410
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/8003
Reviewed-by: Venky Shankar 
Tested-by: Gluster Build System 
Tested-by: Venky Shankar

gsyncd / geo-rep: FSH recommended log locations

2014-06-10T16:55:55+00:00

Upgrading "working_dir" on the fly is a bit unclean yet
(though it works) as currently config upgrade does not
support "old" values to be expanded by using configuration
variables.

Change-Id: I44ed65c281f2e0ce3b6b467addc5c1c88ac674e7
BUG: 1077516
Signed-off-by: Venky Shankar 
Signed-off-by: Kotresh H R 
Signed-off-by: Aravinda VK 
Signed-off-by: Ajeet Jha 
Reviewed-on: http://review.gluster.org/7375
Tested-by: Gluster Build System

gsyncd / geo-rep: Xsync crawl metadata synchronization

2014-06-10T10:21:26+00:00

Added "metadata" record for directory and file creations
during the intial crawl.

Change-Id: I811ae26e0144cadf7249cb64541ec354ab83fe66
BUG: 1106604
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/8018
Reviewed-by: Aravinda VK 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

geo-rep: Changelog History API changes

2014-05-13T17:21:44+00:00

Additional argument added to API gf_history_changelog,
actual_end - The end time till where changelogs are available.

Added sort to history_get_changes API output.

BUG: 1091961
Change-Id: Id043409882a83cd0a7b9adc3d34d5147d17e532e
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/7747
Reviewed-by: ajeet jha 
Reviewed-by: Venky Shankar 
Tested-by: Venky Shankar

geo-rep: Pause and Resume feature for geo-replication

2014-05-09T07:27:40+00:00

Changelog consumption/processing now happens in seperate process
group than monitor. When monitor process group gets SIGSTOP all
worker process, ssh, rsync will be paused except the changelog
processing. When it gets SIGCONT it resumes its operation.

Changelog agent runs as RepceServer, geo-rep worker communicates
with changelog agent using RepceClient.

Change-Id: I35c333e4d8b13d03a7808aed601960eef23cfa04
BUG: 1093602
Signed-off-by: Venky Shankar 
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/7322

geo-rep: Changelog History consumption more fixes

2014-05-09T07:17:59+00:00

Number of parallel threads to process changelog history
is made configurable via sync_jobs

Change-Id: Idcd8e655d9df540cfa48648b9e98af941f95e9d0
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/7660
Reviewed-by: Venky Shankar 
Tested-by: Venky Shankar