glusterfs.git/geo-replication/syncdaemon/syncdutils.py, branch v3.12.5

geo-rep: Fix syncing of hardlink of symlink

2017-08-29T12:42:31+00:00

Problem:
If there is a hardlink to a symlink on master
and if the symlink file is deleted on master,
geo-rep fails to sync the hardlink.

Typical Usecase:
It's easily hit with rsnapshot use case where
it uses hardlinks.

Example Reproducer:
Setup geo-replication between master and slave
volume and in master mount point, do the following.

 1. mkdir /tmp/symlinkbug
 2. ln -f -s /does/not/exist /tmp/symlinkbug/a_symlink
 3. rsync -a /tmp/symlinkbug ./
 4. cp -al symlinkbug symlinkbug.0
 5. ln -f -s /does/not/exist2 /tmp/symlinkbug/a_symlink
 6. rsync -a /tmp/symlinkbug ./
 7. cp -al symlinkbug symlinkbug.1

Cause:
If the source was not present while syncing hardlink,
it was always packing the blob as regular file.

Fix:
If the source was not present while syncing hardlink,
pack the blob based on the mode.

> Change-Id: Iaa12d6f99de47b18e0650e7c4eb455f23f8390f2
> BUG: 1432046
> Signed-off-by: Kotresh HR 
> Reported-by: Christian Lohmaier 
> Reviewed-on: https://review.gluster.org/18011
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Aravinda VK 
(cherry picked from commit e893962deaabab8e934813f8a0443a8f94e009f2)

Change-Id: Iaa12d6f99de47b18e0650e7c4eb455f23f8390f2
BUG: 1486120
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/18127
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

geo-rep: Fix changelog encoding to encode only space and newline

2017-07-21T04:44:30+00:00

libgfchangelog was encoding path using spec rfc3986, but encoding only
required for SPACE and NEWLINE chars since the NEWLINE char is used as
record separator and SPACE as field separator in the parsed changelogs
output.

Changed the encoding function to encode only SPACE and NEWLINE.

BUG: 1451724
Change-Id: I1936efad31788a9e636f912c832ed7d7efea4fe2
Signed-off-by: Aravinda VK 
Reviewed-on: https://review.gluster.org/17787
Reviewed-by: Prashanth Pai 
Reviewed-by: Kotresh HR 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

geo-rep: Structured log support

2017-06-20T06:00:47+00:00

Changed all log messages to structured log format

Change-Id: Idae25f8b4ad0bbae38f4362cbda7bbf51ce7607b
Updates: #240
Signed-off-by: Aravinda VK 
Reviewed-on: https://review.gluster.org/17551
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep: Rsync tunables for performance improvements

2017-05-23T11:15:47+00:00

Flag: --ignore-missing-args
This Rsync flag reduces sync failures if the source file is
unlinked but present in --files-from list. This reduces
Rsync retries in Geo-rep and improves the performance

Flag: --existing
Rsync in Geo-rep never creates target files. Using RPC Geo-rep creates
entry in Slave and rsync --inplace used to prevent creating temporary file
and rename.(To avoid different GFID in Slave). If the entry is missing in
Slave then Geo-rep Rsync gets Permission denied errors when it tries to
create file with name as GFID inside .gfid dir.(Geo-rep rsync syncs data
using GFIDS with aux-gfid-mount)

To disable these flags,

    gluster volume geo-replication  config \
        rsync-opt-ignore-missing-args false
    gluster volume geo-replication  config \
        rsync-opt-existing false

Thanks Kotresh for finding these awesome tunables.

BUG: 1400924
Change-Id: I6a84fb86a589bf6edc8dfd1086456a84b05a64fc
Signed-off-by: Aravinda VK 
Reviewed-on: https://review.gluster.org/16010
Smoke: Gluster Build System 
Reviewed-by: Amar Tumballi 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep: Fix mount cleanup

2017-04-27T05:58:51+00:00

On corner cases, mount cleanup might cause
worker crash. Fixing the same.

Change-Id: I38c0af51d10673765cdb37bc5b17bb37efd043b8
BUG: 1433506
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/17015
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

geo-rep: Fix EBUSY traceback

2017-04-10T06:24:37+00:00

EBUSY was added to retry list of errno_wrap
without importing. Fixing the same.

Change-Id: Ide81a9ccc9b948a96265b6890da078b722b45d51
BUG: 1434018
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/17011
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Aravinda VK

geo-rep: Retry on EBUSY

2017-04-07T06:08:31+00:00

Do not crash on EBUSY error. Add EBUSY
retry errno list. Crash only if the error
persists even after max retries.

Change-Id: Ia067ccc6547731f28f2a315d400705e616cbf662
BUG: 1434018
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/16924
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Aravinda VK 
CentOS-regression: Gluster Build System

geo-rep: Optionally allow access to mounts

2017-03-20T03:33:36+00:00

In order to improve debuggability, it is important
to have access to geo-rep master and slave mounts.
With the default behaviour, geo-rep lazy unmounts
the mounts after changing the current working
directory into the mount point. It also cleans
up the mount points. So only geo-rep worker has
the access and it becomes impossible to take the
client profile info and do any other client statck
analysis. Hence the following new config is being
introduced to allow access to mounts.

gluster vol geo-rep  :: \
config access_mount true

The default value of 'access_mount' is false.

Change-Id: I53dce4ea86a6ffc979c82f9330e8954327180ca3
BUG: 1433506
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/16912
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Vijay Bellur

geo-rep: Use Host UUID to find local Gluster node

2016-12-14T06:38:14+00:00

To spawn workers for each local brick, Geo-rep was collecting all
the machine IPs based on hostname and finds based on the connectivity.

With this patch, Geo-rep finds local brick if host UUID matches with
UUID of the brick from Volume info.

BUG: 1401801
Change-Id: Ic83c65df89e43cb86346e3ede227aa84d17ffd79
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/16035
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep/eventsapi: Additional Events

2016-10-18T10:56:03+00:00

Added following events
EVENT_GEOREP_ACTIVE
    {
        "nodeid": NODEID,
        "ts": TIMESTAMP,
        "event": "GEOREP_ACTIVE",
        "message": {
            "master_volume": MASTER_VOLUME_NAME,
            "slave_host": SLAVE_HOST,
            "slave_volume": SLAVE_VOLUME,
            "brick_path": BRICK_PATH
        }
    }

EVENT_GEOREP_PASSIVE
    {
        "nodeid": NODEID,
        "ts": TIMESTAMP,
        "event": "GEOREP_PASSIVE",
        "message": {
            "master_volume": MASTER_VOLUME_NAME,
            "slave_host": SLAVE_HOST,
            "slave_volume": SLAVE_VOLUME,
            "brick_path": BRICK_PATH
        }
    }
EVENT_GEOREP_CHECKPOINT_COMPLETED
    {
        "nodeid": NODEID,
        "ts": TIMESTAMP,
        "event": "GEOREP_ACTIVE",
        "message": {
            "master_volume": MASTER_VOLUME_NAME,
            "slave_host": SLAVE_HOST,
            "slave_volume": SLAVE_VOLUME,
            "brick_path": BRICK_PATH,
            "checkpoint_time": CHECKPOINT_TIME,
            "checkpoint_completion_time": CHECKPOINT_COMPLETION_TIME
        }
    }

BUG: 1379330
Change-Id: I90716175868c59dd65c8d202e73e0ede90347b6a
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15630
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR 
Tested-by: Kotresh HR