glusterfs.git/geo-replication/syncdaemon/syncdutils.py, branch v4.1.7

geo-rep: Fix issues with gfid conflict handling

2018-08-16T04:21:14+00:00

1. MKDIR/RMDIR is recorded on all bricks. So if
   one brick succeeds creating it, other bricks
   should ignore it. But this was not happening.
   The fix rename of directories in hybrid crawl,
   was trying to rename the directory to itself
   and in the process crashing with ENOENT if the
   directory is removed.

2. If file is created, deleted and a directory is
   created with same name, it was failing to sync.
   Again the issue is around the fix for rename
   of directories in hybrid crawl. Fixed the same.

   If the same case was done with hardlink present
   for the file, it was failing. This patch fixes
   that too.

Backport of:
 > BUG: 1598884
 > Change-Id: I6f3bca44e194e415a3d4de3b9d03cc8976439284
 > Signed-off-by: Kotresh HR 

fixes: bz#1611114
Change-Id: I6f3bca44e194e415a3d4de3b9d03cc8976439284
Signed-off-by: Kotresh HR

geo-rep: Fix geo-rep for older versions of unshare

2018-08-16T04:20:47+00:00

Geo-rep mounts are private to worker. It uses
mount namespace using unshare command to achieve
the same. Well, the unshare command has to support
'--propagation' option. So geo-rep breaks on the
systems with older unshare version. The patch
makes it fall back to lazy umount behaviour if
the unshare does not support propagation option.

Backport of:
 > BUG: 1589782
 > Change-Id: Ia614f068aede288d63ac62fea4461b1865066054
 > Signed-off-by: Kotresh HR 

fixes: bz#1611111
Change-Id: Ia614f068aede288d63ac62fea4461b1865066054
Signed-off-by: Kotresh HR

geo-rep: Remove lazy umount and use mount namespaces

2018-02-22T05:40:35+00:00

Lazy umounting the master volume by worker causes
issues with rsync's usage of getcwd. Henc removing
the lazy umount and using private mount namespace
for the same. On the slave, the lazy umount is
retained as we can't use private namespace in non
root geo-rep setup.

Change-Id: I403375c02cb3cc7d257a5f72bbdb5118b4c8779a
BUG: 1546129
Signed-off-by: Kotresh HR

geo-rep: Support for using Volinfo from Conf file

2018-01-23T03:03:01+00:00

Once Geo-replication is started, it runs Gluster commands to get Volume
info from Master and Slave. With this patch, Georep can get Volume info
from Conf file if `--use-gconf-volinfo` argument is specified to monitor

Create a config(Or add to the config if exists) with following fields

    [vars]
    master-bricks=NODEID:HOSTNAME:PATH,..
    slave-bricks=NODEID:HOSTNAME,..
    master-volume-id=
    slave-volume-id=
    master-replica-count=
    master-disperse_count=

Note: Exising Geo-replication is not affected since this is activated
only when `--use-gconf-volinfo` is passed while spawning `gsyncd
monitor`

Tiering support is not yet added since Tiering + Glusterd2 is still
under discussion.

Fixes: #396
Change-Id: I281baccbad03686c00f6488a8511dd6db0edc57a
Signed-off-by: Aravinda VK

geo-rep: Log message improvements

2017-12-28T14:49:49+00:00

BUG: 1529480
Change-Id: If4775ed9886990c0e1bcf4e44c7dfef95cc4f0c3
Signed-off-by: Aravinda VK

fips/geo-rep: Replace MD5 with SHA256

2017-12-22T15:32:06+00:00

MD5 is not fips compliant. Hence replacing
with SHA256.

NOTE:
The hash is used to form the ctl_path for the ssh connection.
The length of ctl_path for ssh connection should not be > 108.
ssh fails with ctl_path too long if it is so. But when rsync
is piped to ssh, it is not taking > 90. rsync is failing with
error number 12. Hence using first 32 bytes of hash. Hash
collision doesn't matter as only one sock file is created
per directory.

Change-Id: I58aeb32a80b5422f6ac0188cf33fbecccbf08ae7
Updates: #230
Signed-off-by: Kotresh HR

geo-rep: Refactoring Config and Arguments parsing

2017-11-15T05:20:08+00:00

- Fixed Python pep8 issues
- Removed dead code
- Rewritten configuration management
- Rewritten Arguments/subcommands handling
- Added Args upgrade to accommodate all these changes without changing
  glusterd code
- use of md5 removed, which was used to hash the brick path for workdir

Both Master and Slave nodes will have subdir for session in the
format "__

  $GLUSTER_LOGDIR/geo-replication/__
  $GLUSTER_LOGDIR/geo-replication-slaves/__

Log file paths renamed since session info is available with directory
name itself.

  $LOG_DIR_MASTER/
      - gsyncd.log - Gsyncd, Worker monitor logs
      - mnt-.log - Aux mount logs, mounted by each worker
      - changes-.log - Changelog related logs(One per brick)

  $LOG_DIR_SLAVE/
      - gsyncd.log - Slave Gsyncd logs
      - mnt--.log - Aux mount logs,
        mounted for each connection from master-node:master-brick
      - mnt-mbr--.log - Same as above,
        but mountbroker setup

Fixes: #73
Change-Id: I2ec2a21e4e2a92fd92899d026e8543725276f021
Signed-off-by: Aravinda VK

geo-rep: Fix rename of directory in hybrid crawl

2017-11-10T05:36:22+00:00

In hybrid crawl, renames and unlink can't be
synced but directory renames can be detected.
While syncing the directory on slave, if the
gfid already exists, it should be rename.
Hence if directory gfid already exists, rename
it.

Change-Id: Ibf9f99e76a3e02795a3c2befd8cac48a5c365bb6
BUG: 1499566
Signed-off-by: Kotresh HR

geo-rep: Fix syncing of hardlink of symlink

2017-08-24T03:13:47+00:00

Problem:
If there is a hardlink to a symlink on master
and if the symlink file is deleted on master,
geo-rep fails to sync the hardlink.

Typical Usecase:
It's easily hit with rsnapshot use case where
it uses hardlinks.

Example Reproducer:
Setup geo-replication between master and slave
volume and in master mount point, do the following.

 1. mkdir /tmp/symlinkbug
 2. ln -f -s /does/not/exist /tmp/symlinkbug/a_symlink
 3. rsync -a /tmp/symlinkbug ./
 4. cp -al symlinkbug symlinkbug.0
 5. ln -f -s /does/not/exist2 /tmp/symlinkbug/a_symlink
 6. rsync -a /tmp/symlinkbug ./
 7. cp -al symlinkbug symlinkbug.1

Cause:
If the source was not present while syncing hardlink,
it was always packing the blob as regular file.

Fix:
If the source was not present while syncing hardlink,
pack the blob based on the mode.

Change-Id: Iaa12d6f99de47b18e0650e7c4eb455f23f8390f2
BUG: 1432046
Signed-off-by: Kotresh HR 
Reported-by: Christian Lohmaier 
Reviewed-on: https://review.gluster.org/18011
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

geo-rep: Fix changelog encoding to encode only space and newline

2017-07-21T04:44:30+00:00

libgfchangelog was encoding path using spec rfc3986, but encoding only
required for SPACE and NEWLINE chars since the NEWLINE char is used as
record separator and SPACE as field separator in the parsed changelogs
output.

Changed the encoding function to encode only SPACE and NEWLINE.

BUG: 1451724
Change-Id: I1936efad31788a9e636f912c832ed7d7efea4fe2
Signed-off-by: Aravinda VK 
Reviewed-on: https://review.gluster.org/17787
Reviewed-by: Prashanth Pai 
Reviewed-by: Kotresh HR 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System