glusterfs.git/geo-replication, branch v6.7

geo-rep: Fix py2/py3 compatibility in repce

2019-12-24T05:22:21+00:00

Geo-rep fails to start on python2 only machine like
centos6. It fails with "ImportError no module named _io".
This patch fixes the same.

Backport of:
 > Patch: https://review.gluster.org//23702/
 > BUG: 1771577
 > Change-Id: I8228458a853a230546f9faf29a0e9e0f23b3efec
 > Signed-off-by: Kotresh HR 
(cherry picked from commit 9595ecca3de49fdf37d30b151f5c3e071e0a80d0)

fixes: bz#1771842
Change-Id: I8228458a853a230546f9faf29a0e9e0f23b3efec
Signed-off-by: Kotresh HR

libgfchangelog : use find_library to locate shared library

2019-11-08T06:30:36+00:00

Issue:

libgfchangelog.so: cannot open shared object file

Due to hardcoded shared library name runtime loader looks for particular version of
a shared library.

Solution:

Using find_library to locate shared library at runtime solves this issue.

Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 323, in main
    func(args)
  File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 82, in subcmd_worker
    local.service_loop(remote)
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1261, in service_loop
    changelog_agent.init()
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__
    return self.ins(self.meth, *a)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__
    raise res
OSError: libgfchangelog.so: cannot open shared object file: No such file or directory

Backport of:
 > Patch: https://review.gluster.org/22557
 > Change-Id: I3dd013d701ed1cd99ba7ef20d1898f343e1db8f5
 > BUG: 1699394
 > Signed-off-by: Sunny Kumar 
(cherry picked from commit f316c8b797283818bd800569771870a4b9bf1310)

Change-Id: I3dd013d701ed1cd99ba7ef20d1898f343e1db8f5
fixes: bz#1770100
Signed-off-by: Sunny Kumar

geo-rep: Fix Permission denied traceback on non root setup

2019-10-24T15:15:10+00:00

Problem:
While syncing rename of directory in hybrid crawl, geo-rep
crashes as below.

Traceback (most recent call last):
  File "/usr/local/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker
    res = getattr(self.obj, rmeth)(*in_data[2:])
  File "/usr/local/libexec/glusterfs/python/syncdaemon/resource.py", line 588, in entry_ops
    src_entry = get_slv_dir_path(slv_host, slv_volume, gfid)
  File "/usr/local/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, in get_slv_dir_path
    [ENOENT], [ESTALE])
  File "/usr/local/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap
    return call(*arg)
PermissionError: [Errno 13] Permission denied: '/bricks/brick1/b1/.glusterfs/8e/c0/8ec0fcd4-d50f-4a6e-b473-a7943ab66640'

Cause:
Conversion of gfid to path for a directory uses readlink on backend
.glusterfs gfid path. But this fails for non root user with
permission denied.

Fix:
Use gfid2path interface to get the path from gfid

Backport of:
 > Patch: https://review.gluster.org/23570
 > Change-Id: I9d40c713a1b32cea95144cbc0f384ada82972222
 > BUG: 1763439
 > Signed-off-by: Kotresh HR 

Change-Id: I9d40c713a1b32cea95144cbc0f384ada82972222
fixes: bz#1764183
Signed-off-by: Kotresh HR

geo-rep: Fix config upgrade on non-participating node

2019-10-22T12:50:28+00:00

After upgrade, if the config files are of old format, it
gets migrated to new format. Monitor process migrates it.
Since monitor doesn't run on nodes where bricks are not
hosted, it doesn't get migrated there. So this patch fixes
the config upgrade on nodes which doesn't host bricks.
This happens during config either on get/set/reset.

Backport of:
 > Patch: https://review.gluster.org/23555
 > Change-Id: Ibade2f2310b0f3affea21a3baa1ae0eb71162cba
 > Signed-off-by: Kotresh HR 
 > BUG: 1762220

Change-Id: Ibade2f2310b0f3affea21a3baa1ae0eb71162cba
fixes: bz#1763028
Signed-off-by: Kotresh HR

geo-rep: Fix the name of changelog archive file

2019-10-22T12:50:27+00:00

Background:
The processed changelogs are archived each month in a single tar file.
The default format is "archive_YYYYMM.tar" which is specified as "%%Y%%m"
in configuration file.

Problem:
The created changelog archive file didn't have corresponding year
and month. It created as "archive_%Y%m.tar" on python2 only systems.

Cause and Fix:
Geo-rep expects "%Y%m" after the ConfigParser reads it from config file.
Since it was "%%Y%%m" in config file, geo-rep used to get correct value
"%Y%m" in python3 and "%%Y%%m" in python2 which is incorrect.
The fix can be to use "%Y%m" in config file but that fails in python3.
So the fix is to use "RawConfigParser" in geo-rep and use "%Y%m". This
works both in python2 and python3.

Backport of:
 > Patch: https://review.gluster.org/23248
 > Change-Id: Ie5b7d2bc04d0d53cd1769e064c2d67aaf95d557c
 > BUG: 1741890
 > Signed-off-by: Kotresh HR 

Change-Id: Ie5b7d2bc04d0d53cd1769e064c2d67aaf95d557c
fixes: bz#1764176
Signed-off-by: Kotresh HR

geo-rep: Fix Config Get Race

2019-10-22T12:50:27+00:00

When two threads(sync jobs) in Geo-rep worker calls `gconf.get` and
`gconf.getr`(realtime) at the sametime, `getr` resets the conf object
and other one gets None. Thread Lock is introduced to fix the issue.

```
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py",
  line 368, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1987,
  in syncjob
    po = self.sync_engine(pb, self.log_err)
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py",
  line 1444, in rsync
    rconf.ssh_ctl_args + \
AttributeError: 'NoneType' object has no attribute 'split'
```

Backport of:
 > Patch: https://review.gluster.org/23158
 > Change-Id: I9c245e5c36338265354e158f5baa32b119eb2da5
 > BUG: 1737484
 > Signed-off-by: Aravinda VK 

Change-Id: I9c245e5c36338265354e158f5baa32b119eb2da5
fixes: bz#1764174
Signed-off-by: Kotresh HR

geo-rep: Test case for upgrading config file

2019-10-22T12:50:27+00:00

Added test case for the patch
https://review.gluster.org/#/c/glusterfs/+/22894/4

Also updated if else structure in gsyncdconfig.py to avoid
repeated occurance of values in new configfile.

Backport of:
 > Patch: https://review.gluster.org/22982
 > BUG: 1707731
 > Change-Id: If97e1d37ac52dbd17d47be6cb659fc5a3ccab6d7
 > Signed-off-by: Shwetha K Acharya 

fixes: bz#1764171
Change-Id: If97e1d37ac52dbd17d47be6cb659fc5a3ccab6d7
Signed-off-by: Kotresh HR

geo-rep : fix gluster command path for non-root session

2019-10-22T12:50:27+00:00

Problem:
gluster command not found.

Cause:
In Volinfo class we issue command 'gluster vol info' to get information
about volume like getting brick_root to perform various operation.
When geo-rep session is configured for non-root user Volinfo class
fails to issue gluster command due to unavailability of gluster
binary path for non-root user.

Solution:
Use config value 'slave-gluster-command-dir'/'gluster-command-dir' to get path
for gluster command based on caller.

Backport of:
 > Patch: https://review.gluster.org/22920
 > BUG: 1722740
 > Change-Id: I4ec46373da01f5d00ecd160c4e8c6239da8b3859
 > Signed-off-by: Sunny Kumar 

fixes: bz#1764172
Change-Id: I4ec46373da01f5d00ecd160c4e8c6239da8b3859
Signed-off-by: Kotresh HR

geo-rep: Upgrading config file to new version

2019-10-22T12:50:27+00:00

- configuration handling is enhanced with patch
https://review.gluster.org/#/c/glusterfs/+/18257/
- hence, the old configurations are not applied when
Geo-rep session is created in the old version and upgraded.

This patch solves the issue. It,
- checks if the config file is old.
- parses required values from old config file and stores in new
  config file, which ensures that configerations are applied on
  upgrade.
- stores old config file as backup.
- handles changes in options introduced in
  https://review.gluster.org/#/c/glusterfs/+/18257/

Backport of:
 > Patch: https://review.gluster.org/22894
 > BUG: bz#1707731
 > Change-Id: Iad8da6c1e1ae8ecf7c84dfdf8ea3ac6966d8a2a0
 > Signed-off-by: Shwetha K Acharya 

updates: bz#1764171
Change-Id: Iad8da6c1e1ae8ecf7c84dfdf8ea3ac6966d8a2a0
Signed-off-by: Kotresh HR

geo-rep : fix mountbroker setup

2019-09-27T11:32:48+00:00

Problem:

Unable to setup mountbroker root directory while creating geo-replication
session for non-root user.

Casue:
With patch[1] which defines the max-port for glusterd one extra sapce
got added in field of 'option max-port'.
[1]. https://review.gluster.org/#/c/glusterfs/+/21872/

In geo-rep spliting of key-value pair form vol file was done on the
basis of space so this additional space caused "ValueError: too many values
to unpack".

Solution:
Use split so that it can treat consecutive whitespace as a single separator.

Backport of https://review.gluster.org/#/c/glusterfs/+/22716/

>Fixes: bz#1709248
>Change-Id: Ia22070a43f95d66d84cb35487f23f9ee58b68c73
>Signed-off-by: Sunny Kumar 
>(cherry picked from commit 3dd03146bb7037ae2ebea0579d0b81be27fdd927)

Fixes: bz#1750228
Signed-off-by: Sunny Kumar 
Change-Id: Ia22070a43f95d66d84cb35487f23f9ee58b68c73