glusterfs.git/geo-replication/syncdaemon, branch v5.1

geo-rep: Add more intelligence to automatic error handling

2018-11-09T18:46:54+00:00

Geo-rep's automatic error handling does gfid conflict
resolution. But if there are ENOENT errors because the
parent is not synced to slave, it doesn' handle them.
This patch adds the intelligence to create missing
parent directories on slave. It can create the missing
directories upto the depth of 10.

Backport of:

 > Patch: https://review.gluster.org/21498/
 > fixes: bz#1643402
 > Change-Id: Ic97ed1fa5899c087e404d559e04f7963ed7bb54c
 > Signed-off-by: Kotresh HR 
(cherry picked from commit 19775e0445411cca9ddd9d294fd54d0b6fbe6a03)

fixes: bz#1646896
Change-Id: Ic97ed1fa5899c087e404d559e04f7963ed7bb54c
Signed-off-by: Kotresh HR

georep: python2 to python3 compat - scheduler

2018-11-08T14:35:30+00:00

1. scheduler - Popen
2. syncdutils - corner case on failure

Backport of:
> Patch: https://review.gluster.org/21505
> BUG: 1643932
> Change-Id: I65af97a244a8790e976acedc2728db6ebbf2ae10
> Signed-off-by: Kotresh HR 
(cherry picked from commit 33e96100e17e9a293db6d63d9d5449d6c2d69376)

fixes: bz#1644514
Change-Id: I65af97a244a8790e976acedc2728db6ebbf2ae10
Signed-off-by: Kotresh HR

geo-rep: Fix issue in gfid-conflict-resolution

2018-11-08T14:33:17+00:00

Problem:
During gfid-conflict-resolution, geo-rep crashes
with 'ValueError: list.remove(x): x not in list'

Cause and Analysis:
During gfid-conflict-resolution, the entry blob is
passed back to master along with additional
information to verify it's integrity. If everything
looks fine, the entry creation is ignored and is
deleted from the original list.  But it is crashing
during removal of entry from the list saying entry
not in list. The reason is that the stat information
in the entry blob was modified and sent back to
master if present.

Fix:
Send back the correct stat information for
gfid-conflict-resolution.


Backport of:
> BUG: bz#1642865
> Change-Id: I47a6aa60b2a495465aa9314eebcb4085f0b1c4fd
> Signed-off-by: Kotresh HR 
(cherry picked from commit ff18121945bff394f3234e9f1a9d61ac97d4d493)

fixes: bz#1644158
Change-Id: I47a6aa60b2a495465aa9314eebcb4085f0b1c4fd
Signed-off-by: Kotresh HR

georep: python2 to python3 compat - syscalls

2018-10-10T14:09:10+00:00

1. ctypes/syscalls
    A) arguments is expected to be encoded
    B) Raw conversion of return value from bytearray into string
2. struct pack/unpack - Raw converstion of string to bytearray
3. basestring -> str

Updates: #411
Change-Id: I80f939adcdec0ed0022c87c0b76d057ad5559e5a
Signed-off-by: Kotresh HR 
(cherry picked from commit fb6e8d0d0ca21b16d331fa69da9b9dadf6c5c35d)

georep: python3 compatibility (Popen)

2018-10-05T14:22:54+00:00

The file objects for python3 by default is opened
in binary mode where as in python2 it's opened
as text by default.

The geo-rep code parses the output of Popen assuming
it as text, hence used the 'universal_newlines' flag
which provides backward compatibility for the same.

Change-Id: I371a03b6348af9666164cb2e8b93d47475431ad9
Updates: #411
Signed-off-by: Kotresh HR 
(cherry picked from commit 65aed1070cc2e44959cf3a0fbfde635de7e03103)

georep: Fix python3 compatibility (os.pipe)

2018-10-05T14:22:46+00:00

'os.pipe' returns pair of file descriptors
which are non-inheritable by child processes.
But geo-rep uses te inheritable nature of
pipe fds to communicate between parent and
child processes. Hence wrote a compatiable
pipe routine which works well both with python2
and python3 with inheritable nature.

Updates: #411
Change-Id: I869d7a52eeecdecf3851d44ed400e69b32a612d9
Signed-off-by: Kotresh HR 
(cherry picked from commit 173e89a6506bc8c727ce6d8e5ac84b59ad2e21de)

georep: python2 and python3 compat - bytes-str

2018-10-05T14:21:59+00:00

1. Fix fdopen used for pid file
2. Fix sha256 checksum calculation

Updates: #411
Change-Id: Ic173d104a73822c29aca260ba6de872cd8d23f86
Signed-off-by: Kotresh HR

georep: python2 to python3 compatibility-mount write

2018-10-05T14:21:27+00:00

python3 expects byte string for os.write. This works
for both py2 and py3. Fixed the same for geo-rep
mount testing code path.

Updates: #411
Change-Id: I2dfedcb0869457707bcca4d2847ef0d52bff1987
Signed-off-by: Kotresh HR

georep: python2 to python3 compat - pickle

2018-10-05T14:20:40+00:00

Handle py2 and py3 compatibility for pickling and unpickling.
Geo-rep pickles and unpickles reading sys.stdin and sys.stdout streams.
py2 and py3 compatibility expects the streams to be opened in binary
mode but the sys.stdout objects are different in python2
and python3

python2:
>>> type(sys.stdout)


python3:
>>> type(sys.stdout)


So in order to access binary stream, using sys.stdin.buffer in python3

Updates: #411
Change-Id: I1a633ccdddff5baf0cf05a8b493add39ddf75bd7
Signed-off-by: Kotresh HR

geo-rep: Fix issues related config set

2018-09-19T04:10:40+00:00

1. '--ignore-mising-args' option for rsync is not
   being used even though the rsync version is
   greater than 3.1.0. Fixed the same.

2. '--existing' option for rsync is also not being
   used. Fixed the same.

3. geo-rep config fails to set rsync-options as the
   value contains '--'. Interestingly, python argsparse
   treats the value with '--' (e.g., --ignore-missing-args)
   as option. But when passed with something like
   --value=--ignore-missing-args, it succeeds. Fixed the
   same.

Backport of:
 > Patch: https://review.gluster.org/21191
 > Change-Id: Iaeb838acaff1c2920fee9c7f920c99edce13a0a1
 > Signed-off-by: Kotresh HR 
 > BUG: 1629561
(cherry picked from commit b977b44dd0adfcd7a3b432844260de4b8d1c4adf)

Change-Id: Iaeb838acaff1c2920fee9c7f920c99edce13a0a1
Signed-off-by: Kotresh HR 
fixes: bz#1630673