glusterfs.git/geo-replication/syncdaemon/resource.py, branch v5.8

geo-rep: Fix syncing of files with non-ascii filenames

2018-12-26T16:50:01+00:00

Problem:
  Creation of files/directories with non-ascii names fails
  to sync to the slave. It crashes with below traceback on
  slave.
  ...
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py", line 118, in worker
    res = getattr(self.obj, rmeth)(*in_data[2:])
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", line 709, in entry_ops
    [ESTALE, EINVAL, EBUSY])
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap
    return call(*arg)
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", line 83, in lsetxattr
    cls.raise_oserr()
  File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", line 38, in raise_oserr
    raise OSError(errn, os.strerror(errn))
  OSError: [Errno 12] Cannot allocate memory

Cause:
  The length calculation arguments passed to blob creation was done before encoding. Hence
  was failing in gfid-access layer.

Fix:
  It appears that the calculating lenght properly fixes this issue. But it will cause
  issues in other places in 'python2' and not in 'python3'. So encoding and decoding
  each required string to make geo-rep compatible with both 'python2' and 'python3'
  is a nightmare and is not fool proof. Hence kept 'python2' code as is with out
  encode/decode and applied encode/decode only to 'python3'

Added non-ascii filename tests to regression

Backport of:
 > Patch: https://review.gluster.org/21668
 > BUG: 1650893
 > Change-Id: I35cfaf848e07b1a0b5cb93c01b98b472f08271a6
 > Signed-off-by: Kotresh HR 

fixes: bz#1648642
Change-Id: I35cfaf848e07b1a0b5cb93c01b98b472f08271a6
Signed-off-by: Kotresh HR

geo-rep: Fix traceback with symlink metadata sync

2018-12-12T12:54:37+00:00

While syncing metadata, 'os.chmod', 'os.chown',
'os.utime' should be used without de-reference.
But python supports only 'os.chown' without
de-reference. That's mostly because Linux
doesn't support 'chmod' on symlink file itself
but it does support 'chown'.

So while syncing metadata ops, if it's symlink
we should only sync 'chown' and not do 'chmod'
and 'utime'. It will lead to tracebacks with
errors like EROFS, EPERM, ACCESS, ENOENT.
All the three errors (EPERM, ACCESS, ENOENT)
were handled except EROFS. But the way it was
handled was not fool proof. The operation is
tried and failure was handled based on the errors.
All the errors with symlink file for 'chown',
'utime' had to be passed to safe errors list of
'errno_wrap'. This patch handles it better by
avoiding 'chmod' and 'utime' if it's symlink
file.
Backport of :
< Patch: https://review.gluster.org/21546/
> BUG: bz#1646104
> Change-Id: Ic354206455cdc7ab2a87d741d81f4efe1f19d77d
> Signed-off-by: Kotresh HR 
(cherry picked from commit 3c6cf9a4a1b46cab2dc53c1ee0afca0fe993102e)


fixes: bz#1654115
Change-Id: Ic354206455cdc7ab2a87d741d81f4efe1f19d77d
Signed-off-by: Kotresh HR

geo-rep: Fix issue in gfid-conflict-resolution

2018-11-08T14:33:17+00:00

Problem:
During gfid-conflict-resolution, geo-rep crashes
with 'ValueError: list.remove(x): x not in list'

Cause and Analysis:
During gfid-conflict-resolution, the entry blob is
passed back to master along with additional
information to verify it's integrity. If everything
looks fine, the entry creation is ignored and is
deleted from the original list.  But it is crashing
during removal of entry from the list saying entry
not in list. The reason is that the stat information
in the entry blob was modified and sent back to
master if present.

Fix:
Send back the correct stat information for
gfid-conflict-resolution.


Backport of:
> BUG: bz#1642865
> Change-Id: I47a6aa60b2a495465aa9314eebcb4085f0b1c4fd
> Signed-off-by: Kotresh HR 
(cherry picked from commit ff18121945bff394f3234e9f1a9d61ac97d4d493)

fixes: bz#1644158
Change-Id: I47a6aa60b2a495465aa9314eebcb4085f0b1c4fd
Signed-off-by: Kotresh HR

georep: python2 to python3 compat - syscalls

2018-10-10T14:09:10+00:00

1. ctypes/syscalls
    A) arguments is expected to be encoded
    B) Raw conversion of return value from bytearray into string
2. struct pack/unpack - Raw converstion of string to bytearray
3. basestring -> str

Updates: #411
Change-Id: I80f939adcdec0ed0022c87c0b76d057ad5559e5a
Signed-off-by: Kotresh HR 
(cherry picked from commit fb6e8d0d0ca21b16d331fa69da9b9dadf6c5c35d)

georep: python3 compatibility (Popen)

2018-10-05T14:22:54+00:00

The file objects for python3 by default is opened
in binary mode where as in python2 it's opened
as text by default.

The geo-rep code parses the output of Popen assuming
it as text, hence used the 'universal_newlines' flag
which provides backward compatibility for the same.

Change-Id: I371a03b6348af9666164cb2e8b93d47475431ad9
Updates: #411
Signed-off-by: Kotresh HR 
(cherry picked from commit 65aed1070cc2e44959cf3a0fbfde635de7e03103)

georep: Fix python3 compatibility (os.pipe)

2018-10-05T14:22:46+00:00

'os.pipe' returns pair of file descriptors
which are non-inheritable by child processes.
But geo-rep uses te inheritable nature of
pipe fds to communicate between parent and
child processes. Hence wrote a compatiable
pipe routine which works well both with python2
and python3 with inheritable nature.

Updates: #411
Change-Id: I869d7a52eeecdecf3851d44ed400e69b32a612d9
Signed-off-by: Kotresh HR 
(cherry picked from commit 173e89a6506bc8c727ce6d8e5ac84b59ad2e21de)

georep: python2 to python3 compatibility-mount write

2018-10-05T14:21:27+00:00

python3 expects byte string for os.write. This works
for both py2 and py3. Fixed the same for geo-rep
mount testing code path.

Updates: #411
Change-Id: I2dfedcb0869457707bcca4d2847ef0d52bff1987
Signed-off-by: Kotresh HR

All: run codespell on the code and fix issues.

2018-07-22T14:40:16+00:00

Please review, it's not always just the comments that were fixed.
I've had to revert of course all calls to creat() that were changed
to create() ...

Only compile-tested!

Change-Id: I7d02e82d9766e272a7fd9cc68e51901d69e5aab5
updates: bz#1193929
Signed-off-by: Yaniv Kaul

geo-rep: Fix issues with gfid conflict handling

2018-07-20T12:58:04+00:00

1. MKDIR/RMDIR is recorded on all bricks. So if
   one brick succeeds creating it, other bricks
   should ignore it. But this was not happening.
   The fix rename of directories in hybrid crawl,
   was trying to rename the directory to itself
   and in the process crashing with ENOENT if the
   directory is removed.

2. If file is created, deleted and a directory is
   created with same name, it was failing to sync.
   Again the issue is around the fix for rename
   of directories in hybrid crawl. Fixed the same.

   If the same case was done with hardlink present
   for the file, it was failing. This patch fixes
   that too.

fixes: bz#1598884
Change-Id: I6f3bca44e194e415a3d4de3b9d03cc8976439284
Signed-off-by: Kotresh HR

geo-rep: Fix symlink rename syncing issue

2018-07-12T08:33:08+00:00

Problem:
   Geo-rep sometimes fails to sync the rename of symlink
if the I/O is as follows

  1. touch file1
  2. ln -s "./file1" sym_400
  3. mv sym_400 renamed_sym_400
  4. mkdir sym_400

 The file 'renamed_sym_400' failed to sync to slave

Cause:
  Assume there are three distribute subvolume (brick1, brick2, brick3).
  The changelogs are recorded as follows for above I/O pattern.
  Note that the MKDIR is recorded on all bricks.

  1. brick1:
     -------

     CREATE file1
     SYMLINK sym_400
     RENAME sym_400 renamed_sym_400
     MKDIR sym_400

  2. brick2:
     -------

     MKDIR sym_400

  3. brick3:
     -------

     MKDIR sym_400

  The operations on 'brick1' should be processed sequentially. But
  since MKDIR is recorded on all the bricks, The brick 'brick2/brick3'
  processed MKDIR first before 'brick1' causing out of order syncing
  and created directory sym_400 first.

  Now 'brick1' processed it's changelog.

     CREATE file1 -> succeeds
     SYMLINK sym_400 -> No longer present in master. Ignored
     RENAME sym_400 renamed_sym_400
            While processing RENAME, if source('sym_400') doesn't
            present, destination('renamed_sym_400') is created. But
            geo-rep stats the name 'sym_400' to confirm source file's
            presence. In this race, since source name 'sym_400' is
            present as directory, it doesn't create destination.
            Hence RENAME is ignored.

Fix:
  The fix is not rely only on stat of source name during RENAME.
  It should stat the name and if the name is present, gfid should
  be same. Only then it can conclude the presence of source.

fixes: bz#1600405
Change-Id: I9fbec4f13ca6a182798a7f81b356fe2003aff969
Signed-off-by: Kotresh HR