glusterfs.git/tests, branch v6.6

test: fix suspicous non-root geo-rep test failures

2019-10-25T05:40:19+00:00

Export of env variable is required for ssh-copy-id command.

Backport of:

>    fixes: bz#1765426
>    Change-Id: Icaf7a848cb8f4ae9f887d885a8c5bb71f26633b4
>    Signed-off-by: Sunny Kumar 

fixes: bz#1765433
Change-Id: Icaf7a848cb8f4ae9f887d885a8c5bb71f26633b4
Signed-off-by: Sunny Kumar 
(cherry picked from commit bfebfa9f2ec9dfc5dbf4a68c3518f98364ebc461)

geo-rep: Fix Permission denied traceback on non root setup

2019-10-24T15:15:10+00:00

Problem:
While syncing rename of directory in hybrid crawl, geo-rep
crashes as below.

Traceback (most recent call last):
  File "/usr/local/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker
    res = getattr(self.obj, rmeth)(*in_data[2:])
  File "/usr/local/libexec/glusterfs/python/syncdaemon/resource.py", line 588, in entry_ops
    src_entry = get_slv_dir_path(slv_host, slv_volume, gfid)
  File "/usr/local/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, in get_slv_dir_path
    [ENOENT], [ESTALE])
  File "/usr/local/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap
    return call(*arg)
PermissionError: [Errno 13] Permission denied: '/bricks/brick1/b1/.glusterfs/8e/c0/8ec0fcd4-d50f-4a6e-b473-a7943ab66640'

Cause:
Conversion of gfid to path for a directory uses readlink on backend
.glusterfs gfid path. But this fails for non root user with
permission denied.

Fix:
Use gfid2path interface to get the path from gfid

Backport of:
 > Patch: https://review.gluster.org/23570
 > Change-Id: I9d40c713a1b32cea95144cbc0f384ada82972222
 > BUG: 1763439
 > Signed-off-by: Kotresh HR 

Change-Id: I9d40c713a1b32cea95144cbc0f384ada82972222
fixes: bz#1764183
Signed-off-by: Kotresh HR

cluster/ec: fix EIO error for concurrent writes on sparse files

2019-10-24T09:25:08+00:00

EC doesn't allow concurrent writes on overlapping areas, they are
serialized. However non-overlapping writes are serviced in parallel.
When a write is not aligned, EC first needs to read the entire chunk
from disk, apply the modified fragment and write it again.

The problem appears on sparse files because a write to an offset
implicitly creates data on offsets below it (so, in some way, they
are overlapping). For example, if a file is empty and we read 10 bytes
from offset 10, read() will return 0 bytes. Now, if we write one byte
at offset 1M and retry the same read, the system call will return 10
bytes (all containing 0's).

So if we have two writes, the first one at offset 10 and the second one
at offset 1M, EC will send both in parallel because they do not overlap.
However, the first one will try to read missing data from the first chunk
(i.e. offsets 0 to 9) to recombine the entire chunk and do the final write.
This read will happen in parallel with the write to 1M. What could happen
is that half of the bricks process the write before the read, and the
half do the read before the write. Some bricks will return 10 bytes of
data while the otherw will return 0 bytes (because the file on the brick
has not been expanded yet).

When EC tries to recombine the answers from the bricks, it can't, because
it needs more than half consistent answers to recover the data. So this
read fails with EIO error. This error is propagated to the parent write,
which is aborted and EIO is returned to the application.

The issue happened because EC assumed that a write to a given offset
implies that offsets below it exist.

This fix prevents the read of the chunk from bricks if the current size
of the file is smaller than the read chunk offset. This size is
correctly tracked, so this fixes the issue.

Also modifying ec-stripe.t file for Test #13 within it.
In this patch, if a file size is less than the offset we are writing, we
fill zeros in head and tail and do not consider it strip cache miss.
That actually make sense as we know what data that part holds and there is
no need of reading it from bricks.


Backport of:
 > Patch:https://review.gluster.org/#/c/glusterfs/+/23066/
 > Change-Id: Ic342e8c35c555b8534109e9314c9a0710b6225d6
 > BUG: 1730715
 > Signed-off-by: Xavi Hernandez 
 
(cherry picked from commit b01a43586c5abc23a874e5528a063c508f952cbd)

Change-Id: Ic342e8c35c555b8534109e9314c9a0710b6225d6
Fixes: bz#1739451
Signed-off-by: Xavi Hernandez

features/shard: Send correct size when reads are sent beyond file size

2019-10-24T09:24:22+00:00

Change-Id: I0cebaaf55c09eb1fb77a274268ff564e871b743b
fixes bz#1737141
Signed-off-by: Krutika Dhananjay 
(cherry picked from commit 51237eda7c4b3846d08c5d24d1e3fe9b7ffba1d4)

geo-rep: Fix config upgrade on non-participating node

2019-10-22T12:50:28+00:00

After upgrade, if the config files are of old format, it
gets migrated to new format. Monitor process migrates it.
Since monitor doesn't run on nodes where bricks are not
hosted, it doesn't get migrated there. So this patch fixes
the config upgrade on nodes which doesn't host bricks.
This happens during config either on get/set/reset.

Backport of:
 > Patch: https://review.gluster.org/23555
 > Change-Id: Ibade2f2310b0f3affea21a3baa1ae0eb71162cba
 > Signed-off-by: Kotresh HR 
 > BUG: 1762220

Change-Id: Ibade2f2310b0f3affea21a3baa1ae0eb71162cba
fixes: bz#1763028
Signed-off-by: Kotresh HR

tests : test case for non-root geo-rep setup

2019-10-22T12:50:27+00:00

Added test case for non-root geo-rep setup.

Backport of:
 > Patch: https://review.gluster.org/22902
 > Change-Id: Ib6ebee79949a9f61bdc5c7b5e11b51b262750e98
 > BUG: 1717827
 > Signed-off-by: Sunny Kumar 
 > Signed-off-by: Kotresh HR 

Change-Id: Ib6ebee79949a9f61bdc5c7b5e11b51b262750e98
fixes: bz#1764178
Signed-off-by: Kotresh HR

geo-rep: Test case for upgrading config file

2019-10-22T12:50:27+00:00

Added test case for the patch
https://review.gluster.org/#/c/glusterfs/+/22894/4

Also updated if else structure in gsyncdconfig.py to avoid
repeated occurance of values in new configfile.

Backport of:
 > Patch: https://review.gluster.org/22982
 > BUG: 1707731
 > Change-Id: If97e1d37ac52dbd17d47be6cb659fc5a3ccab6d7
 > Signed-off-by: Shwetha K Acharya 

fixes: bz#1764171
Change-Id: If97e1d37ac52dbd17d47be6cb659fc5a3ccab6d7
Signed-off-by: Kotresh HR

cluster/afr: Heal entries when there is a source & no healed_sinks

2019-10-17T10:52:54+00:00

Problem:
In a situation where B1 blames B2, B2 blames B1 and B3 doesn't blame
anything for entry heal, heal will not complete even though we have
clear source and sinks. This will happen because while doing
afr_selfheal_find_direction() only the bricks which are blamed by
non-accused bricks are considered as sinks. Later in
__afr_selfheal_entry_finalize_source() when it tries to mark all the
non-sources as sinks it fails to do so because there won't be any
healed_sinks marked, no witness present and there will be a source.

Fix:
If there is a source and no healed_sinks, then reset all the locked
sources to 0 and healed sinks to 1 to do conservative merge.

Change-Id: If40d8bc95d52a52b2730f55bdcf135109b421548
Fixes: bz#1760706
Signed-off-by: karthik-us

afr: support split-brain CLI for replica 3

2019-10-17T10:51:33+00:00

Ever since we added quorum checks for lookups in afr via commit
bd44d59741bb8c0f5d7a62c5b1094179dd0ce8a4, the split-brain resolution
commands would not work for replica 3 because there would be no
readables for the lookup fop.

The argument was that split-brains do not occur in replica 3 but we do
see (data/metadata) split-brain cases once in a while which indicate that there are
a few bugs/corner cases yet to be discovered and fixed.

Fortunately, commit  8016d51a3bbd410b0b927ed66be50a09574b7982 added
GF_CLIENT_PID_GLFS_HEALD as the pid for all fops made by glfsheal. If we
leverage this and allow lookups in afr when pid is GF_CLIENT_PID_GLFS_HEALD,
split-brain resolution commands will work for replica 3 volumes too.

Likewise, the check is added in shard_lookup as well to permit resolving
split-brains by specifying "/.shard/shard-file.xx" as the file name
(which previously used to fail with EPERM).

Change-Id: I3c543dea79caf7cfbc1633e9089cb1cdd2538ba9
Fixes: bz#1760792
Signed-off-by: Ravishankar N 
(cherry picked from commit 47dbd753187f69b3835d2e42fdbe7485874c4b3e)

ctime/rebalance: Heal ctime xattr on directory during rebalance

2019-09-27T11:34:25+00:00

After add-brick and rebalance, the ctime xattr is not present
on rebalanced directories on new brick. This patch fixes the
same.

Note that ctime still doesn't support consistent time across
distribute sub-volume.

This patch also fixes the in-memory inconsistency of time attributes
when metadata is self healed.

Backport of:
 > Patch: https://review.gluster.org/23127
 > Change-Id: Ia20506f1839021bf61d4753191e7dc34b31bb2df
 > BUG: 1734026
 > Signed-off-by: Kotresh HR 

Patch: https://review.gluster.org/23127
Change-Id: Ia20506f1839021bf61d4753191e7dc34b31bb2df
fixes: bz#1752413
Signed-off-by: Kotresh HR