glusterfs.git/tests, branch v6.7

test: fix non-root test case for geo-rep

2019-12-24T05:20:57+00:00

Problem:
On a freshly installed system non-root geo-rep test case gets blocked.

Solution:

On a freshly installed system, the remote key need to be accepted automatically by ssh-copy-id.

Credits: M. Scherer 

Backport of:

>    Change-Id: I5077f99a6681660f7e3e84c25ef216f521b7c29c
>    Fixes: bz#1779742
>    Signed-off-by: Sunny Kumar 

Change-Id: I5077f99a6681660f7e3e84c25ef216f521b7c29c
Fixes: bz#1784796
Signed-off-by: Sunny Kumar

cluster/ec: Prevent double pre-op xattrops

2019-10-30T08:18:18+00:00

Problem:
Race:
Thread-1                                    Thread-2
1) Does ec_get_size_version() to perform
pre-op fxattrop as part of write-1
                                           2) Calls ec_set_dirty_flag() in
                                              ec_get_size_version() for write-2.
					      This sets dirty[] to 1
3) Completes executing
ec_prepare_update_cbk leading to
ctx->dirty[] = '1'
					   4) Takes LOCK(inode->lock) to check if there are
					      any flags and sets dirty-flag because
				              lock->waiting_flag is 0 now. This leads to
					      fxattrop to increment on-disk dirty[] to '2'

At the end of the writes the file will be marked for heal even when it doesn't need heal.

Fix:
Perform ec_set_dirty_flag() and other checks inside LOCK() to prevent dirty[] to be marked
as '1' in step 2) above

Updates bz#1739446
Change-Id: Icac2ab39c0b1e7e154387800fbededc561612865
Signed-off-by: Pranith Kumar K

test: fix suspicous non-root geo-rep test failures

2019-10-25T05:40:19+00:00

Export of env variable is required for ssh-copy-id command.

Backport of:

>    fixes: bz#1765426
>    Change-Id: Icaf7a848cb8f4ae9f887d885a8c5bb71f26633b4
>    Signed-off-by: Sunny Kumar 

fixes: bz#1765433
Change-Id: Icaf7a848cb8f4ae9f887d885a8c5bb71f26633b4
Signed-off-by: Sunny Kumar 
(cherry picked from commit bfebfa9f2ec9dfc5dbf4a68c3518f98364ebc461)

geo-rep: Fix Permission denied traceback on non root setup

2019-10-24T15:15:10+00:00

Problem:
While syncing rename of directory in hybrid crawl, geo-rep
crashes as below.

Traceback (most recent call last):
  File "/usr/local/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker
    res = getattr(self.obj, rmeth)(*in_data[2:])
  File "/usr/local/libexec/glusterfs/python/syncdaemon/resource.py", line 588, in entry_ops
    src_entry = get_slv_dir_path(slv_host, slv_volume, gfid)
  File "/usr/local/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, in get_slv_dir_path
    [ENOENT], [ESTALE])
  File "/usr/local/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap
    return call(*arg)
PermissionError: [Errno 13] Permission denied: '/bricks/brick1/b1/.glusterfs/8e/c0/8ec0fcd4-d50f-4a6e-b473-a7943ab66640'

Cause:
Conversion of gfid to path for a directory uses readlink on backend
.glusterfs gfid path. But this fails for non root user with
permission denied.

Fix:
Use gfid2path interface to get the path from gfid

Backport of:
 > Patch: https://review.gluster.org/23570
 > Change-Id: I9d40c713a1b32cea95144cbc0f384ada82972222
 > BUG: 1763439
 > Signed-off-by: Kotresh HR 

Change-Id: I9d40c713a1b32cea95144cbc0f384ada82972222
fixes: bz#1764183
Signed-off-by: Kotresh HR

cluster/ec: fix EIO error for concurrent writes on sparse files

2019-10-24T09:25:08+00:00

EC doesn't allow concurrent writes on overlapping areas, they are
serialized. However non-overlapping writes are serviced in parallel.
When a write is not aligned, EC first needs to read the entire chunk
from disk, apply the modified fragment and write it again.

The problem appears on sparse files because a write to an offset
implicitly creates data on offsets below it (so, in some way, they
are overlapping). For example, if a file is empty and we read 10 bytes
from offset 10, read() will return 0 bytes. Now, if we write one byte
at offset 1M and retry the same read, the system call will return 10
bytes (all containing 0's).

So if we have two writes, the first one at offset 10 and the second one
at offset 1M, EC will send both in parallel because they do not overlap.
However, the first one will try to read missing data from the first chunk
(i.e. offsets 0 to 9) to recombine the entire chunk and do the final write.
This read will happen in parallel with the write to 1M. What could happen
is that half of the bricks process the write before the read, and the
half do the read before the write. Some bricks will return 10 bytes of
data while the otherw will return 0 bytes (because the file on the brick
has not been expanded yet).

When EC tries to recombine the answers from the bricks, it can't, because
it needs more than half consistent answers to recover the data. So this
read fails with EIO error. This error is propagated to the parent write,
which is aborted and EIO is returned to the application.

The issue happened because EC assumed that a write to a given offset
implies that offsets below it exist.

This fix prevents the read of the chunk from bricks if the current size
of the file is smaller than the read chunk offset. This size is
correctly tracked, so this fixes the issue.

Also modifying ec-stripe.t file for Test #13 within it.
In this patch, if a file size is less than the offset we are writing, we
fill zeros in head and tail and do not consider it strip cache miss.
That actually make sense as we know what data that part holds and there is
no need of reading it from bricks.


Backport of:
 > Patch:https://review.gluster.org/#/c/glusterfs/+/23066/
 > Change-Id: Ic342e8c35c555b8534109e9314c9a0710b6225d6
 > BUG: 1730715
 > Signed-off-by: Xavi Hernandez 
 
(cherry picked from commit b01a43586c5abc23a874e5528a063c508f952cbd)

Change-Id: Ic342e8c35c555b8534109e9314c9a0710b6225d6
Fixes: bz#1739451
Signed-off-by: Xavi Hernandez

features/shard: Send correct size when reads are sent beyond file size

2019-10-24T09:24:22+00:00

Change-Id: I0cebaaf55c09eb1fb77a274268ff564e871b743b
fixes bz#1737141
Signed-off-by: Krutika Dhananjay 
(cherry picked from commit 51237eda7c4b3846d08c5d24d1e3fe9b7ffba1d4)

geo-rep: Fix config upgrade on non-participating node

2019-10-22T12:50:28+00:00

After upgrade, if the config files are of old format, it
gets migrated to new format. Monitor process migrates it.
Since monitor doesn't run on nodes where bricks are not
hosted, it doesn't get migrated there. So this patch fixes
the config upgrade on nodes which doesn't host bricks.
This happens during config either on get/set/reset.

Backport of:
 > Patch: https://review.gluster.org/23555
 > Change-Id: Ibade2f2310b0f3affea21a3baa1ae0eb71162cba
 > Signed-off-by: Kotresh HR 
 > BUG: 1762220

Change-Id: Ibade2f2310b0f3affea21a3baa1ae0eb71162cba
fixes: bz#1763028
Signed-off-by: Kotresh HR

tests : test case for non-root geo-rep setup

2019-10-22T12:50:27+00:00

Added test case for non-root geo-rep setup.

Backport of:
 > Patch: https://review.gluster.org/22902
 > Change-Id: Ib6ebee79949a9f61bdc5c7b5e11b51b262750e98
 > BUG: 1717827
 > Signed-off-by: Sunny Kumar 
 > Signed-off-by: Kotresh HR 

Change-Id: Ib6ebee79949a9f61bdc5c7b5e11b51b262750e98
fixes: bz#1764178
Signed-off-by: Kotresh HR

geo-rep: Test case for upgrading config file

2019-10-22T12:50:27+00:00

Added test case for the patch
https://review.gluster.org/#/c/glusterfs/+/22894/4

Also updated if else structure in gsyncdconfig.py to avoid
repeated occurance of values in new configfile.

Backport of:
 > Patch: https://review.gluster.org/22982
 > BUG: 1707731
 > Change-Id: If97e1d37ac52dbd17d47be6cb659fc5a3ccab6d7
 > Signed-off-by: Shwetha K Acharya 

fixes: bz#1764171
Change-Id: If97e1d37ac52dbd17d47be6cb659fc5a3ccab6d7
Signed-off-by: Kotresh HR

cluster/afr: Heal entries when there is a source & no healed_sinks

2019-10-17T10:52:54+00:00

Problem:
In a situation where B1 blames B2, B2 blames B1 and B3 doesn't blame
anything for entry heal, heal will not complete even though we have
clear source and sinks. This will happen because while doing
afr_selfheal_find_direction() only the bricks which are blamed by
non-accused bricks are considered as sinks. Later in
__afr_selfheal_entry_finalize_source() when it tries to mark all the
non-sources as sinks it fails to do so because there won't be any
healed_sinks marked, no witness present and there will be a source.

Fix:
If there is a source and no healed_sinks, then reset all the locked
sources to 0 and healed sinks to 1 to do conservative merge.

Change-Id: If40d8bc95d52a52b2730f55bdcf135109b421548
Fixes: bz#1760706
Signed-off-by: karthik-us