glusterfs.git/tests/include.rc, branch v3.11.3

cluster/ec: Non-disruptive upgrade on EC volume fails

2017-07-19T11:26:21+00:00

Problem:
Enabling optimistic changelog on EC volume was not
handling node down scenarios appropriately resulting
in volume data inaccessibility.

Solution:
Update dirty xattr appropriately on good bricks whenever
nodes are down. This would fix the metadata information
as part of heal and thus ensures data accessibility.

>BUG: 1468261
>Change-Id: I08b0d28df386d9b2b49c3de84b4aac1c729ac057
>Signed-off-by: Sunil Kumar Acharya 
>Reviewed-on: https://review.gluster.org/17703
>Smoke: Gluster Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Pranith Kumar Karampuri 

BUG: 1470938
Change-Id: I08b0d28df386d9b2b49c3de84b4aac1c729ac057
Signed-off-by: Sunil Kumar Acharya 
Reviewed-on: https://review.gluster.org/17773
Smoke: Gluster Build System 
Reviewed-by: Ashish Pandey 
Reviewed-by: Pranith Kumar Karampuri 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

cluster/ec: Update xattr and heal size properly

2017-06-07T13:29:39+00:00

Problem-1 : Recursive healing of same file is happening
when IO is going on even after data heal completes.

Solution:
RCA: At the end of the write, when ec_update_size_version
gets called, we send it only on good bricks and not
on healing brick. Due to this, xattr on healing brick
will always remain out of sync and when the background
heal check source and sink, it finds this brick to be
healed and start healing from scratch. That involve
ftruncate and writing all of the data again.

To solve this, send xattrop on all the good bricks as
well as healing bricks.

Problem-2: The above fix exposes the data corruption
during heal. If the write on a file is going on and
heal finishes, we find that the file gets corrupted.

RCA:
The real problem happens in ec_rebuild_data(). Here we receive the
'size' argument which contains the real file size at the time of
starting self-heal and it's assigned to heal->total_size.

After that, a sequence of calls to ec_sync_heal_block() are done. Each
call ends up calling ec_manager_heal_block(), which does the actual work
of healing a block.

First a lock on the inode is taken in state EC_STATE_INIT using
ec_heal_inodelk(). When the lock is acquired, ec_heal_lock_cbk() is
called. This function calls ec_set_inode_size() to store the real size
of the inode (it uses heal->total_size).

The next step is to read the block to be healed. This is done using a
regular ec_readv(). One of the things this call does is to trim the
returned size if the file is smaller than the requested size.

In our case, when we read the last block of a file whose size was = 512
mod 1024 at the time of starting self-heal, ec_readv() will return only
the first 512 bytes, not the whole 1024 bytes.

This isn't a problem since the following ec_writev() sent from the heal
code only attempts to write the amount of data read, so it shouldn't
modify the remaining 512 bytes.

However ec_writev() also checks the file size. If we are writing the
last block of the file (determined by the size stored on the inode that
we have set to heal->total_size), any data beyond the (imposed) end of
file will be cleared with 0's. This causes the 512 bytes after the
heal->total_size to be cleared. Since the file was written after heal
started, the these bytes contained data, so the block written to the
damaged brick will be incorrect.

Solution:
Align heal->total_size to a multiple of the stripe size.

Thanks "Xavier Hernandez" 
to find out the root cause and to fix the issue.

>Change-Id: I6c9f37b3ff9dd7f5dc1858ad6f9845c05b4e204e
>BUG: 1428673
>Signed-off-by: Ashish Pandey 
>Reviewed-on: https://review.gluster.org/16985
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Pranith Kumar Karampuri 
>Reviewed-by: Xavier Hernandez 
>Signed-off-by: Ashish Pandey 

Change-Id: I6c9f37b3ff9dd7f5dc1858ad6f9845c05b4e204e
BUG: 1459392
Signed-off-by: Ashish Pandey 
Reviewed-on: https://review.gluster.org/17482
Smoke: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri 
Tested-by: Pranith Kumar Karampuri 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

glusterd: Make reset-brick work correctly if brick-mux is on

2017-05-16T00:29:48+00:00

Reset brick currently kills of the corresponding brick process.
However, with brick multiplexing enabled, stopping the brick
process would render all bricks attached to it unavailable. To
handle this correctly, we need to make sure that the brick process
is terminated only if brick-multiplexing is disabled. Otherwise,
we should send the GLUSTERD_BRICK_TERMINATE rpc to the respective
brick process to detach the brick that is to be reset.

> Reviewed-on: https://review.gluster.org/17128
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Atin Mukherjee 

(cherry picked from commit 74383e3ec6f8244b3de9bf14016452498c1ddcf0)

Change-Id: I69002d66ffe6ec36ef48af09b66c522c6d35ac58
BUG: 1449933
Signed-off-by: Samikshan Bairagya 
Reviewed-on: https://review.gluster.org/17245
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee

tests: track EW_RETRIES for debugging

2017-04-09T22:11:05+00:00

It can often be useful while debugging to know how many times
EXPECT_WITHIN had to retry a command before it got the result we were
looking for.  This patch just adds a variable EW_RETRIES that can be
inspected to find this info for the last EXPECT_WITHIN.

Change-Id: I1bcb09bb7eb118c3d76c60317ef99e02df6b6ee6
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/16451
Tested-by: Jeff Darcy 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Raghavendra Talur 
Reviewed-by: Vijay Bellur

core: run many bricks within one glusterfsd process

2017-01-31T00:13:58+00:00

This patch adds support for multiple brick translator stacks running
in a single brick server process.  This reduces our per-brick memory usage by
approximately 3x, and our appetite for TCP ports even more.  It also creates
potential to avoid process/thread thrashing, and to improve QoS by scheduling
more carefully across the bricks, but realizing that potential will require
further work.

Multiplexing is controlled by the "cluster.brick-multiplex" global option.  By
default it's off, and bricks are started in separate processes as before.  If
multiplexing is enabled, then *compatible* bricks (mostly those with the same
transport options) will be started in the same process.

Change-Id: I45059454e51d6f4cbb29a4953359c09a408695cb
BUG: 1385758
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/14763
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Vijay Bellur

tests/include : EXPECT_WITHIN takes full time even if expression matches

2017-01-25T06:49:46+00:00

Problem: For all the tests using get_pending_heal_count, EXPECT_WITHIN
is taking full time given to it even if the heal count matches with
expected value.

Solution:
RC - In most of the tests, to check heal count, wildcards are
being used. In EXPECT_WITHIN, in if condition, when we use it in
double quotes (" "), it gives string with wildcards which does not
match with the output of get_pending_heal_count.
For example, (0 =~ ^0$).
So, "while" loop was running for full time and at the end, after
coming out of loop, in next if condition it was matching with the
expression without quotes. That is why it was passing.

Remove double quotes in "if condition" in EXPECT_WITHIN and match
as we are matching it in test_expect_footer.

Change-Id: Ia161594774d05b9b888efb2f7ed1950590d8ac1b
BUG: 1412549
Signed-off-by: Ashish Pandey 
Reviewed-on: https://review.gluster.org/16382
Smoke: Gluster Build System 
Reviewed-by: Jeff Darcy 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra Talur

glusterd: bypass add-brick validation with force

2017-01-19T03:50:33+00:00

Commit c916a2f added a validation to restrict add-brick operation if a
replica configuration is changed and any of the bricks belonging to the
volume is down. However we should bypass this validation with a force
option if users really want to have add-brick to go through at the sake
of the corner cases of data loss issue.

The original problem of add-brick getting failed when layout is not set
will still be a problem with a force option as the issue has to be taken
care in the DHT layer.

Change-Id: I0ed3df91ea712f77674eb8afc6fdfa577f25a7bb
BUG: 1406411
Signed-off-by: Atin Mukherjee 
Reviewed-on: http://review.gluster.org/16358
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Ravishankar N 
CentOS-regression: Gluster Build System

tests: Fix split-brain-favorite-child-policy.t failures

2017-01-03T06:13:51+00:00

Problem:
In CentOS-7, the file was receving an extra removexattr(security.ima)
FOP which changed its ctime, breaking the assumption that a particular brick
had the latest ctime based on the writevs done in the .t

Fix:
1. Compare the ctime of both files in the backend and pick the one with
the latest ctime for the fav-child policy. Also unmount the volume
before comparing, to avoid any further FOPS on the file that
can possibly modify the timestamps.

2. Added floating point handling in stat function. Thanks to Pranith for
the helping debugging the regex.

Change-Id: I06041a0f39a29d2593b867af8685d65c7cd99150
BUG: 1408757
Signed-off-by: Ravishankar N 
Reviewed-on: http://review.gluster.org/16288
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

cluster/afr: Fix missing name indices due to EEXIST error

2016-12-27T11:53:04+00:00

PROBLEM:
Consider a volume with  granular-entry-heal and sharding enabled. When
a replica is down and a shard is created as part of a write, the name
index is correctly created under indices/entry-changes/.
Now when a read on the same region triggers another MKNOD, the fop
fails on the online bricks with EEXIST. By virtue of this being a
symmetric error, the failed_subvols[] array is reset to all zeroes.
Because of this, before post-op, the GF_XATTROP_ENTRY_OUT_KEY will be
set, causing the name index, which was created in the previous MKNOD
operation, to be wrongly deleted in THIS MKNOD operation.

FIX:
The ideal fix would have been for a transaction to delete the name
index ONLY if it knows it is the one that created the index in the first
place. This would involve gathering information as to whether THIS xattrop
created the index from individual bricks, aggregating their responses and
based on the various posisble combinations of responses, decide whether to
delete the index or not. This is rather complex. Simpler fix would be
for post-op to examine local->op_ret in the event of no failed_subvols
to figure out whether to delete the name index or not. This can occasionally
lead to creation of stale name indices but they won't be affecting the IO path
or mess with pending changelogs in any way and self-heal in its crawl of
"entry-changes" directory would take care to delete such indices.

Change-Id: Ic1b5257f4dc9c20cb740a866b9598cf785a1affa
BUG: 1408712
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/16286
Smoke: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

tests: Fix races in open-behind.t

2016-09-27T11:44:27+00:00

Problems:
1) flush-behind is on by default, so just because write completes doesn't mean
   it will be on the disk, it could still be in write-behind's cache. This
   leads to failure where if you write from one mount and expect it to be there
   on the other mount, sometimes it won't be there.
2) Sometimes the graph switch is not completing by the time we issue read which
   is leading to opens not being sent on brick leading to failures.

Fixes:
1) Disable flush-behind
2) Add new functions to check the new graph is there and connected to bricks
   before 'cat' is executed.

BUG: 1379511
Change-Id: I0faed684e0dc70cfd2258ce6fdaed655ee915ae6
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/15575
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System