glusterfs.git/tests, branch v3.7.16

gfapi: redesign the public interface for upcall consumers

2016-10-03T12:20:43+00:00

The glfs_callback_arg and glfs_callback_inode_arg were allocated by
gfapi, and expected to be free()'d by the application. However it is not
reasonable to expect that applications use the same memory allocator to
as the compiled libgfapi.so. For instance, it is possible that gfapi
uses glibc malloc/free, and an application like NFS-Ganesha the versions
from jemalloc. Mismatching of the malloc() and free() functions causes
segmentation faults at best.

In order to prevent problems like this in the future, the API for
applications that consume upcalls has been remodeled. Any of the
structures that gfapi allocates, should be free'd with glfs_free(). The
members of the structures can not be accessed directly anymore, each
has its own function to access now.

Correcting the naming of the functions, structures and constants is a
continuation of commit 2775dc64101ed37c8d9809bf9852dbf0746ee2b6. These
new improvements not only have correct prefixes for the functions and
structures, the naming also reflects more to the upcall framework and
does not use "callback" anymore.

Cherry picked from commit 4721188a154acd9a0a4c096d8d73e97f3bf1b2a9:
> Change-Id: I2b8bd5a0a82036d2abea1a217f5e5975a1d4fe93
> BUG: 1344714
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/14701
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kaleb KEITHLEY 
> Reviewed-by: soumya k 
> Reviewed-by: jiffin tony Thottan 

Once difference with the version of this change in other branches is
that leases are not included in glusterfs-3.7. Hence there is a little
change that drops the handling of GF_UPCALL_RECALL_LEASE.

In addition, this backport contains commit 2775dc6410:
> libgfapi/upcall : prepend "glfs_" to callback_arg, callback_inode_arg
> Reviewed-on: http://review.gluster.org/14702

Change-Id: I2b8bd5a0a82036d2abea1a217f5e5975a1d4fe93
BUG: 1347715
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/15602
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kaushal M

Upcall/cache-invalidation: Use parent stbuf while updating parent entry

2016-10-03T12:15:26+00:00

For *create*  fops (CREATE, MKDIR, MKNOD), we invalidate the parent
entry. Hence send parent attributes in the stat field.

Also "UP_PARENT_DENTRY_FLAGS" has to be set only for the fops which shall
result in two invalidations requests - one for the inode on which fop is
being performed and another on parent entry.

In case of CREATE/MKDIR/MKNOD fops, there shall be only one invalidation
request sent, that too on parent inode. We send invalidation directly on
parent inode's gfid. So there is no necessity to set these flags which
when set shall endup invalidating the parent's parent entry.

Cherry picked from commit f4282bd927e2e0d826d62cf1192102382c5697b2:
> Change-Id: I7514ee08382081e3e060818ede497dbca26987dc
> BUG: 1291259
> Signed-off-by: Soumya Koduri 
> Reviewed-on: http://review.gluster.org/12962
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Smoke: Gluster Build System 
> Reviewed-by: Niels de Vos 

Change-Id: I7514ee08382081e3e060818ede497dbca26987dc
BUG: 1347715
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/15600
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kaushal M

tests: fix rebalance timing issue

2016-09-19T05:35:17+00:00

With a start and stop rebalance, the stop command may fail
as by that time the rebalance process may not come up.
Using the rebalance status commmand to ensure that the rebalance
process is up before stoping rebalance.

>Reviewed-on: http://review.gluster.org/14885
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Jeff Darcy 

Change-Id: I3d5123cd5dfabde2720428455b257d11b980ce21
BUG: 1375049
Signed-off-by: Atin Mukherjee 
Reviewed-on: http://review.gluster.org/15461
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kaushal M

dht: "replica.split-brain-status" attribute value is not correct

2016-09-13T04:49:55+00:00

Problem: In a distributed-replicate volume attribute
         "replica.split-brain-status" value does not display split-brain
           condition though directory is in split-brain.
         If directory is in split brain on mutiple replica-pairs
         it does not show full list of replica pairs.

Solution: Update the dht_aggregate code to aggregate the xattr
          value in this specific condition.

Fix:      1) function getChoices returns the choices from split-brain
             status string.
          2) function add_opt adding the choices to local buffer to
             store in dictionary
          3) For the key "replica.split-brain-status" function dht_aggregate
             call dht_aggregate_split_brain_xattr to prepare the list.

Test:     To verify the patch followed below steps
          1) Create a distributed replica volume and create mount point
          2) Stop heal daemon
          3) Touch file and directories on mount point
             mkdir test{1..5};touch tmp{1..5}
          4) Down brick process on one of the replica set
             pkill -9 glusterfsd
          5) Change permission of dir on mount point
             chmod 755 test{1..5}
          6) Restart brick process on node with force option
          7) kill brick process on other node in same replica set
          8) Change permission of dir again on mount point
             chmod 766 test{1..5}
          9) Reexecute same step from 4-9 on other replica set also
          10) After check heal status on server it will show dir's are
              in split brain on all replica sets
          11) After check the replica.split-brain-status attr on mount
              point it will show wrong status of split brain.
          12) After apply the patch the attribute shows correct value.

> Change-Id: Icdfd72005a4aa82337c342762775a3d1761bbe4a
> Signed-off-by: Mohit Agrawal 
> Reviewed-on: http://review.gluster.org/15201
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Raghavendra G 
> (cherry picked from commit c4e9ec653c946002ab6d4c71ee8e6df056438a04)

Change-Id: Ia5234e8a2291a7e8a7211c82368f4df1c99fa099
Backport of commit c4e9ec653c946002ab6d4c71ee8e6df056438a04
BUG: 1375099
Signed-off-by: Mohit Agrawal 
Reviewed-on: http://review.gluster.org/15466
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

tests/cli: Generate SSL certificates

2016-09-06T07:44:32+00:00

Generate SSL certificates before enabling
management encryption to avoid test failure.

master -
This patch is backport of following two master patches

http://review.gluster.org/#/c/13959/ - bug-1320388.t
was first introduced in this patch

http://review.gluster.org/#/c/15202/ - Modified
bug-1320388.t to create SSL cerificate

Change-Id: Iab23b36703f4653f1d5bb9d14695e4d3fa63ad61
BUG: 1368926
Signed-off-by: Ashish Pandey 
>Change-Id: Iab23b36703f4653f1d5bb9d14695e4d3fa63ad61
>BUG: 1368349
>Signed-off-by: Ashish Pandey 
>Reviewed-on: http://review.gluster.org/15202
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Atin Mukherjee 
>Signed-off-by: Ashish Pandey 
Reviewed-on: http://review.gluster.org/15227
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kaushal M

gfapi: do not cache upcalls if the application is not interested

2016-09-05T09:09:01+00:00

When the volume option 'features.cache-invalidation' is enabled, upcall
events are sent from the brick process to the client. Even if the client
is not interested in upcall events itself, md-cache or other xlators may
benefit from them.

By adding a new 'cache_upcalls' boolean in the 'struct glfs', we can
enable the caching of upcalls when the application called
glfs_h_poll_upcall(). NFS-Ganesha sets up a thread for handling upcalls
in the initialization phase, and calls glfs_h_poll_upcall() before any
NFS-client accesses the NFS-export.

In the future there will be a more flexible registration API for
enabling certain kind of upcall events. Until that is available, this
should work just fine.

Verificatio of this change is not trivial within our current regression
test framework. The bug report contains a description on how to reliably
reproduce the problem with the glusterfs-coreutils.

Cherry picked from commit 218c9b033fa44eacbc27d87491abd830548b362e:
> Change-Id: I818595c92db50e6e48f7bfe287ee05103a4a30a2
> BUG: 1368842
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/15191
> Smoke: Gluster Build System 
> Reviewed-by: Poornima G 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: soumya k 
> Reviewed-by: Kaleb KEITHLEY 

Change-Id: I818595c92db50e6e48f7bfe287ee05103a4a30a2
BUG: 1368843
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/15347
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Poornima G 
Reviewed-by: soumya k

nfs: allow hostnames with dashes in exports/netgroups files

2016-08-25T21:03:11+00:00

Hostnames with dashes (like "vagrant-testVM") are not correctly parsed
when reading the exports/netgroups files. This bacomes obvious when
running ./run-tests-in-vagrant.sh because it causes
tests/basic/mount-nfs-auth.t and tests/basic/netgroup_parsing.t to fail.

The regex for hostname (in exports) and the entry and hostname
(netgroups) parsing does not include the "-" sign, and hence the
hostnames are splitted at it.

Cherry picked from commit e5221d288e41d29d89d52f8deab657d2285a852c:
> BUG: 1350237
> Change-Id: I38146a283561e1fa386cc841c43fd3b1e30a87ad
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/14809
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Jeff Darcy 

Change-Id: I38146a283561e1fa386cc841c43fd3b1e30a87ad
BUG: 1357835
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/14956
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
Reviewed-by: jiffin tony Thottan

gfapi: Fix IO error caused when there is consecutive graph switches

2016-08-25T04:38:22+00:00

Issue:
Consider a simple situation, where glfs_init() is done, i.e. initial
graph is up. Now perform 2 volume sets that results in 2 client side
graph changes. After this perform some IO, the IO fails with ENOTCON.
The only way to recover this client is i guess another graph switch
or restart.

What actually is happening from code perspective:
Initial graph lets say A, followed by 2 consecutive graph switches
to B and C without any IO those two switches.

- graph_setup (A) as a result of GF_EVENT_CHILD_UP, and
fs->next_subvol = A

- glfs_init() results in fs->active_subvol = A, fs->next_subvol = NULL

- graph_setup (B) as a result of GF_EVENT_CHILD_UP, and
fs->next_subvol = B

- graph_setup (C) as a result of GF_EVENT_CHILD_UP, and
fs->next_subvol = C. It also sees that the previous graph B was never
set as fs->active_subvol, i.e. no IO or anything happened on B, so
can safely send GF_EVENT_PARENT_DOWN (by calling glfs_subvol_done(B)).
This parent down on B, results in child_down(B), which is fine.
But child_down also triggers graph_setup(B).

- graph_setup(B) as a result of GF_EVENT_CHILD_DOWN, and
fs->next_subvol = B, and GF_EVENT_PARENT_DOWN on C as explained
above. This again leads to GF_EVENT_CHILD_DOWN on C.

- graph_setup(C) as a result of GF_EVENT_CHILD_DOWN, and
fs->next_subvol = C, and GF_EVENT_PARENT_DOWN on B as explained
above.

Thus both the graphs B and C are disconnected, and hence the ENOTCON

Solution:
Remove the call to graph_setup() when the event is GF_EVENT_CHILD_DOWN.
It don't see any reason why graph_setup should be called when there is
child_down. Not sure what the original reason was, to have graph_setup
in child_down. git hostory shows the first patch itself had this call.

> Reviewed-on: http://review.gluster.org/14656
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Jeff Darcy 

BUG: 1367294
Change-Id: I9de86555f66cc94a05649ac863b40ed3426ffd4b
Signed-off-by: Poornima G 
Signed-off-by: Oleksandr Natalenko 
Reviewed-on: http://review.gluster.org/14835
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kaushal M

cluster/afr: Prevent split-brain when bricks are brought off and on in cyclic order

2016-08-22T10:05:08+00:00

        Backport of: http://review.gluster.org/15080

When the bricks are brought offline and then online in cyclic
order while writes are in progress on a file, thanks to inode
refresh in write txns, AFR will mostly fail the write attempt
when the only good copy is offline. However, there is still a
remote possibility that the file will run into split-brain if
the brick that has the lone good copy goes offline *after* the
inode refresh but *before* the write txn completes (I call it
in-flight split-brain in the patch for ease of reference),
requiring intervention from admin to resolve the split-brain
before the IO can resume normally on the file. To get around this,
the patch does the following things:
i) retains the dirty xattrs on the file
ii) avoids marking the last of the good copies as bad (or accused)
    in case it is the one to go down during the course of a write.
iii) fails that particular write with the appropriate errno.

This way, we still have one good copy left despite the split-brain situation
which when it is back online, will be chosen as source to do the heal.

Change-Id: I7c13c6ddd5b8fe88b0f2684e8ce5f4a9c3a24a08
BUG: 1367270
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/15222
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Oleksandr Natalenko 
Reviewed-by: Pranith Kumar Karampuri

snapshot/snapd: Don't display pid when snapd is offline

2016-08-22T08:34:31+00:00

    Backport of http://review.gluster.org/#/c/14981/

We were previously reading the pidfile, and displaying
the pid even if snapd daemon is not running. Now to fix
it, we re-assign pid value to -1, if snapd is offline.

> Reviewed-on: http://review.gluster.org/14981
> Tested-by: Vijay Bellur 
> NetBSD-regression: NetBSD Build System 
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Jeff Darcy 

(cherry picked from commit ec6925a379c7bee071df1638bc2751b266cee346)

Change-Id: I4baff8d489fe9380061c52aea006db90fa421cd7
BUG: 1360979
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/15032
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Rajesh Joseph