glusterfs.git/libglusterfs, branch v3.8.6

features/shard: Fill loc.pargfid too for named lookups on individual shards

2016-11-09T07:28:19+00:00

        Backport of: http://review.gluster.org/#/c/15788/

On a sharded volume when a brick is replaced while IO is going on, named
lookup on individual shards as part of read/write was failing with
ENOENT on the replaced brick, and as a result AFR initiated name heal in
lookup callback. But since pargfid was empty (which is what this patch
attempts to fix), the resolution of the shards by protocol/server used
to fail and the following pattern of logs was seen:

Brick-logs:

[2016-11-08 07:41:49.387127] W [MSGID: 115009]
[server-resolve.c:566:server_resolve] 0-rep-server: no resolution type
for (null) (LOOKUP)
[2016-11-08 07:41:49.387157] E [MSGID: 115050]
[server-rpc-fops.c:156:server_lookup_cbk] 0-rep-server: 91833: LOOKUP(null)
(00000000-0000-0000-0000-000000000000/16d47463-ece5-4b33-9c93-470be918c0f6.82)
==> (Invalid argument) [Invalid argument]

Client-logs:
[2016-11-08 07:41:27.497687] W [MSGID: 114031]
[client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-0: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[Invalid argument]
[2016-11-08 07:41:27.497755] W [MSGID: 114031]
[client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-1: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[Invalid argument]
[2016-11-08 07:41:27.498500] W [MSGID: 114031]
[client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-2: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[Invalid argument]
[2016-11-08 07:41:27.499680] E [MSGID: 133010]

Also, this patch makes AFR by itself choose a non-NULL pargfid even if
its ancestors fail to initialize all pargfid placeholders.

Change-Id: Ica9e1b5b196ac37aafe6128e7aa0694a07245fdb
BUG: 1392846
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/15796
Reviewed-by: Pranith Kumar Karampuri 
Reviewed-by: Ravishankar N 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

libglusterfs: add gf_get_mem_type()

2016-10-18T07:00:06+00:00

gfapi needs to provide a function towards applications to free memory
that it allocated. Depending on how the application is compiled/linked,
it could use a different memory allocator than Gluster itself. Therefore
it is not safe for gfapi to request applications to free memory with
'standard' free().

Examples for this are Gluster allocated structures with GF_CALLOC() when
memory accounting is enabled (the default). Some gfapi functions use
malloc() to allocate memory as a workaround, but the free() from the
jemalloc implementation should not be combined with the malloc() from
glibc.

Cherry picked from commit db4e26ed71a01e5f760fbc3c7051962426f102c9:
> Change-Id: I626cd1a60abf8965f9263290f4045d1f69fc2093
> BUG: 1344714
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/15108
> Smoke: Gluster Build System 
> Reviewed-by: soumya k 
> Reviewed-by: jiffin tony Thottan 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Shyamsundar Ranganathan 

This also includes commit da2aabe3 which fixes an unused variable
warning.

Change-Id: I626cd1a60abf8965f9263290f4045d1f69fc2093
BUG: 1347717
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/15639
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: jiffin tony Thottan

glusterfsd/main: fix OOM adjustment for older kernels

2016-10-12T11:01:28+00:00

Milind Changire reported that GlusterFS fails to build on RHEL5
because linux/oom.h is unavailable.

Milind's initial patch disables OOM adjustment completely
for those environments that do not have this header. However,
I'd take another approach that:

1) checks for linux/oom.h in compile-time and defines necessary
constants if the header is not present;
2) checks for available OOM API in /proc in run-time and uses it
accordingly.

This allows OOM to be adjusted properly on RHEL5 (the kernel is pretty new
to present /proc API for that) as well as RHEL6 (the kernel has many thing
backported including new /proc API).

> Reviewed-on: http://review.gluster.org/15587
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Niels de Vos 
> Tested-by: Oleksandr Natalenko 

Change-Id: I1bc610586872d208430575c149a7d0c54bd82370
BUG: 1383694
Signed-off-by: Oleksandr Natalenko 
Reviewed-on: http://review.gluster.org/15623
Tested-by: Oleksandr Natalenko 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

afr: Ignore gluster internal (virtual) xattrs in metadata heal check

2016-09-29T07:10:18+00:00

Backport of http://review.gluster.org/#/c/15548/

Problem:
In arbiter configuration, posix-xlator in the arbiter brick always sets the
GF_CONTENT_KEY in the response dict with a value 0. If the file size on the
data bricks is more than quick-read's max-file-size (64kb default), those
bricks don't set the key. Because of this difference in the no. of dict
elements, afr triggers metadata heal in lookup code path, in turn leading to
extra lookups+inodelks.

Fix:
Changed afr dict comparison logic to ignore all virtual xattrs and the on-disk
ones that we should not be healing.

Change-Id: I05730bdd39d8fb0b9a49a5fc9c0bb01f0d3bb308
BUG: 1377193
Signed-off-by: Ravishankar N 
Reviewed-on: http://review.gluster.org/15578
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

quotad: fix potential buffer overflows

2016-08-27T11:10:42+00:00

This converts sprintf to gf_asprintf in following components:                                                                                                          * quotad.c
* dht
* afr
* protocol/client
* rpc/rpc-lib
* rpc/rpc-transport

        This is a backport of http://review.gluster.org/#/c/14102/

> Change-Id: If8a267bab3d91003bdef3a92664077a0136745ee
> BUG: 1332073
> Signed-off-by: Raghavendra G 
> Reviewed-on: http://review.gluster.org/14102
> Tested-by: Manikandan Selvaganesh 
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Manikandan Selvaganesh 

Change-Id: If8a267bab3d91003bdef3a92664077a0136745ee
BUG: 1366746
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/15325
Smoke: Gluster Build System 
Tested-by: Manikandan Selvaganesh 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

cluster/afr: Prevent split-brain when bricks are brought off and on in cyclic order

2016-08-22T10:22:36+00:00

        Backport of: http://review.gluster.org/15080

When the bricks are brought offline and then online in cyclic
order while writes are in progress on a file, thanks to inode
refresh in write txns, AFR will mostly fail the write attempt
when the only good copy is offline. However, there is still a
remote possibility that the file will run into split-brain if
the brick that has the lone good copy goes offline *after* the
inode refresh but *before* the write txn completes (I call it
in-flight split-brain in the patch for ease of reference),
requiring intervention from admin to resolve the split-brain
before the IO can resume normally on the file. To get around this,
the patch does the following things:
i) retains the dirty xattrs on the file
ii) avoids marking the last of the good copies as bad (or accused)
    in case it is the one to go down during the course of a write.
iii) fails that particular write with the appropriate errno.

This way, we still have one good copy left despite the split-brain situation
which when it is back online, will be chosen as source to do the heal.

> Change-Id: I9ca634b026ac830b172bac076437cc3bf1ae7d8a
> BUG: 1363721
> Signed-off-by: Krutika Dhananjay 
> Reviewed-on: http://review.gluster.org/15080
> Tested-by: Pranith Kumar Karampuri 
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Ravishankar N 
> Reviewed-by: Oleksandr Natalenko 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Pranith Kumar Karampuri 
(cherry picked from commit fcb5b70b1099d0379b40c81f35750df8bb9545a5)

Change-Id: I157f1025aebd6624fa3d412abc69a4ae6f2fe9e0
BUG: 1367272
Signed-off-by: Krutika Dhananjay 
Signed-off-by: Oleksandr Natalenko 
Reviewed-on: http://review.gluster.org/15221
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

core: use for makedev(3), major(3), minor(3)

2016-08-22T08:43:46+00:00

gnu glibc (libc) in Fedora 26 (rawhide) has turned off "compat" for
makedev(), major(), and minor(3) decls.

3.8.2 on F26 built with warnings, see:
https://kojipkgs.fedoraproject.org//packages/glusterfs/3.8.2/1.fc26/data/logs/x86_64/build.log

More recent builds of other packages (namely nfs-ganesha) indicate
that these warnings would now be errors, and, will be errors when
3.8.3 is released and built.

Cherry picked from commit c441a0ae5aca1f2977961afecb3236f90eb5b672:
> Change-Id: I3caa0d05f5de695818753e40e754da8b40d2011c
> BUG: 1367527
> Signed-off-by: Kaleb S. KEITHLEY 
> Reviewed-on: http://review.gluster.org/15182
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Niels de Vos 

Change-Id: I3caa0d05f5de695818753e40e754da8b40d2011c
BUG: 1368423
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/15203
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY

inode: Adjust lru_size while retiring entries in lru list

2016-08-13T08:04:01+00:00

As part of inode_table_destroy(), we first retire entries
in the lru list but the lru_size is not adjusted accordingly.
This may result in invalid memory reference in inode_table_prune
if the lru_size > lru_limit.

> Reviewed-on: http://review.gluster.org/15087
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Raghavendra G 
> Reviewed-by: Prashanth Pai 

BUG: 1362540
Change-Id: I29ee3c03b0eaa8a118d06dc0cefba85877daf963
Signed-off-by: Soumya Koduri 
Signed-off-by: Oleksandr Natalenko 
Reviewed-on: http://review.gluster.org/15129
Smoke: Gluster Build System 
Reviewed-by: Prashanth Pai 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

libglusterfs/client_t: Dump the 0th client too

2016-08-11T13:04:22+00:00

> Reviewed-on: http://review.gluster.org/14704
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Jeff Darcy 

BUG: 1365742
Change-Id: I565e81944b6670d26ed1962689dcfd147181b61e
Signed-off-by: Raghavendra G 
Signed-off-by: Oleksandr Natalenko 
Reviewed-on: http://review.gluster.org/15124
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Prashanth Pai 
CentOS-regression: Gluster Build System

dht/rebalance: allocate migrator thread pool dynamically

2016-08-10T10:21:54+00:00

Problems: The maximum number of migratior threads created was static set
to "40". And the number of these threads get created in rebalance depends
on the number of cores user has. If the number of cores exceeds 40, a
crash or memory corruption can be seen.

Fix: Make the migratior thread pool dynamic.

> Change-Id: Ifbdac8a1a396363dd75e2f6bcb454070cfdbf839
> BUG: 1362069
> Reviewed-on: http://review.gluster.org/15000
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Raghavendra G 
(cherry picked from commit b8e8bfc7e4d3eaf76bb637221bc6392ec10ca54b)

Change-Id: Ifbdac8a1a396363dd75e2f6bcb454070cfdbf839
BUG: 1362069
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/15061
Smoke: Gluster Build System 
Tested-by: N Balachandran 
Reviewed-by: N Balachandran 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System