glusterfs.git/xlators/cluster/dht, branch v3.10dev

dht, md-cache, upcall: Add invalidation of IATT when the layout changes

2016-08-31T06:08:54+00:00

Issue:
dht_layout is built as a part of lookup only. The layout can be
modified by rebalance process. Since every IO fop is preceded
by a lookup, there are very less issues of stale layout. But
with enhancements of aggressive caching of stats in md-cache,
the lookup will reduce and expose the stale layout issue often.

Solution:
Since stale layout is already an issue on dht, there is already
a plan to fix this at the dht layer, but this fix is not currently
planned for any release. Until this fix comes out, we can have
a workaround where, the upcall will send a notification to md-cache
when a layout xattr is changed. As a part of layout change notification
the existing cache is invalidated and the next lookup will fetch the
latest layout.

This is not a foolproof solution as the window between the layout change
and the next lookup(after invalidation of stat), where there will be stale
layout. But until the final fix comes in, this reduces the stale layout
window.

Change-Id: Iacf871a38b35880c1fc0bc68fe7ce291265e71d4
BUG: 1369638
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15300
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G

dht/tiering: fix unused variable warnings/errors

2016-08-30T18:45:03+00:00

http://review.gluster.org/14085 fixes a/the "leak" - via the
generated rpc/xdr headers - of pragmas that mask these warnings.

However 14085 won't pass the smoke test until all the warnings are
fixed.

Change-Id: I367a737570dd7d2f6cc25f4bf4299d31bb6826aa
BUG: 1369124
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: http://review.gluster.org/15242
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos 
Smoke: Gluster Build System 
Reviewed-by: Prashanth Pai 
Reviewed-by: Dan Lambright

dht: Implement ipc fop

2016-08-27T12:48:36+00:00

ipc is used by md-cache to communicate the list of xattrs that
it is caching, to the upcall xlator. Hence implement this in
dht, such that it winds to all the bricks if the ipc op is
GF_IPC_MDC_TARGET_UPCALL. The ips should not fail if any of
the bricks is down, as md-cache will replay the ipc late when
the brick comes back up.

Change-Id: Ica551a550c04cbb1240c0d211fe831c2e5eb6017
BUG: 1211863
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15225
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G

quotad: fix potential buffer overflows

2016-08-25T12:18:09+00:00

This converts sprintf to gf_asprintf in following components:                                                                                                          * quotad.c
* dht
* afr
* protocol/client
* rpc/rpc-lib
* rpc/rpc-transport

Change-Id: If8a267bab3d91003bdef3a92664077a0136745ee
BUG: 1332073
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/14102
Tested-by: Manikandan Selvaganesh 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Manikandan Selvaganesh

cluster/tier: dont promote if estimated block consumption > hi watermark

2016-07-29T14:14:37+00:00

Add test to fail promotion if estimated block consumption grows
beyond hi watermark.

Skip file migrations until next cycle if tier_get_fs_stat() fails
in tier_migrate_using_query_file()

Change-Id: Ice04572fa739c09109c4433e65965197482a7beb
BUG: 1349284
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/14780
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: mohammed rafi  kc 
Reviewed-by: Dan Lambright

dht/rebalance: allocate migrator thread pool dynamically

2016-07-28T12:46:30+00:00

Problems: The maximum number of migratior threads created was static set
to "40". And the number of these threads get created in rebalance depends
on the number of cores user has. If the number of cores exceeds 40, a 
crash or memory corruption can be seen.

Fix: Make the migratior thread pool dynamic.

Change-Id: Ifbdac8a1a396363dd75e2f6bcb454070cfdbf839
BUG: 1359711
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/15000
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

cluster/tier: fix statfs for dht/tiered volumes

2016-07-27T13:03:48+00:00

Return the correct size of the tiered volume in statfs. It should
be the size of the cold tier, not the sum of the hot and cold tier,
because the hot tier is a cache and not an extension of the volume's
capacity. The number of free blocks, etc is the cold tier's capacity
subtracted by the sum of utilization on the hot and cold tiers. Note
if both tiers are part of the same file system this must be accounted
for as well.

The patch also fixes a pre-existing bug in the DHT/tier
translators. If statfs was taken on a file, the code only calculated
free space on the cached subvolume, not all subvolumes in the replica
group. With the fix, this is corrected, except in the case
where quota is used with the deem-statfs option set to "on".

Change-Id: I2b8bcb4511edf83f12130960aad0a609fcf8f513
BUG: 1339689
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/14536
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G 
CentOS-regression: Gluster Build System

dict: Don't expose get_new_dict/dict_destroy

2016-07-26T06:47:33+00:00

get_new_dict/dict_destroy is causing confusion where, dict_new/dict_destroy or
get_new_dict/dict_unref are used instead of dict_new/dict_unref.

Change-Id: I4cc69f5b6711d720823395e20fd624a0c6c1168c
BUG: 1296043
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/13183
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Jeff Darcy 
Reviewed-by: Krutika Dhananjay

cluster/dht: Wrong type of function's parameter when calling dht_selfheal_dir_xattr_cbk

2016-06-23T14:33:19+00:00

The second parameter's type is call_frame_t *, and we change
it to be type xlator_t *, it is exactly what we need in this function.

Change-Id: I6a154edcaa5a11084d837ca925efbfac853d0786
BUG: 1346551
Signed-off-by: Zhou Zhengping 
Reviewed-on: http://review.gluster.org/14737
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G 
CentOS-regression: Gluster Build System

cluster/dht: initialize cbk before attempting inode-link

2016-06-17T12:23:50+00:00

Otherwise inode-link failures in selfheal codepath will result in a
crash.

Change-Id: I9061629ae9d1eb1ac945af5f448d0d8b397a5022
BUG: 1345748
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/14707
Reviewed-by: N Balachandran 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Poornima G 
Reviewed-by: Susant Palai 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy