glusterfs.git, branch v5.7 GlusterFS is a distributed file-system capable of scaling to several petabytes. It aggregates various storage bricks over Infiniband RDMA or TCP/IP interconnect into one large parallel network file system. doc: Added release notes for 5.7 2019-07-02T04:53:18+00:00 hari gowtham hgowtham@redhat.com 2019-06-05T16:40:47+00:00 d24b4605fe104f7a9a3c9091154291b9cce44915 Fixes: bz#1697986 Change-Id: I6dc17424665431957152761eaec7b6b2226ae1f0 Signed-off-by: hari gowtham <hgowtham@redhat.com> 
Fixes: bz#1697986

Change-Id: I6dc17424665431957152761eaec7b6b2226ae1f0
Signed-off-by: hari gowtham <hgowtham@redhat.com>
cluster/ec: honor contention notifications for partially acquired locks 2019-06-28T11:07:08+00:00 Xavi Hernandez xhernandez@redhat.com 2019-05-09T09:07:18+00:00 257fee6312513498ee70765b9f343d5df77b1fce EC was ignoring lock contention notifications received while a lock was being acquired. When a lock is partially acquired (some bricks have granted the lock but some others not yet) we can receive notifications from acquired bricks, which should be honored, since we may not receive more notifications after that. Since EC was ignoring them, once the lock was acquired, it was not released until the eager-lock timeout, causing unnecessary delays on other clients. This fix takes into consideration the notifications received before having completed the full lock acquisition. After that, the lock will be releaed as soon as possible. Backport of: > BUG: bz#1708156 > Change-Id: I2a306dbdb29fb557dcab7788a258bd75d826cc12 > Signed-off-by: Xavi Hernandez <xhernandez@redhat.com> Fixes: bz#1717282 Change-Id: I2a306dbdb29fb557dcab7788a258bd75d826cc12 Signed-off-by: Xavi Hernandez <xhernandez@redhat.com> Signed-off-by: Hari Gowtham <hgowtham@redhat.com> 
EC was ignoring lock contention notifications received while a lock was
being acquired. When a lock is partially acquired (some bricks have
granted the lock but some others not yet) we can receive notifications
from acquired bricks, which should be honored, since we may not receive
more notifications after that.

Since EC was ignoring them, once the lock was acquired, it was not
released until the eager-lock timeout, causing unnecessary delays on
other clients.

This fix takes into consideration the notifications received before
having completed the full lock acquisition. After that, the lock will
be releaed as soon as possible.

Backport of:
> BUG: bz#1708156
> Change-Id: I2a306dbdb29fb557dcab7788a258bd75d826cc12
> Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>

Fixes: bz#1717282
Change-Id: I2a306dbdb29fb557dcab7788a258bd75d826cc12
Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
Signed-off-by: Hari Gowtham <hgowtham@redhat.com>
tests/utils: Fix py2/py3 util python scripts 2019-06-28T11:07:08+00:00 Kotresh HR khiremat@redhat.com 2019-06-06T07:24:04+00:00 5355f5a18146183a8e27872ebee501acad7b5eb0 Following files are fixed. tests/bugs/distribute/overlap.py tests/utils/changelogparser.py tests/utils/create-files.py tests/utils/gfid-access.py tests/utils/libcxattr.py Have marked glupy as bad test. Backport of: > Change-Id: I3db857cc19e19163d368d913eaec1269fbc37140 > BUG: 1193929 > Signed-off-by: Kotresh HR <khiremat@redhat.com> Change-Id: I3db857cc19e19163d368d913eaec1269fbc37140 Updates: bz#1629877 Signed-off-by: Kotresh HR <khiremat@redhat.com> 
Following files are fixed.

tests/bugs/distribute/overlap.py
tests/utils/changelogparser.py
tests/utils/create-files.py
tests/utils/gfid-access.py
tests/utils/libcxattr.py

Have marked glupy as bad test.

Backport of:
> Change-Id: I3db857cc19e19163d368d913eaec1269fbc37140
> BUG: 1193929
> Signed-off-by: Kotresh HR <khiremat@redhat.com>

Change-Id: I3db857cc19e19163d368d913eaec1269fbc37140
Updates: bz#1629877
Signed-off-by: Kotresh HR <khiremat@redhat.com>
ec: fix truncate lock to cover the write in tuncate clean 2019-05-08T14:20:13+00:00 Kinglong Mee kinglongmee@gmail.com 2019-04-12T03:35:55+00:00 515866cd49aadf52c9949f8745fd2ba1ea9c99c3 ec_truncate_clean does writing under the lock granted for truncate, but the lock is calculated by ec_adjust_offset_up, so that, the write in ec_truncate_clean is out of lock. Updates: bz#1699500 Change-Id: I15ed1b0807d75c5eb817323f1c227e97d03e0e7c Signed-off-by: Kinglong Mee <kinglongmee@gmail.com> (cherry picked from commit 0e1223491e964096384edfae5032ed0d50d028ad) 
ec_truncate_clean does writing under the lock granted for truncate,
but the lock is calculated by ec_adjust_offset_up, so that,
the write in ec_truncate_clean is out of lock.

Updates: bz#1699500
Change-Id: I15ed1b0807d75c5eb817323f1c227e97d03e0e7c
Signed-off-by: Kinglong Mee <kinglongmee@gmail.com>
(cherry picked from commit 0e1223491e964096384edfae5032ed0d50d028ad)
performance/write-behind: remove request from wip list in wb_writev_cbk 2019-05-08T14:07:56+00:00 Raghavendra G rgowdapp@redhat.com 2019-05-03T04:44:48+00:00 09cb7406c28052c4643b2cb6e0ab986331b9576b There is a race in the way O_DIRECT writes are handled. Assume two overlapping write requests w1 and w2. * w1 is issued and is in wb_inode->wip queue as the response is still pending from bricks. Also wb_request_unref in wb_do_winds is not yet invoked. list_for_each_entry_safe (req, tmp, tasks, winds) { list_del_init (&req->winds); if (req->op_ret == -1) { call_unwind_error_keep_stub (req->stub, req->op_ret, req->op_errno); } else { call_resume_keep_stub (req->stub); } wb_request_unref (req); } * w2 is issued and wb_process_queue is invoked. w2 is not picked up for winding as w1 is still in wb_inode->wip. w1 is added to todo list and wb_writev for w2 returns. * response to w1 is received and invokes wb_request_unref. Assume wb_request_unref in wb_do_winds (see point 1) is not invoked yet. Since there is one more refcount, wb_request_unref in wb_writev_cbk of w1 doesn't remove w1 from wip. * wb_process_queue is invoked as part of wb_writev_cbk of w1. But, it fails to wind w2 as w1 is still in wip. * wb_requet_unref is invoked on w1 as part of wb_do_winds. w1 is removed from all queues including w1. * After this point there is no invocation of wb_process_queue unless new request is issued from application causing w2 to be hung till the next request. This bug is similar to bz 1626780 and bz 1379655. Change-Id: Iaa47437613591699d4c8ad18bc0b32de6affcc31 Signed-off-by: Raghavendra G <rgowdapp@redhat.com> Fixes: bz#1707198 (cherry picked from commit 6454132342c0b549365d92bcf3572ecd914f7fa8) 
There is a race in the way O_DIRECT writes are handled. Assume two
overlapping write requests w1 and w2.

* w1 is issued and is in wb_inode->wip queue as the response is still
  pending from bricks. Also wb_request_unref in wb_do_winds is not yet
  invoked.

       list_for_each_entry_safe (req, tmp, tasks, winds) {
		list_del_init (&req->winds);

                if (req->op_ret == -1) {
			call_unwind_error_keep_stub (req->stub, req->op_ret,
		                                     req->op_errno);
                } else {
                        call_resume_keep_stub (req->stub);
		}

                wb_request_unref (req);
        }

* w2 is issued and wb_process_queue is invoked. w2 is not picked up
  for winding as w1 is still in wb_inode->wip. w1 is added to todo
  list and wb_writev for w2 returns.

* response to w1 is received and invokes wb_request_unref. Assume
  wb_request_unref in wb_do_winds (see point 1) is not invoked
  yet. Since there is one more refcount, wb_request_unref in
  wb_writev_cbk of w1 doesn't remove w1 from wip.

* wb_process_queue is invoked as part of wb_writev_cbk of w1. But, it
  fails to wind w2 as w1 is still in wip.

* wb_requet_unref is invoked on w1 as part of wb_do_winds. w1 is
  removed from all queues including w1.

* After this point there is no invocation of wb_process_queue unless
  new request is issued from application causing w2 to be hung till
  the next request.

This bug is similar to bz 1626780 and bz 1379655.

Change-Id: Iaa47437613591699d4c8ad18bc0b32de6affcc31
Signed-off-by: Raghavendra G <rgowdapp@redhat.com>
Fixes: bz#1707198
(cherry picked from commit 6454132342c0b549365d92bcf3572ecd914f7fa8)
cluster/afr: Remove local from owners_list on failure of lock-acquisition 2019-05-08T14:07:12+00:00 Pranith Kumar K pkarampu@redhat.com 2019-04-04T10:01:56+00:00 65505150d665b207e2eb2143599a772c39a648f8 When eager-lock lock acquisition fails because of say network failures, the local is not being removed from owners_list, this leads to accumulation of waiting frames and the application will hang because the waiting frames are under the assumption that another transaction is in the process of acquiring lock because owner-list is not empty. Handled this case as well in this patch. Added asserts to make it easier to find these problems in future. fixes bz#1699736 Change-Id: I3101393265e9827755725b1f2d94a93d8709e923 Signed-off-by: Pranith Kumar K <pkarampu@redhat.com> 
When eager-lock lock acquisition fails because of say network failures, the
local is not being removed from owners_list, this leads to accumulation of
waiting frames and the application will hang because the waiting frames are
under the assumption that another transaction is in the process of acquiring
lock because owner-list is not empty. Handled this case as well in this patch.
Added asserts to make it easier to find these problems in future.

fixes bz#1699736
Change-Id: I3101393265e9827755725b1f2d94a93d8709e923
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
cluster/dht: Request linkto xattrs in dht_rmdir opendir 2019-04-10T07:51:40+00:00 N Balachandran nbalacha@redhat.com 2019-02-06T04:28:55+00:00 c94acb9efdba4d837f1029cdcbdec051a394447a If parallel-readdir is enabled, the rda xlator is loaded below dht in the graph and proactively lists and caches entries when an opendir is performed. dht_rmdir checks if the directory being deleted contains stale linkto files by performing a readdirp on its child subvols. However, as the entries are actually read in during the opendir operation which does not request the linkto xattr,no linkto xattrs are present for the entries causing dht to incorrectly identify them as data files and fail the rmdir operation with ENOTEMPTY. DHT now always adds the linkto xattr in the list of xattrs requested in the opendir. Change-Id: I0711198e66c59146282eb8b88084170bedfb4018 fixes: bz#1695399 Signed-off-by: N Balachandran <nbalacha@redhat.com> (cherry picked from commit 110006bbcd5bb3e814b4cfe7d74cb41891ac3b0c) 
If parallel-readdir is enabled, the rda xlator is loaded
below dht in the graph and proactively lists and caches
entries when an opendir is performed. dht_rmdir checks if
the directory being deleted contains stale linkto files by
performing a readdirp on its child subvols. However, as
the entries are actually read in during the opendir operation
which does not request the linkto xattr,no linkto xattrs are
present for the entries causing dht to incorrectly identify
them as data files and fail the rmdir operation with ENOTEMPTY.
DHT now always adds the linkto xattr in the list of xattrs
requested in the opendir.

Change-Id: I0711198e66c59146282eb8b88084170bedfb4018
fixes: bz#1695399
Signed-off-by: N Balachandran <nbalacha@redhat.com>
(cherry picked from commit 110006bbcd5bb3e814b4cfe7d74cb41891ac3b0c)
doc: Added release 5.6 notes 2019-04-09T11:31:16+00:00 ShyamsundarR srangana@redhat.com 2019-04-09T11:30:43+00:00 34a2347780c2429284f57232f3aabb78547a9795 Fixes: bz#1693300 Change-Id: I4deaa0ecdd1692fb11f2d90ecc30a2370a659c2f Signed-off-by: ShyamsundarR <srangana@redhat.com> 
Fixes: bz#1693300
Change-Id: I4deaa0ecdd1692fb11f2d90ecc30a2370a659c2f
Signed-off-by: ShyamsundarR <srangana@redhat.com>
glusterd: fix txn-id mem leak 2019-04-09T11:06:46+00:00 Atin Mukherjee amukherj@redhat.com 2019-03-18T10:38:04+00:00 9f415235f6a05280824900c59a8c926fff423cdf This commit ensures the following: 1. Don't send commit op request to the remote nodes when gluster v status all is executed as for the status all transaction the local commit gets the name of the volumes and remote commit ops are technically a no-op. So no need for additional rpc requests. 2. In op state machine flow, if the transaction is in staged state and op_info.skip_locking is true, then no need to set the txn id in the priv->glusterd_txn_opinfo dictionary which never gets freed. Fixes: bz#1694612 Change-Id: Ib6a9300ea29633f501abac2ba53fb72ff648c822 Signed-off-by: Atin Mukherjee <amukherj@redhat.com> (cherry picked from commit 34e010d64905b7387de57840d3fb16a326853c9b) 
This commit ensures the following:
1. Don't send commit op request to the remote nodes when gluster v
status all is executed as for the status all transaction the local
commit gets the name of the volumes and remote commit ops are
technically a no-op. So no need for additional rpc requests.
2. In op state machine flow, if the transaction is in staged state and
op_info.skip_locking is true, then no need to set the txn id in the
priv->glusterd_txn_opinfo dictionary which never gets freed.

Fixes: bz#1694612
Change-Id: Ib6a9300ea29633f501abac2ba53fb72ff648c822
Signed-off-by: Atin Mukherjee <amukherj@redhat.com>
(cherry picked from commit 34e010d64905b7387de57840d3fb16a326853c9b)
cluster-syncop: avoid duplicate unlock of inodelk/entrylk 2019-04-08T14:18:16+00:00 Kinglong Mee kinglongmee@gmail.com 2019-03-18T12:47:54+00:00 b399d33d575bb2e2cb7af5232e96a091d4768682 When using ec, there are many messages at brick log as, [inodelk.c:514:__inode_unlock_lock] 0-test-locks: Matching lock not found for unlock 0-9223372036854775807, lo=68e040a84b7f0000 on 0x7f208c006f78 [MSGID: 115053] [server-rpc-fops_v2.c:280:server4_inodelk_cbk] 0-test-server: 2557439: INODELK <gfid:df4e41be-723f-4289-b7af-b4272b3e880c> (df4e41be-723f-4289-b7af-b4272b3e880c), client: CTX_ID:67d4a7f3-605a-4965-89a5-31309d62d1fa-GRAPH_ID:0-PID:1659-HOST:openfs-node2-PC_NAME:test-client-1-RECON_NO:-28, error-xlator: test-locks [Invalid argument] > Change-Id: Ib164d29ebb071f620a4ca9679c4345ef7c88512a > Signed-off-by: Kinglong Mee <mijinlong@open-fs.com> (cherry-pick of https://review.gluster.org/#/c/glusterfs/+/22377/) Change-Id: I8345ad6c8e1bbb676917eb47e1c5ed72c162f6ce Updates: bz#1690952 Signed-off-by: Kinglong Mee <mijinlong@open-fs.com> 
When using ec, there are many messages at brick log as,

[inodelk.c:514:__inode_unlock_lock] 0-test-locks:  Matching lock not found for unlock 0-9223372036854775807, lo=68e040a84b7f0000 on 0x7f208c006f78
[MSGID: 115053] [server-rpc-fops_v2.c:280:server4_inodelk_cbk] 0-test-server: 2557439: INODELK <gfid:df4e41be-723f-4289-b7af-b4272b3e880c> (df4e41be-723f-4289-b7af-b4272b3e880c), client: CTX_ID:67d4a7f3-605a-4965-89a5-31309d62d1fa-GRAPH_ID:0-PID:1659-HOST:openfs-node2-PC_NAME:test-client-1-RECON_NO:-28, error-xlator: test-locks [Invalid argument]

> Change-Id: Ib164d29ebb071f620a4ca9679c4345ef7c88512a
> Signed-off-by: Kinglong Mee <mijinlong@open-fs.com>
(cherry-pick of https://review.gluster.org/#/c/glusterfs/+/22377/)

Change-Id: I8345ad6c8e1bbb676917eb47e1c5ed72c162f6ce
Updates: bz#1690952
Signed-off-by: Kinglong Mee <mijinlong@open-fs.com>