glusterfs.git/libglusterfs, branch release-8 GlusterFS is a distributed file-system capable of scaling to several petabytes. It aggregates various storage bricks over Infiniband RDMA or TCP/IP interconnect into one large parallel network file system. posix: fix chmod error on symlinks (#2157) 2021-02-12T14:35:05+00:00 Xavi Hernandez xhernandez@users.noreply.github.com 2021-02-12T14:35:05+00:00 d036312a7e59942d8e137b424fa2dc6012dd6ad2 After glibc 2.32, lchmod() is returning EOPNOTSUPP instead of ENOSYS when called on symlinks. The man page says that the returned code is ENOTSUP. They are the same in linux, but this patch correctly handles all errors. Fixes: #2154 Change-Id: Ib3bb3d86d421cba3d7ec8d66b6beb131ef6e0925 Signed-off-by: Xavi Hernandez <xhernandez@redhat.com> 
After glibc 2.32, lchmod() is returning EOPNOTSUPP instead of ENOSYS when
called on symlinks. The man page says that the returned code is ENOTSUP.
They are the same in linux, but this patch correctly handles all errors.

Fixes: #2154
Change-Id: Ib3bb3d86d421cba3d7ec8d66b6beb131ef6e0925
Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
 Events: Fixing coverity issues. 2020-09-15T01:10:18+00:00 Srijan Sivakumar ssivakum@redhat.com 2020-09-01T07:18:48+00:00 85ff28ace3901a5a54d8de42d33ab2f9ac528ed8 Fixing resource leak reported by coverity scan. CID: 1431237 Change-Id: I2bed106b3dc4296c50d80542ee678d32c6928c25 Updates: #1060 Signed-off-by: Srijan Sivakumar <ssivakum@redhat.com> (cherry picked from commit ebc0253269d8a538239dd0b99d42f56ea320b0f0) 
Fixing resource leak reported by coverity scan.

CID: 1431237

Change-Id: I2bed106b3dc4296c50d80542ee678d32c6928c25
Updates: #1060
Signed-off-by: Srijan Sivakumar <ssivakum@redhat.com>
(cherry picked from commit ebc0253269d8a538239dd0b99d42f56ea320b0f0)
Events: Socket creation after getaddrinfo and IPv4 and IPv6 packet capture 2020-09-14T20:01:45+00:00 srijan-sivakumar ssivakum@redhat.com 2020-07-18T00:29:09+00:00 93d48622d9ddb96f07a8590312c2885e11751436 Issue: Currently, the socket creation is done prior to getaddrinfo function being invoked. This can cause mismatch in the protocol and address families of the created socket and the result of the getaddrinfo api. Also, the glustereventsd UDP server by default only captures IPv4 packets hence IPv6 packets are not even captured. Code Changes: 1. Modified the socket creation in such a way that the parameters taken in are dependent upon the result of the getaddrinfo function. 2. Created a subclass for adding address family in glustereventsd.py for both AF_INET and AF_INET6. 3. Modified addresses in the eventsapiconf.py.in Reasoning behind the approach: 1. If we are using getaddrinfo function then socket creation should happen only after we check if we received back valid addresses. Hence socket creation should come after the call to getaddrinfo 2. The listening server which pushes the events to the webhook has to listen for both IPv4 and IPv6 messages as we would not be sure as to what address family is picked in _gf_event. Fixes: #1377 Change-Id: I568dcd1a977c8832f0fef981e1f81cac7043c760 Signed-off-by: srijan-sivakumar <ssivakum@redhat.com> (cherry picked from commit 7c309928591deb8d0188793677958226ac03897a) 
Issue: Currently, the socket creation is done
prior to getaddrinfo function being invoked. This
can cause mismatch in the protocol and address
families of the created socket and the result
of the getaddrinfo api. Also, the glustereventsd
UDP server by default only captures IPv4 packets
hence IPv6 packets are not even captured.

Code Changes:
1. Modified the socket creation in such a way that
the parameters taken in are dependent upon the
result of the getaddrinfo function.
2. Created a subclass for adding address family
in glustereventsd.py for both AF_INET and AF_INET6.
3. Modified addresses in the eventsapiconf.py.in

Reasoning behind the approach:
1. If we are using getaddrinfo function then
socket creation should happen only after we
check if we received back valid addresses.
Hence socket creation should come after the call
to getaddrinfo
2. The listening server which pushes the events
to the webhook has to listen for both IPv4
and IPv6 messages as we would not be sure as to
what address family is picked in _gf_event.

Fixes: #1377
Change-Id: I568dcd1a977c8832f0fef981e1f81cac7043c760
Signed-off-by: srijan-sivakumar <ssivakum@redhat.com>
(cherry picked from commit 7c309928591deb8d0188793677958226ac03897a)
libglusterfs: fix dict leak 2020-09-07T04:52:58+00:00 Ravishankar N ravishankar@redhat.com 2020-09-01T06:06:42+00:00 635dcf82505efcdeaf01c4e0450a157b533099ba Problem: gf_rev_dns_lookup_cached() allocated struct dnscache->dict if it was null but the freeing was left to the caller. Fix: Moved dict allocation and freeing into corresponding init and fini routines so that its easier for the caller to avoid such leaks. Updates: #1000 Change-Id: I90d6a6f85ca2dd4fe0ab461177aaa9ac9c1fbcf9 Signed-off-by: Ravishankar N <ravishankar@redhat.com> (cherry picked from commit 079f7a7d8a2bd85070c1da4dde2452ca82a1cdbb) 
Problem:
gf_rev_dns_lookup_cached() allocated struct dnscache->dict if it was null
but the freeing was left to the caller.

Fix:
Moved dict allocation and freeing into corresponding init and fini
routines so that its easier for the caller to avoid such leaks.

Updates: #1000
Change-Id: I90d6a6f85ca2dd4fe0ab461177aaa9ac9c1fbcf9
Signed-off-by: Ravishankar N <ravishankar@redhat.com>
(cherry picked from commit 079f7a7d8a2bd85070c1da4dde2452ca82a1cdbb)
posix: Implement a janitor thread to close fd 2020-08-21T10:38:07+00:00 Mohit Agrawal moagrawa@redhat.com 2020-07-27T12:38:00+00:00 e173c5b0ee32c210a7d36f03f1847c42218a62e5 Problem: In the commit fb20713b380e1df8d7f9e9df96563be2f9144fd6 we use syntask to close fd but we have found the patch is reducing the performance Solution: Use janitor thread to close fd's and save the pfd ctx into ctx janitor list and also save the posix_xlator into pfd object to avoid the race condition during cleanup in brick_mux environment Change-Id: Ifb3d18a854b267333a3a9e39845bfefb83fbc092 Fixes: #1396 Signed-off-by: Mohit Agrawal <moagrawa@redhat.com> (cherry picked from commit 41b9616435cbdf671805856e487e373060c9455b) 
Problem: In the commit fb20713b380e1df8d7f9e9df96563be2f9144fd6 we use
         syntask to close fd but we have found the patch is reducing the
         performance

Solution: Use janitor thread to close fd's and save the pfd ctx into
          ctx janitor list and also save the posix_xlator into pfd object to
          avoid the race condition during cleanup in brick_mux environment

Change-Id: Ifb3d18a854b267333a3a9e39845bfefb83fbc092
Fixes: #1396
Signed-off-by: Mohit Agrawal <moagrawa@redhat.com>
(cherry picked from commit 41b9616435cbdf671805856e487e373060c9455b)
Issue with gf_fill_iatt_for_dirent 2020-08-19T13:27:22+00:00 Soumya Koduri skoduri@redhat.com 2020-07-01T20:37:56+00:00 6b10b33f8a9bce054fec980583cc597f5a438bb5 In "gf_fill_iatt_for_dirent()", while calculating inode_path for loc, the inode should be of parent's. Instead it is loc.inode which results in error and eventually lookup/readdirp fails. This patch fixes the same. Change-Id: Ied086234a4634e8cb13520521ac547c87b3c76b5 Fixes: #1351 Signed-off-by: Soumya Koduri <skoduri@redhat.com> (cherry picked from commit ab8308333aaf033e07dbbdf2f69f9313a7e311f3) 
In "gf_fill_iatt_for_dirent()", while calculating inode_path for loc,
the inode should be of parent's. Instead it is loc.inode which results in error
 and eventually lookup/readdirp fails.

This patch fixes the same.

Change-Id: Ied086234a4634e8cb13520521ac547c87b3c76b5
Fixes: #1351
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
(cherry picked from commit ab8308333aaf033e07dbbdf2f69f9313a7e311f3)
open-behind: rewrite of internal logic 2020-06-29T12:51:52+00:00 Xavi Hernandez xhernandez@redhat.com 2020-05-12T21:54:54+00:00 80e088be4470d86e5c63750f611a39e487a591b3 There was a critical flaw in the previous implementation of open-behind. When an open is done in the background, it's necessary to take a reference on the fd_t object because once we "fake" the open answer, the fd could be destroyed. However as long as there's a reference, the release function won't be called. So, if the application closes the file descriptor without having actually opened it, there will always remain at least 1 reference, causing a leak. To avoid this problem, the previous implementation didn't take a reference on the fd_t, so there were races where the fd could be destroyed while it was still in use. To fix this, I've implemented a new xlator cbk that gets called from fuse when the application closes a file descriptor. The whole logic of handling background opens have been simplified and it's more efficient now. Only if the fop needs to be delayed until an open completes, a stub is created. Otherwise no memory allocations are needed. Correctly handling the close request while the open is still pending has added a bit of complexity, but overall normal operation is simpler. Change-Id: I6376a5491368e0e1c283cc452849032636261592 Fixes: #1225 Signed-off-by: Xavi Hernandez <xhernandez@redhat.com> 
There was a critical flaw in the previous implementation of open-behind.

When an open is done in the background, it's necessary to take a
reference on the fd_t object because once we "fake" the open answer,
the fd could be destroyed. However as long as there's a reference,
the release function won't be called. So, if the application closes
the file descriptor without having actually opened it, there will
always remain at least 1 reference, causing a leak.

To avoid this problem, the previous implementation didn't take a
reference on the fd_t, so there were races where the fd could be
destroyed while it was still in use.

To fix this, I've implemented a new xlator cbk that gets called from
fuse when the application closes a file descriptor.

The whole logic of handling background opens have been simplified and
it's more efficient now. Only if the fop needs to be delayed until an
open completes, a stub is created. Otherwise no memory allocations are
needed.

Correctly handling the close request while the open is still pending
has added a bit of complexity, but overall normal operation is simpler.

Change-Id: I6376a5491368e0e1c283cc452849032636261592
Fixes: #1225
Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
syncop: improve scaling and implement more tools 2020-05-30T05:17:12+00:00 Xavi Hernandez xhernandez@redhat.com 2020-04-30T09:19:01+00:00 67cd2b1ab5c9a8db637506b9a6be57fc79672c99 The current scaling of the syncop thread pool is not working properly and can leave some tasks in the run queue more time than necessary when the maximum number of threads is not reached. This patch provides a better scaling condition to react faster to pending work. Condition variables and sleep in the context of a synctask have also been implemented. Their purpose is to replace regular condition variables and sleeps that block synctask threads and prevent other tasks to be executed. The new features have been applied to several places in glusterd. Change-Id: Ic50b7c73c104f9e41f08101a357d30b95efccfbf Fixes: #1116 Signed-off-by: Xavi Hernandez <xhernandez@redhat.com> 
The current scaling of the syncop thread pool is not working properly
and can leave some tasks in the run queue more time than necessary
when the maximum number of threads is not reached.

This patch provides a better scaling condition to react faster to
pending work.

Condition variables and sleep in the context of a synctask have also
been implemented. Their purpose is to replace regular condition
variables and sleeps that block synctask threads and prevent other
tasks to be executed.

The new features have been applied to several places in glusterd.

Change-Id: Ic50b7c73c104f9e41f08101a357d30b95efccfbf
Fixes: #1116
Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
tests: skip tests on absence of reflink in xfs 2020-05-29T07:02:11+00:00 Pranith Kumar K pkarampu@redhat.com 2020-05-06T12:06:20+00:00 1572f98486bf3204b5d2d89da880b92e26734c3c Fixes: #1223 Change-Id: I36cb72d920ffd77405051546615c5262c392daef Signed-off-by: Pranith Kumar K <pkarampu@redhat.com> (cherry picked from commit b85f01abab658d1d704cd6caf84dd64eddafbff7) 
Fixes: #1223
Change-Id: I36cb72d920ffd77405051546615c5262c392daef
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
(cherry picked from commit b85f01abab658d1d704cd6caf84dd64eddafbff7)
Posix: Optimize posix code to improve file creation 2020-04-06T10:43:26+00:00 Mohit Agrawal moagrawal@redhat.com 2019-12-19T03:02:19+00:00 f2d90013aaa4652dbcc1e6e3d62a0d31ada89d5d Problem: Before executing a fop in POSIX xlator it builds an internal path based on GFID.To validate the path it call's (l)stat system call and while .glusterfs is heavily loaded kernel takes time to lookup inode and due to that performance drops Solution: In this patch we followed two ways to improve the performance. 1) Keep open fd specific to first level directory(gfid[0]) in .glusterfs, it would force to kernel keep the inodes from all those files in cache. In case of memory pressure kernel won't uncache first level inodes. We need to open 256 fd's per brick to access the entry faster. 2) Use at based call's to access relative path to reduce path based lookup time. Note: To verify the patch we have executed kernel untar 100 times on 6 different clients after enabling metadata group-cache and some other option.We were getting more than 20 percent improvement in kenel untar after applying the patch. Credits: Xavi Hernandez <xhernandez@redhat.com> Change-Id: I1643e6b01ed669b2bb148d02f4e6a8e08da45343 updates: #891 Signed-off-by: Mohit Agrawal <moagrawal@redhat.com> 
Problem: Before executing a fop in POSIX xlator it builds an internal
         path based on GFID.To validate the path it call's (l)stat
         system call and while .glusterfs is heavily loaded kernel takes
         time to lookup inode and due to that performance drops

Solution: In this patch we followed two ways to improve the performance.
          1) Keep open fd specific to first level directory(gfid[0])
             in .glusterfs, it would force to kernel keep the inodes
             from all those files in cache. In case of memory pressure
             kernel won't uncache first level inodes. We need to open
             256 fd's per brick to access the entry faster.
          2) Use at based call's to access relative path to reduce
             path based lookup time.

Note: To verify the patch we have executed kernel untar 100 times on 6
      different clients after enabling metadata group-cache and some
      other option.We were getting more than 20 percent improvement in
      kenel untar after applying the patch.

Credits: Xavi Hernandez <xhernandez@redhat.com>
Change-Id: I1643e6b01ed669b2bb148d02f4e6a8e08da45343
updates: #891
Signed-off-by: Mohit Agrawal <moagrawal@redhat.com>