glusterfs.git/api, branch v3.12.5

gfapi: set lkowner in glfd

2017-10-25T11:33:39+00:00

We need a provision to be able to set lkowner (which is
used to distinguish locks maintained by server) in gfapi.
Since the same lk_owner need to be used to be able to
flush the lock while closing the fd, store the lkowner
in the glfd structure itself.

A new API has been added to be able to set lkowner in glfd.

This is backport of below mainline fix -
 https://review.gluster.org/#/c/18429
 https://review.gluster.org/#/c/18522/

Change-Id: I67591d6b9a89c20b9617d52616513ff9e6c06b47
BUG: 1501956
Signed-off-by: Soumya Koduri

gfapi: Duplicate the buffer sent in setxattr calls

2017-08-12T13:33:55+00:00

Issue:
The caller of glfs_setxattr sends a buffer to set as the value.
We create a dict in which the pointer to the value is set.
Underlying layers like md-cache take a ref on this dict to store
the value for a longer time. But the moment setxattr is complete,
the caller of glfs_setxattr can free the value memory.

Solution:
memcpy the setxattr value to the gluster buffer.

> Reviewed-on: https://review.gluster.org/17967
> Reviewed-by: soumya k 
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Jeff Darcy 
> (cherry picked from commit e11296f8e52b7e3b13d21b41d4fa34baea878edf)

Change-Id: I58753fe702e8b7d0f6c4f058714c65d0ad5d7a0a
BUG: 1479655
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/18001
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

tests/gfapi : add test case for nameless lookups in glfs_resolve_component()

2017-08-04T13:40:15+00:00

Plus address pending comment to add check for entry "" in glfs_resolve_component()

Upstream reference :
>Change-Id: I6063f776ce1cd76cb4c1b1f621b064f3dcc91e5c
>BUG: 1460514
>Signed-off-by: Jiffin Tony Thottan 
>Reviewed-on: https://review.gluster.org/17844
>Smoke: Gluster Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Niels de Vos 
>Reviewed-by: soumya k 
>(cherry picked from commit 5c433f8f5834a4cae62d0375bfdb273242630f01)

Change-Id: I6063f776ce1cd76cb4c1b1f621b064f3dcc91e5c
BUG: 1477994
Signed-off-by: Jiffin Tony Thottan 
Reviewed-on: https://review.gluster.org/17966
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

gfapi: mark glfs_ipc() for internal use only

2017-07-28T14:44:53+00:00

The 3.7 version of glfs_ipc() has never been used by external
applications. There is little use for internal xlator communication that
is triggered from outside of core GlusterFS executables. This function
has now been removed from libgfapi.so.

For Gluster 4.0 a new variation for glfs_ipc() has been added. The
function expects dict_t parameters, which are currently not available
for external applications. There is no sense in providing glfs_ipc() for
non-core GlusterFS executables. Therefore, glfs_ipc() has been marked as
private, and the declaration is now in the glfs-internal.h header.

The Python test case (tests/features/ipctest.py) is not correct and will
be re-written in C to prevent portability issues. This test is currently
disabled (commit d26f0bac149d495fa93710c3f7b6b63c36cb8387).

Cherry picked from commit 40d71d6a201d618e52555a062ac1a429d36ca9ca:
> Change-Id: Idbfe35570d34d45ce8b6b43084627a552ac21f59
> Fixes: #269
> Signed-off-by: Niels de Vos 
> Reviewed-on: https://review.gluster.org/17854
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: soumya k 
> Reviewed-by: Kaleb KEITHLEY 

Change-Id: Idbfe35570d34d45ce8b6b43084627a552ac21f59
Fixes: #269
Signed-off-by: Niels de Vos 
Reviewed-on: https://review.gluster.org/17894
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY 
Reviewed-by: jiffin tony Thottan

mem-pool: initialize pthread_key_t pool_key in mem_pool_init_early()

2017-07-19T20:18:24+00:00

It is not possible to call pthread_key_delete for the pool_key that is
intialized in the constructor for the memory pools. This makes it
difficult to do a full cleanup of all the resources in mem_pools_fini().
For this, the initialization of pool_key should be moved to
mem_pool_init().

However, the glusterfsd binary has a rather complex initialization
procedure. The memory pools need to get initialized partially to get
mem_get() functionality working. But, the pool_sweeper thread can get
killed in case it is started before glusterfsd deamonizes.

In order to solve this, mem_pools_init() is split into two pieces:
1. mem_pools_init_early() for initializing the basic structures
2. mem_pools_init_late() to start the pool_sweeper thread

With the split of mem_pools_init(), and placing the pthread_key_create()
in mem_pools_init_early(), it is now possible to correctly cleanup the
pool_key with pthread_key_delete() in mem_pools_fini().

It seems that there was no memory pool initialization in the CLI. This
has been added as well now. Without it, the CLI will not be able to call
mem_get() successfully which results in a hang of the process.

Change-Id: I1de0153dfe600fd79eac7468cc070e4bd35e71dd
BUG: 1470170
Signed-off-by: Niels de Vos 
Reviewed-on: https://review.gluster.org/17779
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy

libglusterfs: Name threads on creation

2017-07-19T14:16:19+00:00

Set names to threads on creation for easier
debugging.

Output of top -H -p 
Before:
19773 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19774 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19775 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19776 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19777 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19778 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19779 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19780 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19781 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19782 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19783 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19784 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19785 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterfsd
19786 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterfsd
19787 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterfsd
19789 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19790 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
25178 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
 5398 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
 7881 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd

After:
19773 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19774 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustertimer
19775 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19776 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustermemsweep
19777 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustersproc0
19778 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustersproc1
19779 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterepoll0
19780 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusteridxwrker
19781 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusteriotwr0
19782 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterbrssign
19783 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterbrswrker
19784 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterclogecon
19785 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterclogd0
19786 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterclogd1
19787 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterclogd2
19789 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterposixjan
19790 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterposixfsy
25178 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterepoll1
 5398 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterepoll2
 7881 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterposixhc

Change-Id: Id5f333755c1ba168a2ffaa4fce6e71c375e10703
BUG: 1254002
Updates: #271
Signed-off-by: Raghavendra Talur 
Reviewed-on: https://review.gluster.org/11926
Reviewed-by: Prashanth Pai 
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System

gfapi: prevent mem-pool leak in case glfs_new_fs() fails

2017-07-12T09:01:12+00:00

Commit 7039243e187 adds a call to mem_pools_init() so that the memory
pool cleanup thread ("sweeper") is started. However, now it is possible
that users of gfapi can not cleanup this thread because glfs_new() can
return NULL, but the sweeper is still running.

In case glfs_fs_new() fails, mem_pools_fini() needs to be called as
well. This seems more correct than calling mem_pools_init() after
glfs_fs_new(), and this makes using memory pools possible *really* early
in the gfapi initialization.

Change-Id: I1f2fb25cc33e227b3c33ce9d1b03f67bc27e981a
Fixes: 7039243e187 ("gfapi: add mem_pools_init and mem_pools_fini calls")
BUG: 1468863
Signed-off-by: Niels de Vos 
Reviewed-on: https://review.gluster.org/17734
Reviewed-by: Jeff Darcy 
Reviewed-by: Vijay Bellur 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: soumya k 
Reviewed-by: Amar Tumballi

mgtm/core : use sha hash function for volfile check

2017-07-10T05:07:11+00:00

We are storing the entire volfile and using this to check
volfile change. With brick multiplexing there will be lot
of graphs per process which will increase the memory foot
print of the process. So instead of storing the entire
graph we could use sha256 and we can compare the hash to
see whether volfile change happened or not.

Also with Brick multiplexing, the direct comparison of vol
file is not correct. There are two problems.

Problem 1:

We are currently storing one single graph (the last
updated volfile) whereas, what we need is the entire
graph with all atttached bricks.

If we fix this issue, we have second problem

Problem 2:
With multiplexing we have a graph that contains multiple
bricks. But what we are checking as part of the reconfigure
is, comparing the entire graph with one single graph,
which will always fail.

Solution:
We create list in glusterfs_ctx_t that stores sha256 hash
of individual brick graphs. When a graph changes happens
we compare the stored hash and the current hash. If the
hash matches, then no need for reconfigure. Otherwise we
first do the reconfigure and then update the hash.

For now, gfapi has not changed this way. Meaning when gfapi
volfile fetch or reconfigure happens, we still store the
entire graph and compare, each memory.

This is fine, because libgfapi will not load brick graphs.
But changing the libgfapi will make the code similar in
both glusterfsd-mgmt and api. Also it helps to reduce some
memory.

Change-Id: I9df917a771a52b95622ab8f63af34ec390163a77
BUG: 1467986
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: https://review.gluster.org/17709
Smoke: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee 
Reviewed-by: Amar Tumballi

gfapi+libglusterfs: fix mem_pools_fini without mem_pools_init case

2017-07-09T03:57:26+00:00

The change consists of two parts: make sure it doesn't happen (in
glfs.c), and make it harmless if it does (in mem-pool.c).

Change-Id: Icb7dda7a45dd3d1ade2ee3991bb6a22c8ec88424
BUG: 1468863
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/17728
Tested-by: Jeff Darcy 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Jeff Darcy

gfapi: add mem_pools_init and mem_pools_fini calls

2017-07-05T12:27:30+00:00

These are both needed because GFAPI clients are a bit unique.  They can
be long-running, so they can potentially benefit from the memory
reclamation that mem_pools_init will enable.  On the other hand, they
might completely tear down their Gluster environment well before the
process exits, so they need mem_pools_fini as well.

Our main and auxiliary daemons do need mem_pools_init but don't need to
call it themselves because that's handled for them in glusterfsd.c right
after they daemonize.  They don't need mem_pools_fini, because the only
place they could *safely* call it is right before exit and then it won't
do them any good.  Transient processes (e.g. gf_attach) don't benefit
from either because, well, they're transient.  They'll be gone before
memory reclamation matters.

Change-Id: I611cf77d8b408b3ecfc00664e8da294edde9e57a
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/17666
Smoke: Gluster Build System 
Tested-by: Jeff Darcy 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy