glusterfs.git/libglusterfs/src/common-utils.h, branch v3.12.5

glusterd : introduce timer in mgmt_v3_lock

2017-11-06T12:16:15+00:00

Problem:
In a multinode environment, if two of the op-sm transactions
are initiated on one of the receiver nodes at the same time,
there might be a possibility that glusterd  may end up in
stale lock.

Solution:
During mgmt_v3_lock a registration is made to  gf_timer_call_after
which release the lock after certain period of time

>mainline patch : https://review.gluster.org/#/c/18437/

Change-Id: I16cc2e5186a2e8a5e35eca2468b031811e093843
BUG: 1503239
Signed-off-by: Gaurav Yadav

glusterd: Add geo-replication session details to get-state output

2017-08-04T13:32:52+00:00

This commit adds support to the get-state CLI to capture details
on geo-replication session as obtained in
`gluster volume geo-replication status detail` in its output.

> Reviewed-on: https://review.gluster.org/17941
> Smoke: Gluster Build System 
> Reviewed-by: Shubhendu Tripathi 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Atin Mukherjee 

(cherry picked from commit 2e7daeffef05c6100cbcc39f1be62935711db3eb)

Fixes: #291
Change-Id: I2fbcba70bfdaf439522637234805545194777ed4
Signed-off-by: Samikshan Bairagya 
Reviewed-on: https://review.gluster.org/17971
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

logging: localtime logging, cmdline, volume set option

2017-08-03T12:04:50+00:00

Despite the fact that appliances generally use UTC, some
users really want log entries in localtime.

fixes gluster/glusterfs#272

feature page: https://review.gluster.org/17807

Backport from master https://review.gluster.org/#/c/16911/

Change-Id: I5fbf2c3eedd9eb128fb3f851dd67b2f4081c8bba
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: https://review.gluster.org/17928
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

libglusterfs: Name threads on creation

2017-07-19T14:16:19+00:00

Set names to threads on creation for easier
debugging.

Output of top -H -p 
Before:
19773 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19774 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19775 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19776 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19777 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19778 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19779 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19780 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19781 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19782 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19783 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19784 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19785 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterfsd
19786 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterfsd
19787 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterfsd
19789 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19790 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
25178 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
 5398 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
 7881 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd

After:
19773 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19774 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustertimer
19775 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19776 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustermemsweep
19777 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustersproc0
19778 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustersproc1
19779 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterepoll0
19780 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusteridxwrker
19781 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusteriotwr0
19782 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterbrssign
19783 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterbrswrker
19784 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterclogecon
19785 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterclogd0
19786 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterclogd1
19787 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterclogd2
19789 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterposixjan
19790 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterposixfsy
25178 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterepoll1
 5398 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterepoll2
 7881 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterposixhc

Change-Id: Id5f333755c1ba168a2ffaa4fce6e71c375e10703
BUG: 1254002
Updates: #271
Signed-off-by: Raghavendra Talur 
Reviewed-on: https://review.gluster.org/11926
Reviewed-by: Prashanth Pai 
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System

common-utils: Remove fop_enum_to_string, get_fop_int

2017-07-18T17:10:26+00:00

Implementation of these two functions becomes easier by using gf_fop_list[]
array. So implemented that and removed usage of these functions.

BUG: 1472250
Change-Id: I8a592913f9eeb02d965708bcf28a637588ed4988
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/17812
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

storage/posix: New gfid2path infra

2017-07-10T14:51:14+00:00

With this infra, a new xattr is stored on each entry
creation as below.

trusted.gfid2path. = /

If there are hardlinks, multiple xattrs would be present.

Fops which are impacted:
create, mknod, link, symlink, rename, unlink

Option to enable:
gluster vol set  storage.gfid2path on

Updates: #139
Change-Id: I369974cd16703c45ee87f82e6c2ff5a987a6cc6a
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/17488
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK 
Reviewed-by: Amar Tumballi

mgtm/core : use sha hash function for volfile check

2017-07-10T05:07:11+00:00

We are storing the entire volfile and using this to check
volfile change. With brick multiplexing there will be lot
of graphs per process which will increase the memory foot
print of the process. So instead of storing the entire
graph we could use sha256 and we can compare the hash to
see whether volfile change happened or not.

Also with Brick multiplexing, the direct comparison of vol
file is not correct. There are two problems.

Problem 1:

We are currently storing one single graph (the last
updated volfile) whereas, what we need is the entire
graph with all atttached bricks.

If we fix this issue, we have second problem

Problem 2:
With multiplexing we have a graph that contains multiple
bricks. But what we are checking as part of the reconfigure
is, comparing the entire graph with one single graph,
which will always fail.

Solution:
We create list in glusterfs_ctx_t that stores sha256 hash
of individual brick graphs. When a graph changes happens
we compare the stored hash and the current hash. If the
hash matches, then no need for reconfigure. Otherwise we
first do the reconfigure and then update the hash.

For now, gfapi has not changed this way. Meaning when gfapi
volfile fetch or reconfigure happens, we still store the
entire graph and compare, each memory.

This is fine, because libgfapi will not load brick graphs.
But changing the libgfapi will make the code similar in
both glusterfsd-mgmt and api. Also it helps to reduce some
memory.

Change-Id: I9df917a771a52b95622ab8f63af34ec390163a77
BUG: 1467986
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: https://review.gluster.org/17709
Smoke: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee 
Reviewed-by: Amar Tumballi

groups: don't allocate auxiliary gid list on stack

2017-07-06T18:26:08+00:00

When glusterfs wants to retrieve the list of auxiliary gids
of a user, it typically allocates a sufficiently big gid_t
array on stack and calls getgrouplist(3) with it. However,
"sufficiently big" means to be of maximum supported gid list
size, which in GlusterFS is GF_MAX_AUX_GROUPS = 64k.
That means a 64k * sizeof(gid_t) = 256k allocation, which is
big enough to overflow the stack in certain cases.

A further observation is that stack allocation of the gid list
brings no gain, as in all cases the content of the gid list
eventually gets copied over to a heap allocated buffer.

So we add a convenience wrapper of getgrouplist to libglusterfs
called gf_getgrouplist which calls getgrouplist with a sufficiently
big heap allocated buffer (it takes care of the allocation too).
We are porting all the getgrouplist invocations to gf_getgrouplist
and thus eliminate the huge stack allocation.

BUG: 1464327
Change-Id: Icea76d0d74dcf2f87d26cb299acc771ca3b32d2b
Signed-off-by: Csaba Henk 
Reviewed-on: https://review.gluster.org/17706
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
Reviewed-by: Amar Tumballi 
CentOS-regression: Gluster Build System

contrib/xxhash: Add xxhash library

2017-06-30T08:16:57+00:00

xxhash is a faster non-cryptographic hash.
https://github.com/Cyan4973/xxHash

Release Taken: "xxHash v0.6.2"
--------------

Files added:
  contrib/xxhash/xxhash.c
  contrib/xxhash/xxhash.h
  contrib/xxhash/xxhsum.c

Modifications to source:
------------------------
Following functions and data types got 'GF_' prefix
as below to avoid any form of name collisions in future.

    ---- Functions ----
    GF_XXH_versionNumber
    GF_XXH32
    GF_XXH32_createState
    GF_XXH32_freeState
    GF_XXH32_copyState
    GF_XXH32_reset
    GF_XXH32_update
    GF_XXH32_digest
    GF_XXH32_canonicalFromHash
    GF_XXH32_hashFromCanonical
    GF_XXH64
    GF_XXH64_createState
    GF_XXH64_freeState
    GF_XXH64_copyState
    GF_XXH64_reset
    GF_XXH64_update
    GF_XXH64_digest
    GF_XXH64_canonicalFromHash
    GF_XXH64_hashFromCanonical

    ---- Data Types ----
    GF_XXH_errorcode
    GF_XXH32_state_t*
    GF_XXH32_canonical_t*
    GF_XXH32_hash_t
    GF_XXH64_state_t*
    GF_XXH64_canonical_t*
    GF_XXH64_hash_t

It is linked with libglusterfs.so. A wrapper
funtion is also added for the easy usage in
common-utils.c.

xxhash can be used for the all the usecases where
a faster non-cryptographic hash is required.
gfid to path infra would be using this for now.

NOTE:
----
The gluster coding guidelines check is ignored
as maintaining it further would be difficult.

Updates: #253
Change-Id: Ib143f90d91d4ee99864a10246d5983e92900173b
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/17641
Smoke: Gluster Build System 
Reviewed-by: Amar Tumballi 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Niels de Vos

libglusterfs: extract some functionality to functions

2017-05-23T13:21:20+00:00

- code in run.c to close all file descriptors,
  except for specified ones is extracted to

    int close_fds_except (int *fdv, size_t count);

- tokenizing and editing a string that consists
  of comma-separated tokens (as done eg. in
  mount_param_to_flag() of contrib/fuse/mount.c
  is abstacted into the following API:

    char *token_iter_init (char *str, char sep, token_iter_t *tit);
    gf_boolean_t next_token (char **tokenp, token_iter_t *tit);
    void drop_token (char *token, token_iter_t *tit);

Updates #153

Change-Id: I7cb5bda38f680f08882e2a7ef84f9142ffaa54eb
Signed-off-by: Csaba Henk 
Reviewed-on: https://review.gluster.org/17229
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Amar Tumballi