glusterfs.git/rpc, branch v3.12.5

rpc-transport/rdma: Add a mutex for the list of RDMA Memory Region(MR) access

2018-01-02T09:50:51+00:00

Problem: gf_rdma_device_t->all_mr is a __gf_rdma_arena_mr(includes MR content)
	 kind of list in the rdma rpc-transport. The rdma rpc-transport will
	 add/delete items to the list when MRs register, deregister, and free.
	 Because gf_rdma_device_t->all_mr is used by different threads
	 and it is not mutex protected, rdma transport maybe access obsolete
	 items in it.

Solution: Add a mutex protection for the gf_rdma_device_t->all_mr.

> Change-Id: I2b7de0f7aa516b90bb6f3c6aae3aadd23b243900
> BUG: 1522651
> Signed-off-by: Yi Wang 

(cherry picked from commit 8483ed87165c1695b513e223549d33d2d63891d9)
Change-Id: I2b7de0f7aa516b90bb6f3c6aae3aadd23b243900
BUG: 1525850
Signed-off-by: Yi Wang

build: make it possible to build cleanly 2x in a row

2017-10-25T11:40:18+00:00

'make clean' does not cleanup everything, and some of the files get
cleaned too eagerly. Several files are being packaged in a 'make dist'
tarball, that get rebuild each time anyway.

Specifically, this change prevents
 - libglusterfs/src/generator.pyc from laying around
 - keeping rpc/xdr/gen/*.x symlinks
 - modifying tests/basic/{fuse,gfapi}/Makefile each run
 - including tests/env.rc and events/src/eventtypes.py in the tarball

Cherry picked from commit a413c6353dbfff5221ea020ff8e1475d5ee17a81:
> Change-Id: I774dd1abf3a9d3b6a89b938cf6ee7d7792c59a82
> BUG: 1501317
> Reported-by: Patrick Matthäi 
> Signed-off-by: Niels de Vos 

Change-Id: I774dd1abf3a9d3b6a89b938cf6ee7d7792c59a82
BUG: 1494527
Reported-by: Patrick Matthäi 
Signed-off-by: Niels de Vos

rpc: TLSv1_2_method() is deprecated in OpenSSL-1.1

2017-10-06T06:32:50+00:00

Fedora 26 has OpenSSL-1.1. Compile-time warnings indicate
that TLSv1_2_method() is now deprecated. As per the SSL man page:

  TLS_method(), TLS_server_method(), TLS_client_method()
    These are the general-purpose version-flexible SSL/TLS methods.
    The actual protocol version used will be negotiated to the highest
    version mutually supported by the client and the server. The
    supported protocols are SSLv3, TLSv1, TLSv1.1 and TLSv1.2.
    Applications should use these methods, and avoid the version-
    specific methods described below.
  ...
  TLSv1_2_method(), ...
  ...

Note that OpenSSL-1.1 is the version of OpenSSL; Fedora 25 and RHEL 7.3
and other distributions (still) have OpenSSL-1.0.

TLS versions are orthogonal to the OpenSSL version.  TLS_method() is the
new — in OpenSSL-1.1 — version flexible function intended to replace the
TLSv1_2_method() function in OpenSSL-1.0 and the older (?), insecure
TLSv23_method(). (OpenSSL-1.0 does not have TLS_method())

master https://review.gluster.org/18268
master BZ: 1491025

Change-Id: I190363ccffe7c25606ea2cf30a6b9ff1ec186057
BUG: 1491690
Signed-off-by: Kaleb S. KEITHLEY

rpc: destroy transport after client_t

2017-09-07T06:49:51+00:00

Problem:
1. Ref counting increment on the client_t object is done in
   rpcsvc_request_init() which is incorrect.
2. Ref not taken when delegating to grace_time_handler()

Solution:
1. Only fop requests which require processing down the graph via
   stack 'frames' now ref count the request in get_frame_from_request()
2. Take ref on client_t object in server_rpc_notify() but avoid
   dropping in RPCSVC_EVENT_TRANSPORT_DESRTROY. Drop the ref
   unconditionally when exiting out of grace_time_handler().
   Also, avoid dropping ref on client_t in
   RPCSVC_EVENT_TRANSPORT_DESTROY when ref mangement as been
   delegated to grace_time_handler()

mainline:
> BUG: 1481600
> Reported-by: Raghavendra G 
> Signed-off-by: Milind Changire 
> Reviewed-on: https://review.gluster.org/17982
> Tested-by: Raghavendra G 
> Reviewed-by: Raghavendra G 
> CentOS-regression: Gluster Build System 
> Smoke: Gluster Build System 
(cherry picked from commit 24b95089a18a6a40e7703cb344e92025d67f3086)

Change-Id: Ic16246bebc7ea4490545b26564658f4b081675e4
BUG: 1487033
Reported-by: Raghavendra G 
Signed-off-by: Milind Changire 
Reviewed-on: https://review.gluster.org/18156
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G 
Tested-by: Raghavendra G 
CentOS-regression: Gluster Build System

tier: separation of attach-tier from add-brick

2017-08-04T20:34:03+00:00

PROBLEM: Both attach tier and add brick have the same RPC
and set of code. This becomes a hurdle while tring to implement
add brick on a tiered volume.

FIX: This patch separates the add brick and attach tier
giving them separate RPCs.

>Change-Id: Iec57e972be968a9ff00b15b507e56a4f6dc398a2
>BUG: 1376326
>Signed-off-by: hari gowtham 
>Reviewed-on: https://review.gluster.org/15503
>Smoke: Gluster Build System 
>Tested-by: hari gowtham 
>Reviewed-by: Samikshan Bairagya 
>CentOS-regression: Gluster Build System 

Signed-off-by: hari gowtham 
Change-Id: Iec57e972be968a9ff00b15b507e56a4f6dc398a2
BUG: 1478276
Reviewed-on: https://review.gluster.org/17974
Smoke: Gluster Build System 
Tested-by: hari gowtham 
Reviewed-by: Shyamsundar Ranganathan 
CentOS-regression: Gluster Build System

glusterd: Add option to get all volume options through get-state CLI

2017-07-31T17:34:26+00:00

This commit makes the get-state CLI capable to returning the values
for all volume options for all volumes. This is similar to what you
get when you issue a `gluster volume get  all` command.

This is the new usage for the get-state CLI:

 # gluster get-state [] [[odir ] \
[file ]] [detail|volumeoptions]

> Reviewed-on: https://review.gluster.org/17858
> CentOS-regression: Gluster Build System 
> Reviewed-by: Atin Mukherjee 
> Reviewed-by: Gaurav Yadav 
> Smoke: Gluster Build System 

(cherry picked from commit 8dcf91660e0bd10eb75ef25a29ca02ec51c81be4)

Change-Id: Ice52d936a5a389c6fa0ba5ab32416a65cdfde46d
Fixes: #277
Signed-off-by: Samikshan Bairagya 
Reviewed-on: https://review.gluster.org/17874
Reviewed-by: Atin Mukherjee 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

rpc: include current second in timed out frame cleanup

2017-07-21T06:22:21+00:00

Problem:
frames which time out at current second are missed out

Solution:
change test to include frames timing out at current second
i.e. timeout <= current.tv_sec
instead of
timeout < current.tv_sec

Change-Id: I459d47856ade2b657a0289e49f7f63da29186d6e
BUG: 1468433
Signed-off-by: Milind Changire 
Reviewed-on: https://review.gluster.org/17722
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

libglusterfs: Name threads on creation

2017-07-19T14:16:19+00:00

Set names to threads on creation for easier
debugging.

Output of top -H -p 
Before:
19773 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19774 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19775 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19776 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19777 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19778 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19779 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19780 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19781 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19782 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19783 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19784 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19785 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterfsd
19786 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterfsd
19787 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterfsd
19789 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19790 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
25178 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
 5398 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
 7881 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd

After:
19773 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19774 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustertimer
19775 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19776 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustermemsweep
19777 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustersproc0
19778 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustersproc1
19779 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterepoll0
19780 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusteridxwrker
19781 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusteriotwr0
19782 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterbrssign
19783 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterbrswrker
19784 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterclogecon
19785 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterclogd0
19786 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterclogd1
19787 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterclogd2
19789 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterposixjan
19790 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterposixfsy
25178 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterepoll1
 5398 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterepoll2
 7881 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterposixhc

Change-Id: Id5f333755c1ba168a2ffaa4fce6e71c375e10703
BUG: 1254002
Updates: #271
Signed-off-by: Raghavendra Talur 
Reviewed-on: https://review.gluster.org/11926
Reviewed-by: Prashanth Pai 
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System

program/GF-DUMP: Shield ping processing from traffic to Glusterfs

2017-07-18T10:45:29+00:00

Program

Since poller thread bears the brunt of execution till the request is
handed over to io-threads, poller thread experiencies lock
contention(s) in the control flow till io-threads, which slows it
down. This delay invariably affects reading ping requests from network
and responding to them, resulting in increased ping latencies, which
sometimes results in a ping-timer-expiry on client leading to
disconnect of transport. So, this patch aims to free up poller thread
from executing code of Glusterfs Program. We do this by making

* Glusterfs Program registering itself asking rpcsvc to execute its
  actors in its own threads.
* GF-DUMP Program registering itself asking rpcsvc to _NOT_ execute
  its actors in its own threads. Otherwise program's ownthreads become
  bottleneck in processing ping traffic. This means that poller thread
  reads a ping packet, invokes its actor and hands the response msg to
  transport queue.

Change-Id: I526268c10bdd5ef93f322a4f95385137550a6a49
Signed-off-by: Raghavendra G 
BUG: 1421938
Reviewed-on: https://review.gluster.org/17105
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Amar Tumballi 
Reviewed-by: Jeff Darcy

socket: call init_openssl_mt() in init() and add cleanup

2017-07-17T17:09:05+00:00

init_openssl_mt() wasn't explicitly invoked and was run implicitly before
dlopen() returned as it was tagged as __attribute__ ((constructor)). This
function used to call GF_CALLOC() which wasn't available or initialized
when socket.so is dlopen()ed by an external program. The program used to
crash with SIGSEGV as follows:

0x00007ffff5efe1ad in __gf_calloc (nmemb=41, size=40, type=158, typestr=0x7ffff63eb3d6 "gf_sock_mt_lock_array")
at mem-pool.c:109
0x00007ffff63e6acf in init_openssl_mt () at socket.c:4016
0x00007ffff7de90da in call_init.part () from /lib64/ld-linux-x86-64.so.2
0x00007ffff7de91eb in _dl_init () from /lib64/ld-linux-x86-64.so.2
0x00007ffff7dedde1 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
0x00007ffff7de8f84 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
0x00007ffff7ded339 in _dl_open () from /lib64/ld-linux-x86-64.so.2

This change moves call to init_openssl_mt() from being a constructor function
to the init() function and introduces fini_openssl_mt() which cleans up
resources (called in destructor).

BUG: 1193929
Change-Id: Iab690897ec34e24c33f6b43f8d8d9f8fd75ac607
Signed-off-by: Prashanth Pai 
Reviewed-on: https://review.gluster.org/17753
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Amar Tumballi 
Reviewed-by: Jeff Darcy