glusterfs.git/libglusterfs/src/common-utils.c, branch v3.11.3

libglusterfs: extract some functionality to functions

2017-06-20T13:44:34+00:00

- code in run.c to close all file descriptors,
  except for specified ones is extracted to

    int close_fds_except (int *fdv, size_t count);

- tokenizing and editing a string that consists
  of comma-separated tokens (as done eg. in
  mount_param_to_flag() of contrib/fuse/mount.c
  is abstacted into the following API:

    char *token_iter_init (char *str, char sep, token_iter_t *tit);
    gf_boolean_t next_token (char **tokenp, token_iter_t *tit);
    void drop_token (char *token, token_iter_t *tit);

Updates #153

> Change-Id: I7cb5bda38f680f08882e2a7ef84f9142ffaa54eb
> Signed-off-by: Csaba Henk 
> Reviewed-on: https://review.gluster.org/17229
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Amar Tumballi 
(cherry picked from commit 98db583e9b2e7aa8e095a75a6bb5f42b0d65ae79)

Change-Id: I7cb5bda38f680f08882e2a7ef84f9142ffaa54eb
Signed-off-by: Csaba Henk 
Reviewed-on: https://review.gluster.org/17569
Tested-by: Amar Tumballi 
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Amar Tumballi

libglusterfs : Fix crash in glusterd while peer probing

2017-06-13T14:20:32+00:00

glusterd crashes when port is being set explcitly to a
range which is outside greater than short data type range.
Eg. sysctl net.ipv4.ip_local_reserved_ports="49152-49156"
In above case glusterd crashes while parsing the port.

With this fix glusterd will be able to handle port range
between INT_MIN to INT_MAX

> Reviewed-on: https://review.gluster.org/17359
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Samikshan Bairagya 
> Reviewed-by: Atin Mukherjee 
> Reviewed-by: Niels de Vos 
> Reviewed-by: Jeff Darcy 
Change-Id: I7c75ee67937b0e3384502973d96b1c36c89e0fe1
BUG: 1459759
Signed-off-by: Gaurav Yadav 
Reviewed-on: https://review.gluster.org/17496
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Samikshan Bairagya 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

glusterd: socketfile & pidfile related fixes for brick multiplexing feature

2017-05-10T14:05:52+00:00

Problem: While brick-muliplexing is on after restarting glusterd, CLI is
         not showing pid of all brick processes in all volumes.

Solution: While brick-mux is on all local brick process communicated through one
          UNIX socket but as per current code (glusterd_brick_start) it is trying
          to communicate with separate UNIX socket for each volume which is populated
          based on brick-name and vol-name.Because of multiplexing design only one
          UNIX socket is opened so it is throwing poller error and not able to
          fetch correct status of brick process through cli process.
          To resolve the problem write a new function glusterd_set_socket_filepath_for_mux
          that will call by glusterd_brick_start to validate about the existence of socketpath.
          To avoid the continuous EPOLLERR erros in  logs update socket_connect code.

Test:     To reproduce the issue followed below steps
          1) Create two distributed volumes(dist1 and dist2)
          2) Set cluster.brick-multiplex is on
          3) kill glusterd
          4) run command gluster v status
          After apply the patch it shows correct pid for all volumes

> BUG: 1444596
> Change-Id: I5d10af69dea0d0ca19511f43870f34295a54a4d2
> Signed-off-by: Mohit Agrawal 
> Reviewed-on: https://review.gluster.org/17101
> Smoke: Gluster Build System 
> Reviewed-by: Prashanth Pai 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Atin Mukherjee 
> (cherry picked from commit 21c7f7baccfaf644805e63682e5a7d2a9864a1e6)

Change-Id: Ia95b9d36e50566b293a8d6350f8316dafc27033b
BUG: 1449004
Signed-off-by: Mohit Agrawal 
Reviewed-on: https://review.gluster.org/17212
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Atin Mukherjee 
Reviewed-by: Prashanth Pai 
CentOS-regression: Gluster Build System

Halo Replication feature for AFR translator

2017-05-08T05:37:07+00:00

	Backport of https://review.gluster.org/16177
		    https://review.gluster.org/17174

Merged both these patches to make sure IPV6 changes don't make it to 3.11 at all.

Summary:
Halo Geo-replication is a feature which allows Gluster or NFS clients to write
locally to their region (as defined by a latency "halo" or threshold if you
like), and have their writes asynchronously propagate from their origin to the
rest of the cluster.  Clients can also write synchronously to the cluster
simply by specifying a halo-latency which is very large (e.g. 10seconds) which
will include all bricks.

In other words, it allows clients to decide at mount time if they desire
synchronous or asynchronous IO into a cluster and the cluster can support both
of these modes to any number of clients simultaneously.

There are a few new volume options due to this feature:
  halo-shd-latency:  The threshold below which self-heal daemons will
  consider children (bricks) connected.

  halo-nfsd-latency: The threshold below which NFS daemons will consider
  children (bricks) connected.

  halo-latency: The threshold below which all other clients will
  consider children (bricks) connected.

  halo-min-replicas: The minimum number of replicas which are to
  be enforced regardless of latency specified in the above 3 options.
  If the number of children falls below this threshold the next
  best (chosen by latency) shall be swapped in.

New FUSE mount options:
  halo-latency & halo-min-replicas: As descripted above.

This feature combined with multi-threaded SHD support (D1271745) results in
some pretty cool geo-replication possibilities.

Operational Notes:
- Global consistency is gaurenteed for synchronous clients, this is provided by
  the existing entry-locking mechanism.
- Asynchronous clients on the other hand and merely consistent to their region.
  Writes & deletes will be protected via entry-locks as usual preventing
  concurrent writes into files which are undergoing replication.  Read operations
  on the other hand should never block.
- Writes are allowed from _any_ region and propagated from the origin to all
  other regions.  The take away from this is care should be taken to ensure
  multiple writers do not write the same files resulting in a gfid split-brain
  which will require resolution via split-brain policies (majority, mtime &
  size).  Recommended method for preventing this is using the nfs-auth feature to
  define which region for each share has RW permissions, tiers not in the origin
  region should have RO perms.

TODO:
- Synchronous clients (including the SHD) should choose clients from their own
  region as preferred sources for reads.  Most of the plumbing is in place for
  this via the child_latency array.
- Better GFID split brain handling & better dent type split brain handling
  (i.e. create a trash can and move the offending files into it).
- Tagging in addition to latency as a means of defining which children you wish
  to synchronously write to

Test Plan:
- The usual suspects, clang, gcc w/ address sanitizer & valgrind
- Prove tests

Reviewers: jackl, dph, cjh, meyering

Reviewed By: meyering

Subscribers: ethanr

Differential Revision: https://phabricator.fb.com/D1272053

Tasks: 4117827

 >Change-Id: I694a9ab429722da538da171ec528406e77b5e6d1
 >BUG: 1428061
 >Signed-off-by: Kevin Vigor 
 >Reviewed-on: http://review.gluster.org/16099
 >Reviewed-on: https://review.gluster.org/16177
 >Tested-by: Pranith Kumar Karampuri 
 >Smoke: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >CentOS-regression: Gluster Build System 
 >Reviewed-by: Pranith Kumar Karampuri 

BUG: 1448416
Change-Id: I694a9ab429722da538da171ec528406e77b5e6d1
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/17192
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kaushal M

build: clang has __builtin_popcount() and __builtin_ffs()

2017-04-05T06:12:53+00:00

Note: Even though gcc(1) will automatically treat ffs() and popcount()
as built-in, calling them explicitly as __builtin in the source helps
make it easier to find them. (And if no __builtin_ffs use the one in
libc.)

Change-Id: Ib74d9b221ff03a01df5ad05907024da1a83a7a88
BUG: 1438772
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: https://review.gluster.org/16993
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy 
Reviewed-by: Xavier Hernandez

libglusterfs: Initialize old sigset

2017-03-27T05:08:56+00:00

...in gf_thread_create(). Technically, since we pass it as an argument
to pthread_sigmask, initialization is not needed but doing it as a good
practice.

Change-Id: Ie069af07cb07c1784f3841e1fc628ca13dfdcef4
BUG: 1434274
Signed-off-by: Ravishankar N 
Reviewed-on: https://review.gluster.org/16929
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Ashish Pandey 
Reviewed-by: Pranith Kumar Karampuri

posix: use nanosecond accuracy when available

2017-03-07T17:10:02+00:00

Programs that set mtime, such as `rsync -a`, don't work correctly on
GlusterFS, because it sets the nanoseconds to 000. This creates
problems for incremental backups, where files get accidentally copied
again and again.

For example, consider `myfile` on an ext4 system, being copied to a
GlusterFS volume, with `rsync -a` and then `cp -u` in turn. You'd
expect that after the first `rsync -a`, `cp -u` agrees that the file
need not be copied.

BUG: 1422074
Change-Id: I89c7b6a73e2e06c02851ff76b7e5cdfaa271e985
Signed-off-by: Niklas Hambüchen 
Reviewed-on: https://review.gluster.org/16667
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
Tested-by: Jeff Darcy 
NetBSD-regression: NetBSD Build System 
Reviewed-by: jiffin tony Thottan 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy

libglusterfs, gfdb, glusterfs: Add missing breaks

2017-02-26T16:29:26+00:00

A few switches did not have breaks causing fall throughs. Most of them
have been fixed with fall through comments for those that are
intentional.

Change-Id: I84c85726b542f38504b50fefab5eba5dbcd27a07
BUG: 1424894
Signed-off-by: Nigel Babu 
Reviewed-on: https://review.gluster.org/16677
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Jeff Darcy 
Reviewed-by: Shyamsundar Ranganathan

Make sure that buffer is null terminated

2017-02-24T14:37:50+00:00

Coverity warn about it, and while that's unlikely
to be a issue in practice, it is rather important
to not mask more critical problems with false positive.

Change-Id: Ibee1a9c37e216635077f05d5ef5de55ad5e0b051
BUG: 789278
Signed-off-by: Michael Scherer 
Reviewed-on: https://review.gluster.org/16727
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
Tested-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

libglusterfs+transport+io-threads: fix 256KB stack abuse

2017-02-02T00:59:25+00:00

Some functions were allocating 64K booleans, which are (crazily) mapped to
4-byte ints, for a total of 256KB per call.  Changed to use bitfields instead,
so usage is now only 8KB per call.  This was the impediment to changing the
io-threads stack size, so that has been adjusted too.

Change-Id: I8781c4f2c8f2b830f4535e366995fac8dd0a8653
BUG: 1418095
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/15745
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: N Balachandran 
Reviewed-by: Shyamsundar Ranganathan