glusterfs.git/tests/features, branch v3.8rc0

WORM/Retention Translator: Implementation of file level WORM

2016-05-02T01:05:22+00:00

To activate the file level worm feature, the features.read-only and
features.worm options should be switched "off" on the volume and
the features.worm-file-level should be switched "on". Both read-only
and worm or worm-file-level cannot be switched "on" together. The
files which are created when the worm-file-level option is set on the
volume will have their own retention profile.

If both worm and worm-file-level are "on" at that time the worm
which is the volume level worm will have priority over file level
worm. If worm-file level is switched "off" after some time and the
read-only option is switched "on" then read-only will have priority.

The current implementation allows the users to manually transmit
a file to a WORM-Retained state by removing all the write bits of
the file using the chmod command. The file will have a retention
profile which contains the state of the file, mode of retention,
and the default retention time.

The file will be made WORM-Retained for a default of 120 seconds
during which it will be immutable and undeletable and it sets the
atime of the file to the time till which it is retained.
After that period if any fop request comes for that file, will
make the transition from WORM-Retained state to WORM state, where
the file will be immutable but deletable and, it will reset
the atime to the actual atime of the file. If a WORM file needs
to be made undeletable again, it can be done by using the chmod
command with all the write bits removed.

There are two modes of retention:
1. Relax: where the retention time of a WORM-Retained file can be
   increased or decreased.
2. Enterprise: where the retention time of a WORM-Retained file
   can be increased but not be decreased.
Whenever a utime change(touch -a, -t, ...)request comes for a
file it checks the mode of retention before setting the utimes.
This is done only if the file is WORM-Retained but for a WORM file
it will change the utimes.

Lazy auto commit:
Whenever a file gets created it will store the creation time of the
file or if a file already exists then any of the next unlink, link,
truncate or rename fops will set the current time as the start time
in an xattr. The next rename/unlink/truncate/link call will check for the
auto commit period and if is is expired, then it will automatically do
the state transition. If it is a normal file then it gets converted
to WORM-Retained state. If it is a WORM-Retained file and its retention
period is expired, then it gets converted to WORM state.

Added the volume set options for the WORM translator. It allows the users
to change the default values of auto-commit-period, default-retention-period,
retention-mode. To make use of the file-level WORM first we have to set the
'worm-file' option to 'on'. The files which are created when the worm-file
option is set on the volume will get WORM-Retained. Other files will work
as usual and will not be WORMed. The auto-commit-period, retention-mode,
and the default-retention-period values for the file will be set to the values
which are set on the volume when the file is created.

Added the tests to check the basic functionalities of the WORM/Retention feature.

Change-Id: I77bd9777f9395a944d76b5cc35a5b48a3c14d148
BUG: 1326308
Signed-off-by: karthik-us 
Reviewed-on: http://review.gluster.org/13429
Reviewed-by: Niels de Vos 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

glusterd: volume set changes for lock migration

2016-05-02T01:05:03+00:00

Change-Id: I48c6f9cdda47503615ba65882acd5eedf0a70c89
BUG: 1326085
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/14024
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

features/trash: wind mkdir with special pid

2016-04-27T21:24:48+00:00

Recent changes done w.r.t handling of mkdir calls in posix translator
resulted in crashing the brick process from trash translator. This was
due to the changes made in posix translator to return EPERM for every
mkdir calls without 'gfid-req' set in dictionary. In order to avoid
gfid mismatches during directory creation from brick side trash
translator does not set 'gfid-req'. This patch is to have an exemption
for trash based on a special pid set for those mkdir calls originating
from trash translator and to reset it in callback.

This patch also includes a small optimization to the existing test case
for trash feature.

Change-Id: I59f084ac875e54342ecf2bffa6e43ebd84814153
BUG: 1317361
Signed-off-by: Anoop C S 
Reviewed-on: http://review.gluster.org/13776
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy

Revert "glusterd: Allocate fresh port on brick (re)start"

2016-04-14T11:20:53+00:00

This reverts commit 34899d7

Commit 34899d7 introduced a change, where restarting a volume or rebooting
a node result into fresh allocation of brick port. In production
environment generally administrator makes firewall configuration for a
range of ports for a volume. With commit 34899d7, on rebooting of node
or restarting a volume might result into volume start fail because
firewall might block fresh allocated port of a brick and also it will be
difficult in testing because of fresh allocation of port.

Change-Id: I7a90f69e8c267a013dc906b5228ca76e819d84ad
BUG: 1322805
Signed-off-by: Gaurav Kumar Garg 
Reviewed-on: http://review.gluster.org/13989
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee

tests: use trap mechanism to ensure that proper cleanups happen

2016-04-12T12:04:24+00:00

This actually consists of several parts.

  * Added a generic cleanup-scheduling mechanism.  Instead of calling
    "trap ... EXIT" directly, just call "push_trapfunc ..." instead and
    your cleanup function will be called along with any others.

  * Converted a few tests to use push_trapfunc.

  * Added "push_trapfunc cleanup_lvm" to snapshot.rc to address the
    particular problem that's driving this - snapshot tests not calling
    cleanup_lvm on their own and leaving bad state for the next test.

Change-Id: I548a97a26328390992fc71ee1f03c0463703f9d7
Signed-off-by: Jeff Darcy 
Reviewed-on: http://review.gluster.org/13933
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee 
Reviewed-by: Rajesh Joseph

dht: extend time for "nuke" test's janitor-cleanup check

2016-04-12T11:20:53+00:00

The previous check worked out to 50 deletes per second.  That might have
seemed generous, but NetBSD regression tests were failing because it
can't hit that figure reliably.

Change-Id: Ifbd8f4547caf53a8a8d11ad586aa8051f77ddc40
Signed-off-by: Jeff Darcy 
Reviewed-on: http://review.gluster.org/13935
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

dht: add "nuke" functionality for efficient server-side deletion

2016-04-07T15:07:55+00:00

This turns a special xattr into an rmdir with flags set.  When that hits
the posix translator on the server side, that causes the file/directory
to be moved into the special "landfill" directory.  From there, the
posix janitor thread will take care of deleting it entirely on the
server side - traversing it recursively if necessary.  A couple of
secondary issues were fixed to make this effective.

 * FUSE now ensures that setxattr values are NUL terminated.

 * The janitor thread now gets woken up immediately when something is
   placed in 'landfill' instead of only when file descriptors need to be
   closed.

 * The default landfill-emptying interval was reduced to 10s.

To use the feature, issue a setxattr something like this:

   setfattr -n glusterfs.dht.nuke -v "" /mnt/glusterfs/vol/some_dir

The value doesn't actually matter; the mere receipt of a request with
this key is sufficient.  Some day it might be useful to allow setting a
required value as a sort of password, so that only those who know it can
access the underlying special functionality.

Change-Id: I8a343c2cdb40a76d5a06c707191fb67babb8514f
Signed-off-by: Jeff Darcy 
Reviewed-on: http://review.gluster.org/13878
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

glusterd: Allocate fresh port on brick (re)start

2016-04-01T20:38:54+00:00

There is no point of using the same port through the entire volume life cycle
for a particular bricks process since there is no guarantee that the same port
would be free and no other application wouldn't consume it in between the
glusterd/volume restart.

We hit a race where on glusterd restart the daemon services start followed by
brick processes and the time brick process tries to bind with the port which was
allocated by glusterd before a restart is been already consumed by some other
client like NFS/SHD/...

Note : This is a short term solution as here we reduce the race window but don't
eliminate it completely. As a long term solution the port allocation has to be
done by glusterfsd and the same should be communicated back to glusterd for book
keeping

Change-Id: Ibbd1e7ca87e51a7cd9cf216b1fe58ef7783aef24
BUG: 1322805
Signed-off-by: Atin Mukherjee 
Reviewed-on: http://review.gluster.org/13865
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy

experimental: add fdl (Full Data Logging) translator

2016-02-13T13:13:07+00:00

NSR needs logging that is different than our existing changelog in
several ways:

 * Full data, not just metadata

 * Pre-op, not post-op

 * High performance

 * Supports the concept of time-bounded "terms"

Others (for example EC) might need the same thing.  This patch adds such
a translator.  It also adds code to dump the resulting journals, and to replay
them using syncops, plus (very rudimentary) tests for all of the above.

Change-Id: I29680a1b4e0a9e7d5a8497fef302c46434b86636
Signed-off-by: Jeff Darcy 
Reviewed-on: http://review.gluster.org/12450
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

SSL improvements: ECDH, DH, CRL, and accessible options

2015-08-05T11:51:43+00:00

- Introduce ssl.dh-param option to specify a file containinf DH parameters.
  If it is provided, EDH ciphers are available.

- Introduce ssl.ec-curve option to specify an elliptic curve name. If
  unspecified, ECDH ciphers are available using the prime256v1 curve.

- Introduce ssl.crl-path option to specify the directory where the
  CRL hash file can be found. Setting to NULL disable CRL checking,
  just like the default.

- Make all ssl.* options accessible through gluster volume set.

- In default cipher list, exclude weak ciphers instead of listing
  the strong ones.

- Enforce server cipher preference.

- introduce RPC_SET_OPT macro to factor repetitive code in glusterd-volgen.c

- Add ssl-ciphers.t test to check all the features touched by this change.

Change-Id: I7bfd433df6bbf176f4a58e770e06bcdbe22a101a
BUG: 1247152
Signed-off-by: Emmanuel Dreyfus 
Reviewed-on: http://review.gluster.org/11735
Tested-by: NetBSD Build System 
Reviewed-by: Kaushal M 
Tested-by: Gluster Build System 
Reviewed-by: Jeff Darcy