glusterfs.git/libglusterfs/src/statedump.c, branch v3.12.5

core: add more information on dictionary usage

2017-06-05T12:44:28+00:00

when you take the 'statedump', it shows the output like below

-----
[dict]
max-number-of-dict-pairs=13
total-pairs-used=41613
total-dict-used=12629
average-pairs-per-dict=3
------

Updates #220

Change-Id: I71a7eda3a3cd23edf4483234f22f983923bbb081
Signed-off-by: Amar Tumballi 
Reviewed-on: https://review.gluster.org/4035
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy

libglusterfs: make memory pools more thread-friendly

2017-02-02T18:30:19+00:00

Early multiplexing tests revealed *massive* contention on certain
pools' global locks - especially for dictionaries and secondarily for
call stubs.  For the thread counts that multiplexing can create, a
more lock-free solution is clearly needed.  Also, the current mem-pool
implementation does a poor job releasing memory back to the system,
artificially inflating memory usage to match whatever the worst case
was since the process started.  This is bad in general, but especially
so for multiplexing where there are more pools and a major point of
the whole exercise is to reduce memory consumption.

The basic ideas for the new design are these

  There is one pool, globally, for each power-of-two size range.
  Every attempt to create a new pool within this range will instead
  add a reference to the existing pool.

  Instead of adding pools for each translator within each multiplexed
  brick (potentially infinite and quite possibly thousands), we
  allocate one set of size-based pools per *thread* (hundreds at
  worst).

  Each per-thread pool is divided into hot and cold lists.  Every
  allocation first attempts to use the hot list, then the cold list.
  When objects are freed, they always go on the hot list.

  There is one global "pool sweeper" thread, which periodically
  reclaims everything in each pool's cold list and then "demotes" the
  current hot list to be the new cold list.

  For normal allocation activity, only a per-thread lock need be
  taken, and even that only to guard against very rare contention from
  the pool sweeper.  When threads start and stop, a global lock must
  be taken to add them to the pool sweeper's list.  Lock contention is
  therefore extremely low, and the hot/cold lists also provide good
  locality.

A more complete explanation (of a similar earlier design) can be found
here:

 http://www.gluster.org/pipermail/gluster-devel/2016-October/051160.html

Change-Id: I5bc8a1ba57cfb553998f979a498886e0d006e665
BUG: 1385758
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/15645
Reviewed-by: Xavier Hernandez 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

libglusterfs: fix glusterd statedump crash

2016-08-04T15:44:09+00:00

commit 3c04a91 removed setting typeStr to NULL if num_allocs is set to 0, this
has caused this regression. Code has been put back like earlier and to avoid
statedump printing all the NULL values check is modified to see skip the records
if num_allocs is 0 instead of total_allocs

Change-Id: Ib8bcc2fba908e88cf52b641c3f6bcba74f5e667c
BUG: 1359190
Signed-off-by: Atin Mukherjee 
Reviewed-on: http://review.gluster.org/14987
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: N Balachandran 
Reviewed-by: Prashanth Pai 
Reviewed-by: Jeff Darcy

statedump: Prevent (null) typestr to be printed

2016-04-24T13:44:44+00:00

Problem:
After the commits: 7e44c783ad731856956929f6614bbe045c26ea3a - lock: use
spinlock only on multicore systems
a6aecae2cd8171b8538bfe5d2800bdd157380b85 - nfs: fix lock variable type

we see a lot of "[global.glusterfs - usage-type (null) memusage]" in statedump
because lock status is not all-zeros after init, and the memcmp to check that
a datatype is never allocated is invalid.

Fix:
Changed if a datatype is allocated or not check based on total_allocs. Also
removed setting typestr to NULL on gf_free even when num_allocs is 0. Because
even that is leading to 'null' memusage string to be printed in statedump.

BUG: 1329870
Change-Id: If2b01a557cbdc787625db32e276e06cee3ac46ee
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/14054
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

core: use syscall wrappers instead of direct syscalls - libglusterfs

2015-11-16T14:18:10+00:00

various xlators and other components are invoking system calls
directly instead of using the libglusterfs/syscall.[ch] wrappers.

If not using the system call wrappers there should be a comment
in the source explaining why the wrapper isn't used.

Change-Id: Ieeca2d36adbc884e4cfa0026dba40df70310d40b
BUG: 1267967
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: http://review.gluster.org/12275
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Jeff Darcy

libglusterfs: Fixing insecure temp file coverity issue

2015-06-12T07:48:11+00:00

Coverity CID: 1124835

Change-Id: I7e87f2b3bad35cf8a9c64c8502de23662d9f677f
BUG: 789278
Signed-off-by: Nandaja Varma 
Reviewed-on: http://review.gluster.org/9565
Tested-by: Gluster Build System 
Reviewed-by: Sakshi Bansal
Tested-by: NetBSD Build System 
Reviewed-by: Prashanth Pai

core: use reference counting for mem_acct structures

2015-05-09T13:28:13+00:00

When freeing memory, our memory-accounting code expects to be able to
dereference from the (previously) allocated block to its owning
translator.  However, as we have already found once in option
validation and twice in logging, that translator might itself have
been freed and the dereference attempt causes on of our daemons to
crash with SIGSEGV.  This patch attempts to fix that as follows:

 * We no longer embed a struct mem_acct directly in a struct xlator,
   but instead allocate it separately.

 * Allocated memory blocks now contain a pointer to the mem_acct
   instead of the xlator.

 * The mem_acct structure contains a reference count, manipulated in
   both the normal and translator allocate/free code using atomic
   increments and decrements.

 * Because it's now a separate structure, we can defer freeing the
   mem_acct until its reference count reaches zero (either way).

 * Some unit tests were disabled, because they embedded their own
   copies of the implementation for what they were supposedly testing.
   Life's too short to spend time fixing tests that seem designed to
   impede progress by requiring a certain implementation as well as
   behavior.

Change-Id: Id929b11387927136f78626901729296b6c0d0fd7
BUG: 1211749
Signed-off-by: Jeff Darcy 
Reviewed-on: http://review.gluster.org/10417
Tested-by: Gluster Build System 
Reviewed-by: Krishnan Parthasarathi 
Reviewed-by: Niels de Vos 
Reviewed-by: Pranith Kumar Karampuri

statedump: Print curr_stdalloc in mempool statedump ...

2014-08-30T02:31:41+00:00

 ...for, it is curr_stdalloc that is incremented for every mem_get()
and decremented on every call to mem_put() and can be used to detect
leaks, when cold_count is 0.

Change-Id: I418a132c3ea4c0b99ea5c6840ff3024d8d19ddf4
BUG: 1134221
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/8557
Tested-by: Gluster Build System 
Reviewed-by: Humble Devassy Chirammal 
Reviewed-by: Pranith Kumar Karampuri 
Tested-by: Pranith Kumar Karampuri

statedump: Don't print mem-type numbers

2014-07-07T04:21:58+00:00

Change-Id: I381bfa9535fe60c37758761d34b98dbbc4e5f569
BUG: 1114188
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/8239
Reviewed-by: Krishnan Parthasarathi 
Reviewed-by: Humble Devassy Chirammal 
Tested-by: Justin Clift 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

meta: export process state dump

2014-05-06T00:29:27+00:00

Various parts of process state dump are exposed
through most appropriate virual files/dirs

Change-Id: Iff30f8ea76552f2f6d9c98def688c5ec4bc6f2af
BUG: 1089216
Signed-off-by: Anand Avati 
Reviewed-on: http://review.gluster.org/7542
Tested-by: Gluster Build System 
Reviewed-by: Harshavardhana