glusterfs.git/libglusterfs/src/mem-pool.c, branch v3.7.19

libglusterfs:Now mempool is added to ctx pool list under a lock

2016-11-28T11:56:50+00:00

mempool is added to ctx pool list without any lock. This can cause undefined
behaviour in case of multithreaded environment.

Fix: modify the list only under ctx->lock

> Reviewed-on: http://review.gluster.org/15842
> NetBSD-regression: NetBSD Build System 
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Raghavendra Talur 
> Reviewed-by: Shyamsundar Ranganathan 
> Reviewed-by: Poornima G 
> Reviewed-by: Niels de Vos 
(cherry picked from commit 277008a3a8583ef10cec9e4182960792e56c5c10)

Change-Id: I7bdbb3db48a899bb0e41427e149b13c0facaedba
Signed-off-by: Rajesh Joseph 
BUG: 1397662
Reviewed-on: http://review.gluster.org/15910
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Poornima G 
Reviewed-by: Niels de Vos

libglusterfs: add gf_get_mem_type()

2016-10-03T12:16:44+00:00

gfapi needs to provide a function towards applications to free memory
that it allocated. Depending on how the application is compiled/linked,
it could use a different memory allocator than Gluster itself. Therefore
it is not safe for gfapi to request applications to free memory with
'standard' free().

Examples for this are Gluster allocated structures with GF_CALLOC() when
memory accounting is enabled (the default). Some gfapi functions use
malloc() to allocate memory as a workaround, but the free() from the
jemalloc implementation should not be combined with the malloc() from
glibc.

Cherry picked from commit db4e26ed71a01e5f760fbc3c7051962426f102c9:
> Change-Id: I626cd1a60abf8965f9263290f4045d1f69fc2093
> BUG: 1344714
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/15108
> Smoke: Gluster Build System 
> Reviewed-by: soumya k 
> Reviewed-by: jiffin tony Thottan 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Shyamsundar Ranganathan 

Change-Id: I626cd1a60abf8965f9263290f4045d1f69fc2093
BUG: 1347715
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/15601
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Kaushal M

mem-pool,stack,store,syncop,timer/libglusterfs : Porting to a new logging framework

2015-07-14T09:35:13+00:00

        Backport of http://review.gluster.org/10827

Cherry picked from 2b9b3ef3b646989bbc0412dca187b3f5fcad3283
>Change-Id: Idd3dcaf7eeea5207b3a5210676ce3df64153197f
>BUG: 1194640
>Signed-off-by: Mohamed Ashiq 
>Reviewed-on: http://review.gluster.org/10827
>Reviewed-by: Pranith Kumar Karampuri 
>Tested-by: NetBSD Build System 

Change-Id: Idd3dcaf7eeea5207b3a5210676ce3df64153197f
BUG: 1217722
Signed-off-by: Mohamed Ashiq 
Reviewed-on: http://review.gluster.org/11478
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos

core: use reference counting for mem_acct structures

2015-05-09T21:27:36+00:00

When freeing memory, our memory-accounting code expects to be able to
dereference from the (previously) allocated block to its owning
translator.  However, as we have already found once in option
validation and twice in logging, that translator might itself have
been freed and the dereference attempt causes on of our daemons to
crash with SIGSEGV.  This patch attempts to fix that as follows:

 * We no longer embed a struct mem_acct directly in a struct xlator,
   but instead allocate it separately.

 * Allocated memory blocks now contain a pointer to the mem_acct
   instead of the xlator.

 * The mem_acct structure contains a reference count, manipulated in
   both the normal and translator allocate/free code using atomic
   increments and decrements.

 * Because it's now a separate structure, we can defer freeing the
   mem_acct until its reference count reaches zero (either way).

 * Some unit tests were disabled, because they embedded their own
   copies of the implementation for what they were supposedly testing.
   Life's too short to spend time fixing tests that seem designed to
   impede progress by requiring a certain implementation as well as
   behavior.

Change-Id: Id929b11387927136f78626901729296b6c0d0fd7
BUG: 1219026
Signed-off-by: Jeff Darcy 
Reviewed-on: http://review.gluster.org/10417
Tested-by: Gluster Build System 
Reviewed-by: Krishnan Parthasarathi 
Reviewed-by: Niels de Vos 
Reviewed-by: Pranith Kumar Karampuri 
Reviewed-on: http://review.gluster.org/10723
Tested-by: NetBSD Build System

mem-pool: invalidate memory on GF_FREE to aid debugging

2015-03-30T12:37:07+00:00

Debugging where memory gets free'd with help from overwriting the memory
before it is free'd with some structures (repeatedly). The struct
mem_invalid starts with a magic value (0xdeadc0de), followed by a
pointer to the xlator, the mem-type. the size of the GF_?ALLOC()
requested area and the baseaddr pointer to what GF_?ALLOC() returned.

With these details, and the 'struct mem_header' that is placed when
calling GF_?ALLOC(), it is possible to identify overruns and possible
use-after-free. A memory dump (core) or running with a debugger is
needed to read the surrounding memory of corrupt structures.

This additional memory invalidation/poisoning needs to be enabled by
passing --enable-debug to ./configure.

Change-Id: I9f5f37dc4b5b59142adefc90897d32e89be67b82
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/10019
Reviewed-by: Venky Shankar 
Tested-by: Gluster Build System 
Reviewed-by: Emmanuel Dreyfus 
Reviewed-by: Kaleb KEITHLEY

testing: Switch to cmocka the successor of cmockery2

2015-03-05T21:51:37+00:00

This uses https://cmocka.org/ as the unit testing framework.

With this change, unit testing is made optional as well. We assume there
is no cmocka available while building. cmocka will be enabled by default
later on. For now, to build with cmocka run:

    $ ./configure --enable-cmocka

This change is based on the work of Andreas (replacing cmockery2 with
cmocka) and Kaleb (make cmockery2 an optional build dependency).

The only modifications I made, are additional #defines in unittest.h for
making sure the unit tests function as expected.

Change-Id: Iea4cbcdaf09996b49ffcf3680c76731459cb197e
BUG: 1067059
Merged-change: http://review.gluster.org/9762/
Signed-off-by: Andreas Schneider 
Signed-off-by: Kaleb S. KEITHLEY 
Signed-off-by: Niels de Vos 
Change-Id: Ia2e955481c102d5dce17695a9205395a6030e985
Reviewed-on: http://review.gluster.org/9738
Tested-by: Gluster Build System

glusterd: statedump support

2014-10-15T10:02:40+00:00

Although glusterd currently has statedump support but it doesn't dump its
context information. Implementing glusterd_dump_priv function to export per-node
glusterd information would be useful for debugging bugs. Once implemented, we
could enhance sos-report to fetch this information. This would potentially
reduce our time to root cause and data needed for debugability can be dumped
gradually.

Following is the main items of the dump list targeted in this patch :

    * Supported max/min op-version and current op-version
    * Information about peer list
    * Information about peer list involved while a transaction is going on
      (xaction_peers)
    * option dictionary in glusterd_conf_t
    * mgmt_v3_lock in glusterd_conf_t
    * List of connected clients
    * uuid of glusterd
    * A section of rpc related information like live connections and their
      statistics

There are couple of issues which were found during implementation and testing
phase:
 - xaction_peers of glusterd_conf_t was not initialized in init because of which
   traversing through this list head was crashing when there was no active
   transaction
 - gf_free was not setting the typestr to NULL if the the alloc count becomes 0
   for a mem-type earlier allocated.

Change-Id: Ic9bce2d57682fc1771cd2bc6af0b7316ecbc761f
BUG: 1139682
Signed-off-by: Atin Mukherjee 
Reviewed-on: http://review.gluster.org/8665
Tested-by: Gluster Build System 
Reviewed-by: Kaushal M

build: Support for unit tests using Cmockery2

2014-07-18T17:55:51+00:00

This patch will allow for developers to create unit tests for
their code.  Documentation has been added to the patch and
is available here:

doc/hacker-guide/en-US/markdown/unittest.md

Also, unit tests are run when RPM is created.

This patch is a replacement for http://review.gluster.org/#/c/7281
which removed unit test infrastucture from the repo due to multiple
conflicts.  Cmockery2 is now available in Fedora and EPEL, and soon
to be available in Debian and Ubuntu.  For all other operating
systems, please install from the source:

https://github.com/lpabon/cmockery2

BUG: 1067059

Change-Id: I1b36cb1f56fd10916f9bf535e8ad080a3358289f
Signed-off-by: Luis Pabón 
Reviewed-on: http://review.gluster.org/7538
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos 
Reviewed-by: Vijay Bellur

libglusterfs: Fix unused variable warning

2014-05-24T16:51:51+00:00

Change-Id: If385ba0993fdbd3e3b7500b2b6631721911e0ab3
BUG: 1100144
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/7858
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

libglusterfs: Provide a way to disable mempool effects

2014-05-23T06:48:01+00:00

Problems:
 - Mempool comes in the way of debugging mem-leaks/corruptions
   because tools like valgrind can't detect mem-pools.

 - Accessing freed pointers may crash only when the mem-pool
   is completely exhausted.

Solution:
In DEBUG builds set mempool count to 0 irrespective of the
pool size requested. This should help in debugging
memory-leaks/corruptions.

Change-Id: I7c6a2d696e330f12acff57ebdc589cc150e9ab79
BUG: 1100144
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/7835
Reviewed-by: Xavier Hernandez 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur