summaryrefslogtreecommitdiffstats
path: root/lib/ntdb
Commit message (Collapse)AuthorAgeFilesLines
* ntdb: Fix control reaches end of non-void function.Andreas Schneider2014-11-288-0/+48
| | | | | Signed-off-by: Andreas Schneider <asn@samba.org> Reviewed-by: Guenther Deschner <gd@samba.org>
* ntdb:build: improve detection of srcdirMichael Adam2014-06-201-1/+1
| | | | | Signed-off-by: Michael Adam <obnox@samba.org> Reviewed-by: Amitay Isaacs <amitay@gmail.com>
* lib/ntdb: explicitly use allow_warnings=True for ntdb-test-helpersStefan Metzmacher2014-04-021-1/+2
| | | | | Signed-off-by: Stefan Metzmacher <metze@samba.org> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* samba:python - Py_RETURN_NONE remove compatibility code for releases < 2.4Matthias Dieter Wallnöfer2014-01-091-4/+0
| | | | | | | | | http://www.python.org/doc//current/c-api/none.html Reviewed-By: Jelmer Vernooij <jelmer@samba.org> Autobuild-User(master): Matthias Dieter Wallnöfer <mdw@samba.org> Autobuild-Date(master): Thu Jan 9 16:27:47 CET 2014 on sn-devel-104
* lib/ntdb optimize includes in ntdb toolsChristian Ambach2013-12-077-61/+5
| | | | | | | | | | | use the private header (which will use libreplace or system headers) instead of direct includes of system includes Signed-off-by: Christian Ambach <ambi@samba.org> Reviewed-by: Rusty Russell <rusty@samba.org> Autobuild-User(master): Christian Ambach <ambi@samba.org> Autobuild-Date(master): Sat Dec 7 18:38:21 CET 2013 on sn-devel-104
* lib/ntdb optimize includes in ntdb testsChristian Ambach2013-12-0727-97/+20
| | | | | | | | use the private header (which will use libreplace or system headers) instead of direct includes of system includes Signed-off-by: Christian Ambach <ambi@samba.org> Reviewed-by: Rusty Russell <rusty@samba.org>
* lib/ntdb correct includes in private headerChristian Ambach2013-12-071-0/+5
| | | | | | | | include all necessary headers when libreplace is not around Signed-off-by: Christian Ambach <ambi@samba.org> Reviewed-by: Jelmer Vernooij <jelmer@samba.org> Reviewed-by: Rusty Russell <rusty@samba.org>
* lib/ntdb fix compilation when libreplace is not aroundChristian Ambach2013-12-072-1/+10
| | | | | | Signed-off-by: Christian Ambach <ambi@samba.org> Reviewed-by: Jelmer Vernooij <jelmer@samba.org> Reviewed-by: Rusty Russell <rusty@samba.org>
* pyntdb: Add tests for Python API.Jelmer Vernooij2013-12-062-2/+156
| | | | | | Signed-off-by: Jelmer Vernooij <jelmer@samba.org> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* pyntdb: Don't segfault when passing in None as filename (for memory db)Jelmer Vernooij2013-12-061-1/+1
| | | | | | Signed-off-by: Jelmer Vernooij <jelmer@samba.org> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* pyntdb: Don't allow access after a database is closed.Jelmer Vernooij2013-12-062-6/+77
| | | | | | Signed-off-by: Jelmer Vernooij <jelmer@samba.org> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* docs: remove duplicate mention of "ntdbtool" in ntdbtool manpage.Günther Deschner2013-11-191-4/+0
| | | | | | | Guenther Signed-off-by: Günther Deschner <gd@samba.org> Reviewed-by: Andreas Schneider <asn@samba.org>
* Release ntdb 1.0.Jelmer Vernooij2013-11-162-1/+39
| | | | | Autobuild-User(master): Jelmer Vernooij <jelmer@samba.org> Autobuild-Date(master): Sat Nov 16 19:06:59 CET 2013 on sn-devel-104
* ntdb: Make sure variables passed by value are initialized.Andreas Schneider2013-10-111-2/+3
| | | | | | | | | | This fixes a GCC warning. Signed-off-by: Andreas Schneider <asn@samba.org> Reviewed-by: Alexander Bokovoy <ab@samba.org> Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org> Autobuild-Date(master): Fri Oct 11 18:05:19 CEST 2013 on sn-devel-104
* ntdbtool.8.xml: Bump version up to 4.1.Karolin Seeger2013-08-211-1/+1
| | | | | Signed-off-by: Karolin Seeger <kseeger@samba.org> Reviewed-by: Jeremy Allison <jra@samba.org>
* ntdbrestore.8.xml: Bump version up to 4.1.Karolin Seeger2013-08-211-1/+1
| | | | | Signed-off-by: Karolin Seeger <kseeger@samba.org> Reviewed-by: Jeremy Allison <jra@samba.org>
* ntdbdump.8.xml: Bump version up to 4.1.Karolin Seeger2013-08-211-1/+1
| | | | | Signed-off-by: Karolin Seeger <kseeger@samba.org> Reviewed-by: Jeremy Allison <jra@samba.org>
* ntdbbackup.8.xml: Bump version up to 4.1.Karolin Seeger2013-08-211-1/+1
| | | | | Signed-off-by: Karolin Seeger <kseeger@samba.org> Reviewed-by: Jeremy Allison <jra@samba.org>
* ntdb.3.xml: Bump version up to 4.1.Karolin Seeger2013-08-211-1/+1
| | | | | Signed-off-by: Karolin Seeger <kseeger@samba.org> Reviewed-by: Jeremy Allison <jra@samba.org>
* Man pages for ntdb tools missingRusty Russell2013-08-196-5/+708
| | | | | | | | | | Copied and modified from tdb man pages. Avoided reproducing API documentation which is extensively documented in the ntdb.h header already. BUG: https://bugzilla.samba.org/show_bug.cgi?id=10000 Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Reviewed-by: Jeremy Allison <jra@samba.org>
* Fix bug 10025 - Lack of Sanity Checking in calls to malloc()/calloc().Bill Parker2013-07-172-0/+20
| | | | | | | | | | | | | In reviewing various files in Samba-4.0.7, I found a number of instances where malloc()/calloc() were called without the checking the return value for a value of NULL, which would indicate failure. (NB. The changes needed to ccan, iniparser, popt and heimdal will be reported upstream, not patched inside Samba). Reviewed-by: Jeremy Allison <jra@samba.org> Reviewed-by: Simo Source <idra@samba.org>
* build: Remove autoconf build systemAndrew Bartlett2013-05-281-41/+0
| | | | | | | | | | | We are now confident that that waf build system meets enough of our needs that we will work to improve it, rather than maintain two build systems. Andrew Bartlett Reviewed-by: Jelmer Vernooij <jelmer@samba.org> Reviewed-by: David Disseldorp <ddiss@samba.org>
* autoconf: build in NTDB.Rusty Russell2013-04-121-1/+1
| | | | | | | | | | | | | | | | | I previously added -DDISABLE_NTDB to FLAGS, but lib/param/util.c doesn't seem to be compiled with that flag, so it's really not a good solution. So instead, compile in ntdb for the autoconf build. This means: 1) Add -DHAVE_CCAN to cflags. 2) Remove pyntdb from autoconf objects (which is what tdb does) 3) Remove -DDISABLE_NTDB 4) Add ntdb utility objects 5) Link in ntdb everywhere we link in tdb. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Reviewed-by: Jeremy Allison <jra@samba.org>
* ntdb: don't call open hook when re-opening an existing database.Rusty Russell2013-03-232-13/+22
| | | | | | | | | | | In particular, the Samba dbwrap wrapper can do this for schannel_store, with the openhook set to clear the database if it can get the lock (which, being in the same process, it can). Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Autobuild-User(master): Rusty Russell <rusty@rustcorp.com.au> Autobuild-Date(master): Sat Mar 23 09:39:50 CET 2013 on sn-devel-104
* ntdb: Cast getpid() result to unsigned int for GNU/Solaris buildAndrew Bartlett2013-03-152-8/+10
| | | | Reviewed-by: Jeremy Allison <jra@samba.org>
* ntdb: fix database corruption when transaction doesn't change anything.Rusty Russell2013-02-203-15/+91
| | | | | | | | | | | | | | | | | | | | ntdb's transaction code has an optimization which tdb's doesnt: it only writes the parts of blocks whose contents have changed. This means we can actually have a transaction which turns out to need no recovery region. This breaks the recovery setup logic, which sets the current recovery size to 0 if there's no recovery area, and assumes that we'll always create a new recovery area since the recovery will always need > 0 bytes. In fact, if we really haven't changed anything, we can skip the transaction commit altogether: since this happens at least once with Samba, it's worth doing. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
* ntdb: remove unused local variable.Rusty Russell2012-10-081-2/+0
| | | | | | | | Reported-by: Matthieu Patou <mat@samba.org> Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Autobuild-User(master): Rusty Russell <rusty@rustcorp.com.au> Autobuild-Date(master): Mon Oct 8 04:43:37 CEST 2012 on sn-devel-104
* pyntdb: Fix init function for ntdb python module.Pierre Lejeune2012-09-251-2/+2
| | | | | | | Signed-Off-By: Jelmer Vernooij <jelmer@samba.org> Autobuild-User(master): Jelmer Vernooij <jelmer@samba.org> Autobuild-Date(master): Tue Sep 25 22:40:39 CEST 2012 on sn-devel-104
* ntdb: Try to fix the build on Solaris which does not have errAndrew Bartlett2012-09-231-1/+1
|
* lib/ntdb: Fix format string errors found by -Werror=format in ntdb testsAndrew Bartlett2012-07-301-1/+1
|
* ccan: we're subsystems, not a library.Rusty Russell2012-06-291-5/+6
| | | | | | | | | | | | | | | | | | | | Don't expose a libccan.so; it would produce clashes if someone else does the same thing. Unfortunately, if we just change it from a SAMBA_LIBRARY to a SAMBA_SUBSYSTEM, it doesn't create a static library as we'd like, but links all the object files in. This means we get many duplicates (eg. everyone gets a copy of tally, even though only ntdb wants it). So, the solution is twofold: 1) Make the ccan modules separate. 2) Make the ccan modules SAMBA_SUBSYSTEMs not SAMBA_LIBRARYs so we don't build shared libraries which we can't share. 3) Make the places which uses ccan explicit. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Autobuild-User(master): Rusty Russell <rusty@rustcorp.com.au> Autobuild-Date(master): Fri Jun 29 06:22:44 CEST 2012 on sn-devel-104
* ntdb: take advantage of direct access across expand.Rusty Russell2012-06-221-33/+15
| | | | | | | This means we no longer have to unmap if we want to compare a record. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* ntdb: test arbitrary operations during ntdb_parse_record().Rusty Russell2012-06-222-0/+90
| | | | | | | | | | | In particular, this tests that we can store enough records to make the database expand while we map the given record. We use a global lock for this, but it could happen in theory with another process. It also tests the that we can recurse inside ntdb_parse_record(). Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* ntdb: make database read-only during ntdb_parse() callback.Rusty Russell2012-06-226-24/+195
| | | | | | | | | | | Since we have a readlock, any write will grab a write lock: if it happens to be on the same bucket, we'll fail. For that reason, enforce read-only so every write operation fails (even for NTDB_NOLOCK or NTDB_INTERNAL dbs), and document it! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* ntdb: allow direct access for NTDB_INTERNAL dbs during expansion.Rusty Russell2012-06-221-14/+45
| | | | | | | | | | | | NTDB_INTERNAL databases need to malloc and copy to keep old versions around if we expand, in a similar way to the manner in which keep old mmaps around. Of course, it only works for read-only accesses, since the two copies are not synced. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* ntdb: enhancement to allow direct access to the ntdb map during expansion.Rusty Russell2012-06-225-33/+86
| | | | | | | | | | | | | | | This means keeping the old mmap around when we expand the database. We could revert to read/write, except for platforms with incoherent mmap (ie. OpenBSD), where we need to use mmap for all accesses. Thus we keep a linked list of old maps, and unmap them when the last access finally goes away. This is required if we want ntdb_parse_record() callbacks to be able to expand the database. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* ntdb: don't munmap the database on every close.Rusty Russell2012-06-222-13/+16
| | | | | | | | | | Since we can have multiple openers, we should leave the mmap in place for the other openers to use. Enhance the test to check the bug (it still works, because without mmap we fall back to read/write, but performance would be terrible!). Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* ntdb: hand correct error code when alloc_read allocation fails.Rusty Russell2012-06-221-1/+1
| | | | | | | -ECUTNPASTE. This is not a usage error! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* ntdb: respect TDB_NO_FSYNC flag for 'make test'Rusty Russell2012-06-2251-151/+187
| | | | | | | This reduces test time from 31 seconds to 6, on my laptop. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* Add --disable-ntdb option for building.Jelmer Vernooij2012-06-211-0/+1
| | | | | Autobuild-User(master): Jelmer Vernooij <jelmer@samba.org> Autobuild-Date(master): Thu Jun 21 19:59:57 CEST 2012 on sn-devel-104
* ntdb: fix occasional abort in testing.Rusty Russell2012-06-201-1/+7
| | | | | | | | | | | | | | Occasionally, the capability test inserts multiple used records and they clash, but our primitive test layout engine doesn't handle hash clashes and aborts. Force a seed value which we know doesn't clash. Reported-by: Andrew Bartlett <abartlet@samba.org> Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Autobuild-User(master): Rusty Russell <rusty@rustcorp.com.au> Autobuild-Date(master): Wed Jun 20 16:50:20 CEST 2012 on sn-devel-104
* ntdb: add autoconf support.Rusty Russell2012-06-191-0/+41
| | | | | | | | | | | This is copied from tdb; we build the utilities, but as nothing else links against it, we shouldn't be adding anything to the normal samba binary sizes. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Autobuild-User(master): Rusty Russell <rusty@rustcorp.com.au> Autobuild-Date(master): Tue Jun 19 07:31:06 CEST 2012 on sn-devel-104
* ntdb: update documentation.Rusty Russell2012-06-196-5821/+505
| | | | | | | | Update the design.lyx file with the latest status and the change in hashing. Also, refresh and add examples to the TDB_porting.txt file. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* ntdb: optimize ntdb_fetch.Rusty Russell2012-06-197-51/+59
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We access the key on lookup, then access the data in the caller. It makes more sense to access both at once. We also put in a likely() for the case where the hash is not chained. Before: Adding 1000 records: 3644-3724(3675) ns (129656 bytes) Finding 1000 records: 1596-1696(1622) ns (129656 bytes) Missing 1000 records: 1409-1525(1452) ns (129656 bytes) Traversing 1000 records: 1636-1747(1668) ns (129656 bytes) Deleting 1000 records: 3138-3223(3175) ns (129656 bytes) Re-adding 1000 records: 3278-3414(3329) ns (129656 bytes) Appending 1000 records: 5396-5529(5426) ns (253312 bytes) Churning 1000 records: 9451-10095(9584) ns (253312 bytes) smbtorture results (--entries=1000) ntdb speed 183881-191112(188223) ops/sec After: Adding 1000 records: 3590-3701(3640) ns (129656 bytes) Finding 1000 records: 1539-1605(1566) ns (129656 bytes) Missing 1000 records: 1398-1440(1413) ns (129656 bytes) Traversing 1000 records: 1629-2015(1710) ns (129656 bytes) Deleting 1000 records: 3118-3236(3163) ns (129656 bytes) Re-adding 1000 records: 3235-3355(3275) ns (129656 bytes) Appending 1000 records: 5335-5444(5385) ns (253312 bytes) Churning 1000 records: 9350-9955(9494) ns (253312 bytes) smbtorture results (--entries=1000) ntdb speed 180559-199981(195106) ops/sec
* ntdb: add -h arg to ntdbrestoreRusty Russell2012-06-191-8/+28
| | | | | | | | Since our default hashsize is 8192 not 131, we look fat when we convert near-empty TDBs. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* ntdb: reduce default hashsize on ntdbtorture.Rusty Russell2012-06-191-3/+10
| | | | | | | Just like tdbtorture, having a hashsize of 2 stresses us much more! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* ntdb: add NTDB_ATTRIBUTE_HASHSIZERusty Russell2012-06-192-2/+41
| | | | | | | | | Since we've given up on expansion, let them frob the hashsize again. We have attributes, so we should use them for optional stuff like this. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* ntdb: remove hash table trees.Rusty Russell2012-06-1936-1662/+1096
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | TDB2 started with a top-level hash of 1024 entries, divided into 128 groups of 8 buckets. When a bucket filled, the 8 bucket group expanded into pointers into 8 new 64-entry hash tables. When these filled, they expanded in turn, etc. It's a nice idea to automatically expand the hash tables, but it doesn't pay off. Remove it for NTDB. 1) It only beats TDB performance when the database is huge and the TDB hashsize is small. We are about 20% slower on medium-size databases (1000 to 10000 records), worse on really small ones. 2) Since we're 64 bits, our hash tables are already twice as expensive as TDB. 3) Since our hash function is good, it means that all groups tend to fill at the same time, meaning the hash enlarges by a factor of 128 all at once, leading to a very large database at that point. 4) Our efficiency would improve if we enlarged the top level, but that makes our minimum db size even worse: it's already over 8k, and jumps to 1M after about 1000 entries! 5) Making the sub group size larger gives a shallower tree, which performs better, but makes the "hash explosion" problem worse. 6) The code is complicated, having to handle delete and reshuffling groups of hash buckets, and expansion of buckets. 7) We have to handle the case where all the records somehow end up with the same hash value, which requires special code to chain records for that case. On the other hand, it would be nice if we didn't degrade as badly as TDB does when the hash chains get long. This patch removes the hash-growing code, but instead of chaining like TDB does when a bucket fills, we point the bucket to an array of record pointers. Since each on-disk NTDB pointer contains some hash bits from the record (we steal the upper 8 bits of the offset), 99.5% of the time we don't need to load the record to determine if it matches. This makes an array of offsets much more cache-friendly than a linked list. Here are the times (in ns) for tdb_store of N records, tdb_store of N records the second time, and a fetch of all N records. I've also included the final database size and the smbtorture local.[n]tdb_speed results. Benchmark details: 1) Compiled with -O2. 2) assert() was disabled in TDB2 and NTDB. 3) The "optimize fetch" patch was applied to NTDB. 10 runs, using tmpfs (otherwise massive swapping as db hits ~30M, despite plenty of RAM). Insert Re-ins Fetch Size dbspeed (nsec) (nsec) (nsec) (Kb) (ops/sec) TDB (10000 hashsize): 100 records: 3882 3320 1609 53 203204 1000 records: 3651 3281 1571 115 218021 10000 records: 3404 3326 1595 880 202874 100000 records: 4317 3825 2097 8262 126811 1000000 records: 11568 11578 9320 77005 25046 TDB2 (1024 hashsize, expandable): 100 records: 3867 3329 1699 17 187100 1000 records: 4040 3249 1639 154 186255 10000 records: 4143 3300 1695 1226 185110 100000 records: 4481 3425 1800 17848 163483 1000000 records: 4055 3534 1878 106386 160774 NTDB (8192 hashsize) 100 records: 4259 3376 1692 82 190852 1000 records: 3640 3275 1566 130 195106 10000 records: 4337 3438 1614 773 188362 100000 records: 4750 5165 1746 9001 169197 1000000 records: 4897 5180 2341 83838 121901 Analysis: 1) TDB wins on small databases, beating TDB2 by ~15%, NTDB by ~10%. 2) TDB starts to lose when hash chains get 10 long (fetch 10% slower than TDB2/NTDB). 3) TDB does horribly when hash chains get 100 long (fetch 4x slower than NTDB, 5x slower than TDB2, insert about 2-3x slower). 4) TDB2 databases are 40% larger than TDB1. NTDB is about 15% larger than TDB1
* ntdb: special accessor functions for read/write of an offset.Rusty Russell2012-06-193-50/+114
| | | | | | | | | | | | | | | | | | | | | | | | | | | | We also split off the NTDB_CONVERT case (where the ntdb is of a different endian) into its own io function. NTDB speed: Adding 10000 records: 3894-9951(8553) ns (815528 bytes) Finding 10000 records: 1644-4294(3580) ns (815528 bytes) Missing 10000 records: 1497-4018(3303) ns (815528 bytes) Traversing 10000 records: 1585-4225(3505) ns (815528 bytes) Deleting 10000 records: 3088-8154(6927) ns (815528 bytes) Re-adding 10000 records: 3192-8308(7089) ns (815528 bytes) Appending 10000 records: 5187-13307(11365) ns (1274312 bytes) Churning 10000 records: 6772-17567(15078) ns (1274312 bytes) NTDB speed in transaction: Adding 10000 records: 1602-2404(2214) ns (815528 bytes) Finding 10000 records: 456-871(778) ns (815528 bytes) Missing 10000 records: 393-522(503) ns (815528 bytes) Traversing 10000 records: 729-1015(945) ns (815528 bytes) Deleting 10000 records: 1065-1476(1374) ns (815528 bytes) Re-adding 10000 records: 1397-1930(1819) ns (815528 bytes) Appending 10000 records: 2927-3351(3184) ns (1274312 bytes) Churning 10000 records: 3921-4697(4378) ns (1274312 bytes) smbtorture results: ntdb speed 86581-191518(175666) ops/sec Applying patch..increase-top-level.patch
* ntdb: inline oob checkRusty Russell2012-06-196-14/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | The simple "is it in range" check can be inline; complex cases can be handed through to the normal or transaction handler. NTDB speed: Adding 10000 records: 4111-9983(9149) ns (815528 bytes) Finding 10000 records: 1667-4464(3810) ns (815528 bytes) Missing 10000 records: 1511-3992(3546) ns (815528 bytes) Traversing 10000 records: 1698-4254(3724) ns (815528 bytes) Deleting 10000 records: 3608-7998(7358) ns (815528 bytes) Re-adding 10000 records: 3259-8504(7805) ns (815528 bytes) Appending 10000 records: 5393-13579(12356) ns (1274312 bytes) Churning 10000 records: 6966-17813(16136) ns (1274312 bytes) NTDB speed in transaction: Adding 10000 records: 916-2230(2004) ns (815528 bytes) Finding 10000 records: 330-866(770) ns (815528 bytes) Missing 10000 records: 196-520(471) ns (815528 bytes) Traversing 10000 records: 356-879(800) ns (815528 bytes) Deleting 10000 records: 505-1267(1108) ns (815528 bytes) Re-adding 10000 records: 658-1681(1477) ns (815528 bytes) Appending 10000 records: 1088-2827(2498) ns (1274312 bytes) Churning 10000 records: 1636-4267(3785) ns (1274312 bytes) smbtorture results: ntdb speed 85588-189430(157110) ops/sec