| Commit message (Collapse) | Author | Age | Files | Lines |
... | |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The problem was tdb->name is NULL for TDB_INTERNAL databases, and
so it was crashing ...
#0 0xb76944f3 in strlen () from /lib/i686/cmov/libc.so.6
#1 0x0809862b in PyString_FromFormatV (format=0xb72b6a26 "Tdb('%s')", vargs=0xbfc26a94 "")
at ../Objects/stringobject.c:211
#2 0x08098888 in PyString_FromFormat (format=0xb72b6a26 "Tdb('%s')") at ../Objects/stringobject.c:358
#3 0xb72b65f2 in tdb_object_repr (self=0xb759e060) at ./pytdb.c:439
Cc: 597089@bugs.debian.org
Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Jelmer Vernooij <jelmer@samba.org>
(This used to be ctdb commit 3ff413baf04ce28eb54a80141250ae1284b2a521)
|
|
|
|
|
|
|
|
|
|
|
| |
Note, unlike tdb_open where flags is `int', tdb_{add,remove}_flags want
flags as `unsigned', so instead of "i" I used "I" in PyArg_ParseTuple.
Cc: 597386@bugs.debian.org
Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Jelmer Vernooij <jelmer@samba.org>
(This used to be ctdb commit 7389f8a8a634c2fe0f068831326d92e6bfa0d046)
|
|
|
|
|
|
| |
this might help reduce test times and load on test machines
(This used to be ctdb commit 5c4240c364c52073ca64fddf2aa2c1593db0093b)
|
|
|
|
| |
(This used to be ctdb commit f1c06608245ec34493c330d891e04c250ad64b20)
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is Stefan Metzmacher <metze@samba.org>'s patch with minor changes:
1) Use the TDB_MAGIC constant so both hashes aren't of strings.
2) Check the hash in tdb_check (paranoia, really).
3) Additional check in the (unlikely!) case where both examples hash to 0.
4) Cosmetic changes to var names and complaint message.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 63c582c99128c3623e270e8425966cab7744fb2f)
|
|
|
|
|
|
|
|
| |
We must not endian-convert the magic string, just the rest.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 525390863ad39acea08ceb88531dc59d118fcad4)
|
|
|
|
|
|
|
|
|
|
|
|
| |
Commit bc1c82ea137 "Fix tdb_check() to work with read-only tdb databases."
claimed to do this, but tdb_lockall_read() fails on read-only databases.
Also make sure we can still do tdb_check() inside a transaction (weird,
but we previously allowed it so don't break the API).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 2558eb250011893d09dbeaedaffeefa0e397142f)
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We can end up with dead areas when we die during transaction commit;
tdb_check() fails on such a (valid) database.
This is particularly noticable now we no longer truncate on recovery;
if the recovery area was at the end of the file we used to remove it
that way.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit b4162a95ff9ae28cda8d9c76c51c9480104517a7)
|
|
|
|
|
|
| |
This serviceability tool was lost during the migration from the old eventsystem to the tevent system.
(This used to be ctdb commit b4c00b4ac30ec215629f44f802ce9660abcd7a48)
|
|\
| |
| |
| | |
(This used to be ctdb commit 13e58d92f5f1723e850a82ae030d0ca57e89b1ee)
|
| |
| |
| |
| |
| |
| |
| |
| | |
This means we can distinguish which child is logging, esp. via syslog where we have no pid.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 68b3761a0874429b90731741f0531f76dcfbb081)
|
| |
| |
| |
| |
| |
| |
| |
| | |
parts of the recovery daemon.
Initialize all event contexts to allow nesting
(This used to be ctdb commit 5bf6bd5e7f33aabbeb7b9707716ef99cf471e590)
|
|\ \
| | |
| | |
| | | |
(This used to be ctdb commit 1566d2d23ab698896b3b6a76974a5c7452db4a62)
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
In Samba this is now called "tevent", and while we use the backwards
compatibility wrappers they don't offer EVENT_FD_AUTOCLOSE: that is now
a separate tevent_fd_set_auto_close() function.
This is based on Samba version 7f29f817fa939ef1bbb740584f09e76e2ecd5b06.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 85e5e760cc91eb3157d3a88996ce474491646726)
|
| |/
| |
| |
| |
| |
| |
| |
| |
| | |
This is based on SAMBA as at revision 2de63aa2801a907905b3e05557074af5b896d486.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit cecd93be0a0aab868430dd43f8276bfb4e35f02e)
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
(Imported from SAMBA 11ab43084b10cf53b530cdc3a6036c898b79ca38)
We saw tdb_lockall() take 71 seconds under heavy load; this is because Linux
(at least) doesn't prevent new small locks being obtained while we're waiting
for a big log.
The workaround is to do divide and conquer using non-blocking chainlocks: if
we get down to a single chain we block. Using a simple test program where
children did "hold lock for 100ms, sleep for 1 second" the time to do
tdb_lockall() dropped signifiantly. There are ln(hashsize) locks taken in
the contended case, but that's slow anyway.
More analysis is given in my blog at http://rusty.ozlabs.org/?p=120
This may also help transactions, though in that case it's the initial
read lock which uses this gradual locking routine; the update-to-write-lock
code is separate and still tries to update in one go.
Even though ABI doesn't change, minor version bumped so behavior change
can be easily detected.
CQ:S1018154
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 9ec0009443a0ac4187ce5212a5143689daa58a02)
|
| |
| |
| |
| |
| |
| |
| |
| | |
(Import from SAMBA bc1c82ea137e1bf6cb55139a666c56ebb2226b23)
The function tdb_lockall() uses F_WRLCK internally, which doesn't work on
a fd opened with O_RDONLY. Use tdb_lockall_read() instead.
(This used to be ctdb commit a5db1122ec48d7e7384066848457c850c1a6cf3c)
|
| |
| |
| |
| |
| |
| | |
(Imported from SAMBA 2eab1d7fdcb54f9ec27431ca4858eb64cb1bd835)
(This used to be ctdb commit 52a87e608d0406aee9df99f7ac3ce16e834b520b)
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
| |
Commit 207a213c/24fed55d purported to fix the problem of signals during
tdb_new_database (which could cause a spurious short write, hence a failure).
However, the code is wrong: newdb+written is not correct.
Fix this by introducing a general tdb_write_all() and using it here and in
the tracing code.
Cc: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 27ba0e5a6681063225df7244a85aa304c51c6948)
|
|
|
|
|
|
|
|
| |
When tdb throws an error, we didn't report the name of the tdb; we should.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit cfea357c9b2142c8cd8cac1ee712d40b188793e1)
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since idtree assigns sequentially, it rarely reaches high numbers.
But such numbers can be forced with idr_get_new_above(), and that
reveals two bugs:
1) Crash in sub_remove() caused by pa array being too short.
2) Shift by more than 32 in _idr_find(), which is undefined, causing
the "outside the current tree" optimization to misfire and return NULL.
Signed-off-by: Rusty Russell <rusty@rustorp.com.au>
(This used to be ctdb commit 32c04e11ebbcf8239e47016302c6ce802a8b0a6f)
|
|
|
|
|
|
|
|
|
|
|
| |
The Samba tree defines _PUBLIC_ (and _PRIVATE_) for libraries to
control visibility. The last commit absorbed this from their tdb,
but we need to #define to stub it out since ctdb doesn't use it
(and doesn't need to: we only use tdb internally).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(This used to be ctdb commit 54bee0ec670724c337b882e7dda3e999ffa80691)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We now use -fvisibilty=hidden to hide symbols from outside the tdb
shared library.
This also moved tdb_transaction_recover() into the tdb_private.h
header, as it should never have been a public API. For that reason we
are changing the version number. We're only doing a minor version
increment as it is extremely unlikely that anyone was actually using
tdb_transaction_recover() as its locking requirements were rather
unusual.
Pair-Programmed-With: Rusty Russell <rusty@samba.org>
(Imported from commit 773a8afbba27a5e2e48577100f3ca9873b506615)
(This used to be ctdb commit e174dc084f11db0eb239b643affef2c02c711b1c)
|
|
|
|
|
|
|
|
|
|
| |
Upstream subunit makes a ":" after commands optional, so I've fixed any
places where we might trigger commands accidently. I've filed a bug
about this in subunit.
(Imported from commit 7da94cc4a664521be279b019e9f32121cd410193)
(This used to be ctdb commit f1242d9cba676d1d393375bea42613192c70662e)
|
|
|
|
|
|
| |
(Imported from commit c1f6f61f620e865516d1856c9d937b5326a29046)
(This used to be ctdb commit e61ff181085b35cb4aba933350bfe4f64471b359)
|
|
|
|
|
|
| |
(Imported from commit 261c3b4f1beed820647061bacbee3acccbcbb089)
(This used to be ctdb commit 87ced00d6d98be4a34719af58694e7c940b4dd68)
|
|
|
|
|
|
| |
(Imported from commit 59315887a07033316edf91c0c57563eee5ea992d)
(This used to be ctdb commit fa38f818c71c85918e673ff563bf7a91a0c4cc17)
|
|
|
|
|
|
| |
(Imported from commit ea8e0d5d54b020c530e392c4edaeed43e20af303)
(This used to be ctdb commit 7161cb1607bb105cd6f4f32df50f519314e77b3f)
|
|
|
|
|
|
|
|
| |
this should be much more portable
(Imported from commit 088096d1bad51428a2e2d487214995d4fdfc7ccc)
(This used to be ctdb commit 7f1330adc690360a4a8ce85352df0908a93684a5)
|
|
|
|
|
|
| |
(Imported from commit e2c7e5c4f72565fe49265d5b036531926ea1ac92)
(This used to be ctdb commit 3318c74386e202b77ba7e29b4c3e0ce4d88fcd5b)
|
|
|
|
|
|
| |
(Imported from commit fb98f60594b6cabc52d0f2f49eda08f793ba4748)
(This used to be ctdb commit e90aba9967ea4a8ae7f6bdfc19666c47bd92951e)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
tdb transactions were designed to be robust against the machine
powering off, but interestingly were never designed to handle the case
where an administrator kill -9's a process during commit. Because
recovery is only done on tdb_open, processes with the tdb already
mapped will simply use it despite it being corrupt and needing
recovery.
The solution to this is to check for recovery every time we grab a
data lock: we could have gained the lock because a process just died.
This has no measurable cost: here is the time for tdbtorture -s 0 -n 1
-l 10000:
Before:
2.75 2.50 2.81 3.19 2.91 2.53 2.72 2.50 2.78 2.77 = Avg 2.75
After:
2.81 2.57 3.42 2.49 3.02 2.49 2.84 2.48 2.80 2.43 = Avg 2.74
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit ec96ea690edbe3398d690b4a953d487ca1773f1c)
(This used to be ctdb commit 4215c7025d2b29439c5acd19ce4e0fc4e67370b3)
|
|
|
|
|
|
| |
(Imported from commit 1bf482b9ef9ec73dd7ee4387d7087aa3955503dd)
(This used to be ctdb commit 52b0f19636565ef633e63d2726a1cc8c41dccedb)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
To test the case of death of a process during transaction commit, add
a -k (kill random) option to tdbtorture. The easiest way to do this
is to make every worker a child (unless there's only one child), which
is why this patch is bigger than you might expect.
Using -k without -t (always transactions) you expect corruption, though
it doesn't happen every time. With -t, we currently get corruption but
the next patch fixes that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit ececeffd85db1b27c07cdf91a921fd203006daf6)
(This used to be ctdb commit e5af0ce79c74f11daae2a8514e398c40535e2e67)
|
|
|
|
|
|
|
|
|
|
|
|
| |
The current recovery code truncates the tdb file on recovery. This is
fine if recovery is only done on first open, but is a really bad idea
as we move to allowing recovery on "live" databases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit 8c3fda4318adc71899bc41486d5616da3a91a688)
(This used to be ctdb commit 65bc926d1a9cb3af18cae6b1462b832a5bcec561)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Now the transaction code uses the standard allrecord lock, that stops
us from trying to grab any per-record locks anyway. We don't need to
have special noop lock ops for transactions.
This is a nice simplification: if you see brlock, you know it's really
going to grab a lock.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit 9f295eecffd92e55584fc36539cd85cd32c832de)
(This used to be ctdb commit 6d7093cf51d0256245cc6bab24c9550ed3f1d8a5)
|
|
|
|
|
|
|
|
|
|
|
|
| |
tdb_release_extra_locks() is too general: it carefully skips over the
transaction lock, even though the only caller then drops it. Change
this, and rename it to show it's clearly transaction-specific.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit a84222bbaf9ed2c7b9c61b8157b2e3c85f17fa32)
(This used to be ctdb commit 803035716338170896fee15f15b17c32e7ee777e)
|
|
|
|
|
|
|
|
|
|
|
|
| |
Now the transaction allrecord lock is the standard one, and thus is cleaned
in tdb_release_extra_locks(), _tdb_transaction_cancel() doesn't need to
know what type it is.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit dd1b508c63034452673dbfee9956f52a1b6c90a5)
(This used to be ctdb commit 74874ffb2c81e098c1d7935b37557c2151382ca6)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Centralize locking of all chains of the tdb; rename _tdb_lockall to
tdb_allrecord_lock and _tdb_unlockall to tdb_allrecord_unlock, and
tdb_brlock_upgrade to tdb_allrecord_upgrade.
Then we use this in the transaction code. Unfortunately, if the transaction
code records that it has grabbed the allrecord lock read-only, write locks
will fail, so we treat this upgradable lock as a write lock, and mark it
as upgradable using the otherwise-unused offset field.
One subtlety: now the transaction code is using the allrecord_lock, the
tdb_release_extra_locks() function drops it for us, so we no longer need
to do it manually in _tdb_transaction_cancel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit fca1621965c547e2d076eca2a2599e9629f91266)
(This used to be ctdb commit d7fdb38ac05b5f2af9eb485e98673280835273dd)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Records themselves get (read) locked by the traversal code against delete.
Interestingly, this locking isn't done when the allrecord lock has been
taken, though the allrecord lock until recently didn't cover the actual
records (it now goes to end of file).
The write record lock, grabbed by the delete code, is not suppressed
by the allrecord lock. This is now bad: it causes us to punch a hole
in the allrecord lock when we release the write record lock. Make this
consistent: *no* record locks of any kind when the allrecord lock is
taken.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit caaf5c6baa1a4f340c1f38edd99b3a8b56621b8b)
(This used to be ctdb commit 7a99cdf5d0a91764a750c1a264e90e5b66f910a1)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We were previously inconsistent with our "global" lock: the
transaction code grabbed it from FREELIST_TOP to end of file, and the
rest of the code grabbed it from FREELIST_TOP to end of the hash
chains. Change it to always grab to end of file for simplicity and
so we can merge the two.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit 9341f230f8968b4b18e451d15dda5ccbe7787768)
(This used to be ctdb commit 46f2c33357c999c31a8064c159c6162269c28d9d)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This was redundant before this patch series: it mirrored num_lockrecs
exactly. It still does.
Also, skip useless branch when locks == 1: unconditional assignment is
cheaper anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit 1ab8776247f89b143b6e58f4b038ab4bcea20d3a)
(This used to be ctdb commit 587ac01ce836286aab54bfcb7a693a0170c7ebd3)
|
|
|
|
|
|
|
|
|
|
|
| |
This is pure overhead, but it centralizes the locking. Realloc (esp. as
most implementations are lazy) is fast compared to the fnctl anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit d48c3e4982a38fb6b568ed3903e55e07a0fe5ca6)
(This used to be ctdb commit 2e8512403525c14c9b776ce28891d09c17ada91d)
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Use our newly-generic nested lock tracking for the active lock.
Note that the tdb_have_extra_locks() and tdb_release_extra_locks()
functions have to skip over this lock now it is tracked.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit 4738d474c412cc59d26fcea64007e99094e8b675)
(This used to be ctdb commit 0a44584963232b0b1c62e30c9bede0439e68ef7d)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This never nests, so it's overkill, but it centralizes the locking into
lock.c and removes the ugly flag in the transaction code to track whether
we have the lock or not.
Note that we have a temporary hack so this places a real lock, despite
the fact that we are in a transaction.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit 9136818df30c7179e1cffa18201cdfc990ebd7b7)
(This used to be ctdb commit 6812d81907793299e874f121174d885f6500f374)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Rather than a boutique lock and a separate nest count, use our
newly-generic nested lock tracking for the transaction lock.
Note that the tdb_have_extra_locks() and tdb_release_extra_locks()
functions have to skip over this lock now it is tracked.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit e8fa70a321d489b454b07bd65e9b0d95084168de)
(This used to be ctdb commit 4ca1b96a70048b2eaa0d12fb5f0fdb54ec396aa3)
|
|
|
|
|
|
|
|
|
|
|
| |
Factor out two loops which find locks; we are going to introduce a couple
more so a helper makes sense.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit ce41411c84760684ce539b6a302a0623a6a78a72)
(This used to be ctdb commit cfb154dd0f189f37b937e90144c2eb9e66a26420)
|
|
|
|
|
|
|
|
|
|
|
| |
Move locking intelligence back into lock.c, rather than open-coding the
lock release in transaction.c.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit db270734d8b4208e00ce9de5af1af7ee11823f6d)
(This used to be ctdb commit d2dd720b51c4032e5d77d30212da8117d3f119df)
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In many places we check whether locks are held: add a helper to do this.
The _tdb_lockall() case has already checked for the allrecord lock, so
the extra work done by tdb_have_extra_locks() is merely redundant.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit fba42f1fb4f81b8913cce5a23ca5350ba45f40e1)
(This used to be ctdb commit dda3587dfee598f387c2e696f3645486fac65052)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
tdb_transaction_lock() and tdb_transaction_unlock() do nothing if we
hold the allrecord lock. However, the two locks don't overlap, so
this is wrong.
This simplification makes the transaction lock a straight-forward nested
lock.
There are two callers for these functions:
1) The transaction code, which already makes sure the allrecord_lock
isn't held.
2) The traverse code, which wants to stop transactions whether it has the
allrecord lock or not. There have been deadlocks here before, however
this should not bring them back (I hope!)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
(Imported from commit b754f61d235bdc3e410b60014d6be4072645e16f)
(This used to be ctdb commit 495f3554259b9dbf9ee673c4fe420d98e50e4901)
|