summaryrefslogtreecommitdiffstats
path: root/ctdb/common/ctdb_io.c
Commit message (Collapse)AuthorAgeFilesLines
* common/io: Limit the queue buffer size for fair scheduling via teventAmitay Isaacs2013-08-221-12/+31
| | | | | | | | | | | | | | | | | | | | If we process all the data available in a socket buffer, CTDB can stay busy processing lots of packets via immediate event mechanism in tevent. After processing an immediate event, tevent returns without epoll_wait. So as long as there are immediate events, tevent will never poll other FDs. CTDB will report this as "Event handling took xx seconds" warning. This is misleading since CTDB is very busy processing packets, but never gets to the point of polling FDs. The improvement in socket handling made it worse when handling traverse control. There were lots of packets filled in the socket buffer quickly and CTDB stayed busy processing those packets and not polling other FDs and timer events. This can lead to controls timing out and in worse case other nodes marking busy node as disconnected. Signed-off-by: Amitay Isaacs <amitay@gmail.com> (This used to be ctdb commit 92939c1178d04116d842708bc2d6a9c2950e36cc)
* Revert "common/io: Keep queue buffer size multiple of 4K"Amitay Isaacs2013-08-221-22/+6
| | | | | | | | | | | | | | | This reverts commit 5e9b1a7e24d058ff88aaa0563db36a804e866fa9. This is not the best approach. Allowing queue buffer size to grow indefinitely causes large number of CTDB packets to be queued up very quickly which when processed via immediate events will block CTDB from processing events from other FDs. If there are immediate events queued up, tevent will never process any of the FDs till all immediate events are processed. Signed-off-by: Amitay Isaacs <amitay@gmail.com> (This used to be ctdb commit d8b094e804efc53fae9f44c6ef961b7b5797d290)
* common/io: Keep queue buffer size multiple of 4KAmitay Isaacs2013-08-091-6/+22
| | | | | | | | | | | | | | | Currently queue buffer size is realloc'd every time we need to extend the buffer. Small increments can cause memory fragmentation. Instead always extend buffer in multiples of 4K. This should reduce multiple talloc_realloc calls when there are lots of packets in the socket buffer. Also, if queue buffer has grown larger than 64K, throw away the buffer once all the requests in the queue have been processed. That way queue does not hold on to large buffers. Signed-off-by: Amitay Isaacs <amitay@gmail.com> (This used to be ctdb commit 5e9b1a7e24d058ff88aaa0563db36a804e866fa9)
* build: Fix tdb.h path to enable building with system TDB libraryMathieu Parent2013-06-141-1/+1
| | | | (This used to be ctdb commit f8bf99de3a5f56be67aaa67ed836458b1cf73e86)
* common/io: For scheduling immediate events use tevent_schedule_immediateAmitay Isaacs2013-03-061-11/+15
| | | | | | | | | | tevent_schedule_immediate() is much more efficient at handling events that need to be processed immediately rather than creating timed events with timeval_zero(). Signed-off-by: Amitay Isaacs <amitay@gmail.com> (This used to be ctdb commit 11734be353a1e246163eda631d35dfe55d1d6fb1)
* common/io: Rewrite socket handling code to read all available dataAmitay Isaacs2013-02-191-68/+92
| | | | | | | | | | | | | This improves the processing of packets considerably. It has been observed that there can be as many as 10 packets in the socket buffer and the current code of reading a single packet from a socket at a time is not very optimal. This change reads all the bytes from socket buffer and then parses to extract multiple packets. If there are multiple packets, set up a timed event to process next packet. Signed-off-by: Amitay Isaacs <amitay@gmail.com> (This used to be ctdb commit d788bc8f7212b7dc1587ae592242dc8c876f4053)
* Remove explicit include of lib/tevent/tevent.h.Amitay Isaacs2012-04-131-1/+0
| | | | | | Signed-off-by: Amitay Isaacs <amitay@gmail.com> (This used to be ctdb commit 0681014ca5ed2a9b56f63fdace7f894beccf8a9a)
* explain why we use FIONREADRonnie Sahlberg2012-01-041-0/+6
| | | | (This used to be ctdb commit d0f85478c37828eb8a24315d4326eb4eaedb9afc)
* Clean up warnings: remove unused function dump_packet()Martin Schwenke2011-11-091-15/+0
| | | | | | Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit c22e201be15e7d5b788c2f5f7916b553e0faaa2a)
* io: Make queue_io_read() safe for reentryDavid Disseldorp2011-08-051-74/+63
| | | | | | | | | | | | | | | | | | queue_io_read() may be reentered via the queue callback, recoverd is particularly guilty of this. queue_io_read() is not safe for reentry if more than one packet is received and partial chunks follow - data read off the pipe on re-entry is assumed to be the start-of-packet four byte length. This leads to a wrongly aligned stream and the notorious "Invalid packet of length 0" errors. This change fixes queue_io_read() to be safe under reentry, only a single packet is processed per call. https://bugzilla.samba.org/show_bug.cgi?id=8319 (This used to be ctdb commit 9ea41d2fab612772f861270c8a59c01c43bd3a4c)
* event: Update events to latest Samba version 0.9.8Rusty Russell2010-08-181-2/+3
| | | | | | | | | | | | | In Samba this is now called "tevent", and while we use the backwards compatibility wrappers they don't offer EVENT_FD_AUTOCLOSE: that is now a separate tevent_fd_set_auto_close() function. This is based on Samba version 7f29f817fa939ef1bbb740584f09e76e2ecd5b06. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> (This used to be ctdb commit 85e5e760cc91eb3157d3a88996ce474491646726)
* Report client for queue errors.Rusty Russell2010-07-011-16/+43
| | | | | | | | | | We've been seeing "Invalid packet of length 0" errors, but we don't know what is sending them. Add a name for each queue, and print nread. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> (This used to be ctdb commit e6cf0e8f14f4263fbd8b995418909199924827e9)
* libctdb: reorganize headers: remove ctdb.h, add ctdb_client.h and ↵Rusty Russell2010-05-201-1/+1
| | | | | | | | | | | | | | ctdb_protocol.h ctdb_client.h is the existing internal client interface (which was mainly in ctdb.h), and ctdb_protocol.h is the information needed for the wire protocol only. ctdb.h will be the new, shiny, libctdb API. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> (This used to be ctdb commit 4bba6b8cd47b352f98d41f9f06258d5ac3c9adef)
* ctdb: migrate to new dlinklist.h from SambaAndrew Tridgell2010-02-091-4/+3
| | | | (This used to be ctdb commit f63c091f12f8d582e9518673365c7c52479c470c)
* ctdb: move ctdb_io.c to use TLIST_*() macrosAndrew Tridgell2010-02-041-21/+6
| | | | | | This will make large packet queues much more efficient (This used to be ctdb commit e3f198056230073135ea6354bbef30c5bb022f8f)
* We only queued up to 1000 packets per queue before we start droppingRonnie Sahlberg2010-02-041-1/+18
| | | | | | | | | | | | | | | | | | | | packets, to avoid the queue to grow excessively if smbd has blocked. This could cause traverse packets to become discarded in case the main smbd daemon does a traverse of a database while there is a recovery (sending a erconfigured message to smbd, causing an avalanche of unlock messages to be sent across the cluster.) This avalance of messages could cause also the tranversal message to be discarded causing the main smbd process to hang indefinitely waiting for the traversal message that will never arrive. Bump the maximum queue length before starting to discard messages from 1000 to 1000000 and at the same time rework the queueing slightly so we can append messages cheaply to the queue instead of walking the list from head to tail every time. (This used to be ctdb commit 59ba5d7f80e0465e5076533374fb9ee862ed7bb6)
* ctdb_io: fix use-after-free on invalid packetsRusty Russell2009-12-021-1/+19
| | | | | | | | | | | | | | | | | | | | | | | Wolfgang saw a talloc complaint about using freed memory in ctdb_tcp_read_cb. His fix was to remove the talloc_free() in that function, which causes loops when a socket is closed (as it does not get removed from the event system), eg: netcat 192.168.1.2 4379 < /dev/null The real bug is that when we have more than one pending packet in the queue, we loop calling the callback without any safeguards should that callback free the queue (as it tends to do on invalid packets). This can be reproduced by sending more than one bogus packet at once: # Length word at start: 4 == empty packet (assumed little endian) /usr/bin/printf \\4\\0\\0\\0\\4\\0\\0\\0 > /tmp/pkt netcat 192.168.1.2 4379 < /tmp/pkt Using a destructor we can check if the callback frees us, and exit immediately. Elsewhere, we return after the callback anyway. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> (This used to be ctdb commit 4d0523dd94fb07e860b3e8118691f93d1ef8d0fa)
* Suggestion from Volker,Ronnie Sahlberg2009-10-261-6/+5
| | | | | | make ctdb_queue_length() cheaper by using a counter variable instead of counting the number of packets each time. (This used to be ctdb commit 331c6e3afd96d8b5e191153a631efdbdabb6ea33)
* When clients have blocked, perhaps because the node is banned or stopped ↵Ronnie Sahlberg2009-10-211-0/+10
| | | | | | | | | | | | and the client is blocked trying to tdb_fetch() a record, make sure we dont queue up too many REQ_MESSAGES. Add a new tunable to control the maximum queue size we allow to a blocked client before we start discarding REQ_MESSAGES instead of queueing them for delivery. This avoids having queued up very very large number of MESSAGES that samba semds between eachother to nodes that are blocked/banned/stopped for extended periods . (This used to be ctdb commit f76d6fed8f9630450263b9fa4b5fdf3493fb1e11)
* add improvements to tracking memory usage in ctdbd adn the recovery daemonRonnie Sahlberg2008-04-011-1/+1
| | | | | | | and a ctdb command to pull the talloc memory map from a recovery daemon ctdb rddumpmemory (This used to be ctdb commit d23950be7406cf288f48b660c0f57a9b8d7bdd05)
* from tridge: decorate dumpmemory output so that packets that are queued show ↵Ronnie Sahlberg2008-04-011-0/+23
| | | | | | up with a little more information to make memory leak debugging easier (This used to be ctdb commit 890832ba37d92c7996b38735451f93592c37ff79)
* merge from ronnieAndrew Tridgell2008-02-041-5/+5
| | | | (This used to be ctdb commit e7b57d38cf7255be823a223cf15b7526285b4f1c)
* update lib/replace from samba4Andrew Tridgell2007-07-101-3/+2
| | | | (This used to be ctdb commit f0555484105668c01c21f56322992e752e831109)
* ctdb is GPL not LGPLAndrew Tridgell2007-05-311-12/+12
| | | | (This used to be ctdb commit 8624378010d1c2a1438e1e701339dfba7276f960)
* fixed more warnings on 64 bit boxesAndrew Tridgell2007-05-291-2/+2
| | | | (This used to be ctdb commit 2f6eae476203f8a8b28e083553204c01f224c8a5)
* another place where we could send a partial packetAndrew Tridgell2007-05-281-2/+4
| | | | (This used to be ctdb commit 4e0c57718a0c9f51b5b2e02bbf516c63c5be0a35)
* drop any partialialy send packets when we get a socket write errorAndrew Tridgell2007-05-261-0/+7
| | | | (This used to be ctdb commit 4f7b97af8e0075de56059fffc57a86cb0c16df46)
* handle corrupt ctdb packets betterAndrew Tridgell2007-05-261-0/+5
| | | | (This used to be ctdb commit aea4d78645a99eb22860048350b5f20841c17704)
* fixed two more places where we don't correctly handle write errors on socketsAndrew Tridgell2007-05-151-1/+6
| | | | (This used to be ctdb commit f4a71bb63e7f75d21b66f9eaeac997c2029cd146)
* - take advantage of the new EVENT_FD_AUTOCLOSE flagAndrew Tridgell2007-05-051-1/+1
| | | | | | - use the tdb_chainlock_mark() call to allow us to guarantee forward progress in the ctdb_lockwait code (This used to be ctdb commit e201e98aad0fef6a779a80f3b1ae7792953e2d6b)
* first stage of efficient non-blocking ctdb traverseAndrew Tridgell2007-05-031-3/+7
| | | | (This used to be ctdb commit 4c23e6f26bde421bb56b55de9d6cd3e319b2be40)
* yay! finally fixed the bug that volker, ronnie and I have been chasingAndrew Tridgell2007-04-291-5/+17
| | | | | | | | | | | | | | | for 2 days. The main bug was in smbd, but there was a secondary (and more subtle) bug in ctdb that the bug in smbd exposed. When we get send a dmaster reply, we have to correctly update the dmaster in the recipient even if the original requst has timed out, otherwise ctdbd can get into a loop fighting over who will handle a key. This patch also cleans up the packet allocation, and makes ctdbd become a real daemon. (This used to be ctdb commit 59405e59ef522b97d8e20e4b14310a217141ac7c)
* - fix includes to work in both samba4 and ctdb standaloneAndrew Tridgell2007-04-161-1/+1
| | | | | | | - when we do a store_unlock the lock record becomes unlocked, so we must destroy it (or we leak memory) (This used to be ctdb commit d85955640e670dd580073da96b25fb8a10c08d18)
* private -> private_data for samba3Andrew Tridgell2007-04-131-11/+11
| | | | (This used to be ctdb commit 080b6901173afb2ad618dd0621876ff478c7d6e5)
* merge from ronnieAndrew Tridgell2007-04-101-2/+0
|\ | | | | | | (This used to be ctdb commit 886a3e9122168ececc257aa1bef2f5de2f2d8d7e)
| * remove old ifdef that remained from when this was a header fileRonnie sahlberg2007-04-101-4/+0
| | | | | | | | (This used to be ctdb commit 050f302103b32ea3895fc97dae7726f11fbf01e3)
* | made all sockets handle partial IOAndrew Tridgell2007-04-101-36/+222
|/ | | | | | abstract IO via ctdb_queue_*() functions (This used to be ctdb commit 636ae76f4632b29231db87be32c9114f58b37840)
* make normal/deamon mode controllable by a ctdb flag so that the api looks ↵Ronnie sahlberg2007-04-101-0/+119
the same in both modes to a client. send the correct structure back to a client assorted other cleanups (tests/test1.sh now works in daemon mode) (This used to be ctdb commit f4593754cab750dfdb9384884502e2e1b8fde1f0)
generated by cgit (git 2.34.1) at 2024-09-28 09:47:33 +0000