We only queued up to 1000 packets per queue before we start dropping

packets, to avoid the queue to grow excessively if smbd has blocked.

This could cause traverse packets to become discarded in case the main
smbd daemon does a traverse of a database while there is a recovery
(sending a erconfigured message to smbd, causing an avalanche of unlock
messages to be sent across the cluster.)

This avalance of messages could cause also the tranversal message to be
discarded  causing the main smbd process to hang indefinitely waiting
for the traversal message that will never arrive.

Bump the maximum queue length before starting to discard messages from
1000 to 1000000 and at the same time rework the queueing slightly so we
can append messages cheaply to the queue instead of walking the list
from head to tail every time.

(This used to be ctdb commit 59ba5d7f80e0465e5076533374fb9ee862ed7bb6)

1 files changed, 18 insertions, 1 deletions

diff --git a/ctdb/common/ctdb_io.c b/ctdb/common/ctdb_io.c
index 28830d5f37..47681d2a56 100644
--- a/ctdb/common/ctdb_io.c
+++ b/ctdb/common/ctdb_io.c
@@ -46,6 +46,12 @@ struct ctdb_queue {
 	struct ctdb_context *ctdb;
 	struct ctdb_partial partial; /* partial input packet */
 	struct ctdb_queue_pkt *out_queue;
+	/* This field is used to track the last added item so we
+	   can append new items to the end cheaply.
+	   This relies of that items are always appended to the tail
+	   and that when reamoving items we only remove the head.
+	*/
+	struct ctdb_queue_pkt *out_queue_last_added;
 	uint32_t out_queue_length;
 	struct fd_event *fde;
 	int fd;
@@ -294,7 +300,18 @@ int ctdb_queue_send(struct ctdb_queue *queue, uint8_t *data, uint32_t length)
 		EVENT_FD_WRITEABLE(queue->fde);
 	}
 
-	DLIST_ADD_END(queue->out_queue, pkt, struct ctdb_queue_pkt *);
+	/* This relies on that when adding items to the queue, we always add
+	   them to the tail and that when removing items we only remove
+	   the head of queue item.
+	   The last_added item thus allows non n^2 behaviour when appending to
+	   very long queues.
+	*/
+	if (queue->out_queue == NULL) {
+		DLIST_ADD(queue->out_queue, pkt);
+	} else {
+		DLIST_ADD_END(queue->out_queue_last_added, pkt, struct ctdb_queue_pkt *);
+	}
+	queue->out_queue_last_added = pkt;
 	queue->out_queue_length++;
 
 	if (queue->ctdb->tunable.verbose_memory_names != 0) {