tests: Fix flakey behavior of ctdb_fetch test

There were two issues with this test:

1. Since the messages are sent from one node to the next, if a node
   does not register for messages before CTDB on that nodes receives
   the message, it will never be seen by ctdb_fetch and it would
   block on receive and would not send any messages to next node.
   The crude solution is to sleep just before the messages are sent,
   so that ctdb_fetch on all nodes have registered for the messages.

2. If ctdb_fetch stops sending messages after timelimit expiry, the
   next node will keep waiting to receive messages in event_loop_once().
   The default timeout is 30 seconds for event_loop_once(). Adding a
   timed event will always set the timeout value to the time remaining
   for the timed event to expire.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit bc55e09fdac9f743d6428bfe0be77840ad0fd1ba)

1 files changed, 17 insertions, 0 deletions

diff --git a/ctdb/tests/src/ctdb_fetch.c b/ctdb/tests/src/ctdb_fetch.c
index 01a5d76100..db84f5d5c4 100644
--- a/ctdb/tests/src/ctdb_fetch.c
+++ b/ctdb/tests/src/ctdb_fetch.c
@@ -113,6 +113,15 @@ static void message_handler(struct ctdb_context *ctdb, uint64_t srvid,
 
 
 /*
+ * timeout handler - noop
+ */
+static void timeout_handler(struct event_context *ev, struct timed_event *timer,
+			    struct timeval curtime, void *private_data)
+{
+	return;
+}
+
+/*
   benchmark the following:
 
   fetch a record
@@ -129,6 +138,7 @@ static void bench_fetch(struct ctdb_context *ctdb, struct event_context *ev)
 	}
 	
 	start_timer();
+	event_add_timed(ev, ctdb, timeval_current_ofs(timelimit,0), timeout_handler, NULL);
 
 	while (end_timer() < timelimit) {
 		if (pnn == 0 && msg_count % 100 == 0 && end_timer() > 0) {
@@ -235,6 +245,13 @@ int main(int argc, const char *argv[])
 		event_loop_once(ev);
 	}
 
+	/* This test has a race condition. If CTDB receives the message from previous
+	 * node, before this node has registered for that message, this node will never
+	 * receive that message and will block on receive. Sleeping for some time will
+	 * hopefully ensure that the test program on all the nodes register for messages.
+	 */
+	printf("Sleeping for %d seconds\n", num_nodes);
+	sleep(num_nodes);
 	bench_fetch(ctdb, ev);
 
 	key.dptr = discard_const(TESTKEY);