summaryrefslogtreecommitdiffstats
path: root/ctdb/server/ctdb_recoverd.c
Commit message (Collapse)AuthorAgeFilesLines
* ctdb-recoverd: Detach database from recovery daemonAmitay Isaacs2014-04-231-0/+46
| | | | | | | | | | | | As part of vacuuming, recoverd attaches to databases to migrate records. When detaching a database from main daemon, it should be removed from recovery daemon also. Signed-off-by: Amitay Isaacs <amitay@gmail.com> Reviewed-by: Michael Adam <obnox@samba.org> Autobuild-User(master): Michael Adam <obnox@samba.org> Autobuild-Date(master): Wed Apr 23 17:05:45 CEST 2014 on sn-devel-104
* ctdb-daemon: Always update database priority cluster wideAmitay Isaacs2014-04-071-14/+5
| | | | | | | | | | | | | Database priority is a global property and all the nodes should have the priority set for the databases. Just setting priority on one node can lead to problems in the recovery as a database can be frozen at wrong priority and then freezing database would not succeed. Signed-off-by: Amitay Isaacs <amitay@gmail.com> Reviewed-by: David Disseldorp <ddiss@samba.org> Autobuild-User(master): David Disseldorp <ddiss@samba.org> Autobuild-Date(master): Mon Apr 7 14:06:26 CEST 2014 on sn-devel-104
* ctdb/recoverd: Do not refuse disabling takeover runs on inactive nodesMartin Schwenke2014-01-171-7/+0
| | | | | | | | | | | | | | Failure might be expected when disabling takeover runs on banned nodes, since they might be suffering from performance problems or similar. More broadly, administrators who reconfigure a cluster that isn't in a happy state aren't necessarily doing something sensible. However, allowing takeover runs to be disabled on inactive nodes stops reconfiguration of stopped nodes. This is probaby an unreasonable limitation, so drop it. Signed-off-by: Martin Schwenke <martin@meltin.net> Reviewed-by: Amitay Isaacs <amitay@gmail.com>
* ctdb-recoverd: Only respond to currently queued ipreallocated requestsMartin Schwenke2013-11-271-1/+10
| | | | | | | | | | | Otherwise new requests can come in during the latter parts of the takeover run when the IP allocation algorithm has already run, and the new requests will be dequeued even though they haven't really be processed. Signed-off-by: Martin Schwenke <martin@meltin.net> Pair-programmed-with: Amitay Isaacs <amitay@gmail.com> Reviewed-by: Michael Adam <obnox@samba.org>
* ctdb-recoverd: For persistent databases a sequence number of 0 is validMartin Schwenke2013-11-271-2/+3
| | | | | | | | Otherwise recovery ends up done by RSN when it is unnecessary. Signed-off-by: Martin Schwenke <martin@meltin.net> Pair-programmed-with: Amitay Isaacs <amitay@gmail.com> Reviewed-by: Michael Adam <obnox@samba.org>
* ctdb-recoverd: Fix backward compatibility for CTDB_SRVID_TAKEOVER_RUNMartin Schwenke2013-11-271-7/+7
| | | | | | | | | When running a mixed version cluster, compatibility with older versions was was broken during recent refactorisation. Signed-off-by: Martin Schwenke <martin@meltin.net> Pair-programmed-with: Amitay Isaacs <amitay@gmail.com> Reviewed-by: Michael Adam <obnox@samba.org>
* ctdb-recoverd: A node refuses to play against itselfMartin Schwenke2013-11-271-0/+5
| | | | | | Signed-off-by: Martin Schwenke <martin@meltin.net> Pair-programmed-with: Amitay Isaacs <amitay@gmail.com> Reviewed-by: Michael Adam <obnox@samba.org>
* ctdb-recoverd: Remove duplicate code to update flags during recoveryMartin Schwenke2013-11-271-17/+0
| | | | | | | | This also happens earlier in do_recovery() and the nodemap is not updated after that, so this update is redundant. Signed-off-by: Martin Schwenke <martin@meltin.net> Reviewed-by: Michael Adam <obnox@samba.org>
* ctdb-server: Coverity fixesAmitay Isaacs2013-11-191-14/+22
| | | | | Signed-off-by: Amitay Isaacs <amitay@gmail.com> Reviewed-by: Michael Adam <obnox@samba.org>
* recoverd: Rebalancing should be done regardless tunableMartin Schwenke2013-10-301-7/+14
| | | | | | | | | Rebalance target nodes should be set even if a deferred rebalance is not configured. The user can explicitly cause a takeover run. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit afd9b51644af074752d74c412cb4e7ec2eba2c69)
* recoverd: Improve an error message in the election codeMartin Schwenke2013-10-301-1/+1
| | | | | | Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 275ed9ebe287e39d891888c13810c70f347af8ac)
* Revert "if a new node enters the cluster, that node will already be frozen ↵Martin Schwenke2013-10-301-20/+13
| | | | | | | | | | | | | | | | | | | at start" This is unnecessary due to 03e2e436db5cfd29a56d13f5d2101e42389bfc94. Furthermore, if a node doesn't force an election but wins it then it can fail to record that it is the new recovery master. This can lead to a reverse split brain where there is no recovery master. This reverts commit c5035657606283d2e35bea40992505e84ca8e7be. Signed-off-by: Martin Schwenke <martin@meltin.net> Pair-programmed-with: Amitay Isaacs <amitay@gmail.com> Conflicts: server/ctdb_recoverd.c (This used to be ctdb commit c8b542e059a54b8d524bd430cad9d82e5edd864d)
* Revert "recoverd: Disable takeover runs on other nodes for 5 minutes"Martin Schwenke2013-10-291-2/+2
| | | | | | | | | | | 5 minutes is too long to leave the cluster in limbo if the recovery daemon dies during a takeover run, even though this is quite unlikely. We need a new recover master to be able to do takeover runs fairly quickly. This reverts commit 71080676bb4acbd0d9b595a30cf7fe6dddbf426f. (This used to be ctdb commit 3e41170c78fc7a2bf526129c9b7db3739b61c6bf)
* recoverd: Remove function reload_nodes_file()Martin Schwenke2013-10-221-11/+2
| | | | | | | | | It is a 1 line wrapper around ctdb_load_nodes_file(), so use that instead. We need less code... :-) Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 4a5d5935f4410a93a3343d85a24dbcddae2c4c20)
* Revert "null out the pointer before we reload the nodes file"Martin Schwenke2013-10-221-1/+1
| | | | | | | | | | This reverts commit 4b0f32047e8bece0a052bdbe2209afe91b7e8ce3. This is not necessary. It just causes a memory leak. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 25fd05505f61dc595c0ef25bb6e332274d5530e8)
* recoverd: Ignore failed flag updates on inactive nodesAmitay Isaacs2013-10-221-2/+6
| | | | | | | Signed-off-by: Amitay Isaacs <amitay@gmail.com> Pair-programmed-with: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 484c46eaae056480baf050fd91868f2fd0537985)
* recoverd: Fix the VNN lmaster consistency checkMartin Schwenke2013-10-221-5/+11
| | | | | | | | | It does cope with node that don't have the lmaster capability. Signed-off-by: Martin Schwenke <martin@meltin.net> Pair-programmed-with: Amitay Isaacs <amitay@gmail.com> (This used to be ctdb commit 588172bcb6bf267339e2bd09e23d2c4904a27a41)
* recoverd: Disable takeover runs on other nodes for 5 minutesMartin Schwenke2013-09-191-2/+2
| | | | | | | | | 60 seconds might not be long enough to kill all connections and release IPs. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 71080676bb4acbd0d9b595a30cf7fe6dddbf426f)
* recoverd: Improve logging for takeover runsMartin Schwenke2013-09-191-1/+5
| | | | | | | | | Takeover runs are currently silent when they succeed. However, they are important, so log something by default. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit b39aa2e401fbb581207d986bac93778e9c01acdc)
* recoverd: Be careful about freeing the list of IP rebalance target nodesMartin Schwenke2013-09-191-1/+7
| | | | | | | | | | | It can change during a takeover run. If it does then don't free it. There are potentially fancier solutions (e.g. check what PNNs are new to the list) to this issue but this is the simplest. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit e81589b7084c661adf617e166cc2c25b4939f841)
* recoverd: Fix the implementation of CTDB_SRVID_REBALANCE_NODEMartin Schwenke2013-09-191-18/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The current implementation has a few flaws: * A takeover run is called unconditionally when the timer goes even if the recovery master role has moved. This means a node other than the recovery master can incorrectly do a takeover run. * The rebalancing target nodes are cleared in the setup for a takeover run, regardless of whether the takeover run succeeds. * The timer to force a rebalance isn't cleared if another takeover run occurs before the deadline. Any forced rebalancing will happen in the first takeover run and when the timer expires some time later then an unnecessary takeover run will occur. * If the recovery master role moves then the rebalancing data will stay on the original node and affect the next takeover run to occur if the recovery master role should come back to the original node. Instead, store an array of rebalance target nodes in the recovery master context. This is passed as an extra argument to ctdb_takeover_run() each time it is called and is cleared when a takeover run succeeds. The timer hangs off the array of rebalance target nodes, which is cleared if the node isn't the recovery master. This means that it is possible to lose rebalance data if the recovery master role moves. However, that's a difficult problem to solve. The best way of approaching it is probably to try to stop the recovery master role from jumping around unnecesarily when inactive nodes join the cluster. The long term solution is to avoid this nonsense completely. The IP allocation algorithm needs to cache state between runs so that it knows which nodes have just become healthy. This also needs recovery master stability. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit c51c1efe5fc7fa668597f2acd435dee16e410fc9)
* recoverd: Remove unused CTDB_SRVID_RELOAD_ALL_IPS and handlerMartin Schwenke2013-09-191-88/+0
| | | | | | Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 4cd727439a0824ebb8dbcf737d9888ffc3c41184)
* recoverd: Defer ipreallocated requests when takeover runs are disabledMartin Schwenke2013-09-191-1/+2
| | | | | | | | | The takeover run will fail anyway but deferring seems like a cleaner option. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 428f800bcdf3dbfe19de8bb36099fbf01ebeaab4)
* recoverd: Reimplement CTDB_SRVID_DISABLE_IP_CHECKMartin Schwenke2013-09-191-51/+37
| | | | | | | | | Use disable_takeover_runs_handler() instead of maintaining duplicate logic. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 0a51a85915486b2a8fded7ba6444b18c6c1ee8e8)
* recoverd: New SRVID message CTDB_SRVID_DISABLE_TAKEOVER_RUNSMartin Schwenke2013-09-191-25/+150
| | | | | | | | | | | | This implements a superset of CTDB_SRVID_DISABLE_IP_CHECK. It stops the IP checks but also causes any attempted takeover runs to fail and be rescheduled. This is meant to completely stop IP movements. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 00db4de53a0d86013e79e6577e7e6cf3ef864e56)
* recoverd: Factor out the SRVID handling codeMartin Schwenke2013-09-191-54/+99
| | | | | | | | | | | | | The code that handles IP reallocate requests can be reused. This also changes the result back to a SRVID caller to the PNN on success or a negative error code on failure. None of the callers currently look at the result so this is harmless... but it will be useful later. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit e4eae6e3291baa299a1d0f733ab11b138ee699a3)
* recoverd: Make the SRVID request structure genericMartin Schwenke2013-09-191-9/+9
| | | | | | | | No need for a separate one for each SRVID. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit d9c22b04d5aa7938a3965bd3144568664eb772ce)
* recoverd: Move disabling of IP checks into do_takeover_run()Martin Schwenke2013-09-191-0/+26
| | | | | | Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 48b603fbf16311daa47b01e7a33d477ed51da56d)
* recoverd: do_takeover_run() should mark when a takeover run is in progressMartin Schwenke2013-09-191-0/+13
| | | | | | | | Nested takeover runs should never happens so they should fail. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 8ed29c60c0a7dd29f2a6efdf694d38e94281e1c4)
* recoverd: takeover_fail_callback() doesn't need to set rec->need_takeover_runMartin Schwenke2013-09-191-1/+0
| | | | | | | | It is set on every failure anyway. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit e5f94c7857405bdeac233069003c3769b3dc3616)
* recoverd: New function do_takeover_run()Martin Schwenke2013-09-191-21/+31
| | | | | | | | | | | Factor the calling sequence for ctdb_takeover_run() into a new function and call it instead. This changes rec->need_takeover_run to false for each successful takeover run and that seems to be the right thing to do. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 9a3f0c0e61ca5c17e020c6e0463d73c7cf4f7c09)
* recoverd: Stabilise the recovery master roleMartin Schwenke2013-09-191-0/+8
| | | | | | | | | | | | | | | | | | | | | On rare occasions when a node that has been inactive it will trigger an election when it becomes active again. If that node has been up for the longest then it will win the election and the recovery master role will spuriously move. While a node remains inactive we reset the priority time to discourage it from winning elections. The priority time will now reflect roughly how long the node has been active rather than how long it has been up. That means the most stable node is more likely to win elections. Having a stable recovery master means that disabling takeover runs while reloading IPs is more likely to succeed. It also improves the chances of being able to cache information in the recovery master - for example, between takeover runs. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit f0f48f22f45e4c82eba2582efae307e25385de81)
* recoverd: Remove an unused temporary talloc contextMartin Schwenke2013-08-221-3/+0
| | | | | | Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit da22d5e60dc023009854025cc9e6bc4b0a84c60e)
* recoverd: Log more information when interfaces changeMartin Schwenke2013-08-221-2/+15
| | | | | | Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 3ef93a1a3e60cdf5d8954e7a16a988ea6126916b)
* recoverd: Improve log message when nodes disagree on recmasterAmitay Isaacs2013-08-141-1/+1
| | | | | | Signed-off-by: Amitay Isaacs <amitay@gmail.com> (This used to be ctdb commit 7b7aa7b599536cd60ebb84d363607bb4e953248a)
* Revert "recoverd: Use correct tdb flags when creating missing databases"Amitay Isaacs2013-08-141-2/+2
| | | | | | | | | | | This reverts commit 10a057d8e15c8c18e540598a940d3548c731b0b4. This approach would not work when creating local databases since currently there is no control to receive TDB flags for remote databases. Signed-off-by: Amitay Isaacs <amitay@gmail.com> (This used to be ctdb commit ca61eb776ab862bd269e45ee0f9f96e7e1e0e001)
* recoverd: Use correct tdb flags when creating missing databasesAmitay Isaacs2013-08-011-2/+2
| | | | | | | | | | When creating missing databases either locally or remotely, make sure to use the correct tdb flags from other nodes. Without this, volatile databases can get attached without TDB_INCOMPATIBLE_HASH flag. Signed-off-by: Amitay Isaacs <amitay@gmail.com> (This used to be ctdb commit 10a057d8e15c8c18e540598a940d3548c731b0b4)
* recoverd: Make sure to use jenkins hash for recovery databasesAmitay Isaacs2013-08-011-1/+1
| | | | | | Signed-off-by: Amitay Isaacs <amitay@gmail.com> (This used to be ctdb commit 32c83e209823e9a4d6306bb7fd63d4500f3e2668)
* recoverd: Assemble up-to-date node flags information from remote nodesAmitay Isaacs2013-07-301-0/+17
| | | | | | | | | | | Currently nodemap used by recovery master is the one obtained from the local node. This information may have been updated while processing main loop. Before comparing node flags on all the nodes, create up-to-date node flags information based on the information received from all the nodes. Signed-off-by: Amitay Isaacs <amitay@gmail.com> (This used to be ctdb commit fcf77dec5af973a0e32f3999bc012053a6f47a96)
* recoverd: Really fix bogus info in message about changed flagsMartin Schwenke2013-07-111-2/+2
| | | | | | | | | | | | Commit 9119a568c2b4601318f7751f537dca2f92a7230b attempted to fix this. However, this was wrong because old_flags and new_flags were confused. The latter has since been fixed in commit 7eb2f89979360b6cc98ca9b17c48310277fa89fc so this can now be fixed properly. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 40f2825d6e818dc8c745b6385a545969dfb45fbc)
* Fixes for various issues found by CoveritySumit Bose2013-07-111-2/+2
| | | | | | Signed-off-by: Amitay Isaacs <amitay@gmail.com> (This used to be ctdb commit 05bfdbbd0d4abdfbcf28e3930086723508b35952)
* recoverd: Recovery daemon should use ctdb_get_pnn, which can't failMartin Schwenke2013-07-101-5/+1
| | | | | | Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit c6fded59fa4da67f738a90fdacb51900e41801f9)
* ctdbd: Set process names for child processesAmitay Isaacs2013-07-101-0/+3
| | | | | | | | This helps distinguish processes in process list in top, perf, etc. Signed-off-by: Amitay Isaacs <amitay@gmail.com> (This used to be ctdb commit 2493f57ce268d6fe7e4c40a87852c347fd60d29e)
* recoverd: Minor style improvements for ctdb_reload_remote_public_ips()Martin Schwenke2013-07-051-20/+25
| | | | | | | | | | | | | * Add a variable to the loop to make the code more readable and have it generally fit into 80 columns. * Improve comments. * Improve log messages. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 0a292fa8939a1343e44cadaa8ed9f3c0f18ca82f)
* recoverd: Clean up log messages in remote IP verificationMartin Schwenke2013-07-051-7/+7
| | | | | | | | | | | | | | | The log messages in verify_remote_ip_allocation() are confusing because they don't include the PNN of the problem node, because it is not known in this function. Add the PNN of the node being verified as a function argument and then shuffle the log messages around to make them clearer. Also fold 3 nested if statements into just one. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit f0942fa01cd422133fc9398f56b4855397d7bc86)
* recoverd: Fix an unclear log message - "Restart recovery process"Martin Schwenke2013-07-051-1/+1
| | | | | | | | | | | When the recovery master notices a node in recovery mode it starts the recovery process, it doesn't restart it. Update documentation to match. Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 298c4d2c3b4ea3d900c91f5a0a5aca2952a13d61)
* recoverd: Fix an incorrect commentMartin Schwenke2013-07-051-3/+1
| | | | | | Signed-off-by: Martin Schwenke <martin@meltin.net> (This used to be ctdb commit 9f6cd8b0bea619991c9f3bf35188c5950dabf8f4)
* recoverd: Send the result from child process only onceAmitay Isaacs2013-07-041-1/+0
| | | | | | | | | The result has been sent before the child keeps waiting for parent ctdbd process. Signed-off-by: Amitay Isaacs <amitay@gmail.com> (This used to be ctdb commit 9aa13bcedd83d463c871e3cf1f3a65da3cd83992)
* recoverd: when the recmaster is banned, use that information when forcing an ↵Michael Adam2013-07-021-0/+6
| | | | | | | | | | | | | | | election When we trigger an election because the recmaster considers itself inactive, update our local nodemap with the recmaster's flags before calling force_election(). This way, we don't send the inactive node freeze commands (e.g.) that may fail and then lead to ourselves getting banned. The theory is that this should help avoiding banning loops. Signed-off-by: Michael Adam <obnox@samba.org> (This used to be ctdb commit 932360992b08a5483d90c0590218ba0fd756119e)
* recoverd: fix a comment typoMichael Adam2013-07-021-1/+1
| | | | | | Signed-off-by: Michael Adam <obnox@samba.org> (This used to be ctdb commit 741944f118e98f178b860194eecb215180949d18)
generated by cgit (git 2.34.1) at 2024-09-21 14:03:32 +0000