Resolves: bug 487425

Bug Description: slapd crashes after changelog is moved
Reviewed by: nkinder, nhosoi (Thanks!)
Fix Description: There are a number of real fixes, mixed in with many changes for debugging and instrumentation.
1) When the update thread gets the changelog iterator, it will use _cl5AddThread to increment the count of threads holding an open handle to the changelog.  When it releases the iterator, or if there were some error acquiring the database handle, it will decrement the thread count.  The way it used to work was that it would increment the thread count when retrieving the DB object, but then would immediately decrement it, meaning it had an open handle to the database, but there was no way for the changelog code to know that (except via the reference count on the DB object itself).
2) Changed the AddThread code to increment the thread count outside of the state lock - this better fits the semantics of the other uses of threadcount which are outside of the lock.
3) The changelog code that closes the databases was not closing things down in the correct order.  The first thing it must do is wait for all threads with open database handles or otherwise accessing the database to terminate.  Once that is done, it can call _cl5DBClose() to actually close all of the databases.  Otherwise, a race condition could cause a database to be accessed after it has been closed.
4) Added clcache cleanup code, and made it possible to re-init the clcache.  The clcache was not designed to be dynamically closed and opened.
clcache is init-ed in _cl5Open
clcache_init is re-entrant
Added more code to clean up the clcache
Delete the clcache in _cl5Delete
5) The clcache stores the current buffer in a thread private storage area.  If the clcache has been re-initialized, this buffer is also invalid and the clcache code must get a new buffer.
Platforms tested: RHEL5
Flag Day: no
Doc impact: no

1 files changed, 9 insertions, 2 deletions

diff --git a/ldap/servers/plugins/replication/cl5_config.c b/ldap/servers/plugins/replication/cl5_config.c
index 54150d5e..7fb11a4d 100644
--- a/ldap/servers/plugins/replication/cl5_config.c
+++ b/ldap/servers/plugins/replication/cl5_config.c
@@ -497,8 +497,6 @@ changelog5_config_modify (Slapi_PBlock *pb, Slapi_Entry* entryBefore, Slapi_Entr
 					"old dir - %s, new dir - %s; recreating changelog.\n",
 					currentDir, config.dir);
 
-			/* this call will block until all threads using changelog
-			   release changelog by calling cl5RemoveThread () */
 			rc = cl5Close ();
 			if (rc != CL5_SUCCESS)
 			{
@@ -511,6 +509,9 @@ changelog5_config_modify (Slapi_PBlock *pb, Slapi_Entry* entryBefore, Slapi_Entr
 				slapi_log_error(SLAPI_LOG_FATAL, repl_plugin_name_cl, 
 					"changelog5_config_modify: failed to close changelog\n");
 				goto done;
+			} else {
+				slapi_log_error(SLAPI_LOG_REPL, repl_plugin_name_cl, 
+								"changelog5_config_modify: closed the changelog\n");
 			}
 
 			rc = cl5Delete (currentDir);
@@ -525,6 +526,9 @@ changelog5_config_modify (Slapi_PBlock *pb, Slapi_Entry* entryBefore, Slapi_Entr
 				slapi_log_error(SLAPI_LOG_FATAL, repl_plugin_name_cl, 
 					"changelog5_config_modify: failed to remove changelog\n");
 				goto done;
+			} else {
+				slapi_log_error(SLAPI_LOG_REPL, repl_plugin_name_cl, 
+								"changelog5_config_modify: deleted the changelog at %s\n", currentDir);
 			}
 
 			rc = cl5Open (config.dir, &config.dbconfig);
@@ -544,6 +548,9 @@ changelog5_config_modify (Slapi_PBlock *pb, Slapi_Entry* entryBefore, Slapi_Entr
 									"changelog5_config_modify: failed to restore previous changelog\n");
 				}
 				goto done;
+			} else {
+				slapi_log_error(SLAPI_LOG_REPL, repl_plugin_name_cl, 
+								"changelog5_config_modify: opened the changelog at %s\n", config.dir);
 			}
 		}
 	}