summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorRainer Gerhards <rgerhards@adiscon.com>2008-01-31 08:25:28 +0000
committerRainer Gerhards <rgerhards@adiscon.com>2008-01-31 08:25:28 +0000
commit0218ac3b17cf22f44895cd06e8501c93e922407a (patch)
tree06fecc7ab1c422709ea65a957cc030a112c70638
parent82428234502ee97ad4da8648dbc658b092f013b9 (diff)
downloadrsyslog-0218ac3b17cf22f44895cd06e8501c93e922407a.tar.gz
rsyslog-0218ac3b17cf22f44895cd06e8501c93e922407a.tar.xz
rsyslog-0218ac3b17cf22f44895cd06e8501c93e922407a.zip
updated to new 3.11.0 features
-rw-r--r--doc/rsyslog_high_database_rate.html50
1 files changed, 46 insertions, 4 deletions
diff --git a/doc/rsyslog_high_database_rate.html b/doc/rsyslog_high_database_rate.html
index 54961f5d..5b0a29ae 100644
--- a/doc/rsyslog_high_database_rate.html
+++ b/doc/rsyslog_high_database_rate.html
@@ -14,7 +14,7 @@
<a href="http://www.gerhards.net/rainer">Rainer
- Gerhards</a> (2008-01-28)</i></small></P>
+ Gerhards</a> (2008-01-31)</i></small></P>
<h2>Abstract</h2>
@@ -72,16 +72,54 @@ restarts, so even a reboot of the system will not cause any message loss.</p>
disk queue and then configure your action. There is nothing else to do. With the
following simple config file, you log anything you receive to a MySQL database
and have buffering applied automatically.</p>
-<textarea rows="11" cols="80">$ModLoad ommysql.so # load the output driver (use ompgsql.so for PostgreSQL)
-$ModLoad imudp.so # network reception
-$UDPServerRun 514 # start a udp server at port 514
+<textarea rows="11" cols="80">$ModLoad ommysql.so # load the output driver (use ompgsql.so for PostgreSQL)
+$ModLoad imudp.so # network reception
+$UDPServerRun 514 # start a udp server at port 514
$ModLoad imuxsock.so # local message reception
$WorkDirectory /rsyslog/work # default location for work (spool) files
$MainMsgQueueFileName mainq # set file name, also enables disk mode
+$ActionResumeRetryCount -1 # infinite retries on insert failure
+# for PostgreSQL replace :ommysql: by :ompgsql: below:
*.* :ommysql:hostname,dbname,userid,password;
</textarea>
+<p>The simple setup above has one drawback: the write database action is
+executed together with all other actions. Typically, local files are also
+written. These local file writes are now bound to the speed of the database
+action. So if the database is down, or threre is a large backlog, local files
+are also not (or late) written.</p>
+<p><b>There is an easy way to avoid this with rsyslog.</b> It involves a
+slightly more complicated setup. In rsyslog, each action can utilize its own
+queue. If so, messages are simply pulled over from the main queue and then the
+action queue handles action processing on its own. This way, main processing and
+the action are de-coupled. In the above example, this means that local file
+writes will happen immediately while the database writes are queued. As a
+side-note, each action can have its own queue, so if you would like to more than
+a single database or send messages reliably to another host, you can do all of
+this on their own queues, de-coupling their processing speeds.</p>
+<p>The configuration for the de-coupled database write involves just a few more
+commands:</p>
+<textarea rows="11" cols="80">$ModLoad ommysql.so # load the output driver (use ompgsql.so for PostgreSQL)
+$ModLoad imudp.so # network reception
+$UDPServerRun 514 # start a udp server at port 514
+$ModLoad imuxsock.so # local message reception
+
+$WorkDirectory /rsyslog/work # default location for work (spool) files
+
+$ActionQueueType LinkedList # use asynchronous processing
+$ActionQueueFileName dbq # set file name, also enables disk mode
+$ActionResumeRetryCount -1 # infinite retries on insert failure
+# for PostgreSQL replace :ommysql: by :ompgsql: below:
+*.* :ommysql:hostname,dbname,userid,password;
+</textarea>
+<p><b>This is the recommended configuration for this use case.</b> It requires
+rsyslog 3.11.0 or above.</p>
+<p>In this example, the main message queue is NOT disk-assisted (there is no
+$MainMsgQueueFileName directive). We still could do that, but have not done it
+because there seems to be no need. The only slow running action is the database
+writer and it has its own queue. So there is no real reason to use a large main
+message queue (except, of course, if you expect *really* heavy traffic bursts).</p>
<p>Note that you can modify a lot of queue performance parameters, but the above
config will get you going with default values. If you consider using this on a real
busy server, it is strongly recommended to invest some time in setting the tuning
@@ -102,6 +140,10 @@ comments or find bugs (I *do* bugs - no way... ;)), please
<li>2008-01-28 *
<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> * Initial Version created</li>
+ <li>2008-01-28 *
+
+ <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a>
+ * Updated to new v3.11.0 capabilities</li>
</ul>
<h2>Copyright</h2>