diff options
Diffstat (limited to 'doc')
81 files changed, 7608 insertions, 1602 deletions
diff --git a/doc/.cvsignore b/doc/.cvsignore deleted file mode 100644 index 282522db..00000000 --- a/doc/.cvsignore +++ /dev/null @@ -1,2 +0,0 @@ -Makefile -Makefile.in diff --git a/doc/Makefile.am b/doc/Makefile.am index 5dba5e89..d38d4d0b 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -1,26 +1,63 @@ html_files = \ + index.html \ bugs.html \ + debug.html \ features.html \ generic_design.html \ + expression.html \ history.html \ how2help.html \ install.html \ + build_from_repo.html \ ipv6.html \ + log_rotation_fix_size.html \ manual.html \ man_rsyslogd.html \ modules.html \ property_replacer.html \ + rsyslog_ng_comparison.html \ rsyslog_conf.html \ rsyslog-example.conf \ rsyslog_mysql.html \ rsyslog_packages.html \ + rsyslog_high_database_rate.html \ rsyslog_php_syslog_ng.html \ rsyslog_recording_pri.html \ + rsyslog_tls.html \ + rsyslog_reliable_forwarding.html \ rsyslog_stunnel.html \ - professional_support.html \ - syslog-protocol.html \ + syslog_protocol.html \ version_naming.html \ contributors.html \ + dev_queue.html \ + omsnmp.html \ + ommysql.html \ + omlibdbi.html \ + imfile.html \ + imtcp.html \ + imgssapi.html \ + imrelp.html \ + imuxsock.html \ + imklog.html \ + professional_support.html \ + queues.html \ + src/queueWorkerLogic.dia \ + queueWorkerLogic.jpg \ + queueWorkerLogic_small.jpg \ + tls_cert_100.jpg \ + tls_cert_ca.jpg \ + tls_cert.jpg \ + tls_cert_errmsgs.html \ + rsyslog_secure_tls.html \ + tls_cert_server.html \ + tls_cert_ca.html \ + tls_cert_summary.html \ + tls_cert_machine.html \ + tls_cert_udp_relay.html \ + tls_cert_client.html \ + tls_cert_scenario.html \ + rainerscript.html \ + rscript_abnf.html \ rsconf1_actionexeconlywhenpreviousissuspended.html \ rsconf1_actionresumeinterval.html \ rsconf1_allowedsender.html \ @@ -44,10 +81,24 @@ html_files = \ rsconf1_gssmode.html \ rsconf1_includeconfig.html \ rsconf1_mainmsgqueuesize.html \ + rsconf1_markmessageperiod.html \ rsconf1_modload.html \ rsconf1_moddir.html \ rsconf1_repeatedmsgreduction.html \ rsconf1_resetconfigvariables.html \ - rsconf1_umask.html + rsconf1_umask.html \ + v3compatibility.html \ + im3195.html \ + netstream.html \ + ns_gtls.html \ + ns_ptcp.html \ + src/tls_cert.dia \ + gssapi.html \ + licensing.html \ + ommail.html \ + omrelp.html \ + syslog_parsing.html \ + troubleshoot.html \ + src/classes.dia EXTRA_DIST = $(html_files) diff --git a/doc/build_from_repo.html b/doc/build_from_repo.html new file mode 100644 index 00000000..8d3b20fe --- /dev/null +++ b/doc/build_from_repo.html @@ -0,0 +1,54 @@ +<html><head> +<title>Building rsyslog from the source repository</title> +</head> +<body> +<h1>Building rsyslog from the source repository</h1> +<p>In most cases, people install rsyslog either via a package or use an "official" +distribution tarball to generate it. But there may be situations where it is desirable +to build directly from the source repository. This is useful for people who would like to +participate in development or who would like to use the latest, not-yet-released code. +The later may especially be the case if you are asked to try out an experimental version. +<p>Building from the repsitory is not much different than building from the source +tarball, but some files are missing because they are output files and thus do not +belong into the repository. +<h2>Obtaining the Source</h2> +<p>First of all, you need to download the sources. Rsyslog is currently kept in a git +repository. You can clone this repository either via http or git protocol (with the later +being much faster. URLS are: +<ul> +<li>git://git.adiscon.com/git/rsyslog.git +<li>http://git.adiscon.com/git/rsyslog.git +</ul> +<p>There is also a browsable version (gitweb) available at +<a href="http://git.adiscon.com/?p=rsyslog.git;a=summary">http://git.adiscon.com/?p=rsyslog.git;a=summary</a>. +This version also offers snapshots of each commit for easy download. You can use these if +you do not have git present on your system. +<p>After you have cloned the repository, you are in the master branch by default. This +is where we keep the devel branch. If you need any other branch, you need to do +a "git checkout --track -b branch origin/branch". For example, the command to check out +the beta branch is "git checkout --track -b beta origin/beta". +<h2>Prequisites</h2> +<p>To build the compilation system, you need the <b>pkg-config</b> package (an utility for +autotools) present on your system. Otherwise, configure will fail with something like +<pre><code> +checking for SYSLOG_UNIXAF support... yes +checking for FSSTND support... yes +./configure: line 25895: syntax error near unexpected token `RELP,' +./configure: line 25895: ` PKG_CHECK_MODULES(RELP, relp >= 0.1.1)' +</code></pre> +<h2>Creating the Build Environment</h2> +<p>This is fairly easy: just issue "<b>autoreconf -fvi</b>", which should do everything you need. +Once this is done, you can follow the usual ./configure steps just like when +you downloaded an official distribution tarball (see the +<a href="install.html">rsyslog install guide</a>, starting at step 2, +for further details about that). + +<p>[<a href="manual.html">manual index</a>] +[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 1.2 or higher.</font></p> +</body> +</html> diff --git a/doc/debug.html b/doc/debug.html new file mode 100644 index 00000000..de77f04a --- /dev/null +++ b/doc/debug.html @@ -0,0 +1,41 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>Debug Support</title></head> +<body> +<h1>Debug Support</h1> +<p> +Rsyslog provides a number of debug aids. Some of them are activated by +adding the --enable-rtinst ./configure option ("rtinst" means runtime +instrumentation). Turning debugging on obviously costs some performance +(in some cases considerable). +</p> +<p>This is document is just being created and thus terse.</p> +<p style="font-weight: bold;">Signals supported</p> +<p>SIGUSR1 - turns debug messages on and off (expect this signal +to go away over time)</p> +<p>SIGUSR2 - outputs debug information (including active threads +and a call stack) for the state when SIGUSR2 was received. This is a +one-time output. Can be sent as often as the user likes.</p> +<p style="font-weight: bold;">Environment Variables</p> +<p>There are two environment variables that set several debug settings. The "RSYSLOG_DEBUGLOG" (sample: RSYSLOG_DEBUGLOG="/path/to/debuglog/") +writes (allmost) +all debug message to the specified log file in addition to stdout. Some +system messages (e.g. segfault or abort message) are not written to the +file as we can not capture them. Runtime debug support is controlled by +"RSYSLOG_DEBUG". It contains an option string with the following +options possible (all are case insensitive):</p><ul><li><span style="font-weight: bold;">LogFuncFlow</span> - print out the logical flow of functions (entering and exiting them)</li><li><span style="font-weight: bold;">FileTrace</span> - specifies which files to trace LogFuncFlow. If <span style="font-weight: bold;">not</span> +set (the default), a LogFuncFlow trace is provided for all files. Set +to limit it to the files specified. FileTrace may be specified multiple +times, one file each (e.g. export RSYSLOG_DEBUG="LogFuncFlow +FileTrace=vm.c FileTrace=expr.c"</li><li><span style="font-weight: bold;">PrintFuncDB</span> - print the content of the debug function database whenever debug information is printed (e.g. abort case)!</li><li><span style="font-weight: bold;">PrintAllDebugInfoOnExit</span> - print all debug information immediately before rsyslogd exits (<span style="font-weight: bold; font-style: italic;">currently not implemented!</span>)</li><li><span style="font-weight: bold;">PrintMutexAction</span> - print mutex action as it happens. Useful for finding deadlocks and such.</li><li><span style="font-weight: bold;">NoLogTimeStamp</span> - do not prefix log lines with a timestamp (default is to do that).</li><li><span style="font-weight: bold;">NoStdOut</span> - do not emit debug messages to stdout. If RSYSLOG_DEBUGLOG is not set, this means no messages will be displayed at all.</li><li><span style="font-weight: bold;">help</span> - display a very short list of commands - hopefully a life saver if you can't access the documentation...</li></ul> +<ul> +</ul> +<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html>
\ No newline at end of file diff --git a/doc/dev_queue.html b/doc/dev_queue.html new file mode 100644 index 00000000..bf2af7f0 --- /dev/null +++ b/doc/dev_queue.html @@ -0,0 +1,250 @@ +<html> +<head> +<title>rsyslog queue object</title> +</head> +<body> +<h1>The rsyslog queue object</h1> +<p>This page reflects the status as of 2008-01-17. The documentation is still incomplete. +Target audience is developers and users who would like to get an in-depth understanding of +queues as used in <a href="http://www.rsyslog.com/">rsyslog</a>.</p> +<p><b>Please note that this document is outdated and does not longer reflect the +specifics of the queue object. However, I have decided to leave it in the doc +set, as the overall picture provided still is quite OK. I intend to update this +document somewhat later when I have reached the "store-and-forward" milestone.</b></p> +<h1>Some definitions</h1> +<p>A queue is DA-enabled if it is configured to use disk-assisted mode when +there is need to. A queue is in DA mode (or DA run mode), when it actually runs +disk assisted.</p> +<h1>Implementation Details</h1> +<h2>Disk-Assisted Mode</h2> +<p>Memory-Type queues may utilize disk-assisted (DA) mode. DA mode is enabled +whenever a queue file name prefix is provided. This is called DA-enabled mode. +If DA-enabled, the queue operates as a regular memory queue until a high water +mark is reached. If that happens, the queue activates disk assistance (called +"runs disk assisted" or "runs DA" - you can find that often in source file +comments). To do so, it creates a helper queue instance (the DA queue). At that +point, there are two queues running - the primary queue's consumer changes to a +shuffle-to-DA-queue consumer and the original primary consumer is assigned to +the DA queue. Existing and new messages are spooled to the disk queue, where the +DA worker takes them from and passes them for execution to the actual consumer. +In essence, the primary queue has now become a memory buffer for the DA queue. +The primary queue will be drained until a low water mark is reached. At that +point, processing is held. New messages enqueued to the primary queue will not +be processed but kept in memory. Processing resumes when either the high water +mark is reached again or the DA queue indicates it is empty. If the DA queue is +empty, it is shut down and processing of the primary queue continues as a +regular in-memory queue (aka "DA mode is shut down"). The whole thing iterates +once the high water mark is hit again.</p> +<p>There is one special case: if the primary queue is shut down and could not +finish processing all messages within the configured timeout periods, the DA +queue is instantiated to take up the remaining messages. These will be preserved +and be processed during the next run. During that period, the DA queue runs in +"enqueue-only" mode and does not execute any consumer. Draining the primary +queue is typically very fast. If that behaviour is not desired, it can be turned +of via parameters. In that case, any remaining in-memory messages are lost.</p> +<p>Due to the fact that when running DA two queues work closely together and +worker threads (including the DA worker) may shut down at any time (due to +timeout), processing synchronization and startup and shutdown is somewhat +complex. I'll outline the exact conditions and steps down here. I also do this +so that I know clearly what to develop to, so please be patient if the +information is a bit too in-depth ;)</p> +<h2>DA Run Mode Initialization</h2> +<p>Three cases:</p> +<ol> + <li>any time during queueEnqObj() when the high water mark is hit</li> + <li>at queue startup if there is an on-disk queue present (presence of QI + file indicates presence of queue data)</li> + <li>at queue shutdown if remaining in-memory data needs to be persisted to + disk</li> +</ol> +<p>In <b>case 1</b>, the worker pool is running. When switching to DA mode, all +regular workers are sent termination commands. The DA worker is initiated. +Regular workers may run in parallel to the DA worker until they terminate. +Regular workers shall terminate as soon as their current consumer has completed. +They shall not execute the DA consumer.</p> +<p>In <b>case 2</b>, the worker pool is not yet running and is NOT started. The +DA worker is initiated.</p> +<p>In <b>case 3</b>, the worker pool is already shut down. The DA worker is +initiated. The DA queue runs in enqueue-only mode.</p> +<p>In all cases, the DA worker starts up and checks if DA mode is already fully +initialized. If not, it initializes it, what most importantly means construction +of the queue.</p> +<p>Then, regular worker processing is carried out. That is, the queue worker +will wait on empty queue and terminate after an timeout. However, If any message +is received, the DA consumer is executed. That consumer checks the low water +mark. If the low water mark is reached, it stops processing until either the +high water mark is reached again or the DA queue indicates it is empty (there is +a pthread_cond_t for this synchronization).</p> +<p>In theory, a <b>case-2</b> startup could lead to the worker becoming inactive +and terminating while waiting on the primary queue to fill. In practice, this is +highly unlikely (but only for the main message queue) because rsyslog issues a +startup message. HOWEVER, we can not rely on that, it would introduce a race. If +the primary rsyslog thread (the one that issues the message) is scheduled very +late and there is a low inactivty timeout for queue workers, the queue worker +may terminate before the startup message is issued. And if the on-disk queue +holds only a few messages, it may become empty before the DA worker is +re-initiated again. So it is possible that the DA run mode termination criteria +occurs while no DA worker is running on the primary queue.</p> +<p>In cases 1 and 3, the DA worker can never become inactive without hitting the +DA shutdown criteria. In <b>case 1</b>, it either shuffles messages from the +primary to the DA queue or it waits because it has the hit low water mark. </p> +<p>In <b>case 3</b>, it always shuffles messages between the queues (because, +that's the sole purpose of that run). In order for this to happen, the high +water mark has been set to the value of 1 when DA run mode has been initialized. +This ensures that the regular logic can be applied to drain the primary queue. +To prevent a hold due to reaching the low water mark, that mark must be changed +to 0 before the DA worker starts.</p> +<h2>DA Run Mode Shutdown</h2> +<p>In essence, DA run mode is terminated when the DA queue is empty and the +primary worker queue size is below the high water mark. It is also terminated +when the primary queue is shut down. The decision to switch back to regular +(non-DA) run mode is typically made by the DA worker. If it switches, the DA +queue is destructed and the regular worker pool is restarted. In some cases, the +queue shutdown process may initiate the "switch" (in this case more or less a +clean shutdown of the DA queue).</p> +<p>One might think that it would be more natural for the DA queue to detect +being idle and shut down itself. However, there are some issues associated with +that. Most importantly, all queue worker threads need to be shut down during +queue destruction. Only after that has happend, final destruction steps can +happen (else we would have a myriad of races). However, it is the DA queues +worker thread that detects it is empty (empty queue detection always happens at +the consumer side and must so). That would lead to the DA queue worker thread to +initiate DA queue destruction which in turn would lead to that very same thread +being canceled (because workers must shut down before the queue can be +destructed). Obviously, this does not work out (and I didn't even mention the +other issues - so let's forget about it). As such, the thread that enqueues +messages must destruct the queue - and that is the primary queue's DA worker +thread.</p> +<p>There are some subleties due to thread synchronization and the fact that the +DA consumer may not be running (in a <b>case-2 startup</b>). So it is not +trivial to reliably change the queue back from DA run mode to regular run mode. +The priority is a clean switch. We accept the fact that there may be situations +where we cleanly shut down DA run mode, just to re-enable it with the very next +message being enqueued. While unlikely, this will happen from time to time and +is considered perfectly legal. We can't predict the future and it would +introduce too great complexity to try to do something against that (that would +most probably even lead to worse performance under regular conditions).</p> +<p>The primary queue's DA worker thread may wait at two different places:</p> +<ol> + <li>after reaching the low water mark and waiting for either high water or + DA queue empty</li> + <li>at the regular pthread_cond_wait() on an empty primary queue</li> +</ol> +<p>Case 2 is unlikely, but may happen (see info above on a case 2 startup).</p> +<p><b>The DA worker may also not wait at all,</b> because it is actively +executing and shuffeling messages between the queues. In that case, however, the +program flow passes both of the two wait conditions but simply does not wait.</p> +<p><b>Finally, the DA worker may be inactive </b>(again, with a case-2 startup). +In that case no work(er) at all is executed. Most importantly, without the DA +worker being active, nobody will ever detect the need to change back to regular +mode. If we have this situation, the very next message enqueued will cause the +switch, because then the DA run mode shutdown criteria is met. However, it may +take close to eternal for this message to arrive. During that time, disk and +memory resources for the DA queue remain allocated. This also leaves processing +in a sub-optimal state and it may take longer than necessary to switch back to +regular queue mode when a message burst happens. In extreme cases, this could +even lead to shutdown of DA run mode, which takes so long that the high water +mark is passed and DA run mode is immediately re-initialized - while with an +immediate switch, the message burst may have been able to be processed by the +in-memory queue without DA support.</p> +<p>So in short, it is desirable switch to regular run mode as soon as possible. +To do this, we need an active DA worker. The easy solution is to initiate DA +worker startup from the DA queue's worker once it detects empty condition. To do +so, the DA queue's worker must call into a "<i>DA worker startup initiation</i>" +routine inside the main queue. As a reminder, the DA worker will most probably +not receive the "DA queue empty" signal in that case, because it will be long +sent (in most cases) before the DA worker even waits for it. So <b>it is vital +that DA run mode termination checks be done in the DA worker before it goes into +any wait condition</b>.</p> +<p>Please note that the "<i>DA worker startup initiation</i>" routine may be +called concurrently from multiple initiators. <b>To prevent a race, it must be +guarded by the queue mutex </b>and return without any action (and no error +code!) if the DA worker is already initiated.</p> +<p>All other cases can be handled by checking the termination criteria +immediately at the start of the worker and then once again for each run. The +logic follows this simplified flow diagram:</p> +<p align="center"><a href="queueWorkerLogic.jpg"> +<img border="0" src="queueWorkerLogic_small.jpg" width="431" height="605"></a></p> +<p>Some of the more subtle aspects of worker processing (e.g. enqueue thread +signaling and other fine things) have been left out in order to get the big +picture. What is called "check DA mode switchback..." right after "worker init" +is actually a check for the worker's termination criteria. Typically, <b>the +worker termination criteria is a shutdown request</b>. However, <b>for a DA +worker, termination is also requested if the queue size is below the high water +mark AND the DA queue is empty</b>. There is also a third termination criteria +and it is not even on the chart: that is the inactivity timeout, which exists in +all modes. Note that while the inactivity timeout shuts down a thread, it +logically does not terminate the worker pool (or DA worker): workers are +restarted on an as-needed basis. However, inactivity timeouts are very important +because they require us to restart workers in some situations where we may +expect a running one. So always keep them on your mind.</p> +<h2>Queue Destruction</h2> +<p>Now let's consider <b>the case of destruction of the primary queue. </b>During +destruction, our focus is on loosing as few messages as possible. If the +queue is not DA-enabled, there is nothing but the configured timeouts to handle +that situation. However, with a DA-enabled queue there are more options.</p> +<p>If the queue is DA-enabled, it may be <i>configured to persist messages to +disk before it is terminated</i>. In that case, loss of messages never occurs +(at the price of a potentially lengthy shutdown). Even if that setting is not +applied, the queue should drain as many messages as possible to the disk. For +that reason, it makes no sense to wait on a low water mark. Also, if the queue +is already in DA run mode, it does not make any sense to switch back to regular +run mode during termination and then try to process some messages via the +regular consumer. It is much more appropriate the try completely drain the queue +during the remaining timeout period. For the same reason, it is preferred that +no new consumers be activated (via the DA queue's worker), as they only cost +valuable CPU cycles and, more importantly, would potentially be long(er)-running +and possibly be needed to be cancelled. To prevent all of that, <b>queue +parameters are changed for DA-enabled queues:</b> the high water mark is to 1 +and the low water mark to 0 on the primary queue. The DA queue is commanded to +run in enqueue-only mode. If the primary queue is <i>configured to persist +messages to disk before it is terminated</i>, its SHUTDOWN timeout is changed to +to eternal. These parameters will cause the queue to drain as much as possible +to disk (and they may cause a case 3 DA run mode initiation). Please note that +once the primary queue has been drained, the DA queue's worker will +automatically switch back to regular (non-DA) run mode. <b>It must be ensured +that no worker cancellation occurs during that switchback</b>. Please note that +the queue may not switch back to regular run mode if it is not <i>configured to +persist messages to disk before it is terminated</i>. In order to apply the new +parameters, <b>worker threads must be awakened.</b> Remember we may not be in DA +run mode at this stage. In that case, the regular workers must be awakened, which +then will switch to DA run mode. No worker may be active, in that case one must +be initiated. If in DA run mode and the DA worker is inactive, the "<i>DA +worker startup initiation</i>" must be called to activate it. That routine +ensures only one DA worker is started even with multiple concurrent callers - +this may be the case here. The DA queue's worker may have requested DA worker +startup in order to terminate on empty queue (which will probably not be honored +as we have changed the low water mark).</p> +<p>After all this is done, the queue destructor requests termination of the +queue's worker threads. It will use the normal timeouts and potentially cancel +too-long running worker threads. <b>The shutdown process must ensure that all +workers reach running state before they are commanded to terminate</b>. +Otherwise it may run into a race condition that could lead to a false shutdown +with workers running asynchronously. As a few workers may have just been started +to initialize (to apply new parameter settings), the probability for this race +condition is extremely high, especially on single-CPU systems.</p> +<p>After all workers have been shut down (or cancelled), the queue may still be +in DA run mode. If so, this must be terminated, which now can simply be done by +destructing the DA queue object. This is not a real switchback to regular run +mode, but that doesn't matter because the queue object will soon be gone away.</p> +<p>Finally, the queue is mostly shut down and ready to be actually destructed. +As a last try, the queuePersists() entry point is called. It is used to persists +a non-DA-enabled queue in whatever way is possible for that queue. There may be +no implementation for the specific queue type. Please note that this is not just +a theoretical construct. This is an extremely important code path when the DA +queue itself is destructed. Remember that it is a queue object in its own right. +The DA queue is obviously not DA-enabled, so it calls into queuePersists() +during its destruction - this is what enables us to persist the disk queue!</p> +<p>After that point, left over queue resources (mutexes, dynamic memory, ...) +are freed and the queue object is actually destructed.</p> +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> +and <a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p>Permission is granted to copy, distribute and/or modify this document under +the terms of the GNU Free Documentation License, Version 1.2 or any later +version published by the Free Software Foundation; with no Invariant Sections, +no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be +viewed at <a href="http://www.gnu.org/copyleft/fdl.html"> +http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body> +</html>
\ No newline at end of file diff --git a/doc/expression.html b/doc/expression.html new file mode 100644 index 00000000..e7eb7842 --- /dev/null +++ b/doc/expression.html @@ -0,0 +1,16 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>Expressions</title></head> +<body> +<h1>Expressions</h1> +<p>Rsyslog supports expressions at a growing number of places. So +far, they are supported for filtering messages.</p><p>Expression support is provided by RainerScript. For now, please see the formal expression definition in <a href="rainerscript.html">RainerScript ABNF</a>. It is the "expr" node.</p><p>C-like comments (/* some comment */) are supported <span style="font-weight: bold;">inside</span> the expression, but not yet in the rest of the configuration file.</p><p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html>
\ No newline at end of file diff --git a/doc/features.html b/doc/features.html index c71194dc..d221eb77 100644 --- a/doc/features.html +++ b/doc/features.html @@ -1,66 +1,137 @@ -<html> -<head> -<title>rsyslog features</title> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>rsyslog features</title> + </head> <body> <h1>RSyslog - Features</h1> -<p><b>This page lists both current features as well as those being considered -for future versions of rsyslog.</b> If you think a feature is missing, drop -<a href="mailto:rgerhards@adiscon.com">Rainer</a> a note. Rsyslog is a vital -project. Features are added each few days. If you would like to keep up of what -is going on, you can also subscribe to the <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">rsyslog mailing list</a>. +<p><b>This page lists both current features as well as +those being considered for future versions of rsyslog.</b> If you +think a feature is missing, drop +<a href="mailto:rgerhards@adiscon.com">Rainer</a> a +note. Rsyslog is a vital project. Features are added each few days. If +you would like to keep up of what is going on, you can also subscribe +to the <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">rsyslog +mailing list</a>.</p> +<p><span style="font-weight: bold;">A better +structured feature list is now contained in our </span><a style="font-weight: bold;" href="rsyslog_ng_comparison.html">rsyslog +vs. syslog-ng comparison</a><span style="font-weight: bold;">. +</span>Probably that page will replace this one in the +future. </p> <h2>Current Features</h2> <ul> - - <li>native support for <a href="rsyslog_mysql.html">writing to MySQL databases</a><li> - native support for writing to Postgres databases<li>support for (plain) tcp - based syslog - much better reliability<li>support for sending and receiving - compressed syslog messages<li>ability to configure backup syslog/database - servers - if the primary fails, control is switched to a prioritized list of - backups<li>support for receiving messages via
reliable <a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php">
RFC 3195</a> delivery<li>ability to generate file names and directories (log targets) - dynamically, based on many different properties<li>control of log output format, - including ability to present channel and priority as visible log data<li>good timestamp format control; at a minimum, ISO 8601/RFC 3339 - second-resolution UTC zone<li>ability to reformat message contents and work with substrings<li>support for - log files larger than 2gb<li>support for file size limitation and automatic - rollover command execution<li>support for running multiple rsyslogd - instances on a single machine<li>support for <a href="rsyslog_stunnel.html"> - ssl-protected syslog</a> (via stunnel)<li>ability to filter on any part of - the message, not just facility and severity<li>ability to use regular - expressions in filters<li>support for discarding - messages based on filters<li>ability to execute shell scripts on received - messages<li>control of whether the local hostname or the hostname of the - origin of the data is shown as the hostname in the output<li>ability to - preserve the original hostname in NAT environments and relay chains - <li>ability to limit the allowed network senders<li>powerful BSD-style
hostname and program name blocks for easy multi-host support<li>
multi-threaded (<a href="http://rgerhards.blogspot.com/2007/08/why-is-rsyslog-multi-threaded-and-is-it.html">is - this important? why?</a>)<li>very
experimental and volatile support for <a href="syslog-protocol.html">syslog-protocol</a>
compliant messages (it is volatile because standardization is currently
underway and this is a proof-of-concept implementation to aid this effort)<li> - experimental support for syslog-transport-tls based framing on syslog/tcp - connections<li> - a copy of klogd.c has been included under the name of rklogd for those Linux - systems that need one. So rsyslog is a full replacement for the sysklogd - package<li> - support for IPv6<li> - ability to control repeated line reduction ("last message repeated n times") - on a per selector-line basis<li> - supports sub-configuration files, which can be automatically read from - directories. Includes are specified in the main configuration file<li> - supports multiple actions per selector/filter condition<li> - MySQL and Postgres SQL functionality as a dynamically loadable plug-in<li> - support for GSS-API<li> - modular design for outputs - easily extensible</ul> -<p> </p> +<li>native support for <a href="rsyslog_mysql.html">writing +to MySQL databases</a></li> +<li> native support for writing to Postgres databases</li> +<li>direct support for Firebird/Interbase, +OpenTDS (MS SQL, Sybase), SQLLite, Ingres, Oracle, and mSQL via libdbi, +a database abstraction layer (almost as good as native)</li> +<li>native support for <a href="ommail.html">sending +mail messages</a> (first seen in 3.17.0)</li> +<li>support for (plain) tcp based syslog - much better +reliability</li> +<li>support for sending and receiving compressed syslog messages</li> +<li>support for on-demand on-disk spooling of messages that can +not be processed fast enough (a great feature for <a href="rsyslog_high_database_rate.html">writing massive +amounts of syslog messages to a database</a>)</li> +<li>support for selectively <a href="http://wiki.rsyslog.com/index.php/OffPeakHours">processing +messages only during specific timeframes</a> and spooling them to +disk otherwise</li> +<li>ability to monitor text files and convert their contents +into syslog messages (one per line)</li> +<li>ability to configure backup syslog/database servers - if +the primary fails, control is switched to a prioritized list of backups</li> +<li>support for receiving messages via reliable <a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php"> +RFC 3195</a> delivery (a bit clumpsy to build right now...)</li> +<li>ability to generate file names and directories (log +targets) dynamically, based on many different properties</li> +<li>control of log output format, including ability to present +channel and priority as visible log data</li> +<li>good timestamp format control; at a minimum, ISO 8601/RFC +3339 second-resolution UTC zone</li> +<li>ability to reformat message contents and work with +substrings</li> +<li>support for log files larger than 2gb</li> +<li>support for file size limitation and automatic rollover +command execution</li> +<li>support for running multiple rsyslogd instances on a single +machine</li> +<li>support for <a href="rsyslog_tls.html">TLS-protected +syslog</a> (both <a href="rsyslog_tls.html">natively</a> +and via <a href="rsyslog_stunnel.html">stunnel</a>)</li> +<li>ability to filter on any part of the message, not just +facility and severity</li> +<li>ability to use regular expressions in filters</li> +<li>support for discarding messages based on filters</li> +<li>ability to execute shell scripts on received messages</li> +<li>control of whether the local hostname or the hostname of +the origin of the data is shown as the hostname in the output</li> +<li>ability to preserve the original hostname in NAT +environments and relay chains </li> +<li>ability to limit the allowed network senders</li> +<li>powerful BSD-style hostname and program name blocks for +easy multi-host support</li> +<li> massively multi-threaded with dynamic work thread pools +that start up and shut themselves down on an as-needed basis (great for +high log volume on multicore machines)</li> +<li>very experimental and volatile support for <a href="syslog_protocol.html">syslog-protocol</a> +compliant messages (it is volatile because standardization is currently +underway and this is a proof-of-concept implementation to aid this +effort)</li> +<li> world's first implementation of syslog-transport-tls</li> +<li> the sysklogd's klogd functionality is implemented as the <i>imklog</i> +input plug-in. So rsyslog is a full replacement for the sysklogd package</li> +<li> support for IPv6</li> +<li> ability to control repeated line reduction ("last message +repeated n times") on a per selector-line basis</li> +<li> supports sub-configuration files, which can be +automatically read from directories. Includes are specified in the main +configuration file</li> +<li> supports multiple actions per selector/filter condition</li> +<li> MySQL and Postgres SQL functionality as a dynamically +loadable plug-in</li> +<li> modular design for inputs and outputs - easily extensible +via custom plugins</li> +<li> an easy-to-write to plugin interface</li> +<li> ability to send SNMP trap messages</li> +<li> ability to filter out messages based on sequence of arrival</li> +<li>support for arbitrary complex boolean, string and +arithmetic expressions in message filters</li> +</ul> +<h2>World's first</h2> +Rsyslog has an interesting number of "world's firsts" - things that +were implemented for the first time ever in rsyslog. Some of them are still features not available elsewhere.<br><ul> +<li>world's first implementation of IETF I-D syslog-protocol (February 2006, version 1.12.2 and above)</li><li>world's first implementation of dynamic syslog on-the-wire compression (December 2006, version 1.13.0 and above)</li><li>world's first open-source implementation of a disk-queueing syslogd (January 2008, version 3.11.0 and above)</li> +<li>world's first implementation of IETF I-D +syslog-transport-tls (May 2008, version 3.19.0 and above)</li> +</ul> <h2>Upcoming Features</h2> -<p>The list below is something like a repository of ideas we'd like to -implement. Features on this list are typically NOT scheduled for immediate -inclusion. We maintain a -<a href="http://bugzilla.adiscon.com/rsyslog-feature.html">feature -request tracker at our bugzilla</a>. This tracker has things typically within -reach of implementation. Users are encouraged to submit feature requests there -(or via our forums). <b>Please note that rsyslog v2 is feature-complete. New -features will be implemented in the v3 branch only. </b>Version 3 already has a -number of very exciting additional features.</p> +<p>The list below is something like a repository of ideas we'd +like to implement. Features on this list are typically NOT scheduled +for immediate inclusion. We maintain a +<a href="http://bugzilla.adiscon.com/rsyslog-feature.html">feature +request tracker at our bugzilla</a>. This tracker has things +typically within reach of implementation. Users are encouraged to +submit feature requests there (or via our forums). If we like them but +they look quite long-lived (aka "not soon to be implemented"), they +will possibly be migrated to this list here and at some time moved back +to the bugzilla tracker.</p> +<ul> +<li>port it to more *nix variants (eg AIX and HP UX) - this +needs volunteers with access to those machines and knowledge </li> +<li>pcre filtering - maybe (depending on feedback) - +simple regex already partly added. So far, this seems sufficient so +that there is no urgent need to do pcre. If done, it will be a loadable RainerScript function.</li> +<li>support for <a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php">RFC +3195</a> as a sender - this is currently unlikely to happen, +because there is no real demand for it. Any work on RFC 3195 has been +suspend until we see some real interest in it. It is probably +much better to use TCP-based syslog, which is interoperable with a +large number of applications. You may also read my blog post on the +future of liblogging, which contains interesting information about the <a href="http://rgerhards.blogspot.com/2007/09/where-is-liblogging-heading-to.html"> +future of RFC 3195 in rsyslog</a>.</li> +</ul> <p>To see when each feature was added, see the -<a href="http://www.rsyslog.com/Topic4.phtml">rsyslog change log</a> (online -only).</p> -</body> -</html> +<a href="http://www.rsyslog.com/Topic4.phtml">rsyslog +change log</a> (online only).</p> +</body></html> diff --git a/doc/free_support.html b/doc/free_support.html new file mode 100644 index 00000000..182a82cd --- /dev/null +++ b/doc/free_support.html @@ -0,0 +1,56 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>Free Support for Rsyslog</title> + +</head> +<body> +<h1>Free Services for Rsyslog</h1> +<p><i>A personal word from Rainer, the lead developer of rsyslog:</i> +<p><b>The rsyslog community provides ample free support resources. Please see our +<a href="troubleshoot.html">troubleshooting guide</a> to get started.</b></p> +<p>Every now and then I receive private mail with support questions. I appreciate +any feedback, but I must limit my resources so that I can help driver a great logging +system forward. +<p>To do so, I have decided not to reply to unsolicited support emails, at least not +with a solution (but rather a link to this page ;)). I hope this does not offend you. The +reason is quite simple: If I do personal support, you gain some advantage without +contributing something back. Think about it: if you ask your question on the public +forum or mailing list, other with the same problem can you and, most importantly, even +years later find your post (and the answer) and get the problem solved. So by +solving your issue in public, you help create a great community ressource and also +help your fellow users finding solutions quicker. In the long term, this +also contributes to improved code because the more questions users can find +solutions to themselves, the fewer I need to look at. +<p>But it comes even better: the rsyslog community is much broader than Rainer ;) - there +are helpful other members hanging around at the public places. They often answer +questions, so that I do not need to look at them (btw, once again a big "thank you", folks!). +And, more important, those folks have different background than me. So they often +either know better how to solve your problem (e.g. because it is distro-specific) +or they know how to better phrase it (after all, I like abstract terms and concepts ;)). +So you do yourself a favor if you use the public places. +<p>An excellent place to go to is the +<a href="http://kb.monitorware.com/rsyslog-f40.html">rsyslog forum</a> inside the +knowledge base (which in itself is a great place to visit!). For those used to +mailing lists, the +<a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">rsyslog mailing list</a> +also offers excellent advise. +<p><b>Don't like to post your question in a public place?</b> Well, then you should +consider purchasing <a href="professional_support.html">rsyslog professional support</a>. +The fees are very low and help fund the project. If you use rsyslog seriously inside +a corporate environment, there is no excuse for not getting one of the support +packages ;) +<p>Of course, things are different when I ask you to mail me privately. I'll usually do +that when I think it makes sense, for example when we exchange debug logs. +<p>I hope you now understand the free support options and the reasoning for them. +I hope I haven't offended you with my words - this is not my intension. I just needed to +make clear why there are some limits on my responsiveness. Happy logging! +<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/gssapi.html b/doc/gssapi.html new file mode 100644 index 00000000..3ad7d07b --- /dev/null +++ b/doc/gssapi.html @@ -0,0 +1,118 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>GSSAPI module support in rsyslog v3</title> + +</head> +<body> +<h1>GSSAPI module support in rsyslog v3</h1> +<p style="font-weight: bold;">What is it good for.</p> +<ul style="margin-left: 1.25cm;"> +<li> +client-serverauthentication </li> +<li> +Log +messages encryption </li> +</ul> +<p class="P5"> </p> +<p class="P3"><span style="font-weight: bold;">Requirements.</span> +</p> +<ul> +<li>Kerberos infrastructure</li> +<li>rsyslog, rsyslog-gssapi</li> +</ul> +<p> </p> +<p><span style="font-weight: bold;">Configuration.</span> +</p> +<p>Let's assume there are 3 machines in kerberos Realm: </p> +<ul> +<li>the +first is running KDC (Kerberos Authentication Service and Key +Distribution Center),</li> +<li>the second is a client sending its logs to the server,</li> +<li>the third is receiver, gathering all logs.</li> +</ul> +<p class="P7"> </p> +<p class="P10"><span style="font-style: italic;">1. +KDC:</span> </p> +<ul> +<li>Kerberos +database must be properly set-up on KDC machine first. Use +kadmin/kadmin.local to do that. Two principals need to be add in our +case:</li> +</ul> +<ol style="margin-left: 1.25cm; list-style-type: decimal;"> +<li> +<p>sender@REALM.ORG +</p> +</li> +</ol> +<ul> +<li>client must have ticket for pricipal sender</li> +<li>REALM.ORG is kerberos Realm</li> +</ul> +<ol style="margin-left: 1.25cm; list-style-type: decimal;"> +<li>host/receiver.mydomain.com@REALM.ORG - service principal</li> +</ol> +<ul> +<li>Use ktadd to export service principal and transfer it to +/etc/krb5.keytab +on receiver </li> +</ul> +<p><span style="font-style: italic;">2. CLIENT:</span> +</p> +<ul> +<li>set-up rsyslog, in /etc/rsyslog.conf</li> +<li>$ModLoad omgssapi - load output gss module </li> +<li>$GSSForwardServiceName +otherThanHost - set the name of service principal, "host" is the +default one</li> +<li>*.* :omgssapi:receiver.mydomain.com - action line, forward +logs to receiver</li> +<li>kinit root - get the TGT ticket</li> +<li>service rsyslog start +<p class="P14" style="margin-left: 0.25cm;"> </p> +</li> +</ul> +<p><span style="font-style: italic;">3. SERVER:</span> +</p> +<ul> +<li class="P14" style="margin-left: 0cm;"> +<p class="P14" style="margin-left: 0.25cm;">set-up +rsyslog, in /etc/rsyslog.conf </p> +</li> +<li class="P16"> +<p class="P16" style="margin-left: 0.25cm;">$ModLoad +<a href="imgssapi.html">imgssapi</a> - load input gss module </p> +</li> +<li class="P16"> +<p class="P16" style="margin-left: 0.25cm;">$InputGSSServerServiceName +otherThanHost - set the name of service principal, "host" is the +default one </p> +</li> +<li class="P16"> +<p class="P16" style="margin-left: 0.25cm;">$InputGSSServerPermitPlainTCP +on - accept GSS and TCP connections (not authenticated senders), off by +default </p> +</li> +<li class="P16"> +<p class="P16" style="margin-left: 0.25cm;">$InputGSSServerRun +514 - run server on port </p> +</li> +<li class="P14" style="margin-left: 0cm;"> +<p class="P14" style="margin-left: 0.25cm;">service +rsyslog start </p> +</li> +</ul> +<span style="font-weight: bold;">The picture demonstrate +how things work.</span> +<p class="P18"> </p> +<img src="gssapi.png" alt="rsyslog gssapi support"> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/gssapi.png b/doc/gssapi.png Binary files differnew file mode 100644 index 00000000..c82baa52 --- /dev/null +++ b/doc/gssapi.png diff --git a/doc/history.html b/doc/history.html index 0f9dbffa..a06aaf5d 100644 --- a/doc/history.html +++ b/doc/history.html @@ -1,7 +1,6 @@ -<html> -<head> -<title>rsyslog history</title> -</head> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>rsyslog history</title></head> <body> <h1>RSyslog - History</h1> @@ -10,28 +9,32 @@ reliable syslog over TCP, writing to MySQL databases and fully configurable output formats (including great timestamps).</b> Rsyslog was initiated by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a>. If you are interested to learn why Rainer initiated the project, you -may want to read his blog posting on "<a href="http://rgerhards.blogspot.com/2007/08/why-does-world-need-another-syslogd.html">why -the world needs another syslogd</a>".<p>Rsyslog has +may want to read his blog posting on "<a href="http://rgerhards.blogspot.com/2007/08/why-does-world-need-another-syslogd.html">why +the world needs another syslogd</a>".<p>Rsyslog has been forked in <b>2004</b> from the <a href="http://www.infodrom.org/projects/sysklogd/">sysklogd standard package</a>. The goal of the rsyslog project is to provide a feature-richer and reliable -syslog daemon while retaining drop-in replacement capabilities to stock syslogd. By "reliable", we mean support for reliable transmission +syslog daemon while retaining drop-in replacement capabilities to stock +syslogd. By "reliable", we mean support for reliable transmission modes like TCP or <a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php">RFC 3195</a> (syslog-reliable). We do NOT imply that the sysklogd package is unreliable.</p> <p>The name "rsyslog" stems back to the planned support for syslog-reliable. Ironically, the initial release of rsyslog did NEITHER support syslog-reliable NOR tcp based syslog. Instead, it contained enhanced configurability and other enhancements -(like database support). The reason for this is that full support for RFC 3195 would require even more changes and especially fundamental architectural +(like database support). The reason for this is that full support for +RFC 3195 would require even more changes and especially fundamental +architectural changes. Also, questions asked on the loganalysis list and at other places indicated that RFC3195 is NOT a prime priority for users, but rather better control over the output format. So there we were, with a rsyslogd that covers a lot of enhancements, but not a single one -of these that made its name ;) Since version 0.9.2, receiving syslog messages -via plain tcp is finally supported, a bit later sending via TCP, too. Starting -with 1.11.0, RFC 3195 is finally support at the receiving side (a.k.a. "listener"). -Support for sending via RFC 3195 is still due. Anyhow, rsyslog has come much -closer to what it name promises.</p> +of these that made its name ;) Since version 0.9.2, receiving syslog +messages via plain tcp is finally supported, a bit later sending via +TCP, too. Starting with 1.11.0, RFC 3195 is finally supported at the +receiving side (a.k.a. "listener"). Support for sending via RFC 3195 is +still due. Anyhow, rsyslog has come much closer to what it name +promises.</p> <p> The database support was initially included so that our web-based syslog interface could be used. This is another open source project which can be found @@ -50,31 +53,31 @@ the syslogd binary with the one that comes with rsyslog. Of course, in order to use any of the new features, you must re-write your syslog.conf. To learn how to do this, please review our commented <a href="sample.conf.php">sample.conf</a> file. It outlines the enhancements over stock syslogd. Discussion has often -arisen of whether having an "old syslogd" logfile format is good or evil. So +arisen of whether having an "old syslogd" logfile format is good or evil. So far, this has not been solved (but Rainer likes the idea of a new format), so we need to live with it for the time being. It is planned to be reconsidered in the 3.x release time frame. -<p>If you are interested in the <a href="http://en.wikipedia.org/wiki/IHE">IHE</a> +</p><p>If you are interested in the <a href="http://en.wikipedia.org/wiki/IHE">IHE</a> environment, you might be interested to hear that rsyslog supports message with sizes of 32k and more. This feature has been tested, but by default is turned off (as it has some memory footprint that we didn't want to put on users not -actually requiring it). Search the file syslogd.c and search for "IHE" - you +actually requiring it). Search the file syslogd.c and search for "IHE" - you will find easy and precise instructions on what you need to change (it's just one line of code!). Please note that RFC 3195/COOKED supports 1K message sizes only. It'll probably support longer messages in the future, but it is our believe that using larger messages with current RFC 3195 is a violation of the -standard.<p>In <b>February 2007</b>, 1.13.1 was released and served for quite a +standard.</p><p>In <b>February 2007</b>, 1.13.1 was released and served for quite a while as a stable reference. Unfortunately, it was not later released as stable, -so the stable build became quite outdated.<p>In <b>June 2007</b>, Peter Vrabec from Red Hat helped us to create +so the stable build became quite outdated.</p><p>In <b>June 2007</b>, Peter Vrabec from Red Hat helped us to create RPM files for Fedora as well as supporting IPv6. There also seemed to be some interest from the Red Hat community. This interest and new ideas resulted in a -very busy time with many great additions.<p>In <b>July 2007</b>, Andrew +very busy time with many great additions.</p><p>In <b>July 2007</b>, Andrew Pantyukhin added BSD ports files for rsyslog and liblogging. We were strongly encouraged by this too. It looks like rsyslog is getting more and more momentum. -Let's see what comes next...<p>Also in <b>July 2007</b> (and beginning of +Let's see what comes next...</p><p>Also in <b>July 2007</b> (and beginning of August), Rainer remodeled the output part of rsyslog. It got a clean object model and is now prepared for a plug-in architecture. During that time, some base -ideas for the overall new object model appeared.<p>In <b>August 2007</b> +ideas for the overall new object model appeared.</p><p>In <b>August 2007</b> community involvement grew more and more. Also, more packages appeared. We were quite happy about that. To facilitate user contributions, we set up a <a href="http://wiki.rsyslog.com/">wiki</a> on August 10th, 2007. Also in August @@ -82,7 +85,7 @@ quite happy about that. To facilitate user contributions, we set up a 2.0.0 release. With its appearance, the pace of changes was deliberately reduced, in order to allow it to mature (see Rainers's <a href="http://rgerhards.blogspot.com/2007/07/pace-of-changes-in-rsyslog.html"> -blog post</a> on this topic, written a bit early, but covering the essence).<p> +blog post</a> on this topic, written a bit early, but covering the essence).</p><p> In <b>November 2007</b>, rsyslog became the default syslogd in Fedora 8. Obviously, that was something we *really* liked. Community involvement also is still growing. There is one sad thing to note: ever since summer, there is an @@ -90,7 +93,7 @@ extremely hard to find segfault bug. It happens on very rare occasions only and never in lab. We are hunting this bug for month now, but still could not get hold of it. Unfortunately, this also affects the new features schedule. It makes limited sense to implement new features if problems with existing ones are not -really understood.<p><b>December 2007</b> showed the appearance of a postgres +really understood.</p><p><b>December 2007</b> showed the appearance of a postgres output module, contributed by sur5r. With 1.20.0, December is also the first time since the bug hunt that we introduce other new features. It has been decided that we carefully will add features in order to not affect the overall project @@ -98,13 +101,24 @@ by these rare bugs. Still, the bug hunt is top priority, but we need to have mor data to analyze. At then end of December, it looked like the bug was found (a race condition), but further confirmation from the field is required before declaring victory. December also brings the initial development on <b>rsyslog v3</b>, -resulting in loadable input modules, now running on a separate thread each.<p>On +resulting in loadable input modules, now running on a separate thread each.</p><p>On <b>January, 2nd 2008</b>, rsyslog 1.21.2 is re-released as rsyslog v2.0.0 stable. This is a major milestone as far as the stable build is concerned. v3 is not yet officially announced. Other than the stable v2 build, v3 will not be backwards compatibile (including missing compatibility to stock sysklogd) for quite a while. Config file changes are required and some command line options do -no longer work due to the new design.<p>Be sure to visit Rainer's <a href="http://rgerhards.blogspot.com/">syslog blog</a> +no longer work due to the new design.</p><p>On <span style="font-weight: bold;">January, 31st 2008</span> +the new massively-multithreaded queue engine was released for the first +time. It was a major milestone, implementing a feature I dreamed of for +more than a year.</p><p>End of <span style="font-weight: bold;">February 2008</span> +saw the first note about RainerScript, a way to configure rsyslogd via +a script-language like configuration format. This effort evolved out of +the need to have complex expression support, which was also the first +use case. On February, 28th rsyslog 3.12.0 was released, the first +version to contain expression support. This also meant that rsyslog +from that date on supported all syslog-ng major features, but had a +number of major features exlusive to it. With 3.12.0, I consider +rsyslog fully superior to syslog-ng (except for platform support).</p><p>Be sure to visit Rainer's <a href="http://rgerhards.blogspot.com/">syslog blog</a> to get some more insight into the development and futures of rsyslog and syslog in general. Don't be shy to post to either the blog or the <a href="http://www.rsyslog.com/PNphpBB2.phtml">rsyslog forums</a>.</p> @@ -112,5 +126,4 @@ Don't be shy to post to either the blog or the <ul> <li><a href="http://www.rsyslog.com/Topic4.phtml">the rsyslog change log</a></li> </ul> -</body> -</html> +</body></html>
\ No newline at end of file diff --git a/doc/im3195.html b/doc/im3195.html new file mode 100644 index 00000000..d6f2f2ed --- /dev/null +++ b/doc/im3195.html @@ -0,0 +1,46 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>RFC3195 Input Module (im3195)</title> + +</head> +<body> +<h1>RFC3195 Input Module</h1> +<p><b>Module Name: im3195</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>Receives syslog messages via RFC 3195. The RAW profile is fully implemented and the +COOKED profile is provided in an experimental state. This module uses +<a href="http://www.liblogging.org">liblogging</a> for the actual protocol handling.</p> +<p><b>Configuration Directives</b>:</p> +<ul> +<li><strong>$Input3195ListenPort <port></strong><br> +The port on which imklog listens for RFC 3195 messages. The default port is 601 +(the IANA-assigned port)</li> +</ul> +<b>Caveats/Known Bugs:</b> +<p>Due to no demand at all for RFC3195, we have converted rfc3195d +to this input module, but we have NOT conducted any testing. Also, +the module does not yet properly handle the recovery case. If someone +intends to put this module into production, good testing should be +cunducted. It also is a good idea to notify the rsyslog project that you intend to use +it in production. In this case, we'll probably give the module another +cleanup. We don't do this now because so far it looks just like a big +waste of time. +<p>Currently only a single listener can be defined. That one binds to all interfaces.</p> +<p><b>Sample:</b></p> +<p>The following sample accepts syslog messages via RFC 3195 on port 1601. +<br> +</p> +<textarea rows="15" cols="60">$ModLoad im3195 +$Input3195ListenPort 1601 +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/imfile.html b/doc/imfile.html new file mode 100644 index 00000000..5bdbce5c --- /dev/null +++ b/doc/imfile.html @@ -0,0 +1,132 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>Text File Input Monitor</title></head> +<body> +<h1>Text File Input Module</h1> +<p><b>Module Name: imfile</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>Provides the ability to convert any standard text file into +a syslog message. A standard +text file is a file consisting of printable characters with lines +being delimited by LF.</p> +<p>The file is read line-by-line and any line read is passed to +rsyslog's rule engine. The rule engine applies filter conditons and +selects which actions needs to be carried out.</p> +<p>As new lines are written they are taken from the file and +processed. Please note that this happens based on a polling interval +and not immediately. The file monitor support file rotation. To fully +work, rsyslogd must run while the file is rotated. Then, any remaining +lines from the old file are read and processed and when done with that, +the new file is being processed from the beginning. If rsyslogd is +stopped during rotation, the new file is read, but any not-yet-reported +lines from the previous file can no longer be obtained.</p> +<p>When rsyslogd is stopped while monitoring a text file, it +records the last processed location and continues to work from there +upon restart. So no data is lost during a restart (except, as noted +above, if the file is rotated just in this very moment).</p> +<p>Currently, the file must have a fixed name and location +(directory). It is planned to add support for dynamically generating +file names in the future.</p> +<p>Multiple files may be monitored by specifying +$InputRunFileMonitor multiple times. +</p> +<p><b>Configuration Directives</b>:</p> +<ul> +<li><strong>$InputFileName /path/to/file</strong><br> +The file being monitored. So far, this must be an absolute name (no +macros or templates)</li> +<li><span style="font-weight: bold;">$InputFileTag +tag:</span><br> +The tag to be used for messages that originate from this file. If you +would like to see the colon after the tag, you need to specify it here +(as shown above).</li> +<li><span style="font-weight: bold;">$InputFileStateFile +<name-of-state-file></span><br> +Rsyslog must keep track of which parts of the to be monitored file it +already processed. This is done in the state file. This file always is +created in the rsyslog working directory (configurable via +$WorkDirectory). Be careful to use unique names for different files +being monitored. If there are duplicates, all sorts of "interesting" +things may happen. Rsyslog currently does not check if a name is +specified multiple times.</li> +<li><span style="font-weight: bold;">$InputFileFacility +facility</span><br> +The syslog facility to be assigned to lines read. Can be specified in +textual form (e.g. "local0", "local1", ...) or as numbers (e.g. 128 for +"local0"). Textual form is suggested. <span style="font-weight: bold;">Default</span> is +"local0".<span style="font-weight: bold;"></span></li> +<li><span style="font-weight: bold;">$InputFileSeverity</span><br> +The +syslog severity to be assigned to lines read. Can be specified in +textual form (e.g. "info", "warning", ...) or as numbers (e.g. 4 for +"info"). Textual form is suggested. <span style="font-weight: bold;">Default</span> +is "notice".</li> +<li><span style="font-weight: bold;">$InputRunFileMonitor</span><br> +This <span style="font-weight: bold;">activates</span> +the current monitor. It has no parameters. If you forget this +directive, no file monitoring will take place.</li> +<li><span style="font-weight: bold;">$InputFilePollInterval +seconds</span><br> +This is a global setting. It specifies how often files are to be polled +for new data. The time specified is in seconds. The <span style="font-weight: bold;">default value</span> is 10 +seconds. Please note that future +releases of imfile may support per-file polling intervals, but +currently this is not the case. If multiple $InputFilePollInterval +statements are present in rsyslog.conf, only the last one is used.<br> +A short poll interval provides more rapid message forwarding, but +requires more system ressources. While it is possible, we stongly +recommend not to set the polling interval to 0 seconds. That will make +rsyslogd become a CPU hog, taking up considerable ressources. It is +supported, however, for the few very unusual situations where this +level may be needed. Even if you need quick response, 1 seconds should +be well enough. Please note that imfile keeps reading files as long as +there is any data in them. So a "polling sleep" will only happen when +nothing is left to be processed.</li> +</ul> +<b>Caveats/Known Bugs:</b> +<p>So far, only 100 files can be monitored. If more are needed, +the source needs to be patched. See define MAX_INPUT_FILES in imfile.c</p><p>Powertop +users may want to notice that imfile utilizes polling. Thus, it is no +good citizen when it comes to conserving system power consumption. We +are currently evaluating to move to inotify(). However, there are a +number of subtle issues, which needs to be worked out first. We will +make the change as soon as we can. If you can afford it, we recommend +using a long polling interval in the mean time. +</p> +<p><b>Sample:</b></p> +<p>The following sample monitors two files. If you need just one, +remove the second one. If you need more, add them according to the +sample ;). This code must be placed in /etc/rsyslog.conf (or wherever +your distro puts rsyslog's config files). Note that only commands +actually needed need to be specified. The second file uses less +commands and uses defaults instead.<br> +</p> +<textarea rows="15" cols="60">$ModLoad imfile # +needs to be done just once +# File 1 +$InputFileName /path/to/file1 +$InputFileTag tag1: +$InputFileStateFile stat-file1 +$InputFileSeverity error +$InputFileFacility local7 +$InputRunFileMonitor +# File 2 +$InputFileName /path/to/file2 +$InputFileTag tag2: +$InputFileStateFile stat-file2 +$InputRunFileMonitor +# ... and so on ... +# +# check for new lines every 10 seconds +$InputFilePollingInterval 10 +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/imgssapi.html b/doc/imgssapi.html new file mode 100644 index 00000000..d644303e --- /dev/null +++ b/doc/imgssapi.html @@ -0,0 +1,52 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>GSSAPI Syslog Input Module</title> + +</head> +<body> +<h1>GSSAPI Syslog Input Module</h1> +<p><b>Module Name: imgssapi</b></p> +<p><b>Author: </b>varmojfekoj</p> +<p><b>Description</b>:</p> +<p>Provides the ability to receive syslog messages from the +network protected via Kerberos 5 encryption and authentication. This +module also accept plain tcp syslog messages on the same port if configured to do so. If you need just plain tcp, use <a href="imtcp.html">imtcp</a> instead.</p> +<p>There is also an <a href="gssapi.html">overview of gssapi support in rsyslog</a> available. We recommend reading +it before digging into the configuration parameters.</p> +<p><b>Configuration Directives</b>:</p> +<ul> +<li>InputGSSServerRun <port><br> +Starts a GSSAPI server on selected port - note that this runs +independently from the TCP server.</li> +<li>InputGSSServerServiceName <name><br> +The service name to use for the GSS server.</li> +<li>$InputGSSServerPermitPlainTCP on|<span style="font-weight: bold;">off</span><br> +Permits the server to receive plain tcp syslog (without GSS) on the +same port</li> +<li>$InputGSSServerMaxSessions <number><br> +Sets the maximum number of sessions supported</li> +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>module always binds to all interfaces</li> +<li>only a single listener can be bound</li> + +</ul> +<p><b>Sample:</b></p> +<p>This sets up a GSS server on port 1514 that also permits to +receive plain tcp syslog messages (on the same port):<br> +</p> +<textarea rows="15" cols="60">$ModLoad imtcp # needs to be done just once +$InputGSSServerRun 1514 +$InputGSSServerPermitPlainTCP on +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/imklog.html b/doc/imklog.html new file mode 100644 index 00000000..b5b21e84 --- /dev/null +++ b/doc/imklog.html @@ -0,0 +1,74 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>Kernel Log Input Module (imklog)</title> + +</head> +<body> +<h1>Kernel Log Input Module</h1> +<p><b>Module Name: imklog</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>Reads messages from the kernel log and submits them to the +syslog engine.</p> +<p><b>Configuration Directives</b>:</p> +<ul> +<li><strong>$KLogInternalMsgFacility +<facility></strong><br> +The facility which messages internally generated by imklog will have. +imklog generates some messages of itself (e.g. on problems, startup and +shutdown) and these do not stem from the kernel. Historically, under +Linux, these too have "kern" facility. Thus, on Linux platforms the +default is "kern" while on others it is "syslogd". You usually do not +need to specify this configuratin directive - it is included primarily +for few limited cases where it is needed for good reason. Bottom line: +if you don't have a good idea why you should use this setting, do not +touch it.</li> +<li><span style="font-weight: bold;">$KLogPermitNonKernelFacility +[on/<span style="font-style: italic;">off</span>]<br> +</span>At least under BSD the kernel log may contain entries +with non-kernel facilities. This setting controls how those are +handled. The default is "off", in which case these messages are +ignored. Switch it to on to submit non-kernel messages to rsyslog +processing.<span style="font-weight: bold;"></span></li> +<li><span style="font-weight: bold;"></span>$DebugPrintKernelSymbols +(imklog) [on/<b>off</b>]<br> +Linux only, ignored on other platforms (but may be specified)</li> +<li>$klogSymbolLookup (imklog) [on/<b>off</b>] -- +disables imklog kernel symbol translation (former klogd -x option). NOTE that +this option is counter-productive on recent kernels (>= 2.6) because the +kernel already does the symbol translation and this option breaks the information.<br> +<b>This option is scheduled for removal, probably with version 4.x.</b> Do not use +it except if you have a very good reason. If you have one, let us know +because otherwise new versions will no longer support it.<br> +Linux only, ignored on other platforms (but may be specified)</li> +<li>$klogUseSyscallInterface (imklog) [on/<b>off</b>] +-- former klogd -s option<br> +Linux only, ignored on other platforms (but may be specified)</li> +<li>$klogSymbolsTwice (imklog) [on/<b>off</b>] -- +former klogd -2 option<br> +Linux only, ignored on other platforms (but may be specified)<br style="font-weight: bold;"> +</li> +</ul> +<b>Caveats/Known Bugs:</b> +<p>This is obviously platform specific and requires platform +drivers. +Currently, imklog functionality is available on Linux and BSD.</p> +<p><b>Sample:</b></p> +<p>The following sample pulls messages from the kernel log. All +parameters are left by default, which is usually a good idea. Please +note that loading the plugin is sufficient to activate it. No directive +is needed to start pulling kernel messages.<br> +</p> +<textarea rows="15" cols="60">$ModLoad imklog +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/imrelp.html b/doc/imrelp.html new file mode 100644 index 00000000..bfdaad84 --- /dev/null +++ b/doc/imrelp.html @@ -0,0 +1,52 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>RELP Input Module</title> + +</head> +<body> +<h1>RELP Input Module</h1> +<p><b>Module Name: imrelp</b></p> +<p><b>Author: Rainer Gerhards</b></p> +<p><b>Description</b>:</p> +<p>Provides the ability to receive syslog messages via the +reliable RELP protocol. This module requires <a href="http://www.librelp.com">librelp</a> to be +present on the system. From the user's point of view, imrelp works much +like imtcp or imgssapi, except that no message loss can occur. Please +note that with the currently supported relp protocol version, a minor +message duplication may occur if a network connection between the relp +client and relp server breaks after the client could successfully send +some messages but the server could not acknowledge them. The window of +opportunity is very slim, but in theory this is possible. Future +versions of RELP will prevent this. Please also note that rsyslogd may +lose a few messages if rsyslog is shutdown while a network conneciton +to the server is broken and could not yet be recovered. Future version +of RELP support in rsyslog will prevent that. Please note that both +scenarios also exists with plain tcp syslog. RELP, even with the small +nits outlined above, is a much more reliable solution than plain tcp +syslog and so it is highly suggested to use RELP instead of plain tcp. +Clients send messages to the RELP server via omrelp.</p> +<p><b>Configuration Directives</b>:</p> +<ul> +<li>InputRELPServerRun <port><br> +Starts a RELP server on selected port</li> +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>see description</li> +</ul> +<p><b>Sample:</b></p> +<p>This sets up a RELP server on port 20514.<br> +</p> +<textarea rows="15" cols="60">$ModLoad imrelp # needs to be done just once +$InputRELPServerRun 20514 +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/imtcp.html b/doc/imtcp.html new file mode 100644 index 00000000..ecc72748 --- /dev/null +++ b/doc/imtcp.html @@ -0,0 +1,54 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>TCP Syslog Input Module</title></head> +<body> +<h1>TCP Syslog Input Module</h1> +<p><b>Module Name: imtcp</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>Provides the ability to receive syslog messages via TCP. +Encryption can be provided by using <a href="rsyslog_stunnel.html">stunnel</a> +(an alternative is the use +the <a href="imgssapi.html">imgssapi</a> +modul).</p> +<p>In the future, multiple receivers may be configured by +specifying +$InputTCPServerRun multiple times. This is not currently supported. +</p> +<p><b>Configuration Directives</b>:</p> +<ul> +<li>$InputTCPServerRun <port><br> +Starts a TCP server on selected port</li> +<li><ul><li>$InputTCPMaxSessions <number></li></ul> +Sets the maximum number of sessions supported</li><li>$InputTCPServerStreamDriverMode <number><br> +Sets the driver mode for the currently selected <a href="netstream.html">network stream driver</a>. <number> is driver specifc.</li><li>$InputTCPServerStreamDriverAuthMode <mode-string><br> +Sets the authentication mode for the currently selected <a href="netstream.html">network stream driver</a>. <mode-string> is driver specifc.</li><li>$InputTCPServerStreamDriverPermittedPeer <id-string><br> +Sets permitted peer IDs. Only these peers are able to connect to the +listener. <id-string> semantics depend on the currently selected +AuthMode and <a href="netstream.html">network stream driver</a>. PermittedPeers may not be set in anonymous modes.</li> +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>module always binds to all interfaces</li> +<li>only a single listener can be bound</li> +<li>can not be loaded together with <a href="imgssapi.html">imgssapi</a> +(which includes the functionality of imtcp)</li> +</ul> +<p><b>Sample:</b></p> +<p>This sets up a TCP server on port 514:<br> +</p> +<textarea rows="15" cols="60">$ModLoad imtcp # +needs to be done just once +$InputTCPServerRun 514 +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/imuxsock.html b/doc/imuxsock.html new file mode 100644 index 00000000..77491992 --- /dev/null +++ b/doc/imuxsock.html @@ -0,0 +1,81 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>Unix Socket Input</title> +</head> +<body> +<h1>Unix Socket Input</h1> +<p><b>Module Name: imuxsock</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p><b>Provides the ability to accept syslog messages via local Unix +sockets. Most importantly, this is the mechanism by which the syslog(3) +call delivers syslog messages to rsyslogd.</b> So you need to have this +module loaded to read the system log socket and be able to process log +messages from applications running on the local system.</p> +<p><b>Application-provided +timestamps are ignored by default.</b> This is needed, as some programs +(e.g. sshd) log with inconsistent timezone information, what +messes up the local logs (which by default don't even contain time zone +information). This seems to be consistent with what sysklogd did for +the past four years. Alternate behaviour may be desirable if +gateway-like processes send messages via the local log slot - in this +case, it can be enabled via the +$InputUnixListenSocketIgnoreMsgTimestamp and $SystemLogSocketIgnoreMsgTimestamp config directives</p> +<p><b>Unix log sockets can be flow-controlled.</b> That is, if processing queues fill up, +the unix socket reader is blocked for a short while. This may be useful to prevent overruning +the queues (which may cause exessive disk-io where it actually would not be needed). However, +flow-controlling a log socket (and especially the system log socket) can lead to a very +unresponsive system. As such, flow control is disabled by default. That means any log records +are places as quickly as possible into the processing queues. If you would like to have +flow control, you need to enable it via the $SystemLogSocketFlowControl and +$InputUnixListenSocketFlowControl config directives. Just make sure you thought about +the implications. Note that for many systems, turning on flow control does not hurt. +<p><b>Configuration Directives</b>:</p> +<ul> +<li><b>$InputUnixListenSocketIgnoreMsgTimestamp</b> [<b>on</b>/off] +<br>Ignore timestamps included in the message. Applies to the next socket being added.</li> +<li><b>$InputUnixListenSocketFlowControl</b> [on/<b>off</b>] - specifies if flow control should be applied +to the next socket.</li> +<li><b>$SystemLogSocketIgnoreMsgTimestamp</b> [<b>on</b>/off]<br> +Ignore timestamps included in the messages, applies to messages received via the system log socket.</li> +<li><b>$OmitLocalLogging</b> (imuxsock) [on/<b>off</b>] -- former -o option</li> +<li><b>$SystemLogSocketName</b> <name-of-socket> -- former -p option</li> +<li><b>$SystemLogFlowControl</b> [on/<b>off</b>] - specifies if flow control should be applied +to the system log socket.</li> +<li><b>$AddUnixListenSocket</b> <name-of-socket> adds additional unix socket, default none -- former -a option</li> +<li><b>$InputUnixListenSocketHostName</b> <hostname> permits to override the hostname that +shall be used inside messages taken from the <b>next</b> $AddUnixListenSocket socket. Note that +the hostname must be specified before the $AddUnixListenSocket configuration directive, and it +will only affect the next one and then automatically be reset. This functionality is provided so +that the local hostname can be overridden in cases where that is desired.</li> +</ul> +<b>Caveats/Known Bugs:</b><br> +<br> +This documentation is sparse and incomplete. +<p><b>Sample:</b></p> +<p>The following sample is the minimum setup required to accept syslog messages from applications running on the local system.<br> +</p> +<textarea rows="2" cols="70">$ModLoad imuxsock # needs to be done just once +$SystemLogSocketFlowControl on # enable flow control (use if needed) +</textarea> +<p>The following sample is a configuration where rsyslogd pulls logs from two +jails, and assigns different hostnames to each of the jails: </p> +<textarea rows="6" cols="60">$ModLoad imuxsock # needs to be done just once + +$InputUnixListenSocketHostName jail1.example.net +$AddUnixListenSocket /jail/1/dev/log +$InputUnixListenSocketHostName jail2.example.net +$AddUnixListenSocket /jail/2/dev/log +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/index.html b/doc/index.html new file mode 100644 index 00000000..b3b336a7 --- /dev/null +++ b/doc/index.html @@ -0,0 +1,31 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Welcome to rsyslog</title></head> +<body> +<h1>Welcome to rsyslog</h1> +<p><b><a href="http://www.rsyslog.com/">Rsyslog</a> +is an enhanced syslogd suitable both for small systems as +well as large enterprises.</b> +<p>This page provide a few quick pointers which hopefully make your +experience with rsyslog a pleasant one. These are +<ul> +<li><b>Most importantly, the <a href="manual.html">rsyslog manual</a></b> - this points to locally +installed documentation which exactly matches the version you have installed. +It is highly suggested to at least briefly look over these files. +<li>The <a href="http://www.rsyslog.com">rsyslog web site</a> which offers +probably every information you'll ever need (ok, just kidding...). +<li>The <a href="http://www.rsyslog.com/doc-status.html">project status page</a> provides +information on current releases +<li>and the <a href="troubleshoot.html">troubleshooting guide</a> hopefully helps if +things do not immediately work out +</ul> +<p>In general, rsyslog supports plain old syslog.conf format, except that the +config file is now called rsyslog.conf. This should help you get started +quickly. +To do the really cool things, though, +you need to learn a bit about its new features. +The man pages offer a bare minimum of information (and are still quite long). Read the +<a href="manual.html">html documentation</a> instead. +When you change the configuration, remember to restart rsyslogd, because otherwise +it will not use your new settings (and you'll end up totally puzzled why this great +config of yours does not even work a bit...;)) +</body></html> diff --git a/doc/install.html b/doc/install.html index bee136ce..48b7f649 100644 --- a/doc/install.html +++ b/doc/install.html @@ -1,5 +1,5 @@ <html><head> -<title>SSL Encrypting syslog with stunnel</title> +<title>A guide on HOWTO install rsyslog</title> <meta name="KEYWORDS" content="syslog encryption, rsyslog, stunnel, secure syslog, tcp, reliable, howto, ssl"> </head> <body> @@ -24,7 +24,11 @@ you volunteer to create one, <a href="mailto:rgerhards@adiscon.com">drop me a line</a>). Thus, this guide focuses on installing from the source, which thankfully is <b>quite easy</b>.</p> <h3>Step 1 - Download Software</h3> -<p>For obvious reasons, you need to download rsyslog. Load the most recent build +<p>For obvious reasons, you need to download rsyslog. Here, I assume that you +use a distribution tarball. If you would like to use a version directly from +the repository, see <a href="build_from_repo.html">build rsyslog from repository</a> +instead. +<p>Load the most recent build from <a href="http://www.rsyslog.com/downloads">http://www.rsyslog.com/downloads</a>. Extract the software with "tar xzf -nameOfDownloadSet-". This will create a new subdirectory rsyslog-version in the current working directory. CD into that. </p> @@ -53,7 +57,18 @@ the rsyslogd and the man pages to the relevant directories.</p> <p>In this step, you tell rsyslogd what to do with received messages. If you are upgrading from stock syslogd, /etc/syslog.conf is probably a good starting point. Rsyslogd understands stock syslogd syntax, so you can simply copy over -/etc/syslog.conf to /etc/rsyslog.conf. Then, edit rsyslog.conf for any +/etc/syslog.conf to /etc/rsyslog.conf. Note since version 3 rsyslog requires +to load plug-in modules to perform useful work (more about +<a href="v3compatibility.html">compatibilty notes v3</a>). To load the most common plug-ins, +add the following to the top of rsyslog.conf:</p> +<p> +$ModLoad immark # provides --MARK-- message capability <br /> +$ModLoad imudp # provides UDP syslog reception <br /> +$ModLoad imtcp # provides TCP syslog reception and GSS-API (if compiled to support it) <br /> +$ModLoad imuxsock # provides support for local system logging (e.g. via logger command) <br /> +$ModLoad imklog # provides kernel logging support (previously done by rklogd) <br /> +</p> +Change rsyslog.conf for any further enhancements you would like to see. For example, you can add database writing as outlined in the paper "<a href="rsyslog_mysql.html">Writing syslog Data to MySQL</a>" (remember you need to enable MySQL support during step 2 if you want to do @@ -139,9 +154,12 @@ comments or bug sighting reports are very welcome. Please <li>2007-07-13 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> * updated to new autotools-based build system</li> + <li>2008-10-01 * + <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> + * added info on building from source repository</li> </ul> <h3>Copyright</h3> -<p>Copyright (c) 2005, 2007 +<p>Copyright © 2005-2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> and <a href="http://www.adiscon.com/en/">Adiscon</a>.</p> <p> Permission is granted to copy, distribute and/or modify this document @@ -151,6 +169,12 @@ comments or bug sighting reports are very welcome. Please Texts. A copy of the license can be viewed at <a href="http://www.gnu.org/copyleft/fdl.html"> http://www.gnu.org/copyleft/fdl.html</a>.</p> - +<p>[<a href="manual.html">manual index</a>] +[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 1.2 or higher.</font></p> </body> </html> diff --git a/doc/licensing.html b/doc/licensing.html new file mode 100644 index 00000000..93a50930 --- /dev/null +++ b/doc/licensing.html @@ -0,0 +1,72 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>rsyslog licensing</title> + +</head> +<body> +<h1>rsyslog licensing</h1> +<p><b>Most important things first: if you intend to use rsyslog inside a GPLv3 compatible project, you are free to do so.</b> You don't even need to continue reading. +If you intend to use rsyslog inside a non-GPLv3 +compatible project, rsyslog offers you some liberties to do that, too. However, you then need +to study the licensing details in depth. +<p>The project hopes this is a good compromise, which also gives a boost to fellow free +software developers who release under GPLv3. +<p>And now on to the dirty and boring license details, still on a executive summary level. For the +real details, check source files and the files COPYING and COPYING.LESSER inside the distribution. +<p>The rsyslog package contains several components: +<ul> +<li>the rsyslog core programs (like rsyslogd) +<li>plugins (like imklog, omrelp, ...) +<li>the rsyslog runtime library +</ul> +<p>Each of these components can be thought of as individual projects. In fact, some of the +plugins have different main authors than the rest of the rsyslog package. All of these +components are currently put together into a single "rsyslog" package (tarball) for +convinience: this makes it easier to distribute a consistent version where everything +is included (and in the right versions) to build a full system. Platform package +maintainers in general take the overall package and split off the individual components, so that +users can install only what they need. In source installations, this can be done via the +proper ./configure switches. +<p>However, while it is convenient to package all parts in a single tarball, it does not +imply all of them are necessarily covered by the same license. Traditionally, GPL licenses +are used for rsyslog, because the project would like to provide free software. GPLv3 has been +used since around 2008 to help fight for our freedom. All rsyslog core programs are +released under GPLv3. But, from the beginning on, plugins were separate projects and we did not +impose and license restrictions on them. So even though all plugins that currently ship with +the rsyslog package are also placed under GPLv3, this can not taken for granted. You need +to check each plugins license terms if in question - this is especially important for +plugins that do NOT ship as part of the rsyslog tarball. +<p>In order to make rsyslog technology available to a broader range of applications, +the rsyslog runtime is, at least partly, licensed under LGPL. If in doubt, check the source file +licensing comments. As of now, the following files are licensed under LGPL: +<ul> +<li>queue.c/.h +<li>wti.c/.h +<li>wtp.c/.h +<li>vm.c/.h +<li>vmop.c/.h +<li>vmprg.c/.h +<li>vmstk.c/.h +<li>expr.c/.h +<li>sysvar.c/.h +<li>ctok.c/.h +<li>ctok_token.c/.h +<li>regexp.c/.h +<li>sync.c/.h +<li>stream.c/.h +<li>var.c/.h +</ul> +This list will change as time of the runtime modularization. At some point in the future, there will +be a well-designed set of files inside a runtime library branch and all of these will be LGPL. Some +select extras will probably still be covered by GPL. We are following a similar licensing +model in GnuTLS, which makes effort to reserve some functionality exclusively to open source +projects. +<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Last Update: 2008-04-15. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/log_rotation_fix_size.html b/doc/log_rotation_fix_size.html new file mode 100644 index 00000000..0b9a3b2e --- /dev/null +++ b/doc/log_rotation_fix_size.html @@ -0,0 +1,59 @@ +<html><head> +<title>Keep the log file size accurate with log rotation</title> +<meta name="KEYWORDS" content="log rotation, howto, guide, fixed-size log"> +</head> +<body> +<h1>Log rotation with rsyslog</h1> + <P><small><i>Written by + Michael Meckelein</i></small></P> +<h2>Situation</h2> + +<p>Your environment does not allow you to store tons of logs? +You have limited disc space available for logging, for example +you want to log to a 124 MB RAM usb stick? Or you do not want to +keep all the logs for months, logs from the last days is sufficient? +Think about log rotation.</p> + +<h2>Log rotation based on a fixed log size</h2> + +<p>This small but hopefully useful article will show you the way +to keep your logs at a given size. The following sample is based on +rsyslog illustrating a simple but effective log rotation with a +maximum size condition.</p> + +<h2>Use Output Channels for fixed-length syslog files</h2> + +<p>Lets assume you do not want to spend more than 100 MB hard +disc space for you logs. With rsyslog you can configure Output +Channels to achieve this. Putting the following directive</p> + +<p><pre> +# start log rotation via outchannel +# outchannel definiation +$outchannel log_rotation,/var/log/log_rotation.log, 52428800,/home/me/./log_rotation_script +# activate the channel and log everything to it +*.* $log_rotation +# end log rotation via outchannel +</pre></p> + +<p>to ryslog.conf instruct rsyslog to log everything to the destination file +'/var/log/log_rotation.log' until the give file size of 50 MB is reached. If +the max file size is reached it will perform an action. In our case it executes +the script /home/me/log_rotation_script which contains a single command:</p> + +<p><pre> +mv -f /var/log/log_rotation.log /var/log/log_rotation.log.1 +</p></pre> + +<p>This moves the original log to a kind of backup log file. +After the action was successfully performed rsyslog creates a new /var/log/log_rotation.log +file and fill it up with new logs. So the latest logs are always in log_roatation.log.</p> + +<h2>Conclusion</h2> + +<p>With this approach two files for logging are used, each with a maximum size of 50 MB. So +we can say we have successfully configured a log rotation which satisfies our requirement. +We keep the logs at a fixed-size level of100 MB.</p> + +</body> +</html> diff --git a/doc/manual.html b/doc/manual.html index 245d1ae8..22fd63b8 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -1,121 +1,101 @@ -<html> - -<head> - -<title>rsyslog documentation</title> - -</head> - +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>rsyslog documentation</title></head> <body> - <h1>RSyslog - Documentation</h1> - -<p><b><a href="http://www.rsyslog.com/">Rsyslog</a> is an enhanced syslogd - -supporting, among others, <a href="rsyslog_mysql.html">MySQL</a>, -PostgreSQL, -<a href="http://wiki.rsyslog.com/index.php/FailoverSyslogServer">failover log -destinations</a>, syslog/tcp, - -fine grain output format control, and the ability to filter on any message part.</b> - -It is quite compatible to stock - -sysklogd and can be used as a drop-in replacement. Its <a href="features.html"> - -advanced features</a> make it suitable for enterprise-class, - -<a href="rsyslog_stunnel.html">encryption protected syslog</a> - -relay chains while at the same time being very easy to setup - -for the novice user. And as we know what enterprise users really need, there is also <a href="professional_support.html">professional rsyslog support</a> available directly from the source!</p> - -<p><b>This documentation is for version 2.0.7 of rsyslog.</b> +<p><b><a href="http://www.rsyslog.com/">Rsyslog</a> +is an enhanced syslogd +supporting, among others, <a href="rsyslog_mysql.html">MySQL</a>, +PostgreSQL, <a href="http://wiki.rsyslog.com/index.php/FailoverSyslogServer">failover +log destinations</a>, syslog/tcp, fine grain output format +control, high precision timestamps, queued operations and the ability to filter on any message +part.</b> +It is quite compatible to stock sysklogd and can be used as a drop-in +replacement. Its <a href="features.html"> +advanced features</a> make it suitable for enterprise-class, <a href="rsyslog_tls.html">encryption protected syslog</a> +relay chains while at the same time being very easy to setup for the +novice user. And as we know what enterprise users really need, there is +also <a href="professional_support.html">professional +rsyslog support</a> available directly from the source!</p> +<p><b>This documentation is for version 3.22.1 (v3-stable branch) of rsyslog.</b> Visit the <i> <a href="http://www.rsyslog.com/doc-status.html">rsyslog status page</a></i></b> to obtain current -version information and project status.</p> - -version information and ports. -<p><b>If you like rsyslog, you might want to lend us - -a helping hand. </b>It doesn't require a lot of time - even a single mouse click - -helps. Learn <a href="how2help.html">how to help the rsyslog project</a>.</p> - -<p><b>Follow the links below for the</b></p> - -<ul> - -<li><a href="man_rsyslogd.html">rsyslogd man page</a> - -<li><a href="rsyslog_conf.html">configuration file syntax (rsyslog.conf)</a><li> -<a href="property_replacer.html">property replacer, an important core component</a><li>a commented <a href="sample.conf.html">sample rsyslog.conf</a> - -<li><a href="bugs.html">rsyslog bug list</a><li><a href="rsyslog_packages.html"> -rsyslog packages</a><li><a href="generic_design.html">backgrounder on generic -syslog application design</a><!-- re-enable when updated <li><a href="contributors.html">contributor "Hall -of Fame"</a> --><li><a href="modules.html">description of rsyslog modules</a></ul> - +version information and project status. +</p><p><b>If you like rsyslog, you might +want to lend us a helping hand. </b>It doesn't require a lot of +time - even a single mouse click helps. Learn <a href="how2help.html">how to help the rsyslog project</a>. +Due to popular demand, there is now a <a href="rsyslog_ng_comparison.html">side-by-side comparison +between rsyslog and syslog-ng</a>.</p> +<p>If you are upgrading from rsyslog v2 or stock sysklogd, +<a href="v3compatibility.html">be +sure to read the rsyslog v3 compatibility document!</a> It will work even +if you do not read the doc, but doing so will definitely improve your experience.</p> +<p><span style="font-weight: bold;"></span><b>Follow +the links below for the</b><br></p><ul> + +<li><a href="troubleshoot.html">troubleshooting rsyslog problems</a></li> +<li><a href="rsyslog_conf.html">configuration file syntax (rsyslog.conf)</a></li> +<li> <a href="property_replacer.html">property replacer, an important core component</a></li> +<li><a href="http://www.rsyslog.com/tool-regex">a regular expression checker/generator tool for rsyslog</a></li> +<li>a commented <a href="sample.conf.html">sample rsyslog.conf</a> </li> +<li><a href="bugs.html">rsyslog bug list</a></li> +<li><a href="rsyslog_packages.html"> rsyslog packages</a></li> +<li><a href="generic_design.html">backgrounder on +generic syslog application design</a><!-- not good as it currently is ;) <li><a href="contributors.html">contributor "Hall of Fame"</a>--></li> +<li><a href="modules.html">description of rsyslog modules</a></li> +</ul> <p><b>We have some in-depth papers on</b></p> - <ul> - - <li><a href="install.html">installing rsyslog</a></li> - <li><a href="ipv6.html">rsyslog and IPv6</a> (which is fully supported)</li> - - <li><a href="rsyslog_stunnel.html">ssl-encrypting syslog with stunnel</a></li> - - <li><a href="rsyslog_mysql.html">writing syslog messages to MySQL</a></li> - - <li><a href="rsyslog_php_syslog_ng.html">using php-syslog-ng with rsyslog</a></li> - <li><a href="rsyslog_recording_pri.html">recording the syslog priority - (severity and facility) to the log file</a></li> - <li><a href="http://www.rsyslog.com/Article19.phtml">preserving syslog - sender over NAT</a> (online only)</li> - +<li><a href="install.html">installing rsyslog</a></li> +<li><a href="build_from_repo.html">obtaining rsyslog from the source repository</a></li> +<li><a href="ipv6.html">rsyslog and IPv6</a> (which is fully supported)</li> +<li><a href="rsyslog_secure_tls.html">native TLS encryption for syslog</a></li> +<li><a href="rsyslog_stunnel.html">ssl-encrypting syslog with stunnel</a></li> +<li><a href="rsyslog_mysql.html">writing syslog messages to MySQL (and other databases as well)</a></li> +<li><a href="rsyslog_high_database_rate.html">writing massive amounts of syslog messages to a database</a></li> +<li><a href="rsyslog_reliable_forwarding.html">reliable forwarding to a remote server</a></li> +<li><a href="rsyslog_php_syslog_ng.html">using +php-syslog-ng with rsyslog</a></li> +<li><a href="rsyslog_recording_pri.html">recording +the syslog priority (severity and facility) to the log file</a></li> +<li><a href="http://www.rsyslog.com/Article19.phtml">preserving +syslog sender over NAT</a> (online only)</li> +<li><a href="gssapi.html">an overview and howto of rsyslog gssapi support</a></li> +<li><a href="debug.html">debug support in rsyslog</a></li> +<li><a href="dev_queue.html">the rsyslog message queue object (developer's view)</a></li> </ul> - -<p>Our <a href="history.html">rsyslog history</a> page is for you if you would like to learn a little more - -on why there is an rsyslog at all. If you are interested why you should care -about rsyslog at all, you may want to read Rainer's essay on "<a href="http://rgerhards.blogspot.com/2007/08/why-does-world-need-another-syslogd.html">why -the world needs another syslogd</a>".</p> - -<p>Documentation is added continuously. Please note that the documentation here - -matches only the current version of rsyslog. If you use an older version, be sure - -to use the doc that came with it.</p> - +<p>Our <a href="history.html">rsyslog history</a> +page is for you if you would like to learn a little more +on why there is an rsyslog at all. If you are interested why you should +care about rsyslog at all, you may want to read Rainer's essay on "<a href="http://rgerhards.blogspot.com/2007/08/why-does-world-need-another-syslogd.html">why +the world needs another syslogd</a>".</p> +<p>Documentation is added continuously. Please note that the +documentation here +matches only the current version of rsyslog. If you use an older +version, be sure to use the doc that came with it.</p> <p><b>You can also browse the following online resources:</b></p> - <ul> +<li>the <a href="http://wiki.rsyslog.com/">rsyslog +wiki</a>, a community resource which includes <a href="http://wiki.rsyslog.com/index.php/Configuration_Samples">rsyslog configuration examples</a></li> +<li><a href="http://www.rsyslog.com/module-Static_Docs-view-f-manual.html.phtml">rsyslog +online documentation (most current version only)</a></li> -<li>the <a href="http://wiki.rsyslog.com/">rsyslog wiki</a>, a community -resource</li> - -<li><a href="http://www.rsyslog.com/module-Static_Docs-view-f-manual.html.phtml">rsyslog online documentation</a></li> - -<li><a href="http://www.rsyslog.com/Topic3.phtml">rsyslog FAQ</a></li> - -<li><a href="http://www.rsyslog.com/PNphpBB2.phtml">rsyslog discussion forum</a></li> - +<li><a href="http://kb.monitorware.com/rsyslog-f40.html">rsyslog discussion forum - use this for technical support</a></li> +<li><a href="http://www.rsyslog.com/Topic8.phtml">rsyslog video tutorials</a></li> <li><a href="http://www.rsyslog.com/Topic4.phtml">rsyslog change log</a></li> - +<li><a href="http://www.rsyslog.com/Topic3.phtml">rsyslog FAQ</a></li> <li><a href="http://www.monitorware.com/en/syslog-enabled-products/">syslog device configuration guide</a> (off-site)</li> - +<li><a href="http://www.rsyslog.com/PNphpBB2.phtml">rsyslog discussion forum - use this for technical support</a></li> +<li><a href="http://kb.monitorware.com/rsyslog-f49.html">deutsches rsyslog forum</a> (forum in German language)</li> </ul> - -<p>And don't forget about the <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">rsyslog mailing list</a>. - -If you are interested in the "backstage", you may find - +<p>And don't forget about the <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">rsyslog +mailing list</a>. If you are interested in the "backstage", you +may find <a href="http://www.gerhards.net/rainer">Rainer</a>'s - -<a href="http://rgerhards.blogspot.com/">blog</a> an interesting read (filter on -syslog and rsyslog tags).</p> - -</body> - -</html> - +<a href="http://rgerhards.blogspot.com/">blog</a> an +interesting read (filter on syslog and rsyslog tags). +If you would like to use rsyslog source code inside your open source project, you can do that without +any restriction as long as your license is GPLv3 compatible. If your license is incompatible to GPLv3, +you may even be still permitted to use rsyslog source code. However, then you need to look at the way +<a href="licensing.html">rsyslog is licensed</a>.</p> +<p>Feedback is always welcome, but if you have a support question, please do not +mail Rainer directly (<a href="free_support.html">why not?</a>). +</body></html> diff --git a/doc/netstream.html b/doc/netstream.html new file mode 100644 index 00000000..e7d54c12 --- /dev/null +++ b/doc/netstream.html @@ -0,0 +1,21 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Network Stream Drivers</title> + +</head> +<body> +<h1>Network Stream Drivers</h1><p>Network stream drivers are a layer +between various parts of rsyslogd (e.g. the imtcp module) and the +transport layer. They provide sequenced delivery, authentication and +confidentiality to the upper layers. Drivers implement different +capabilities.</p><p> Users need to know about netstream drivers because +they need to configure the proper driver, and proper driver properties, +to achieve desired results (e.g. a <a href="rsyslog_tls.html">TLS-protected syslog transmission</a>).</p><p>The following drivers exist:</p><ul><li><a href="ns_ptcp.html">ptcp</a> - the plain tcp network transport (no security)</li><li><a href="ns_gtls.html">gtls</a> - a secure TLS transport implemented via the GnuTLS library</li></ul>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>] +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html>
\ No newline at end of file diff --git a/doc/ns_gtls.html b/doc/ns_gtls.html new file mode 100644 index 00000000..0d02ad02 --- /dev/null +++ b/doc/ns_gtls.html @@ -0,0 +1,59 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>gtls Network Stream Driver</title> + +</head> +<body> +<h1>gtls Network Stream Driver</h1> +<p>This <a href="netstream.html">network stream +driver</a> implements a TLS protected transport via the <a href="http://www.gnu.org/software/gnutls/" target="_blank">GnuTLS +library</a>.</p> +<p><b>Available since:</b> 3.19.0 (suggested minimum 3.19.8 and above)</p> +<p style="font-weight: bold;">Supported Driver Modes</p> +<ul> +<li>0 - unencrypted trasmission (just like <a href="ns_ptcp.html">ptcp</a> driver)</li> +<li>1 - TLS-protected operation</li> +</ul> +Note: mode 0 does not provide any benefit over the ptcp driver. This +mode exists for technical reasons, but should not be used. It may be +removed in the future.<br> +<span style="font-weight: bold;">Supported Authentication +Modes</span><br> +<ul> +<li><span style="font-weight: bold;">anon</span> +- anonymous authentication as +described in IETF's draft-ietf-syslog-transport-tls-12 Internet draft</li> +<li><span style="font-weight: bold;">x509/fingerprint</span> +- certificate fingerprint authentication as +described in IETF's draft-ietf-syslog-transport-tls-12 Internet draft</li> +<li><span style="font-weight: bold;">x509/certvalid</span> +- certificate validation only</li> +<li><span style="font-weight: bold;">x509/name</span> +- certificate validation and subject name authentication as +described in IETF's draft-ietf-syslog-transport-tls-12 Internet draft +</li> +</ul> +Note: "anon" does not permit to authenticate the remote peer. As such, +this mode is vulnerable to man in the middle attacks as well as +unauthorized access. It is recommended NOT to use this mode.</p> +<p>x509/certvalid is a nonstandard mode. It validates the remote +peers certificate, but does not check the subject name. This is +weak authentication that may be useful in scenarios where multiple +devices are deployed and it is sufficient proof of authenticy when +their certificates are signed by the CA the server trusts. This is +better than anon authentication, but still not recommended. +<b>Known Problems</b><br> +<p>Even in x509/fingerprint mode, both the client and sever +certificate currently must be signed by the same root CA. This is an +artifact of the underlying GnuTLS library and the way we use it. It is +expected that we can resolve this issue in the future.</p> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>] +</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/ns_ptcp.html b/doc/ns_ptcp.html new file mode 100644 index 00000000..c028d6c0 --- /dev/null +++ b/doc/ns_ptcp.html @@ -0,0 +1,16 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>ptcp Network Stream Driver</title> + +</head> +<body> +<h1>ptcp Network Stream Driver</h1> +<p>This <a href="netstream.html">network stream driver</a> implement a plain tcp transport without security properties.</p><p>Supported Driver Modes</p><ul><li>0 - unencrypted trasmission</li></ul>Supported Authentication Modes<br><ul><li>"anon" - no authentication</li></ul>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>] +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html>
\ No newline at end of file diff --git a/doc/omlibdbi.html b/doc/omlibdbi.html new file mode 100644 index 00000000..8ff74371 --- /dev/null +++ b/doc/omlibdbi.html @@ -0,0 +1,126 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>Generic Database Output Module (omlibdbi)</title> + +</head> +<body> +<h1>Generic Database Output Module (omlibdbi)</h1> +<p><b>Module Name: omlibdbi</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>This modules supports a large number of database systems via <a href="http://libdbi.sourceforge.net/">libdbi</a>. +Libdbi abstracts the database layer and provides drivers for many +systems. Drivers are available via the <a href="http://libdbi-drivers.sourceforge.net/">libdbi-drivers</a> +project. As of this writing, the following drivers are available:</p> +<ul> +<li><a href="http://www.firebird.sourceforge.net/">Firebird/Interbase</a></li> +<li><a href="http://www.freetds.org/">FreeTDS</a> +(provides access to <a href="http://www.microsoft.com/sql">MS +SQL Server</a> and <a href="http://www.sybase.com/products/informationmanagement/adaptiveserverenterprise">Sybase</a>)</li> +<li><a href="http://www.mysql.com/">MySQL</a> +(also +supported via the native ommysql plugin in rsyslog)</li> +<li><a href="http://www.postgresql.org/">PostgreSQL</a>(also +supported via the native +ommysql plugin in rsyslog)</li> +<li><a href="http://www.sqlite.org/">SQLite/SQLite3</a></li> +</ul> +<p>The following drivers are in various stages of completion:</p> +<ul> +<li><a href="http://ingres.com/">Ingres</a></li> +<li><a href="http://www.hughes.com.au/">mSQL</a></li> +<li><a href="http://www.oracle.com/">Oracle</a></li> +</ul> +<p>These drivers seem to be quite usable, at +least from an rsyslog point of view.</p> +<p>Libdbi provides a slim layer between rsyslog and the actual +database engine. We have not yet done any performance testing (e.g. +omlibdbi vs. ommysql) but honestly believe that the performance impact +should be irrelevant, if at all measurable. Part of that assumption is +that rsyslog just does the "insert" and most of the time is spent +either in the database engine or rsyslog itself. It's hard to think of +any considerable time spent in the libdbi abstraction layer.</p> +<p><span style="font-weight: bold;">Setup</span></p> +<p>In order for this plugin to work, you need to have libdbi, the +libdbi driver for your database backend and the client software for +your database backend installed. There are libdbi packages for many +distributions. Please note that rsyslogd requires a quite recent +version (0.8.3) of libdbi. It may work with older versions, but these +need some special ./configure options to support being called from a +dlopen()ed plugin (as omlibdbi is). So in short, you probably save you +a lot of headache if you make sure you have at least libdbi version +0.8.3 on your system. +</p> +<p><b>Configuration Directives</b>:</p> +<ul> +<li><span style="font-weight: bold;">$ActionLibdbiDriverDirectory /path/to/dbd/drivers</span><br>This +is a global setting. It points libdbi to its driver directory. Usually, +you do not need to set it. If you installed libdbi-driver's at a +non-standard location, you may need to specify the directory here. If +you are unsure, do <span style="font-weight: bold;">not</span> use this configuration directive. Usually, everything works just fine.<strong></strong></li><li><strong>$ActionLibdbiDriver drivername</strong><br> +Name of the dbidriver to use, see libdbi-drivers documentation. As a +quick excerpt, at least those were available at the time of this +writiting "mysql" (suggest to use ommysql instead), "firebird" (Firbird +and InterBase), "ingres", "msql", "Oracle", "sqlite", "sqlite3", +"freetds" (for Microsoft SQL and Sybase) and "pgsql" (suggest to use +ompgsql instead).</li> +<li><span style="font-weight: bold;">$ActionLibdbiHost +hostname</span><br> +The host to connect to.</li> +<li><span style="font-weight: bold;">$ActionLibdbiUserName +user</span><br> +The user used to connect to the database.</li> +<li><span style="font-weight: bold;">$ActionlibdbiPassword</span><br> +That user's password.</li> +<li><span style="font-weight: bold;">$ActionlibdbiDBName +db</span><br> +The database that shall be written to.</li> +<li><span style="font-weight: bold;">selector +line: :omlibdbi:<span style="font-style: italic;">;template</span></span><br> +executes the recently configured omlibdbi action. The ;template part is +optional. If no template is provided, a default template is used (which +is currently optimized for MySQL - sorry, folks...)</li> +</ul> +<b>Caveats/Known Bugs:</b> +<p>You must make sure that any templates used for omlibdbi +properly escape strings. This is usually done by supplying the SQL (or +STDSQL) option to the template. Omlibdbi rejects templates without this +option for security reasons. However, omlibdbi does not detect if you +used the right option for your backend. Future versions of rsyslog +(with full expression support) will provide advanced +ways of handling this situation. So far, you must be careful. The +default template provided by rsyslog is suitable for MySQL, but not +necessarily for your database backend. Be careful!</p> +<p>If you receive the rsyslog error message "libdbi or libdbi +drivers not present on this system" you may either not have libdbi and +its drivers installed or (very probably) the version is earlier than +0.8.3. In this case, you need to make sure you have at least 0.8.3 and +the libdbi driver for your database backend present on your system.</p><p>I +do not have most of the database supported by omlibdbi in my lab. So it +received limited cross-platform tests. If you run into troubles, be +sure the let us know at <a href="http://www.rsyslog.com">http://www.rsyslog.com</a>.</p> +<p><b>Sample:</b></p> +<p>The following sample writes all syslog messages to the +database "syslog_db" on mysqlsever.example.com. The server is MySQL and +being accessed under the account of "user" with password "pwd" (if you +have empty passwords, just remove the $ActionLibdbiPassword line).<br> +</p> +<textarea rows="15" cols="60">$ModLoad omlibdbi +$ActionLibdbiDriver mysql +$ActionLibdbiHost mysqlserver.example.com +$ActionLibdbiUserName user +$ActionLibdbiPassword pwd +$ActionLibdbiDBName syslog_db +*.* :omlibdbi: +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/ommail.html b/doc/ommail.html new file mode 100644 index 00000000..c18cf3f8 --- /dev/null +++ b/doc/ommail.html @@ -0,0 +1,145 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>mail output module - sending syslog messages via mail</title> + +</head> +<body> +<h1>Mail Output Module (ommail)</h1> +<p><b>Module Name: ommail</b></p> +<p><b>Available since: </b> 3.17.0</p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>This module supports sending syslog messages via mail. Each +syslog message is sent via its own mail. Obviously, you will want to +apply rigorous filtering, otherwise your mailbox (and mail server) will +be heavily spammed. The ommail plugin is primarily meant for alerting +users. As such, it is assume that mails will only be sent in an +extremely limited number of cases.</p> +<p>Please note that ommail is especially well-suited to work in +tandem with <a href="imfile.html">imfile</a> to +watch files for the occurence of specific things to be alerted on. So +its scope is far broader than forwarding syslog messages to mail +recipients.</p> +Ommail uses two templates, one for the mail body and one for the +subject line. If neither is provided, a quite meaningless subject line +is used and the mail body will be a syslog message just as if it were +written to a file. It is expected that the users customizes both +messages. In an effort to support cell phones (including SMS gateways), +there is an option to turn off the body part at all. This is considered +to be useful to send a short alert to a pager-like device.<br> +<br> +It is highly recommended to use the "<span style="font-weight: bold;">$ActionExecOnlyOnceEveryInterval +<seconds></span>" directive to limit the amount of +mails that potentially be generated. With it, mails are sent at most in +a <seconds> interval. This may be your life safer. And +remember that an hour has 3,600 seconds, so if you would like to +receive mails at most once every two hours, include a +"$ActionExecOnlyOnceEveryInterval 7200" immediately before the ommail +action. Messages sent more frequently are simpy discarded.<span style="font-weight: bold;"></span> +<p><b>Configuration Directives</b>:</p> +<ul> +<li><span style="font-weight: bold;">$ActionMailSMTPServer</span><br> +Name or IP address of the SMTP server to be used. Must currently be +set. The default is 127.0.0.1, the SMTP server on the local machine. +Obviously it is not good to expect one to be present on each machine, +so this value should be specified.<br> +</li> +<li><span style="font-weight: bold;">$ActionMailSMTPPort</span><br> +Port number or name of the SMTP port to be used. The default is 25, the +standard SMTP port.</li> +<li><span style="font-weight: bold;">$ActionMailFrom</span><br> +The email address used as the senders address. There is no default.</li> +<li><span style="font-weight: bold;">$ActionMailTo</span><br> +The recipient email addresses. There is no default. To specify multiple +recpients, repeat this directive as often as needed. Note: <b>This directive +must be specified for each new action and is automatically reset.</b> +[Multiple recipients are supported for 3.21.2 and above.]</li> +<li><span style="font-weight: bold;">$ActionMailSubject</span><br> +The name of the <span style="font-weight: bold;">template</span> +to be used as the mail subject. If this is not specified, a more or +less meaningless mail subject is generated (we don't tell you the exact +text because that can change - if you want to have something specific, +configure it!).</li> +<li><span style="font-weight: bold;">$ActionMailEnableBody</span><br> +Setting this to "off" permits to exclude the actual message body. This +may be useful for pager-like devices or cell phone SMS messages. The +default is "on", which is appropriate for allmost all cases. Turn it +off only if you know exactly what you do!</li> +</ul> +<b>Caveats/Known Bugs:</b> +<p>The current ommail implementation supports <span style="font-weight: bold;">SMTP-direct mode</span> +only. In that mode, the plugin talks to the mail server via SMTP +protocol. No other process is involved. This mode offers best +reliability as it is not depending on any external entity except the +mail server. Mail server downtime is acceptable if the action is put +onto its own action queue, so that it may wait for the SMTP server to +come back online. However, the module implements only the bare SMTP +essentials. Most importantly, it does not provide any authentication +capabilities. So your mail server must be configured to accept incoming +mail from ommail without any authentication needs (this may be change +in the future as need arises, but you may also be referred to +sendmail-mode).</p> +<p>In theory, ommail should also offer a mode where it uses the +sendmail utility to send its mail (<span style="font-weight: bold;">sendmail-mode</span>). +This is somewhat less reliable (because we depend on an entity we do +not have close control over - sendmail). It also requires dramatically +more system ressources, as we need to load the external process (but +that should be no problem given the expected infrequent number of calls +into this plugin). The big advantage of sendmail mode is that it +supports all the bells and whistles of a full-blown SMTP implementation +and may even work for local delivery without a SMTP server being +present. Sendmail mode will be implemented as need arises. So if you +need it, please drop us a line (I nobody does, sendmail mode will +probably never be implemented).</p> +<p><b>Sample:</b></p> +<p>The following sample alerts the operator if the string "hard +disk fatal failure" is present inside a syslog message. The mail server +at mail.example.net is used and the subject shall be "disk problem on +<hostname>". Note how \r\n is included inside the body +text +to create line breaks. A message is sent at most once every 6 hours, +any other messages are silently discarded (or, to be precise, not being +forwarded - they are still being processed by the rest of the +configuration file).<br> +</p> +<textarea rows="15" cols="80">$ModLoad ommail +$ActionMailSMTPServer mail.example.net +$ActionMailFrom rsyslog@example.net +$ActionMailTo operator@example.net +$template mailSubject,"disk problem on %hostname%" +$template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'" +$ActionMailSubject mailSubject +# make sure we receive a mail only once in six +# hours (21,600 seconds ;)) +$ActionExecOnlyOnceEveryInterval 21600 +# the if ... then ... mailBody mus be on one line! +if $msg contains 'hard disk fatal failure' then :ommail:;mailBody +</textarea> +<p>The sample below is the same, but sends mail to two recipients:</p> +<textarea rows="15" cols="80">$ModLoad ommail +$ActionMailSMTPServer mail.example.net +$ActionMailFrom rsyslog@example.net +$ActionMailTo operator@example.net +$ActionMailTo admin@example.net +$template mailSubject,"disk problem on %hostname%" +$template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'" +$ActionMailSubject mailSubject +# make sure we receive a mail only once in six +# hours (21,600 seconds ;)) +$ActionExecOnlyOnceEveryInterval 21600 +# the if ... then ... mailBody mus be on one line! +if $msg contains 'hard disk fatal failure' then :ommail:;mailBody +</textarea> +<p>A more advanced example plus a discussion on using the email feature +inside a reliable system can be found in Rainer's blogpost +"<a style="font-style: italic;" href="http://rgerhards.blogspot.com/2008/04/why-is-native-email-capability.html">Why +is native email capability an advantage for a syslogd?</a>" +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/ommysql.html b/doc/ommysql.html new file mode 100644 index 00000000..e81ce532 --- /dev/null +++ b/doc/ommysql.html @@ -0,0 +1,57 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>MySQL Database Output Module</title> +</head> + +<body> +<h1>MySQL Database Output Module</h1> +<p><b>Module Name: ommysql</b></p> +<p><b>Author: </b>Michael Meckelein (Initial Author) / Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>This module provides native support for logging to MySQL databases. It offers +superior performance over the more generic <a href="omlibdbi.html">omlibdbi</a> module. +</p> +<p><b>Configuration Directives</b>:</p> +<p>ommysql mostly uses the "old style" configuration, with almost everything on the +action line itself. A few newer features are being migrated to the new style-config +directive configuration system. +<ul> +<li><b>$ActionOmmysqlServerPort <port></b><br>Permits to select +a non-standard port for the MySQL server. The default is 0, which means the +system default port is used. There is no need to specify this directive unless +you know the server is running on a non-standard listen port. +<li>Action parameters: +<br><b>:ommysql:database-server,database-name,database-userid,database-password</b> +<br>All parameters should be filled in for a successful connect. +</ul> +<p>Note rsyslog contains a canned default template to write to the MySQL +database. It works on the MonitorWare schema. This template is: +<p> +<textarea rows="5" cols="80">$template tpl,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL +</textarea> +<p>As you can see, the template is an actual SQL statement. Note the ",SQL" option: it tells the +template processor that the template is used for SQL processing, thus quote characters are quoted +to prevent security issues. You can not assign a template without ",SQL" to a MySQL output action. +<p>If you would like to change fields contents or add or delete your own fields, you +can simply do so by modifying the schema (if required) and creating your own custom +template. +<p><b>Sample:</b></p> +<p>The following sample writes all syslog messages to the +database "syslog_db" on mysqlsever.example.com. The server is +being accessed under the account of "user" with password "pwd". +</p> +<textarea rows="5" cols="80">$ModLoad ommysql +$ActionOmmysqlServerPort 1234 # use non-standard port +*.* :ommysql:mysqlserver.example.com,syslog_db,user,pwd +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/omrelp.html b/doc/omrelp.html new file mode 100644 index 00000000..d5437a70 --- /dev/null +++ b/doc/omrelp.html @@ -0,0 +1,54 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>RELP Output Module (omrelp)</title> + +</head> +<body> +<h1>RELP Output Module (omrelp)</h1> +<p><b>Module Name: omrelp</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>This module supports sending syslog messages over the reliable +RELP protocol. For RELP's advantages over plain tcp syslog, please see +the documentation for <a href="imrelp.html">imrelp</a> +(the server counterpart). </p> +<span style="font-weight: bold;">Setup</span> +<p>Please note the <a href="http://www.librelp.com">librelp</a> +is required for imrelp (it provides the core relp protocol +implementation).</p> +<p><b>Configuration Directives</b>:</p> +<p>This module uses old-style action configuration to keep +consistent with the forwarding rule. So far, no additional +configuration directives can be specified. To send a message via RELP, +use</p> +<p>*.* + :omrelp:<sever>:<port>;<template></p> +<p>just as you use </p> +<p>*.* + @@<sever>:<port>;<template></p> +<p>to forward a message via plain tcp syslog.</p> +<b>Caveats/Known Bugs:</b> +<p>See <a href="imrelp.html">imrelp</a>, +which documents them. </p> +<p><b>Sample:</b></p> +<p>The following sample sends all messages to the central server +"centralserv" at port 2514 (note that that server must run imrelp on +port 2514). Rsyslog's high-precision timestamp format is used, thus the +special "RSYSLOG_ForwardFormat" (case sensitive!) template is used.<br> +</p> +<textarea rows="15" cols="60">$ModLoad omrelp +# forward messages to the remote server "myserv" on +# port 2514 +*.* :omrelp:centralserv:2514;RSYSLOG_ForwardFormat +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/omsnmp.html b/doc/omsnmp.html new file mode 100644 index 00000000..31aaef24 --- /dev/null +++ b/doc/omsnmp.html @@ -0,0 +1,174 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>SNMP Output Module</title></head> + +<body> + +<h1>SNMP Output Module</h1> +<p><b>Module Name: omsnmp</b></p> +<p><b>Author: Andre Lorbach <alorbach@adiscon.com></b></p> +<p><b>Description</b>:</p> +<p>Provides the ability to send syslog messages as an SNMPv1 & v2c traps. By +default, SNMPv2c is preferred. The syslog message is wrapped into a OCTED +STRING variable. This module uses the <a target="_blank" href="http://net-snmp.sourceforge.net/"> +NET-SNMP</a> library. In order to compile this module, you will need to have the +<a target="_blank" href="http://net-snmp.sourceforge.net/">NET-SNMP</a> +developer (headers) package installed. </p> +<p> </p> +<p><b>Action Line:</b></p> +<p>%omsnmp% without any further parameters.</p> +<p> </p> +<p><b>Configuration Directives</b>:</p> +<ul> + <li><strong>$actionsnmptransport </strong>(This parameter is optional, the + default value is "udp")<br> + <br> + Defines the transport type you wish to use. Technically we can support all + transport types which are supported by NET-SNMP. <br> + To name a few possible values: <br> + <br> + udp, tcp, udp6, tcp6, icmp, icmp6 ...<br> + <br> + Example: <strong>$actionsnmptransport udp<br> + </strong></li> + <li><strong>$actionsnmptarget</strong><br> + <br> + This can be a hostname or ip address, and is our snmp target host. This + parameter is required, if the snmptarget is not defined, nothing will be + send. <br> + <br> + Example: <strong>$actionsnmptarget server.domain.xxx</strong><br> + </li> + <li><strong>$actionsnmptargetport </strong>(This parameter is optional, the + default value is "162")<br> + <br> + The port which will be used, common values are port 162 or 161. <br> + <br> + Example: <strong>$actionsnmptargetport 162</strong><br> + </li> + <li><strong>$actionsnmpversion </strong>(This parameter is optional, the + default value is "1")<br> + <br> + There can only be two choices for this parameter for now. <br> + 0 means SNMPv1 will be used.<br> + 1 means SNMPv2c will be used. <br> + Any other value will default to 1. <br> + <br> + Example: <strong>$actionsnmpversion 1</strong><br> + </li> + <li><strong>$actionsnmpcommunity </strong>(This parameter is optional, the + default value is "public")<br> + <br> + This sets the used SNMP Community.<br> + <br> + Example:<strong> $actionsnmpcommunity public<br> + </strong><br> + </li> + <li><strong>$actionsnmptrapoid </strong>(This parameter is + optional, the default value is "1.3.6.1.4.1.19406.1.2.1" which means + "ADISCON-MONITORWARE-MIB::syslogtrap")<br> + This configuration parameter is used for <strong>SNMPv2</strong> only.<br> + <br> + This is the OID which defines the trap-type, or notifcation-type rsyslog + uses to send the trap. <br> + In order to decode this OID, you will need to have the + ADISCON-MONITORWARE-MIB and ADISCON-MIB mibs installed on the receiver side. Downloads of these mib files + can be found here: <br> + <a href="http://www.adiscon.org/download/ADISCON-MIB.txt"> + http://www.adiscon.org/download/ADISCON-MIB.txt</a><br> + <a href="http://www.adiscon.org/download/ADISCON-MONITORWARE-MIB.txt"> + http://www.adiscon.org/download/ADISCON-MONITORWARE-MIB.txt</a><br> + <br> + Thanks to the net-snmp + mailinglist for the help and the recommendations ;).<br> + <br> + Example: <strong>$actionsnmptrapoid 1.3.6.1.4.1.19406.1.2.1<br> + </strong>If you have this MIBS installed, you can also configured with the + OID Name: <strong>$actionsnmptrapoid ADISCON-MONITORWARE-MIB::syslogtrap<br> + </strong> + </li> + <li><strong>$actionsnmpsyslogmessageoid </strong>(This parameter is + optional, the default value is "1.3.6.1.4.1.19406.1.1.2.1" which means + "ADISCON-MONITORWARE-MIB::syslogMsg")<br> + <br> + This OID will be used as a variable, type "OCTET STRING". This variable will + contain up to 255 characters of the original syslog message including syslog header. It is recommend to + use the default OID. <br> + In order to decode this OID, you will need to have the + ADISCON-MONITORWARE-MIB and ADISCON-MIB mibs installed on the receiver side. + To download these custom mibs, see the description of <strong>$actionsnmptrapoid. + </strong><br> + <br> + Example: <strong>$actionsnmpsyslogmessageoid 1.3.6.1.4.1.19406.1.1.2.1<br> + </strong>If you have this MIBS installed, you can also configured with the + OID Name: <strong>$actionsnmpsyslogmessageoid + ADISCON-MONITORWARE-MIB::syslogMsg<br> + </strong><br> + </li> + <li><strong>$actionsnmpenterpriseoid </strong>(This parameter is optional, + the default value is "1.3.6.1.4.1.3.1.1" which means "enterprises.cmu.1.1")<br> + <br> + Customize this value if needed. I recommend to use the default value unless + you require to use a different OID. <br> + This configuration parameter is used for <strong>SNMPv1</strong> only. It + has no effect if <strong>SNMPv2</strong> is used. <br> + <br> + Example: <strong>$actionsnmpenterpriseoid 1.3.6.1.4.1.3.1.1 <br> + </strong><br> + </li> + <li><strong>$actionsnmpspecifictype </strong>(This parameter is optional, + the default value is "0")<strong> </strong><br> + <br> + This is the specific trap number. This configuration parameter is used for + <strong>SNMPv1</strong> only. It has no effect if <strong>SNMPv2</strong> is + used. <br> + <br> + Example: <strong>$actionsnmpspecifictype 0<br> + </strong><br> + </li> + <li><strong>$actionsnmptraptype</strong> (This parameter is optional, the + default value is "6" which means SNMP_TRAP_ENTERPRISESPECIFIC) <br> + <br> + There are only 7 Possible trap types defined which can be used here. These + trap types are: <br> + 0 = SNMP_TRAP_COLDSTART<br> + 1 = SNMP_TRAP_WARMSTART<br> + 2 = SNMP_TRAP_LINKDOWN<br> + 3 = SNMP_TRAP_LINKUP<br> + 4 = SNMP_TRAP_AUTHFAIL<br> + 5 = SNMP_TRAP_EGPNEIGHBORLOSS<br> + 6 = SNMP_TRAP_ENTERPRISESPECIFIC<br> + <br> + Any other value will default to 6 automatically. This configuration + parameter is used for <strong>SNMPv1</strong> only. It has no effect if + <strong>SNMPv2</strong> is used. <br> + <br> + Example: <strong>$actionsnmptraptype 6</strong><br> + </li> +</ul> +<p> </p> +<p><b>Caveats/Known Bugs:</b></p><ul><li>In order to decode the custom OIDs, you + will need to have the adiscon mibs installed. </li></ul> +<p><b>Sample:</b></p> +<p>The following commands send every message as a snmp trap.</p> +<textarea rows="10" cols="60">$ModLoad omsnmp + +$actionsnmptransport udp +$actionsnmptarget localhost +$actionsnmptargetport 162 +$actionsnmpversion 1 +$actionsnmpcommunity public + +*.* :omsnmp: +</textarea> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> + +</body></html> diff --git a/doc/professional_support.html b/doc/professional_support.html index 7f5e8371..2cb6c1e1 100644 --- a/doc/professional_support.html +++ b/doc/professional_support.html @@ -4,11 +4,11 @@ </head> <body> -<h1>Professional Support for Rsyslog</h1> -<p>Professional Support is offered by <a href="http://www.adiscon.com">Adiscon</a>, the company -that sponsors rsyslog development. For details, please contact <a href="mailto:info%40adiscon.com">Adiscon Sales</a>.</p> -<p></p> -<h3><u>EMail Support Service</u></h3> +<h1>Professional Services for Rsyslog</h1> +<p>Professional services are being offered by <a href="http://www.adiscon.com">Adiscon</a>, the company +that sponsors rsyslog development. For details, please contact <a href="mailto:info%40adiscon.com">Adiscon Sales</a>. </p> + +<h3>EMail Support Service</h3> Price: 99.00 EURO <br> Duration: 180 days <br> @@ -19,14 +19,19 @@ need to provide proof of software support in your organization. This contract provides</p> <ul> <li>unlimited email support tickets during validity -</li><li><span style="font-weight: bold;">fixes for</span> +</li> +<li><span style="font-weight: bold;">fixes for</span> current and <span style="font-weight: bold;">past rsyslog releases</span> -</li><li>advise on how to implement rsyslog in the best possible way. -</li></ul> +</li> +<li>advise on how to implement rsyslog in the best possible +way. +</li> +</ul> <p>Under this contract, fixes for old rsyslog releases will be provided / created, provided that it is possible to do that with the -code base in question. Phone support is not included.</p><h3><u>Custom-Written Config File</u></h3> +code base in question. Phone support is not included.</p> +<h3>Custom-Written Config File</h3> Price: 29.00 EURO <br> Duration: N/A @@ -43,9 +48,35 @@ faster). For security reasons, we will not put passwords into the configuration file, but will place easy to read comments in the places where you need to put them in. The agreement is governed under German law. You may also purchase this service if you would like to have your -own configuration file reviewed, e.g. for auditing purposes.</p><br><p>All agreements are +own configuration file reviewed, e.g. for auditing purposes.</p> +<h3>Custom Development</h3> +<p>Do you need an exotic feature that otherwise would not be implemented? +Do you need something really quick, quicker than it is available via +the regular development schedule? Then, you may consider funding +development for a specific functionality. We are always looking for +interesting projects. If you hire us to to do the job, you can be sure +to get the best possible and probably quickest solution, because we are +obviously at the heart of the source code. No need to get aquainted to +anything, no risk of misunderstanding program concepts. Benefit from +our vast syslog experience.</p> +<p>Please note that custom development is not limited to rsyslog. We offer +a number of logging solutions and can also work as part of your time +for specific requirements. The opportunities are endless, just ask. We +will work with you on your requirements and provide a quote on the +estimated cost. Just write to <a href="mailto:sales@adiscon.com">sales@adiscon.com</a> for details.</p><h3>Consulting Services</h3> +<p>Do you have demanding logging requirements? Why not talk to a +real logging professional? Instead of trying to find the solution +like a needle in the haystack, talk to the team that brought rsyslog, +phpLogCon, the Windows MonitorWare products and other logging +solutions. We sweat logging for over 15 years now and can help quickly. +Depending on your needs, consulting can be carried out via email, the +phone or on your premises (for larger or local projects). Everything is +possible, it just depends on your needs. Consulting services are +available in English and German. Just mail <a href="mailto:sales@adiscon.com">sales@adiscon.com</a> what you are interested in and we will work with you on a proposal that fits your needs. +</p><p></p><p>All agreements are governed under German law. -</p><p></p> +</p> + <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> @@ -53,5 +84,5 @@ project.<br> Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. -Released under the GNU GPL version 2 or higher.</font></p> -</body></html> +Released under the GNU GPL version 3 or higher.</font></p> +</body></html>
\ No newline at end of file diff --git a/doc/property_replacer.html b/doc/property_replacer.html index 8b777a73..c2a0c0d2 100644 --- a/doc/property_replacer.html +++ b/doc/property_replacer.html @@ -1,175 +1,413 @@ -<html> -<head> -<title>The Rsyslogd Property Replacer</title> -</head> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>The Rsyslogd Property Replacer</title></head> <body> <h1>The Property Replacer</h1> -<p><b>The property replacer is a core component in rsyslogd's output system.</b> -A syslog message has a number of well-defined properties (see below). Each of -this properties can be accessed <b>and</b> manipulated by the property replacer. -With it, it is easy to use only part of a property value or manipulate the value, -e.g. by converting all characters to lower case.</p> +<p><b>The property replacer is a core component in +rsyslogd's output system.</b> A syslog message has a number of +well-defined properties (see below). Each of this properties can be +accessed <b>and</b> manipulated by the property replacer. +With it, it is easy to use only part of a property value or manipulate +the value, e.g. by converting all characters to lower case.</p> <h1>Accessing Properties</h1> -<p>Syslog message properties are used inside templates. They are accessed by putting them between percent signs. Properties can be modified by -the property replacer. The full syntax is as follows:</p> +<p>Syslog message properties are used inside templates. They are +accessed by putting them between percent signs. Properties can be +modified by the property replacer. The full syntax is as follows:</p> <blockquote><b><code>%propname:fromChar:toChar:options%</code></b></blockquote> <h2>Available Properties</h2> -<p><b><code>propname</code></b> is the name of the property to access. It is case-sensitive. +<p><b><code>propname</code></b> is the +name of the property to access. It is case-insensitive (prior to 3.17.0, they were case-senstive). Currently supported are:</p> <table> -<tr><td><b>msg</b></td><td>the MSG part of the message (aka "the message" ;))</td></tr> -<tr><td><b>rawmsg</b></td><td>the message excactly as it was received from the -socket. Should be useful for debugging.</td></tr> -<tr><td><b>UxTradMsg</b></td><td>will disappear soon - do NOT use!</td></tr> -<tr><td><b>HOSTNAME</b></td><td>hostname from the message</td></tr> -<tr><td><b>source</b></td><td>alias for HOSTNAME</td></tr> -<tr><td><b>FROMHOST</b></td><td>hostname of the system the message was received - from (in a relay chain, this is the system immediately in front of us and - not necessarily the original sender)</td></tr> -<tr><td><b>syslogtag</b></td><td>TAG from the message</td></tr> -<tr><td><b>programname</b></td><td>the "static" part of the tag, as defined by -BSD syslogd. For example, when TAG is "named[12345]", programname is "named".</td></tr> -<tr><td><b>PRI</b></td><td>PRI part of the message - undecoded (single value)</td></tr> -<tr><td><b>PRI-text</b></td><td>the PRI part of the message in a textual form - (e.g. "syslog.info")</td></tr> -<tr><td><b>IUT</b></td><td>the monitorware InfoUnitType - used when talking -to a <a href="http://www.monitorware.com">MonitorWare</a> backend (also for - <a href="http://www.phplogcon.org/">phpLogCon</a>)</td></tr> -<tr><td><b>syslogfacility</b></td><td>the facility from the message - in numerical form</td></tr> -<tr> - <td><b>syslogfacility-text</b></td><td>the facility from the message - in - text form</td> -</tr> -<tr> - <td><b>syslogseverity</b></td><td>severity from the - message - in numerical form</td> -</tr> -<tr><td><b>syslogseverity-text</b></td><td>severity from the - message - in text form</td></tr> -<tr><td><b>syslogpriority</b></td><td>an alias for syslogseverity - included for - historical reasons (be careful: it still is the severity, not PRI!)</td></tr> -<tr><td><b>syslogpriority-text</b></td><td>an alias for syslogseverity-text</td></tr> -<tr><td><b>timegenerated</b></td><td>timestamp when the message was RECEIVED. Always in - high resolution</td></tr> -<tr><td><b>timereported</b></td><td>timestamp from the message. Resolution depends on +<tbody> +<tr> +<td><b>msg</b></td> +<td>the MSG part of the message (aka "the message" ;))</td> +</tr> +<tr> +<td><b>rawmsg</b></td> +<td>the message excactly as it was received from the +socket. Should be useful for debugging.</td> +</tr> +<tr> +<td><b>uxtradmsg</b></td> +<td>will disappear soon - do NOT use!</td> +</tr> +<tr> +<td><b>hostname</b></td> +<td>hostname from the message</td> +</tr> +<tr> +<td><b>source</b></td> +<td>alias for HOSTNAME</td> +</tr> +<tr> +<td><b>fromhost</b></td> +<td>hostname of the system the message was received from +(in a relay chain, this is the system immediately in front of us and +not necessarily the original sender). This is a DNS-resolved name, except +if that is not possible or DNS resolution has been disabled.</td> +</tr> +<tr> +<td><b>fromhost-ip</b></td> +<td>The same as fromhost, but alsways as an IP address. Local inputs +(like imklog) use 127.0.0.1 in this property.</td> +</tr> +<tr> +<td><b>syslogtag</b></td> +<td>TAG from the message</td> +</tr> +<tr> +<td><b>programname</b></td> +<td>the "static" part of the tag, as defined by +BSD syslogd. For example, when TAG is "named[12345]", programname is +"named".</td> +</tr> +<tr> +<td><b>pri</b></td> +<td>PRI part of the message - undecoded (single value)</td> +</tr> +<tr> +<td><b>pri-text</b></td> +<td>the PRI part of the message in a textual form (e.g. +"syslog.info")</td> +</tr> +<tr> +<td><b>iut</b></td> +<td>the monitorware InfoUnitType - used when talking +to a <a href="http://www.monitorware.com">MonitorWare</a> +backend (also for <a href="http://www.phplogcon.org/">phpLogCon</a>)</td> +</tr> +<tr> +<td><b>syslogfacility</b></td> +<td>the facility from the message - in numerical form</td> +</tr> +<tr> +<td><b>syslogfacility-text</b></td> +<td>the facility from the message - in text form</td> +</tr> +<tr> +<td><b>syslogseverity</b></td> +<td>severity from the message - in numerical form</td> +</tr> +<tr> +<td><b>syslogseverity-text</b></td> +<td>severity from the message - in text form</td> +</tr> +<tr> +<td><b>syslogpriority</b></td> +<td>an alias for syslogseverity - included for historical +reasons (be careful: it still is the severity, not PRI!)</td> +</tr> +<tr> +<td><b>syslogpriority-text</b></td> +<td>an alias for syslogseverity-text</td> +</tr> +<tr> +<td><b>timegenerated</b></td> +<td>timestamp when the message was RECEIVED. Always in high +resolution</td> +</tr> +<tr> +<td><b>timereported</b></td> +<td>timestamp from the message. Resolution depends on what was provided in the message (in most cases, -only seconds)</td></tr> -<tr><td><b>TIMESTAMP</b></td><td>alias for timereported</td></tr> -<tr><td><b>PROTOCOL-VERSION</b></td><td>The contents of the PROTCOL-VERSION - field from IETF draft draft-ietf-syslog-protcol</td></tr> -<tr><td><b>STRUCTURED-DATA</b></td><td>The contents of the STRUCTURED-DATA field - from IETF draft draft-ietf-syslog-protocol</td></tr> -<tr><td><b>APP-NAME</b></td><td>The contents of the APP-NAME field from IETF - draft draft-ietf-syslog-protocol</td></tr> -<tr><td><b>PROCID</b></td><td>The contents of the PROCID field from IETF draft - draft-ietf-syslog-protocol</td></tr> -<tr><td height="24"><b>MSGID</b></td><td height="24">The contents of the MSGID field from IETF draft - draft-ietf-syslog-protocol</td></tr> -<tr><td><b>$NOW</b></td><td>The current date stamp in the format YYYY-MM-DD</td></tr> -<tr><td><b>$YEAR</b></td><td>The current year (4-digit)</td></tr> -<tr><td><b>$MONTH</b></td><td>The current month (2-digit)</td></tr> -<tr><td><b>$DAY</b></td><td>The current day of the month (2-digit)</td></tr> -<tr><td><b>$HOUR</b></td><td>The current hour in military (24 hour) time - (2-digit)</td></tr> -<tr> -<td><b>$HHOUR</b></td> +only seconds)</td> +</tr> +<tr> +<td><b>timestamp</b></td> +<td>alias for timereported</td> +</tr> +<tr> +<td><b>protocol-version</b></td> +<td>The contents of the PROTCOL-VERSION field from IETF +draft draft-ietf-syslog-protcol</td> +</tr> +<tr> +<td><b>structured-data</b></td> +<td>The contents of the STRUCTURED-DATA field from IETF +draft draft-ietf-syslog-protocol</td> +</tr> +<tr> +<td><b>app-name</b></td> +<td>The contents of the APP-NAME field from IETF draft +draft-ietf-syslog-protocol</td> +</tr> +<tr> +<td><b>procid</b></td> +<td>The contents of the PROCID field from IETF draft +draft-ietf-syslog-protocol</td> +</tr> +<tr> +<td><b>msgid</b></td> +<td>The contents of the MSGID field from +IETF draft draft-ietf-syslog-protocol</td> +</tr> +<td><b>inputname</b></td> +<td>The name of the input module that generated the +message (e.g. "imuxsock", "imudp"). Note that not all modules +necessarily provide this property. If not provided, it is an +empty string. Also note that the input module may provide +any value of its liking. Most importantly, it is <b>not</b> +necessarily the module input name. Internal sources can also +provide inputnames. Currently, "rsyslogd" is defined as inputname +for messages internally generated by rsyslogd, for example startup +and shutdown and error messages. +This property is considered useful when trying to filter messages +based on where they originated - e.g. locally generated messages +("rsyslogd", "imuxsock", "imklog") should go to a different place +than messages generated somewhere. +</td> +</tr> +<tr> +<td><b>$now</b></td> +<td>The current date stamp in the format YYYY-MM-DD</td> +</tr> +<tr> +<td><b>$year</b></td> +<td>The current year (4-digit)</td> +</tr> +<tr> +<td><b>$month</b></td> +<td>The current month (2-digit)</td> +</tr> +<tr> +<td><b>$day</b></td> +<td>The current day of the month (2-digit)</td> +</tr> +<tr> +<td><b>$hour</b></td> +<td>The current hour in military (24 hour) time (2-digit)</td> +</tr> +<tr> +<td><b>$hhour</b></td> <td>The current half hour we are in. From minute 0 to 29, this is always 0 while from 30 to 59 it is always 1.</td> </tr> <tr> -<td><b>$QHOUR</b></td> +<td><b>$qhour</b></td> <td>The current quarter hour we are in. Much like $HHOUR, but values range from 0 to 3 (for the four quater hours that are in each hour)</td> </tr> <tr> -<tr><td><b>$MINUTE</b></td><td>The current minute (2-digit)</td></tr> +<td><b>$minute</b></td> +<td>The current minute (2-digit)</td> +</tr> +<tr> +<td><b>$myhostname</b></td> +<td>The name of the current host as it knows itself (probably useful +for filtering in a generic way)</td> +</tr> +</tbody> </table> -<p>Properties starting with a $-sign are so-called system properties. These do -NOT stem from the message but are rather internally-generated.</p> +<p>Properties starting with a $-sign are so-called system +properties. These do NOT stem from the message but are rather +internally-generated.</p> <h2>Character Positions</h2> -<p><b><code>FromChar</code></b> and <b><code>toChar</code></b> are used to build substrings. They specify the offset within -the string that should be copied. Offset counting starts at 1, so if you need to -obtain the first 2 characters of the message text, you can use this syntax: -"%msg:1:2%". If you do not whish to specify from and to, but you want to specify -options, you still need to include the colons. For example, if you would like to -convert the full message text to lower case, use "%msg:::lowercase%". -If you would like to extract from a position until the end of the string, you -can place a dollar-sign ("$") in toChar (e.g. %msg:10:$%, which will extract -from position 10 to the end of the string).<p> -There is also support for <b>regular expressions</b>. To use them, you need to -place a "R" into FromChar. This tells rsyslog that a regular expression instead -of position-based extraction is desired. The actual regular expression must then -be provided in toChar. The regular expression <b>must</b> be followed by the -string "--end". It denotes the end of the regular expression and will not become -part of it. If you are using regular expressions, the property replacer will -return the part of the property text that matches the regular expression. An -example for a property replacer sequence with a regular expression is: "%msg:R:.*Sev:. -\(.*\) \[.*--end%"<br> +<p><b><code>FromChar</code></b> and <b><code>toChar</code></b> +are used to build substrings. They specify the offset within the string +that should be copied. Offset counting starts at 1, so if you need to +obtain the first 2 characters of the message text, you can use this +syntax: "%msg:1:2%". If you do not whish to specify from and to, but +you want to specify options, you still need to include the colons. For +example, if you would like to convert the full message text to lower +case, use "%msg:::lowercase%". If you would like to extract from a +position until the end of the string, you can place a dollar-sign ("$") +in toChar (e.g. %msg:10:$%, which will extract from position 10 to the +end of the string).</p> +<p>There is also support for <b>regular expressions</b>. +To use them, you need to place a "R" into FromChar. This tells rsyslog +that a regular expression instead of position-based extraction is +desired. The actual regular expression must then be provided in toChar. +The regular expression <b>must</b> be followed by the +string "--end". It denotes the end of the regular expression and will +not become part of it. If you are using regular expressions, the +property replacer will return the part of the property text that +matches the regular expression. An example for a property replacer +sequence with a regular expression is: "%msg:R:.*Sev:. \(.*\) +\[.*--end%"</p> +<p>It is possible to specify some parametes after the "R". These are +comma-separated. They are: +<p>R,<regexp-type>,<submatch>,<nomatch>,<match-number> +<p>regexp-type is either "BRE" for Posix basic regular expressions or +"ERE" for extended ones. The string must be given in upper case. The +default is "BRE" to be consistent with earlier versions of rsyslog that +did not support ERE. The submatch identifies the submatch to be used +with the result. A single digit is supported. Match 0 is the full match, +while 1 to 9 are the acutal submatches. The match-number identifies which match to +use, if the expression occurs more than once inside the string. Please note +that the first match is number 0, the second 1 and so on. Up to 10 matches +(up to number 9) are supported. Please note that it would be more +natural to have the match-number in front of submatch, but this would break +backward-compatibility. So the match-number must be specified after "nomatch". +<p>nomatch is either "DFLT", "BLANK", ZERO or "FIELD" (all upper case!). It tells +what to use if no match is found. With "DFLT", the strig "**NO MATCH**" is +used. This was the only supported value up to rsyslog 3.19.5. With "BLANK" +a blank text is used (""). With "ZERO", "0" is used. +Finally, "FIELD" uses the full property text +instead of the expression. Some folks have requested that, so it seems +to be useful. +<p>The following is a sample of an ERE expression that takes the first +submatch from the message string and replaces the expression with +the full field if no match is found: +<p>%msg:R,ERE,1,FIELD:for (vlan[0-9]*):--end% +<p>and this takes the first submatch of the second match of said expression: +<p>%msg:R,ERE,1,FIELD,1:for (vlan[0-9]*):--end% +<p><b>Please note: there is also a +<a href="http://www.rsyslog.com/tool-regex">rsyslog regular expression checker/generator</a> +online tool available.</b> With that tool, you can check your regular expressions and +also generate a valid property replacer sequence. Usage of this tool is recommended. +Depending on the version offered, the tool may not cover all subleties that can +be done with the property replacer. It concentrates on the most often used cases. So it +is still useful to hand-craft expressions for demanding environments. +<p><b>Also, extraction can be done based on so-called +"fields"</b>. To do so, place a "F" into FromChar. A field in its +current definition is anything that is delimited by a delimiter +character. The delimiter by default is TAB (US-ASCII value 9). However, +if can be changed to any other US-ASCII character by specifying a comma +and the <b>decimal</b> US-ASCII value of the delimiter +immediately after the "F". For example, to use comma (",") as a +delimiter, use this field specifier: "F,44". If your syslog +data is delimited, this is a quicker way to extract than via regular +expressions (actually, a *much* quicker way). Field counting starts at +1. Field zero is accepted, but will always lead to a "field not found" +error. The same happens if a field number higher than the number of +fields in the property is requested. The field number must be placed in +the "ToChar" parameter. An example where the 3rd field (delimited by +TAB) from the msg property is extracted is as follows: "%msg:F:3%". The +same example with semicolon as delimiter is "%msg:F,59:3%".</p> +<p>Please note that the special characters "F" and "R" are +case-sensitive. Only upper case works, lower case will return an error. +There are no white spaces permitted inside the sequence (that will lead +to error messages and will NOT provide the intended result).</p> +<p>Each occurence of the field delimiter starts a new field. However, +if you add a plus sign ("+") after the field delimiter, multiple +delimiters, one immediately after the others, are treated as separate +fields. This can be useful in cases where the syslog message contains +such sequences. A frequent case may be with code that is written as +follows:</p> +<code><pre> +int n, m; +... +syslog(LOG_ERR, "%d test %6d", n, m); +</pre></code> +<p>This will result into things like this in syslog messages: +"1 test 2", +"1 test 23", +"1 test 234567" +<p>As you can see, the fields are delimited by space characters, but +their exact number is unknown. They can properly be extracted as follows: <p> -<b>Also, extraction can be done based on so-called "fields"</b>. To do so, place -a "F" into FromChar. A field in its current definition is anything -that is delimited by a delimiter character. The delimiter by default is TAB -(US-ASCII value 9). However, if can be changed to any other US-ASCII character -by specifying a comma and the <b>decimal</b> US-ASCII value of the delimiter immediately after the -"F". For example, to use comma (",") as a delimiter, use this field specifier: -"F,44". If your syslog data is delimited, -this is a quicker way to extract than via regular expressions (actually, a *much* -quicker way). Field counting starts at 1. Field zero is accepted, but will -always lead to a "field not found" error. The same happens if a field number -higher than the number of fields in the property is requested. The field number -must be placed in the "ToChar" parameter. An example where the 3rd field -(delimited by TAB) from -the msg property is extracted is as follows: "%msg:F:3%". The same -example with semicolon as delimiter is "%msg:F,59:3%".<p> -Please note that the special characters "F" and "R" are case-sensitive. Only -upper case works, lower case will return an error. There are no white spaces -permitted inside the sequence (that will lead to error messages and will NOT -provide the intended result).<br> +"%msg:F,32:2%" to "%msg:F,32+:2%". +<p>This feature was suggested by Zhuang Yuyao and implemented by him. +It is modeled after perl compatible regular expressions. +</p> + <h2>Property Options</h2> -<b><code>property options</code></b> are case-insensitive. Currently, the following options -are defined:</p> +<b><code>property options</code></b> are +case-insensitive. Currently, the following options are defined: +<p></p> <table> -<tr><td><b>uppercase</b></td><td>convert property to lowercase only</td></tr> -<tr><td><b>lowercase</b></td><td>convert property text to uppercase only</td></tr> -<tr><td><b>drop-last-lf</b></td><td>The last LF in the message (if any), is dropped. - Especially useful for PIX.</td></tr> -<tr><td><b>date-mysql</b></td><td>format as mysql date</td></tr> -<tr><td><b>date-rfc3164</b></td><td>format as RFC 3164 date</td></tr> -<tr><td><b>date-rfc3339</b></td><td>format as RFC 3339 date</td></tr> -<tr> - <td><b>escape-cc</b></td><td>replace control characters (ASCII value 127 and - values less then 32) with an escape sequence. The sequence is "#<charval>" - where charval is the 3-digit decimal value of the control character. For - example, a tabulator would be replaced by "#009".<br> - Note: using this option requires that - <a href="../../rsyslog/doc/rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> - is set to off.</td> -</tr> -<tr> - <td><b>space-cc</b></td><td>replace control characters by spaces<br> - Note: using this option requires that - <a href="../../rsyslog/doc/rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> - is set to off.</td> -</tr> -<tr> - <td><b>drop-cc</b></td><td>drop control characters - the resulting string - will neither contain control characters, escape sequences nor any other - replacement character like space.<br> - Note: using this option requires that - <a href="../../rsyslog/doc/rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> - is set to off.</td> +<tbody> +<tr> +<td><b>uppercase</b></td> +<td>convert property to lowercase only</td> </tr> +<tr> +<td><b>lowercase</b></td> +<td>convert property text to uppercase only</td> +</tr> +<tr> +<td><b>drop-last-lf</b></td> +<td>The last LF in the message (if any), is dropped. +Especially useful for PIX.</td> +</tr> +<tr> +<td><b>date-mysql</b></td> +<td>format as mysql date</td> +</tr> +<tr> +<td><b>date-rfc3164</b></td> +<td>format as RFC 3164 date</td> +</tr> +<tr> +<td><b>date-rfc3339</b></td> +<td>format as RFC 3339 date</td> +</tr> +<tr> +<td><b>date-subseconds</b></td> +<td>just the subseconds of a timestamp (always 0 for a low precision timestamp)</td> +</tr> +<tr> +<td valign="top"><b>escape-cc</b></td> +<td>replace control characters (ASCII value 127 and values +less then 32) with an escape sequence. The sequnce is +"#<charval>" where charval is the 3-digit decimal value +of the control character. For example, a tabulator would be replaced by +"#009".<br> +Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> +is set to off.</td> +</tr> +<tr> +<td valign="top"><b>space-cc</b></td> +<td>replace control characters by spaces<br> +Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> +is set to off.</td> +</tr> +<tr> +<td valign="top"><b>drop-cc</b></td> +<td>drop control characters - the resulting string will +neither contain control characters, escape sequences nor any other +replacement character like space.<br> +Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a> +is set to off.</td> +</tr> +<tr> +<td valign="top"><b>sp-if-no-1st-sp</b></td> +<td>This option looks scary and should probably not be used by a user. For any field +given, it returns either a single space character or no character at all. Field content +is never returned. A space is returned if (and only if) the first character of the +field's content is NOT a space. This option is kind of a hack to solve a problem rooted +in RFC 3164: 3164 specifies no delimiter between the syslog tag sequence and the actual +message text. Almost all implementation in fact delemit the two by a space. As of +RFC 3164, this space is part of the message text itself. This leads to a problem when +building the message (e.g. when writing to disk or forwarding). Should a delimiting +space be included if the message does not start with one? If not, the tag is immediately +followed by another non-space character, which can lead some log parsers to misinterpret +what is the tag and what the message. The problem finally surfaced when the klog module +was restructured and the tag correctly written. It exists with other message sources, +too. The solution was the introduction of this special property replacer option. Now, +the default template can contain a conditional space, which exists only if the +message does not start with one. While this does not solve all issues, it should +work good enough in the far majority of all cases. If you read this text and have +no idea of what it is talking about - relax: this is a good indication you will never +need this option. Simply forget about it ;) +</td> +</tr> +<tr> +<td valign="top"><b>secpath-drop</b></td> +<td>Drops slashes inside the field (e.g. "a/b" becomes "ab"). +Useful for secure pathname generation (with dynafiles). +</td> +</tr> +<tr> +<td valign="top"><b>secpath-replace</b></td> +<td>Replace slashes inside the field by an underscore. (e.g. "a/b" becomes "a_b"). +Useful for secure pathname generation (with dynafiles). +</td> +</tr> +</tbody> </table> - +<p>To use multiple options, simply place them one after each other with a comma delmimiting +them. For example "escape-cc,sp-if-no-1st-sp". If you use conflicting options together, +the last one will override the previous one. For example, using "escape-cc,drop-cc" will +use drop-cc and "drop-cc,escape-cc" will use escape-cc mode. <h2>Further Links</h2> <ul> - <li>Article on "<a href="rsyslog_recording_pri.html">Recording the Priority of - Syslog Messages</a>" (describes use of templates to record severity and - facility of a message)</li> - <li><a href="rsyslog_conf.html">Configuration file syntax</a>, this is where you - actually use the property replacer.</li> +<li>Article on "<a href="rsyslog_recording_pri.html">Recording +the Priority of Syslog Messages</a>" (describes use of templates +to record severity and facility of a message)</li> +<li><a href="rsyslog_conf.html">Configuration file +syntax</a>, this is where you actually use the property replacer.</li> </ul> - -</body> -</html> +</body></html> diff --git a/doc/queueWorkerLogic.jpg b/doc/queueWorkerLogic.jpg Binary files differnew file mode 100644 index 00000000..fb143c4a --- /dev/null +++ b/doc/queueWorkerLogic.jpg diff --git a/doc/queueWorkerLogic_small.jpg b/doc/queueWorkerLogic_small.jpg Binary files differnew file mode 100644 index 00000000..4fae6d27 --- /dev/null +++ b/doc/queueWorkerLogic_small.jpg diff --git a/doc/queues.html b/doc/queues.html new file mode 100644 index 00000000..a2074d36 --- /dev/null +++ b/doc/queues.html @@ -0,0 +1,360 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="de"> +<title>Understanding rsyslog queues</title></head> +<body> + +<h1>Understanding rsyslog Queues</h1> +<p>Rsyslog uses queues whenever two activities need to be loosely coupled. With a +queue, one part of the system "produces" something while another part "consumes" +this something. The "something" is most often syslog messages, but queues may +also be used for other purposes.</p> +<p>The most prominent example is the main message queue. Whenever rsyslog +receives a message (e.g. locally, via UDP, TCP or in whatever else way), it +places these messages into the main message queue. Later, it is dequeued by the +rule processor, which then evaluates which actions are to be carried out. In +front of each action, there is also a queue, which potentially de-couples the +filter processing from the actual action (e.g. writing to file, database or +forwarding to another host).</p> +<h1>Queue Modes</h1> +<p>Rsyslog supports different queue modes, some with submodes. Each of them has +specific advantages and disadvantages. Selecting the right queue mode is quite +important when tuning rsyslogd. The queue mode (aka "type") is set via the "<i>$<object>QueueType</i>" +config directive.</p> +<h2>Direct Queues</h2> +<p>Direct queues are <b>non</b>-queuing queues. A queue in direct mode does +neither queue nor buffer any of the queue elements but rather passes the element +directly (and immediately) from the producer to the consumer. This sounds strange, +but there is a good reason for this queue type.</p> +<p>Direct mode queues allow to use queues generically, even in places where +queuing is not always desired. A good example is the queue in front of output +actions. While it makes perfect sense to buffer forwarding actions or database +writes, it makes only limited sense to build up a queue in front of simple local +file writes. Yet, rsyslog still has a queue in front of every action. So for +file writes, the queue mode can simply be set to "direct", in which case no +queuing happens.</p> +<p>Please note that a direct queue also is the only queue type that passes back +the execution return code (success/failure) from the consumer to the producer. +This, for example, is needed for the backup action logic. Consequently, backup +actions require the to-be-checked action to use a "direct" mode queue.</p> +<p>To create a direct queue, use the "<i>$<object>QueueType Direct</i>" config +directive.</p> +<h2>Disk Queues</h2> +<p>Disk queues use disk drives for buffering. The important fact is that the +always use the disk and do not buffer anything in memory. Thus, the queue is +ultra-reliable, but by far the slowest mode. For regular use cases, this queue +mode is not recommended. It is useful if log data is so important that it must +not be lost, even in extreme cases.</p> +<p>When a disk queue is written, it is done in chunks. Each chunk receives its +individual file. Files are named with a prefix (set via the "<i>$<object>QueueFilename</i>" +config directive) and followed by a 7-digit number (starting at one and +incremented for each file). Chunks are 10mb by default, a different size can be +set via the"<i>$<object>QueueMaxFileSize</i>" config directive. Note that +the size limit is not a sharp one: rsyslog always writes one complete queue +entry, even if it violates the size limit. So chunks are actually a little but +(usually less than 1k) larger then the configured size. Each chunk also has a +different size for the same reason. If you observe different chunk sizes, you +can relax: this is not a problem.</p> +<p>Writing in chunks is used so that processed data can quickly be deleted and +is free for other uses - while at the same time keeping no artificial upper +limit on disk space used. If a disk quota is set (instructions further below), +be sure that the quota/chunk size allows at least two chunks to be written. +Rsyslog currently does not check that and will fail miserably if a single chunk +is over the quota.</p> +<p>Creating new chunks costs performance but provides quicker ability to free +disk space. The 10mb default is considered a good compromise between these two. +However, it may make sense to adapt these settings to local policies. For +example, if a disk queue is written on a dedicated 200gb disk, it may make sense +to use a 2gb (or even larger) chunk size.</p> +<p>Please note, however, that the disk queue by default does not update its +housekeeping structures every time it writes to disk. This is for performance +reasons. In the event of failure, data will still be lost (except when manually +is mangled with the file structures). However, disk queues can be set to write +bookkeeping information on checkpoints (every n records), so that this can be +made ultra-reliable, too. If the checkpoint interval is set to one, no data can +be lost, but the queue is exceptionally slow.</p> +<p>Each queue can be placed on a different disk for best performance and/or +isolation. This is currently selected by specifying different <i>$WorkDirectory</i> +config directives before the queue creation statement.</p> +<p>To create a disk queue, use the "<i>$<object>QueueType Disk</i>" config +directive. Checkpoint intervals can be specified via "<i>$<object>QueueCheckpointInterval</i>", +with 0 meaning no checkpoints. </p> +<h2>In-Memory Queues</h2> +<p>In-memory queue mode is what most people have on their mind when they think +about computing queues. Here, the enqueued data elements are held in memory. +Consequently, in-memory queues are very fast. But of course, they do not survive +any program or operating system abort (what usually is tolerable and unlikely). +Be sure to use an UPS if you use in-memory mode and your log data is important +to you. Note that even in-memory queues may hold data for an infinite amount of +time when e.g. an output destination system is down and there is no reason to move +the data out of memory (lying around in memory for an extended period of time is +NOT a reason). Pure in-memory queues can't even store queue elements anywhere +else than in core memory. </p> +<p>There exist two different in-memory queue modes: LinkedList and FixedArray. +Both are quite similar from the user's point of view, but utilize different +algorithms. </p> +<p>A FixedArray queue uses a fixed, pre-allocated array that holds pointers to +queue elements. The majority of space is taken up by the actual user data +elements, to which the pointers in the array point. The pointer array itself is +comparatively small. However, it has a certain memory footprint even if the +queue is empty. As there is no need to dynamically allocate any housekeeping +structures, FixedArray offers the best run time performance (uses the least CPU +cycle). FixedArray is best if there is a relatively low number of queue elements +expected and performance is desired. It is the default mode for the main message +queue (with a limit of 10,000 elements).</p> +<p>A LinkedList queue is quite the opposite. All housekeeping structures are +dynamically allocated (in a linked list, as its name implies). This requires +somewhat more runtime processing overhead, but ensures that memory is only +allocated in cases where it is needed. LinkedList queues are especially +well-suited for queues where only occasionally a than-high number of elements +need to be queued. A use case may be occasional message burst. Memory +permitting, it could be limited to e.g. 200,000 elements which would take up +only memory if in use. A FixedArray queue may have a too large static memory +footprint in such cases.</p> +<p><b>In general, it is advised to use LinkedList mode if in doubt</b>. The +processing overhead compared to FixedArray is low and may be +<span style="font-size: 12pt; line-height: 115%; font-family: 'Times New Roman',serif;" lang="EN-US"> +outweigh </span>by the reduction in memory use. Paging in most-often-unused +pointer array pages can be much slower than dynamically allocating them.</p> +<p>To create an in-memory queue, use the "<i>$<object>QueueType LinkedList</i>" +or "<i>$<object>QueueType FixedArray</i>" config directive.</p> +<h3>Disk-Assisted Memory Queues</h3> +<p>If a disk queue name is defined for in-memory queues (via <i> +$<object>QueueFileName</i>), they automatically +become "disk-assisted" (DA). In that mode, data is written to disk (and read +back) on an as-needed basis.</p> +<p>Actually, the regular memory queue (called the +"primary queue") and a disk queue (called the "DA queue") work in tandem in this +mode. Most importantly, the disk queue is activated if the primary queue is full +or needs to be persisted on shutdown. Disk-assisted queues combine the +advantages of pure memory queues with those of pure disk queues. Under normal +operations, they are very fast and messages will never touch the disk. But if +there is need to, an unlimited amount of messages can be buffered (actually +limited by free disk space only) and data can be persisted between rsyslogd runs.</p> +<p>With a DA-queue, both disk-specific and in-memory specific configuration +parameters can be set. From the user's point of view, think of a DA queue like a +"super-queue" which does all within a single queue [from the code perspective, +there is some specific handling for this case, so it is actually much like a +single object].</p> +<p>DA queues are typically used to de-couple potentially long-running and +unreliable actions (to make them reliable). For example, it is recommended to +use a disk-assisted linked list in-memory queue in front of each database and +"send via tcp" action. Doing so makes these actions reliable and de-couples +their potential low execution speed from the rest of your rules (e.g. the local +file writes). There is a howto on <a href="rsyslog_high_database_rate.html"> +massive database inserts</a> which nicely describes this use case. It may even +be a good read if you do not intend to use databases.</p> +<p>With DA queues, we do not simply write out everything to disk and then run as +a disk queue once the in-memory queue is full. A much smarter algorithm is used, +which involves a "high watermark" and a "low watermark". Both specify numbers of +queued items. If the queue size reaches high watermark elements, the queue +begins to write data elements to disk. It does so until it reaches the low water +mark elements. At this point, it stops writing until either high water mark is +reached again or the on-disk queue becomes empty, in which case the queue +reverts back to in-memory mode, only. While holding at the low watermark, new +elements are actually enqueued in memory. They are eventually written to disk, +but only if the high water mark is ever reached again. If it isn't, these items +never touch the disk. So even when a queue runs disk-assisted, there is +in-memory data present (this is a big difference to pure disk queues!).</p> +<p>This algorithm prevents unnecessary disk writes, but also leaves some +additional buffer space for message bursts. Remember that creating disk files +and writing to them is a lengthy operation. It is too lengthy to e.g. block +receiving UDP messages. Doing so would result in message loss. Thus, the queue +initiates DA mode, but still is able to receive messages and enqueue them - as +long as the maximum queue size is not reached. The number of elements between +the high water mark and the maximum queue size serves as this "emergency +buffer". Size it according to your needs, if traffic is very bursty you will +probably need a large buffer here. Keep in mind, though, that under normal +operations these queue elements will probably never be used. Setting the high +water mark too low will cause disk-assistance to be turned on more often than +actually needed.</p> +<p>The water marks can be set via the "<i>$<object>QueueHighWatermark</i>" and +"<i>$<object>QueueHighWatermark</i>" configuration file directives. Note that +these are actual numbers, not precentages. Be sure they make sense (also in +respect to "<i>$<object>QueueSize</i>"), as rsyslodg does currently not perform +any checks on the numbers provided. It is easy to screw up the system here (yes, +a feature enhancement request is filed ;)).</p> +<h1>Limiting the Queue Size</h1> +<p>All queues, including disk queues, have a limit of the number of elements +they can enqueue. This is set via the "<i>$<object>QueueSize</i>" config +parameter. Note that the size is specified in number of enqueued elements, not +their actual memory size. Memory size limits can not be set. A conservative +assumption is that a single syslog messages takes up 512 bytes on average +(in-memory, NOT on the wire, this *is* a difference).</p> +<p>Disk assisted queues are special in that they do <b>not</b> have any size +limit. The enqueue an unlimited amount of elements. To prevent running out of +space, disk and disk-assisted queues can be size-limited via the "<i>$<object>QueueMaxDiskSpace</i>" +configuration parameter. If it is not set, the limit is only available free +space (and reaching this limit is currently not very gracefully handled, so +avoid running into it!). If a limit is set, the queue can not grow larger than +it. Note, however, that the limit is approximate. The engine always writes +complete records. As such, it is possible that slightly more than the set limit +is used (usually less than 1k, given the average message size). Keeping strictly +on the limit would be a performance hurt, and thus the design decision was to +favour performance. If you don't like that policy, simply specify a slightly +lower limit (e.g. 999,999K instead of 1G).</p> +<p>In general, it is a good idea to limit the pysical disk space even if you +dedicate a whole disk to rsyslog. That way, you prevent it from running out of +space (future version will have an auto-size-limit logic, that then kicks in in +such situations).</p> +<h1>Worker Thread Pools</h1> +<p>Each queue (except in "direct" mode) has an associated pool of worker +threads. Worker threads carry out the action to be performed on the data +elements enqueued. As an actual sample, the main message queue's worker task is +to apply filter logic to each incoming message and enqueue them to the relevant +output queues (actions).</p> +<p>Worker threads are started and stopped on an as-needed basis. On a system +without activity, there may be no worker at all running. One is automatically +started when a message comes in. Similarily, additional workers are started if +the queue grows above a specific size. The "<i>$<object>QueueWorkerThreadMinimumMessages</i>" +config parameter controls worker startup. If it is set to the minimum number of +elements that must be enqueued in order to justify a new worker startup. For +example, let's assume it is set to 100. As long as no more than 100 messages are +in the queue, a single worker will be used. When more than 100 messages arrive, +a new worker thread is automatically started. Similarily, a third worker will be +started when there are at least 300 messages, a forth when reaching 400 and so +on.</p> +<p>It, however, does not make sense to have too many worker threads running in +parall. Thus, the upper limit ca be set via "<i>$<object>QueueWorkerThreads</i>". +If it, for example, is set to four, no more than four workers will ever be +started, no matter how many elements are enqueued. </p> +<p>Worker threads that have been started are kept running until an inactivity +timeout happens. The timeout can be set via "<i>$<object>QueueWorkerTimeoutShutdown</i>" +and is specified in milliseconds. If you do not like to keep the workers +running, simply set it to 0, which means immediate timeout and thus immediate +shutdown. But consider that creating threads involves some overhead, and this is +why we keep them running.</p> +<h2>Discarding Messages</h2> +<p>If the queue reaches the so called "discard watermark" (a number of queued +elements), less important messages can automatically be discarded. This is in an +effort to save queue space for more important messages, which you even less like +to loose. Please note that whenever there are more than "discard watermark" +messages, both newly incoming as well as already enqueued low-priority messages +are discarded. The algorithm discards messages newly coming in and those at the +front of the queue.</p> +<p>The discard watermark is a last resort setting. It should be set sufficiently +high, but low enough to allow for large message burst. Please note that it take +effect immediately and thus shows effect promptly - but that doesn't help if the +burst mainly consist of high-priority messages...</p> +<p>The discard watermark is set via the "<i>$<object>QueueDiscardMark</i>" +directive. The priority of messages to be discarded is set via "<i>$<object>QueueDiscardSeverity</i>". +This directive accepts both the usual textual severity as well as a +numerical one. To understand it, you must be aware of the numerical +severity values. They are defined in RFC 3164:</p> +<pre> Numerical Severity<br> Code<br><br> 0 Emergency: system is unusable<br> 1 Alert: action must be taken immediately<br> 2 Critical: critical conditions<br> 3 Error: error conditions<br> 4 Warning: warning conditions<br> 5 Notice: normal but significant condition<br> 6 Informational: informational messages<br> 7 Debug: debug-level messages</pre> +<p>Anything of the specified severity and (numerically) above it is +discarded. To turn message discarding off, simply specify the discard +watermark to be higher than the queue size. An alternative is to +specify the numerical value 8 as DiscardSeverity. This is also the +default setting to prevent unintentional message loss. So if you would +like to use message discarding, you need to set" <i>$<object>QueueDiscardSeverity</i>" to an actual value.</p> +<p>An interesting application is with disk-assisted queues: if the discard +watermark is set lower than the high watermark, message discarding will start +before the queue becomes disk-assisted. This may be a good thing if you would +like to switch to disk-assisted mode only in cases where it is absolutely +unavoidable and you prefer to discard less important messages first.</p> +<h1>Filled-Up Queues</h1> +<p>If the queue has either reached its configured maximum number of entries or +disk space, it is finally full. If so, rsyslogd throttles the data element +submitter. If that, for example, is a reliable input (TCP, local log socket), +that will slow down the message originator which is a good +<span style="font-size: 12pt; line-height: 115%; font-family: 'Times New Roman',serif;" lang="EN-US"> +resolution </span>for this scenario.</p> +<p>During +<span style="font-size: 12pt; line-height: 115%; font-family: 'Times New Roman',serif;" lang="EN-US"> +throtteling</span>, a disk-assisted queue continues to write to disk and +messages are also discarded based on severity as well as regular dequeuing and +processing continues. So chances are good the situation will be resolved by +simply throttling. Note, though, that throtteling is highly undesirable for +unreliable sources, like UDP message reception. So it is not a good thing to run +into throtteling mode at all.</p> +<p>We can not hold processing +<span style="font-size: 12pt; line-height: 115%; font-family: 'Times New Roman',serif;" lang="EN-US"> +infinitely</span>, not even when throtteling. For example, throtteling the local +log socket too long would cause the system at whole come to a standstill. To +prevent this, rsyslogd times out after a configured period ("<i>$<object>QueueTimeoutEnqueue</i>", +specified in milliseconds) if no space becomes available. As a last resort, it +then discards the newly arrived message.</p> +<p>If you do not like throtteling, set the timeout to 0 - the message will then +immediately be discarded. If you use a high timeout, be sure you know what you +do. If a high main message queue enqueue timeout is set, it can lead to +something like a complete hang of the system. The same problem does not apply to +action queues.</p> +<h2>Rate Limiting</h2> +<p>Rate limiting provides a way to prevent rsyslogd from processing things too +fast. It can, for example, prevent overruning a receiver system.</p> +<p>Currently, there are only limited rate-limiting features available. The "<i>$<object>QueueDequeueSlowdown</i>" +directive allows to specify how long (in microseconds) dequeueing should be +delayed. While simple, it still is powerful. For example, using a +DequeueSlowdown delay of 1,000 microseconds on a UDP send action ensures that no +more than 1,000 messages can be sent within a second (actually less, as there is +also some time needed for the processing itself).</p><h2>Processing Timeframes</h2><p>Queues +can be set to dequeue (process) messages only during certain +timeframes. This is useful if you, for example, would like to transfer +the bulk of messages only during off-peak hours, e.g. when you have +only limited bandwidth on the network path the the central server.</p><p>Currently, +only a single timeframe is supported and, even worse, it can only be +specified by the hour. It is not hard to extend rsyslog's capabilities +in this regard - it was just not requested so far. So if you need more +fine-grained control, let us know and we'll probably implement it. +There are two configuration directives, both should be used together or +results are unpredictable:" <i>$<object>QueueDequeueTimeBegin <hour></i>" and "<i>$<object>QueueDequeueTimeEnd <hour></i>". The hour parameter must be specified in 24-hour format (so 10pm is 22). A use case for this parameter can be found in the <a href="http://wiki.rsyslog.com/index.php/OffPeakHours">rsyslog wiki</a>. </p> +<h2>Terminating Queues</h2> +<p>Terminating a process sounds easy, but can be complex. +<span style="font-size: 12pt; line-height: 115%; font-family: 'Times New Roman',serif;" lang="EN-US"> +Terminating </span>a running queue is in fact the most complex operation a queue +object can perform. You don't see that from a user's point of view, but its +quite hard work for the developer to do everything in the right order.</p> +<p>The complexity arises when the queue has still data enqueued when it +finishes. Rsyslog tries to preserve as much of it as possible. As a first +measure, there is a regular queue time out ("<i>$<object>QueueTimeoutShutdown</i>", +specified in milliseconds): the queue workers are given that time period to +finish processing the queue.</p> +<p>If after that period there is still data in the queue, workers are instructed +to finish the current data element and then terminate. This essentially means +any other data is lost. There is another timeout ("<i>$<object>QueueTimeoutActionCompletion</i>", +also specified in milliseconds) that specifies how long the workers have to +finish the current element. If that timeout expires, any remaining workers are +cancelled and the queue is brought down.</p> +<p>If you do not like to lose data on shutdown, the "<i>$<object>QueueSaveOnShutdown</i>" +parameter can be set to "on". This requires either a disk or disk-assisted +queue. If set, rsyslogd ensures that any queue elements are saved to disk before +it terminates. This includes data elements there were begun being processed by +workers that needed to be cancelled due to too-long processing. For a large +queue, this operation may be lengthy. No timeout applies to a required shutdown +save.</p> +<h1>Where are Queues Used?</h1> +<p> Currently, queues are used for the main message queue and for the +actions.</p> +<p>There is a single main message queue inside rsyslog. Each input module +delivers messages to it. The main message queue worker filters messages based on +rules specified in rsyslog.conf and dispatches them to the individual action +queues. Once a message is in an action queue, it is deleted from the main +message queue.</p> +<p>There are multiple action queues, one for each configured action. By default, +these queues operate in direct (non-queueing) mode. Action queues are fully +configurable and thus can be changed to whatever is best for the given use case.</p> +<p>Future versions of rsyslog will most probably utilize queues at other places, +too.</p> +<p> +<span style="font-size: 12pt; line-height: 115%; font-family: 'Times New Roman',serif;" lang="EN-US"> +Wherever </span>"<i><object></i>" was used above in the config file +statements, substitute "<i><object></i>" with either "MainMsg" or "Action". The +former will set main message queue +<span style="font-size: 12pt; line-height: 115%; font-family: 'Times New Roman',serif;" lang="EN-US"> +parameters</span>, the later parameters for the next action that will be +created. Action queue parameters can not be modified once the action has been +specified. For example, to tell the main message queue to save its content on +shutdown, use <i>$MainMsgQueueSaveOnShutdown on</i>".</p> +<p>If the same parameter is specified multiple times before a queue is created, +the last one specified takes precedence. The main message queue is created after +parsing the config file and all of its potential includes. An action queue is +created each time an action selector is specified. Action queue parameters are +reset to default after an action queue has been created (to provide a clean +environment for the next action).</p> +<p>Not all queues necessarily support the full set of queue configuration +parameters, because not all are applicable. For example, in current output +module design, actions do not support multi-threading. Consequently, the number +of worker threads is fixed to one for action queues and can not be changed.</p> + +</body></html>
\ No newline at end of file diff --git a/doc/rainerscript.html b/doc/rainerscript.html new file mode 100644 index 00000000..ef0e41cb --- /dev/null +++ b/doc/rainerscript.html @@ -0,0 +1,63 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>RainerScript</title> + +</head> +<body> +<h1>RainerScript</h1> +<p><b>RainerScript is a scripting language specifically +designed and well-suited +for processing network events and configuring event processors</b> +(with the most prominent sample being syslog). While RainerScript is +theoritically usable with various softwares, it currently is being +used, and developed for, rsyslog. Please note that RainerScript may not +be abreviated as rscript, because that's somebody elses trademark.</p> +<p>RainerScript is currently under development. It has its first +appearance in rsyslog 3.12.0, where it provides complex expression +support. However, this is only a very partial implementatio of the +scripting language. Due to technical restrictions, the final +implementation will have a slightly different syntax. So while you are +invited to use the full power of expresssions, you unfortunatley need +to be prepared to change your configuration files at some later points. +Maintaining backwards-compatibility at this point would cause us to +make too much compromise. Defering the release until everything is +perfect is also not a good option. So use your own judgement.</p> +<p>A formal definition of the language can be found in <a href="rscript_abnf.html">RainerScript ABNF</a>. The +rest of this document describes the language from the user's point of +view. Please note that this doc is also currently under development and +can (and will) probably improve as time progresses. If you have +questions, use the rsyslog forum. Feedback is also always welcome.</p> +<h2>Data Types</h2> +RainerScript is a typeless language. That doesn't imply you don't need +to care about types. Of course, expressions like "A" + "B" will not +return a valid result, as you can't really add two letters (to +concatenate them, use the concatenation operator &). + However, all type conversions are automatically done by the +script interpreter when there is need to do so.<br> +<h2>Expressions</h2> +The language supports arbitrary complex expressions. All usual +operators are supported. The precedence of operations is as follows +(with operations being higher in the list being carried out before +those lower in the list, e.g. multiplications are done before additions.<br> +<ul> +<li>expressions in parenthesis</li><li>not, unary minus</li><li>*, /, % (modulus, as in C)</li><li>+, -, & (string concatenation)</li><li>==, !=, <>, <, >, <=, >=, contains (strings!), startswith (strings!)</li><li>and</li><li>or</li> +</ul>For example, "not a == b" probably returns not what you intended. +The script processor will first evaluate "not a" and then compare the +resulting boolean to the value of b. What you probably intended to do +is "not (a == b)". And if you just want to test for inequality, we +highly suggest to use "!=" or "<>". Both are exactly the same and +are provided so that you can pick whichever you like best. So inquality +of a and b should be tested as "a <> b". The "not" operator +should be reserved to cases where it actually is needed to form a +complex boolean expression. In those cases, parenthesis are highly +recommended. +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html>
\ No newline at end of file diff --git a/doc/rsconf1_actionexeconlywhenpreviousissuspended.html b/doc/rsconf1_actionexeconlywhenpreviousissuspended.html index 3f18e243..d5cf8b14 100644 --- a/doc/rsconf1_actionexeconlywhenpreviousissuspended.html +++ b/doc/rsconf1_actionexeconlywhenpreviousissuspended.html @@ -9,14 +9,14 @@ <p><b>Description:</b></p> <p>This directive allows to specify if actions should always be executed ("off," the default) or only if the previous action is suspended ("on"). This directive works hand-in-hand with the multiple actions per selector feature. It can be used, for example, to create rules that automatically switch destination servers or databases to a (set of) backup(s), if the primary server fails. Note that this feature depends on proper implementation of the suspend feature in the output module. All built-in output modules properly support it (most importantly the database write and the syslog message forwarder).</p> <p>This selector processes all messages it receives (*.*). It tries to forward every message to primary-syslog.example.com (via tcp). If it can not reach that server, it tries secondary-1-syslog.example.com, if that fails too, it tries secondary-2-syslog.example.com. If neither of these servers can be connected, the data is stored in /var/log/localbuffer. Please note that the secondaries and the local log buffer are only used if the one before them does not work. So ideally, /var/log/localbuffer will never receive a message. If one of the servers resumes operation, it automatically takes over processing again.</p> -<p>We strongly advise not to use repeated line reduction together with ActionExecOnlyIfPreviousIsSuspended. It may lead to "interesting" and undesired results (but you can try it if you like).</p> +<p>We strongly advise not to use repeated line reduction together with ActionExecOnlyWhenPreviousIsSuspended. It may lead to "interesting" and undesired results (but you can try it if you like).</p> <p><b>Sample:</b></p> <p><code><b>*.* @@primary-syslog.example.com -<br>$ActionExecOnlyIfPreviousIsSuspended on +<br>$ActionExecOnlyWhenPreviousIsSuspended on <br>& @@secondary-1-syslog.example.com # & is used to have more than one action for <br>& @@secondary-2-syslog.example.com # the same selector - the mult-action feature <br>& /var/log/localbuffer -<br>$ActionExecOnlyIfPreviousIsSuspended off # to re-set it for the next selector </b></code></p> +<br>$ActionExecOnlyWhenPreviousIsSuspended off # to re-set it for the next selector </b></code></p> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> @@ -26,4 +26,4 @@ Copyright © 2007 by <a href="http://www.gerhards.net/rainer">Rainer Gerhard <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 2 or higher.</font></p> </body> -</html>
\ No newline at end of file +</html> diff --git a/doc/rsconf1_dirgroup.html b/doc/rsconf1_dirgroup.html index 868e5ecd..4575a7e8 100644 --- a/doc/rsconf1_dirgroup.html +++ b/doc/rsconf1_dirgroup.html @@ -7,7 +7,7 @@ <p><b>Type:</b> global configuration directive</p> <p><b>Default:</b> </p> <p><b>Description:</b></p> -<p>Set the group for directories newly created. Please note that this setting does not affect the group of directories already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd on startup and on HUPing. Interim changes to the user mapping are not detected.</p> +<p>Set the group for directories newly created. Please note that this setting does not affect the group of directories already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd on during startup processing. Interim changes to the user mapping are not detected.</p> <p><b>Sample:</b></p> <p><code><b>$DirGroup loggroup</b></code></p> @@ -19,4 +19,4 @@ Copyright © 2007 by <a href="http://www.gerhards.net/rainer">Rainer Gerhard <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 2 or higher.</font></p> </body> -</html>
\ No newline at end of file +</html> diff --git a/doc/rsconf1_dirowner.html b/doc/rsconf1_dirowner.html index e85a5122..34291856 100644 --- a/doc/rsconf1_dirowner.html +++ b/doc/rsconf1_dirowner.html @@ -7,7 +7,7 @@ <p><b>Type:</b> global configuration directive</p> <p><b>Default:</b> </p> <p><b>Description:</b></p> -<p>Set the file owner for directories newly created. Please note that this setting does not affect the owner of directories already existing. The parameter is a user name, for which the userid is obtained by rsyslogd on startup and on HUPing. Interim changes to the user mapping are not detected.</p> +<p>Set the file owner for directories newly created. Please note that this setting does not affect the owner of directories already existing. The parameter is a user name, for which the userid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.</p> <p><b>Sample:</b></p> <p><code><b>$DirOwner loguser</b></code></p> @@ -19,4 +19,4 @@ Copyright © 2007 by <a href="http://www.gerhards.net/rainer">Rainer Gerhard <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 2 or higher.</font></p> </body> -</html>
\ No newline at end of file +</html> diff --git a/doc/rsconf1_filecreatemode.html b/doc/rsconf1_filecreatemode.html index 7c6f1713..c8440864 100644 --- a/doc/rsconf1_filecreatemode.html +++ b/doc/rsconf1_filecreatemode.html @@ -30,6 +30,6 @@ index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> Copyright © 2007 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL -version 2 or higher.</font></p> +version 3 or higher.</font></p> </body> </html> diff --git a/doc/rsconf1_filegroup.html b/doc/rsconf1_filegroup.html index b9acaab7..92e4813b 100644 --- a/doc/rsconf1_filegroup.html +++ b/doc/rsconf1_filegroup.html @@ -7,7 +7,7 @@ <p><b>Type:</b> global configuration directive</p> <p><b>Default:</b> </p> <p><b>Description:</b></p> -<p>Set the group for dynaFiles newly created. Please note that this setting does not affect the group of files already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd on startup and on HUPing. Interim changes to the user mapping are not detected.</p> +<p>Set the group for dynaFiles newly created. Please note that this setting does not affect the group of files already existing. The parameter is a group name, for which the groupid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.</p> <p><b>Sample:</b></p> <p><code><b>$FileGroup loggroup</b></code></p> @@ -19,4 +19,4 @@ Copyright © 2007 by <a href="http://www.gerhards.net/rainer">Rainer Gerhard <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 2 or higher.</font></p> </body> -</html>
\ No newline at end of file +</html> diff --git a/doc/rsconf1_fileowner.html b/doc/rsconf1_fileowner.html index 7a9cbbc7..f8d5ebf3 100644 --- a/doc/rsconf1_fileowner.html +++ b/doc/rsconf1_fileowner.html @@ -7,7 +7,7 @@ <p><b>Type:</b> global configuration directive</p> <p><b>Default:</b> </p> <p><b>Description:</b></p> -<p>Set the file owner for dynaFiles newly created. Please note that this setting does not affect the owner of files already existing. The parameter is a user name, for which the userid is obtained by rsyslogd on startup and on HUPing. Interim changes to the user mapping are not detected.</p> +<p>Set the file owner for dynaFiles newly created. Please note that this setting does not affect the owner of files already existing. The parameter is a user name, for which the userid is obtained by rsyslogd during startup processing. Interim changes to the user mapping are not detected.</p> <p><b>Sample:</b></p> <p><code><b>$FileOwner loguser</b></code></p> @@ -19,4 +19,4 @@ Copyright © 2007 by <a href="http://www.gerhards.net/rainer">Rainer Gerhard <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 2 or higher.</font></p> </body> -</html>
\ No newline at end of file +</html> diff --git a/doc/rsconf1_markmessageperiod.html b/doc/rsconf1_markmessageperiod.html new file mode 100644 index 00000000..9b6590cd --- /dev/null +++ b/doc/rsconf1_markmessageperiod.html @@ -0,0 +1,30 @@ +<html> +<head> +<title>rsyslog.conf file</title> +</head> +<body> +<h2>$MarkMessagePeriod</h2> +<p><b>Type:</b> specific to immark input module</p> +<p><b>Default:</b> 1800 (20 minutes)</p> +<p><b>Description:</b></p> +<p>This specifies when mark messages are to be written to output modules. The +time specified is in seconds. Specifying 0 is possible and disables mark +messages. In that case, however, it is more efficient to NOT load the immark +input module.</p> +<p>So far, there is only one mark message process and any subsequent +$MarkMessagePeriod overwrites the previous.</p> +<p><b>This directive is only available after the immark input module has been +loaded.</b></p> +<p><b>Sample:</b></p> +<p><code><b>$MarkMessagePeriod 600 # mark messages appear every 10 Minutes</b></code></p> +<p><b>Available since:</b> rsyslog 3.0.0</p> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2007 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 2 or higher.</font></p> +</body> +</html>
\ No newline at end of file diff --git a/doc/rsconf1_modload.html b/doc/rsconf1_modload.html index 37623a1f..a2b8087a 100644 --- a/doc/rsconf1_modload.html +++ b/doc/rsconf1_modload.html @@ -18,7 +18,7 @@ $ModDir</a> directive.</p> <p>If a full path name is specified, the module is loaded from that path. The default module directory is ignored in that case.</p> <p><b>Sample:</b></p> -<p><code><b>$ModLoad MySQL # load MySQL functionality<br> +<p><code><b>$ModLoad ommysql # load MySQL functionality<br> $ModLoad /rsyslog/modules/ompgsql.so # load the postgres module via absolute path</b></code></p> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual diff --git a/doc/rscript_abnf.html b/doc/rscript_abnf.html new file mode 100644 index 00000000..278fb59c --- /dev/null +++ b/doc/rscript_abnf.html @@ -0,0 +1,41 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"><title>RainerScript ABNF</title></head> +<body> +<h1>RainerScript ABNF</h1> +<p>This is the formal definition of RainerScript, as supported by +rsyslog configuration. Please note that this currently is working +document and the actual implementation may be quite different.</p> +<p>The +first glimpse of RainerScript will be available as part of rsyslog +3.12.0 expression support. However, the 3.12. series of rsyslog will +have a partial script implementaiton, which will not necessariy be +compatible with the later full implementation. So if you use it, be +prepared for some config file changes as RainerScript evolves.</p> +<p>C-like comments (/* some comment */) are supported in all pure +RainerScript lines. However, legacy-mapped lines do not support them. +All lines support the hash mark "#" as a comment initiator. Everything +between the hash and the end of line is a comment (just like // in C++ +and many other languages).</p> +<h2>Formal Definition</h2> +<p>Below is the formal language definitionin ABNF (RFC 2234) +format: <br> +</p> +<pre>; <span style="font-weight: bold;">all of this is a working document and may change!</span> -- rgerhards, 2008-02-24<br><br>script := *stmt<br>stmt := (if_stmt / block / vardef / run_s / load_s)<br>vardef := "var" ["scope" = ("global" / "event")] <br>block := "begin" stmt "end"<br>load_s := "load" constraint ("module") modpath params ; load mod only if expr is true<br>run_s := "run" constraint ("input") name<br>constraint:= "if" expr ; constrains some one-time commands<br>modpath := expr<br>params := ["params" *1param *("," param) "endparams"]<br>param := paramname) "=" expr<br>paramname := [*(obqualifier ".") name]<br>modpath:= ; path to module<br>?line? := cfsysline / cfli<br>cfsysline:= BOL "$" *char EOL ; how to handle the first line? (no EOL in front!)<br>BOL := ; Begin of Line - implicitely set on file beginning and after each EOL<br>EOL := 0x0a ;LF<br>if_stmt := "if" expr "then"<br>old_filter:= BOL facility "." severity ; no whitespace allowed between BOL and facility!<br>facility := "*" / "auth" / "authpriv" / "cron" / "daemon" / "kern" / "lpr" / <br> "mail" / "mark" / "news" / "security" / "syslog" / "user" / "uucp" / <br> "local0" .. "local7" / "mark"<br> ; The keyword security should not be used anymore<br> ; mark is just internal<br>severity := TBD ; not really relevant in this context<br><br>; and now the actual expression<br>expr := e_and *("or" e_and)<br>e_and := e_cmp *("and" e_cmp)<br>e_cmp := val 0*1(cmp_op val)<br>val := term *(("+" / "-" / "&") term)<br>term := factor *(("*" / "/" / "%") factor)<br>factor := ["not"] ["-"] terminal<br>terminal := var / constant / function / ( "(" expr ")" )<br>function := name "(" *("," expr) ")"<br>var := "$" varname<br>varname := msgvar / sysvar<br>msgvar := name<br>sysvar := "$" name<br>name := alpha *(alnum)<br>constant := string / number<br>string := simpstr / tplstr ; tplstr will be implemented in next phase<br>simpstr := "'" *char "'" ; use your imagination for char ;)<br>tplstr := '"' template '"' ; not initially implemented<br>number := ["-"] 1*digit ; 0nn = octal, 0xnn = hex, nn = decimal<br>cmp_op := "==" / "!=" / "<>" / "<" / ">" / "<=" / ">=" / "contains" / "contains_i" / "startswith" / "startswith_i"<br>digit := %x30-39<br>alpha := "a" ... "z" # all letters<br>alnum :* alpha / digit / "_" /"-" # "-" necessary to cover currently-existing message properties<br></pre> +<h2>Samples</h2> +<p>Some samples of RainerScript:</p><p>define function IsLinux<br>begin<br> if $environ contains "linux" then return true else return false<br>end</p><p>load if IsLinux() 'imklog.so' params name='klog' endparams /* load klog under linux only */<br>run if IsLinux() input 'klog'<br>load 'ommysql.so'</p><p>if $message contains "error" then<br> action<br> type='ommysql.so', queue.mode='disk', queue.highwatermark = 300,<br> action.dbname='events', action.dbuser='uid',<br> + [?action.template='templatename'?] or [?action.sql='insert into +table... values('&$facility&','&$severity&...?]<br> endaction<br><br>... or ...</p><p>define action writeMySQL<br> type='ommysql.so', queue.mode='disk', queue.highwatermark = 300,<br> action.dbname='events', action.dbuser='uid',<br> [?action.template='templatename'?] or [?action.sql='insert into table... values('<span style="font-family: monospace;"> &</span> $facility & ',' & $severity &...?]<br> endaction</p><p>if $message contains "error" then action writeMySQL</p><p>ALTERNATE APPROACH</p><p>define function IsLinux(<br> if $environ contains "linux" then return true else return false<br>)</p><p>load if IsLinux() 'imklog.so' params name='klog' endparams /* load klog under linux only */<br>run if IsLinux() input 'klog'<br>load 'ommysql.so'</p><p>if $message contains "error" then<br> action(<br> type='ommysql.so', queue.mode='disk', queue.highwatermark = 300,<br> action.dbname='events', action.dbuser='uid',<br> + [?action.template='templatename'?] or [?action.sql='insert into +table... values('&$facility&','&$severity&...?]<br> )<br><br>... or ...</p><p>define action writeMySQL(<br> type='ommysql.so', queue.mode='disk', queue.highwatermark = 300,<br> action.dbname='events', action.dbuser='uid',<br> + [?action.template='templatename'?] or [?action.sql='insert into +table... values('&$facility&','&$severity&...?]<br> )</p><p>if $message contains "error" then action writeMySQL(action.dbname='differentDB')</p><p></p><p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/rsyslog-example.conf b/doc/rsyslog-example.conf index 495bc566..a3ec2f1f 100644 --- a/doc/rsyslog-example.conf +++ b/doc/rsyslog-example.conf @@ -34,7 +34,7 @@ $IncludeConfig /etc/rsyslog.d/ # whole directory (must contain t # $ModLoad - Dynamically loads a plug-in and activates it # -------- -$ModLoad MySQL # load MySQL functionality +$ModLoad ommysql # load MySQL functionality $ModLoad /rsyslog/modules/somemodule.so # load a module via absolute path diff --git a/doc/rsyslog_conf.html b/doc/rsyslog_conf.html index 25298d33..2dc8aa24 100644 --- a/doc/rsyslog_conf.html +++ b/doc/rsyslog_conf.html @@ -1,540 +1,925 @@ -<html> -<head> -<title>rsyslog.conf file</title> -</head> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>rsyslog.conf file</title></head> <body> <h1>rsyslog.conf configuration file</h1> -<p><b>This document is currently being enhanced. Please pardon its current -appearance.</b></p> -<p><b>Rsyslogd is configured via the rsyslog.conf file</b>, typically found in -/etc. By default, rsyslogd reads the file /etc/rsyslog.conf. This may be changed -by a command line option.</p> +<p><b>This document is currently being enhanced. Please +pardon its current appearance.</b></p> +<p><b>Rsyslogd is configured via the rsyslog.conf file</b>, +typically found in /etc. By default, rsyslogd reads the file +/etc/rsyslog.conf. This may be changed by a command line option.</p> <p><a href="http://wiki.rsyslog.com/index.php/Configuration_Samples"> Configuration file examples can be found in the rsyslog wiki</a>.</p> -<p>There is also one sample file provided together with the documentation set. -If you do not like to read, be sure to have at least a quick look at -<a href="rsyslog-example.conf">rsyslog-example.conf</a>. </p> -<p>While rsyslogd contains enhancements over standard syslogd, efforts have been -made to keep the configuration file as compatible as possible. While, for -obvious reasons, <a href="features.html">enhanced features</a> require a -different config file syntax, rsyslogd should be able to work with a standard -syslog.conf file. This is especially useful while you are migrating from syslogd -to rsyslogd.</p> +<p>There is also one sample file provided together with the +documentation set. If you do not like to read, be sure to have at least +a quick look at +<a href="rsyslog-example.conf">rsyslog-example.conf</a>. +</p> +<p>While rsyslogd contains enhancements over standard syslogd, +efforts have been made to keep the configuration file as compatible as +possible. While, for obvious reasons, <a href="features.html">enhanced +features</a> require a different config file syntax, rsyslogd +should be able to work with a standard syslog.conf file. This is +especially useful while you are migrating from syslogd to rsyslogd.</p> +<h2>Modules</h2> +<p>Rsyslog has a modular design. Consequently, there is a growing +number of modules. Here is the entry point to their documentation and +what they do (list is currently not complete)</p> +<ul> +<li><a href="omsnmp.html">omsnmp</a> - SNMP +trap output module</li> +<li><a href="omrelp.html">omrelp</a> - RELP +output module</li> +<li>omgssapi - output module for GSS-enabled syslog</li> +<li><a href="ommysql.html">ommysql</a> - output module for MySQL</li> +<li>ompgsql - output module for PostgreSQL</li> +<li><a href="omlibdbi.html">omlibdbi</a> - +generic database output module (Firebird/Interbase, MS SQL, Sybase, +SQLLite, Ingres, Oracle, mSQL)</li> +<li><a href="ommail.html">ommail</a> - +permits rsyslog to alert folks by mail if something important happens</li> +<li><a href="imfile.html">imfile</a> +- input module for text files</li> +<li><a href="imrelp.html">imrelp</a> - RELP +input module</li> +<li>imudp - udp syslog message input</li> +<li><a href="imtcp.html">imtcp</a> - input +plugin for plain tcp syslog</li> +<li><a href="imgssapi.html">imgssapi</a> - +input plugin for plain tcp and GSS-enabled syslog</li> +<li>immark - support for mark messages</li> +<li><a href="imklog.html">imklog</a> - kernel logging</li> +<li><a href="imuxsock.html">imuxsock</a> - +unix sockets, including the system log socket</li> +<li><a href="im3195.html">im3195</a> - +accepts syslog messages via RFC 3195</li> +</ul> +<p>Please note that each module provides configuration +directives, which are NOT necessarily being listed below. Also +remember, that a modules configuration directive (and functionality) is +only available if it has been loaded (using $ModLoad).</p> +<h2>Lines</h2> +Lines can be continued by specifying a backslash ("\") as the last +character of the line.<br> <h2>Global Directives</h2> -<p>All global directives need to be specified on a line by their own and must -start with a dollar-sign. Here is a list in alphabetical order. Follow links for -a description.</p> +<p>All global directives need to be specified on a line by their +own and must start with a dollar-sign. Here is a list in alphabetical +order. Follow links for a description.</p> +<p>Not all directives have an in-depth description right now. +Default values for them are in bold. A more in-depth description will +appear as implementation progresses. Directives may change during that +process, we will NOT try hard to maintain backwards compatibility +(after all, v3 is still very early in development and quite +unstable...). So you have been warned ;)</p> +<p><b>Be sure to read information about <a href="queues.html">queues in rsyslog</a></b> - +many parameter settings modify queue parameters. If in doubt, use the +default, it is usually well-chosen and applicable in most cases.</p> <ul> - <li><a href="rsconf1_actionexeconlywhenpreviousissuspended.html">$ActionExecOnlyWhenPreviousIsSuspended</a></li> - <li><a href="rsconf1_actionresumeinterval.html">$ActionResumeInterval</a></li> - <li><a href="rsconf1_allowedsender.html">$AllowedSender</a></li> - <li><a href="rsconf1_controlcharacterescapeprefix.html">$ControlCharacterEscapePrefix</a></li> - <li><a href="rsconf1_debugprintcfsyslinehandlerlist.html">$DebugPrintCFSyslineHandlerList</a></li> - <li><a href="rsconf1_debugprintmodulelist.html">$DebugPrintModuleList</a></li> - <li><a href="rsconf1_debugprinttemplatelist.html">$DebugPrintTemplateList</a></li> - <li><a href="rsconf1_dircreatemode.html">$DirCreateMode</a></li> - <li><a href="rsconf1_dirgroup.html">$DirGroup</a></li> - <li><a href="rsconf1_dirowner.html">$DirOwner</a></li> - <li><a href="rsconf1_dropmsgswithmaliciousdnsptrrecords.html">$DropMsgsWithMaliciousDnsPTRRecords</a></li> - <li><a href="rsconf1_droptrailinglfonreception.html">$DropTrailingLFOnReception</a></li> - <li><a href="rsconf1_dynafilecachesize.html">$DynaFileCacheSize</a></li> - <li><a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a></li> - <li><a href="rsconf1_failonchownfailure.html">$FailOnChownFailure</a></li> - <li><a href="rsconf1_filecreatemode.html">$FileCreateMode</a></li> - <li><a href="rsconf1_filegroup.html">$FileGroup</a></li> - <li><a href="rsconf1_fileowner.html">$FileOwner</a></li> - <li><a href="rsconf1_gssforwardservicename.html">$GssForwardServiceName</a></li> - <li><a href="rsconf1_gsslistenservicename.html">$GssListenServiceName</a></li> - <li><a href="rsconf1_gssmode.html">$GssMode</a></li> - <li><a href="rsconf1_includeconfig.html">$IncludeConfig</a></li> - <li><a href="rsconf1_mainmsgqueuesize.html">$MainMsgQueueSize</a></li> - <li><a href="rsconf1_moddir.html">$ModDir</a></li> - <li><a href="rsconf1_modload.html">$ModLoad</a></li> - <li><a href="rsconf1_repeatedmsgreduction.html">$RepeatedMsgReduction</a></li> - <li><a href="rsconf1_resetconfigvariables.html">$ResetConfigVariables</a></li> - <li><a href="rsconf1_umask.html">$UMASK</a></li> +<li><a href="rsconf1_actionexeconlywhenpreviousissuspended.html">$ActionExecOnlyWhenPreviousIsSuspended</a></li> +<li>$ActionExecOnlyOnceEveryInterval <seconds> - +execute action only if the last execute is at last +<seconds> seconds in the past (more info in <a href="ommail.html">ommail</a>, +but may be used with any action)</li> +<li><i><b>$ActionExecOnlyEveryNthTime</b> <number></i> - If configured, the next action will +only be executed every n-th time. For example, if configured to 3, the first two messages +that go into the action will be dropped, the 3rd will actually cause the action to execute, +the 4th and 5th will be dropped, the 6th executed under the action, ... and so on. Note: +this setting is automatically re-set when the actual action is defined.</li> +<li><i><b>$ActionExecOnlyEveryNthTimeTimeout</b> <number-of-seconds></i> - has a meaning only if +$ActionExecOnlyEveryNthTime is also configured for the same action. If so, the timeout +setting specifies after which period the counting of "previous actions" expires and +a new action count is begun. Specify 0 (the default) to disable timeouts. +<br> +<i>Why is this option needed?</i> Consider this case: a message comes in at, eg., 10am. That's +count 1. Then, nothing happens for the next 10 hours. At 8pm, the next +one occurs. That's count 2. Another 5 hours later, the next message +occurs, bringing the total count to 3. Thus, this message now triggers +the rule. +<br> +The question is if this is desired behavior? Or should the rule only be +triggered if the messages occur within an e.g. 20 minute window? If the +later is the case, you need a +<br> +$ActionExecOnlyEveryNthTimeTimeout 1200 +<br> +This directive will timeout previous messages seen if they are older +than 20 minutes. In the example above, the count would now be always 1 +and consequently no rule would ever be triggered. + +<li>$ActionFileDefaultTemplate [templateName] - sets a new default template for file actions</li> +<li>$ActionFileEnableSync [on/<span style="font-weight: bold;">off</span>] - enables file +syncing capability of omfile</li> +<li>$ActionForwardDefaultTemplate [templateName] - sets a new +default template for UDP and plain TCP forwarding action</li> +<li>$ActionGSSForwardDefaultTemplate [templateName] - sets a +new default template for GSS-API forwarding action</li> +<li>$ActionQueueCheckpointInterval <number></li> +<li>$ActionQueueDequeueSlowdown <number> [number +is timeout in <i> micro</i>seconds (1000000us is 1sec!), +default 0 (no delay). Simple rate-limiting!]</li> +<li>$ActionQueueDiscardMark <number> [default +9750]</li> +<li>$ActionQueueDiscardSeverity <number> +[*numerical* severity! default 4 (warning)]</li> +<li>$ActionQueueFileName <name></li> +<li>$ActionQueueHighWaterMark <number> [default +8000]</li> +<li>$ActionQueueImmediateShutdown [on/<b>off</b>]</li> +<li>$ActionQueueSize <number></li> +<li>$ActionQueueLowWaterMark <number> [default +2000]</li> +<li>$ActionQueueMaxFileSize <size_nbr>, default 1m</li> +<li>$ActionQueueTimeoutActionCompletion <number> +[number is timeout in ms (1000ms is 1sec!), default 1000, 0 means +immediate!]</li> +<li>$ActionQueueTimeoutEnqueue <number> [number +is timeout in ms (1000ms is 1sec!), default 2000, 0 means indefinite]</li> +<li>$ActionQueueTimeoutShutdown <number> [number +is timeout in ms (1000ms is 1sec!), default 0 (indefinite)]</li> +<li>$ActionQueueWorkerTimeoutThreadShutdown +<number> [number is timeout in ms (1000ms is 1sec!), +default 60000 (1 minute)]</li> +<li>$ActionQueueType [FixedArray/LinkedList/<b>Direct</b>/Disk]</li> +<li>$ActionQueueSaveOnShutdown [on/<b>off</b>] +</li> +<li>$ActionQueueWorkerThreads <number>, num worker threads, default 1, recommended 1</li> +<li>$ActionQueueWorkerThreadMinumumMessages <number>, default 100</li> +<li><a href="rsconf1_actionresumeinterval.html">$ActionResumeInterval</a></li> +<li>$ActionResumeRetryCount <number> [default 0, -1 means eternal]</li> +<li>$ActionSendResendLastMsgOnReconn <[on/<b>off</b>]> specifies if the last message is to be resend when a connecition broken and has been reconnedcted. May increase reliability, but comes at the risk of message duplication. +<li>$ActionSendStreamDriver <driver basename> just like $DefaultNetstreamDriver, but for the specific action. +</li><li>$ActionSendStreamDriverMode <mode>, default 0, mode to use with the stream driver +(driver-specific)</li><li>$ActionSendStreamDriverAuthMode <mode>, authentication mode to use with the stream driver. Note that this directive requires TLS +netstream drivers. For all others, it will be ignored. +(driver-specific)</li><li>$ActionSendStreamDriverPermittedPeer <ID>, accepted fingerprint (SHA1) or name of remote peer. Note that this directive requires TLS +netstream drivers. For all others, it will be ignored. +(driver-specific) -<span style="font-weight: bold;"> directive may go away</span>!</li> +<li><a href="rsconf1_allowedsender.html">$AllowedSender</a></li> +<li><a href="rsconf1_controlcharacterescapeprefix.html">$ControlCharacterEscapePrefix</a></li> +<li><a href="rsconf1_debugprintcfsyslinehandlerlist.html">$DebugPrintCFSyslineHandlerList</a></li> + +<li><a href="rsconf1_debugprintmodulelist.html">$DebugPrintModuleList</a></li> +<li><a href="rsconf1_debugprinttemplatelist.html">$DebugPrintTemplateList</a></li> +<li>$DefaultNetstreamDriver <drivername>, the default <a href="netstream.html">network stream driver</a> to use. Defaults to ptcp.$DefaultNetstreamDriverCAFile </path/to/cafile.pem></li> +<li>$DefaultNetstreamDriverCertFile </path/to/certfile.pem></li> +<li>$DefaultNetstreamDriverKeyFile </path/to/keyfile.pem></li> +<li><a href="rsconf1_dircreatemode.html">$DirCreateMode</a></li> +<li><a href="rsconf1_dirgroup.html">$DirGroup</a></li> +<li><a href="rsconf1_dirowner.html">$DirOwner</a></li> +<li><a href="rsconf1_dropmsgswithmaliciousdnsptrrecords.html">$DropMsgsWithMaliciousDnsPTRRecords</a></li> +<li><a href="rsconf1_droptrailinglfonreception.html">$DropTrailingLFOnReception</a></li> +<li><a href="rsconf1_dynafilecachesize.html">$DynaFileCacheSize</a></li> +<li><a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a></li> +<li>$ErrorMessagesToStderr [<b>on</b>|off] - direct rsyslogd error message to stderr (in addition to other targets)</li> +<li><a href="rsconf1_failonchownfailure.html">$FailOnChownFailure</a></li> +<li><a href="rsconf1_filecreatemode.html">$FileCreateMode</a></li> +<li><a href="rsconf1_filegroup.html">$FileGroup</a></li> +<li><a href="rsconf1_fileowner.html">$FileOwner</a></li> +<li><a href="rsconf1_gssforwardservicename.html">$GssForwardServiceName</a></li> +<li><a href="rsconf1_gsslistenservicename.html">$GssListenServiceName</a></li> +<li><a href="rsconf1_gssmode.html">$GssMode</a></li> +<li><a href="rsconf1_includeconfig.html">$IncludeConfig</a></li><li>MainMsgQueueCheckpointInterval <number></li> +<li>$MainMsgQueueDequeueSlowdown <number> [number +is timeout in <i> micro</i>seconds (1000000us is 1sec!), +default 0 (no delay). Simple rate-limiting!]</li> +<li>$MainMsgQueueDiscardMark <number> [default +9750]</li> +<li>$MainMsgQueueDiscardSeverity <severity> +[either a textual or numerical severity! default 4 (warning)]</li> +<li>$MainMsgQueueFileName <name></li> +<li>$MainMsgQueueHighWaterMark <number> [default +8000]</li> +<li>$MainMsgQueueImmediateShutdown [on/<b>off</b>]</li> +<li><a href="rsconf1_mainmsgqueuesize.html">$MainMsgQueueSize</a></li> +<li>$MainMsgQueueLowWaterMark <number> [default +2000]</li> +<li>$MainMsgQueueMaxFileSize <size_nbr>, default +1m</li> +<li>$MainMsgQueueTimeoutActionCompletion +<number> [number is timeout in ms (1000ms is 1sec!), +default +1000, 0 means immediate!]</li> +<li>$MainMsgQueueTimeoutEnqueue <number> [number +is timeout in ms (1000ms is 1sec!), default 2000, 0 means indefinite]</li> +<li>$MainMsgQueueTimeoutShutdown <number> [number +is timeout in ms (1000ms is 1sec!), default 0 (indefinite)]</li> +<li>$MainMsgQueueWorkerTimeoutThreadShutdown +<number> [number is timeout in ms (1000ms is 1sec!), +default 60000 (1 minute)]</li> +<li>$MainMsgQueueType [<b>FixedArray</b>/LinkedList/Direct/Disk]</li> +<li>$MainMsgQueueSaveOnShutdown [on/<b>off</b>] +</li> +<li>$MainMsgQueueWorkerThreads <number>, num +worker threads, default 1, recommended 1</li> +<li>$MainMsgQueueWorkerThreadMinumumMessages <number>, default 100</li> +<li><a href="rsconf1_markmessageperiod.html">$MarkMessagePeriod</a> (immark)</li> +<li><b><i>$MaxMessageSize</i></b> <size_nbr>, default 2k - allows to specify maximum supported message size +(both for sending and receiving). The default +should be sufficient for almost all cases. Do not set this below 1k, as it would cause +interoperability problems with other syslog implementations.<br> +Change the setting to e.g. 32768 if you would like to +support large message sizes for IHE (32k is the current maximum +needed for IHE). I was initially tempted to set the default to 32k, +but there is a some memory footprint with the current +implementation in rsyslog. +<br>If you intend to receive Windows Event Log data (e.g. via +<a href="http://www.eventreporter.com/">EventReporter</a>), you might want to +increase this number to an even higher value, as event +log messages can be very lengthy ("$MaxMessageSize 64k" is not a bad idea). +Note: testing showed that 4k seems to be +the typical maximum for <b>UDP</b> based syslog. This is an IP stack +restriction. Not always ... but very often. If you go beyond +that value, be sure to test that rsyslogd actually does what +you think it should do ;) It is highly suggested to use a TCP based transport +instead of UDP (plain TCP syslog, RELP). This resolves the UDP stack size restrictions. +<br>Note that 2k, the current default, is the smallest size that must be +supported in order to be compliant to the upcoming new syslog RFC series. +</li> +<li><a href="rsconf1_moddir.html">$ModDir</a></li> +<li><a href="rsconf1_modload.html">$ModLoad</a></li> +<li><a href="rsconf1_repeatedmsgreduction.html">$RepeatedMsgReduction</a></li> +<li><a href="rsconf1_resetconfigvariables.html">$ResetConfigVariables</a></li> +<li>$WorkDirectory <name> (directory for spool +and other work files)</li> +<li>$UDPServerAddress <IP> (imudp) -- local IP +address (or name) the UDP listens should bind to</li> +<li>$UDPServerRun <port> (imudp) -- former +-r<port> option, default 514, start UDP server on this +port, "*" means all addresses</li> +<li><a href="rsconf1_umask.html">$UMASK</a></li> </ul> +<p><b>Where <size_nbr> is specified above,</b> +modifiers can be used after the number part. For example, 1k means +1024. Supported are k(ilo), m(ega), g(iga), t(era), p(eta) and e(xa). +Lower case letters refer to the traditional binary defintion (e.g. 1m +equals 1,048,576) whereas upper case letters refer to their new +1000-based definition (e.g 1M equals 1,000,000).</p> +<p>Numbers may include '.' and ',' for readability. So you can +for example specify either "1000" or "1,000" with the same result. +Please note that rsyslogd simply ignores the punctuation. Form it's +point of view, "1,,0.0.,.,0" also has the value 1000. </p> <h2>Basic Structure</h2> -<p>Rsyslog supports standard sysklogd's configuration file format and extends -it. So in general, you can take a "normal" syslog.conf and use it together with -rsyslogd. It will understand everything. However, to use most of rsyslogd's -unique features, you need to add extended configuration directives.<p>Rsyslogd -supports the classical, selector-based rule lines. They are still at the heart -of it and all actions are initiated via rule lines. A rule lines is any line not -starting with a $ or the comment sign (#). Lines starting with $ carry -rsyslog-specific directives.<p>Every rule line consists of two fields, a selector field and an action field. -These two fields are separated by one or more spaces or tabs. The selector field -specifies a pattern of facilities and priorities belonging to the specified -action.<br> -<br> -Lines starting with a hash mark ("#'') and empty lines are ignored. - +<p>Rsyslog supports standard sysklogd's configuration file format +and extends it. So in general, you can take a "normal" syslog.conf and +use it together with rsyslogd. It will understand everything. However, +to use most of rsyslogd's unique features, you need to add extended +configuration directives.</p> +<p>Rsyslogd supports the classical, selector-based rule lines. +They are still at the heart of it and all actions are initiated via +rule lines. A rule lines is any line not starting with a $ or the +comment sign (#). Lines starting with $ carry rsyslog-specific +directives.</p> +<p>Every rule line consists of two fields, a selector field and +an action field. These two fields are separated by one or more spaces +or tabs. The selector field specifies a pattern of facilities and +priorities belonging to the specified action.<br> +<br> +Lines starting with a hash mark ("#'') and empty lines are ignored. +</p> <h2>Templates</h2> -<p>Templates are a key feature of rsyslog. They allow to specify any format a user -might want. They are also used for dynamic file name generation. Every output in rsyslog uses templates - this holds true for files, -user messages and so on. The database writer expects its template to be a proper -SQL statement - so this is highly customizable too. You might ask how does all -of this work when no templates at all are specified. Good question ;) The answer -is simple, though. Templates compatible with the stock syslogd formats are -hardcoded into rsyslogd. So if no template is specified, we use one of these -hardcoded templates. Search for "template_" in syslogd.c and you will find the +<p>Templates are a key feature of rsyslog. They allow to specify +any +format a user might want. They are also used for dynamic file name +generation. Every output in rsyslog uses templates - this holds true +for files, user messages and so on. +Please note that there is an +<a href="http://www.rsyslog.com/Article354.phtml">online tutorial on rsyslog templates</a> +available on the web. We recommend viewing it. +The database writer expects its +template to be a proper SQL statement - so this is highly customizable +too. You might ask how does all of this work when no templates at all +are specified. Good question ;) The answer is simple, though. Templates +compatible with the stock syslogd formats are hardcoded into rsyslogd. +So if no template is specified, we use one of these hardcoded +templates. Search for "template_" in syslogd.c and you will find the hardcoded ones.</p> -<p>A template consists of a template directive, a name, the actual template text -and optional options. A sample is:</p> -<blockquote><code>$template MyTemplateName,"\7Text %property% some more text\n",<options></code></blockquote> -<p>The "$template" is the template directive. It tells rsyslog that this line -contains a template. "MyTemplateName" is the template name. All -other config lines refer to this name. The text within quotes is the actual -template text. The backslash is an escape character, much as it is in C. It does -all these "cool" things. For example, \7 rings the bell (this is an ASCII -value), \n is a new line. C programmers and perl coders have the advantage of -knowing this, but the set in rsyslog is a bit restricted currently. -<p> -All text in the template is used literally, except for things within percent -signs. These are properties and allow you access to the contents of the syslog -message. Properties are accessed via the property replacer (nice name, huh) and -it can do cool things, too. For example, it can pick a substring or do -date-specific formatting. More on this is below, on some lines of the property -replacer.<br> -<br> -The <options> part is optional. It carries options influencing the template as -whole. See details below. Be sure NOT to mistake template options with property -options - the later ones are processed by the property replacer and apply to a -SINGLE property, only (and not the whole template).<br> +<p>A template consists of a template directive, a name, the +actual template text and optional options. A sample is:</p> +<blockquote><code>$template MyTemplateName,"\7Text +%property% some more text\n",<options></code></blockquote> +<p>The "$template" is the template directive. It tells rsyslog +that this line contains a template. "MyTemplateName" is the template +name. All +other config lines refer to this name. The text within quotes is the +actual template text. The backslash is an escape character, much as it +is in C. It does all these "cool" things. For example, \7 rings the +bell (this is an ASCII value), \n is a new line. C programmers and perl +coders have the advantage of knowing this, but the set in rsyslog is a +bit restricted currently. +</p> +<p>All text in the template is used literally, except for things +within percent signs. These are properties and allow you access to the +contents of the syslog message. Properties are accessed via the +property replacer (nice name, huh) and it can do cool things, too. For +example, it can pick a substring or do date-specific formatting. More +on this is below, on some lines of the property replacer.<br> +<br> +The <options> part is optional. It carries options +influencing the template as whole. See details below. Be sure NOT to +mistake template options with property options - the later ones are +processed by the property replacer and apply to a SINGLE property, only +(and not the whole template).<br> <br> Template options are case-insensitive. Currently defined are: </p> -<p><b>sql</b> - format the string suitable for a SQL statement in MySQL format. This will -replace single quotes ("'") and the backslash character by their -backslash-escaped counterpart ("\'" and "\\") inside each field. Please note -that in MySQL configuration, the <code class="literal">NO_BACKSLASH_ESCAPES</code> +<p><b>sql</b> - format the string suitable for a SQL +statement in MySQL format. This will replace single quotes ("'") and +the backslash character by their backslash-escaped counterpart ("\'" +and "\\") inside each field. Please note that in MySQL configuration, +the <code class="literal">NO_BACKSLASH_ESCAPES</code> mode must be turned off for this format to work (this is the default).</p> -<p><b>stdsql</b> - format the string suitable for a SQL statement that is to be -sent to a standards-compliant sql server. This will -replace single quotes ("'") by two single quotes ("''") inside each field. -You must use stdsql together with MySQL if in MySQL configuration the -<code class="literal">NO_BACKSLASH_ESCAPES</code> is turned on.</p> -<p>Either the <b>sql</b> or <b>stdsql</b> -option <b>must</b> be specified when a template is used for writing to a database, -otherwise injection might occur. Please note that due to the unfortunate fact -that several vendors have violated the sql standard and introduced their own -escape methods, it is impossible to have a single option doing all the work. -So you yourself must make sure you are using the right format. <b>If you choose +<p><b>stdsql</b> - format the string suitable for a +SQL statement that is to be sent to a standards-compliant sql server. +This will replace single quotes ("'") by two single quotes ("''") +inside each field. You must use stdsql together with MySQL if in MySQL +configuration the +<code class="literal">NO_BACKSLASH_ESCAPES</code> is +turned on.</p> +<p>Either the <b>sql</b> or <b>stdsql</b> +option <b>must</b> be specified when a template is used +for writing to a database, otherwise injection might occur. Please note +that due to the unfortunate fact that several vendors have violated the +sql standard and introduced their own escape methods, it is impossible +to have a single option doing all the work. So you yourself +must make sure you are using the right format. <b>If you choose the wrong one, you are still vulnerable to sql injection.</b><br> <br> -Please note that the database writer *checks* that the sql option is present in -the template. If it is not present, the write database action is disabled. This -is to guard you against accidental forgetting it and then becoming vulnerable -to SQL injection. The sql option can also be useful with files - especially if -you want to import them into a database on another machine for performance -reasons. However, do NOT use it if you do not have a real need for it - among -others, it takes some toll on the processing time. Not much, but on a really -busy system you might notice it ;)</p> -<p>The default template for the write to database action has the sql option set. -As we currently support only MySQL and the sql option matches the default MySQL -configuration, this is a good choice. However, if you have turned on -<code class="literal">NO_BACKSLASH_ESCAPES</code> in your MySQL config, you need -to supply a template with the stdsql option. Otherwise you will become -vulnerable to SQL injection. <br> +Please note that the database writer *checks* that the sql option is +present in the template. If it is not present, the write database +action is disabled. This is to guard you against accidental forgetting +it and then becoming vulnerable to SQL injection. The sql option can +also be useful with files - especially if you want to import them into +a database on another machine for performance reasons. However, do NOT +use it if you do not have a real need for it - among others, it takes +some toll on the processing time. Not much, but on a really busy system +you might notice it ;)</p> +<p>The default template for the write to database action has the +sql option set. As we currently support only MySQL and the sql option +matches the default MySQL configuration, this is a good choice. +However, if you have turned on +<code class="literal">NO_BACKSLASH_ESCAPES</code> in +your MySQL config, you need to supply a template with the stdsql +option. Otherwise you will become vulnerable to SQL injection. <br> <br> To escape:<br> % = \%<br> \ = \\ --> '\' is used to escape (as in C)<br> -$template TraditionalFormat,%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"<br> -<br> -Properties can be accessed by the <a href="property_replacer.html">property replacer</a> -(see there for details).</p> -<p><b>Please note that as of 1.15.0, templates can also by used to generate -selector lines with dynamic file names.</b> For example, if you would like to -split syslog messages from different hosts to different files (one per host), -you can define the following template:</p> -<blockquote><code>$template DynFile,"/var/log/system-%HOSTNAME%.log"</code></blockquote> -<p>This template can then be used when defining an output selector line. It will -result in something like "/var/log/system-localhost.log"</p> +$template TraditionalFormat,%timegenerated% %HOSTNAME% +%syslogtag%%msg%\n"<br> +<br> +Properties can be accessed by the <a href="property_replacer.html">property +replacer</a> (see there for details).</p> +<p><b>Please note that templates can also by +used to generate selector lines with dynamic file names.</b> For +example, if you would like to split syslog messages from different +hosts to different files (one per host), you can define the following +template:</p> +<blockquote><code>$template +DynFile,"/var/log/system-%HOSTNAME%.log"</code></blockquote> +<p>This template can then be used when defining an output +selector line. It will result in something like +"/var/log/system-localhost.log"</p> +<p>Template +names beginning with "RSYSLOG_" are reserved for rsyslog use. Do NOT +use them if, otherwise you may receive a conflict in the future (and +quite unpredictable behaviour). There is a small set of pre-defined +templates that you can use without the need to define it:</p> +<ul> +<li><span style="font-weight: bold;">RSYSLOG_TraditionalFileFormat</span> +- the "old style" default log file format with low-precision timestamps</li> +<li><span style="font-weight: bold;">RSYSLOG_FileFormat</span> +- a modern-style logfile format similar to TraditionalFileFormat, buth +with high-precision timestamps and timezone information</li> +<li><span style="font-weight: bold;">RSYSLOG_TraditionalForwardFormat</span> +- the traditional forwarding format with low-precision timestamps. Most +useful if you send messages to other syslogd's or rsyslogd +below +version 3.12.5.</li> +<li><span style="font-weight: bold;">RSYSLOG_ForwardFormat</span> +- a new high-precision forwarding format very similar to the +traditional one, but with high-precision timestamps and timezone +information. Recommended to be used when sending messages to rsyslog +3.12.5 or above.</li> +<li><span style="font-weight: bold;">RSYSLOG_SyslogProtocol23Format</span> +- the format specified in IETF's internet-draft +ietf-syslog-protocol-23, which is assumed to be come the new syslog +standard RFC. This format includes several improvements. The rsyslog +message parser understands this format, so you can use it together with +all relatively recent versions of rsyslog. Other syslogd's may get +hopelessly confused if receiving that format, so check before you use +it. Note that the format is unlikely to change when the final RFC comes +out, but this may happen.</li> +<li><span style="font-weight: bold;">RSYSLOG_DebugFormat</span> +- a special format used for troubleshooting property problems. This format +is meant to be written to a log file. Do <b>not</b> use for production or remote +forwarding.</li> +</ul> <h2>Output Channels</h2> -<p>Output Channels are a new concept first introduced in rsyslog 0.9.0. <b>As of this -writing, it is most likely that they will be replaced by something different in -the future.</b> So if you -use them, be prepared to change you configuration file syntax when you upgrade -to a later release.<br> -<br> -The idea behind output channel definitions is that it shall provide an umbrella -for any type of output that the user might want. In essence,<br> -this is the "file" part of selector lines (and this is why we are not sure -output channel syntax will stay after the next review). There is a<br> -difference, though: selector channels both have filter conditions (currently -facility and severity) as well as the output destination. Output channels define -the output definition, only. As of this build, they can only be used to write to -files - not pipes, ttys or whatever else. If we stick with output channels, this -will change over time.</p> -<p>In concept, an output channel includes everything needed to know about an -output actions. In practice, the current implementation only carries<br> -a filename, a maximum file size and a command to be issued when this file size -is reached. More things might be present in future version, which might also -change the syntax of the directive.</p> -<p>Output channels are defined via an $outchannel directive. It's syntax is as -follows:<br> +<p>Output Channels are a new concept first introduced in rsyslog +0.9.0. <b>As of this writing, it is most likely that they will +be replaced by something different in the future.</b> So if you +use them, be prepared to change you configuration file syntax when you +upgrade to a later release.<br> +<br> +The idea behind output channel definitions is that it shall provide an +umbrella for any type of output that the user might want. In essence,<br> +this is the "file" part of selector lines (and this is why we are not +sure output channel syntax will stay after the next review). There is a<br> +difference, though: selector channels both have filter conditions +(currently facility and severity) as well as the output destination. +Output channels define the output definition, only. As of this build, +they can only be used to write to files - not pipes, ttys or whatever +else. If we stick with output channels, this will change over time.</p> +<p>In concept, an output channel includes everything needed to +know about an output actions. In practice, the current implementation +only carries<br> +a filename, a maximum file size and a command to be issued when this +file size is reached. More things might be present in future version, +which might also change the syntax of the directive.</p> +<p>Output channels are defined via an $outchannel directive. It's +syntax is as follows:<br> <br> $outchannel name,file-name,max-size,action-on-max-size<br> <br> -name is the name of the output channel (not the file), file-name is the file -name to be written to, max-size the maximum allowed size and action-on-max-size -a command to be issued when the max size is reached. This command always has -exactly one parameter. The binary is that part of action-on-max-size before the -first space, its parameter is everything behind that space.<br> -<br> -Please note that max-size is queried BEFORE writing the log message to the file. -So be sure to set this limit reasonably low so that any message might fit. For -the current release, setting it 1k lower than you expected is helpful. The -max-size must always be specified in bytes - there are no special symbols (like -1k, 1m,...) at this point of development.<br> -<br> -Keep in mind that $outchannel just defines a channel with "name". It does not -activate it. To do so, you must use a selector line (see below). That selector -line includes the channel name plus an $ sign in front of it. A sample might be:<br> +name is the name of the output channel (not the file), file-name is the +file name to be written to, max-size the maximum allowed size and +action-on-max-size a command to be issued when the max size is reached. +This command always has exactly one parameter. The binary is that part +of action-on-max-size before the first space, its parameter is +everything behind that space.<br> +<br> +Please note that max-size is queried BEFORE writing the log message to +the file. So be sure to set this limit reasonably low so that any +message might fit. For the current release, setting it 1k lower than +you expected is helpful. The max-size must always be specified in bytes +- there are no special symbols (like 1k, 1m,...) at this point of +development.<br> +<br> +Keep in mind that $outchannel just defines a channel with "name". It +does not activate it. To do so, you must use a selector line (see +below). That selector line includes the channel name plus an $ sign in +front of it. A sample might be:<br> <br> *.* $mychannel<br> <br> -In its current form, output channels primarily provide the ability to size-limit -an output file. To do so, specify a maximum size. When this size is reached, -rsyslogd will execute the action-on-max-size command and then reopen the file -and retry. The command should be something like a log rotation script or a -similar thing.</p> -<p>If there is no action-on-max-size command or the command did not resolve the -situation, the file is closed and never reopened by rsyslogd (except, of course, -by huping it). This logic was integrated when we first experienced severe issues -with files larger 2gb, which could lead to rsyslogd dumping core. In such cases, -it is more appropriate to stop writing to a single file. Meanwhile, rsyslogd has -been fixed to support files larger 2gb, but obviously only on file systems and -operating system versions that do so. So it can still make sense to enforce a -2gb file size limit.</p> +In its current form, output channels primarily provide the ability to +size-limit an output file. To do so, specify a maximum size. When this +size is reached, rsyslogd will execute the action-on-max-size command +and then reopen the file and retry. The command should be something +like a <a href="log_rotation_fix_size.html">log rotation +script</a> or a similar thing.</p> +<p>If there is no action-on-max-size command or the command did +not resolve the situation, the file is closed and never reopened by +rsyslogd (except, of course, by huping it). This logic was integrated +when we first experienced severe issues with files larger 2gb, which +could lead to rsyslogd dumping core. In such cases, it is more +appropriate to stop writing to a single file. Meanwhile, rsyslogd has +been fixed to support files larger 2gb, but obviously only on file +systems and operating system versions that do so. So it can still make +sense to enforce a 2gb file size limit.</p> <h2>Filter Conditions</h2> -<p>Rsyslog offers two different types "filter conditions":</p> +<p>Rsyslog offers four different types "filter conditions":</p> <ul> - <li>"traditional" severity and facility based selectors</li> - <li>property-based filters</li> +<li>BSD-style blocks</li> +<li>"traditional" severity and facility based selectors</li> +<li>property-based filters</li> +<li>expression-based filters</li> </ul> <h3>Blocks</h3> -<p>Rsyslogd supports BSD-style blocks inside rsyslog.conf. Each block of lines -is separated from the previous block by a program or hostname specification. A -block will only log messages corresponding to the most recent program and -hostname specifications given. Thus, a block which selects ‘ppp’ as the program, -directly followed by a block that selects messages from the hostname ‘dialhost’, -then the second block will only log messages from the ppp program on dialhost. +<p>Rsyslogd supports BSD-style blocks inside rsyslog.conf. Each +block of lines is separated from the previous block by a program or +hostname specification. A block will only log messages corresponding to +the most recent program and hostname specifications given. Thus, a +block which selects ‘ppp’ as the program, directly followed by a block +that selects messages from the hostname ‘dialhost’, then the second +block will only log messages from the ppp program on dialhost. </p> -<p>A program specification is a line beginning with ‘!prog’ and the following -blocks will be associated with calls to syslog from that specific program. A -program specification for ‘foo’ will also match any message logged by the kernel -with the prefix ‘foo: ’. Alternatively, a program specification ‘-foo’ causes the -following blocks to be applied to messages from any program but the one specified. - -A hostname specification of the form ‘+hostname’ and -the following blocks will be applied to messages received from the specified -hostname. Alternatively, a hostname specification ‘-hostname’ causes the -following blocks to be applied to messages from any host but the one specified. - -If the hostname is given as ‘@’, the local hostname will be used. (NOT YET -IMPLEMENTED) A program or hostname specification may be reset by giving the -program or hostname as ‘*’.</p> -<p>Please note that the "#!prog", "#+hostname" and "#-hostname" syntax available -in BSD syslogd is not supported by rsyslogd. By default, no hostname or program -is set.</p> +<p>A program specification is a line beginning with ‘!prog’ and +the following blocks will be associated with calls to syslog from that +specific program. A program specification for ‘foo’ will also match any +message logged by the kernel with the prefix ‘foo: ’. Alternatively, a +program specification ‘-foo’ causes the following blocks to be applied +to messages from any program but the one specified. A hostname +specification of the form ‘+hostname’ and the following blocks will be +applied to messages received from the specified hostname. +Alternatively, a hostname specification ‘-hostname’ causes the +following blocks to be applied to messages from any host but the one +specified. If the hostname is given as ‘@’, the local hostname will be +used. (NOT YET IMPLEMENTED) A program or hostname specification may be +reset by giving the program or hostname as ‘*’.</p> +<p>Please note that the "#!prog", "#+hostname" and "#-hostname" +syntax available in BSD syslogd is not supported by rsyslogd. By +default, no hostname or program is set.</p> <h3>Selectors</h3> -<p><b>Selectors are the traditional way of filtering syslog messages.</b> They -have been kept in rsyslog with their original syntax, because it is well-known, -highly effective and also needed for compatibility with stock syslogd -configuration files. If you just need to filter based on priority and facility, -you should do this with selector lines. They are <b>not</b> second-class -citizens in rsyslog and offer the best performance for this job.</p> -<p>The selector field itself again consists of two parts, a facility and a -priority, separated by a period (``.''). Both parts are case insensitive and can -also be specified as decimal numbers, but don't do that, you have been warned. -Both facilities and priorities are described in rsyslog(3). The names mentioned -below correspond to the similar LOG_-values in /usr/include/rsyslog.h.<br><br>The facility is one of the following keywords: auth, authpriv, cron, daemon, -kern, lpr, mail, mark, news, security (same as auth), syslog, user, uucp and -local0 through local7. The keyword security should not be used anymore and mark -is only for internal use and therefore should not be used in applications. -Anyway, you may want to specify and redirect these messages here. The facility -specifies the subsystem that produced the message, i.e. all mail programs log -with the mail facility (LOG_MAIL) if they log using syslog.<br><br>Please note that the upcoming next syslog-RFC specifies many more facilities. -Support for them will be added in a future version of rsyslog, which might -require changes to existing configuration files.<br><br>The priority is one of the following keywords, in ascending order: debug, info, -notice, warning, warn (same as warning), err, error (same as err), crit, alert, -emerg, panic (same as emerg). The keywords error, warn and panic are deprecated -and should not be used anymore. The priority defines the severity of the message<br> -<br>The behavior of the original BSD syslogd is that all messages of the specified -priority and higher are logged according to the given action. Rsyslogd behaves the same, but has some extensions.<br><br>In addition to the above mentioned names the rsyslogd(8) understands the -following extensions: An asterisk (``*'') stands for all facilities or all -priorities, depending on where it is used (before or after the period). The -keyword none stands for no priority of the given facility.<br><br>You can specify multiple facilities with the same priority pattern in one -statement using the comma (``,'') operator. You may specify as much facilities -as you want. Remember that only the facility part from such a statement is -taken, a priority part would be skipped.</p> -<p>Multiple selectors may be specified for a single action using the semicolon -(``;'') separator. Remember that each selector in the selector field is capable -to overwrite the preceding ones. Using this behavior you can exclude some -priorities from the pattern.</p> -<p>Rsyslogd has a syntax extension to the original BSD source, that makes its -use more intuitively. You may precede every priority with an equation sign -(``='') to specify only this single priority and not any of the above. You may -also (both is valid, too) precede the priority with an exclamation mark (``!'') -to ignore all that priorities, either exact this one or this and any higher -priority. If you use both extensions than the exclamation mark must occur before -the equation sign, just use it intuitively.</p> +<p><b>Selectors are the traditional way of filtering syslog +messages.</b> They have been kept in rsyslog with their original +syntax, because it is well-known, highly effective and also needed for +compatibility with stock syslogd configuration files. If you just need +to filter based on priority and facility, you should do this with +selector lines. They are <b>not</b> second-class citizens +in rsyslog and offer the best performance for this job.</p> +<p>The selector field itself again consists of two parts, a +facility and a priority, separated by a period (".''). Both parts are +case insensitive and can also be specified as decimal numbers, but +don't do that, you have been warned. Both facilities and priorities are +described in rsyslog(3). The names mentioned below correspond to the +similar LOG_-values in /usr/include/rsyslog.h.<br> +<br> +The facility is one of the following keywords: auth, authpriv, cron, +daemon, kern, lpr, mail, mark, news, security (same as auth), syslog, +user, uucp and local0 through local7. The keyword security should not +be used anymore and mark is only for internal use and therefore should +not be used in applications. Anyway, you may want to specify and +redirect these messages here. The facility specifies the subsystem that +produced the message, i.e. all mail programs log with the mail facility +(LOG_MAIL) if they log using syslog.<br> +<br> +The priority is one of the following keywords, in ascending order: +debug, info, notice, warning, warn (same as warning), err, error (same +as err), crit, alert, emerg, panic (same as emerg). The keywords error, +warn and panic are deprecated and should not be used anymore. The +priority defines the severity of the message.<br> +<br> +The behavior of the original BSD syslogd is that all messages of the +specified priority and higher are logged according to the given action. +Rsyslogd behaves the same, but has some extensions.<br> +<br> +In addition to the above mentioned names the rsyslogd(8) understands +the following extensions: An asterisk ("*'') stands for all facilities +or all priorities, depending on where it is used (before or after the +period). The keyword none stands for no priority of the given facility.<br> +<br> +You can specify multiple facilities with the same priority pattern in +one statement using the comma (",'') operator. You may specify as much +facilities as you want. Remember that only the facility part from such +a statement is taken, a priority part would be skipped.</p> +<p>Multiple selectors may be specified for a single action using +the semicolon (";'') separator. Remember that each selector in the +selector field is capable to overwrite the preceding ones. Using this +behavior you can exclude some priorities from the pattern.</p> +<p>Rsyslogd has a syntax extension to the original BSD source, +that makes its use more intuitively. You may precede every priority +with an equation sign ("='') to specify only this single priority and +not any of the above. You may also (both is valid, too) precede the +priority with an exclamation mark ("!'') to ignore all that +priorities, either exact this one or this and any higher priority. If +you use both extensions than the exclamation mark must occur before the +equation sign, just use it intuitively.</p> <h3>Property-Based Filters</h3> -<p>Property-based filters are unique to rsyslogd. They allow to filter on any -property, like HOSTNAME, syslogtag and msg. A list of all currently-supported -properties can be found in the <a href="property_replacer.html">property -replacer documentation</a> (but keep in mind that only the properties, not the -replacer is supported). With this filter, each properties can be checked against -a specified value, using a specified compare operation. Currently, there is only -a single compare operation (contains) available, but additional operations will be added in the -future.</p> -<p>A property-based filter must start with a colon in column 0. This tells -rsyslogd that it is the new filter type. The colon must be followed by the -property name, a comma, the name of the compare operation to carry out, another -comma and then the value to compare against. This value must be quoted. There -can be spaces and tabs between the commas. Property names and compare operations -are case-sensitive, so "msg" works, while "MSG" is an invalid property name. In -brief, the syntax is as follows:</p> -<p><code><b>:property, [!]compare-operation, "value"</b></code></p> -<p>The following <b>compare-operations</b> are currently supported:</p> -<table border="1" width="100%" id="table1"> - <tr> - <td>contains</td> - <td>Checks if the string provided in value is contained in the property. - There must be an exact match, wildcards are not supported.</td> - </tr> - <tr> - <td>isequal</td> - <td>Compares the "value" string provided and the property contents. - These two values must be exactly equal to match. The difference to - contains is that contains searches for the value anywhere inside the - property value, whereas all characters must be identical for isequal. As - such, isequal is most useful for fields like syslogtag or FROMHOST, - where you probably know the exact contents.</td> - </tr> - <tr> - <td>startswith</td> - <td>Checks if the value is found exactly at the beginning of the - property value. For example, if you search for "val" with<p><code><b>:msg, - startswith, "val"</b></code></p> - <p>it will be a match if msg contains "values are in this message" but - it won't match if the msg contains "There are values in this message" - (in the later case, contains would match). Please note that "startswith" - is by far faster than regular expressions. So even once they are - implemented, it can make very much sense (performance-wise) to use "startswith".</td> - </tr> - <tr> - <td>regex</td> - <td>Compares the property against the provided regular expression.</td> - </tr> +<p>Property-based filters are unique to rsyslogd. They allow to +filter on any property, like HOSTNAME, syslogtag and msg. A list of all +currently-supported properties can be found in the <a href="property_replacer.html">property replacer documentation</a> +(but keep in mind that only the properties, not the replacer is +supported). With this filter, each properties can be checked against a +specified value, using a specified compare operation.</p> +<p>A property-based filter must start with a colon in column 0. +This tells rsyslogd that it is the new filter type. The colon must be +followed by the property name, a comma, the name of the compare +operation to carry out, another comma and then the value to compare +against. This value must be quoted. There can be spaces and tabs +between the commas. Property names and compare operations are +case-sensitive, so "msg" works, while "MSG" is an invalid property +name. In brief, the syntax is as follows:</p> +<p><code><b>:property, [!]compare-operation, "value"</b></code></p> +<p>The following <b>compare-operations</b> are +currently supported:</p> +<table id="table1" border="1" width="100%"> +<tbody> +<tr> +<td>contains</td> +<td>Checks if the string provided in value is contained in +the property. There must be an exact match, wildcards are not supported.</td> +</tr> +<tr> +<td>isequal</td> +<td>Compares the "value" string provided and the property +contents. These two values must be exactly equal to match. The +difference to contains is that contains searches for the value anywhere +inside the property value, whereas all characters must be identical for +isequal. As such, isequal is most useful for fields like syslogtag or +FROMHOST, where you probably know the exact contents.</td> +</tr> +<tr> +<td>startswith</td> +<td>Checks if the value is found exactly at the beginning +of the property value. For example, if you search for "val" with +<p><code><b>:msg, startswith, "val"</b></code></p> +<p>it will be a match if msg contains "values are in this +message" but it won't match if the msg contains "There are values in +this message" (in the later case, contains would match). Please note +that "startswith" is by far faster than regular expressions. So even +once they are implemented, it can make very much sense +(performance-wise) to use "startswith".</p> +</td> +</tr> +<tr> +<td>regex</td> +<td>Compares the property against the provided POSIX +regular +expression.</td> +</tr> +</tbody> </table> -<p>You can use the bang-character (!) immediately in front of a -compare-operation, the outcome of this operation is negated. For example, if msg -contains "This is an informative message", the following sample would not match:</p> -<p><code><b>:msg, contains, "error"</b></code></p> +<p>You can use the bang-character (!) immediately in front of a +compare-operation, the outcome of this operation is negated. For +example, if msg contains "This is an informative message", the +following sample would not match:</p> +<p><code><b>:msg, contains, "error"</b></code></p> <p>but this one matches:</p> -<p><code><b>:msg, !contains, "error"</b></code></p> -<p>Using negation can be useful if you would like to do some generic processing -but exclude some specific events. You can use the discard action in conjunction -with that. A sample would be:</p> -<p><code><b>*.* /var/log/allmsgs-including-informational.log<br> -:msg, contains, "informational" <font color="#FF0000" size="4">~</font> -<br>*.* /var/log/allmsgs-but-informational.log</b></code></p> -<p>Do not overlook the red tilde in line 2! In this sample, all messages are -written to the file allmsgs-including-informational.log. Then, all messages -containing the string "informational" are discarded. That means the config file -lines below the "discard line" (number 2 in our sample) will not be applied to -this message. Then, all remaining lines will also be written to the file -allmsgs-but-informational.log.</p> -<p><b>Value</b> is a quoted string. It supports some escape sequences:</p> -<p>\" - the quote character (e.g. "String with \"Quotes\"")<br> -\\ - the backslash character (e.g. "C:\\tmp")</p> -<p>Escape sequences always start with a backslash. Additional escape sequences -might be added in the future. Backslash characters <b>must</b> be escaped. Any -other sequence then those outlined above is invalid and may lead to -unpredictable results.</p> -<p>Probably, "msg" is the most prominent use case of property based filters. It -is the actual message text. If you would like to filter based on some message -content (e.g. the presence of a specific code), this can be done easily by:</p> -<p><code><b>:msg, contains, "ID-4711"</b></code></p> -<p>This filter will match when the message contains the string "ID-4711". Please -note that the comparison is case-sensitive, so it would not match if "id-4711" -would be contained in the message.</p> -<p>Getting property-based filters right can sometimes be challenging. In order -to help you do it with as minimal effort as possible, rsyslogd spits out debug -information for all property-based filters during their evaluation. To enable -this, run rsyslogd in foreground and specify the "-d" option.</p> -<p>Boolean operations inside property based filters (like 'message contains -"ID17" or message contains "ID18"') are currently not supported -(except for "not" as outlined above). Please note -that while it is possible to query facility and severity via property-based filters, -it is far more advisable to use classic selectors (see above) for those -cases.</p> +<p><code><b>:msg, !contains, "error"</b></code></p> +<p>Using negation can be useful if you would like to do some +generic processing but exclude some specific events. You can use the +discard action in conjunction with that. A sample would be:</p> +<p><code><b>*.* +/var/log/allmsgs-including-informational.log<br> +:msg, contains, "informational" <font color="#ff0000" size="4">~</font> +<br> +*.* /var/log/allmsgs-but-informational.log</b></code></p> +<p>Do not overlook the red tilde in line 2! In this sample, all +messages are written to the file allmsgs-including-informational.log. +Then, all messages containing the string "informational" are discarded. +That means the config file lines below the "discard line" (number 2 in +our sample) will not be applied to this message. Then, all remaining +lines will also be written to the file allmsgs-but-informational.log.</p> +<p><b>Value</b> is a quoted string. It supports some +escape sequences:</p> +<p>\" - the quote character (e.g. "String with \"Quotes\"")<br> +\\ - the backslash character (e.g. "C:\\tmp")</p> +<p>Escape sequences always start with a backslash. Additional +escape sequences might be added in the future. Backslash characters <b>must</b> +be escaped. Any other sequence then those outlined above is invalid and +may lead to unpredictable results.</p> +<p>Probably, "msg" is the most prominent use case of property +based filters. It is the actual message text. If you would like to +filter based on some message content (e.g. the presence of a specific +code), this can be done easily by:</p> +<p><code><b>:msg, contains, "ID-4711"</b></code></p> +<p>This filter will match when the message contains the string +"ID-4711". Please note that the comparison is case-sensitive, so it +would not match if "id-4711" would be contained in the message.</p> +<p><code><b>:msg, regex, "fatal .* error"</b></code></p> +<p>This filter uses a POSIX regular expression. It matches when +the +string contains the words "fatal" and "error" with anything in between +(e.g. "fatal net error" and "fatal lib error" but not "fatal error" as +two spaces are required by the regular expression!).</p> +<p>Getting property-based filters right can sometimes be +challenging. In order to help you do it with as minimal effort as +possible, rsyslogd spits out debug information for all property-based +filters during their evaluation. To enable this, run rsyslogd in +foreground and specify the "-d" option.</p> +<p>Boolean operations inside property based filters (like +'message contains "ID17" or message contains "ID18"') are currently not +supported (except for "not" as outlined above). Please note that while +it is possible to query facility and severity via property-based +filters, it is far more advisable to use classic selectors (see above) +for those cases.</p> +<h3>Expression-Based Filters</h3> +Expression based filters allow +filtering on arbitrary complex expressions, which can include boolean, +arithmetic and string operations. Expression filters will evolve into a +full configuration scripting language. Unfortunately, their syntax will +slightly change during that process. So if you use them now, you need +to be prepared to change your configuration files some time later. +However, we try to implement the scripting facility as soon as possible +(also in respect to stage work needed). So the window of exposure is +probably not too long.<br> +<br> +Expression based filters are indicated by the keyword "if" in column 1 +of a new line. They have this format:<br> +<br> +if expr then action-part-of-selector-line<br> +<br> +"If" and "then" are fixed keywords that mus be present. "expr" is a +(potentially quite complex) expression. So the <a href="expression.h">expression documentation</a> for +details. "action-part-of-selector-line" is an action, just as you know +it (e.g. "/var/log/logfile" to write to that file).<br> +<br> +A few quick samples:<br> +<br> +<code> +*.* /var/log/file1 # the traditional way<br> +if $msg contains 'error' /var/log/errlog # the expression-based way<br> +</code> +<br> +Right now, you need to specify numerical values if you would like to +check for facilities and severity. These can be found in <a href="http://www.ietf.org/rfc/rfc3164.txt">RFC 3164</a>. +If you don't like that, you can of course also use the textual property +- just be sure to use the right one. As expression support is enhanced, +this will change. For example, if you would like to filter on message +that have facility local0, start with "DEVNAME" and have either +"error1" or "error0" in their message content, you could use the +following filter:<br> +<br> +<code> +if $syslogfacility-text == 'local0' and $msg +startswith 'DEVNAME' and ($msg contains 'error1' or $msg contains +'error0') then /var/log/somelog<br> +</code> +<br> +Please note that the above <span style="font-weight: bold;">must +all be on one line</span>! And if you would like to store all +messages except those that contain "error1" or "error0", you just need +to add a "not":<br> +<br> +<code> +if $syslogfacility-text == 'local0' and $msg +startswith 'DEVNAME' and <span style="font-weight: bold;">not</span> +($msg contains 'error1' or $msg contains +'error0') then /var/log/somelog<br> +</code> +<br> +If you would like to do case-insensitive comparisons, use +"contains_i" instead of "contains" and "startswith_i" instead of +"startswith".<br> +<br> +Note that regular expressions are currently NOT +supported in expression-based filters. These will be added later when +function support is added to the expression engine (the reason is that +regular expressions will be a separate loadable module, which requires +some more prequisites before it can be implemented).<br> <h2>ACTIONS</h2> -<p>The action field of a rule describes what to do with the message. In general, -message content is written to a kind of "logfile". But also other actions might -be done, like writing to a database table or forwarding to another host.<br> -<br> -Templates can be used with all actions. If used, the specified template is used -to generate the message content (instead of the default template). To specify a -template, write a semicolon after the action value immediately followed by the -template name.<br> -<br> -Beware: templates MUST be defined BEFORE they are used. It is OK to define some -templates, then use them in selector lines, define more templates and use use -them in the following selector lines. But it is NOT permitted to use a template -in a selector line that is above its definition. If you do this, the action will be ignored.</p> -<p><b>You can have multiple actions for a single selector </b> (or more -precisely a single filter of such a selector line). Each action must be on its -own line and the line must start with an ampersand ('&') character and have no -filters. An example would be</p> +<p>The action field of a rule describes what to do with the +message. In general, message content is written to a kind of "logfile". +But also other actions might be done, like writing to a database table +or forwarding to another host.<br> +<br> +Templates can be used with all actions. If used, the specified template +is used to generate the message content (instead of the default +template). To specify a template, write a semicolon after the action +value immediately followed by the template name.<br> +<br> +Beware: templates MUST be defined BEFORE they are used. It is OK to +define some templates, then use them in selector lines, define more +templates and use use them in the following selector lines. But it is +NOT permitted to use a template in a selector line that is above its +definition. If you do this, the action will be ignored.</p> +<p><b>You can have multiple actions for a single selector </b> (or +more precisely a single filter of such a selector line). Each action +must be on its own line and the line must start with an ampersand +('&') character and have no filters. An example would be</p> <p><code><b>*.=crit rger<br> & root<br> & /var/log/critmsgs</b></code></p> -<p>These three lines send critical messages to the user rger and root and also -store them in /var/log/critmsgs. <b>Using multiple actions per selector is</b> -convenient and also <b>offers a performance benefit</b>. As the filter needs to -be evaluated only once, there is less computation required to process the -directive compared to the otherwise-equal config directives below:</p> +<p>These three lines send critical messages to the user rger and +root and also store them in /var/log/critmsgs. <b>Using multiple +actions per selector is</b> convenient and also <b>offers +a performance benefit</b>. As the filter needs to be evaluated +only once, there is less computation required to process the directive +compared to the otherwise-equal config directives below:</p> <p><code><b>*.=crit rger<br> *.=crit root<br> *.=crit /var/log/critmsgs</b></code></p> <p> </p> <h3>Regular File</h3> -<p>Typically messages are logged to real files. The file has to be specified with -full pathname, beginning with a slash "/''.<br> +<p>Typically messages are logged to real files. The file has to +be specified with full pathname, beginning with a slash "/''.<br> <br> -You may prefix each entry with the minus ``-'' sign to omit syncing the file -after every logging. Note that you might lose information if the system crashes -right behind a write attempt. Nevertheless this might give you back some -performance, especially if you run programs that use +You may prefix each entry with the minus "-'' sign to omit syncing the +file after every logging. Note that you might lose information if the +system crashes right behind a write attempt. Nevertheless this might +give you back some performance, especially if you run programs that use logging in a very verbose manner.</p> -<p>If your system is connected to a reliable UPS and you receive lots of log -data (e.g. firewall logs), it might be a very good idea to turn of -syncing by specifying the "-" in front of the file name. </p> -<p><b>The filename can be either static </b>(always the same) or <b>dynamic</b> -(different based on message received). The later is useful if you would -automatically split messages into different files based on some message -criteria. For example, dynamic file name selectors allow you to split messages -into different files based on the host that sent them. With dynamic file names, -everything is automatic and you do not need any filters. </p> -<p>It works via the template system. First, you define a template for the file -name. An example can be seen above in the description of template. We will use -the "DynFile" template defined there. Dynamic filenames are indicated by -specifying a questions mark "?" instead of a slash, followed by the template -name. Thus, the selector line for our dynamic file name would look as follows:</p> +<p>If your system is connected to a reliable UPS and you receive +lots of log data (e.g. firewall logs), it might be a very good idea to +turn of +syncing by specifying the "-" in front of the file name. </p> +<p><b>The filename can be either static </b>(always +the same) or <b>dynamic</b> (different based on message +received). The later is useful if you would automatically split +messages into different files based on some message criteria. For +example, dynamic file name selectors allow you to split messages into +different files based on the host that sent them. With dynamic file +names, everything is automatic and you do not need any filters. </p> +<p>It works via the template system. First, you define a template +for the file name. An example can be seen above in the description of +template. We will use the "DynFile" template defined there. Dynamic +filenames are indicated by specifying a questions mark "?" instead of a +slash, followed by the template name. Thus, the selector line for our +dynamic file name would look as follows:</p> <blockquote> <code>*.* ?DynFile</code> </blockquote> -<p>That's all you need to do. Rsyslog will now automatically generate file names -for you and store the right messages into the right files. Please note that the -minus sign also works with dynamic file name selectors. Thus, to avoid syncing, -you may use</p> +<p>That's all you need to do. Rsyslog will now automatically +generate file names for you and store the right messages into the right +files. Please note that the minus sign also works with dynamic file +name selectors. Thus, to avoid syncing, you may use</p> <blockquote> <code>*.* -?DynFile</code></blockquote> -<p>And of course you can use templates to specify the output format:</p> +<p>And of course you can use templates to specify the output +format:</p> <blockquote> <code>*.* ?DynFile;MyTemplate</code></blockquote> -<p><b>A word of caution:</b> rsyslog creates files as needed. So if a new host -is using your syslog server, rsyslog will automatically create a new file for -it.</p> - -<p><b>Creating directories is also supported</b>. For example you can use the hostname as directory -and the program name as file name:</p> +<p><b>A word of caution:</b> rsyslog creates files as +needed. So if a new host is using your syslog server, rsyslog will +automatically create a new file for it.</p> +<p><b>Creating directories is also supported</b>. For +example you can use the hostname as directory and the program name as +file name:</p> <blockquote> <code>$template DynFile,"/var/log/%HOSTNAME%/%programname%.log"</code></blockquote> - <h3>Named Pipes</h3> -<p>This version of rsyslogd(8) has support for logging output to named pipes (fifos). -A fifo or named pipe can be used as a destination for log messages by prepending -a pipe symbol (``|'') to the name of the file. This is handy for debugging. Note -that the fifo must be created with the mkfifo(1) command before rsyslogd(8) is -started.</p> +<p>This version of rsyslogd(8) has support for logging output to +named pipes (fifos). A fifo or named pipe can be used as a destination +for log messages by prepending a pipe symbol ("|'') to the name of the +file. This is handy for debugging. Note that the fifo must be created +with the mkfifo(1) command before rsyslogd(8) is started.</p> <h3>Terminal and Console</h3> -<p>If the file you specified is a tty, special tty-handling is done, same with -/dev/console.</p> +<p>If the file you specified is a tty, special tty-handling is +done, same with /dev/console.</p> <h3>Remote Machine</h3> -<p>Rsyslogd provides full remote logging, i.e. is able to send messages to a -remote host running rsyslogd(8) and to receive messages from remote hosts. -Using this feature you're able to control all syslog messages on one host, if -all other machines will log remotely to that. This tears down<br> +<p>Rsyslogd provides full remote logging, i.e. is able to send +messages to a remote host running rsyslogd(8) and to receive messages +from remote hosts. Using this feature you're able to control all syslog +messages on one host, if all other machines will log remotely to that. +This tears down<br> administration needs.<br> <br> -<b>Please note that this version of rsyslogd by default does NOT forward messages -it has received from the network to another host. Specify the "-h" option to enable this.</b></p> -<p>To forward messages to another host, prepend the hostname with the at sign ("@"). -A single at sign means that messages will be forwarded via UDP protocol (the -standard for syslog). If you prepend two at signs ("@@"), the messages will be -transmitted via TCP. Please note that plain TCP based syslog is not officially -standardized, but most major syslogds support it (e.g. syslog-ng or WinSyslog). -The forwarding action indicator (at-sign) can be followed by one or more options. -If they are given, they must be immediately (without a space) following the -final at sign and be enclosed in parenthesis. The individual options must be -separated by commas. The following options are right now defined:</p> -<table border="1" width="100%" id="table2"> - <tr> - <td> - <p align="center"><b>z<number></b></td> - <td>Enable zlib-compression for the message. The <number> is the - compression level. It can be 1 (lowest gain, lowest CPU overhead) to 9 (maximum - compression, highest CPU overhead). The level can also be 0, which means - "no compression". If given, the "z" option is ignored. So this does not - make an awful lot of sense. There is hardly a difference between level 1 - and 9 for typical syslog messages. You can expect a compression gain - between 0% and 30% for typical messages. Very chatty messages may - compress up to 50%, but this is seldom seen with typically traffic. - Please note that rsyslogd checks the compression gain. Messages with 60 - bytes or less will never be compressed. This is because compression gain - is pretty unlikely and we prefer to save CPU cycles. Messages over that - size are always compressed. However, it is checked if there is a gain in - compression and only if there is, the compressed message is transmitted. - Otherwise, the uncompressed messages is transmitted. This saves the - receiver CPU cycles for decompression. It also prevents small message to - actually become larger in compressed form.<p><b>Please note that when a - TCP transport is used, compression will also turn on - syslog-transport-tls framing. See the "o" option for important - information on the implications.</b></p> - <p>Compressed messages are automatically detected and decompressed by - the receiver. There is nothing that needs to be configured on the - receiver side.</td> - </tr> - <tr> - <td> - <p align="center"><b>o</b></td> - <td><b>This option is experimental. Use at your own risk and only if you - know why you need it! If in doubt, do NOT turn it on.</b><p>This option - is only valid for plain TCP based transports. It selects a different - framing based on IETF internet draft syslog-transport-tls-06. This - framing offers some benefits over traditional LF-based framing. However, - the standardization effort is not yet complete. There may be changes in - upcoming versions of this standard. Rsyslog will be kept in line with - the standard. There is some chance that upcoming changes will be - incompatible to the current specification. In this case, all systems - using -transport-tls framing must be upgraded. There will be no effort - made to retain compatibility between different versions of rsyslog. The - primary reason for that is that it seems technically impossible to - provide compatibility between some of those changes. So you should take - this note very serious. It is not something we do not *like* to do (and - may change our mind if enough people beg...), it is something we most - probably *can not* do for technical reasons (aka: you can beg as much as - you like, it won't change anything...).</p> - <p>The most important implication is that compressed syslog messages via - TCP must be considered with care. Unfortunately, it is technically - impossible to transfer compressed records over traditional syslog plain - tcp transports, so you are left with two evil choices...</td> - </tr> +<b>Please note that this version of rsyslogd by default does NOT +forward messages it has received from the network to another host. +Specify the "-h" option to enable this.</b></p> +<p>To forward messages to another host, prepend the hostname with +the at sign ("@"). A single at sign means that messages will +be forwarded via UDP protocol (the standard for syslog). If you prepend +two at signs ("@@"), the messages will be transmitted via TCP. Please +note that plain TCP based syslog is not officially standardized, but +most major syslogds support it (e.g. syslog-ng or WinSyslog). The +forwarding action indicator (at-sign) can be followed by one or more +options. If they are given, they must be immediately (without a space) +following the final at sign and be enclosed in parenthesis. The +individual options must be separated by commas. The following options +are right now defined:</p> +<table id="table2" border="1" width="100%"> +<tbody> +<tr> +<td> +<p align="center"><b>z<number></b></p> +</td> +<td>Enable zlib-compression for the message. The +<number> is the compression level. It can be 1 (lowest +gain, lowest CPU overhead) to 9 (maximum compression, highest CPU +overhead). The level can also be 0, which means "no compression". If +given, the "z" option is ignored. So this does not make an awful lot of +sense. There is hardly a difference between level 1 and 9 for typical +syslog messages. You can expect a compression gain between 0% and 30% +for typical messages. Very chatty messages may compress up to 50%, but +this is seldom seen with typically traffic. Please note that rsyslogd +checks the compression gain. Messages with 60 bytes or less will never +be compressed. This is because compression gain is pretty unlikely and +we prefer to save CPU cycles. Messages over that size are always +compressed. However, it is checked if there is a gain in compression +and only if there is, the compressed message is transmitted. Otherwise, +the uncompressed messages is transmitted. This saves the receiver CPU +cycles for decompression. It also prevents small message to actually +become larger in compressed form. +<p><b>Please note that when a TCP transport is used, +compression will also turn on syslog-transport-tls framing. See the "o" +option for important information on the implications.</b></p> +<p>Compressed messages are automatically detected and +decompressed by the receiver. There is nothing that needs to be +configured on the receiver side.</p> +</td> +</tr> +<tr> +<td> +<p align="center"><b>o</b></p> +</td> +<td><b>This option is experimental. Use at your own +risk and only if you know why you need it! If in doubt, do NOT turn it +on.</b> +<p>This option is only valid for plain TCP based +transports. It selects a different framing based on IETF internet draft +syslog-transport-tls-06. This framing offers some benefits over +traditional LF-based framing. However, the standardization effort is +not yet complete. There may be changes in upcoming versions of this +standard. Rsyslog will be kept in line with the standard. There is some +chance that upcoming changes will be incompatible to the current +specification. In this case, all systems using -transport-tls framing +must be upgraded. There will be no effort made to retain compatibility +between different versions of rsyslog. The primary reason for that is +that it seems technically impossible to provide compatibility between +some of those changes. So you should take this note very serious. It is +not something we do not *like* to do (and may change our mind if enough +people beg...), it is something we most probably *can not* do for +technical reasons (aka: you can beg as much as you like, it won't +change anything...).</p> +<p>The most important implication is that compressed syslog +messages via TCP must be considered with care. Unfortunately, it is +technically impossible to transfer compressed records over traditional +syslog plain tcp transports, so you are left with two evil choices...</p> +</td> +</tr> +</tbody> </table> <p><br> The hostname may be followed by a colon and the destination port.</p> <p>The following is an example selector line with forwarding:</p> <p>*.* @@(o,z9)192.168.0.1:1470</p> -<p>In this example, messages are forwarded via plain TCP with experimental -framing and maximum compression to the host 192.168.0.1 at port 1470.</p> +<p>In this example, messages are forwarded via plain TCP with +experimental framing and maximum compression to the host 192.168.0.1 at +port 1470.</p> <p>*.* @192.168.0.1</p> -<p>In the example above, messages are forwarded via UDP to the machine -192.168.0.1, the destination port defaults to 514. Messages will not be -compressed.</p> +<p>In the example above, messages are forwarded via UDP to the +machine 192.168.0.1, the destination port defaults to 514. Messages +will not be compressed.</p> <p>Note that IPv6 addresses contain colons. So if an IPv6 address is specified in the hostname part, rsyslogd could not detect where the IP address ends and where the port starts. There is a syntax extension to support this: @@ -544,167 +929,182 @@ brackets also work with real host names and IPv4 addresses, too. is as follows: <p>*.* @[2001::1]:515 <p>This works with TCP, too. -<p><b>Note to sysklogd users:</b> sysklogd does <b>not</b> support RFC 3164 -format, which is the default forwarding template in rsyslog. As such, you will -experience duplicate hostnames if rsyslog is the sender and sysklogd is the -receiver. The fix is simple: you need to use a different template. Use that one:</p> -<p class="MsoPlainText">$template sysklogd,"<%PRI%>%TIMESTAMP% -%syslogtag%%msg%\""<br> +<p><b>Note to sysklogd users:</b> sysklogd does <b>not</b> +support RFC 3164 format, which is the default forwarding template in +rsyslog. As such, you will experience duplicate hostnames if rsyslog is +the sender and sysklogd is the receiver. The fix is simple: you need to +use a different template. Use that one:</p> +<p class="MsoPlainText">$template +sysklogd,"<%PRI%>%TIMESTAMP% %syslogtag%%msg%\""<br> *.* @192.168.0.1;sysklogd</p> <h3>List of Users</h3> -<p>Usually critical messages are also directed to ``root'' on that machine. You can -specify a list of users that shall get the message by simply writing the login. -You may specify more than one user by separating them with commas (",''). If -they're logged in they get the message. Don't think a mail would be sent, that -might be too late.</p> +<p>Usually critical messages are also directed to "root'' on +that machine. You can specify a list of users that shall get the +message by simply writing the login. You may specify more than one user +by separating them with commas (",''). If they're logged in they get +the message. Don't think a mail would be sent, that might be too late.</p> <h3>Everyone logged on</h3> -<p>Emergency messages often go to all users currently online to notify them that -something strange is happening with the system. To specify this wall(1)-feature -use an asterisk ("*'').</p> +<p>Emergency messages often go to all users currently online to +notify them that something strange is happening with the system. To +specify this wall(1)-feature use an asterisk ("*'').</p> <h3>Call Plugin</h3> -<p>This is a generic way to call an output plugin. The plugin must support this -functionality. Actual parameters depend on the module, so see the module's doc -on what to supply. The general syntax is as follows:</p> +<p>This is a generic way to call an output plugin. The plugin +must support this functionality. Actual parameters depend on the +module, so see the module's doc on what to supply. The general syntax +is as follows:</p> <p>:modname:params;template</p> -<p>Currently, the ommysql database output module supports this syntax (in -addtion to the ">" syntax it traditionally supported). For ommysql, the module -name is "ommysql" and the params are the traditional ones. The ;template part is -not module specific, it is generic rsyslog functionality available to all -modules.</p> +<p>Currently, the ommysql database output module supports this +syntax (in addtion to the ">" syntax it traditionally +supported). For ommysql, the module name is "ommysql" and the params +are the traditional ones. The ;template part is not module specific, it +is generic rsyslog functionality available to all modules.</p> <p>As an example, the ommysql module may be called as follows:</p> <p>:ommysql:dbhost,dbname,dbuser,dbpassword;dbtemplate</p> -<p>For details, please see the "Database Table" section of this documentation.</p> -<p>Note: as of this writing, the ":modname:" part is hardcoded into the module. -So the name to use is not necessarily the name the module's plugin file is -called.</p> +<p>For details, please see the "Database Table" section of this +documentation.</p> +<p>Note: as of this writing, the ":modname:" part is hardcoded +into the module. So the name to use is not necessarily the name the +module's plugin file is called.</p> <h3>Database Table</h3> -<p>This allows logging of the message to a database table. Currently, only MySQL -databases are supported. However, other database drivers will most probably be -developed as plugins. By default, a <a href="http://www.monitorware.com/">MonitorWare</a>-compatible schema is required -for this to work. You can create that schema with the createDB.SQL file that -came with the rsyslog package. You can also<br> -use any other schema of your liking - you just need to define a proper template -and assign this template to the action.<br> -<br> -The database writer is called by specifying a greater-then sign (">") in front -of the database connect information. Immediately after that<br> -sign the database host name must be given, a comma, the database name, another -comma, the database user, a comma and then the user's password. If a specific -template is to be used, a semicolon followed by the template name can follow -the connect information. This is as follows:<br> +<p>This allows logging of the message to a database table. +Currently, only MySQL databases are supported. However, other database +drivers will most probably be developed as plugins. By default, a <a href="http://www.monitorware.com/">MonitorWare</a>-compatible +schema is required for this to work. You can create that schema with +the createDB.SQL file that came with the rsyslog package. You can also<br> +use any other schema of your liking - you just need to define a proper +template and assign this template to the action.<br> +<br> +The database writer is called by specifying a greater-then sign +(">") in front of the database connect information. Immediately +after that<br> +sign the database host name must be given, a comma, the database name, +another comma, the database user, a comma and then the user's password. +If a specific template is to be used, a semicolon followed by the +template name can follow the connect information. This is as follows:<br> <br> >dbhost,dbname,dbuser,dbpassword;dbtemplate</p> -<p><b>Important: to use the database functionality, the MySQL output module must be -loaded in the config file</b> BEFORE the first database table action is used. This is done by -placing the</p> -<p><code><b>$ModLoad MySQL</b></code></p> -<p>directive some place above the first use of the database write (we recommend -doing at the the beginning of the config file).</p> +<p><b>Important: to use the database functionality, the +MySQL output module must be loaded in the config file</b> BEFORE +the first database table action is used. This is done by placing the</p> +<p><code><b>$ModLoad ommysql</b></code></p> +<p>directive some place above the first use of the database write +(we recommend doing at the the beginning of the config file).</p> <h3>Discard</h3> -<p>If the discard action is carried out, the received message is immediately -discarded. No further processing of it occurs. Discard has primarily been added -to filter out messages before carrying on any further processing. For obvious -reasons, the results of "discard" are depending on where in the configuration -file it is being used. Please note that once a message has been discarded there -is no way to retrieve it in later configuration file lines.</p> -<p>Discard can be highly effective if you want to filter out some annoying -messages that otherwise would fill your log files. To do that, place the discard -actions early in your log files. This often plays well with property-based -filters, giving you great freedom in specifying what you do not want.</p> -<p>Discard is just the single tilde character with no further parameters:</p> +<p>If the discard action is carried out, the received message is +immediately discarded. No further processing of it occurs. Discard has +primarily been added to filter out messages before carrying on any +further processing. For obvious reasons, the results of "discard" are +depending on where in the configuration file it is being used. Please +note that once a message has been discarded there is no way to retrieve +it in later configuration file lines.</p> +<p>Discard can be highly effective if you want to filter out some +annoying messages that otherwise would fill your log files. To do that, +place the discard actions early in your log files. This often plays +well with property-based filters, giving you great freedom in +specifying what you do not want.</p> +<p>Discard is just the single tilde character with no further +parameters:</p> <p>~</p> <p>For example,</p> <p>*.* ~</p> -<p>discards everything (ok, you can achive the same by not running rsyslogd at -all...).</p> +<p>discards everything (ok, you can achive the same by not +running rsyslogd at all...).</p> <h3>Output Channel</h3> -<p>Binds an output channel definition (see there for details) to this action. -Output channel actions must start with a $-sign, e.g. if you would like to bind -your output channel definition "mychannel" to the action, use "$mychannel". -Output channels support template definitions like all all other actions.</p> +<p>Binds an output channel definition (see there for details) to +this action. Output channel actions must start with a $-sign, e.g. if +you would like to bind your output channel definition "mychannel" to +the action, use "$mychannel". Output channels support template +definitions like all all other actions.</p> <h3>Shell Execute</h3> -<p>This executes a program in a subshell. The program is passed the -template-generated message as the only command line parameter. Rsyslog waits -until the program terminates and only then continues to run.</p> +<p>This executes a program in a subshell. The program is passed +the template-generated message as the only command line parameter. +Rsyslog waits until the program terminates and only then continues to +run.</p> <p>^program-to-execute;template</p> -<p>The program-to-execute can be any valid executable. It receives the template -string as a single parameter (argv[1]).</p> -<p><b>WARNING:</b> The Shell Execute action was added to serve an urgent need. -While it is considered reasonable save when used with some thinking, its -implications must be considered. The current implementation uses a system() call -to execute the command. This is not the best way to do it (and will hopefully -changed in further releases). Also, proper escaping of special characters is -done to prevent command injection. However, attackers always find smart ways to -circumvent escaping, so we can not say if the escaping applied will really safe -you from all hassles. Lastly, rsyslog will wait until the shell command -terminates. Thus, a program error in it (e.g. an infinite loop) can actually -disable rsyslog. Even without that, during the programs run-time no messages are -processed by rsyslog. As the IP stacks buffers are quickly overflowed, this -bears an increased risk of message loss. You must be aware of these implications. -Even though they are severe, there are several cases where the "shell execute" -action is very useful. This is the reason why we have included it in its current -form. To mitigate its risks, always a) test your program thoroughly, b) make -sure its runtime is as short as possible (if it requires a longer run-time, you -might want to spawn your own sub-shell asynchronously), c) apply proper -firewalling so that only known senders can send syslog messages to rsyslog. -Point c) is especially important: if rsyslog is accepting message from any hosts, -chances are much higher that an attacker might try to exploit the "shell execute" -action.</p> +<p>The program-to-execute can be any valid executable. It +receives the template string as a single parameter (argv[1]).</p> +<p><b>WARNING:</b> The Shell Execute action was added +to serve an urgent need. While it is considered reasonable save when +used with some thinking, its implications must be considered. The +current implementation uses a system() call to execute the command. +This is not the best way to do it (and will hopefully changed in +further releases). Also, proper escaping of special characters is done +to prevent command injection. However, attackers always find smart ways +to circumvent escaping, so we can not say if the escaping applied will +really safe you from all hassles. Lastly, rsyslog will wait until the +shell command terminates. Thus, a program error in it (e.g. an infinite +loop) can actually disable rsyslog. Even without that, during the +programs run-time no messages are processed by rsyslog. As the IP +stacks buffers are quickly overflowed, this bears an increased risk of +message loss. You must be aware of these implications. Even though they +are severe, there are several cases where the "shell execute" action is +very useful. This is the reason why we have included it in its current +form. To mitigate its risks, always a) test your program thoroughly, b) +make sure its runtime is as short as possible (if it requires a longer +run-time, you might want to spawn your own sub-shell asynchronously), +c) apply proper firewalling so that only known senders can send syslog +messages to rsyslog. Point c) is especially important: if rsyslog is +accepting message from any hosts, chances are much higher that an +attacker might try to exploit the "shell execute" action.</p> <h2>TEMPLATE NAME</h2> -<p>Every ACTION can be followed by a template name. If so, that template is used -for message formatting. If no name is given, a hard-coded default template is -used for the action. There can only be one template name for each given action. -The default template is specific to each action. For a description of what a -template is and what you can do with it, see "TEMPLATES" at the top of this -document.</p> +<p>Every ACTION can be followed by a template name. If so, that +template is used for message formatting. If no name is given, a +hard-coded default template is used for the action. There can only be +one template name for each given action. The default template is +specific to each action. For a description of what a template is and +what you can do with it, see "TEMPLATES" at the top of this document.</p> <h2>EXAMPLES</h2> -<p>Below are example for templates and selector lines. I hope they are -self-explanatory. If not, please see www.monitorware.com/rsyslog/ for advise.</p> +<p>Below are example for templates and selector lines. I hope +they are self-explanatory. If not, please see +www.monitorware.com/rsyslog/ for advise.</p> <h3>TEMPLATES</h3> -<p>Please note that the samples are split across multiple lines. A template MUST -NOT actually be split across multiple lines.<br> +<p>Please note that the samples are split across multiple lines. +A template MUST NOT actually be split across multiple lines.<br> <br> A template that resembles traditional syslogd file output:<br> -$template TraditionalFormat,"%timegenerated% %HOSTNAME%<br> -%syslogtag%%msg:::drop-last-lf%\n"<br> +$template TraditionalFormat,"%timegenerated% %HOSTNAME%<br> +%syslogtag%%msg:::drop-last-lf%\n"<br> <br> A template that tells you a little more about the message:<br> -$template precise,"%syslogpriority%,%syslogfacility%,%timegenerated%,%HOSTNAME%,<br> -%syslogtag%,%msg%\n"<br> +$template +precise,"%syslogpriority%,%syslogfacility%,%timegenerated%,%HOSTNAME%,<br> +%syslogtag%,%msg%\n"<br> <br> A template for RFC 3164 format:<br> -$template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%"<br> +$template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% +%syslogtag%%msg%"<br> <br> A template for the format traditonally used for user messages:<br> -$template usermsg," XXXX%syslogtag%%msg%\n\r"<br> +$template usermsg," XXXX%syslogtag%%msg%\n\r"<br> <br> And a template with the traditonal wall-message format:<br> -$template wallmsg,"\r\n\7Message from syslogd@%HOSTNAME% at %timegenerated%<br> +$template wallmsg,"\r\n\7Message from syslogd@%HOSTNAME% at +%timegenerated%<br> <br> A template that can be used for the database write (please note the SQL<br> template option)<br> -$template MySQLInsert,"insert iut, message, receivedat values<br> +$template MySQLInsert,"insert iut, message, receivedat values<br> ('%iut%', '%msg:::UPPERCASE%', '%timegenerated:::date-mysql%')<br> -into systemevents\r\n", SQL<br> +into systemevents\r\n", SQL<br> <br> -The following template emulates <a href="http://www.winsyslog.com/en/">WinSyslog</a> -format (it's an <a href="http://www.adiscon.com/en/">Adiscon</a> format, you do -not feel bad if you don't know it ;)). It's interesting to see how it takes -different parts out of the date stamps. What happens is that the date stamp is -split into the actual date and time and the these two are combined with just a -comma in between them.<br> +The following template emulates <a href="http://www.winsyslog.com/en/">WinSyslog</a> +format (it's an <a href="http://www.adiscon.com/en/">Adiscon</a> +format, you do not feel bad if you don't know it ;)). It's interesting +to see how it takes different parts out of the date stamps. What +happens is that the date stamp is split into the actual date and time +and the these two are combined with just a comma in between them.<br> <br> -$template WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,<br> +$template WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,<br> %timegenerated:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%,<br> %timegenerated:12:19:date-rfc3339%,%syslogfacility%,%syslogpriority%,<br> -%syslogtag%%msg%\n"</p> +%syslogtag%%msg%\n"</p> <h3>SELECTOR LINES</h3> <p># Store critical stuff in critical<br> #<br> *.=crit;kern.none /var/adm/critical<br> <br> -This will store all messages with the priority crit in the file /var/adm/critical, -except for any kernel message.<br> +This will store all messages with the priority crit in the file +/var/adm/critical, except for any kernel message.<br> <br> <br> # Kernel messages are first, stored in the kernel<br> @@ -718,20 +1118,21 @@ kern.crit @finlandia;RFC3164fmt<br> kern.crit /dev/console<br> kern.info;kern.!err /var/adm/kernel-info<br> <br> -The first rule direct any message that has the kernel facility to the file /var/adm/kernel.<br> +The first rule direct any message that has the kernel facility to the +file /var/adm/kernel.<br> <br> -The second statement directs all kernel messages of the priority crit and higher -to the remote host finlandia. This is useful, because if the host crashes and -the disks get irreparable errors you might not be able to read the stored -messages. If they're on a remote host, too, you still can try to find out the -reason for the crash.<br> +The second statement directs all kernel messages of the priority crit +and higher to the remote host finlandia. This is useful, because if the +host crashes and the disks get irreparable errors you might not be able +to read the stored messages. If they're on a remote host, too, you +still can try to find out the reason for the crash.<br> <br> -The third rule directs these messages to the actual console, so the person who -works on the machine will get them, too.<br> +The third rule directs these messages to the actual console, so the +person who works on the machine will get them, too.<br> <br> -The fourth line tells rsyslogd to save all kernel messages that come with -priorities from info up to warning in the file /var/adm/kernel-info. Everything -from err and higher is excluded.<br> +The fourth line tells rsyslogd to save all kernel messages that come +with priorities from info up to warning in the file +/var/adm/kernel-info. Everything from err and higher is excluded.<br> <br> <br> # The tcp wrapper loggs with mail.info, we display<br> @@ -739,25 +1140,26 @@ from err and higher is excluded.<br> #<br> mail.=info /dev/tty12<br> <br> -This directs all messages that uses mail.info (in source LOG_MAIL | LOG_INFO) to -/dev/tty12, the 12th console. For example the tcpwrapper tcpd(8) uses this as -it's default.<br> +This directs all messages that uses mail.info (in source LOG_MAIL | +LOG_INFO) to /dev/tty12, the 12th console. For example the tcpwrapper +tcpd(8) uses this as it's default.<br> <br> <br> # Store all mail concerning stuff in a file<br> #<br> mail.*;mail.!=info /var/adm/mail<br> <br> -This pattern matches all messages that come with the mail facility, except for -the info priority. These will be stored in the file /var/adm/mail.<br> +This pattern matches all messages that come with the mail facility, +except for the info priority. These will be stored in the file +/var/adm/mail.<br> <br> <br> # Log all mail.info and news.info messages to info<br> #<br> mail,news.=info /var/adm/info<br> <br> -This will extract all messages that come either with mail.info or with news.info -and store them in the file /var/adm/info.<br> +This will extract all messages that come either with mail.info or with +news.info and store them in the file /var/adm/info.<br> <br> <br> # Log info and notice messages to messages file<br> @@ -765,8 +1167,8 @@ and store them in the file /var/adm/info.<br> *.=info;*.=notice;\<br> mail.none /var/log/messages<br> <br> -This lets rsyslogd log all messages that come with either the info or the notice -facility into the file /var/log/messages, except for all<br> +This lets rsyslogd log all messages that come with either the info or +the notice facility into the file /var/log/messages, except for all<br> messages that use the mail facility.<br> <br> <br> @@ -775,17 +1177,17 @@ messages that use the mail facility.<br> *.=info;\<br> mail,news.none /var/log/messages<br> <br> -This statement causes rsyslogd to log all messages that come with the info -priority to the file /var/log/messages. But any message coming either with the -mail or the news facility will not be stored.<br> +This statement causes rsyslogd to log all messages that come with the +info priority to the file /var/log/messages. But any message coming +either with the mail or the news facility will not be stored.<br> <br> <br> # Emergency messages will be displayed using wall<br> #<br> *.=emerg *<br> <br> -This rule tells rsyslogd to write all emergency messages to all currently logged -in users. This is the wall action.<br> +This rule tells rsyslogd to write all emergency messages to all +currently logged in users. This is the wall action.<br> <br> <br> # Messages of the priority alert will be directed<br> @@ -793,37 +1195,38 @@ in users. This is the wall action.<br> #<br> *.alert root,rgerhards<br> <br> -This rule directs all messages with a priority of alert or higher to the -terminals of the operator, i.e. of the users ``root'' and ``rgerhards'' if -they're logged in.<br> +This rule directs all messages with a priority of alert or higher to +the terminals of the operator, i.e. of the users "root'' and +"rgerhards'' if they're logged in.<br> <br> <br> *.* @finlandia<br> <br> -This rule would redirect all messages to a remote host called finlandia. This is -useful especially in a cluster of machines where all syslog messages will be -stored on only one machine.<br> +This rule would redirect all messages to a remote host called +finlandia. This is useful especially in a cluster of machines where all +syslog messages will be stored on only one machine.<br> <br> -In the format shown above, UDP is used for transmitting the message. The -destination port is set to the default auf 514. Rsyslog is also capable of using -much more secure and reliable TCP sessions for message forwarding. Also, the -destination port can be specified. To select TCP, simply add one additional @ in -front of the host name (that is, @host is UPD, @@host is TCP). For example:<br> +In the format shown above, UDP is used for transmitting the message. +The destination port is set to the default auf 514. Rsyslog is also +capable of using much more secure and reliable TCP sessions for message +forwarding. Also, the destination port can be specified. To select TCP, +simply add one additional @ in front of the host name (that is, @host +is UPD, @@host is TCP). For example:<br> <br> <br> *.* @@finlandia<br> <br> -To specify the destination port on the remote machine, use a colon followed by -the port number after the machine name. The following forwards to port 1514 on -finlandia:<br> +To specify the destination port on the remote machine, use a colon +followed by the port number after the machine name. The following +forwards to port 1514 on finlandia:<br> <br> <br> *.* @@finlandia:1514<br> <br> -This syntax works both with TCP and UDP based syslog. However, you will probably -primarily need it for TCP, as there is no well-accepted port for this transport -(it is non-standard). For UDP, you can usually stick with the default auf 514, -but might want to modify it for security rea-<br> +This syntax works both with TCP and UDP based syslog. However, you will +probably primarily need it for TCP, as there is no well-accepted port +for this transport (it is non-standard). For UDP, you can usually stick +with the default auf 514, but might want to modify it for security rea-<br> sons. If you would like to do that, it's quite easy:<br> <br> <br> @@ -833,27 +1236,28 @@ sons. If you would like to do that, it's quite easy:<br> <br> *.* >dbhost,dbname,dbuser,dbpassword;dbtemplate<br> <br> -This rule writes all message to the database "dbname" hosted on "dbhost". The -login is done with user "dbuser" and password "dbpassword". The actual table -that is updated is specified within the template (which contains the insert -statement). The template is called "dbtemplate" in this case.</p> -<p>:msg,contains,"error" @errorServer</p> -<p>This rule forwards all messages that contain the word "error" in the msg part -to the server "errorServer". Forwarding is via UDP. Please note the colon in -fron</p> +This rule writes all message to the database "dbname" hosted on +"dbhost". The login is done with user "dbuser" and password +"dbpassword". The actual table that is updated is specified within the +template (which contains the insert statement). The template is called +"dbtemplate" in this case.</p> +<p>:msg,contains,"error" @errorServer</p> +<p>This rule forwards all messages that contain the word "error" +in the msg part to the server "errorServer". Forwarding is via UDP. +Please note the colon in fron</p> <h2>CONFIGURATION FILE SYNTAX DIFFERENCES</h2> -<p>Rsyslogd uses a slightly different syntax for its configuration file than the -original BSD sources. Originally all messages of a specific priority and above -were forwarded to the log file. The modifiers ``='', ``!'' and ``-'' were added -to make rsyslogd more flexible and to use it in a more intuitive manner.<br> -<br> -The original BSD syslogd doesn't understand spaces as separators between the -selector and the action field.<br> -<br> -When compared to syslogd from sysklogd package, rsyslogd offers additional -<a href="features.html">features</a> (like template and database support). For obvious reasons, the syntax for -defining such features is available -in rsyslogd, only.<br> - </p> -</body> -</html> +<p>Rsyslogd uses a slightly different syntax for its +configuration file than the original BSD sources. Originally all +messages of a specific priority and above were forwarded to the log +file. The modifiers "='', "!'' and "!-'' were added to make rsyslogd +more flexible and to use it in a more intuitive manner.<br> +<br> +The original BSD syslogd doesn't understand spaces as separators +between the selector and the action field.<br> +<br> +When compared to syslogd from sysklogd package, rsyslogd offers +additional +<a href="features.html">features</a> (like template +and database support). For obvious reasons, the syntax for defining +such features is available in rsyslogd, only.</p> +</body></html> diff --git a/doc/rsyslog_high_database_rate.html b/doc/rsyslog_high_database_rate.html new file mode 100644 index 00000000..158a4df6 --- /dev/null +++ b/doc/rsyslog_high_database_rate.html @@ -0,0 +1,177 @@ +<html><head> + +<title>Handling a massive syslog database insert rate with Rsyslog</title> + +<meta name="KEYWORDS" content="syslog, rsyslog, reliable, howto, database, postgresql, mysql, buffering, disk, queue"> + +</head> + +<body> + +<h1>Handling a massive syslog database insert rate with Rsyslog</h1> + + <P><small><i>Written by + + <a href="http://www.gerhards.net/rainer">Rainer + + Gerhards</a> (2008-01-31)</i></small></P> + +<h2>Abstract</h2> + +<p><i><b>In this paper, I describe how log massive amounts of +<a href="http://www.monitorware.com/en/topics/syslog/">syslog</a> + +messages to a database. </b>This HOWTO is currently under development and thus a +bit brief. Updates are promised ;).</i></p> + +<h2>The Intention</h2> + +<p>Database updates are inherently slow when it comes to storing syslog +messages. However, there are a number of applications where it is handy to have +the message inside a database. Rsyslog supports native database writing via +output plugins. As of this writing, there are plugins available for MySQL an +PostgreSQL. Maybe additional plugins have become available by the time you read +this. Be sure to check.</p> +<p>In order to successfully write messages to a database backend, the backend +must be capable to record messages at the expected average arrival rate. This is +the rate if you take all messages that can arrive within a day and divide it by +86400 (the number of seconds per day). Let's say you expect 43,200,000 messages +per day. That's an average rate of 500 messages per second (mps). Your database +server MUST be able to handle that amount of message per second on a sustained +rate. If it doesn't, you either need to add an additional server, lower the +number of message - or forget about it.</p> +<p>However, this is probably not your peak rate. Let's simply assume your +systems work only half a day, that's 12 hours (and, yes, I know this is +unrealistic, but you'll get the point soon). So your average rate is actually +1,000 mps during work hours and 0 mps during non-work hours. To make matters +worse, workload is not divided evenly during the day. So you may have peaks of +up to 10,000mps while at other times the load may go down to maybe just 100mps. +Peaks may stay well above 2,000mps for a few minutes.</p> +<p>So how the hack you will be able to handle all of this traffic (including the +peaks) with a database server that is just capable of inserting a maximum of +500mps?</p> +<p>The key here is buffering. Messages that the database server is not capable +to handle will be buffered until it is. Of course, that means database insert +are NOT real-time. If you need real-time inserts, you need to make sure your +database server can handle traffic at the actual peak rate. But lets assume you +are OK with some delay.</p> +<p>Buffering is fine. But how about these massive amounts of data? That can't be +hold in memory, so don't we run out of luck with buffering? The key here is that +rsyslog can not only buffer in memory but also buffer to disk (this may remind +you of "spooling" which gets you the right idea). There are several queuing +modes available, offering differnent throughput. In general, the idea is to +buffer in memory until the memory buffer is exhausted and switch to +disk-buffering when needed (and only as long as needed). All of this is handled +automatically and transparently by rsyslog.</p> +<p>With our above scenario, the disk buffer would build up during the day and +rsyslog would use the night to drain it. Obviously, this is an extreme example, +but it shows what can be done. Please note that queue content survies rsyslogd +restarts, so even a reboot of the system will not cause any message loss.</p> +<h2>How To Setup</h2> +<p>Frankly, it's quite easy. You just need to do is instruct rsyslog to use a +disk queue and then configure your action. There is nothing else to do. With the +following simple config file, you log anything you receive to a MySQL database +and have buffering applied automatically.</p> +<textarea rows="11" cols="80"> +$ModLoad ommysql # load the output driver (use ompgsql for PostgreSQL) +$ModLoad imudp # network reception +$UDPServerRun 514 # start a udp server at port 514 +$ModLoad imuxsock # local message reception + +$WorkDirectory /rsyslog/work # default location for work (spool) files +$MainMsgQueueFileName mainq # set file name, also enables disk mode + +$ActionResumeRetryCount -1 # infinite retries on insert failure +# for PostgreSQL replace :ommysql: by :ompgsql: below: +*.* :ommysql:hostname,dbname,userid,password; +</textarea> +<p>The simple setup above has one drawback: the write database action is +executed together with all other actions. Typically, local files are also +written. These local file writes are now bound to the speed of the database +action. So if the database is down, or threre is a large backlog, local files +are also not (or late) written.</p> +<p><b>There is an easy way to avoid this with rsyslog.</b> It involves a +slightly more complicated setup. In rsyslog, each action can utilize its own +queue. If so, messages are simply pulled over from the main queue and then the +action queue handles action processing on its own. This way, main processing and +the action are de-coupled. In the above example, this means that local file +writes will happen immediately while the database writes are queued. As a +side-note, each action can have its own queue, so if you would like to more than +a single database or send messages reliably to another host, you can do all of +this on their own queues, de-coupling their processing speeds.</p> +<p>The configuration for the de-coupled database write involves just a few more +commands:</p> +<textarea rows="11" cols="80"> +$ModLoad ommysql # load the output driver (use ompgsql for PostgreSQL) +$ModLoad imudp # network reception +$UDPServerRun 514 # start a udp server at port 514 +$ModLoad imuxsock # local message reception + +$WorkDirectory /rsyslog/work # default location for work (spool) files + +$ActionQueueType LinkedList # use asynchronous processing +$ActionQueueFileName dbq # set file name, also enables disk mode +$ActionResumeRetryCount -1 # infinite retries on insert failure +# for PostgreSQL replace :ommysql: by :ompgsql: below: +*.* :ommysql:hostname,dbname,userid,password; +</textarea> +<p><b>This is the recommended configuration for this use case.</b> It requires +rsyslog 3.11.0 or above.</p> +<p>In this example, the main message queue is NOT disk-assisted (there is no +$MainMsgQueueFileName directive). We still could do that, but have not done it +because there seems to be no need. The only slow running action is the database +writer and it has its own queue. So there is no real reason to use a large main +message queue (except, of course, if you expect *really* heavy traffic bursts).</p> +<p>Note that you can modify a lot of queue performance parameters, but the above +config will get you going with default values. If you consider using this on a real +busy server, it is strongly recommended to invest some time in setting the tuning +parameters to appropriate values.</p> + +<h3>Feedback requested</h3> + +<P>I would appreciate feedback on this tutorial. If you have additional ideas, + +comments or find bugs (I *do* bugs - no way... ;)), please + +<a href="mailto:rgerhards@adiscon.com">let me know</a>.</P> + +<h2>Revision History</h2> + +<ul> + + <li>2008-01-28 * + + <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> * Initial Version created</li> + <li>2008-01-28 * + + <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> + * Updated to new v3.11.0 capabilities</li> + +</ul> +<h2>Copyright</h2> + +<p>Copyright (c) 2008 + +<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> and + +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> + +<p> Permission is granted to copy, distribute and/or modify this document + + under the terms of the GNU Free Documentation License, Version 1.2 + + or any later version published by the Free Software Foundation; + + with no Invariant Sections, no Front-Cover Texts, and no Back-Cover + + Texts. A copy of the license can be viewed at + +<a href="http://www.gnu.org/copyleft/fdl.html"> + +http://www.gnu.org/copyleft/fdl.html</a>.</p> + + + +</body> + +</html> diff --git a/doc/rsyslog_mysql.html b/doc/rsyslog_mysql.html index 53ee30cc..753c86ec 100644 --- a/doc/rsyslog_mysql.html +++ b/doc/rsyslog_mysql.html @@ -1,249 +1,262 @@ -<html><head> -<title>Writing syslog Data to MySQL</title> -<meta name="KEYWORDS" content="syslog, mysql, syslog to mysql, howto"> -</head> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Writing syslog Data to MySQL</title> + +<meta name="KEYWORDS" content="syslog, mysql, syslog to mysql, howto"></head> <body> <h1>Writing syslog messages to MySQL</h1> - <P><small><i>Written by - <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer - Gerhards</a> (2005-08-02)</i></small></P> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-02-28)</i></small></p> <h2>Abstract</h2> <p><i><b>In this paper, I describe how to write <a href="http://www.monitorware.com/en/topics/syslog/">syslog</a> -messages to a <a href="http://www.mysql.com">MySQL</a> database.</b> Having -syslog messages in a database is often handy, especially when you intend to set -up a front-end for viewing them. This paper describes an approach -with <a href="http://www.rsyslog.com/">rsyslogd</a>, an alternative enhanced -syslog daemon natively supporting MySQL. I describe the components needed -to be installed and how to configure them.</i></p> +messages to a <a href="http://www.mysql.com">MySQL</a> +database.</b> Having syslog messages in a database is often +handy, especially when you intend to set up a front-end for viewing +them. This paper describes an approach with <a href="http://www.rsyslog.com/">rsyslogd</a>, +an +alternative enhanced syslog daemon natively supporting MySQL. I +describe the components needed to be installed and how to configure +them. Please note that as of this writing, rsyslog supports a variety +of databases. While this guide is still MySQL-focussed, you +can probably use it together with other ones too. You just need to +modify a few settings.</i></p> <h2>Background</h2> -<p>In many cases, syslog data is simply written to text files. This approach has -some advantages, most notably it is very fast and efficient. However, data -stored in text files is not readily accessible for real-time viewing and analysis. -To do that, the messages need to be in a database. There are various -ways to store syslog messages in a database. For example, some have the syslogd -write text files which are later feed via a separate script into the database. -Others have written scripts taking the data (via a pipe) from a -non-database-aware syslogd and store them as they appear. Some others use -database-aware syslogds and make them write the data directly to the database. -In this paper, I use that "direct write" approach. I think it is superior, -because the syslogd itself knows the status of the database connection and thus -can handle it intelligently (well ... hopefully ;)). I use rsyslogd to acomplish -this, simply because I have initiated the rsyslog project with -database-awareness as one goal.</p> -<p><b>One word of caution:</b> while message storage in the database provides an -excellent foundation for interactive analysis, it comes at a cost. Database i/o -is considerably slower than text file i/o. As such, directly writing to -the database makes sense only if your message volume is low enough to allow a) -the syslogd, b) the network, and c) the database server to catch -up with it. Some time ago, I have written a paper on -<a href="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php"> -optimizing syslog server performance</a>. While this paper talks about -Window-based solutions, the ideas in it are generic enough to apply here, too. -So it might be worth reading if you anticipate medium high to high traffic. If you -anticipate really high traffic (or very large traffic spikes), you should -seriously consider forgetting about direct database writes - in my opinion, such -a situation needs either a very specialised system or a different approach (the -text-file-to-database approach might work better for you in this case). +<p>In many cases, syslog data is simply written to text files. +This approach has some advantages, most notably it is very fast and +efficient. However, data stored in text files is not readily accessible +for real-time viewing and analysis. To do that, the messages need to be +in a database. There are various ways to store syslog messages in a +database. For example, some have the syslogd write text files which are +later feed via a separate script into the database. Others have written +scripts taking the data (via a pipe) from a non-database-aware syslogd +and store them as they appear. Some others use database-aware syslogds +and make them write the data directly to the database. In this paper, I +use that "direct write" approach. I think it is superior, because the +syslogd itself knows the status of the database connection and thus can +handle it intelligently (well ... hopefully ;)). I use rsyslogd to +acomplish this, simply because I have initiated the rsyslog project +with database-awareness as one goal.</p> +<p><b>One word of caution:</b> while message storage +in the database provides an excellent foundation for interactive +analysis, it comes at a cost. Database i/o is considerably slower than +text file i/o. As such, directly writing to the database makes sense +only if your message volume is low enough to allow a) the syslogd, b) +the network, and c) the database server to catch up with it. Some time +ago, I have written a paper on +<a href="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php">optimizing +syslog server performance</a>. While this paper talks about +Window-based solutions, the ideas in it are generic enough to apply +here, too. So it might be worth reading if you anticipate medium high +to high traffic. If you anticipate really high traffic (or very large +traffic spikes), you should seriously consider forgetting about direct +database writes - in my opinion, such a situation needs either a very +specialised system or a different approach (the text-file-to-database +approach might work better for you in this case). </p> <h2>Overall System Setup</h2> -<P>In this paper, I concentrate on the server side. If you are thinking about -interactive syslog message review, you probably want to centralize syslog. In -such a scenario, you have multiple machines (the so-called clients) send their -data to a central machine (called server in this context). While I expect such a -setup to be typical when you are interested in storing messages in the database, -I do not describe how to set it up. This is beyond the scope of this paper. If -you search a little, you will probably find many good descriptions on how to -centralize syslog. If you do that, it might be a good idea to do it securely, so -you might also be interested in my paper on <a href="rsyslog_stunnel.html"> -ssl-encrypting syslog message transfer</a>.</P> -<P>No matter how the messages arrive at the server, their processing is always the -same. So you can use this paper in combination with any description for -centralized syslog reporting.</P> -<P>As I already said, I use rsyslogd on the server. It has intrinsic support for -talking to MySQL databases. For obvious reasons, we also need an instance of -MySQL running. To keep us focussed, the setup of MySQL itself is also beyond the -scope of this paper. I assume that you have successfully installed MySQL and -also have a front-end at hand to work with it (for example, -<a href="http://www.phpmyadmin.net/">phpMyAdmin</a>). Please make sure that this -is installed, actually working and you have a basic understanding of how to -handle it.</P> +<p>In this paper, I concentrate on the server side. If you are +thinking about interactive syslog message review, you probably want to +centralize syslog. In such a scenario, you have multiple machines (the +so-called clients) send their data to a central machine (called server +in this context). While I expect such a setup to be typical when you +are interested in storing messages in the database, I do not describe +how to set it up. This is beyond the scope of this paper. If you search +a little, you will probably find many good descriptions on how to +centralize syslog. If you do that, it might be a good idea to do it +securely, so you might also be interested in my paper on <a href="rsyslog_stunnel.html"> +ssl-encrypting syslog message transfer</a>.</p> +<p>No matter how the messages arrive at the server, their +processing is always the same. So you can use this paper in combination +with any description for centralized syslog reporting.</p> +<p>As I already said, I use rsyslogd on the server. It has +intrinsic support for talking to MySQL databases. For obvious reasons, +we also need an instance of MySQL running. To keep us focussed, the +setup of MySQL itself is also beyond the scope of this paper. I assume +that you have successfully installed MySQL and also have a front-end at +hand to work with it (for example, +<a href="http://www.phpmyadmin.net/">phpMyAdmin</a>). +Please make sure that this is installed, actually working and you have +a basic understanding of how to handle it.</p> <h2>Setting up the system</h2> -<p>You need to download and install rsyslogd first. Obtain it from the -<a href="http://www.rsyslog.com/">rsyslog site</a>. Make sure that you disable -stock syslogd, otherwise you will experience some difficulties.</p> -<p>It is important to understand how rsyslogd talks to the database. In rsyslogd, -there is the concept of "templates". Basically, a template is a string that -includes some replacement characters, which are called "properties" in rsyslog. -Properties are accessed via the "<a href="property_replacer.html">Property -Replacer</a>". Simply said, you access properties by including their name -between percent signs inside the template. For example, if the syslog message is -"Test", the template "%msg%" would be expanded to "Test". Rsyslogd supports -sending template text as a SQL statement to MySQL. As such, the template must be -a valid SQL statement. There is no limit in what the statement might be, but -there are some obvious and not so obvious choices. For example, a template "drop -table xxx" is possible, but does not make an awful lot of sense. In practice, -you will always use an "insert" statment inside the template.</p> -<p>An example: if you would just like to store the msg part of the full syslog -message, you have probably created a table "syslog" with a single column "message". -In such a case, a good template would be "insert into syslog(message) values ('%msg%')". -With the example above, that would be expanded to "insert into syslog(message) -values('Test')". This expanded string is then sent to the database. It's that -easy, no special magic. The only thing you must ensure is that your template -expands to a proper SQL statement and that this statement matches your database -design.</p> -<p>Does that mean you need to create database schema yourself and also must -fully understand rsyslogd's properties? No, that's not needed. Because we -anticipated that folks are probably more interested in getting things going instead -of designing them from scratch. So we have provided a default schema as well -as build-in support for it. This schema also offers an additional -benefit: rsyslog is part of <a href="http://www.adiscon.com/en/">Adiscon</a>'s -<a href="http://www.monitorware.com/en/">MonitorWare product line</a> (which -includes open source and closed source members). All of these tools share the -same default schema and know how to operate on it. For this reason, the default -schema is also called the "MonitorWare Schema". If you use it, you can simply -add <a href="http://www.phplogcon.org/">phpLogCon, a GPLed syslog web interface</a>, -to your system and have instant interactive access to your database. So there -are some benefits in using the provided schema.</p> -<p>The schema definition is contained in the file "createDB.sql". It comes with -the rsyslog package. Review it to check that the database name is acceptable for -you. Be sure to leave the table and field names unmodified, because -otherwise you need to customize rsyslogd's default sql template, which we do not -do -in this paper. Then, run the script with your favourite MySQL tool. Double-check -that the table was successfully created.</p> - -<p>MySQL support in rsyslog is integrated via a loadable plug-in module. To use the database -functionality, MySQL must be enabled in the config file BEFORE the first database table action is +<p>You need to download and install rsyslogd first. Obtain it +from the +<a href="http://www.rsyslog.com/">rsyslog site</a>. +Make sure that you disable stock syslogd, otherwise you will experience +some difficulties. On some distributions (Fedora 8 and above, for +example), rsyslog may already by the default syslogd, in which case you +obviously do not need to do anything specific. For many others, there +are prebuild packages available. If you use either, please make sure +that you have the required database plugins for your database +available. It usually is a separate package and typically <span style="font-weight: bold;">not</span> installed by default.</p> +<p>It is important to understand how rsyslogd talks to the +database. In rsyslogd, there is the concept of "templates". Basically, +a template is a string that includes some replacement characters, which +are called "properties" in rsyslog. Properties are accessed via the "<a href="property_replacer.html">Property Replacer</a>". +Simply said, you access properties by including their name between +percent signs inside the template. For example, if the syslog message +is "Test", the template "%msg%" would be expanded to "Test". Rsyslogd +supports sending template text as a SQL statement to MySQL. As such, +the template must be a valid SQL statement. There is no limit in what +the statement might be, but there are some obvious and not so obvious +choices. For example, a template "drop table xxx" is possible, but does +not make an awful lot of sense. In practice, you will always use an +"insert" statment inside the template.</p> +<p>An example: if you would just like to store the msg part of +the full syslog message, you have probably created a table "syslog" +with a single column "message". In such a case, a good template would +be "insert into syslog(message) values ('%msg%')". With the example +above, that would be expanded to "insert into syslog(message) +values('Test')". This expanded string is then sent to the database. +It's that easy, no special magic. The only thing you must ensure is +that your template expands to a proper SQL statement and that this +statement matches your database design.</p> +<p>Does that mean you need to create database schema yourself and +also must fully understand rsyslogd's properties? No, that's not +needed. Because we anticipated that folks are probably more interested +in getting things going instead of designing them from scratch. So we +have provided a default schema as well as build-in support for it. This +schema also offers an additional benefit: rsyslog is part of <a href="http://www.adiscon.com/en/">Adiscon</a>'s +<a href="http://www.monitorware.com/en/">MonitorWare +product line</a> (which includes open source and closed source +members). All of these tools share the same default schema and know how +to operate on it. For this reason, the default schema is also called +the "MonitorWare Schema". If you use it, you can simply add <a href="http://www.phplogcon.org/">phpLogCon, a GPLed syslog +web interface</a>, to your system and have instant interactive +access to your database. So there are some benefits in using the +provided schema.</p> +<p>The schema definition is contained in the file "createDB.sql". +It comes with the rsyslog package. Review it to check that the database +name is acceptable for you. Be sure to leave the table and field names +unmodified, because otherwise you need to customize rsyslogd's default +sql template, which we do not do in this paper. Then, run the script +with your favourite MySQL tool. Double-check that the table was +successfully created.</p> +<p>MySQL support in rsyslog is integrated via a loadable plug-in +module. To use the database +functionality, MySQL must be enabled in the config file BEFORE the +first database table action is used. This is done by placing the</p> <blockquote> - <p><code>$ModLoad MySQL</code></p> +<p><code>$ModLoad ommysql</code></p> </blockquote> -<p>directive at the begining of /etc/rsyslog.conf</p> - -<p>Next, we need to tell rsyslogd to write data to the database. As we use -the default schema, we do NOT need to define a template for this. We can use the -hardcoded one (rsyslogd handles the proper template linking). So all we need to -do is add a simple selector line to /etc/rsyslog.conf:</p> +<p>directive at the begining of /etc/rsyslog.conf. For other databases, use their plugin name (e.g. ompgsql).</p> +<p>Next, we need to tell rsyslogd to write data to the database. +As we use the default schema, we do NOT need to define a template for +this. We can use the hardcoded one (rsyslogd handles the proper +template linking). So all we need to do is add a simple selector line +to /etc/rsyslog.conf:</p> <blockquote> - <p><code>*.* - >database-server,database-name,database-userid,database-password</code></p> +<p><code>*.* :ommysql:database-server,database-name,database-userid,database-password</code></p> </blockquote> -<p>In many cases, MySQL will run on the local machine. In this case, you can -simply use "127.0.0.1" for <i>database-server</i>. This can be especially -advisable, if you do not need to expose MySQL to any process outside of the -local machine. In this case, you can simply bind it to 127.0.0.1, which provides -a quite secure setup. Of course, also supports remote MySQL instances. In that -case, use the remote server name (e.g. mysql.example.com) or IP-address. The <i> -database-name</i> by default is "syslog". If you have modified the default, use -your name here. <i>Database-userid</i> and <i>-password</i> are the credentials -used to connect to the database. As they are stored in clear text in -rsyslog.conf, that user should have only the least possible privileges. It is -sufficient to grant it INSERT privileges to the systemevents table, only. As a -side note, it is strongly advisable to make the rsyslog.conf file readable by -root only - if you make it world-readable, everybody could obtain the password -(and eventually other vital information from it). In our example, let's assume -you have created a MySQL user named "syslogwriter" with a password of -"topsecret" (just to say it bluntly: such a password is NOT a good idea...). If -your MySQL database is on the local machine, your rsyslog.conf line might look -like in this sample:</p> +<p>Again, other databases have other selector names, e.g. ":ompgsql:" +instead of ":ommysql:". See the output plugin's documentation for +details.</p><p>In many cases, MySQL will run on the local machine. In this +case, you can simply use "127.0.0.1" for <i>database-server</i>. +This can be especially advisable, if you do not need to expose MySQL to +any process outside of the local machine. In this case, you can simply +bind it to 127.0.0.1, which provides a quite secure setup. Of course, +also supports remote MySQL instances. In that case, use the remote +server name (e.g. mysql.example.com) or IP-address. The <i> +database-name</i> by default is "syslog". If you have modified +the default, use your name here. <i>Database-userid</i> +and <i>-password</i> are the credentials used to connect +to the database. As they are stored in clear text in rsyslog.conf, that +user should have only the least possible privileges. It is sufficient +to grant it INSERT privileges to the systemevents table, only. As a +side note, it is strongly advisable to make the rsyslog.conf file +readable by root only - if you make it world-readable, everybody could +obtain the password (and eventually other vital information from it). +In our example, let's assume you have created a MySQL user named +"syslogwriter" with a password of "topsecret" (just to say it bluntly: +such a password is NOT a good idea...). If your MySQL database is on +the local machine, your rsyslog.conf line might look like in this +sample:</p> <blockquote> - <p><code>*.* - >127.0.0.1,syslog,syslogwriter,topsecret</code></p> +<p><code>*.* :ommysql:127.0.0.1,Syslog,syslogwriter,topsecret</code></p> </blockquote> -<p>Save rsyslog.conf, restart rsyslogd - and you should see syslog messages -being stored in the "systemevents" table!</p> -<p>The example line stores every message to the database. Especially if you have -a high traffic volume, you will probably limit the amount of messages being -logged. This is easy to acomplish: the "write database" action is just a regular -selector line. As such, you can apply normal selector-line filtering. If, for -example, you are only interested in messages from the mail subsystem, you can -use the following selector line:</p> +<p>Save rsyslog.conf, restart rsyslogd - and you should see +syslog messages being stored in the "systemevents" table!</p> +<p>The example line stores every message to the database. +Especially if you have a high traffic volume, you will probably limit +the amount of messages being logged. This is easy to acomplish: the +"write database" action is just a regular selector line. As such, you +can apply normal selector-line filtering. If, for example, you are only +interested in messages from the mail subsystem, you can use the +following selector line:</p> <blockquote> - <p><code>mail.* - >127.0.0.1,syslog,syslogwriter,topsecret</code></p> +<p><code>mail.* </code><code>:ommysql:</code><code>127.0.0.1,syslog,syslogwriter,topsecret</code></p> </blockquote> -<p>Review the <a href="rsyslog_conf.html">rsyslog.conf</a> documentation for -details on selector lines and their filtering.</p> -<p><b>You have now completed everything necessary to store syslog messages to -the MySQL database.</b> If you would like to try out a front-end, you might want -to look at <a href="http://www.phplogcon.org/">phpLogCon</a>, which displays -syslog data in a browser. As of this writing, phpLogCon is not yet a powerful -tool, but it's open source, so it might be a starting point for your own -solution.</p> +<p>Review the <a href="rsyslog_conf.html">rsyslog.conf</a> +documentation for details on selector lines and their filtering.</p> +<p><b>You have now completed everything necessary to store +syslog messages to the MySQL database.</b> If you would like to +try out a front-end, you might want to look at <a href="http://www.phplogcon.org/">phpLogCon</a>, which +displays syslog data in a browser. As of this writing, phpLogCon is not +yet a powerful tool, but it's open source, so it might be a starting +point for your own solution.</p> <h2>On Reliability...</h2> -<p><b>This section needs updating. You can now solve the issue with failover -database servers. Read the <a href="rsyslog_conf.html">rsyslog.conf </a>doc on -that</b>.</p> -<p>Rsyslogd writes syslog messages directly to the database. This implies that -the database must be available at the time of message arrival. If the database -is offline, no space is left or something else goes wrong - rsyslogd can not -write the database record. If rsyslogd is unable to store a message, it performs -one retry. This is helpful if the database server was restarted. In this case, -the previous connection was broken but a reconnect immediately succeeds. However, -if the database is down for an extended period of time, an immediate retry does -not help. While rsyslogd could retry until it finally succeeds, that would have -negative impact. Syslog messages keep coming in. If rsyslogd would be busy -retrying the database, it would not be able to process these messages. -Ultimately, this would lead to loss of newly arrived messages.</p> -<p>In most cases, rsyslogd is configured not only to write to the database but -to perform other actions as well. In the always-retry scenario, that would mean -no other actions would be carried out. As such, the design of rsyslogd is -limited to a single retry. If that does not succeed, the current message is will -not be written to the database and the MySQL database writer be suspended for a -short period of time. Obviously, this leads to the loss of the current message -as well as all messages received during the suspension period. But they are only -lost in regard to the database, all other actions are correctly carried out. -While not perfect, we consider this to be a better approach then the potential -loss of all messages in all actions.</p> -<p><b>In short: try to avoid database downtime if you do not want to experience -message loss.</b></p> -<p>Please note that this restriction is not rsyslogd specific. All approaches to -real-time database storage share this problem area.</p> +<p>Rsyslogd writes syslog messages directly to the database. This +implies that the database must be available at the time of message +arrival. If the database is offline, no space is left or something else +goes wrong - rsyslogd can not write the database record. If rsyslogd is +unable to store a message, it performs one retry. This is helpful if +the database server was restarted. In this case, the previous +connection was broken but a reconnect immediately succeeds. However, if +the database is down for an extended period of time, an immediate retry +does not help.</p> +<p>Message loss in this scenario can easily be prevented with +rsyslog. All you need to do is run the database writer in queued mode. +This is now described in a generic way and I do not intend to duplicate +it here. So please be sure to read "<a href="rsyslog_high_database_rate.html">Handling a massive +syslog database insert rate with Rsyslog</a>", which describes +the scenario and also includes configuration examples.</p> <h2>Conclusion</h2> -<P>With minimal effort, you can use rsyslogd to write syslog messages to a MySQL -database. Once the messages are arrived there, you can interactivley review and -analyse them. In practice, the messages are also stored in text files for -longer-term archival and the databases are cleared out after some time (to avoid -becoming too slow). If you expect an extremely high syslog message volume, -storing it in real-time to the database may outperform your database server. In -such cases, either filter out some messages or think about alternate approaches -involving non-real-time database writing (beyond the scope of this paper).</P> -<P>The method outlined in this paper provides an easy to setup and maintain -solution for most use cases, especially with low and medium syslog message -volume (or fast database servers).</P> +<p>With minimal effort, you can use rsyslogd to write syslog +messages to a MySQL database. You can even make it absolutely fail-safe +and protect it against database server downtime. Once the messages are +arrived there, you +can interactivley review and analyse them. In practice, the messages +are also stored in text files for longer-term archival and the +databases are cleared out after some time (to avoid becoming too slow). +If you expect an extremely high syslog message volume, storing it in +real-time to the database may outperform your database server. In such +cases, either filter out some messages or used queued mode (which in +general is recommended with databases).</p> +<p>The method outlined in this paper provides an easy to setup +and maintain solution for most use cases.</p> <h3>Feedback Requested</h3> -<P>I would appreciate feedback on this paper. If you have additional ideas, -comments or find bugs, please -<a href="mailto:rgerhards@adiscon.com">let me know</a>.</P> +<p>I would appreciate feedback on this paper. If you have +additional ideas, comments or find bugs, please +<a href="mailto:rgerhards@adiscon.com">let me know</a>.</p> <h2>References and Additional Material</h2> <ul> - <li><a href="http://www.rsyslog.com">www.rsyslog.com</a> - the rsyslog site</li> - <li> - <a href="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php"> - Paper on Syslog Server Optimization</a></li> +<li><a href="http://www.rsyslog.com">www.rsyslog.com</a> +- the rsyslog site</li> +<li> <a href="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php"> +Paper on Syslog Server Optimization</a></li> </ul> <h2>Revision History</h2> <ul> - <li>2005-08-02 * - <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> * - initial version created</li> - <li>2005-08-03 * - <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> - * added references to demo site</li> - <li>2007-06-13 * - <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> - * removed demo site - was torn down because too expensive for usage count</li> +<li>2005-08-02 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> * initial version created</li> +<li>2005-08-03 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> * added references to demo site</li> +<li>2007-06-13 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> * removed demo site - was torn down because too +expensive for usage count</li> +<li>2008-02-21 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> * updated reliability section, can now be done with +on-demand disk queues</li><li>2008-02-28 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> * added info on other databases, updated syntax to more recent one</li> </ul> <h2>Copyright</h2> -<p>Copyright (c) 2005-2007 -<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> -and <a href="http://www.adiscon.com/en/">Adiscon</a>.</p> -<p>Permission is granted to copy, distribute and/or modify this document under -the terms of the GNU Free Documentation License, Version 1.2 or any later -version published by the Free Software Foundation; with no Invariant Sections, -no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be -viewed at <a href="http://www.gnu.org/copyleft/fdl.html"> +<p>Copyright (c) 2005-2008 +<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and <a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p>Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at <a href="http://www.gnu.org/copyleft/fdl.html"> http://www.gnu.org/copyleft/fdl.html</a>.</p> -</body> -</html> +</body></html> diff --git a/doc/rsyslog_ng_comparison.html b/doc/rsyslog_ng_comparison.html new file mode 100644 index 00000000..2f383f78 --- /dev/null +++ b/doc/rsyslog_ng_comparison.html @@ -0,0 +1,587 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>rsyslog vs. syslog-ng - a comparison</title></head> +<body> +<h1>rsyslog vs. syslog-ng</h1> +<p><small><i>Written by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> +(2008-05-06)</i></small></p> +<p>We have often been asked about a comparison sheet between +rsyslog and syslog-ng. Unfortunately, I do not know much about +syslog-ng, I did not even use it once. Also, there seems to be no +comprehensive feature sheet available for syslog-ng (that recently +changed, see below). So I started this +comparison, but it probably is not complete. For sure, I miss some +syslog-ng features. This is not an attempt to let rsyslog shine more +than it should. I just used the <a href="features.html">rsyslog +feature sheet</a> as a starting point, simply because it was +available. If you would like to add anything to the chart, or correct +it, please simply <a href="mailto:rgerhards@adiscon.com">drop +me a line</a>. I would love to see a real honest and up-to-date +comparison sheet, so please don't be shy ;)</p> +<table border="1"> +<tbody> +<tr> +<td valign="top"><b>Feature</b></td> +<td valign="top"><b>rsyslog</b></td> +<td valign="top"><b>syslog-ng</b></td> +</tr> +<tr> +<td colspan="3" valign="top"><br> +<b>Input Sources</b><br> +</td> +</tr> +<tr> +<td valign="top">UNIX domain socket</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +<td></td> +</tr> +<tr> +<td valign="top">UDP</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +<td></td> +</tr> +<tr> +<td valign="top">TCP</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +<td></td> +</tr> +<tr> +<td valign="top"><a href="http://www.librelp.com">RELP</a></td> +<td valign="top">yes</td> +<td valign="top">no</td> +<td></td> +</tr> +<tr> +<td valign="top">RFC 3195/BEEP</td> +<td valign="top">yes (via <a href="im3195.html">im3195</a>)</td> +<td valign="top">no</td> +<td></td> +</tr> +<tr> +<td valign="top">kernel log</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +<td></td> +</tr> +<tr> +<td valign="top">file</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +<td></td> +</tr> +<tr> +<td valign="top">mark message generator as an +optional input</td> +<td valign="top">yes</td> +<td valign="top">no (?)</td> +<td></td> +</tr> +<tr> +<td valign="top">Windows Event Log</td> +<td valign="top">via <a href="http://www.eventreporter.com">EventReporter</a> +or <a href="http://www.mwagent.com">MonitorWare Agent</a> +(both commercial software)</td> +<td valign="top">via separate Windows agent, paid +edition only</td> +</tr> +<tr> +<td colspan="3" valign="top"><b><br> +Network (Protocol) Support</b><br> +</td> +</tr> +<tr> +<td valign="top">support for (plain) tcp based syslog</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">support for GSS-API</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td valign="top">ability to limit the allowed +network senders (syslog ACLs)</td> +<td valign="top">yes</td> +<td valign="top">yes (?)</td> +</tr> +<tr> +<td valign="top">support for syslog-transport-tls +based framing on syslog/tcp connections</td> +<td valign="top">yes</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td valign="top">udp syslog</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">syslog over RELP<br> +truly reliable message delivery (<a href="http://blog.gerhards.net/2008/05/why-you-cant-build-reliable-tcp.html">Why +is plain tcp syslog not reliable?</a>)</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td valign="top">on the wire (zlib) message +compression</td> +<td valign="top">yes</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td valign="top">support for receiving messages via +reliable <a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php">RFC +3195</a> delivery</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td valign="top">support for <a href="rsyslog_tls.html">TLS/SSL-protected +syslog</a> </td> +<td valign="top"><a href="rsyslog_tls.html">natively</a> (since 3.19.0)<br><a href="rsyslog_stunnel.html">via +stunnel</a></td> +<td valign="top">via stunnel<br> +paid edition natively</td> +</tr> +<tr> +<td valign="top">support for IETF's new syslog-protocol draft</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td valign="top">support for IETF's new syslog-transport-tls draft</td> +<td valign="top">yes<br>(since 3.19.0 - world's first implementation)</td> +<td valign="top">no</td> +</tr> +<tr> +<td valign="top">support for IPv6</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">native ability to send SNMP traps</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td valign="top">ability to preserve the original +hostname in NAT environments and relay chains</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td colspan="3" valign="top"><br> +<b>Message Filtering</b><br> +</td> +</tr> +<tr> +<td valign="top">Filtering for syslog facility and +priority</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +<td></td> +</tr> +<tr> +<td valign="top">Filtering for hostname</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +<td></td> +</tr> +<tr> +<td valign="top">Filtering for application</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +<td></td> +</tr> +<tr> +<td valign="top">Filtering for message contents</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +<td></td> +</tr> +<tr> +<td valign="top">Filtering for sending IP address</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +<td></td> +</tr> +<tr> +<td valign="top">ability to filter on any other message +field not mentioned above (including substrings and the like)</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td>support for complex filters, using full boolean algebra +with and/or/not operators and parenthesis</td> +<td>yes</td> +<td>yes</td> +</tr> +<tr> +<td>Support for reusable filters: specify a filter once and +use it in multiple selector lines</td> +<td>no</td> +<td>yes</td> +</tr> +<tr> +<td>support for arbritrary complex arithmetic and string +expressions inside filters</td> +<td>yes</td> +<td>no</td> +</tr> +<tr> +<td valign="top">ability to use regular expressions +in filters</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">support for discarding messages +based on filters</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +<td></td> +</tr> +<tr> +<td valign="top">ability to filter out messages based on sequence of appearing</td> +<td valign="top">yes (starting with 3.21.3)</td> +<td valign="top">no</td> +<td></td> +</tr> +<tr> +<td valign="top">powerful BSD-style hostname and +program name blocks for easy multi-host support</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td></td> +<td></td> +<td></td> +</tr> +<tr> +<td colspan="3" valign="top"><br> +<b>Supported Database Outputs</b><br> +</td> +</tr> +<tr> +<td valign="top">MySQL</td> +<td valign="top"><a href="rsyslog_mysql.html">yes</a> +(native ommysql, <a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">yes (via libdibi)</td> +</tr> +<tr> +<td valign="top">PostgreSQL</td> +<td valign="top">yes (native ompgsql, <a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">yes (via libdibi)</td> +</tr> +<tr> +<td valign="top">Oracle</td> +<td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">yes (via libdibi)</td> +</tr> +<tr> +<td valign="top">SQLite</td> +<td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">yes (via libdibi)</td> +</tr> +<tr> +<td valign="top">Microsoft SQL (Open TDS)</td> +<td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td valign="top">Sybase (Open TDS)</td> +<td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td valign="top">Firebird/Interbase</td> +<td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td valign="top">Ingres</td> +<td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td valign="top">mSQL</td> +<td valign="top">yes (<a href="omlibdbi.html">omlibdbi</a>)</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td colspan="3" valign="top"><br> +<b>Enterprise Features</b><br> +</td> +</tr> +<tr> +<td valign="top">support for on-demand on-disk +spooling of messages</td> +<td valign="top">yes</td> +<td valign="top">paid edition only</td> +</tr> +<tr> +<td valign="top">ability to limit disk space used +by spool files</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">each action can use its own, +independant +set of spool files</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td valign="top">different sets of spool files can +be placed on different disk</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td valign="top">ability to process spooled +messages only during a configured timeframe (e.g. process messages only +during off-peak hours, during peak hours they are enqueued only)</td> +<td valign="top"><a href="http://wiki.rsyslog.com/index.php/OffPeakHours">yes</a><br> +(can independently be configured for the main queue and each action +queue)</td> +<td valign="top">no</td> +</tr> +<tr> +<td valign="top">ability to configure backup +syslog/database servers </td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td>Professional Support</td> +<td><a href="professional_support.html">yes</a></td> +<td>yes</td> +</tr> +<tr> +<td colspan="3" valign="top"><br> +<b>Config File</b><br> +</td> +</tr> +<tr> +<td valign="top">config file format</td> +<td valign="top">compatible to legacy syslogd but +ugly</td> +<td valign="top">clean but not backwards compatible</td> +</tr> +<tr> +<td valign="top">ability to include config file from +within other config files</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td height="25" valign="top">ability to +include all config files +existing in a specific directory</td> +<td height="25" valign="top">yes</td> +<td height="25" valign="top">no</td> +</tr> +<tr> +<td colspan="3" valign="top"><br> +<b>Extensibility</b><br> +</td> +</tr> +<tr> +<td valign="top">Functionality split in separately +loadable +modules</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td valign="top">Support for third-party input +plugins</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +</tr> +<tr> +<td valign="top">Support for third-party output +plugins</td> +<td valign="top">yes</td> +<td valign="top">no</td> +</tr> +<tr> +<td colspan="3" valign="top"><br> +<b>Other Features</b><br> +</td> +</tr> +<tr> +</tr> +<tr> +<td valign="top">ability to generate file names and +directories (log targets) dynamically</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">control of log output format, +including ability to present channel and priority as visible log data</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr><td valign="top">native ability to send mail messages</td> +<td valign="top">yes (<a href="ommail.html">ommail</a>, introduced in 3.17.0)</td> +<td valign="top">no (only via piped external process)</td> +</tr> +<tr> +<td valign="top">good timestamp format control; at a +minimum, ISO 8601/RFC 3339 second-resolution UTC zone</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">ability to reformat message +contents and work with substrings</td> +<td valign="top">yes</td> +<td valign="top">I think yes</td> +</tr> +<tr> +<td valign="top">support for log files larger than +2gb</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">support for log file size +limitation +and automatic rollover command execution</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">support for running multiple +syslogd instances on a single machine</td> +<td valign="top">yes</td> +<td valign="top">? (but I think yes)</td> +</tr> +<tr> +<td valign="top">ability to execute shell scripts on +received messages</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">ability to pipe messages to a +continously running program</td> +<td valign="top">no</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">massively multi-threaded for +tomorrow's multi-core machines</td> +<td valign="top">yes</td> +<td valign="top">no (only multithreaded with +database destinations)</td> +</tr> +<tr> +<td valign="top">ability to control repeated line +reduction ("last message repeated n times") on a per selector-line basis</td> +<td valign="top">yes</td> +<td valign="top">yes (?)</td> +</tr> +<tr> +<td valign="top">supports multiple actions per +selector/filter condition</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +<td></td> +</tr> +<tr> +<td valign="top">web interface</td> +<td valign="top"><a href="http://www.phplogcon.org">phpLogCon</a><br> +[also works with <a href="http://freshmeat.net/projects/php-syslog-ng/"> +php-syslog-ng</a>]</td> +<td valign="top"><a href="http://freshmeat.net/projects/php-syslog-ng/"> +php-syslog-ng</a></td> +</tr> +<tr> +<td valign="top">using text files as input source</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">rate-limiting output actions</td> +<td valign="top">yes</td> +<td valign="top">yes</td> +</tr> +<tr> +<td valign="top">discard low-priority messages under +system stress</td> +<td valign="top">yes</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td height="43" valign="top">flow control +(slow down message reception when system is busy)</td> +<td height="43" valign="top">yes (advanced, +with multiple ways to slow down inputs depending on individual input +capabilities, based on watermarks)</td> +<td height="43" valign="top">yes (limited? +"stops accepting messages")</td> +</tr> +<tr> +<td valign="top">rewriting messages</td> +<td valign="top">yes</td> +<td valign="top">yes (at least I think so...)</td> +</tr> +<tr> +<td valign="top">output data into various formats</td> +<td valign="top">yes</td> +<td valign="top">yes (looks somewhat limited to me)</td> +</tr> +<tr> +<td valign="top">ability to control "message +repeated n times" generation</td> +<td valign="top">yes</td> +<td valign="top">no (?)</td> +</tr> +<tr> +<td valign="top">license</td> +<td valign="top">GPLv3 (GPLv2 for v2 branch)</td> +<td valign="top">GPL (paid edition is closed source)</td> +</tr> +<tr> +<td valign="top">supported platforms</td> +<td valign="top">Linux, BSD, anecdotical seen on +Solaris; compilation and basic testing done on HP UX</td> +<td valign="top">many popular *nixes</td> +</tr> +<tr> +<td valign="top">DNS cache</td> +<td valign="top">no</td> +<td valign="top">yes</td> +</tr> +</tbody> +</table> +<p>While the <span style="font-weight: bold;">rsyslog</span> +project was initiated in 2004, it <span style="font-weight: bold;">is +build on the main author's (Rainer Gerhards) 12+ years of +logging experience</span>. Rainer, for example, also +wrote the first <a href="http://www.winsyslog.com/Common/en/News/WinSyslog-1996-03-31.php">Windows +syslog server</a> in early 1996 and invented the <a href="http://www.eventreporter.com/Common/en/News/EvntSLog-1997-03-23.php">eventlog-to-syslog</a> +class of applications in early 1997. He did custom logging development +and consulting even before he wrote these products. Rsyslog draws on +that vast experience and sometimes even on the code.</p> +<p>Based on a discussion I had, I also wrote about the <b>political +argument why it is good to have another strong syslogd besides syslog-ng</b>. +You may want to read it at my blog at "<a href="http://rgerhards.blogspot.com/2007/08/why-does-world-need-another-syslogd.html">Why +does the world need another syslogd?</a>".</p> +<p>Balabit, the vendor of syslog-ng, has just recently done a +feature sheet. I have not yet been able to fully work through it. In +the mean time, you may want to read it in parallel. It is available at +<a href="http://www.balabit.com/network-security/syslog-ng/features/detailed/">Balabit's +site</a>.</p> +</body></html> diff --git a/doc/rsyslog_packages.html b/doc/rsyslog_packages.html index 38c43c93..80ba96c5 100644 --- a/doc/rsyslog_packages.html +++ b/doc/rsyslog_packages.html @@ -5,43 +5,72 @@ <body> <h1>rsyslog packages</h1> <p><b>Thanks to some volunteers, rsyslog is also available in package form on -some distributions.</b> All available packages are listed below. If you would +some distributions.</b> All currently known packages are listed below. If I have forgotten +one or if you would like to maintain a package for a new distribution, please mail me at <a href="mailto:rgerhards@adiscon.com">rgerhards@adiscon.com</a>. Any help is *deeply* appreciated. While I create the core daemon, the package maintainers are really filling it with life, making it available to the average user. I am very grateful for that!</p> -<p>This list has last been updated on 2007-07-06 by +<p>This list has last been updated on 2008-07-11 by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a>. New packages may appear at any time, so be sure to check this page whenever you need a new one.</p> -<p>Red Hat has recently begun to build RPMs for rsyslog. The URL changes, but a -good place to look for them is at -<a href="http://freshmeat.net/projects/rsyslog/">freshmeat's rsyslog info page</a>. -Please note that the Red Hat RPMs do currently <b>not</b> include MySQL -functionality. If you need that, you need to compile from the source tarball -(which honestly is quite easy).</p> -<h2>BSD</h2> -<p>Give <a href="http://www.freshports.org/sysutils/rsyslog/"> -http://www.freshports.org/sysutils/rsyslog/</a> a try.</p> - -<h2>CentOS 4.3</h2> -<a href="http://www.se-community.com/~james/rsyslog/"> -http://www.se-community.com/~james/rsyslog/</a></p> -<p>Maintained by<b> James Bergamin.</b></p> - -<h2>openSUSE</h2> -<a href="http://download.opensuse.org/repositories/home:/darix/"> -http://download.opensuse.org/repositories/home:/darix/</a></p> -<p>Maintained by<b> darix</b></p> - -<h2>Almost any Linux</h2> -<p><b>Bennet Todd</b> maintains packages that should work on almost any Linux. +<ul> +<li><b>BSD</b> (maintained by infofarmer) + <ul> + <li><a href="http://www.freshports.org/sysutils/rsyslog/"> http://www.freshports.org/sysutils/rsyslog/</a> + </ul> + +<li><b>CentOS 4.3</b> (maintained by James Bergamin) + <ul> + <li><a href="http://www.se-community.com/~james/rsyslog/"> +http://www.se-community.com/~james/rsyslog/</a> + </ul> + +<li><b>Debian</b> (maintained by Michael Biebl) + <ul> + <li><a href="http://packages.debian.org/sid/rsyslog">http://packages.debian.org/sid/rsyslog</a> + </ul> + +<li><b>Fedora</b> + <ul> + <li>Starting with Fedora 8, rsyslog is available as part of the core distribution. + </ul> + +<li><b>openSUSE</b> (maintained by darix) + <ul> + <li><a href="http://download.opensuse.org/repositories/home:/darix/">http://download.opensuse.org/repositories/home:/darix/</a> + </ul> + +<li><b>Red Hat Enterprise Linux</b> + <ul> + <li>Starting with RHEL 5.2, rsyslog is available as part of the core distribution. + </ul> + +<li><b>Ubuntu</b> + <ul> + <li>Starting with hardy, rsyslog is available from the universe repository. + </ul> + +<li>Almost any Linux</h2> + <ul> + <li>Bennet Todd maintains packages that should work on almost any Linux. He keeps a current i386 tree. There is also a PPC tree, but that one is not paid -much attention for (anyhow, it is known to typically work well, too).</p> -<p>Please visit <a href="http://bent.latency.net/bent/"> +much attention for (anyhow, it is known to typically work well, too). +Please visit <a href="http://bent.latency.net/bent/"> http://bent.latency.net/bent/</a>, select the relevant tree and then do a search -for rsyslog.</p> +for rsyslog. +Please note, however, that as of this writing the versions in this repository +have been aged a bit. So it may be worth trying to find some other places first. + </ul> +</ul> + +<p>Just in case you are interested, the list of distribution is sorted by alphabetic order +of the distribution name. +<p>If you do not find a suitable package for your distribution, there is no reason +to panic. It is quite simple to install rsyslog from the source tarball, so you +should consider that. </body> </html> diff --git a/doc/rsyslog_php_syslog_ng.html b/doc/rsyslog_php_syslog_ng.html index 9e722755..bf48a1eb 100644 --- a/doc/rsyslog_php_syslog_ng.html +++ b/doc/rsyslog_php_syslog_ng.html @@ -107,7 +107,7 @@ server machine, "syslog" is the database name (default from the schema and "pass" are the logon credentials. Use a user with low privileges, insert into the logs table is sufficient. "syslog-ng" is the template name and tells rsyslogd to use the SQL statement shown above.</p> -<p>Once you have made the changes, all you need to do is reload (or HUP) +<p>Once you have made the changes, all you need to do is restart rsyslogd. Then, you should see syslog messages flow into your database - and show up in php-syslog-ng.</p> <h2>Conclusion</h2> @@ -148,4 +148,4 @@ no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be viewed at <a href="http://www.gnu.org/copyleft/fdl.html"> http://www.gnu.org/copyleft/fdl.html</a>.</p> </body> -</html>
\ No newline at end of file +</html> diff --git a/doc/rsyslog_recording_pri.html b/doc/rsyslog_recording_pri.html index 48852ca2..cf11e3e5 100644 --- a/doc/rsyslog_recording_pri.html +++ b/doc/rsyslog_recording_pri.html @@ -1,5 +1,5 @@ <html><head> -<title>Writing syslog Data to MySQL</title> +<title>Recording the Priority of Syslog Messages</title> <meta name="KEYWORDS" content="syslog, mysql, syslog to mysql, howto"> </head> <body> @@ -62,7 +62,7 @@ semicolon:</p> <p>That's all you need to do. There is one common pitfall: you need to define the template before you use it in a selector line. Otherwise, you will receive an error.</p> -<p>Once you have applied the changes, you need to restart or HUP rsyslogd. It +<p>Once you have applied the changes, you need to restart rsyslogd. It will then pick the new configuration.</p> <h2>What if I do not want rsyslogd to be the standard syslogd?</h2> <p>If you do not want to switch to rsyslog, you can still use it as a setup aid. diff --git a/doc/rsyslog_reliable_forwarding.html b/doc/rsyslog_reliable_forwarding.html new file mode 100644 index 00000000..d04d9ead --- /dev/null +++ b/doc/rsyslog_reliable_forwarding.html @@ -0,0 +1,152 @@ +<html><head> +<title>Reliable Forwarding of syslog Messages (via plain TCP syslog)</title> +</head> +<body> +<h1>Reliable Forwarding of syslog Messages with Rsyslog</h1> + <P><small><i>Written by + <a href="http://www.gerhards.net/rainer">Rainer + Gerhards</a> (2008-06-27)</i></small></P> +<h2>Abstract</h2> +<p><i><b>In this paper, I describe how to forward +<a href="http://www.monitorware.com/en/topics/syslog/">syslog</a> + + messages (quite) reliable to a central rsyslog server.</b> +This depends on rsyslog being installed on the client system and +it is recommended to have it installed on the server system. Please note +that industry-standard +<a href="http://blog.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html">plain TCP syslog protocol is not fully reliable</a> +(thus the "quite reliable"). If you need a truely reliable solution, you need +to look into RELP (natively supported by rsyslog).</i></p> + +<h2>The Intention</h2> +<p>Whenever two systems talk over a network, something can go wrong. +For example, the communications link may go down, or a client or server may abort. +Even in regular cases, the server may be offline for a short period of time +because of routine maintenance. +<p>A logging system should be capable of avoiding message loss in situations where the +server is not reachable. To do so, unsent data needs to be buffered at the client while the +server is offline. Then, once the server is up again, this data is to be sent. +<p>This can easily be acomplished by rsyslog. In rsyslog, every action runs on its own queue +and each queue can be set to buffer data if the action is not ready. Of course, +you must be able to detect that "the action is not ready", which means the remote +server is offline. This can be detected with plain TCP syslog and RELP, but not with UDP. +So you need to use either of the two. In this howto, we use plain TCP syslog. +<p>Please note that we are using rsyslog-specific features. The are required on the +client, but not on the server. So the client system must run rsyslog (at least version 3.12.0), while on the +server another syslogd may be running, as long as it supports plain tcp syslog. +<p><b>The rsyslog queueing subsystem tries to buffer to memory. So even if the +remote server goes +offline, no disk file is generated.</b> File on disk are created only if there is +need to, for example if rsyslog runs out of (configured) memory queue space or needs +to shutdown (and thus persist yet unsent messages). Using main memory and going to the +disk when needed is a huge performance benefit. You do not need to care about it, +because, all of it is handled automatically and transparently by rsyslog.</p> +<h2>How To Setup</h2> +<p>First, you need to create a working directory for rsyslog. This is where it +stores its queue files (should need arise). You may use any location on your +local system. +<p>Next, you need to do is instruct rsyslog to use a +disk queue and then configure your action. There is nothing else to do. With the +following simple config file, you forward anything you receive to a remote server +and have buffering applied automatically when it goes down. This must be done on the +client machine.</p> +<textarea rows="9" cols="80"> +$ModLoad imuxsock # local message reception + +$WorkDirectory /rsyslog/work # default location for work (spool) files + +$ActionQueueType LinkedList # use asynchronous processing +$ActionQueueFileName srvrfwd # set file name, also enables disk mode +$ActionResumeRetryCount -1 # infinite retries on insert failure +$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down +*.* @@server:port +</textarea> +<p>The port given above is optional. It may not be specified, in which case you only +provide the server name. The "$ActionQueueFileName" is used to create queue files, should need +arise. This value must be unique inside rsyslog.conf. No two rules must use the same queue file. +Also, for obvious reasons, it must only contain those characters that can be used inside a +valid file name. Rsyslog possibly adds some characters in front and/or at the end of that name +when it creates files. So that name should not be at the file size name length limit (which +should not be a problem these days). +<p>Please note that actual spool files are only created if the remote server is down +<b>and</b> there is no more space in the in-memory queue. By default, a short failure +of the remote server will never result in the creation of a disk file as a couple of +hundered messages can be held in memory by default. [These parameters can be fine-tuned. However, +then you need to either fully understand how the queue works +(<a href="http://www.rsyslog.com/doc-queues.html">read elaborate doc</a>) or +use <a href="http://www.rsyslog.com/doc-professional_support.html">professional services</a> +to have it done based on +your specs ;) - what that means is that fine-tuning queue parameters is far from +being trivial...] +<p>If you would like to test if your buffering scenario works, you need to +stop, wait a while and restart you central server. Do <b>not</b> watch for files being created, +as this usually does not happen and never happens immediately. + +<h3>Forwarding to More than One Server</h3> +<p>If you have more than one server you would like to forward to, that's quickly done. +Rsyslog has no limit on the number or type of actions, so you can define as many targets +as you like. What is important to know, however, is that the full set of directives make +up an action. So you can not simply add (just) a second forwarding rule, but need to +duplicate the rule configuration as well. Be careful that you use different queue +file names for the second action, else you will mess up your system. +<p>A sample for forwarding to two hosts looks like this: +<p> +<textarea rows="20" cols="80"> +$ModLoad imuxsock # local message reception + +$WorkDirectory /rsyslog/work # default location for work (spool) files + +# start forwarding rule 1 +$ActionQueueType LinkedList # use asynchronous processing +$ActionQueueFileName srvrfwd1 # set file name, also enables disk mode +$ActionResumeRetryCount -1 # infinite retries on insert failure +$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down +*.* @@server1:port +# end forwarding rule 1 + +# start forwarding rule 2 +$ActionQueueType LinkedList # use asynchronous processing +$ActionQueueFileName srvrfwd2 # set file name, also enables disk mode +$ActionResumeRetryCount -1 # infinite retries on insert failure +$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down +*.* @@server2 +# end forwarding rule 2 +</textarea> +<p>Note the filename used for the first rule it is "srvrfwd1" and for the second it +is "srvrfwd2". I have used a server without port name in the second forwarding rule. +This was just to illustrate how this can be done. You can also specify a port there +(or drop the port from server1). +<p>When there are multiple action queues, they all work independently. Thus, if server1 +goes down, server2 still receives data in real-time. The client will <b>not</b> block +and wait for server1 to come back online. Similarily, server1's operation will not +be affected by server2's state. + +<h2>Some Final Words on Reliability ...</h2> +<p>Using plain TCP syslog provides a lot of reliability over UDP syslog. However, +plain TCP syslog is <b>not</b> a fully reliable transport. In order to get full reliability, +you need to use the RELP protocol. +<p>Folow the next link to learn more about +<a href="http://blog.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html">the +problems you may encounter with plain tcp syslog</a>. +<h3>Feedback requested</h3> +<P>I would appreciate feedback on this tutorial. If you have additional ideas, +comments or find bugs (I *do* bugs - no way... ;)), please +<a href="mailto:rgerhards@adiscon.com">let me know</a>.</P> +<h2>Revision History</h2> +<ul> + <li>2008-06-27 * + <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> * Initial Version created</li> +</ul> +<h2>Copyright</h2> +<p>Copyright (c) 2008 +<a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this document + under the terms of the GNU Free Documentation License, Version 1.2 + or any later version published by the Free Software Foundation; + with no Invariant Sections, no Front-Cover Texts, and no Back-Cover + Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html"> +http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body> +</html> diff --git a/doc/rsyslog_secure_tls.html b/doc/rsyslog_secure_tls.html new file mode 100644 index 00000000..be2811f4 --- /dev/null +++ b/doc/rsyslog_secure_tls.html @@ -0,0 +1,127 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: recommended scenario</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-06-17)</i></small></p> +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +<li><a href="tls_cert_errmsgs.html">Frequently seen Error Messages</a> +</ul> + +<h2>Overview</h2> +<p>This document describes a secure way to set up rsyslog TLS. A secure logging +environment requires more than just encrypting the transmission channel. This document +provides one possible way to create such a secure system. +<p>Rsyslog's TLS authentication can be used very flexible and thus supports a +wide range of security policies. This section tries to give some advise on a +scenario that works well for many environments. However, it may not be suitable +for you - please assess you security needs before using the recommendations +below. Do not blame us if it doesn't provide what you need ;)</p> +<p>Our policy offers these security benefits:</p> +<ul> + <li>syslog messages are encrypted while traveling on the wire</li> + <li>the syslog sender authenticates to the syslog receiver; thus, the + receiver knows who is talking to it</li> + <li>the syslog receiver authenticates to the syslog sender; thus, the sender + can check if it indeed is sending to the expected receiver</li> + <li>the mutual authentication prevents man-in-the-middle attacks</li> +</ul> +<p>Our secrity goals are achived via public/private key security. As such, it is +vital that private keys are well protected and not accessible to third parties. +<span style="float: left"> +<script type="text/javascript"><!-- +google_ad_client = "pub-3204610807458280"; +/* rsyslog doc inline */ +google_ad_slot = "5958614527"; +google_ad_width = 125; +google_ad_height = 125; +//--> +</script> +<script type="text/javascript" +src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> +</script> +</span> +I private keys have become known to third parties, the system does not provide +any security at all. Also, our solution bases on X.509 certificates and a (very +limited) chain of trust. We have one instance (the CA) that issues all machine +certificates. The machine certificate indentifies a particular machine. hile in +theory (and practice), there could be several "sub-CA" that issues machine +certificates for a specific adminitrative domain, we do not include this in our +"simple yet secure" setup. If you intend to use this, rsyslog supports it, but +then you need to dig a bit more into the documentation (or use the forum to ask). +In general, if you depart from our simple model, you should have good reasons +for doing so and know quite well what you are doing - otherwise you may +compromise your system security.</p> +<p>Please note that security never comes without effort. In the scenario +described here, we have limited the effort as much as possible. What remains is +some setup work for the central CA, the certificate setup for each machine as +well as a few configuration commands that need to be applied to all of them. +Proably the most important limiting factor in our setup is that all senders and +receivers must support IETF's syslog-transport-tls standard (which is not +finalized yet). We use mandatory-to-implement technology, yet you may have +trouble finding all required features in some implementations. More often, +unfortunately, you will find that an implementation does not support the +upcoming IETF standard at all - especially in the "early days" (starting May +2008) when rsyslog is the only implementation of said standard.</p> +<p>Fortunately, rsyslog supports allmost every protocol that is out there in the +syslog world. So in cases where transport-tls is not available on a sender, we +recommend to use rsyslog as the initial relay. In that mode, the not-capabe +sender sends to rsyslog via another protocol, which then relays the message via +transport-tls to either another interim relay or the final destination (which, +of course, must by transport-tls capable). In such a scenario, it is best to try +see what the sender support. Maybe it is possible to use industry-standard plain +tcp syslog with it. Often you can even combine it with stunnel, which then, too, +enables a secure delivery to the first rsyslog relay. If all of that is not +possible, you can (and often must...) resort to UDP. Even though this is now +lossy and insecure, this is better than not having the ability to listen to that +device at all. It may even be reasonale secure if the uncapable sender and the +first rsyslog relay communicate via a private channel, e.g. a dedicated network +link.</p> +<p>One final word of caution: transport-tls protects the connection between the +sender and the receiver. It does not necessarily protect against attacks that +are present in the message itself. Especially in a relay environment, the +message may have been originated from a malicious system, which placed invalid +hostnames and/or other content into it. If there is no provisioning against such +things, these records may show up in the receivers' repository. -transport-tls +does not protect against this (but it may help, properly used). Keep in mind +that syslog-transport-tls provides hop-by-hop security. It does not provide +end-to-end security and it does not authenticate the message itself (just the +last sender).</p> +<h3>A very quick Intro</h3> +<p>If you'd like to get all information very rapidly, the graphic below contains +everything you need to know (from the certificate perspective) in a very condensed +manner. It is no surprise if the graphic puzzles you. In this case, <a href="tls_cert_scenario.html">simply read on</a> +for full instructions. +<p> +<img align="center" alt="TLS/SSL protected syslog" src="tls_cert.jpg"> +<h3>Feedback requested</h3> +<p>I would appreciate feedback on this tutorial. If you have +additional ideas, comments or find bugs (I *do* bugs - no way... ;)), +please +<a href="mailto:rgerhards@adiscon.com">let me know</a>.</p> +<h2>Revision History</h2> +<ul> +<li>2008-06-06 * <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> * Initial Version created</li> +<li>2008-06-18 * <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> * Greatly enhanced and modularized the doc</li> +</ul> +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/rsyslog_stunnel.html b/doc/rsyslog_stunnel.html index 9d944521..1d024934 100644 --- a/doc/rsyslog_stunnel.html +++ b/doc/rsyslog_stunnel.html @@ -1,248 +1,240 @@ -<html><head>
-<title>SSL Encrypting syslog with stunnel</title>
-<meta name="KEYWORDS" content="syslog encryption, rsyslog, stunnel, secure syslog, tcp, reliable, howto, ssl">
-</head>
-<body>
-<h1>SSL Encrypting Syslog with Stunnel</h1>
- <P><small><i>Written by
- <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer
- Gerhards</a> (2005-07-22)</i></small></P>
-<h2>Abstract</h2>
-<p><i><b>In this paper, I describe how to encrypt <a href="http://www.monitorware.com/en/topics/syslog/">syslog</a>
-messages on the network.</b> Encryption
-is vital to keep the confidiental content of syslog messages secure. I describe the overall
-approach and provide an HOWTO do it with the help of
-<a href="http://www.rsyslog.com">rsyslogd</a> and <a href="http://www.stunnel.org">stunnel</a>.</i></p>
-<h2>Background</h2>
-<P><b>Syslog is a
-clear-text protocol. That means anyone with a sniffer can have
-a peek at your data.</b> In some environments, this is no problem at all. In
-others, it is a huge setback, probably even preventing deployment of syslog
-solutions. Thankfully, there is an easy way to encrypt syslog communication. I
-will describe one approach in this paper.</P>
-<P>The most straigthforward solution would be that the syslogd itself encrypts
-messages. Unfortuantely, encryption is only standardized in
-<a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php">RFC 3195</a>. But there
-is currently no syslogd that implements RFC 3195's encryption features,
-so this route leads to nothing. Another approach would be to use vendor- or
-project-specific syslog extensions. There are a few around, but the problem here
-is that they have compatibility issues. However, there is one surprisingly easy
-and interoperable solution: though not standardized, many vendors and projects
-implement plain tcp syslog. In a nutshell, plain tcp syslog is a mode where
-standard syslog messages are transmitted via tcp and records are separated by
-newline characters. This mode is supported by all major syslogd's (both on Linux/Unix
-and Windows) as well as log sources (for example,
-<a href="http://www.eventreporter.com/en/">EventReporter</a> for Windows
-Event Log forwarding). Plain tcp syslog offers reliability, but it does not
-offer encryption in itself. However, since it operates on a tcp stream, it is now easy
-to add encryption. There are various ways to do that. In this paper, I will
-describe how it is done with stunnel (an
-other alternative would be <a href="http://en.wikipedia.org/wiki/IPSec">IPSec</a>, for example).</P>
-<P>Stunnel is open source and it is available both for Unix/Linux and Windows.
-It provides a way to
- use ssl communication for any non-ssl aware client and server - in this case,
- our syslogd.</P>
- <P>Stunnel works much like a wrapper. Both on the client and on the server machine,
- tunnel portals are created. The non-ssl aware client and server software is
- configured to not directly talk to the remote partner, but to the local
- (s)tunnel portal instead. Stunnel, in turn, takes the data received from the
- client, encrypts it via ssl, sends it to the remote tunnel portal and that
- remote portal sends it to the recipient process on the remote machine. The
- transfer to the portals is done via unencrypted communication. As such,
- it is vital that
- the portal and the respective program that is talking to it are on the same
- machine, otherwise data would travel partly unencrypted. Tunneling, as done by stunnel,
- requires connection oriented communication. This is why you need to use
- tcp-based syslog. As a side-note, you can also encrypt a plain-text RFC
- 3195 session via stunnel, though this definitely is not what the
- protocol designers had on their mind ;)</P>
-<P>In the rest of this document, I assume that you use rsyslog on both the
-client and the server. For the samples, I use <a href="http://www.debian.org/">Debian</a>.
-Interestingly, there are
-some annoying differences between stunnel implementations. For example, on
-Debian a comment line starts with a semicolon (';'). On
-<a href="http://www.redhat.com">Red Hat</a>, it starts with
-a hash sign ('#'). So you need to watch out for subtle issues when setting up
-your system.</P>
-<h2>Overall System Setup</h2>
-<P>In ths paper, I assume two machines, one named "client" and the other named "server".
-It is obvious that, in practice, you will probably have multiple clients but
-only one server. Syslog traffic shall be transmitted via stunnel over the
-network. Port 60514 is to be used for that purpose. The machines are set up as
-follows:</P>
-<P><b>Client</b></P>
-<ul>
- <li>rsyslog forwards message to stunnel local portal at port 61514</li>
- <li>local stunnel forwards data via the network to port 60514 to its remote
- peer</li>
-</ul>
-<P><b>Server</b></P>
-<ul>
- <li>stunnel listens on port 60514 to connections from its client peers</li>
- <li>all connections are forwarded to the locally-running rsyslog listening
- at port 61514</li>
-</ul>
-<h2>Setting up the system</h2>
-<P>For Debian, you need the "stunnel4" package. The "stunnel" package is the
-older 3.x release, which will not support the configuration I describe below.
-Other distributions might have other names. For example, on Red Hat it is just "stunnel".
-Make sure that you install the appropriate package on both the client and the
-server. It is also a good idea to check if there are updates for either stunnel
-or openssl (which stunnel uses) - there are often security fixes available and
-often the latest fixes are not included in the default package.</P>
-<P>In my sample setup, I use only the bare minimum of options. For example, I do
-not make the server check client cerficiates. Also, I do not talk much about
-certificates at all. If you intend to really secure your system, you should
-probably learn about certificates and how to manage and deploy them. This is
-beyond the scope of this paper. For additional information,
-<a href="http://www.stunnel.org/faq/certs.html">
-http://www.stunnel.org/faq/certs.html</a> is a good starting point.</P>
-<P>You also need to install rsyslogd on both machines. Do this before starting
-with the configuration. You should also familarize yourself with its
-configuration file syntax, so that you know which actions you can trigger with
-it. Rsyslogd can work as a drop-in replacement for stock
-<a href="http://www.infodrom.org/projects/sysklogd/">sysklogd</a>. So if you know
-the standard syslog.conf syntax, you do not need to learn any more to follow
-this paper.</P>
-<h3>Server Setup</h3>
-<P>At the server, you need to have a digital certificate. That certificate
-enables SSL operation, as it provides the necessary crypto keys being used to
-secure the connection. Many versions of stunnel come with a default certificate,
-often found in /etc/stunnel/stunnel.pem. If you have it, it is good for testing
-only. If you use it in production, it is very easy to break into your secure
-channel as everybody is able to get hold of your private key. I didn't find an
-stunnel.pem on my Debian machine. I guess the Debian folks removed it because of
-its insecurity.</P>
-<P>You can create your own certificate with a simple openssl tool - you need to
-do it if you have none and I highly recommend to create one in any case. To
-create it, cd to /etc/stunnel and type:</P>
-<p><blockquote><code>openssl req -new -x509 -days 3650 -nodes -out
-stunnel.pem -keyout stunnel.pem</code></blockquote></p>
-<P>That command will ask you a number of questions. Provide some answer for
-them. If you are unsure, read
-<a href="http://www.stunnel.org/faq/certs.html">
-http://www.stunnel.org/faq/certs.html</a>. After the command has finished, you
-should have a usable stunnel.pem in your working directory.</P>
-<P>Next is to create a configuration file for stunnel. It will direct stunnel
-what to do. You can used the following basic file:</P>
-<P><blockquote><code><pre>; Certificate/key is needed in server mode
-cert = /etc/stunnel/stunnel.pem
-
-<i>; Some debugging stuff useful for troubleshooting
-debug = 7
-foreground=yes</i>
-
-[ssyslog]
-accept = 60514
-connect = 61514</pre>
-</code></blockquote></P>
-<p>Save this file to e.g. /etc/stunnel/syslog-server.conf. Please note that the
-settings in <i>italics</i> are for debugging only. They run stunnel
-with a lot of debug information in the foreground. This is very valuable while
-you setup the system - and very useless once everything works well. So be sure
-to remove these lines when going to production.</p>
-<p>Finally, you need to start the stunnel daemon. Under Debian, this is done via
-"stunnel /etc/stunnel/syslog.server.conf". If you have enabled the debug
-settings, you will immediately see a lot of nice messages.</p>
-<p>Now you have stunnel running, but it obviously unable to talk to rsyslog -
-because it is not yet running. If not already done, configure it so that it does
-everything you want. If in doubt, you can simply copy /etc/syslog.conf to /etc/rsyslog.conf
-and you probably have what you want. The really important thing in rsyslogd
-configuration is that you must make it listen to tcp port 61514 (remember: this
-is where stunnel send the messages to). Thankfully, this is easy to achive: just
-add "-t 61514" to the rsyslogd startup options in your system startup script.
-After done so, start (or restart) rsyslogd.</p>
-<p>The server should now be fully operational.</p>
-<h3>Client Setup</h3>
-<P>The client setup is simpler. Most importantly, you do not need a certificate
-(of course, you can use one if you would like to authenticate the client, but
-this is beyond the scope of this paper). So the basic thing you need to do is
-create the stunnel configuration file.</P>
-<P><blockquote><code><pre><i>; Some debugging stuff useful for troubleshooting
-debug = 7
-foreground=yes</i>
-
-<b>client=yes</b>
-
-[ssyslog]
-accept = 127.0.0.1:61514
-connect = <font color="#FF0000">192.0.2.1</font>:60514
-</pre>
-</code></blockquote></P>
-<P>Again, the text in <i>italics</i> is for debugging purposes only. I suggest
-you leave it in during your initial testing and then remove it. The most
-important difference to the server configuration outlined above is the "client=yes"
-directive. It is what makes this stunnel behave like a client. The accept
-directive binds stunnel only to the local host, so that it is protected from
-receiving messages from the network (somebody might fake to be the local sender).
-The address "192.0.2.1" is the address of the server machine. You must change it
-to match your configuration. Save this file to /etc/stunnel/syslog-client.conf.</P>
-<P>Then, start stunnel via "stunnel4 /etc/stunnel/syslog-client.conf". Now
-you should see some startup messages. If no errors appear, you have a running
-client stunnel instance.</P>
-<P>Finally, you need to tell rsyslogd to send data to the remote host. In stock
-syslogd, you do this via the "@host" forwarding directive. The same works with
-rsyslog, but it suppports extensions to use tcp. Add the following line to your
-/etc/rsyslog.conf:</P>
-<P><blockquote><code><pre>*.* @<font color="#FF0000">@</font>127.0.0.1:61514
-</pre>
-</code></blockquote><i></P>
-
-</i>
-
-<P>Please note the double at-sign (@@). This is no typo. It tells rsyslog to use
-tcp instead of udp delivery. In this sample, all messages are forwarded to the
-remote host. Obviously, you may want to limit this via the usual rsyslog.conf
-settings (if in doubt, use man rsyslog.con).</P>
-<P>You do not need to add any special startup settings to rsyslog on the client.
-Start or restart rsyslog so that the new configuration setting takes place.</P>
-<h3>Done</h3>
-<P>After following these steps, you should have a working secure syslog
-forwarding system. To verify, you can type "logger test" or a similar smart
-command on the client. It should show up in the respective server log file. If
-you dig out you sniffer, you should see that the traffic on the wire is actually
-protected. In the configuration use above, the two stunnel endpoints should be
-quite chatty, so that you can follow the action going on on your system.</P>
-<P>If you have only basic security needs, you can probably just remove the debug
-settings and take the rest of the configuration to production. If you are
-security-sensitve, you should have a look at the various stunnel settings that
-help you further secure the system.</P>
-<h2>Preventing Systems from talking directly to the rsyslog Server</h2>
-<P>It is possible that remote systems (or attackers) talk to the rsyslog server
-by directly connecting to its port 61514. Currently (July of 2005), rsyslog does
-not offer the ability to bind to the local host, only. This feature is planned,
-but as long as it is missing, rsyslog must be protected via a firewall. This can
-easily be done via e.g iptables. Just be sure not to forget it.</P>
-<h2>Conclusion</h2>
-<P>With minumal effort, you can set up a secure logging infrastructure employing
-ssl encrypted syslog message transmission. As a side note, you also have the
-benefit of reliable tcp delivery which is far less prone to message loss than
-udp.</P>
-<h3>Feedback requested</h3>
-<P>I would appreciate feedback on this tutorial. If you have additional ideas,
-comments or find bugs (I *do* bugs - no way... ;)), please
-<a href="mailto:rgerhards@adiscon.com">let me know</a>.</P>
-<h2>Revision History</h2>
-<ul>
- <li>2005-07-22 *
- <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> * Initial Version created</li>
- <li>2005-07-26 *
- <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> * Some text brush-up, hyperlinks added</li>
- <li>2005-08-03 *
- <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a>
- * license added</li>
-</ul>
-<h2>Copyright</h2>
-<p>Copyright (c) 2005
-<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> and
-<a href="http://www.adiscon.com/en/">Adiscon</a>.</p>
-<p> Permission is granted to copy, distribute and/or modify this document
- under the terms of the GNU Free Documentation License, Version 1.2
- or any later version published by the Free Software Foundation;
- with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
- Texts. A copy of the license can be viewed at
-<a href="http://www.gnu.org/copyleft/fdl.html">
-http://www.gnu.org/copyleft/fdl.html</a>.</p>
-
-</body>
-</html>
\ No newline at end of file +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> + +<title>SSL Encrypting syslog with stunnel</title><meta name="KEYWORDS" content="syslog encryption, rsyslog, stunnel, secure syslog, tcp, reliable, howto, ssl"></head><body> +<h1>SSL Encrypting Syslog with Stunnel</h1> + <p><small><i>Written by + <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer + Gerhards</a> (2005-07-22)</i></small></p> +<h2>Abstract</h2> +<p><i><b>In this paper, I describe how to encrypt <a href="http://www.monitorware.com/en/topics/syslog/">syslog</a> +messages on the network.</b> Encryption +is vital to keep the confidiental content of syslog messages secure. I describe the overall +approach and provide an HOWTO do it with the help of +<a href="http://www.rsyslog.com">rsyslogd</a> and <a href="http://www.stunnel.org">stunnel</a>.</i></p><p><span style="font-weight: bold;">Please note that starting with rsyslog 3.19.0, </span><a style="font-weight: bold;" href="rsyslog_tls.html">rsyslog provides native TLS/SSL encryption</a><span style="font-weight: bold;"> <span style="font-style: italic;">without</span> the need of stunnel. </span>I +strongly recomend to use that feature instead of stunnel. The stunnel +documentation here is mostly provided for backwards compatibility. New +deployments are advised to use native TLS mode.<i></i></p> +<h2>Background</h2> +<p><b>Syslog is a +clear-text protocol. That means anyone with a sniffer can have +a peek at your data.</b> In some environments, this is no problem at all. In +others, it is a huge setback, probably even preventing deployment of syslog +solutions. Thankfully, there is an easy way to encrypt syslog communication. I +will describe one approach in this paper.</p> +<p>The most straightforward solution would be that the syslogd itself encrypts +messages. Unfortuantely, encryption is only standardized in +<a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php">RFC 3195</a>. But there +is currently no syslogd that implements RFC 3195's encryption features, +so this route leads to nothing. Another approach would be to use vendor- or +project-specific syslog extensions. There are a few around, but the problem here +is that they have compatibility issues. However, there is one surprisingly easy +and interoperable solution: though not standardized, many vendors and projects +implement plain tcp syslog. In a nutshell, plain tcp syslog is a mode where +standard syslog messages are transmitted via tcp and records are separated by +newline characters. This mode is supported by all major syslogd's (both on Linux/Unix +and Windows) as well as log sources (for example, +<a href="http://www.eventreporter.com/en/">EventReporter</a> for Windows +Event Log forwarding). Plain tcp syslog offers reliability, but it does not +offer encryption in itself. However, since it operates on a tcp stream, it is now easy +to add encryption. There are various ways to do that. In this paper, I will +describe how it is done with stunnel (an +other alternative would be <a href="http://en.wikipedia.org/wiki/IPSec">IPSec</a>, for example).</p> +<p>Stunnel is open source and it is available both for Unix/Linux and Windows. +It provides a way to + use ssl communication for any non-ssl aware client and server - in this case, + our syslogd.</p> + <p>Stunnel works much like a wrapper. Both on the client and on the server machine, + tunnel portals are created. The non-ssl aware client and server software is + configured to not directly talk to the remote partner, but to the local + (s)tunnel portal instead. Stunnel, in turn, takes the data received from the + client, encrypts it via ssl, sends it to the remote tunnel portal and that + remote portal sends it to the recipient process on the remote machine. The + transfer to the portals is done via unencrypted communication. As such, + it is vital that + the portal and the respective program that is talking to it are on the same + machine, otherwise data would travel partly unencrypted. Tunneling, as done by stunnel, + requires connection oriented communication. This is why you need to use + tcp-based syslog. As a side-note, you can also encrypt a plain-text RFC + 3195 session via stunnel, though this definitely is not what the + protocol designers had on their mind ;)</p> +<p>In the rest of this document, I assume that you use rsyslog on both the +client and the server. For the samples, I use <a href="http://www.debian.org/">Debian</a>. +Interestingly, there are +some annoying differences between stunnel implementations. For example, on +Debian a comment line starts with a semicolon (';'). On +<a href="http://www.redhat.com">Red Hat</a>, it starts with +a hash sign ('#'). So you need to watch out for subtle issues when setting up +your system.</p> +<h2>Overall System Setup</h2> +<p>In ths paper, I assume two machines, one named "client" and the other named "server". +It is obvious that, in practice, you will probably have multiple clients but +only one server. Syslog traffic shall be transmitted via stunnel over the +network. Port 60514 is to be used for that purpose. The machines are set up as +follows:</p> +<p><b>Client</b></p> +<ul> + <li>rsyslog forwards message to stunnel local portal at port 61514</li> + <li>local stunnel forwards data via the network to port 60514 to its remote + peer</li> +</ul> +<p><b>Server</b></p> +<ul> + <li>stunnel listens on port 60514 to connections from its client peers</li> + <li>all connections are forwarded to the locally-running rsyslog listening + at port 61514</li> +</ul> +<h2>Setting up the system</h2> +<p>For Debian, you need the "stunnel4" package. The "stunnel" package is the +older 3.x release, which will not support the configuration I describe below. +Other distributions might have other names. For example, on Red Hat it is just "stunnel". +Make sure that you install the appropriate package on both the client and the +server. It is also a good idea to check if there are updates for either stunnel +or openssl (which stunnel uses) - there are often security fixes available and +often the latest fixes are not included in the default package.</p> +<p>In my sample setup, I use only the bare minimum of options. For example, I do +not make the server check client cerficiates. Also, I do not talk much about +certificates at all. If you intend to really secure your system, you should +probably learn about certificates and how to manage and deploy them. This is +beyond the scope of this paper. For additional information, +<a href="http://www.stunnel.org/faq/certs.html"> +http://www.stunnel.org/faq/certs.html</a> is a good starting point.</p> +<p>You also need to install rsyslogd on both machines. Do this before starting +with the configuration. You should also familarize yourself with its +configuration file syntax, so that you know which actions you can trigger with +it. Rsyslogd can work as a drop-in replacement for stock +<a href="http://www.infodrom.org/projects/sysklogd/">sysklogd</a>. So if you know +the standard syslog.conf syntax, you do not need to learn any more to follow +this paper.</p> +<h3>Server Setup</h3> +<p>At the server, you need to have a digital certificate. That certificate +enables SSL operation, as it provides the necessary crypto keys being used to +secure the connection. Many versions of stunnel come with a default certificate, +often found in /etc/stunnel/stunnel.pem. If you have it, it is good for testing +only. If you use it in production, it is very easy to break into your secure +channel as everybody is able to get hold of your private key. I didn't find an +stunnel.pem on my Debian machine. I guess the Debian folks removed it because of +its insecurity.</p> +<p>You can create your own certificate with a simple openssl tool - you need to +do it if you have none and I highly recommend to create one in any case. To +create it, cd to /etc/stunnel and type:</p> +<p></p><blockquote><code>openssl req -new -x509 -days 3650 -nodes -out +stunnel.pem -keyout stunnel.pem</code></blockquote><p></p> +<p>That command will ask you a number of questions. Provide some answer for +them. If you are unsure, read +<a href="http://www.stunnel.org/faq/certs.html"> +http://www.stunnel.org/faq/certs.html</a>. After the command has finished, you +should have a usable stunnel.pem in your working directory.</p> +<p>Next is to create a configuration file for stunnel. It will direct stunnel +what to do. You can used the following basic file:</p> +<p></p><blockquote><code></code><pre>; Certificate/key is needed in server mode<br>cert = /etc/stunnel/stunnel.pem<br><br><i>; Some debugging stuff useful for troubleshooting<br>debug = 7<br>foreground=yes</i> + +[ssyslog] +accept = 60514 +connect = 61514</pre> +</blockquote><p></p> +<p>Save this file to e.g. /etc/stunnel/syslog-server.conf. Please note that the +settings in <i>italics</i> are for debugging only. They run stunnel +with a lot of debug information in the foreground. This is very valuable while +you setup the system - and very useless once everything works well. So be sure +to remove these lines when going to production.</p> +<p>Finally, you need to start the stunnel daemon. Under Debian, this is done via +"stunnel /etc/stunnel/syslog.server.conf". If you have enabled the debug +settings, you will immediately see a lot of nice messages.</p> +<p>Now you have stunnel running, but it obviously unable to talk to rsyslog - +because it is not yet running. If not already done, configure it so that it does +everything you want. If in doubt, you can simply copy /etc/syslog.conf to /etc/rsyslog.conf +and you probably have what you want. The really important thing in rsyslogd +configuration is that you must make it listen to tcp port 61514 (remember: this +is where stunnel send the messages to). Thankfully, this is easy to achive: just +add "-t 61514" to the rsyslogd startup options in your system startup script. +After done so, start (or restart) rsyslogd.</p> +<p>The server should now be fully operational.</p> +<h3>Client Setup</h3> +<p>The client setup is simpler. Most importantly, you do not need a certificate +(of course, you can use one if you would like to authenticate the client, but +this is beyond the scope of this paper). So the basic thing you need to do is +create the stunnel configuration file.</p> +<p></p><blockquote><code></code><pre><i>; Some debugging stuff useful for troubleshooting<br>debug = 7<br>foreground=yes</i> + +<b>client=yes</b> + +[ssyslog] +accept = 127.0.0.1:61514 +connect = <font color="#ff0000">192.0.2.1</font>:60514<br></pre> +</blockquote><p></p> +<p>Again, the text in <i>italics</i> is for debugging purposes only. I suggest +you leave it in during your initial testing and then remove it. The most +important difference to the server configuration outlined above is the "client=yes" +directive. It is what makes this stunnel behave like a client. The accept +directive binds stunnel only to the local host, so that it is protected from +receiving messages from the network (somebody might fake to be the local sender). +The address "192.0.2.1" is the address of the server machine. You must change it +to match your configuration. Save this file to /etc/stunnel/syslog-client.conf.</p> +<p>Then, start stunnel via "stunnel4 /etc/stunnel/syslog-client.conf". Now +you should see some startup messages. If no errors appear, you have a running +client stunnel instance.</p> +<p>Finally, you need to tell rsyslogd to send data to the remote host. In stock +syslogd, you do this via the "@host" forwarding directive. The same works with +rsyslog, but it suppports extensions to use tcp. Add the following line to your +/etc/rsyslog.conf:</p> +<p></p><blockquote><code></code><pre>*.* @<font color="#ff0000">@</font>127.0.0.1:61514<br></pre> +</blockquote><i><p></p> + +</i> + +<p>Please note the double at-sign (@@). This is no typo. It tells rsyslog to use +tcp instead of udp delivery. In this sample, all messages are forwarded to the +remote host. Obviously, you may want to limit this via the usual rsyslog.conf +settings (if in doubt, use man rsyslog.con).</p> +<p>You do not need to add any special startup settings to rsyslog on the client. +Start or restart rsyslog so that the new configuration setting takes place.</p> +<h3>Done</h3> +<p>After following these steps, you should have a working secure syslog +forwarding system. To verify, you can type "logger test" or a similar smart +command on the client. It should show up in the respective server log file. If +you dig out you sniffer, you should see that the traffic on the wire is actually +protected. In the configuration use above, the two stunnel endpoints should be +quite chatty, so that you can follow the action going on on your system.</p> +<p>If you have only basic security needs, you can probably just remove the debug +settings and take the rest of the configuration to production. If you are +security-sensitve, you should have a look at the various stunnel settings that +help you further secure the system.</p> +<h2>Preventing Systems from talking directly to the rsyslog Server</h2> +<p>It is possible that remote systems (or attackers) talk to the rsyslog server +by directly connecting to its port 61514. Currently (July of 2005), rsyslog does +not offer the ability to bind to the local host, only. This feature is planned, +but as long as it is missing, rsyslog must be protected via a firewall. This can +easily be done via e.g iptables. Just be sure not to forget it.</p> +<h2>Conclusion</h2> +<p>With minumal effort, you can set up a secure logging infrastructure employing +ssl encrypted syslog message transmission. As a side note, you also have the +benefit of reliable tcp delivery which is far less prone to message loss than +udp.</p> +<h3>Feedback requested</h3> +<p>I would appreciate feedback on this tutorial. If you have additional ideas, +comments or find bugs (I *do* bugs - no way... ;)), please +<a href="mailto:rgerhards@adiscon.com">let me know</a>.</p> +<h2>Revision History</h2> +<ul> + <li>2005-07-22 * + <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> * Initial Version created</li> + <li>2005-07-26 * + <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> * Some text brush-up, hyperlinks added</li> + <li>2005-08-03 * + <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> + * license added</li><li>2008-05-05 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> + * updated to reflect native TLS capability of rsyslog 3.19.0 and above</li> +</ul> +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this document + under the terms of the GNU Free Documentation License, Version 1.2 + or any later version published by the Free Software Foundation; + with no Invariant Sections, no Front-Cover Texts, and no Back-Cover + Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html"> +http://www.gnu.org/copyleft/fdl.html</a>.</p> + +</body></html> diff --git a/doc/rsyslog_tls.html b/doc/rsyslog_tls.html new file mode 100644 index 00000000..0a27b90e --- /dev/null +++ b/doc/rsyslog_tls.html @@ -0,0 +1,307 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS (SSL) Encrypting syslog</title> + +<meta name="KEYWORDS" content="syslog encryption, rsyslog, secure syslog, tcp, reliable, howto, ssl, tls"> +</head> +<body> +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-05-06)</i></small></p> +<h2>Abstract</h2> +<p><i><b>In this paper, I describe how to encrypt <a href="http://www.monitorware.com/en/topics/syslog/">syslog</a> +messages on the network.</b> Encryption +is vital to keep the confidiental content of syslog messages secure. I +describe the overall +approach and provide an HOWTO do it with <a href="http://www.rsyslog.com">rsyslog's</a> TLS +features. </i></p> +<p>Please +note that TLS is the more secure successor of SSL. While people often +talk about "SSL encryption" they actually mean "TLS encryption". So +don't look any further if you look for how to SSL-encrypt syslog. You +have found the right spot.</p> +<p>This is a quick guide. There is a more elaborate guide currently +under construction which provides a much more secure environment. It +is highly recommended to +<a href="rsyslog_secure_tls.html">at least have a look at it</a>. +<h2>Background</h2> +<p><b>Traditional syslog is a clear-text protocol. That +means anyone with a sniffer can have a peek at your data.</b> In +some environments, this is no problem at all. In others, it is a huge +setback, probably even preventing deployment of syslog solutions. +Thankfully, there are easy ways to encrypt syslog +communication. </p> +The traditional approach involves <a href="rsyslog_stunnel.html">running +a wrapper like stunnel around the syslog session</a>. This works +quite well and is in widespread use. However, it is not thightly +coupled with the main syslogd and some, even severe, problems can +result from this (follow a mailing list thread that describes <a href="http://lists.adiscon.net/pipermail/rsyslog/2008-March/000580.html">total +loss of syslog messages due to stunnel mode</a> and the <a href="http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp-syslog.html">unreliability +of TCP syslog</a>). +<p><a href="gssapi.html">Rsyslog supports syslog via +GSSAP</a>I since long to overcome these limitatinos. However, +syslog via GSSAPI is a rsyslog-exclusive transfer mode and it requires +a proper Kerberos environment. As such, it isn't a really universal +solution. The <a href="http://www.ietf.org/">IETF</a> +has begun standardizing syslog over plain tcp over +TLS for a while now. While I am not fully satisfied with the results so +far, this obviously has the potential to become the long-term +solution. The Internet Draft in question, syslog-transport-tls has been +dormant for some time but is now (May of 2008) again being worked on. I +expect it to turn into a RFC within the next 12 month (but don't take +this for granted ;)). I didn't want to wait for it, because there +obviously is need for TLS syslog right now (and, honestly, I have +waited long enough...). Consequently, I have +implemented the current draft, with some interpretations I made (there +will be a compliance doc soon). So in essence, a TLS-protected syslog +transfer mode is available right now. As a side-note, Rsyslog +is the world's first +implementation of syslog-transport-tls.</p> +<p>Please note that in theory it should be compatible with other, +non IETF syslog-transport-tls implementations. If you would like to run +it with something else, please let us know so that we can create a +compatibility list (and implement compatbility where it doesn't yet +exist). </p> +<h2>Overall System Setup</h2> +<p>Encryption requires a reliable stream. So It will not work +over UDP syslog. In rsyslog, network transports utilize a so-called +"network stream layer" (netstream for short). This layer provides a +unified view of the transport to the application layer. The plain TCP +syslog sender and receiver are the upper layer. The driver layer +currently consists of the "ptcp" and "gtls" library plugins. "ptcp" +stands for "plain tcp" and is used for unencrypted message transfer. It +is also used internally by the gtls driver, so it must always be +present on a system. The "gtls" driver is for GnutTLS, a TLS library. +It is used for encrypted message transfer. In the future, additional +drivers will become available (most importantly, we would like to +include a driver for NSS).</p> +<p>What you need to do to build an encrypted syslog channel is to +simply use the proper netstream drivers on both the client and the +server. Client, in the sense of this document, is the rsyslog system +that is sending syslog messages to a remote (central) loghost, which is +called the server. In short, the setup is as follows:</p> +<p><b>Client</b></p> +<ul> +<li>forwards messages via plain tcp syslog using gtls netstream +driver to central sever on port 10514<br> +</li> +</ul> +<p><b>Server</b></p> +<ul> +<li>accept incoming messages via plain tcp syslog using gtls +netstream driver on port 10514</li> +</ul> +<h2>Setting up the system</h2> +<h3>Server Setup</h3> +<p>At the server, you need to have a digital certificate. That +certificate enables SSL operation, as it provides the necessary crypto +keys being used to secure the connection. There is a set of default +certificates in ./contrib/gnutls. These are key.pem and cert.pem. These +are good for testing. If you use it in production, +it is very easy to break into your secure channel as everybody is able +to get hold of your private key. So it is a good idea to +generate the key and certificate yourself.</p> +<p>You also need a root CA certificate. Again, there is a sample +CA certificate in ./contrib/gnutls, named ca.cert. It is suggested to +generate your own.</p> +<p>To configure the server, you need to tell it where are its +certificate files, to use the gtls driver and start up a listener. This +is done as follows:<br> +</p> +<blockquote><code></code> +<pre># make gtls driver the default +$DefaultNetstreamDriver gtls + +# certificate files +$DefaultNetstreamDriverCAFile /path/to/contrib/gnutls/ca.pem +$DefaultNetstreamDriverCertFile /path/to/contrib/gnutls/cert.pem +$DefaultNetstreamDriverKeyFile /path/to/contrib/gnutls/key.pem + +$ModLoad imtcp # load TCP listener + +$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode +$InputTCPServerStreamDriverAuthMode anon # client is NOT authenticated +$InputTCPServerRun 10514 # start up listener at port 10514 +</pre> +</blockquote> +This is all you need to do. You can use the rest of your rsyslog.conf +together with this configuration. The way messages are received does +not interfer with any other option, so you are able to do anything else +you like without any restrictions. +<p>Restart rsyslogd. The server should now be fully +operational.</p> +<h3>Client Setup</h3> +<p>The client setup is equally simple. You need less +certificates, just the CA cert. </p> +<blockquote> +<pre># certificate files - just CA for a client +$DefaultNetstreamDriverCAFile /path/to/contrib/gnutls/ca.pem + +# set up the action +$DefaultNetstreamDriver gtls # use gtls netstream driver +$ActionSendStreamDriverMode 1 # require TLS for the connection +$ActionSendStreamDriverAuthMode anon # server is NOT authenticated +*.* @@(o)server.example.net:10514 # send (all) messages + +</pre> +</blockquote> +<p>Note that we use the regular TCP forwarding syntax (@@) here. +There is nothing special, because the encryption is handled by the +netstream driver. So I have just forwarded every message (*.*) for +simplicity - you can use any of rsyslog's filtering capabilities (like +epxression-based filters or regular expressions). Note that the "(o)" +part is not strictly necessary. It selects octet-based framing, which +provides compatiblity to IETF's syslog-transport-tls draft. Besides +compatibility, this is also a more reliable transfer mode, so I suggest +to always use it.</p> +<h3>Done</h3> +<p>After +following these steps, you should have a working secure +syslog forwarding system. To verify, you can type "logger test" or a +similar "smart" command on the client. It should show up in the +respective server log file. If you dig out your sniffer, you should see +that the traffic on the wire is actually protected.</p> +<h3>Limitations</h3> +<p>The current implementation has a number of limitations. These +are +being worked on. Most importantly, neither the client nor the server +are authenticated. So while the message transfer is encrypted, you can +not be sure which peer you are talking to. Please note that this is a +limitation found in most real-world SSL syslog systems. Of course, that +is not an excuse for not yet providing this feature - but it tells you +that it is acceptable and can be worked around by proper firewalling, +ACLs and other organizational measures. Mutual authentication will be +added shortly to rsyslog.</p> +<p>Secondly, the plain tcp syslog listener +can currently listen to a single port, in a single mode. So if you use +a TLS-based listener, you can not run unencrypted syslog on the same +instance at the same time. A work-around is to run a second rsyslogd +instance. This limitation, too, is scheduled to be removed soon.</p> +<p>The +RELP transport can currently not be protected by TLS. A work-around is +to use stunnel. TLS support for RELP will be added once plain TCP +syslog has sufficiently matured.</p> +<h2>Certificates</h2> +<p>In order to be really secure, certificates are needed. This is +a short summary on how to generate the necessary certificates with +GnuTLS' certtool. You can also generate certificates via other tools, +but as we currently support GnuTLS as the only TLS library, we thought +it is a good idea to use their tools.<br> +</p> +<p>Note that this section aims at people who are not involved +with PKI at all. The main goal is to get them going in a reasonable +secure way. </p> +<h3>CA Certificate</h3> +<p>This is used to sign all of your other certificates. The CA +cert must be trusted by all clients and servers. The private key must +be well-protected and not given to any third parties. The certificate +itself can (and must) be distributed. To generate it, do the following:</p> +<ol> +<li>generate the private key: +<pre>certtool --generate-privkey --outfile ca-key.pem</pre> +<br> +This takes a short while. Be sure to do some work on your workstation, +it waits for radom input. Switching between windows is sufficient ;) +</li> +<li>now create the (self-signed) CA certificate itself:<br> +<pre>certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem</pre> +This generates the CA certificate. This command queries you for a +number of things. Use appropriate responses. When it comes to +certificate validity, keep in mind that you need to recreate all +certificates when this one expires. So it may be a good idea to use a +long period, eg. 3650 days (roughly 10 years). You need to specify that +the certificates belongs to an authrity. The certificate is used to +sign other certificates.<br> +</li> +<li>You need to distribute this certificate +to all peers and you need to point to it via the +$DefaultNetstreamDriverCAFile config directive. All other certificates +will be issued by this CA.<br> +Important: do only distribute the ca.pem, NOT ca-key.pem (the private +key). Distributing the CA private key would totally breach security as +everybody could issue new certificates on the behalf of this CA. +</li> +</ol> +<h3>Individual Peer Certificate</h3> +<p>Each peer (be it client, server or both), needs a certificate +that conveys its identity. Access control is based on these +certificates. You can, for example, configure a server to accept +connections only from configured clients. The client ID is taken from +the client instances certificate. So as a general rule of thumb, you +need to create a certificate for each instance of rsyslogd that you +run. That instance also needs the private key, so that it can properly +decrypt the traffic. Safeguard the peer's private key file. If somebody +gets hold of it, it can malicously pretend to be the compromised host. +If such happens, regenerate the certificate and make sure you use a +different name instead of the compromised one (if you use name-based +authentication). </p> +<p>These are the steps to generate the indivudual certificates +(repeat: you need to do this for every instance, do NOT share the +certificates created in this step):</p> +<ol> +<li>generate a private key (do NOT mistake this with the CA's +private key - this one is different):<br> +<pre>certtool --generate-privkey --outfile key.pem</pre> +Again, this takes a short while.</li> +<li>generate a certificate request:<br> +<pre>certtool --generate-request --load-privkey key.pem --outfile request.pem</pre> +If you do not have the CA's private key (because you are not authorized +for this), you can send the certificate request to the responsible +person. If you do this, you can skip the remaining steps, as the CA +will provide you with the final certificate. If you submit the request +to the CA, you need to tell the CA the answers that you would normally +provide in step 3 below. +</li> +<li>Sign (validate, authorize) the certificate request and +generate the instances certificate. You need to have the CA's +certificate and private key for this:<br> +<pre>certtool --generate-certificate --load-request request.pem --outfile cert.pem \<br> --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem</pre> +Answer questions as follows: Cert does not belogn to an authority; it +is a TLS web server and client certificate; the dnsName MUST be the +name of the peer in question (e.g. centralserver.example.net) - this is +the name used for authenticating the peers. Please note that you may +use an IP address in dnsName. This is a good idea if you would like to +use default server authentication and you use selector lines with IP +addresses (e.g. "*.* @@192.168.0.1") - in that case you need to select +a dnsName of 192.168.0.1. But, of course, changing the server IP then +requires generating a new certificate.</li> +</ol> +After you have generated the certificate, you need to place it onto the +local machine running rsyslogd. Specify the certificate and key via the +$DefaultNetstreamDriverCertFile /path/to/cert.pem and +$DefaultNetstreamDriverKeyFile /path/to/key.pem configuration +directives. Make sure that nobody has access to key.pem, as that would +breach security. And, once again: do NOT use these files on more than +one instance. Doing so would prevent you from distinguising between the +instances and thus would disable useful authentication. +<h3>Troubleshooting Certificates</h3> +<p>If you experience trouble with your certificate setup, it may +be +useful to get some information on what is contained in a specific +certificate (file). To obtain that information, do </p> +<pre>$ certtool --certificate-info --infile cert.pem</pre> +<p>where "cert.pem" can be replaced by the various certificate pem files (but it does not work with the key files).</p> +<h2>Conclusion</h2> +<p>With minumal effort, you can set up a secure logging +infrastructure employing TLS encrypted syslog message transmission.</p> +<h3>Feedback requested</h3> +<p>I would appreciate feedback on this tutorial. If you have +additional ideas, comments or find bugs (I *do* bugs - no way... ;)), +please +<a href="mailto:rgerhards@adiscon.com">let me know</a>.</p> +<h2>Revision History</h2> +<ul> +<li>2008-05-06 * <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> * Initial Version created</li><li>2008-05-26 * <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> * added information about certificates</li> +</ul> +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/src/classes.dia b/doc/src/classes.dia Binary files differnew file mode 100644 index 00000000..70e91566 --- /dev/null +++ b/doc/src/classes.dia diff --git a/doc/src/queueWorkerLogic.dia b/doc/src/queueWorkerLogic.dia Binary files differnew file mode 100644 index 00000000..068ea50c --- /dev/null +++ b/doc/src/queueWorkerLogic.dia diff --git a/doc/src/tls.dia b/doc/src/tls.dia Binary files differnew file mode 100644 index 00000000..77e5d185 --- /dev/null +++ b/doc/src/tls.dia diff --git a/doc/src/tls_cert.dia b/doc/src/tls_cert.dia Binary files differnew file mode 100644 index 00000000..e76431df --- /dev/null +++ b/doc/src/tls_cert.dia diff --git a/doc/src/tls_cert_100.dia b/doc/src/tls_cert_100.dia Binary files differnew file mode 100644 index 00000000..baed5e0f --- /dev/null +++ b/doc/src/tls_cert_100.dia diff --git a/doc/src/tls_cert_ca.dia b/doc/src/tls_cert_ca.dia Binary files differnew file mode 100644 index 00000000..7ce27a8d --- /dev/null +++ b/doc/src/tls_cert_ca.dia diff --git a/doc/status.html b/doc/status.html index cc197318..5c8409ec 100644 --- a/doc/status.html +++ b/doc/status.html @@ -1,48 +1,53 @@ -<html> -<head> -<title>rsyslog status page</title> -</head> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>rsyslog status page</title></head> <body> <h2>rsyslog status page</h2> -<p>This page reflects the status as of 2008-03-27.</p> +<p>This page reflects the status as of 2008-10-22.</p> <h2>Current Releases</h2> -p><b>development:</b> 3.12.4 - -<a href="http://www.rsyslog.com/Article195.phtml">change -log</a> - -<a href="http://www.rsyslog.com/Downloads-index-req-getit-lid-89.phtml">download</a></p> -<p><b><font color="#ff0000"><a href="v3compatibility.html">If you used version 2, be sure to read the rsyslog v3 -compatibility document!</a></font></b><br> -Documentation for 3.x is currently partly sparse. If you need -assistance, please -<a href="http://www.rsyslog.com/PNphpBB2.phtml">post in -the rsyslog forums</a>!</p> -<p><b>stable:</b> 2.0.4 - <a href="http://www.rsyslog.com/Article197.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-index-req-getit-lid-90.phtml">download</a></p> -<p> (<a href="version_naming.html">How are versions named?</a>)</p> + +<p><b>development:</b> 3.21.6 [2008-10-22] - +<a href="http://www.rsyslog.com/Article292.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-135.phtml">download</a> + +<br><b>beta:</b> 3.19.12 [2008-10-16] - +<a href="http://www.rsyslog.com/Article283.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-134.phtml">download</a></p> + +<p><b>v3 stable:</b> 3.18.5 [2008-10-09] - <a href="http://www.rsyslog.com/Article281.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-133.phtml">download</a> + +<br><b>v2 stable:</b> 2.0.6 [2008-08-07] - <a href="http://www.rsyslog.com/Article266.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-125.phtml">download</a> +<br>v0 and v1 are deprecated and no longer supported. If you absolutely do not like to +upgrade, you may consider purchasing a +<a href="professional_support.html">commercial rsyslog support package</a>. Just let us point +out that it is really not a good idea to still run a v0 version. + +<p><a href="v3compatibility.html">If you updgrade from version 2, be sure to read the rsyslog v3 +compatibility document.</a></p> +<p>(<a href="version_naming.html">How are versions named?</a>)</p> + <h2>Platforms</h2> -<p>Thankfully, a number of folks have begin to build packages and help port -rsyslog to other platforms. As such, -<a href="http://wiki.rsyslog.com/index.php/Platforms">the platform list is now -maintained inside the rsyslog wiki</a>. Platform maintainers perhaps have posted -extra information there. If you do platform-specific work, feel free to add -information to the wiki.</p> -<p>Rsyslog is the default syslogd in Fedora 8.</p> +<p>Thankfully, a number of folks have begin to build packages and +help port rsyslog to other platforms. As such, +<a href="http://wiki.rsyslog.com/index.php/Platforms">the +platform list is now maintained inside the rsyslog wiki</a>. +Platform maintainers perhaps have posted extra information there. If +you do platform-specific work, feel free to add information to the wiki.</p> +<p>Rsyslog is the default syslogd in Fedora 8 and above.</p> <h2>Additional information</h2> <p><b>Currently supported features are listed on the <a href="features.html">rsyslog features page</a>.</b></p> <ul> - <li>The rsyslog home page is <a href="http://www.rsyslog.com">www.rsyslog.com</a>.</li> - <li>Mailing list info can be found at - <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">http://lists.adiscon.com/rsyslog</a>.</li> - <li>The change log can be found at - <a href="http://www.rsyslog.com/Topic4.phtml"> - http://www.rsyslog.com/Topic4.phtml</a>. </li> - <li>Online documentation is available at - <a href="http://www.rsyslog.com/doc">http://www.rsyslog.com/doc</a>.</li> - <li>You may also find <a href="http://rgerhards.blogspot.com/">Rainer's blog</a> an interesting read.</li> +<li>The rsyslog home page is <a href="http://www.rsyslog.com">www.rsyslog.com</a>.</li> +<li>Mailing list info can be found at <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">http://lists.adiscon.com/rsyslog</a>.</li> +<li>The change log can be found at <a href="http://www.rsyslog.com/Topic4.phtml"> +http://www.rsyslog.com/Topic4.phtml</a>. </li> +<li>Online documentation is available at <a href="http://www.rsyslog.com/doc">http://www.rsyslog.com/doc</a>.</li> +<li>You may also find <a href="http://rgerhards.blogspot.com/">Rainer's blog</a> +an interesting read.</li> </ul> -<p>The project was initiated in 2004 by +<p>The project was initiated in 2003 and seriouosly begun in 2004 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> -and is currently being maintained by him. See the <a href="history.html">history -page</a> for more background information.</p> -</body> -</html> +and is currently being maintained by him. See the <a href="history.html">history page</a> for more +background information.</p> +</body></html> diff --git a/doc/syslog_parsing.html b/doc/syslog_parsing.html new file mode 100644 index 00000000..57da6657 --- /dev/null +++ b/doc/syslog_parsing.html @@ -0,0 +1,196 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>syslog parsing in rsyslog</title> +</head> +<body> +<h1>syslog parsing in rsyslog</h1> +<p><small><i>Written by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> +(2008-09-23)</i></small></p> +<p><b>We regularly receive messages asking why <a href="http://www.rsyslog.com">rsyslog</a> +parses this or that message incorrectly.</b> Of course, it turns out that rsyslog does +the right thing, but the message sender does not. And also of course, this is not even +of the slightest help to the end user experiencing the problem ;). So I thought I write this +paper. It describes the problem source and shows potential solutions (aha!). +<h2>Syslog Standardization</h2> +The syslog protocol has not been standardized until relatively recently.The first document "smelling" a bit +like a standard is <a href="http://www.ietf.org/rfc/rfc3164.txt">RFC 3164</a>, which dates back +to August 2001. The problem is that this document is no real standard. It has assigned "informational" +status by the <a href="http://www.ietf.org">IETF</a> which means it provides some hopefully +useful information but does not demand anything. It is impossible to "comply" to an informational +document. This, of course, doesn't stop marketing guys from telling they comply to RFC3164 and +it also does not stop some techs to tell you "this and that does not comply to RFC3164, so it is +<anybody else but them>'s fault". +<p>Then, there is <a href="http://www.ietf.org/rfc/rfc3195.txt">RFC3195</a>, which is +a real standard. In it's section 3 it makes (a somewhat questionable) reference to (informational) +RFC 3164 which may be interpreted in a way that RFC3195 standardizes the format layed out +in RFC 3164 by virtue of referencing them. So RFC3195 seems to extend its standardization +domain to the concepts layed out in RFC 3164 (which is why I tend to find that refrence +questionable). In that sense, RFC3195 standardizes the format informationally described in +RFC3164, Section 4. But it demands it only for the scope of RFC3195, which is syslog over +BEEP - and NOT syslog over UDP. So one may argue whether or not the RFC3164 format could +be considered a standard for any non-BEEP (including UDP) syslog, too. In the strict view +I tend to have, it does not. Refering to the RFC3195 context usually does not help, +because there are virtually no RFC3195 implementations available (at this time, +I would consider this RFC a failure). +<p>Now let's for a short moment assume that RFC3195 would somehow be able to demand +RFC3164 format for non-BEEP syslog. So we could use RFC3164 format as a standard. But does +that really help? Let's cite RFC 3164, right at the begining of section 4 (actually, this +is the first sentence): +<blockquote> +<pre> + The payload of any IP packet that has a UDP destination port of 514 + MUST be treated as a syslog message. +<pre> +</blockquote> +<p>Think a bit about it: this means that whatever is send to port 514 must be considered +a valid syslog message. No format at all is demanded. So if "this is junk" is sent to +UDP port 514 - voila, we have a valid message (interestingly, it is no longer a syslog +message if it is sent to port 515 ;)). You may now argue that I am overdoing. So let's +cite RFC 3164, Section 5.4, Example 2: +<blockquote> +<pre> + Example 2 + + Use the BFG! + + While this is a valid message, it has extraordinarily little useful + information. +</pre> +</blockquote> +<p>As you can see, RFC3164 explicitely states that no format at all is required. +<p>Now a side-note is due: all of this does not mean that the RFC3164 authors +did not know what they were doing. No, right the contrary is true: RFC3164 mission +is to describe what has been seen in practice as syslog messages and the +conclusion is quite right that there is no common understanding on the +message format. This is also the reason why RFC3164 is an informational document: +it provides useful information, but does not precisely specify anything. +<p>After all of this bashing, I now have to admit that RFC3164 has some format +recommendations layed out in section 4. The format described has quite some +value in it and implementors recently try to follow it. This format is usually meant +when someone tells you that a software is "RFC3164 compliant" or expects "RFC3164 compliant messages". +I also have to admit that rsyslog also uses this format and, in the sense outlined here, +expects messages received to be "RFC3164 compliant" (knowingly that such a beast does not +exist - I am simply lying here ;)). +<p>Please note that there is some relief of the situation in reach. There is a new normative +syslog RFC series upcoming, and it specifies a standard message format. At the time of +this writing, the main documents are sitting in the RFC editor queue waiting for a transport +mapping to be completed. I personally expect them to be assigned RFC numbers in 2009. +<h2>Practical Format Requirements</h2> +<p>From a practical point of view, the message format expected (and generated by +default in legacy mode) is: +<pre><code> +<PRI>TIMESTAMP SP HOST SP TAG MSG(Freetext) +</code></pre> +<p>SP is the ASCII "space" character and the definition of the rest of the fields +can be taken from RFC3164. Please note that there also is a lot of confusion on what +syntax and semantics the TAG actually has. This format is called "legacy syslog" because +it is not well specified (as you know by now) and has been "inherited from the real world". +<p>Rsyslog offers two parsers: one for the upcoming RFC series and one for legacy format. We +concentrate on the later. That parser applies some logic to detect missing hostnames, +is able to handle various ways the TIMESTAMP is typically malformed. In short it applies +a lot of guesswork in trying to figure out what a message really means. I am sure the +guessing algorithm can be improved, and I am always trying that when I see new malformed +messages (and there is an ample set of them...). However, this finds its limits where +it is not possible to differentiate between two entities which could be either. +For example, look at this message: +<pre><code> +<144>Tue Sep 23 11:40:01 taghost sample message +</code></pre> +<p>Does it contain a hostname? Mabye. The value "taghost" is a valid hostname. Of course, it is +also a valid tag. If it is a hostname, the tag's value is "sample" and the msg value is "message". +Or is the hostname missing, the tag is "taghost" and msg is "sample message"? As a human, I tend +to say the later interpretation is correct. But that's hard to tell the message parser (and, no, I do +not intend to apply artificial intelligence just to guess what the hostname value is...). +<p>One approach is to configure the parser so that it never expects hostnames. This becomes problematic +if you receive messages from multiple devices. Over time, I may implement parser conditionals, +but this is not yet available and I am not really sure if it is needed comlexity... +<p>Things like this, happen. Even more scary formats happen in practice. Even from mainstream +vendors. For example, I was just asked about this message (which, btw, finally made me +write this article here): +<pre></code> +"<130> [ERROR] iapp_socket_task.c 399: iappSocketTask: iappRecvPkt returned error" +</code></pre> +<p>If you compare it with the format RFC3164 "suggests", you'll quickly notice that +the message is "a bit" malformed. Actually, even my human intelligence is not sufficient +to guess if there is a TAG or not (is "[ERROR]" a tag or part of the message). I may not be +the smartest guy, but don't expect me to program a parser that is smarter than me. +<p>To the best of my konwledge, these vendor's device's syslog format can be configured, so it +would proabably be a good idea to include a (sufficiently well-formed) timestamp, +the sending hostname and (maybe?) a tag to make this message well parseable. +I will also once again take this sample and see if we can apply some guesswork. +For example, "[" can not be part of a well-formed TIMESTAMP, so logic can conclude +there is not TIMESTAMP. Also, "[" can not be used inside a valid hostname, so +logic can conclude that the message contains no hostname. Even if I implement this +logic (which I will probably do), this is a partial solution: it is impossible to +guess if there is a tag or not (honestly!). And, even worse, it is a solution only for +those set of messages that can be handled by the logic described. Now consider this +hypothetical message: +<pre></code> +"<130> [ERROR] host.example.net 2008-09-23 11-40-22 PST iapp_socket_task.c 399: iappSocketTask: iappRecvPkt returned error" +</code></pre> +<p>Obviously, it requires additional guesswork. If we iterate over all the cases, we +can very quickly see that it is impossible to guess everything correct. In the example above +we can not even surely tell if PST should be a timezone or some other message property. +<p>A potential solution is to generate a parser-table based parser, but this requires +considerable effort and also has quite some runtime overhead. I try to avoid this for +now (but I may do it, especially if someone sponsors this work ;)). Side-note: if you want +to be a bit scared about potential formats, you may want to have a look at my paper +<i>"<a href="http://www.monitorware.com/en/workinprogress/nature-of-syslog-data.php">On the Nature of Syslog Data</a>"</i>. +<h2>Work-Around</h2> +<p><b>The number one work-around is to configure your devices so that they emit +(sufficiently) well-formed messages.</b> You should by now know what these look +like. +<p>If that cure is not available, there are some things you can do in rsyslog to +handle the situation. First of all, be sure to read about +<a href="rsyslog_conf.html">rsyslog.conf format</a> +and the <a href="property_replacer.html">property replacer and properties</a> specifically. +You need to understand that everything is configured in rsyslog. And that the message is parsed +into properties. There are also properties available which do not stem back directly to parsing. +Most importantly, %fromhost% property holds the name of the system rsyslog received +the message from. In non-relay cases, this can be used instead of hostname. In relay cases, +there is no cure other than to either fix the orginal sender or at least one of the +relays in front of the rsyslog instance in question. Similarly, you can use %timegenerated% +instead of %timereported%. Timegenerated is the time the message hit rsyslog for the first +time. For non-relayed, locally connected peers, Timegenerated should be a very close approximation +of the actual time a message was formed at the sender (depending, of course, on potential +internal queueing inside the sender). +Also, you may use the +%rawmsg% property together with the several extraction modes the property replacer supports. +Rawmsg contains the message as it is received from the remote peer. In a sense, you can +implement a post-parser with this method. +<p>To use these properties, you need to define your own templates and assign them. Details +can be found in the above-quoted documentation. Just let's do a quick example. Let's say +you have the horrible message shown above and can not fix the sending device for +some good reason. In rsyslog.conf, you used to say: +<pre><code> +*.* /var/log/somefile +</code></pre> +<p>Of course, things do not work out well with that ill-formed message. So you decide +to dump the rawmsg to the file and pull the remote host and time of message generation +from rsyslog's internal properties (which, btw, is clever, because otherwise there is no +indication of these two properties...). So you need to define a template for that and +make sure the template is used with your file logging action. This is how it may look: +<pre><code> +$template, MalfromedMsgFormater,"%timegenerated% %fromhost% %rawmsg:::drop-last-lf%\n" +*.* /var/log/somefile;MalformedMsgFormatter +</code></pre> +<p>This will make your log much nicer, but not look perfect. Experiment a bit +with the available properties and replacer extraction options to fine-tune it +to your needs. +<h2>Wrap-Up</h2> +<p>Syslog message format is not sufficiently standardized. There exists a weak +"standard" format, which is used by a good number of implementations. However, there +exist many others, including mainstream vendor implementations, which have a +(sometimes horribly) different format. Rsyslog tries to deal with anomalies but +can not guess right in all instances. If possible, the sender should be configured +to submit well-formed messages. If that is not possible, you can work around these +issues with rsyslog's property replacer and template system. +<p>I hope this is a useful guide. You may also have a look at the +<a href="troubleshoot.html">rsyslog troubleshooting guide</a> for further help and places where +to ask questions. +<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/syslog-protocol.html b/doc/syslog_protocol.html index 72de5c27..72de5c27 100644 --- a/doc/syslog-protocol.html +++ b/doc/syslog_protocol.html diff --git a/doc/tls_cert.jpg b/doc/tls_cert.jpg Binary files differnew file mode 100644 index 00000000..920e998d --- /dev/null +++ b/doc/tls_cert.jpg diff --git a/doc/tls_cert_100.jpg b/doc/tls_cert_100.jpg Binary files differnew file mode 100644 index 00000000..beeedc58 --- /dev/null +++ b/doc/tls_cert_100.jpg diff --git a/doc/tls_cert_ca.html b/doc/tls_cert_ca.html new file mode 100644 index 00000000..2cae4040 --- /dev/null +++ b/doc/tls_cert_ca.html @@ -0,0 +1,168 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: scenario</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-06-17)</i></small></p> + +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +</ul> + +<h3>Setting up the CA</h3> +<p>The first step is to set up a certificate authority (CA). It must be +maintained by a trustworthy person (or group) and approves the indentities of +all machines. It does so by issuing their certificates. In a small setup, the +administrator can provide the CA function. What is important is the the CA's +<span style="float: left"> +<script type="text/javascript"><!-- +google_ad_client = "pub-3204610807458280"; +/* rsyslog doc inline */ +google_ad_slot = "5958614527"; +google_ad_width = 125; +google_ad_height = 125; +//--> +</script> +<script type="text/javascript" +src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> +</script> +</span> +private key is well-protocted and machine certificates are only issued if it is +know they are valid (in a single-admin case that means the admin should not +issue certificates to anyone else except himself).</p> +<p>The CA creates a so-called self-signed certificate. That is, it approves its +own authenticy. This sounds useless, but the key point to understand is that +every machine will be provided a copy of the CA's certificate. Accepting this +certificate is a matter of trust. So by configuring the CA certificate, the +administrator tells <a href="http://www.rsyslog.com">rsyslog</a> which certificates to trust. This is the root of all +trust under this model. That is why the CA's private key is so important - +everyone getting hold of it is trusted by our rsyslog instances.</p> +<center><img src="tls_cert_ca.jpg"></center> +<p>To create a self-signed certificate, use the following commands with GnuTLS (which +is currently the only supported TLS library, what may change in the future). +Please note that GnuTLS' tools are not installed by default on many platforms. Also, +the tools do not necessarily come with the GnuTLS core package. If you do not +have certtool on your system, check if there is package for the GnuTLS tools available +(under Fedora, for example, this is named gnutls-utils-<version> and +it is NOT installed by default). </p> +<ol> +<li>generate the private key: +<pre>certtool --generate-privkey --outfile ca-key.pem</pre> +<br> +This takes a short while. Be sure to do some work on your workstation, +it waits for radom input. Switching between windows is sufficient ;) +</li> +<li>now create the (self-signed) CA certificate itself:<br> +<pre>certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem</pre> +This generates the CA certificate. This command queries you for a +number of things. Use appropriate responses. When it comes to +certificate validity, keep in mind that you need to recreate all +certificates when this one expires. So it may be a good idea to use a +long period, eg. 3650 days (roughly 10 years). You need to specify that +the certificates belongs to an authority. The certificate is used to +sign other certificates.<br> +</li> +</ol> +<h3>Sample Screen Session</h3> +<p>Text in red is user input. Please note that for some questions, there is no +user input given. This means the default was accepted by simply pressing the +enter key. +<code><pre> +[root@rgf9dev sample]# <font color="red">certtool --generate-privkey --outfile ca-key.pem --bits 2048</font> +Generating a 2048 bit RSA private key... +[root@rgf9dev sample]# <font color="red">certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem</font> +Generating a self signed certificate... +Please enter the details of the certificate's distinguished name. Just press enter to ignore a field. +Country name (2 chars): <font color="red">US</font> +Organization name: <font color="red">SomeOrg</font> +Organizational unit name: <font color="red">SomeOU</font> +Locality name: <font color="red">Somewhere</font> +State or province name: <font color="red">CA</font> +Common name: <font color="red">someName (not necessarily DNS!)</font> +UID: +This field should not be used in new certificates. +E-mail: +Enter the certificate's serial number (decimal): + + +Activation/Expiration time. +The certificate will expire in (days): <font color="red">3650</font> + + +Extensions. +Does the certificate belong to an authority? (Y/N): <font color="red">y</font> +Path length constraint (decimal, -1 for no constraint): +Is this a TLS web client certificate? (Y/N): +Is this also a TLS web server certificate? (Y/N): +Enter the e-mail of the subject of the certificate: <font color="red">someone@example.net</font> +Will the certificate be used to sign other certificates? (Y/N): <font color="red">y</font> +Will the certificate be used to sign CRLs? (Y/N): +Will the certificate be used to sign code? (Y/N): +Will the certificate be used to sign OCSP requests? (Y/N): +Will the certificate be used for time stamping? (Y/N): +Enter the URI of the CRL distribution point: +X.509 Certificate Information: + Version: 3 + Serial Number (hex): 485a365e + Validity: + Not Before: Thu Jun 19 10:35:12 UTC 2008 + Not After: Sun Jun 17 10:35:25 UTC 2018 + Subject: C=US,O=SomeOrg,OU=SomeOU,L=Somewhere,ST=CA,CN=someName (not necessarily DNS!) + Subject Public Key Algorithm: RSA + Modulus (bits 2048): + d9:9c:82:46:24:7f:34:8f:60:cf:05:77:71:82:61:66 + 05:13:28:06:7a:70:41:bf:32:85:12:5c:25:a7:1a:5a + 28:11:02:1a:78:c1:da:34:ee:b4:7e:12:9b:81:24:70 + ff:e4:89:88:ca:05:30:0a:3f:d7:58:0b:38:24:a9:b7 + 2e:a2:b6:8a:1d:60:53:2f:ec:e9:38:36:3b:9b:77:93 + 5d:64:76:31:07:30:a5:31:0c:e2:ec:e3:8d:5d:13:01 + 11:3d:0b:5e:3c:4a:32:d8:f3:b3:56:22:32:cb:de:7d + 64:9a:2b:91:d9:f0:0b:82:c1:29:d4:15:2c:41:0b:97 + Exponent: + 01:00:01 + Extensions: + Basic Constraints (critical): + Certificate Authority (CA): TRUE + Subject Alternative Name (not critical): + RFC822name: someone@example.net + Key Usage (critical): + Certificate signing. + Subject Key Identifier (not critical): + fbfe968d10a73ae5b70d7b434886c8f872997b89 +Other Information: + Public Key Id: + fbfe968d10a73ae5b70d7b434886c8f872997b89 + +Is the above information ok? (Y/N): <font color="red">y</font> + + +Signing certificate... +[root@rgf9dev sample]# <font color="red">chmod 400 ca-key.pem</font> +[root@rgf9dev sample]# <font color="red">ls -l</font> +total 8 +-r-------- 1 root root 887 2008-06-19 12:33 ca-key.pem +-rw-r--r-- 1 root root 1029 2008-06-19 12:36 ca.pem +[root@rgf9dev sample]# +</pre></code> +<p><font color="red"><b>Be sure to safeguard ca-key.pem!</b> Nobody except the CA itself +needs to have it. If some third party obtains it, you security is broken!</font> +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/tls_cert_ca.jpg b/doc/tls_cert_ca.jpg Binary files differnew file mode 100644 index 00000000..f2da0454 --- /dev/null +++ b/doc/tls_cert_ca.jpg diff --git a/doc/tls_cert_client.html b/doc/tls_cert_client.html new file mode 100644 index 00000000..dbe7961b --- /dev/null +++ b/doc/tls_cert_client.html @@ -0,0 +1,91 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: client setup</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-07-03)</i></small></p> + +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +</ul> + +<h3>Setting up a client</h3> +<p>In this step, we configure a client machine. We from our scenario, we use +zuse.example.net. You need to do the same steps for all other clients, too (in the +example, that meanst turng.example.net). The client check's the server's identity and +talks to it only if it is the expected server. This is a very important step. +Without it, you would not detect man-in-the-middle attacks or simple malicious servers +who try to get hold of your valuable log data. +<span style="float: left"> +<script type="text/javascript"><!-- +google_ad_client = "pub-3204610807458280"; +/* rsyslog doc inline */ +google_ad_slot = "5958614527"; +google_ad_width = 125; +google_ad_height = 125; +//--> +</script> +<script type="text/javascript" +src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> +</script> +</span> +<p><center><img src="tls_cert_100.jpg"></center> +<p>Steps to do: +<ul> +<li>make sure you have a functional CA (<a href="tls_cert_ca.html">Setting up the CA</a>) +<li>generate a machine certificate for zuse.example.net (follow instructions in + <a href="tls_cert_machine.html">Generating Machine Certificates</a>) +<li>make sure you copy over ca.pem, machine-key.pem ad machine-cert.pem to the client. +Ensure that no user except root can access them (<b>even read permissions are really bad</b>). +<li>configure the client so that it checks the server identity and sends messages only +if the server identity is known. Please note that you have the same options as when +configuring a server. However, we now use a single name only, because there is only one +central server. No using wildcards make sure that we will exclusively talk to that server +(otherwise, a compromised client may take over its role). If you load-balance to different +server identies, you obviously need to allow all of them. It still is suggested to use +explcit names. +</ul> +<p><b>At this point, please be reminded once again that your security needs may be quite different from +what we assume in this tutorial. Evaluate your options based on your security needs.</b> +<h3>Sample syslog.conf</h3> +<p>Keep in mind that this rsyslog.conf sends messages via TCP, only. Also, we do not +show any rules to write local files. Feel free to add them. +<code><pre> +# make gtls driver the default +$DefaultNetstreamDriver gtls + +# certificate files +$DefaultNetstreamDriverCAFile /rsyslog/protected/ca.pem +$DefaultNetstreamDriverCertFile /rsyslog/protected/machine-cert.pem +$DefaultNetstreamDriverKeyFile /rsyslog/protected/machine-key.pem + +$ActionSendStreamDriverAuthMode x509/name +$ActionSendStreamDriverPermittedPeer central.example.net +$ActionSendStreamDriverMode 1 # run driver in TLS-only mode +*.* @@central.example.net:10514 # forward everything to remote server +</pre></code> +<p>Note: the example above forwards every message to the remote server. Of course, +you can use the normal filters to restrict the set of information that is sent. +Depending on your message volume and needs, this may be a smart thing to do. +<p><font color="red"><b>Be sure to safeguard at least the private key (machine-key.pem)!</b> +If some third party obtains it, you security is broken!</font> +<h2>Copyright</h2> +<p>Copyright © 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/tls_cert_errmsgs.html b/doc/tls_cert_errmsgs.html new file mode 100644 index 00000000..d002174c --- /dev/null +++ b/doc/tls_cert_errmsgs.html @@ -0,0 +1,103 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: error messages</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-06-17)</i></small></p> + +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +<li><a href="tls_cert_errmsgs.html">Frequently seen Error Messages</a> +</ul> + +<h3>Error Messages</h3> +<p>This page covers error message you may see when setting up +<span style="float: left"> +<script type="text/javascript"><!-- +google_ad_client = "pub-3204610807458280"; +/* rsyslog doc inline */ +google_ad_slot = "5958614527"; +google_ad_width = 125; +google_ad_height = 125; +//--> +</script> +<script type="text/javascript" +src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> +</script> +</span> +<a href="http://www.rsyslog.com">rsyslog</a> with TLS. Please note that many +of the message stem back to the TLS library being used. In those cases, there is +not always a good explanation available in rsyslog alone. +<p>A single error typically results in two or more message being emitted: (at +least) one is the actual error cause, followed by usually one message with additional +information (like certificate contents). In a typical system, these message should +immediately follow each other in your log. Kepp in mind that they are reported +as syslog.err, so you need to capture these to actually see errors (the default +rsyslog.conf's shipped by many systems will do that, recording them e.g. in +/etc/messages). +<h3>certificate invalid</h3> +<p>Sample: +<code> +not permitted to talk to peer, certificate invalid: <font color="red">insecure algorithm</font> +</code> +<p>This message may occur during connection setup. It indicates that the remote peer's +certificate can not be accepted. The reason for this is given in the message part that +is shown in red. Please note that this red part directly stems back to the TLS library, +so rsyslog does acutally not have any more information about the reason. +<p>With GnuTLS, the following reasons have been seen in practice: +<h4>insecure algorith</h4> +<p>The certificate contains information on which encryption algorithms are to be used. +This information is entered when the certificate is created. +Some older alogrithms are no longer secure and the TLS library does not accept +them. Thus the connection request failed. The cure is to use a certificate with sufficiently secure +alogorithms. +<p>Please note that noi encryption algorithm is totally secure. It only is secure based +on our current knowledge AND on computing power available. As computers get more and more +powerful, previously secure algorithms become insecure over time. As such, algorithms +considered secure today may not be accepted by the TLS library in the future. +<p>So in theory, after a system upgrade, a connection request may fail with the "insecure +algorithm" failure without any change in rsyslog configuration or certificates. This could be +caused by a new perception of the TLS library of what is secure and what not. +<h3>GnuTLS error -64</h3> +<p>Sample: <code>unexpected GnuTLS error -64 in nsd_gtls.c:517: Error while reading file.</code> +<p>This error points to an encoding error witht the pem file in question. It means "base 64 encoding error". +From my experience, it can be caused by a couple of things, some of them not obvious: +<ul> +<li>You specified a wrong file, which is not actually in .pem format +<li>The file was incorrectly generated +<li>I think I have also seen this when I accidently swapped private key files and +certificate files. So double-check the type of file you are using. +<li>It may even be a result of an access (permission) problem. In theory, that +should lead to another error, but in practice it sometimes seems to lead to +this -64 error. +</ul> +<h3>info on invalid cert</h3> +<p>Sample: +<code> +info on invalid cert: peer provided 1 certificate(s). Certificate 1 info: certificate valid from Wed Jun 18 11:45:44 2008 to Sat Jun 16 11:45:53 2018; Certificate public key: RSA; DN: C=US,O=Sample Corp,OU=Certs,L=Somehwere,ST=CA,CN=somename; Issuer DN: C=US,O=Sample Corp,OU=Certs,L=Somewhere,ST=CA,CN=somename,EMAIL=xxx@example.com; SAN:DNSname: machine.example.net; +</code> +<p>This is <b>not</b> an error message in itself. It always follows the actual error message and +tells you what is seen in the peer's certificate. This is done to give you a chance to evaluate +the certificate and better understand why the initial error message was issued. +<p>Please note that you can NOT diagnose problems based on this message alone. It follows +in a number of error cases and does not pinpoint any problems by itself. +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/tls_cert_machine.html b/doc/tls_cert_machine.html new file mode 100644 index 00000000..5ecde0d1 --- /dev/null +++ b/doc/tls_cert_machine.html @@ -0,0 +1,172 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: generating the machine certificate</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-06-18)</i></small></p> + +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +</ul> + +<h3>generating the machine certificate</h3> +<p>In this step, we generate certificates for each of the machines. Please note +that both clients and servers need certificates. The certificate identifies each +machine to the remote peer. The DNSName specified inside the certificate can +<span style="float: left"> +<script type="text/javascript"><!-- +google_ad_client = "pub-3204610807458280"; +/* rsyslog doc inline */ +google_ad_slot = "5958614527"; +google_ad_width = 125; +google_ad_height = 125; +//--> +</script> +<script type="text/javascript" +src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> +</script> +</span> +be specified inside the $<object>PermittedPeer config statements. +<p>For now, we assume that a single person (or group) is responsible for the whole +rsyslog system and thus it is OK if that single person is in posession of all +machine's private keys. This simplification permits us to use a somewhat less +complicated way of generating the machine certificates. So, we generate both the private +and public key on the CA (which is NOT a server!) and then copy them over to the +respective machines. +<p>If the roles of machine and CA administrators are split, the private key must +be generated by the machine administrator. This is done via a certificate request. +This request is then sent to the CA admin, which in turn generates the certificate +(containing the public key). The CA admin then sends back the certificate to the +machine admin, who installs it. That way, the CA admin never get's hold of the +machine's private key. Instructions for this mode will be given in a later revision +of this document. +<p><b>In any case, it is vital that the machine's private key is protected. Anybody +able to obtain that private key can imporsonate as the machine to which it belongs, thus +breaching your security.</b> +<h3>Sample Screen Session</h3> +<p>Text in red is user input. Please note that for some questions, there is no +user input given. This means the default was accepted by simply pressing the +enter key. +<p><b>Please note:</b> you need to substitute the names specified below with values +that match your environment. Most importantly, machine.example.net must be replaced +by the actual name of the machine that will be using this certificate. For example, +if you generate a certificate for a machine named "server.example.com", you need +to use that name. If you generate a certificate for "client.example.com", you need +to use this name. Make sure that each machine certificate has a unique name. If not, +you can not apply proper access control. +<code><pre> +[root@rgf9dev sample]# <font color="red">certtool --generate-privkey --outfile key.pem --bits 2048</font> +Generating a 2048 bit RSA private key... +[root@rgf9dev sample]# <font color="red">certtool --generate-request --load-privkey key.pem --outfile request.pem</font> +Generating a PKCS #10 certificate request... +Country name (2 chars): <font color="red">US</font> +Organization name: <font color="red">SomeOrg</font> +Organizational unit name: <font color="red">SomeOU</font> +Locality name: <font color="red">Somewhere</font> +State or province name: <font color="red">CA</font> +Common name: <font color="red">machine.example.net</font> +UID: +Enter a challenge password: +[root@rgf9dev sample]# <font color="red">certtool --generate-certificate --load-request request.pem --outfile cert.pem --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem</font> +Generating a signed certificate... +Enter the certificate's serial number (decimal): + + +Activation/Expiration time. +The certificate will expire in (days): 1000 + + +Extensions. +Does the certificate belong to an authority? (Y/N): <font color="red">n</font> +Is this a TLS web client certificate? (Y/N): <font color="red">y</font> +Is this also a TLS web server certificate? (Y/N): <font color="red">y</font> +Enter the dnsName of the subject of the certificate: <font color="red">machine.example.net</font> <i>{This is the name of the machine that will use the certificate}</i> +Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (Y/N): +Will the certificate be used for encryption (RSA ciphersuites)? (Y/N): +X.509 Certificate Information: + Version: 3 + Serial Number (hex): 485a3819 + Validity: + Not Before: Thu Jun 19 10:42:54 UTC 2008 + Not After: Wed Mar 16 10:42:57 UTC 2011 + Subject: C=US,O=SomeOrg,OU=SomeOU,L=Somewhere,ST=CA,CN=machine.example.net + Subject Public Key Algorithm: RSA + Modulus (bits 2048): + b2:4e:5b:a9:48:1e:ff:2e:73:a1:33:ee:d8:a2:af:ae + 2f:23:76:91:b8:39:94:00:23:f2:6f:25:ad:c9:6a:ab + 2d:e6:f3:62:d8:3e:6e:8a:d6:1e:3f:72:e5:d8:b9:e0 + d0:79:c2:94:21:65:0b:10:53:66:b0:36:a6:a7:cd:46 + 1e:2c:6a:9b:79:c6:ee:c6:e2:ed:b0:a9:59:e2:49:da + c7:e3:f0:1c:e0:53:98:87:0d:d5:28:db:a4:82:36:ed + 3a:1e:d1:5c:07:13:95:5d:b3:28:05:17:2a:2b:b6:8e + 8e:78:d2:cf:ac:87:13:15:fc:17:43:6b:15:c3:7d:b9 + Exponent: + 01:00:01 + Extensions: + Basic Constraints (critical): + Certificate Authority (CA): FALSE + Key Purpose (not critical): + TLS WWW Client. + TLS WWW Server. + Subject Alternative Name (not critical): + DNSname: machine.example.net + Subject Key Identifier (not critical): + 0ce1c3dbd19d31fa035b07afe2e0ef22d90b28ac + Authority Key Identifier (not critical): + fbfe968d10a73ae5b70d7b434886c8f872997b89 +Other Information: + Public Key Id: + 0ce1c3dbd19d31fa035b07afe2e0ef22d90b28ac + +Is the above information ok? (Y/N): <font color="red">y</font> + + +Signing certificate... +[root@rgf9dev sample]# <font color="red">rm -f request.pem</font> +[root@rgf9dev sample]# <font color="red">ls -l</font> +total 16 +-r-------- 1 root root 887 2008-06-19 12:33 ca-key.pem +-rw-r--r-- 1 root root 1029 2008-06-19 12:36 ca.pem +-rw-r--r-- 1 root root 1074 2008-06-19 12:43 cert.pem +-rw-r--r-- 1 root root 887 2008-06-19 12:40 key.pem +[root@rgf9dev sample]# # it may be a good idea to rename the files to indicate where they belong to +[root@rgf9dev sample]# <font color="red">mv cert.pem machine-cert.pem</font> +[root@rgf9dev sample]# <font color="red">mv key.pem machine-key.pem</font> +[root@rgf9dev sample]# +</pre></code> +<h3>Distributing Files</h3> +<p>Provide the machine with: +<ul> +<li>a copy of ca.pem +<li>cert.pem +<li>key.pem +</ul> +<p>This is how the relevant part of rsyslog.conf looks on the target machine: +<p> +<code><pre> +$DefaultNetstreamDriverCAFile /home/rger/proj/rsyslog/sample/ca.pem +$DefaultNetstreamDriverCertFile /home/rger/proj/rsyslog/sample/machine-cert.pem +$DefaultNetstreamDriverKeyFile /home/rger/proj/rsyslog/sample/machine-key.pem +</pre></code> +<p><b><font color="red">Never</font> provide anyone with ca-key.pem!</b> Also, make sure +nobody but the machine in question gets hold of key.pem. +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/tls_cert_scenario.html b/doc/tls_cert_scenario.html new file mode 100644 index 00000000..7973532b --- /dev/null +++ b/doc/tls_cert_scenario.html @@ -0,0 +1,63 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: scenario</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-06-17)</i></small></p> + +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +<li><a href="tls_cert_errmsgs.html">Frequently seen Error Messages</a> +</ul> + +<h3>Sample Scenario</h3> +<p>We have a quite simple scenario. There is one central syslog server, +<span style="float: left"> +<script type="text/javascript"><!-- +google_ad_client = "pub-3204610807458280"; +/* rsyslog doc inline */ +google_ad_slot = "5958614527"; +google_ad_width = 125; +google_ad_height = 125; +//--> +</script> +<script type="text/javascript" +src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> +</script> +</span> +named central.example.net. These server is being reported to by two Linux +machines with name zuse.example.net and turing.example.net. Also, there is a +third client - ada.example.net - which send both its own messages to the central +server but also forwards messages receive from an UDP-only capable router. We +hav decided to use ada.example.net because it is in the same local network +segment as the router and so we enjoy TLS' security benefits for forwarding the +router messages inside the corporate network. All systems (except the router) use +<a href="http://www.rsyslog.com/">rsyslog</a> as the syslog software.</p> +<p><center><img src="tls_cert_100.jpg"></center> +<p>Please note that the CA must not necessarily be connected to the rest of the +network. Actually, it may be considered a security plus if it is not. If the CA +is reachable via the regular network, it should be sufficiently secured (firewal +rules et al). Keep in mind that if the CA's security is breached, your overall +system security is breached. +<p>In case the CA is compromised, you need to regenerate the CA's certificate as well +as all individual machines certificates. +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/tls_cert_server.html b/doc/tls_cert_server.html new file mode 100644 index 00000000..9c68db5d --- /dev/null +++ b/doc/tls_cert_server.html @@ -0,0 +1,118 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: central server setup</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-06-18)</i></small></p> + +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +</ul> + +<h3>Setting up the Central Server</h3> +<p>In this step, we configure the central server. We assume it accepts messages only +via TLS protected plain tcp based syslog from those peers that are explicitely permitted +to send to it. The picture below show our configuration. This step configures +the server central.example.net. +<span style="float: left"> +<script type="text/javascript"><!-- +google_ad_client = "pub-3204610807458280"; +/* rsyslog doc inline */ +google_ad_slot = "5958614527"; +google_ad_width = 125; +google_ad_height = 125; +//--> +</script> +<script type="text/javascript" +src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> +</script> +</span> +<p><center><img src="tls_cert_100.jpg"></center> +<p>Steps to do: +<ul> +<li>make sure you have a functional CA (<a href="tls_cert_ca.html">Setting up the CA</a>) +<li>generate a machine certificate for central.example.net (follow instructions in + <a href="tls_cert_machine.html">Generating Machine Certificates</a>) +<li>make sure you copy over ca.pem, machine-key.pem ad machine-cert.pem to the central server. +Ensure that no user except root can access them (<b>even read permissions are really bad</b>). +<li>configure the server so that it accepts messages from all machines in the +example.net domain that have certificates from your CA. Alternatively, you may also +precisely define from which machine names messages are accepted. See sample rsyslog.conf +below. +</ul> +In this setup, we use wildcards to ease adding new systems. We permit the server to accept +messages from systems whos names match *.example.net. +<pre><code> +$InputTCPServerStreamDriverPermittedPeer *.example.net +</code></pre> +This will match zuse.example.net and +turing.example.net, but NOT pascal.otherdepartment.example.net. If the later would be desired, +you can (and need) to include additional permitted peer config statments: +<pre><code> +$InputTCPServerStreamDriverPermittedPeer *.example.net +$InputTCPServerStreamDriverPermittedPeer *.otherdepartment.example.net +$InputTCPServerStreamDriverPermittedPeer *.example.com +</code></pre> +<p>As can be seen with example.com, the different permitted peers need NOT to be in a single +domain tree. Also, individual machines can be configured. For example, if only zuse, turing +and ada should be able to talk to the server, you can achive this by: +<pre><code> +$InputTCPServerStreamDriverPermittedPeer zuse.example.net +$InputTCPServerStreamDriverPermittedPeer turing.example.net +$InputTCPServerStreamDriverPermittedPeer ada.example.net +</code></pre> +<p>As an extension to the (upcoming) IETF syslog/tls standard, you can specify some text +together with a domain component wildcard. So "*server.example.net", "server*.example.net" +are valid permitted peers. However "server*Fix.example.net" is NOT a valid wildcard. The +IETF standard permits no text along the wildcards. +<p>The reason we use wildcards in the default setup is that it makes it easy to add systems +without the need to change the central server's configuration. It is important to understand that +the central server will accept names <b>only</b> (no exception) if the client certificate was +signed by the CA we set up. So if someone tries to create a malicious certificate with +a name "zuse.example.net", the server will <b>not</b> accept it. So a wildcard is safe +as long as you ensure CA security is not breached. Actually, you authorize a client by issuing +the certificate to it. +<p><b>At this point, please be reminded once again that your security needs may be quite different from +what we assume in this tutorial. Evaluate your options based on your security needs.</b> +<h3>Sample syslog.conf</h3> +<p>Keep in mind that this rsyslog.conf accepts messages via TCP, only. The only other +source accepted is messages from the server itself. +<code><pre> +$ModLoad imuxsock # local messages +$ModLoad imtcp # TCP listener + +# make gtls driver the default +$DefaultNetstreamDriver gtls + +# certificate files +$DefaultNetstreamDriverCAFile /rsyslog/protected/ca.pem +$DefaultNetstreamDriverCertFile /rsyslog/protected/machine-cert.pem +$DefaultNetstreamDriverKeyFile /rsyslog/protected/machine-key.pem + +$InputTCPServerStreamDriverAuthMode x509/name +$InputTCPServerStreamDriverPermittedPeer *.example.net +$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode +$InputTCPServerRun 10514 # start up listener at port 10514 +</pre></code> +<p><font color="red"><b>Be sure to safeguard at least the private key (machine-key.pem)!</b> +If some third party obtains it, you security is broken!</font> +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/tls_cert_summary.html b/doc/tls_cert_summary.html new file mode 100644 index 00000000..8e003bc8 --- /dev/null +++ b/doc/tls_cert_summary.html @@ -0,0 +1,66 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: Summary</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-07-03)</i></small></p> + +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +</ul> + +<h3>Summary</h3> +<p>If you followed the steps outlined in this documentation set, you now have +<span style="float: left"> +<script type="text/javascript"><!-- +google_ad_client = "pub-3204610807458280"; +/* rsyslog doc inline */ +google_ad_slot = "5958614527"; +google_ad_width = 125; +google_ad_height = 125; +//--> +</script> +<script type="text/javascript" +src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> +</script> +</span> +a reasonable (for most needs) secure setup for the following environment: +<center><img src="tls_cert_100.jpg"></center> +<p>You have learned about the security decisions involved and which we +made in this example. <b>Be once again reminded that you must make sure yourself +that whatever you do matches your security needs!</b> There is no guarantee that +what we generally find useful actually is. It may even be totally unsuitable for +your environment. +<p>In the example, we created a rsyslog certificate authority (CA). Guard the CA's +files. You need them whenever you need to create a new machine certificate. We also saw how +to generate the machine certificates themselfs and distribute them to the individual +machines. Also, you have found some configuration samples for a sever, a client and +a syslog relay. Hopefully, this will enable you to set up a similar system in many +environments. +<p>Please be warned that you defined some expiration dates for the certificates. +After they are reached, the certificates are no longer valid and rsyslog will NOT +accept them. At that point, syslog messages will no longer be transmitted (and rsyslogd +will heavily begin to complain). So it is a good idea to make sure that you renew the +certificates before they expire. Recording a reminder somewhere is probably a good +idea. +<p>If you have any more questions, please visit the <a href="http://kb.monitorware.com/rsyslog-f40.html">rsyslog forum</a> and simply ask ;) +<h2>Copyright</h2> +<p>Copyright (c) 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/tls_cert_udp_relay.html b/doc/tls_cert_udp_relay.html new file mode 100644 index 00000000..f4740ce7 --- /dev/null +++ b/doc/tls_cert_udp_relay.html @@ -0,0 +1,105 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>TLS-protected syslog: UDP relay setup</title> +</head> +<body> + +<h1>Encrypting Syslog Traffic with TLS (SSL)</h1> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2008-07-03)</i></small></p> + +<ul> +<li><a href="rsyslog_secure_tls.html">Overview</a> +<li><a href="tls_cert_scenario.html">Sample Scenario</a> +<li><a href="tls_cert_ca.html">Setting up the CA</a> +<li><a href="tls_cert_machine.html">Generating Machine Certificates</a> +<li><a href="tls_cert_server.html">Setting up the Central Server</a> +<li><a href="tls_cert_client.html">Setting up syslog Clients</a> +<li><a href="tls_cert_udp_relay.html">Setting up the UDP syslog relay</a> +<li><a href="tls_cert_summary.html">Wrapping it all up</a> +</ul> + +<h3>Setting up the UDP syslog relay</h3> +<p>In this step, we configure the UDP relay ada.example.net. +As a reminder, that machine relays messages from a local router, which only +supports UDP syslog, to the central syslog server. The router does not talk +directly to it, because we would like to have TLS protection for its sensitve +logs. If the router and the syslog relay are on a sufficiently secure private +network, this setup can be considered reasonable secure. In any case, it is the +best alternative among the possible configuration scenarios. +<span style="float: left"> +<script type="text/javascript"><!-- +google_ad_client = "pub-3204610807458280"; +/* rsyslog doc inline */ +google_ad_slot = "5958614527"; +google_ad_width = 125; +google_ad_height = 125; +//--> +</script> +<script type="text/javascript" +src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> +</script> +</span> +<p><center><img src="tls_cert_100.jpg"></center> +<p>Steps to do: +<ul> +<li>make sure you have a functional CA (<a href="tls_cert_ca.html">Setting up the CA</a>) +<li>generate a machine certificate for ada.example.net (follow instructions in + <a href="tls_cert_machine.html">Generating Machine Certificates</a>) +<li>make sure you copy over ca.pem, machine-key.pem ad machine-cert.pem to the client. +Ensure that no user except root can access them (<b>even read permissions are really bad</b>). +<li>configure the client so that it checks the server identity and sends messages only +if the server identity is known. +</ul> +<p>These were essentially the same steps as for any +<a href="tls_cert_client.html">TLS syslog client</a>. We now need to add the +capability to forward the router logs: +<ul> +<li>make sure that the firewall rules permit message recpetion on UDP port 514 (if you use +a non-standard port for UDP syslog, make sure that port number is permitted). +<li>you may want to limit who can send syslog messages via UDP. A great place to do this +is inside the firewall, but you can also do it in rsyslog.conf via an $AllowedSender +directive. We have used one in the sample config below. Please be aware that this is +a kind of weak authentication, but definitely better than nothing... +<li>add the UDP input plugin to rsyslog's config and start a UDP listener +<li>make sure that your forwarding-filter permits to forward messages received +from the remote router to the server. In our sample scenario, we do not need to +add anything special, because all messages are forwarded. This includes messages +received from remote hosts. +</ul> +<p><b>At this point, please be reminded once again that your security needs may be quite different from +what we assume in this tutorial. Evaluate your options based on your security needs.</b> +<h3>Sample syslog.conf</h3> +<p>Keep in mind that this rsyslog.conf sends messages via TCP, only. Also, we do not +show any rules to write local files. Feel free to add them. +<code><pre> +# start a UDP listener for the remote router +$ModLoad imudp # load UDP server plugin +$AllowedSender UDP, 192.0.2.1 # permit only the router +$UDPServerRun 514 # listen on default syslog UDP port 514 + +# make gtls driver the default +$DefaultNetstreamDriver gtls + +# certificate files +$DefaultNetstreamDriverCAFile /rsyslog/protected/ca.pem +$DefaultNetstreamDriverCertFile /rsyslog/protected/machine-cert.pem +$DefaultNetstreamDriverKeyFile /rsyslog/protected/machine-key.pem + +$ActionSendStreamDriverAuthMode x509/name +$ActionSendStreamDriverPermittedPeer central.example.net +$ActionSendStreamDriverMode 1 # run driver in TLS-only mode +*.* @@central.example.net:10514 # forward everything to remote server +</pre></code> +<p><font color="red"><b>Be sure to safeguard at least the private key (machine-key.pem)!</b> +If some third party obtains it, you security is broken!</font> +<h2>Copyright</h2> +<p>Copyright © 2008 <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p> Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at +<a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a>.</p> +</body></html> diff --git a/doc/troubleshoot.html b/doc/troubleshoot.html new file mode 100644 index 00000000..e655c2ef --- /dev/null +++ b/doc/troubleshoot.html @@ -0,0 +1,98 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>troubleshooting rsyslog</title></head> +<body> +<h2>troubleshooting rsyslog</h2> +<p><b>Having trouble with <a href="http://www.rsyslog.com">rsyslog</a>?</b> +This page provides some tips on where to look for help and what to do +if you need to ask for assistance. This page is continously being expanded. +<p>Useful troubleshooting ressources are: +<ul> +<li>The <a href="http://www.rsyslog.com/doc">rsyslog documentation</a> - note that the online version always covers +the most recent development version. However, there is a version-specific +doc set in each tarball. If you installed rsyslog from a package, there usually +is a rsyslog-doc package, that often needs to be installed separately. +<li>The <a href="http://wiki.rsyslog.com">rsyslog wiki</a> provides user tips and experiences. +<li>Check <a href="http://bugzilla.adiscon.com">the bugzilla</a> to see if your problem is a known +(and even fixed ;)) bug. +</ul> +<p><b>Malformed Messages and Message Properties</b> +<p>A common trouble source are <a href="syslog_parsing.html">ill-formed syslog messages</a>, which +lead to to all sorts of interesting problems, including malformed hostnames and dates. +Read the quoted guide to find relief. +<p><b>Configuration Problems</b> +<p>Rsyslog 3.21.1 and above has been enhanced to support extended configuration checking. +It offers a special command line switch (-N1) that puts it into "config verfication mode". +In that mode, it interprets and check the configuration file, but does not startup. This +mode can be used in parallel to a running instance of rsyslogd. +<p>To enable it, run rsyslog interactively as follows: +<p><b><i>/path/to/rsyslogd -f/path/to/config-file -N1</i></b> +<p>You should also specify other options you usually give (like -c3 and whatever else). +Any problems experienced are reported to stderr [aka "your screen" (if not redirected)]. +<p><b>Asking for Help</b> +<p>If you can't find the answer yourself, you should look at these places for +community help. +<ul> +<li>The <a href="http://kb.monitorware.com/rsyslog-f40.html">rsyslog forum</a>. This is +the preferred method of obtaining support. +<li>The <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">rsyslog mailing list</a>. +This is a low-volume list which occasional gets traffic spikes. +The mailing list is probably a good place for complex questions. +</ul> +<p><b>Debug Log</b> +<p>If you ask for help, there are chances that we need to ask for an rsyslog debug log. +The debug log is a detailled report of what rsyslog does during processing. As such, it may +even be useful for your very own troubleshooting. People have seen things inside their debug +log that enabled them to find problems they did not see before. So having a look at the +debug log, even before asking for help, may be useful. +<p>Note that the debug log contains most of those things we consider useful. This is a lot +of information, but may still be too few. So it sometimes may happen that you will be asked +to run a specific version which has additional debug output. Also, we revise from time to +time what is worth putting into the standard debug log. As such, log content may change +from version to version. We do not guarantee any specific debug log contents, so do not +rely on that. The amount of debug logging can also be controlled via some environment +options. Please see <a href="debug.html">debugging support</a> for further details. +<p>In general, it is advisable to run rsyslogd in the foreground to obtain the log. +To do so, make sure you know which options are usually used when you start rsyslogd +as a background daemon. Let's assume "-c3" is the only option used. Then, do the following: +<ul> +<li>make sure rsyslogd as a daemon is stopped (verify with ps -ef|grep rsyslogd) +<li>make sure you have a console session with root permissions +<li>run rsyslogd interactively: /sbin/rsyslogd ..your options.. -dn > logfile +<br>where "your options" is what you usually use. /sbin/rsyslogd is the full path +to the rsyslogd binary (location different depending on distro). +In our case, the command would be +<br>/sbin/rsyslogd -c3 -dn > logfile +<li>press ctrl-C when you have sufficient data (e.g. a device logged a record) +<br><b>NOTE: rsyslogd will NOT stop automatically - you need to ctrl-c out of it!</b> +<li>Once you have done all that, you can review logfile. It contains the debug output. +<li>When you are done, make sure you re-enable (and start) the background daemon! +</ul> +<p>If you need to submit the logfile, you may want to check if it contains any +passwords or other sensitive data. If it does, you can change it to some <b>consistent</b> +meaningless value. <b>Do not delete the lines</b>, as this renders the debug log +unusable (and makes Rainer quite angry for wasted time, aka significantly reduces the chance +he will remain motivated to look at your problem ;)). For the same reason, make sure +whatever you change is change consistently. Really! +<p>Debug log file can get quite large. Before submitting them, it is a good idea to zip them. +Rainer has handled files of around 1 to 2 GB. If your's is larger ask before submitting. Often, +it is sufficient to submit the first 2,000 lines of the log file and around another 1,000 around +the area where you see a problem. Also, +ask you can submit a file via private mail. Private mail is usually a good way to go for large files +or files with sensitive content. However, do NOT send anything sensitive that you do not want +the outside to be known. While Rainer so far made effort no to leak any sensitive information, +there is no guarantee that doesn't happen. If you need a guarantee, you are probably a +candidate for a <a href="professional_support.html">commercial support contract</a>. Free support +comes without any guarantees, include no guarantee on confidentiality +[aka "we don't want to be sued for work were are not even paid for ;)]. +<b>So if you submit debug logs, do so at your sole risk</b>. By submitting them, you accept +this policy. +<p>[<a href="manual.html">manual index</a>] +[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 2 or higher.</font></p> +</body> +</html> + diff --git a/doc/v3compatibility.html b/doc/v3compatibility.html new file mode 100644 index 00000000..ad8776bb --- /dev/null +++ b/doc/v3compatibility.html @@ -0,0 +1,196 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Compatibility notes for rsyslog v3</title> + +<meta name="KEYWORDS" content="syslog, mysql, syslog to mysql, howto"></head> +<body> +<h1>Compatibility Notes for rsyslog v3</h1> +<p><small><i>Written by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> +(2008-03-28)</i></small></p> +<p>Rsyslog aims to be a drop-in replacement for sysklogd. +However, version 3 has some considerable enhancements, which lead to +some backward compatibility issues both in regard to sysklogd and +rsyslog v1 and v2. Most of these issues are avoided by default by not +specifying the -c option on the rsyslog command line. That will enable +backwards-compatibility mode. However, please note that things may be +suboptimal in backward compatibility mode, so the advise is to work +through this document, update your rsyslog.conf, remove the no longer +supported startup options and then add -c3 as the first option to the +rsyslog command line. That will enable native mode.</p> +<p>Please note that rsyslogd helps you during that process by +logging appropriate messages about compatibility mode and +backwards-compatibility statemtents automatically generated. You may +want your syslogd log for those. They immediately follow rsyslogd's +startup message.</p> +<h2>Inputs</h2> +<p>With v2 and below, inputs were automatically started together +with rsyslog. In v3, inputs are optional! They come in the form of +plug-in modules. +<font color="#ff0000"><b>At least one input module +must be loaded to make rsyslog do any useful work.</b></font> +The config file directives doc briefly lists which config statements +are available by which modules.</p> +<p>It is suggested that input modules be loaded in the top part +of the config file. Here is an example, also highlighting the most +important modules:</p> +<p><b>$ModLoad immark # provides --MARK-- +message capability<br> +$ModLoad imudp # provides UDP syslog reception<br> +$ModLoad imtcp # provides TCP syslog reception<br> +</b><b>$ModLoad imgssapi # provides GSSAPI syslog +reception<br> +</b><b>$ModLoad imuxsock # provides support for local +system logging (e.g. +via logger command)<br> +$ModLoad imklog # provides kernel logging support (previously done +by rklogd)</b></p> +<h2>Command Line Options</h2> +<p>A number of command line options have been removed. New config +file directives have been added for them. The -h and -e option have +been removed even in compatibility mode. They are ignored but an +informative message is logged. Please note that -h was never supported +in v2, but was silently ignored. It disappeared some time ago in the +final v1 builds. It can be replaced by applying proper filtering inside +syslog.conf.</p> +<h2>-c option / Compatibility Mode</h2> +<p>The -c option is new and tells rsyslogd about the desired +backward compatibility mode. It must always be the first option on the +command line, as it influences processing of the other options. To use +the rsyslog v3 native +interface, specify -c3. To use compatibility mode , +either do not use -c at all or use -c<vers> where vers is +the +rsyslog version that it shall be compatible to. Use -c0 to be +command-line compatible to sysklogd.</p><p><span style="font-weight: bold;">Please note that rsyslogd issues warning messages if the -c3 command line option is not given.</span> +This is to alert you that your are running in compatibility mode. +Compatibility mode interfers with you rsyslog.conf commands and may +cause some undesired side-effects. It is meant to be used with a plain +old rsyslog.conf - if you use new features, things become messy. So the +best advise is to work through this document, convert your options and +config file and then use rsyslog in native mode. In order to aid you in +this process, rsyslog logs every compatibility-mode config file +directive it has generated. So you can simply copy them from your +logfile and paste them to the config.</p> +<h2>-e Option</h2> +This option is no longer supported, as the "last message repeated n +times" feature is now turned off by default. We changed this default +because this feature is causing a lot of trouble and we need to make it +either go away or change the way it works. For more information, please +see our dedicted <a href="http://www.rsyslog.com/PNphpBB2-viewtopic-p-1130.phtml">forum +thread on "last message repeated n times"</a>. This thread also +contains information on how to configure rsyslogd so that it continues +to support this feature (as long as it is not totally removed). +<h2>-m Option</h2> +<p>The -m command line option is emulated in compatibiltiy mode. +To replace it, use the following config directives (compatibility mode +auto-generates them):</p> +<p><b>$ModLoad immark<br> +$MarkMessageInterval 1800 # 30 minutes</b></p> +<h2>-r Option</h2> +<p>Is no longer available in native mode. However, it +is +understood in compatibility mode (if no -c option is given). Use the <b>$UDPSeverRun +<port></b> config file directives. You can now also +set the local address the server should listen to via <b>$UDPServerAddress +<ip></b> config directive.</p> +<p>The following example configures an UDP syslog server at the +local address 192.0.2.1 on port 514:</p> +<p><b>$ModLoad imudp<br> +$UDPServerAddress 192.0.2.1 # this MUST be before the $UDPServerRun +directive!<br> +$UDPServerRun 514</b></p> +<p>"$UDPServerAddress *" means listen on all local interfaces. +This is the default if no directive is specified.</p> +<p>Please note that now multiple listeners are supported. For +example, you can do the following:</p> +<p><b>$ModLoad imudp<br> +$UDPServerAddress 192.0.2.1 # this MUST be before the $UDPServerRun +directive!<br> +$UDPServerRun 514<br> +$UDPServerAddress * # all local interfaces<br> +$UDPServerRun 1514</b></p> +<p>These config file settings run two listeners: one +at 192.0.2.1:514 and one on port 1514, which listens on all local +interfaces.</p> +<h2>Default port for UDP (and TCP) Servers</h2> +<p>Please note that with pre-v3 rsyslogd, a service database +lookup was made when a UDP server was started and no port was +configured. Only if that failed, the IANA default of 514 was used. For +TCP servers, this lookup was never done and 514 always used if no +specific port was configured. For consitency, both TCP and UDP now use +port 514 as default. If a lookup is desired, you need to specify it in +the "Run" directive, e.g. "<i>$UDPServerRun syslog</i>".</p> +<h2>klogd</h2> +<p>klogd has (finally) been replaced by a loadable input module. +To enable klogd functionality, do</p> +<p><b>$ModLoad imklog</b></p> +<p>Note that this can not be handled by the compatibility layer, +as klogd was a separate binary.A limited set of klogd command line +settings is now supported +via rsyslog.conf. That set of configuration directives is to be +expanded. </p> +<h2>Output File Syncing</h2> +Rsyslogd tries to keep as compatible to +stock syslogd as possible. As such, it retained stock syslogd's default +of syncing every file write if not specified otherwise (by placing a +dash in front of the output file name). While this was a useful feature +in past days where hardware was much less reliable and UPS seldom, this +no longer is useful in today's worl. Instead, the syncing is a high +performace hit. With it, rsyslogd writes files around 50 *times* slower +than without it. It also affects overall system performance due to the +high IO activity. In rsyslog v3, syncing has been turned off by +default. This is done via a specific configuration directive +"$ActionFileEnableSync on/off" which is off by default. So even if +rsyslogd finds sync selector lines, it ignores them by default. In +order to enable file syncing, the administrator must specify +"$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures +that syncing only happens in some installations where the administrator +actually wanted that (performance-intense) feature. In the fast +majority of cases (if not all), this dramatically increases rsyslogd +performance without any negative effects. +<h2>Output File Format</h2> +<p>Rsyslog supports high precision RFC 3339 timestamps and puts these into +local log files by default. This is a departure from previous syslogd +behaviour. We decided to sacrify some backward-compatibility in an +effort to provide a better logging solution. Rsyslog has been +supporting the high-precision timestamps for over three years as of +this writing, but nobody used them because they were not default (one +may also assume that most people didn't even know about them). Now, we +are writing the great high-precision time stamps, which greatly aid in +getting the right sequence of logging events. If you do not like that, +you can easily turn them off by placing +</p><p style="font-weight: bold;"><code>$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat</code> +</p><p>right at the start of your rsyslog.conf. This will use the +previous format. Please note that the name is case-sensitive and must +be specificed exactly as shown above. Please also note that you can of +course use any other format of your liking. To do so, simply specify +the template to use or set a new default template via the +$ActionFileDefaultTemplate directive. Keep in mind, though, that +templates must be defined before they are used.</p><p>Keep in mind that +when receiving messages from remote hosts, the timestamp is just as +precise as the remote host provided it. In most cases, this means you +will only a receive a standard timestamp with second precision. If +rsyslog is running at the remote end, you can configure it to provide +high-precision timestamps (see below).</p><h2>Forwarding Format</h2><p>When +forwarding messages to remote syslog servers, rsyslogd by default uses +the plain old syslog format with second-level resolution inside the +timestamps. We could have made it emit high precision timestamps. +However, that would have broken almost all receivers, including earlier +versions of rsyslog. To avoid this hassle, high-precision timestamps +need to be explicitely enabled. To make this as painless as possible, +rsyslog comes with a canned template that contains everything +necessary. To enable high-precision timestamps, just use:</p><p style="font-weight: bold;"><code>$ActionForwardDefaultTemplate RSYSLOG_ForwardFormat # for plain TCP and UDP</code></p><p style="font-weight: bold;"><code>$ActionGSSForwardDefaultTemplate RSYSLOG_ForwardFormat # for GSS-API</code></p><p>And, of course, you can always set different forwarding formats by just specifying the right template.</p><p>If +you are running in a system with only rsyslog 3.12.5 and above in the +receiver roles, it is suggested to add one (or both) of the above +statements to the top of your rsyslog.conf (but after the $ModLoad's!) +- that will enable you to use the best in timestamp support availble. +Please note that when you use this format with other receivers, they +will probably become pretty confused and not detect the timestamp at +all. In earlier rsyslog versions, for example, that leads to +duplication of timestamp and hostname fields and disables the detection +of the orignal hostname in a relayed/NATed environment. So use the new +format with care. </p><h2>Queue Modes for the Main Message Queue</h2> +<p>Either "FixedArray" or "LinkedList" is recommended. "Direct" +is available, but should not be used except for a very good reason +("Direct" disables queueing and will potentially lead to message loss +on the input side).</p> +</body></html> diff --git a/doc/version_naming.html b/doc/version_naming.html index a685f5ff..8c1b9187 100644 --- a/doc/version_naming.html +++ b/doc/version_naming.html @@ -1,33 +1,130 @@ -<html> -<head> -<title>rsyslog bugs and annoyances</title> -</head> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>rsyslog version naming</title></head> <body> <h1>Version Naming</h1> -<p>This document briefly outlines the strategy for naming versions. It applies -to versions 1.0.0 and above. Versions below that are all unstable and have a -different naming schema.</p> -<p><b>Please note that version naming is currently being changed. There is a -<a href="http://rgerhards.blogspot.com/2007/08/on-rsyslog-versions.html">blog +<p style="font-weight: bold;">This is the proposal on how versions should be named in the future:</p><p>Rsyslog version naming has undergone a number of changes in +the past. Our sincere hopes is that the scheme outlined here will serve +us well for the future. In general, a three-number versioning scheme +with a potential development state indication is used. It follows this +pattern:</p> +<p>major.minor.patchlevel[-devstate]</p> +<p>where devstate has some forther structure: +-<releaseReason><releaseNumber></p> +<p>All stable builds come without the devstate part. All unstable +development version come with it.</p> +<p>The <span style="font-weight: bold;">major</span> +version is incremented whenever something really important happens. A +single new feature, even if important, does not justify an increase in +the major version. There is no hard rule when the major version needs +an increment. It mostly is a soft factor, when the developers and/or +the community think there has been sufficient change to justify that. +Major version increments are expected to happen quite infrequently, +maybe around once a year. A major version increment has important +implications from the support side: without support contracts, the +current major version's last stable release and the last stable release +of the version immediately below it are supported (Adiscon, the rsyslog +sponsor, offers <a href="professional_support.html">support contracts</a> covering all other versions).</p> +<p>The <span style="font-weight: bold;">minor</span> version is +incremented whenever a non-trivial new feature is planned to be added. +Triviality of a feature is simply determined by time estimated to +implement a feature. If that's more than a few days, it is considered a +non-trivial feature. Whenever a new minor version is begun, the desired +feature is identified and will be the primary focus of that major.minor +version. Trivial features may justify a new minor version if they +either do not look trivial from the user's point of view or change +something quite considerable (so we need to alert users). A minor +version increment may also be done for some other good reasons that the +developers have.</p> +<p>The <span style="font-weight: bold;">patchlevel</span> is incremented whenever there is a bugfix or very minor feature added to a (stable or development) release.</p><p>The <span style="font-weight: bold;">devstate</span> +is important during development of a feature. It helps the developers +to release versions with new features to the general public and in the +hope that this will result in some testing. To understand how it works, +we need to look at the release cycle: As already said, at the start of +a new minor version, a new non-trivial feature to be implemented in +that version is selected. Development on this feature begins. At the +current pace of development, getting initial support for such a +non-trivial feature typically takes between two and four weeks. During +this time, new feature requests come in. Also, we may find out that it +may be just the right time to implement some not yet targeted feature +requests. A reason for this is that the minor release's feature focus +is easier to implement if the other feature is implemented first. This +is a quite common thing to happen. So development on the primary focus +may hold for a short period while we implement something else. Even +unrelated, but very trivial feature requests (maybe an hour's worth of +time to implement), may be done in between. Once we have implemented +these things, we would like to release as quickly as possible (even +more if someone has asked for the feature). So we do not like to wait +for the original focus feature to be ready (what could take maybe three +more weeks). As a result, we release the new features. But that version +will also include partial code of the focus feature. Typically this +doesn't hurt as long as noone tries to use it (what of course would +miserably fail). But still, part of the new code is already in it. When +we release such a "minor-feature enhanced" but "focus-feature not yet +completed" version, we need a way to flag it. In current thinking, that +is using a "<span style="font-weight: bold;">-mf<version></span>" <span style="font-weight: bold;">devstate</span> +in the version number ("mf" stands for "minor feature"). Version +numbers for -mf releases start at 0 for the first release and are +monotonically incremented. Once the focus feature has been fully +implemented, a new version now actually supporting that feature will be +released. Now, the release reason is changed to the well-know "<span style="font-weight: bold;">-rc<version></span>" +where "rc" stands for release candidate. For the first release +candidate, the version starts at 0 again and is incremented +monotonically for each subsequent release. Please note that a -rc0 may +only have bare functionality but later -rc's have a richer one. If new +minor features are implemented and released once we have reached rc +stage, still a new rc version is issued. The difference between "mf" +and "rc" is simply the presence of the desired feature. No support is +provided for -mf versions once the first -rc version has been released. +And only the most current -rc version is supported.</p><p>The -rc is +removed and the version declared stable when we think it has undergone +sufficient testing and look sufficiently well. Then, it'll turn into a +stable release. Stable minor releases never receive non-trivial new +features. There may be more than one -rc releases without a stable +release present at the same time. In fact, most often we will work on +the next minor development version while the previous minor version is +still a -rc because it is not yet considered sufficiently stable.</p><p>Note: <span style="font-weight: bold;">the +absence of the -devstate part indicates that a release is stable. +Following the same logic, any release with a -devstate part is unstable.</span></p><p>A quick sample: </p><p>4.0.0 +is the stable release. We begin to implement relp, moving to +major.minor to 4.1. While we develop it, someone requests a trivial +feature, which we implement. We need to release, so we will have +4.1.0-mf0. Another new feature is requested, move to 4.1.0-mf2. A first +version of RELP is implemented: 4.1.0-rc0. A new trivial feature is +implemented: 4.1.0-rc1. Relp is being enhanced: 4.1.0-rc2. We now feel +RELP is good enough for the time being and begin to implement TLS on +plain /Tcp syslog: logical increment to 4.2. Now another new feature in +that tree: 4.2.0-mf0. Note that we now have 4.0.0 (stable) and +4.1.0-rc2 and 4.1.0-mf0 (both devel). We find a big bug in RELP coding. +Two new releases: 4.1.0-rc3, 4.2.0-mf1 (the bug fix acts like a +non-focus feature change). We release TLS: 4.2.0-rc0. Another RELP bug +fix 4.1.0-rc4, 4.2.0-rc1. After a while, RELP is matured: 4.1.0 +(stable). Now support for 4.0.x stable ends. It, however, is still +provided for 3.x.x (in the actual case 2.x.x, because v3 was under the +old naming scheme and now stable v3 was ever released).</p><p style="font-weight: bold;">This is how it is done so far:</p><p>This document briefly outlines the strategy for naming +versions. It applies to versions 1.0.0 and above. Versions below that +are all unstable and have a different naming schema.</p> +<p><b>Please note that version naming is currently being +changed. There is a +<a href="http://rgerhards.blogspot.com/2007/08/on-rsyslog-versions.html">blog post about future rsyslog versions</a>.</b></p> -<p>The major version is incremented whenever a considerate, major features have -been added. This is expected to happen quite infrequently.</p> -<p>The minor version number is incremented whenever there is "sufficient need" -(at the discretion of the developers). There is a notable difference between -stable and unstable branches. The <b>stable branch</b> always has a minor -version number in the range from 0 to 9. It is expected that the stable branch -will receive bug and security fixes only. So the range of minor version numbers -should be quite sufficient.</p> -<p>For the <b>unstable branch</b>, minor version numbers always start at 10 and -are incremented as needed (again, at the discretion of the developers). Here, -new minor versions include both fixes as well as new features (hopefully most of -the time). They are expected to be released quite often.</p> -<p>The patch level (third number) is incremented whenever a really minor thing -must be added to an existing version. This is expected to happen quite -infrequently.</p> -<p>In general, the unstable branch carries all new development. Once it -concludes with a sufficiently-enhanced, quite stable version, a new major stable -version is assigned.</p> - -</body> -</html> +<p>The major version is incremented whenever a considerate, major +features have been added. This is expected to happen quite infrequently.</p> +<p>The minor version number is incremented whenever there is +"sufficient need" (at the discretion of the developers). There is a +notable difference between stable and unstable branches. The <b>stable +branch</b> always has a minor version number in the range from 0 +to 9. It is expected that the stable branch will receive bug and +security fixes only. So the range of minor version numbers should be +quite sufficient.</p> +<p>For the <b>unstable branch</b>, minor version +numbers always start at 10 and are incremented as needed (again, at the +discretion of the developers). Here, new minor versions include both +fixes as well as new features (hopefully most of the time). They are +expected to be released quite often.</p> +<p>The patch level (third number) is incremented whenever a +really minor thing must be added to an existing version. This is +expected to happen quite infrequently.</p> +<p>In general, the unstable branch carries all new development. +Once it concludes with a sufficiently-enhanced, quite stable version, a +new major stable version is assigned.</p> +</body></html>
\ No newline at end of file |
