diff options
Diffstat (limited to 'doc')
29 files changed, 1127 insertions, 202 deletions
diff --git a/doc/Makefile.am b/doc/Makefile.am index a447cddf..4832e757 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -90,7 +90,9 @@ html_files = \ rsconf1_moddir.html \ rsconf1_repeatedmsgreduction.html \ rsconf1_resetconfigvariables.html \ + rsconf1_rulesetcreatemainqueue.html \ rsconf1_umask.html \ + rulesetparser.html \ v3compatibility.html \ v4compatibility.html \ v5compatibility.html \ @@ -130,6 +132,8 @@ grfx_files = \ dataflow.png \ queue_analogy_tv.png \ gssapi.png \ + rfc5424layers.png \ + src/rfc5424layers.dia \ rsyslog-vers.png EXTRA_DIST = $(html_files) $(grfx_files) diff --git a/doc/debug.html b/doc/debug.html index de77f04a..46759986 100644 --- a/doc/debug.html +++ b/doc/debug.html @@ -1,41 +1,145 @@ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head> -<meta http-equiv="Content-Language" content="en"><title>Debug Support</title></head> +<html> +<head> +<meta http-equiv="Content-Language" content="en"> +<title>Rsyslog Debug Support</title></head> <body> -<h1>Debug Support</h1> +<h1>Rsyslog Debug Support</h1> <p> -Rsyslog provides a number of debug aids. Some of them are activated by +Rsyslog provides a number of debug aides. Some of them are activated by adding the --enable-rtinst ./configure option ("rtinst" means runtime instrumentation). Turning debugging on obviously costs some performance (in some cases considerable). </p> <p>This is document is just being created and thus terse.</p> -<p style="font-weight: bold;">Signals supported</p> -<p>SIGUSR1 - turns debug messages on and off (expect this signal -to go away over time)</p> -<p>SIGUSR2 - outputs debug information (including active threads +<h2>Signals supported</h2> +<p><b>SIGUSR1</b> - turns debug messages on and off. Note that for this +signal to work, rsyslogd must be running with debugging enabled, either +via the -d command line switch or the environment options specified below. +It is <b>not</b> required that rsyslog was compiled with debugging enabled +(but depending on the settings this may lead to better debug info). +<p><b>SIGUSR2</b> - outputs debug information (including active threads and a call stack) for the state when SIGUSR2 was received. This is a -one-time output. Can be sent as often as the user likes.</p> -<p style="font-weight: bold;">Environment Variables</p> -<p>There are two environment variables that set several debug settings. The "RSYSLOG_DEBUGLOG" (sample: RSYSLOG_DEBUGLOG="/path/to/debuglog/") +one-time output. Can be sent as often as the user likes. +<p><b>Note:</b> this signal <b>may go away</b> in later releases and may +be replaced by something else.</p> +<h2>Environment Variables</h2> +<p>There are two environment variables that set several debug settings: +<ul> +<li>The "RSYSLOG_DEBUGLOG" (sample: RSYSLOG_DEBUGLOG="/path/to/debuglog/") writes (allmost) all debug message to the specified log file in addition to stdout. Some system messages (e.g. segfault or abort message) are not written to the -file as we can not capture them. Runtime debug support is controlled by -"RSYSLOG_DEBUG". It contains an option string with the following -options possible (all are case insensitive):</p><ul><li><span style="font-weight: bold;">LogFuncFlow</span> - print out the logical flow of functions (entering and exiting them)</li><li><span style="font-weight: bold;">FileTrace</span> - specifies which files to trace LogFuncFlow. If <span style="font-weight: bold;">not</span> +file as we can not capture them. +<li>Runtime debug support is controlled by "RSYSLOG_DEBUG". +<p>The "RSYSLOG_DEBUG" environment variable contains an option string with the following +options possible (all are case insensitive):</p> +<ul> +<li><b>LogFuncFlow</b> - print out the logical flow of functions (entering and exiting them)</li> +<li><b>FileTrace</b> - specifies which files to trace LogFuncFlow. If <b>not</b> set (the default), a LogFuncFlow trace is provided for all files. Set to limit it to the files specified. FileTrace may be specified multiple times, one file each (e.g. export RSYSLOG_DEBUG="LogFuncFlow -FileTrace=vm.c FileTrace=expr.c"</li><li><span style="font-weight: bold;">PrintFuncDB</span> - print the content of the debug function database whenever debug information is printed (e.g. abort case)!</li><li><span style="font-weight: bold;">PrintAllDebugInfoOnExit</span> - print all debug information immediately before rsyslogd exits (<span style="font-weight: bold; font-style: italic;">currently not implemented!</span>)</li><li><span style="font-weight: bold;">PrintMutexAction</span> - print mutex action as it happens. Useful for finding deadlocks and such.</li><li><span style="font-weight: bold;">NoLogTimeStamp</span> - do not prefix log lines with a timestamp (default is to do that).</li><li><span style="font-weight: bold;">NoStdOut</span> - do not emit debug messages to stdout. If RSYSLOG_DEBUGLOG is not set, this means no messages will be displayed at all.</li><li><span style="font-weight: bold;">help</span> - display a very short list of commands - hopefully a life saver if you can't access the documentation...</li></ul> +FileTrace=vm.c FileTrace=expr.c"</li> +<li><b>PrintFuncDB</b> - print the content of the debug function database whenever debug information is printed (e.g. abort case)!</li> +<li><b>PrintAllDebugInfoOnExit</b> - print all debug information immediately before rsyslogd exits (<span style="font-weight: bold; font-style: italic;">currently not implemented!</span>)</li> +<li><b>PrintMutexAction</b> - print mutex action as it happens. Useful for finding deadlocks and such.</li> +<li><b>NoLogTimeStamp</b> - do not prefix log lines with a timestamp (default is to do that).</li> +<li><b>NoStdOut</b> - do not emit debug messages to stdout. If RSYSLOG_DEBUGLOG is not set, this means no messages will be displayed at all.</li> +<li><b>Debug</b> - if present, turns on the debug system and enables debug output +<li><b>DebugOnDemand</b> - if present, turns on the debug system but does not enable +debug output itself. You need to send SIGUSR1 to turn it on when desired. +<li><b>help</b> - display a very short list of commands - hopefully a life saver if you can't access the documentation...</li> +</ul> +</ul> +<h3>Why Environment Variables?</h3> +<p>You may ask why we use environment variables for debug-system parameters and not +the usual rsyslog.conf configuration commands. After all, environment variables force one +to change distro-specific configuration files, whereas regular configuration directives +would fit nicely into the one central rsyslog.conf. +<p>The problem here is that many settings of the debug system must be initialized +before the full rsyslog engine starts up. At that point, there is no such thing like +rsyslog.conf or the objects needed to process it present in an running instance. +And even if we would enable to change settings some time later, that would mean that +we can not correctly monitor (and debug) the initial startup phase of rsyslogd. What +makes matters worse is that during this startup phase (and never again later!) some +of the base debug structure needs to be created, at least if the build is +configured for that (many of these things only happen in --enable-rtinst mode). So +if we do not initialize the debug system <b>before</b> actually startig up the +rsyslog core, we get a number of data structures wrong. +<p>For these reasons, we utilize environment variables to initialize and configure +the debugging system. We understand this may be somewhat painful, but now you know +there are at least some good reasons for doing so. +<h2>Getting debug information from a running Instance</h2> +<p>It is possible to obtain debugging information from a running instance, but this requires +some setup. We assume that the instance runs in the background, so debug output to +stdout is not desired. As such, all debug information needs to go into a log file. +<p>To create this setup, you need to <ul> +<li>point the RSYSLOG_DEBUGLOG environment variable to a file that is accessible +during the while runtime (we strongly suggest a file in the local file system!) +<li>set RSYSLOG_DEBUG at least to "DebugOnDeman NoStdOut" +<li>make sure these environment variables are set in the correct (distro-specifc) +startup script if you do not run rsyslogd interactively </ul> +<p>These settings enable the capability to react to SIGUSR1. The signal will toggle +debug status when received. So send it one to turn debug loggin on, and send it again +to turn debug logging off again. The third time it will be turned on again ... and so on. +<p>On a typical system, you can signal rsyslogd as follows: +<pre> +kill -USR1 `cat /var/run/rsyslogd.pid` +</pre> +Important: there are backticks around the "cat"-command. If you use the regular +quote it won't work. The debug log will show whether debug logging has been turned +on or off. There is no other indication of the status. +<p>Note: running with DebugOnDemand by itself does in practice not have any performance +toll. However, switching debug logging on has a severe performance toll. Also, debug +logging synchronizes much of the code, removing a lot of concurrency and thus +potential race conditions. As such, the very same running instance may behave +very differently with debug logging turned on vs. off. The on-demand debug log +functionality is considered to be very valuable to analyze hard-to-find bugs that +only manifest after a long runtime. Turning debug logging on a failing instance +may reveal the cause of the failure. However, depending on the failure, debug logging +may not even be successfully be turned on. Also note that with this rsyslog version we cannot +obtain any debug information on events that happened <i>before</i> debug logging was +turned on. +<p>If an instance hangs, it is possible to obtain some useful information about the current +threads and their calling stack by sending SIGUSR2. However, the usefulness of that +information is very much depending on rsyslog compile-time settings, must importantly +the --enable-rtinst configure flag. Note that activating this option causes additional overhead +and slows down rsyslgod considerable. So if you do that, you need to check if it is +capable to handle the workload. Also, threading behavior is modified by the +runtime instrumentation. +<p>Sending SIGUSR2 writes new process state information to the log file each time +it is sent. So it may be useful to do that from time to time. It probably is most +useful if the process seems to hang, in which case it may (may!) be able to output +some diagnostic information on the current processing state. In that case, turning +on the mutex debugging options (see above) is probably useful. +<h2>Interpreting the Logs</h2> +<p>Debug logs are primarily meant for rsyslog developers. But they may still provide valuable +information to users. Just be warned that logs sometimes contains informaton the looks like +an error, but actually is none. We put a lot of extra information into the logs, and there +are some cases where it is OK for an error to happen, we just wanted to record it inside +the log. The code handles many cases automatically. So, in short, the log may not make sense to +you, but it (hopefully) makes sense to a developer. Note that we developers often need +many lines of the log file, it is relatively rare that a problem can be diagnosed by +looking at just a couple of (hundered) log records. +<h2>Security Risks</h2> +<p>The debug log will reveal potentially sensible information, including user accounts and +passwords, to anyone able to read the log file. As such, it is recommended to properly +guard access to the log file. Also, an instance running with debug log enabled runs much +slower than one without. An attacker may use this to place carry out a denial-of-service +attack or try to hide some information from the log file. As such, it is suggested to +enable DebugOnDemand mode only for a reason. Note that when no debug mode is enabled, +SIGUSR1 and SIGUSR2 are completely ignored. +<p>When running in any of the debug modes (including on demand mode), an interactive +instance of rsyslogd can be aborted by pressing ctl-c. +<p> <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> -project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer -Gerhards</a> and +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> -</body></html>
\ No newline at end of file +</body> +</html> diff --git a/doc/design.tex b/doc/design.tex index 53d25313..a3ec8f45 100644 --- a/doc/design.tex +++ b/doc/design.tex @@ -22,7 +22,7 @@ rgerhards@adiscon.com} \maketitle \begin{abstract} -This paper describes rsyslog design and internals. It is created to facilitate a discussion about the implementation of "batched queue processing". As such, it does not describe the full design of rsyslog but rather those elements that are relevant to queues. However, the document may be expanded in the future. +This paper describes rsyslog design and internals. It is created to facilitate a discussion about the implementation of "batched queue processing". As such, it does not describe the full design of rsyslog but rather those elements that are relevant to queues. However, the document may be expanded in the future. This is work in progress and should be considered with care. It is NOT updated during all phases of development. \end{abstract} \tableofcontents @@ -811,9 +811,37 @@ b) we push the failed message back to the main queue, but with an indication that it failed in an action. This is harder to implement and most importantly harder to understand/configure, but more flexible +\section{Network Stream Subsystem} +The idea of network streams was introduced when we implemented RFC5425 (syslog over TLS) in 2008. The core idea is to encapsulate all stream-oriented network data transfer into a single transport layer and make the upper layers independent of actual transport being used. This is in line with the traditional layer approaches in communication systems. + +Under this system, the upper layer provides plugins to send and receive streams of syslog data. Framing is provided by the upper layer. The upper layer itself is integrated in input and output plugins, which then are used to provide application-level syslog message objects to and from the rsyslog core. To these upper layers, the netstream layer provides reliable and sequenced message delivery with much of the same semantics as a usual TCP stream. + +\begin{figure} +\begin{center} +\includegraphics[scale=0.4]{tls.jpeg} +\end{center} +\caption{Objects at the Network Stream Layer} +\label{fig_netstream_objects} +\end{figure} + +At the netstream layer, we have a small set of generic classes, which are used for setup of the drivers and driver parameters. This is a very thin layer, mostly a wrapper. Once an actual lower-level netstream driver has been loaded, all parameters are passed through to it. + +Please note that both in theory and practice netstream drivers may call back into different netstream drivers. For example, the GnuTLS RFC5425 driver loads and calls back into the plain tcp driver, simply because that driver provides part of the required functionality and there is no point in re-implementing it for GnuTLS. + +The netstream driver layer does not only provide read and write calls but supports i/o multiplexing. To do so, it offers an interface that follows select() semantics. That permits an upper-layer comonent to request being blocked unless some data arrives. Note that due to the subleties in TLS processing, the upper layer may be awoken while there is no upper-layer work to do. This will properly be indicated by the netstream subsystem, is not an error and must be accepted and poperly handled by the upper layer. + +Using the nestream layer, we do not need to modify the input and output plugins while at the same time we can add additional transport providers. One weak spot in this design is the current configuration process. With the current system, we need to provide one configuration statement per driver property and we need to hardcode this. So if a new driver would require new properties, we still would need to modify the upper layers. This is unfortunate, but the current config system does not provide for any better way to handle the situation. Once we are able to create a new config system, we will address this by providing the ability to pass a string of parameters onto the driver, which will then have the ability to parse its content. So once we do this, we need to modify the driver interface, but the end result would be a simlification. + +So far, only drivers for GnuTLS and plain tcp are provided. However, during the design of the layer we also looked at openssl and Mozilla Network Security Services as well as kept an eye on the needs of Kerberos. In theory, it should not be a major problem to write drivers for these systems (but it most probably still is a lot of work to do). + +A final note on Kerberos: in order to keep compatible with previous protocol handling and due to constraints in testing environment and knowledge, we still support Kerberos not via the netstream layer but via special extension into the input and output modules. That, too, is unfortunate, but given the current resources at hand, there is no alternative to handling in that way. We would be very interested in moving over Kerberos to a netstream driver and any volunteer would be very welcome. + \section{Future Development} This section covers topics that can not currently be developed, but where important thoughts came up in discussions. For obvious reasons, the section has brainstorming character. +\subsection{Lock-Free Queuing} +On a very busy system, lock contention can limit performance. We should investigate ways to apply lock-free algorithms inside rsyslog. It is believed that at least for some scenarios, lock-free algorigthms can be applied with great benefit. To do so, we should introduce new queue modes, which will use very different semantics from what is described so far for the queue engine. Most importantly, in lock-free mode we will have limits on the number of producers and we will most probably not be able to guarantee audit-grade processing. The later is not a problem, because there are ample use cases that do not require audit-gradeness. + \subsection{Audit-Grade High Performance Queue Storage Driver} An audit grade driver must ensure that no message is lost, but should also be able to handle large workloads. The sequential disk driver does not support the later. diff --git a/doc/expression.html b/doc/expression.html index 9e37cb7a..c401d9ab 100644 --- a/doc/expression.html +++ b/doc/expression.html @@ -1,17 +1,22 @@ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html><head> -<meta http-equiv="Content-Language" content="en"><title>Expressions</title></head> +<meta http-equiv="Content-Language" content="en"> +<title>Expressions in rsyslog</title></head> <body> -<a href="rsyslog_conf_filter.html">back</a> -<h1>Expressions</h1> +<a href="rsyslog_conf_filter.html">back to rsyslog filter conditions</a> +<h1>Expressions in rsyslog</h1> <p>Rsyslog supports expressions at a growing number of places. So -far, they are supported for filtering messages.</p><p>Expression support is provided by RainerScript. For now, please see the formal expression definition in <a href="rainerscript.html">RainerScript ABNF</a>. It is the "expr" node.</p><p>C-like comments (/* some comment */) are supported <span style="font-weight: bold;">inside</span> the expression, but not yet in the rest of the configuration file.</p><p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +far, they are supported for filtering messages.</p> +<p>Expression support is provided by RainerScript. Please see the +<a href="rainerscript.html">RainerScript documentation</a> for more details.</p> +<p>C-like comments (/* some comment */) are supported <b>inside</b> the expression, +but not yet in the rest of the configuration file.</p> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> -project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer -Gerhards</a> and +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body></html> diff --git a/doc/imtcp.html b/doc/imtcp.html index 0ccdecc7..434b3903 100644 --- a/doc/imtcp.html +++ b/doc/imtcp.html @@ -1,21 +1,23 @@ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head> -<meta http-equiv="Content-Language" content="en"><title>TCP Syslog Input Module</title></head> +<html> +<head> +<meta http-equiv="Content-Language" content="en"> +<title>TCP Syslog Input Module</title> +</head> + <body> -<a href="rsyslog_conf_modules.html">back</a> +<a href="rsyslog_conf_modules.html">back to rsyslog module overview</a> <h1>TCP Syslog Input Module</h1> <p><b>Module Name: imtcp</b></p> -<p><b>Author: </b>Rainer Gerhards -<rgerhards@adiscon.com></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Multi-Ruleset Support: </b>since 4.5.0 and 5.1.1 <p><b>Description</b>:</p> <p>Provides the ability to receive syslog messages via TCP. -Encryption can be provided by using <a href="rsyslog_stunnel.html">stunnel</a> -(an alternative is the use -the <a href="imgssapi.html">imgssapi</a> -modul).</p> -<p>Multiple receivers may be configured by -specifying +Encryption is natively provided by selecting the approprioate network stream driver and +can also be provided by using <a href="rsyslog_stunnel.html">stunnel</a> +(an alternative is the use the <a href="imgssapi.html">imgssapi</a> module).</p> +<p>Multiple receivers may be configured by specifying $InputTCPServerRun multiple times. This is available since version 4.3.1, earlier versions do NOT support it. </p> @@ -49,8 +51,7 @@ after loading imtcp, otherwise it may have no effect.</li> Starts a TCP server on selected port</li> <li>$InputTCPMaxListeners <number><br> Sets the maximum number of listeners (server ports) supported. Default is 20. This must be set before the first $InputTCPServerRun directive.</li> -<li>$InputTCPMaxSessions <number><br> -Sets the maximum number of sessions supported. Default is 200. This must be set before the first $InputTCPServerRun directive</li> +<li>$InputTCPMaxSessions <number><br> Sets the maximum number of sessions supported. Default is 200. This must be set before the first $InputTCPServerRun directive</li> <li>$InputTCPServerStreamDriverMode <number><br> Sets the driver mode for the currently selected <a href="netstream.html">network stream driver</a>. <number> is driver specifc.</li> <li>$InputTCPServerInputName <name><br> @@ -63,6 +64,8 @@ Sets the authentication mode for the currently selected <a href="netstream.html" Sets permitted peer IDs. Only these peers are able to connect to the listener. <id-string> semantics depend on the currently selected AuthMode and <a href="netstream.html">network stream driver</a>. PermittedPeers may not be set in anonymous modes.</li> +<li>$InputTCPServerBindRuleset <ruleset><br> +Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> </ul> <b>Caveats/Known Bugs:</b> <ul> @@ -70,20 +73,22 @@ AuthMode and <a href="netstream.html">network stream driver</a>. Permitted <li>can not be loaded together with <a href="imgssapi.html">imgssapi</a> (which includes the functionality of imtcp)</li> </ul> -<p><b>Sample:</b></p> -<p>This sets up a TCP server on port 514:<br> +<p><b>Example:</b></p> +<p>This sets up a TCP server on port 514 and permits it to accept up to 500 connections:<br> </p> -<textarea rows="15" cols="60">$ModLoad imtcp # -needs to be done just once +<textarea rows="15" cols="60">$ModLoad imtcp # needs to be done just once +$InputTCPMaxSessions 500 $InputTCPServerRun 514 </textarea> +<p>Note that the parameters (here: max sessions) need to be set <b>before</b> the listener +is activated. Otherwise, the parameters will not apply. +</p> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> -<p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> +<p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer -Gerhards</a> and +Copyright © 2008,2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> -</body></html> +</body> +</html> diff --git a/doc/imudp.html b/doc/imudp.html new file mode 100644 index 00000000..f0e86307 --- /dev/null +++ b/doc/imudp.html @@ -0,0 +1,58 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html> +<head> +<meta http-equiv="Content-Language" content="en"> +<title>TCP Syslog Input Module</title> +</head> + +<body> +<a href="rsyslog_conf_modules.html">back to rsyslog module overview</a> + +<h1>UDP Syslog Input Module</h1> +<p><b>Module Name: imudp</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Multi-Ruleset Support: </b>since 5.3.2 +<p><b>Description</b>:</p> +<p>Provides the ability to receive syslog messages via UDP. +<p>Multiple receivers may be configured by specifying +$UDPServerRun multiple times. +</p> +<p><b>Configuration Directives</b>:</p> +<ul> +<li>$UDPServerAddress <IP><br> +local IP address (or name) the UDP listens should bind to</li> +<li>$UDPServerRun <port><br> +former -r<port> option, default 514, start UDP server on this +port, "*" means all addresses</li> +<li>$UDPServerTimeRequery <nbr-of-times><br> +this is a performance +optimization. Getting the system time is very costly. With this setting, imudp can +be instructed to obtain the precise time only once every n-times. This logic is +only activated if messages come in at a very fast rate, so doing less frequent +time calls should usually be acceptable. The default value is two, because we have +seen that even without optimization the kernel often returns twice the identical time. +You can set this value as high as you like, but do so at your own risk. The higher +the value, the less precise the timestamp. +<li>$InputUDPServerBindRuleset <ruleset><br> +Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>currently none known</li> +</ul> +<p><b>Sample:</b></p> +<p>This sets up an UPD server on port 514:<br> +</p> +<textarea rows="15" cols="60">$ModLoad imudp # needs to be done just once +$UDPServerRun 514 +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/imuxsock.html b/doc/imuxsock.html index 472470a0..15c365a6 100644 --- a/doc/imuxsock.html +++ b/doc/imuxsock.html @@ -46,6 +46,18 @@ Ignore timestamps included in the messages, applies to messages received via the <li><b>$SystemLogSocketName</b> <name-of-socket> -- former -p option</li> <li><b>$SystemLogFlowControl</b> [on/<b>off</b>] - specifies if flow control should be applied to the system log socket.</li> +<li><b>$InputUnixListenSocketCreatePath</b> [on/<b>off</b>] - create directories in the socket path +if they do not already exist. They are created with 0755 permissions with the owner being the process under +which rsyslogd runs. The default is not to create directories. Keep in mind, though, that rsyslogd always +creates the socket itself if it does not exist (just not the directories by default). +<br>Note that this statement affects the +next $AddUnixListenSocket directive that follows in sequence in the configuration file. It never works +on the system log socket (where it is deemed unnecessary). Also note that it is automatically +being reset to "off" after the $AddUnixListenSocket directive, so if you would have it active +for two additional listen sockets, you need to specify it in front of each one. This option is primarily considered +useful for defining additional sockets that reside on non-permanent file systems. As rsyslogd probably starts +up before the daemons that create these sockets, it is a vehicle to enable rsyslogd to listen to those +sockets even though their directories do not yet exist. [available since 4.7.0 and 5.3.0]</li> <li><b>$AddUnixListenSocket</b> <name-of-socket> adds additional unix socket, default none -- former -a option</li> <li><b>$InputUnixListenSocketHostName</b> <hostname> permits to override the hostname that shall be used inside messages taken from the <b>next</b> $AddUnixListenSocket socket. Note that @@ -57,20 +69,32 @@ that the local hostname can be overridden in cases where that is desired.</li> <br> This documentation is sparse and incomplete. <p><b>Sample:</b></p> -<p>The following sample is the minimum setup required to accept syslog messages from applications running on the local system.<br> +<p>The following sample is the minimum setup required to accept syslog messages from applications running +on the local system.<br> </p> <textarea rows="2" cols="70">$ModLoad imuxsock # needs to be done just once $SystemLogSocketFlowControl on # enable flow control (use if needed) </textarea> <p>The following sample is a configuration where rsyslogd pulls logs from two jails, and assigns different hostnames to each of the jails: </p> -<textarea rows="6" cols="60">$ModLoad imuxsock # needs to be done just once +<textarea rows="6" cols="70">$ModLoad imuxsock # needs to be done just once $InputUnixListenSocketHostName jail1.example.net $AddUnixListenSocket /jail/1/dev/log $InputUnixListenSocketHostName jail2.example.net $AddUnixListenSocket /jail/2/dev/log </textarea> +<p>The following sample is a configuration where rsyslogd reads the openssh log +messages via a separate socket, but this socket is created on a temporary file +system. As rsyslogd starts up before the sshd, it needs to create the socket +directories, because it otherwise can not open the socket and thus not listen +to openssh messages. Note that it is vital not to place any other socket between +the $InputUnixListenSocketCreatePath and the $InputUnixListenSocketHostName.</p> +<textarea rows="6" cols="70">$ModLoad imuxsock # needs to be done just once + +$InputUnixListenSocketCreatePath on # turn on for *next* socket +$InputUnixListenSocketHostName /var/run/sshd/dev/log +</textarea> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the diff --git a/doc/manual.html b/doc/manual.html index 90af6940..57f3d4a6 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -19,7 +19,7 @@ rsyslog support</a> available directly from the source!</p> <p><b>Please visit the <a href="http://www.rsyslog.com/sponsors">rsyslog sponsor's page</a> to honor the project sponsors or become one yourself!</b> We are very grateful for any help towards the project goals.</p> -<p><b>This documentation is for version 5.1.6 (devel branch) of rsyslog.</b> +<p><b>This documentation is for version 5.3.5 (devel branch) of rsyslog.</b> Visit the <i><a href="http://www.rsyslog.com/doc-status.html">rsyslog status page</a></i></b> to obtain current version information and project status. </p><p><b>If you like rsyslog, you might @@ -43,9 +43,10 @@ if you do not read the doc, but doing so will definitely improve your experience <li> <a href="property_replacer.html">property replacer, an important core component</a></li> <li>a commented <a href="sample.conf.html">sample rsyslog.conf</a> </li> <li><a href="bugs.html">rsyslog bug list</a></li> -<li><a href="rsyslog_packages.html"> rsyslog packages</a></li> +<li><a href="messageparser.html">understanding rsyslog message parsers</a></li> <li><a href="generic_design.html">backgrounder on generic syslog application design</a></li> <li><a href="modules.html">description of rsyslog modules</a></li> +<li><a href="rsyslog_packages.html">rsyslog packages</a></li> </ul> <p><b>We have some in-depth papers on</b></p> <ul> diff --git a/doc/messageparser.html b/doc/messageparser.html new file mode 100644 index 00000000..370db59f --- /dev/null +++ b/doc/messageparser.html @@ -0,0 +1,222 @@ +<html> +<head> +<title>Message parsers in rsyslog</title> +</head> +<body> +<a href="manual.html">rsyslog documentation</a> + +<h1>Message parsers in rsyslog</h1> +<p><small><i>Written by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> +(2009-11-06)</i></small></p> +<h2>Intro</h2> +<p>Message parsers are a feature of rsyslog 5.3.4 and above. In this article, I describe what +message parsers are, what they can do and how they relate to the relevant standards. I will +also describe what you can not do with time. Finally, I give some advice on implementing your +own custom parser. + +<h2>What are message parsers?</h2> +<p>Well, the quick answer is that message parsers are the component of rsyslog that +parses the syslog message after it is being received. Prior to rsyslog 5.3.4, message parsers +where built in into the rsyslog core itself and could not be modified (other than by modifying +the rsyslog code). +<p>In 5.3.4, we changed that: message parsers are now loadable modules (just +like input and output modules). That means that new message parsers can be added without +modifying the rsyslog core, even without contributing something back to the +project. +<p>But that doesn't answer what a message parser really is. What does ist mean to "parse a +message" and, maybe more importantly, what is a message? To answer these questions correctly, +we need to dig down into the relevant standards. +<a href="http://tools.ietf.org/html/rfc5424">RFC5424</a> specifies a layered architecture +for the syslog protocol: +<p align="center"><img src="rfc5424layers.png" alt="RFC5424 syslog protocol layers"> +<p>For us important is the distinction between the syslog transport and the upper layers. +The transport layer specifies how a stream of messages is assembled at the sender side and +how this stream of messages is disassembled into the individual messages at the receiver +side. In networking terminology, this is called "framing". The core idea is that +each message is put into a so-called "frame", which then is transmitted over the communications +link. +<p>The framing used is depending on the protocol. For example, in UDP the "frame"-equivalent is +a packet that is being sent (this also means that no two messages can travel within a single +UDP packet). In "plain tcp syslog", the industry standard, LF is used as a frame delimiter +(which also means that no multi-line message can properly be transmitted, a "design" flaw +in plain tcp syslog). In <a href="http://tools.ietf.org/html/rfc5425">RFC5425</a> there is +a header in front of each frame that contains the size of the message. With this framing, +any message content can properly be transferred. +<p>And now comes the important part: <b>message parsers do NOT operate at the transport +layer</b>, they operate, as their name implies, on messages. So we can not use message +parsers to change the underlying framing. For example, if a sender splits (for whatever +reason) a single message into two and encapsulates these into two frames, there is no way +a message parser could undo that. +<p>A typical example may be a multi-line message: let's assume some originator has generated +a message for the format "A\nB" (where \n means LF). If that message is being transmitted +via plain tcp syslog, the frame delimiter is LF. So the sender will delimite the frame with +LF, but otherwise send the message unmodified onto the wire (because that is how things are +-unfortunately- done in plain tcp syslog...). So wire will see "A\nB\n". When this +arrives at the receiver, the transport layer will undo the framing. When it sees the LF +after A, it thinks it finds a valid frame delimiter (in fact, this is the correct view!). So +the receive will extract one complete message A and one complete message B, not knowing +that they once were both part of a large multi-line message. These two messages are then +passed to the upper layers, where the message parsers receive them and extract information. +However, the message parsers never know (or even have a chance to see) that A and B +belonged together. Even further, in rsyslog there is no guarnatee that A will be parsed +before B - concurrent operations may cause the reverse order (and do so very validly). +<p>The important lesson is: <b>message parsers can not be used to fix a broken framing</b>. +You need a full protocol implementation to do that, what is the domain of input and +output modules. +<p>I have now told you what you can not do with message parsers. But what they are good for? +Thankfully, broken framing is not the primary problem of the syslog world. A wealth of different +formats is. Unfortunately, many real-world implementations violate the relevant standards +in one way or another. That makes it often very hard to extract meaningful information from +a message or to process messages from different sources by the same rules. In my article +<a href="syslog_parsing.html">syslog parsing in rsyslog</a> I have elaborated on all +the real-world evil that you can usually see. So I won't repeat that here. But in short, the +real problem is not the framing, but how to make malformed messages well-looking. +<p><b>This is what message parsers permit you to do: take a (well-known) malformed message, parse +it according to its semantics and generate perfectly valid internal message representations +from it.</b> So as long as messages are consistenly in the same wrong format (and they usually +are!), a message parser can look at that format, parse it, and make the message processable just +like it were wellformed in the first place. Plus, one can abuse the interface to do some other +"intersting" tricks, but that would take us to far. +<p>While this functionality may not sound exciting, it actually solves a very big issue (that you +only really understand if you have managed a system with various different syslog sources). +Note that we were often able to process malformed messages in the past with the help of the +property replacer and regular expressions. While this is nice, it has a performance hit. A +message parser is a C code, compiled to native language, and thus typically much faster than +any regular expression based method (depending, of course, on the quality of the implementation...). + +<h2>How are message parsers used?</h2> +<p>In a simlified view, rsyslog +<ol> +<li>first receives messages (via the input module), +<li><i>then parses them (at the message level!)</i> and +<li>then processes them (operating on the internal message representation). +</ol> +Message parsers are utilized in the second step (written in italics). +Thus, they take the raw message (NOT frame!) received from the remote system and create +the internal structure out of it that the other parts of rsyslog need in order to perform +their processing. Parsing is vital, because an unparsed message can not be processed in the +third stage, the actual application-level processing (like forwarding or writing to files). +<h3>Parser Chains and how they Operate</h3> +Rsyslog chains parsers together to provide flexibility. +A <b>parser chain</b> +contains all parsers that can potentially be used to parse a message. +It is assumed that there is some +way a parser can detect if the message it is being presented is supported by it. If so, the parser +will tell the rsyslog engine and parse the message. The rsyslog engine now calls each parser +inside the chain (in sequence!) until the first parser is able to parse the message. After one +parser has been found, the message is considered parsed and no others parsers are called on that +message. +<p>Side-note: this method implies there are some "not-so-dirty" tricks available to modify +the message by a parser module that declares itself as "unable to parse" but still does +some message modification. This was not a primary design goal, but may be utilized, and the +interface probably extended, to support generic filter modules. These would need to go +to the root of the parser chain. As mentioned, the current system already supports this. +<p>The position inside the parser chain can be thought of as a priority: parser sitting +earlier in the chain take precedence over those sitting later in it. So more specific +parser should go ealier in the chain. A good example of how this works is the default parser +set provided by rsyslog: rsyslog.rfc5424 and rsyslog.rfc3164, each one parses according to the +rfc that has named it. RFC5424 was designed to be distinguishable from RFC3164 message by the +sequence "1 " immediately after the so-called PRI-part (don't worry about these words, it is +sufficient if you understand there is a well-defined sequence used to indentify RFC5424 +messages). In contrary, RFC3164 actually permits everything as a valid message. Thus the +RFC3164 parser will always parse a message, sometimes with quite unexpected outcome (there is +a lot of guesswork involved in that parser, which unfortunately is unavoidable due to +existing techology limits). So the default parser chain is to try the RFC5424 parser first +and after it the RFC3164 parser. If we have a 5424-formatted message, that parser will +identify and parse it and the rsyslog engine will stop processing. But if we receive a +legacy syslog message, the RFC5424 will detect that it can not parse it, return this status +to the engine which then calls the next parser inside the chain. That usually happens to be +the RFC3164 parser, which will always process the message. But there could also be any other +parser inside the chain, and then each one would be called unless one that is able to parse +can be found. +<p>If we reversed the parser order, RFC5424 messages would incorrectly parsed. Why? Because the +RFC3164 parser will always parse every message, so if it were asked first, it would parse +(and misinterpret) the 5424-formatted message, return it did so and the rsyslog engine would +never call the 5424 parser. So oder of sequence is very important. +<p>What happens if no parser in the chain could parse a message? Well, then we could not +obtain the in-memory representation that is needed to further process the message. In that +case, rsyslog has no other choice than to discard the message. If it does so, it will emit +a warning message, but only in the first 1,000 incidents. This limit is a safety measure +against message-loops, which otherwise could quickly result from a parser chain +misconfiguration. <b>If you do not tolerate loss of unparsable messages, you must ensure +that each message can be parsed.</b> You can easily achive this by always using the +"rsyslog-rfc3164" parser as the <i>last</i> parser inside parser chains. That may result +in invalid parsing, but you will have a chance to see the invalid message (in debug mode, +a warning message will be written to the debug log each time a message is dropped due to +inability to parse it). +<h3>Where are parser chains used?</h3> +<p>We now know what parser chains are and how they operate. The question is now how many +parser chains can be active and how it is decicded which parser chain is used on which message. +This is controlled via <a href="multi_ruleset.html">rsyslog's rulesets</a>. In short, multiple +rulesets can be defined and there always exist at least one ruleset (for specifcs, follow +the <a href="multi_ruleset.html">link</a>). A parser chain is bound to a specific ruleset. +This is done by virtue of defining parsers via the +<a href="rsconf1_rulesetparser.html">$RulesetParser</a> configuration directive (for specifics, +see there). If no such directive is specified, the default parser chain is used. As of this +writing, the default parser chain always consists of "rsyslog.rfc5424", "rsyslog.rfc3164", in +that order. As soon as a parser is configured, the default list is cleared and the new parser +is added to the end of the (initially empty) ruleset's parser chain. +<p>The important point to know is that parser chains are defined on a per-ruleset basis. +<h3>Can I use different parser chains for different devices?</h3> +<p>The correct answer is: generally yes, but it depends. First of all, remember that input +modules (and specific listeners) may be bound to specific rulesets. As parser chains "reside" +in rulesets, binding to a ruleset also binds to the parser chain that is bound to that ruleset. +As a number one prequisite, the input module must support binding to different rulesets. Not +all do, but their number is growing. For example, the important +<a href="imudp.html">imudp</a> and <a href="imtcp.html">imtcp</a> input modules support +that functionality. Those that do not (for example <a href="im3195">im3195</a>) can only +utilize the default ruleset and thus the parser chain defined in that ruleset. +<p>If you do not know if the input module in question supports ruleset binding, check +its documentation page. Those that support it have the requiered directives. +<p>Note that it is currently under evaluation if rsyslog will support binding parser chains +to specific inputs directly, without depending on the ruleset. There are some concerns that +this may not be necessary but adds considerable complexity to the configuration. So this may +or may not be possible in the future. In any case, if we decide to add it, input modules +need to support it, so this functionality would require some time to implement. +<p>The coockbook recipe for using different parsers for different devices is given +as an actual in-depth example in the <a href="rscon1_rulesetsparser.html">$RulesetParser</a> +configuration directive doc page. In short, it is acomplished by defining specific rulesets +for the required parser chains, definining different listener ports for each of the devices +with different format and binding these listeners to the correct ruleset (and thus parser +chains). Using that approach, a variety of different message formats can be supported +via a single rsyslog instance. + +<h2>Which message parsers are available</h2> +<p>As of this writing, there exist only two message parsers, one for RFC5424 format and one for +legacy syslog (loosely described in +<a href="http://tools.ietf.org/html/rfc3164">RFC3164</a>). These parsers are built-in and +must not be explicitely loaded. However, message parsers can be added with relative ease +by anyone knowing to code in C. Then, they can be loaded via $ModLoad just like any +other loadable module. It is expected that the rsyslog project will be contributed additional +message parsers over time, so that at some point there hopefully is a rich choice of them +(I intend to add a browsable repository as soon as new parsers pop up). +<h3>How to write a message parser?</h3> +<p>As a prequisite, you need to know the exact format that the device is sending. Then, you need +moderate C coding skills, and a little bit of rsyslog internals. I guess the rsyslog specific part +should not be that hard, as almost all information can be gained from the existing parsers. They +are rather simple in structure and can be found under the "./tools" directory. They are named +pmrfc3164.c and pmrfc5424.c. You need to follow the usual loadable module guidelines. +It is my expectation that writing a parser should typically not take longer than a single +day, with maybe a day more to get aquainted with rsyslog. Of course, I am not sure if the number +is actually right. +<p>If you can not program or have no time to do it, Adiscon can also write a message parser +for you as +part of the <a href="http://www.rsyslog/professional-services">rsyslog professional services +offering</a>. +<h2>Conclusion</h2> +<p>Malformed syslog messages are a pain and unfortunately often seen in practice. Message parsers +provide a fast and efficient solution for this problem. Different parsers can be defined for +different devices, and they all convert message information into rsyslog's well-defined +internal format. Message parsers were first introduced in rsyslog 5.3.4 and also offer +some interesting ideas that may be explored in the future - up to full message normalization +capabilities. It is strongly recommended that anyone with a heterogenous environment take +a look at message parser capabilities. + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> +</body> +</html> diff --git a/doc/multi_ruleset.html b/doc/multi_ruleset.html index 8d8c614f..da65b4ba 100644 --- a/doc/multi_ruleset.html +++ b/doc/multi_ruleset.html @@ -23,7 +23,7 @@ write it to a file or forward it to a remote logging server. <p>A traditional configuration file is made up of one or more of these rules. When a new message arrives, its processing starts with the first rule (in order of appearance in rsyslog.conf) and continues for each rule until either all rules have been processed or -a so-called "e;discard" action happens, in which case processing stops and the +a so-called "discard" action happens, in which case processing stops and the message is thrown away (what also happens after the last rule has been processed). <p>The <b>multi-ruleset</b> support now permits to specify more than one such rule sequence. @@ -66,11 +66,18 @@ to seperate the messages by any other method. <pre>$InputTCPServerBindRuleset <name> </pre> -directive. Note that "name"e; must be the name of a ruleset that is already defined +directive. Note that "name" must be the name of a ruleset that is already defined at the time the bind directive is given. There are many ways to make sure this happens, but I personally think that it is best to define all rule sets at the top of rsyslog.conf and define the inputs at the bottom. This kind of reverses the traditional recommended ordering, but seems to be a really useful and straightforward way of doing things. +<h2>Why are rulesets important for different parser configurations?</h2> +<p>Custom message parsers, used to handle differnet (and potentially otherwise-invalid) +message formats, can be bound to rulesets. So multiple rulesets can be a very useful +way to handle devices sending messages in different malformed formats in a consistent +way. Unfortunately, this is not uncommon in the syslog world. An in-depth explanation +with configuration sample can be found at the +<a href="rsconf1_rulesetparser.html">$RulesetParser</a> configuration directive. <h2>Can I use a different Ruleset as the default?</h2> <p>This is possible by using the @@ -249,11 +256,12 @@ $InputTCPServerBindRuleset remote10516 $InputTCPServerRun 10516 </pre> -<p>Note that the "mail.*" rule inside the "remote10516"e; ruleset does +<p>Note that the "mail.*" rule inside the "remote10516" ruleset does not affect processing inside any other rule set, including the default rule set. <h2>Performance</h2> +<h3>Fewer Filters</h3> <p>No rule processing can be faster than not processing a rule at all. As such, it is useful for a high performance system to identify disjunct actions and try to split these off to different rule sets. In the example section, we had a case where three different tcp listeners @@ -263,6 +271,25 @@ is no need to check the reception service - instead messages are automatically p right rule set and can be processed by very simple rules (maybe even with "*.*"-filters, the fastest ones available). +<h3>Partitioning of Input Data</h3> +<p>Starting with rsyslog 5.3.4, rulesets permit higher concurrency. They offer the ability +to run on their own "main" queue. What that means is that a own queue is associated +with a specific rule set. That means that inputs bound to that ruleset do no longer need +to compete with each other when they enqueue a data element into the queue. Instead, enqueue +operations can be completed in parallel. +<p>An example: let us assume we have three TCP listeners. Without rulesets, each of them +needs to insert messages into the main message queue. So if each of them wants to +submit a newly arrived message into the queue at the same time, only one can do so +while the others need to wait. With multiple rulesets, its own queue can be created for each +ruleset. If now each listener is bound to its own ruleset, concurrent message submission is +possible. On a machine with a sufficiently large number of corse, this can result in +dramatic performance improvement. +<p>It is highly advised that high-performance systems define a dedicated ruleset, with a +dedicated queue for each of the inputs. +<p>By default, rulesets do <b>not</b> have their own queue. It must be activated via the +<a href="rsconf1_rulesetcreatemainqueue.html">$RulesetCreateMainQueue</a> directive. + +<h3>Future Enhancements</h3> <p>In the long term, multiple rule sets will probably lay the foundation for even better optimizations. So it is not a bad idea to get aquainted with them. diff --git a/doc/omoracle.html b/doc/omoracle.html index cfcf277f..2bb6aa5d 100644 --- a/doc/omoracle.html +++ b/doc/omoracle.html @@ -13,10 +13,11 @@ <p><b>Available since: </b>: 4.3.0 <p><b>Status: </b>: contributed module, not maitained by rsyslog core authors <p><b>Description</b>:</p> -<p>This module provides native support for logging to Oracle databases. It offers -superior performance over the more generic <a href="omlibdbi.html">omlibdbi</a> module. -It also includes a number of enhancements, most importantly prepared statements and -batching, what provides a big performance improvements. +<p>This module provides native support for logging to Oracle +databases. It offers superior performance over the more +generic <a href="omlibdbi.html">omlibdbi</a> module. It also includes +a number of enhancements, most importantly prepared statements and +batching, what provides a big performance improvement. </p> <p>Note that this module is maintained by its original author. If you need assistance with it, it is suggested to post questions to the @@ -63,7 +64,7 @@ it is suggested to post questions to the $OmoracleStatement \ insert into foo(hostname,message)values(:host,:message) - Also note that identifiers to placeholders are arbitrarry. You + Also note that identifiers to placeholders are arbitrary. You need to define the properties on the template in the correct order you want them passed to the statement! </pre> diff --git a/doc/omruleset.html b/doc/omruleset.html new file mode 100644 index 00000000..41d6ccfc --- /dev/null +++ b/doc/omruleset.html @@ -0,0 +1,140 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>ruleset output module (omruleset)</title> +</head> +<body> +<a href="rsyslog_conf_modules.html">rsyslog module reference</a> + +<h1>ruleset output/including module (omruleset)</h1> +<p><b>Module Name: omruleset</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Available Since</b>: 5.3.4</p> +<p><b>Description</b>:</p> +<p>This is a very special "output" module. It permits to pass a message object +to another rule set. While this is a very simple action, it enables very +complex configurations, e.g. it supports high-speed "and" conditions, sending +data to the same file in a non-racy way, include-ruleset functionality as well as +some high-performance optimizations (in case the rule sets have the necessary +queue definitions). +<p>While it leads to a lot of power, this output module offers seamingly easy functionaltiy. +The complexity (and capablities) arise from how everthing can be combined. +<p>With this module, a message can be sent to processing to another ruleset. This is somewhat +similar to a "#include" in the C programming language. However, one needs to keep +on the mind that a ruleset can contain its own queue and that a queue can run in various modes. +<p>Note that if no queue is defined in the ruleset, the message is enqueued into the main message +queue. This most often is not optimal and means that message processing may be severely defered. +Also note that when the ruleset's target queue is full and no free space can be aquired +within the usual timeout, the message object may actually be lost. This is an extreme scenario, +but users building an audit-grade system need to know this restriction. For regular installations, +it should not really be relevant. +<p><b>At minimum, be sure you understand the +<a href="rsconf1_rulesetcreatemainqueue.html">$RulesetCreateMainQueue</a> +directive as well as the importance of statement order in rsyslog.conf before using omruleset!</b> +<p><b>Recommended Use:</b> +<ul> +<li>create rulesets specifically for omruleset +<li>create these rulesets with their own main queue +<li> decent queueing parameters (sizes, threads, etc) should be used +for the ruleset main queue. If in doubt, use the same parameters as for the +overall main queue. +<li>if you use multiple levels of ruleset nesting, double check for endless loops - the rsyslog +engine does not detect these +</ul> + +<p><b>Configuration Directives</b>:</p> +<ul> +<li><b>$ActionOmrulesetRulesetName</b> ruleset-to-submit-to<br> +This directive specifies the name of the ruleset that the message +provided to omruleset should be submitted to. This ruleset must already have +been defined. Note that the directive is automatically reset after each +:omruleset: action and there is no default. This is done to prevent accidential +loops in ruleset definition, what can happen very quickly. +The :omruleset: action will NOT be honored if no ruleset name has been defined. As usual, +the ruleset name must be specified in front of the action that it modifies. +</ul> +<p><b>Examples:</b></p> +<p>This example creates a ruleset for a write-to-file action. The idea here +is that the same file is written based on multiple filters, problems occur if the file is used +together with a buffer. That is because file buffers are action-specific, and so some partial +buffers would be written. With omruleset, we create a single action inside its own ruleset and +then pass all messages to it whenever we need to do so. Of course, such a simple situation could +also be solved by a more complex filter, but the method used here can also be utilized in +more complex scenarios (e.g. with multiple listeners). The example tries to keep it simple. +Note that we create a ruleset-specific main queue (for simplicity with the default main queue +parameters) in order to avoid re-queueing messages back into the main queue. +</p> +<textarea rows="30" cols="80">$ModLoad omruleset + +# define ruleset for commonly written file +$RuleSet commonAction +$RulesetCreateMainQueue on +*.* /path/to/file.log + +#switch back to default ruleset +$ruleset RSYSLOG_DefaultRuleset + +# begin first action +# note that we must first specify which ruleset to use for omruleset: +$ActionOmrulesetRulesetName CommonAction +mail.info :omruleset: +#end first action + +# begin second action +# note that we must first specify which ruleset to use for omruleset: +$ActionOmrulesetRulesetName CommonAction +:FROMHOST, isequal, "myhost.example.com" :omruleset: +#end second action + +# of course, we can have "regular" actions alongside :omrulset: actions +*.* /path/to/general-message-file.log +</textarea> +<p>The next example is used to creat a high-performance nested and filter condition. Here, +it is first checked if the message contains a string "error". If so, the message is forwarded +to another ruleset which then applies some filters. The advantage of this is that we can use +high-performance filters where we otherwise would need to use the (much slower) expression-based +filters. Also, this enables pipeline processing, in that second ruleset is executed in +parallel to the first one.</p> +<textarea rows="30" cols="80">$ModLoad omruleset + +# define "second" ruleset +$RuleSet nested +$RulesetCreateMainQueue on # again, we use our own queue +mail.* /path/to/mailerr.log +kernel.* /path/to/kernelerr.log +auth.* /path/to/autherr.log + +#switch back to default ruleset +$ruleset RSYSLOG_DefaultRuleset + +# begin first action - here we filter on "error" +# note that we must first specify which ruleset to use for omruleset: +$ActionOmrulesetRulesetName nested +:msg, contains, "error :omruleset: +#end first action + +# begin second action - as an example we can do anything else in +# this processing. Note that these actions are processed concurrently +# to the ruleset "nested" +:FROMHOST, isequal, "myhost.example.com" /path/to/host.log +#end second action + +# of course, we can have "regular" actions alongside :omrulset: actions +*.* /path/to/general-message-file.log +</textarea> +<p><b>Caveats/Known Bugs:</b> +<p>The current configuration file language is not really adequate for a complex construct +like omruleset. Unfortunately, more important work is currently preventing me from redoing the +config language. So use extreme care when nesting rulesets and be sure to test-run your +config before putting it into production, ensuring you have a suffciently large probe +of the traffic run over it. If problems arise, the +<a href="troubleshoot.html">rsyslog debug log</a> is your friend. +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/omstdout.html b/doc/omstdout.html new file mode 100644 index 00000000..0bd10cfb --- /dev/null +++ b/doc/omstdout.html @@ -0,0 +1,42 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>stdout output module (omstdout)</title> +</head> +<body> +<a href="rsyslog_conf_modules.html">rsyslog module reference</a> + +<h1>stdout output module (stdout)</h1> +<p><b>Module Name: omstdout</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Available Since</b>: 4.1.6</p> +<p><b>Description</b>:</p> +<p>This module writes any messages that are passed to it to stdout. +It was developed for the rsyslog test suite. However, there may +(limited) other uses exists. Please not that we do not put too much +effort into the quality of this module as we do not expect it to +be used in real deployments. If you do, please drop us a note so +that we can enhance its priority! +<p><b>Configuration Directives</b>:</p> +<ul> +<li><b>$ActionOMStdoutArrayInterface</b> [on|<b>off</b><br> +This setting instructs omstdout to use the alternate +array based method of parameter passing. If used, the values +will be output with commas between the values but no other padding bytes. +This is a test aid for the alternate calling interface. +<li><b>$ActionOMStdoutEnsureLFEnding</b> [<b>on</b>|off<br> +Makes sure that each message is written with a terminating LF. This is needed for +the automatted tests. If the message contains a trailing LF, none is added. +</ul> +<b>Caveats/Known Bugs:</b> +<p>Currently none known. +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/omudpspoof.html b/doc/omudpspoof.html index e5f963c7..64f83752 100644 --- a/doc/omudpspoof.html +++ b/doc/omudpspoof.html @@ -16,23 +16,23 @@ spoof the sender address. Also, it enables to circle through a number of source ports. <p><b>Configuration Directives</b>:</p> <ul> -<li><b>$ActionUDPSpoofSourceNameTemplate</b> <templatename><br> +<li><b>$ActionOMUDPSpoofSourceNameTemplate</b> <templatename><br> This MUST be specified. It is the name of the template that contains a numerical IP address that is to be used as the source system IP address. While it may often be a constant value, it can be generated as usual via the property replacer, as long as it is a valid IPv4 address. -<li><b>$ActionUDPSpoofTargetHost</b> <hostname><br> +<li><b>$ActionOMUDPSpoofTargetHost</b> <hostname><br> Host that the messages shall be sent to. <li><b>$ActionUDPSpoofTargetPort</b> <port><br> Remote port that the messages shall be sent to. -<li><b>$ActionUDPSpoofDefaultTemplate</b> <templatename><br> +<li><b>$ActionOMUDPSpoofDefaultTemplate</b> <templatename><br> This setting instructs omudpspoof to use a template different from the default template for all of its actions that do not have a template specified explicitely. -<li><b>$ActionUDPSpoofSourcePortStart</b> <number><br> +<li><b>$ActionOMUDPSpoofSourcePortStart</b> <number><br> Specifies the start value for circeling the source ports. Must be less than or equal to the end value. Default is 32000. -<li><b>$ActionUDPSpoofSourcePortEnd</b> <number><br> +<li><b>$ActionOMUDPSpoofSourcePortEnd</b> <number><br> Specifies the ending value for circeling the source ports. Must be less than or equal to the start value. Default is 42000. </ul> diff --git a/doc/professional_support.html b/doc/professional_support.html index 7724ede8..de3ac800 100644 --- a/doc/professional_support.html +++ b/doc/professional_support.html @@ -1,112 +1,27 @@ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head> -<meta http-equiv="Content-Language" content="en"><title>Professional Support for Rsyslog</title> - +<html> +<head> +<meta http-equiv="Content-Language" content="en"> +<title>Professional Support for Rsyslog</title> </head> + <body> -<h1>Professional Services for Rsyslog</h1> +<h1><a href="http://www.rsyslog.com/Article412.phtml">Professional Services for Rsyslog</a></h1> <p>Professional services are being offered by <a href="http://www.adiscon.com">Adiscon</a>, the company -that sponsors rsyslog development. For details, please contact <a href="mailto:info%40adiscon.com">Adiscon Sales</a>. </p> +that sponsors rsyslog development. For details, please contact <a href="mailto:info%40adiscon.com">Adiscon Sales</a>.</p> -<h3>EMail Support Service</h3> -Price: 99.00 EURO <br> -Duration: 180 days -<br> -Support level: 8x5 -<p>Purchase rsyslog support directly from the source. This -contract provides priority email support. It is a great option if you -need to provide proof of software support in your organization. This -contract provides</p> -<ul> -<li>unlimited email support tickets during validity -</li> -<li><span style="font-weight: bold;">fixes for</span> -current and <span style="font-weight: bold;">past rsyslog -releases</span> -</li> -<li>advise on how to implement rsyslog in the best possible -way. -</li> -</ul> -<p>Under this contract, fixes for old rsyslog releases will be -provided / created, provided that it is possible to do that with the -code base in question. Phone support is not included.</p> -<h3>Custom-Written Config File</h3> -Price: 29.00 EURO -<br> -Duration: N/A -<br> -Support level: 8x5 -<p>Creating rsyslog config files is easy - but if you would like -to have that extra feature and have no time to do it, this service is -for you. Important: BEFORE you purchase this service, contact us and -inquire (via <a href="mailto:info%40adiscon.com">info@adiscon.com</a>) -whether or not your desired result can be achieved via rsyslog. Once -this is clear, order the service and we will ship a custom-made -configuration file within 5 working days (at latest, most often much -faster). For security reasons, we will not put passwords into the -configuration file, but will place easy to read comments in the places -where you need to put them in. The agreement is governed under German -law. You may also purchase this service if you would like to have your -own configuration file reviewed, e.g. for auditing purposes.</p> -<h3>Local Installation Support</h3> -<p>If you intend to install rsyslog on your system but would like -to do so with minimal effort and according to your specification, you -can ask us to perform the installation for you. You get a perfect -installation, exactly like you needed, but without a need to -touch the system. This is a perfect choice for the busy administrator! -<p>In order to perform this work, we just need ssh access to your -system and the proper permissions. -<p>We charge a low one-time fee for this service. For details, please -contact <a href="mailto:info@adiscon.com">info@adiscon.com</a>. -<h3>Local Installation Maintenance</h3> -<p>If you used our services to set up the system, why not keep it -running perfectly with maintenance support? Under this contract, we -assure you run a recent build that does not interfere with your -environment and we even carry out change requests you may have. So this -is a hassle-free, everything cared about solution. -<p>Again, all we need to have is ssh access and the proper permissions -to your machine. Of course, work will only be carried out when you -expect us to do so. You are always in control of what happens. This -is a perfect outsourcing solution for those who would like to run -a great logging system but can not afford the time to keep it -in perfect shape! -<p>We charge a low monthly fee for this service. For details, please -contact <a href="mailto:info@adiscon.com">info@adiscon.com</a>. -<h3>Custom Development</h3> -<p>Do you need an exotic feature that otherwise would not be implemented? -Do you need something really quick, quicker than it is available via -the regular development schedule? Then, you may consider funding -development for a specific functionality. We are always looking for -interesting projects. If you hire us to to do the job, you can be sure -to get the best possible and probably quickest solution, because we are -obviously at the heart of the source code. No need to get aquainted to -anything, no risk of misunderstanding program concepts. Benefit from -our vast syslog experience.</p> -<p>Please note that custom development is not limited to rsyslog. We offer -a number of logging solutions and can also work as part of your time -for specific requirements. The opportunities are endless, just ask. We -will work with you on your requirements and provide a quote on the -estimated cost. Just write to <a href="mailto:sales@adiscon.com">sales@adiscon.com</a> for details.</p><h3>Consulting Services</h3> -<p>Do you have demanding logging requirements? Why not talk to a -real logging professional? Instead of trying to find the solution -like a needle in the haystack, talk to the team that brought rsyslog, -phpLogCon, the Windows MonitorWare products and other logging -solutions. We sweat logging for over 15 years now and can help quickly. -Depending on your needs, consulting can be carried out via email, the -phone or on your premises (for larger or local projects). Everything is -possible, it just depends on your needs. Consulting services are -available in English and German. Just mail <a href="mailto:sales@adiscon.com">sales@adiscon.com</a> what you are interested in and we will work with you on a proposal that fits your needs. -</p><p></p><p>All agreements are -governed under German law. -</p> +<p>In an effort to keep the documentation set clean and enable Adiscon to enhance the +professional offerings, the actual +<a href="http://www.rsyslog.com/Article412.phtml">rsyslog service offering</a> +has been moved to a +<a href="http://www.rsyslog.com/Article412.phtml">separate page</a>. +Please follow that link for details. <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> -<p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> +<p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer -Gerhards</a> and +Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> -</body></html> +</body> +</html> diff --git a/doc/rainerscript.html b/doc/rainerscript.html index ef0e41cb..63a79040 100644 --- a/doc/rainerscript.html +++ b/doc/rainerscript.html @@ -51,13 +51,25 @@ of a and b should be tested as "a <> b". The "not" operator should be reserved to cases where it actually is needed to form a complex boolean expression. In those cases, parenthesis are highly recommended. +<h2>Functions</h2> +<p>RainerScript supports a currently quite limited set of functions: +<ul> +<li>getenv(str) - like the OS call, returns the value of the environment +variable, if it exists. Returns an empty string if it does not exist. +<li>strlen(str) - returns the length of the provided string +<li>tolower(str) - converts the provided string into lowercase +</ul> +<p>The following example can be used to build a dynamic filter based on some environment +variable: +<pre> +if $msg contains getenv('TRIGGERVAR') then /path/to/errfile +</pre> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer -Gerhards</a> and +Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> -</body></html>
\ No newline at end of file +</body></html> diff --git a/doc/rfc5424layers.png b/doc/rfc5424layers.png Binary files differnew file mode 100644 index 00000000..70192cc0 --- /dev/null +++ b/doc/rfc5424layers.png diff --git a/doc/rsconf1_abortonuncleanconfig.html b/doc/rsconf1_abortonuncleanconfig.html new file mode 100644 index 00000000..77526c07 --- /dev/null +++ b/doc/rsconf1_abortonuncleanconfig.html @@ -0,0 +1,37 @@ +<html> +<head> +<title>rsyslog.conf file</title> +</head> +<body> +<a href="rsyslog_conf_global.html">rsyslog.conf configuration directive</a> + +<h2>$AboortOnUncleanConfig</h2> +<p><b>Type:</b> global configuration directive</p> +<p><b>Parameter Values:</b> boolean (on/off, yes/no)</p> +<p><b>Available since:</b> 5.3.1+</p> +<p><b>Default:</b> off</p> +<p><b>Description:</b></p> +<p>This directive permits to prevent rsyslog from running when the configuration file +is not clean. "Not Clean" means there are errors or some other annoyances that rsyslgod +reports on startup. This is a user-requested feature to have a strict startup mode. Note +that with the current code base it is not always possible to differentiate between an +real error and a warning-like condition. As such, the startup will also prevented if +warnings are present. I consider this a good thing in being "strict", but I admit +there also currently is no other way of doing it. +<p><b>Caveats:</b></p> +Note that the consequences of a failed rsyslogd startup can be much more serious than a +startup with only partial configuration. For example, log data may be lost or systems that +depend on the log server in question will not be able to send logs, what in the ultimate +result could result in a system hang on those systems. Also, the local system may hang when +the local log socket has become full and is not read. There exist many such scenarios. +As such, it is strongly recommended not to turn on this directive. + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 2 or higher.</font></p> +</body> +</html> diff --git a/doc/rsconf1_omfileforcechown.html b/doc/rsconf1_omfileforcechown.html new file mode 100644 index 00000000..7415a6f6 --- /dev/null +++ b/doc/rsconf1_omfileforcechown.html @@ -0,0 +1,64 @@ +<html> +<head> +<title>rsyslog.conf file</title> +</head> +<body> +<a href="rsyslog_conf_global.html">back</a> + +<h2>$omfileForceChown</h2> +<p><b>Type:</b> global configuration directive</p> +<p><b>Parameter Values:</b> boolean (on/off, yes/no)</p> +<p><b>Available since:</b> 4.7.0+, 5.3.0+</p> +<p><b>Default:</b> off</p> +<p><b>Description:</b></p> +<p>Forces rsyslogd to change the ownership for output files that already exist. Please note +that this tries to fix a potential problem that exists outside the scope of rsyslog. Actually, +it tries to fix invalid ownership/permission settings set by the original file creator. +<p>Rsyslog changes the ownership during initial execution with root privileges. When a privelege +drop is configured, privileges are dropped after the file owner ship is changed. Not that this currently +is a limitation in rsyslog's privilege drop code, which is on the TODO list to be removed. See Caveats +section below for the important implications. +<p><b>Caveats:</b></p> +<p>This directive tries to fix a problem that actually is outside the scope of rsyslog. As such, +there are a couple of restrictions and situations in which it will not work. <b>Users are strongly +encouraged to fix their system instead of turning this directive on</b> - it should only be used +as a last resort. +<p>At least in the following scenario, this directive will fail expectedly: +<p>It does not address +the situation that someone changes the ownership *after* rsyslogd has started. +Let's, for example, consider a log rotation script. +<ul> +<li>rsyslog is started +<li>ownership is changed +<li>privileges dropped +<li>log rotation (lr) script starts +<li>lr removes files +<li>lr creates new files with root:adm (or whatever else) +<li>lr HUPs rsyslogd +<li>rsyslogd closes files +<li>rsyslogd tries to open files +<li>rsyslogd tries to change ownership --> fail as we are non-root now +<li>file open fails +</ul> + +Please note that once the privilege drop code is refactored, this directive will +no longer work, because then privileges will be dropped before any action is performed, +and thus we will no longer be able to chown files that do not belong to the +user rsyslogd is configured to run under. + +<p>So <b>expect the directive to go away</b>. It will not +be removed in version 4, but may disappear at any time for any version greater than 4. + +<p><b>Sample:</b></p> +<p><code><b>$FileOwner loguser</b> +<br><b>$omfileForceChown on</b></code></p> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2007 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 2 or higher.</font></p> +</body> +</html> diff --git a/doc/rsconf1_rulesetcreatemainqueue.html b/doc/rsconf1_rulesetcreatemainqueue.html new file mode 100644 index 00000000..5c1e0dec --- /dev/null +++ b/doc/rsconf1_rulesetcreatemainqueue.html @@ -0,0 +1,83 @@ +<html> +<head> +<title>RulesetCreateMainQueue - rsyslog.conf file</title> +</head> +<body> +<a href="rsyslog_conf_global.html">rsyslog.conf configuration directive</a> + +<h2>$RulesetCreateMainQueue</h2> +<p><b>Type:</b> ruleset-specific configuration directive</p> +<p><b>Parameter Values:</b> boolean (on/off, yes/no)</p> +<p><b>Available since:</b> 5.3.5+</p> +<p><b>Default:</b> off</p> +<p><b>Description:</b></p> +<p> +Rulesets may use their own "main" message queue for message submission. Specifying +this directive, <b>inside a ruleset definition</b>, turns this on. This is both a performance +enhancement and also permits different rulesets (and thus different inputs within the same +rsyslogd instance) to use different types of main message queues. +<p>The ruleset queue is created with the parameters that are specified for the main message +queue at the time the directive is given. If different queue configurations are desired, +different main message queue directives must be used in front of the $RulesetCreateMainQueue +directive. Note that this directive may only be given once per ruleset. If multiple statements +are specified, only the first is used and for the others error messages are emitted. +<p>Note that the final set of ruleset configuration directives specifies the parameters for +the default main message queue. +<p>To learn more about this feature, please be sure to read about +<a href="multi_ruleset.html">multi-ruleset support in rsyslog</a>. +<p><b>Caveats:</b></p> +The configuration statement "$RulesetCreateMainQueue off" has no effect at all. +The capability to specify this is an artifact of the current (ugly!) configuration +language. + +<p><b>Example:</b></p> +<p>This example sets up a tcp server with three listeners. Each of these +three listener is bound to a specific ruleset. As a performance optimization, +the rulesets all receive their own private queue. The result is that received messages +can be independently processed. With only a single main message queue, we would have +some lock contention between the messages. This does not happen here. Note that in this +example, we use different processing. Of course, all messages could also have been +processed in the same way ($IncludeConfig may be useful in that case!). +</p> +<textarea rows="30" cols="60">$ModLoad imtcp +# at first, this is a copy of the unmodified rsyslog.conf +#define rulesets first +$RuleSet remote10514 +$RulesetCreateMainQueue on # create ruleset-specific queue +*.* /var/log/remote10514 + +$RuleSet remote10515 +$RulesetCreateMainQueue on # create ruleset-specific queue +*.* /var/log/remote10515 + +$RuleSet remote10516 +$RulesetCreateMainQueue on # create ruleset-specific queue +mail.* /var/log/mail10516 +& ~ +# note that the discard-action will prevent this messag from +# being written to the remote10516 file - as usual... +*.* /var/log/remote10516 + +# and now define listners bound to the relevant ruleset +$InputTCPServerBindRuleset remote10514 +$InputTCPServerRun 10514 + +$InputTCPServerBindRuleset remote10515 +$InputTCPServerRun 10515 + +$InputTCPServerBindRuleset remote10516 +$InputTCPServerRun 10516 +</textarea> +<p>Note the positions of the directives. With the current config language, +position is very important. This is ugly, but unfortunately the way it currently +works. +</p> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 2 or higher.</font></p> +</body> +</html> diff --git a/doc/rsconf1_rulesetparser.html b/doc/rsconf1_rulesetparser.html new file mode 100644 index 00000000..84350431 --- /dev/null +++ b/doc/rsconf1_rulesetparser.html @@ -0,0 +1,119 @@ +<html> +<head> +<title>RulesetParser - rsyslog.conf file</title> +</head> +<body> +<a href="rsyslog_conf_global.html">rsyslog.conf configuration directive</a> + +<h2>$RulesetParser</h2> +<p><b>Type:</b> ruleset-specific configuration directive</p> +<p><b>Parameter Values:</b> string</p> +<p><b>Available since:</b> 5.3.4+</p> +<p><b>Default:</b> rsyslog.rfc5424;rsyslog.rfc5425</p> +<p><b>Description:</b></p> +<p> +This directive permits to specify which +<a href="messageparser.html">message parsers</a> should be used for the ruleset +in question. It no ruleset is explicitely specified, the default ruleset is used. Message +parsers are contained in (loadable) parser modules with the most common cases +(RFC3164 and RFC5424) being build-in into rsyslogd. +<p>When this directive is specified the first time for a ruleset, it will not only add the +parser to the ruleset's parser chain, it will also wipe out the default parser chain. +So if you need to have +them in addition to the custom parser, you need to specify those as well. +<p>Order of directives is important. Parsers are tried one after another, in the order +they are specified inside the config. As soon as a parser is able to parse the message, +it will do so and no other parsers will be executed. If no matching parser can be found, +the message will be discarded and a warning message be issued (but only for the first +1,000 instances of this problem, to prevent message generation loops). +<p>Note that the rfc3164 parser will <b>always</b> be able to parse a message - it may +just not be the format that you like. This has two important implications: 1) always place +that parser at the END of the parser list, or the other parsers after it will never +be tried and 2) if you would like to make sure no message is lost, placing the rfc3164 +parser at the end of the parser list ensures that. +<p>Multiple parser modules are very useful if you have various devices that emit +messages that are malformed in various ways. The route to take then is +<ul> +<li>make sure you find a custom parser for that device; if there is no one, you +may consider writing one yourself (it is not that hard) or getting one written +as part of +<a href="http://www.rsyslog.com/professional-servcies">Adiscon's professional services +for rsyslog</a>. +<li>load your custom parsers via $ModLoad +<li>create a ruleset for each malformed format; assign the custom parser to it +<li>create a specific listening port for all devices that emit the same +malformed format +<li>bind the listener to the ruleset with the required parser +</ul> +<p>Note that it may be cumbersome to add all rules to all rulesets. To avoid this, +you can either use $Include or <a href="omruleset.html">omruleset</a> +(what probably provides the best solution). +<p>More information about rulesets in general can be found in +<a href="multi_ruleset.html">multi-ruleset support in rsyslog</a>. +<p><b>Caveats:</b></p> +<p>currently none known</p> + +<p><b>Example:</b></p> +<p>This example assumes there are two devices emiting malformed messages via UDP. +We have two custom parsers for them, named "device1.parser" and +"device2.parser". In addition to that, we have a number of other +devices sending wellformed messages, also via UDP. +<p>The solution is to listen for data from the two devices on two special +ports (10514 and 10515 in this example), create a ruleset for each and +assign the custom parsers to them. The rest of the messages are received via +port 514 using the regular parsers. Processing shall be equal for all messages. +So we simply forward the malformed messages to the regular queue once they are parsed (keep +in mind that a message is never again parsed once any parser properly processed it). +</p> +<textarea rows="40" cols="80">$ModLoad imudp +$ModLoad pmdevice1 # load parser "device1.parser" for device 1 +$ModLoad pmdevice2 # load parser "device2.parser" for device 2 + +# define ruleset for the first device sending malformed data +$Ruleset maldev1 +$RulesetCreateMainQueue on # create ruleset-specific queue +$RulesetParser "device1.parser" # note: this deactivates the default parsers +# forward all messages to default ruleset: +$ActionOmrulesetRulesetName RSYSLOG_DefaultRuleset +*.* :omruleset: + +# define ruleset for the second device sending malformed data +$Ruleset maldev2 +$RulesetCreateMainQueue on # create ruleset-specific queue +$RulesetParser "device2.parser" # note: this deactivates the default parsers +# forward all messages to default ruleset: +$ActionOmrulesetRulesetName RSYSLOG_DefaultRuleset +*.* :omruleset: + +# switch back to default ruleset +$Ruleset RSYSLOG_DefaultRuleset +*.* /path/to/file +auth.info @authlogger.example.net +# whatever else you usually do... + + +# now define the inputs and bind them to the rulesets +# first the default listener (utilizing the default ruleset) +$UDPServerRun 514 + +# now the one with the parser for device type 1: +$InputUDPServerBindRuleset maldev1 +$UDPServerRun 10514 + +# and finally the one for device type 2: +$InputUDPServerBindRuleset maldev2 +$UDPServerRun 10515 +</textarea> +<p>Note the positions of the directives. With the current config language, +<b>sequence of statements is very important</b>. This is ugly, but unfortunately +the way it currently works. +</p> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 2 or higher.</font></p> +</body> +</html> diff --git a/doc/rsyslog_conf_global.html b/doc/rsyslog_conf_global.html index 45eeabe6..beb90e02 100644 --- a/doc/rsyslog_conf_global.html +++ b/doc/rsyslog_conf_global.html @@ -17,6 +17,8 @@ appear as implementation progresses. many parameter settings modify queue parameters. If in doubt, use the default, it is usually well-chosen and applicable in most cases.</p> <ul> +<li><a href="rsconf1_abortonuncleanconfig.html">$AbortOnUncleanConfig</a> - abort startup if there is +any issue with the config file</li> <li><a href="rsconf1_actionexeconlywhenpreviousissuspended.html">$ActionExecOnlyWhenPreviousIsSuspended</a></li> <li>$ActionName <a_single_word> - used primarily for documentation, e.g. when generating a configuration graph. Available sice 4.3.1. @@ -158,7 +160,12 @@ Usually that should not be a big issue, as the restart-type HUP can easily be re something along the lines of "/etc/init.d/rsyslog restart". </li> <li><a href="rsconf1_includeconfig.html">$IncludeConfig</a></li><li>MainMsgQueueCheckpointInterval <number></li> -<li>$MainMsgQueueDequeueBatchSize <number> [default 32]</li> +<li><b>$LogRSyslogStatusMessages</b> [<b>on</b>/off] - If set to on (the default), +rsyslog emits message on startup and shutdown as well as when it is HUPed. +This information might be needed by some log analyzers. If set to off, no such +status messages are logged, what may be useful for other scenarios. +[available since 4.7.0 and 5.3.0] +<li><b>$MainMsgQueueDequeueBatchSize</b> <number> [default 32]</li> <li>$MainMsgQueueDequeueSlowdown <number> [number is timeout in <i> micro</i>seconds (1000000us is 1sec!), default 0 (no delay). Simple rate-limiting!]</li> @@ -231,6 +238,7 @@ error recovery thus can handle write errors without data loss. Note that this op severely reduces the effect of zip compression and should be switched to off for that use case. Note that the default -off- is primarily an aid to preserve the traditional syslogd behaviour.</li> +<li><a href="rsconf1_omfileforcechown.html">$omfileForceChown</a> - force ownership change for all files</li> <li><b>$RepeatedMsgContainsOriginalMsg</b> [on/<b>off</b>] - "last message repeated n times" messages, if generated, have a different format that contains the message that is being repeated. Note that only the first "n" characters are included, with n to be at least 80 characters, most @@ -241,10 +249,14 @@ large enough for the whole message. (Introduced with 4.1.5). Once set, it affect <li><a href="rsconf1_resetconfigvariables.html">$ResetConfigVariables</a></li> <li><b>$Ruleset</b> <i>name</i> - starts a new ruleset or switches back to one already defined. All following actions belong to that new rule set. -the <i>name</i> does not yet exist, it is created. To swith back to rsyslog's +the <i>name</i> does not yet exist, it is created. To switch back to rsyslog's default ruleset, specify "RSYSLOG_DefaultRuleset") as the name. All following actions belong to that new rule set. It is advised to also read our paper on <a href="multi_ruleset.html">using multiple rule sets in rsyslog</a>.</li> +<li><b><a href="rsconf1_rulesetcreatemainqueue.html">$RulesetCreateMainQueue</a></b> on - creates +a ruleset-specific main queue. +<li><b><a href="rsconf1_rulesetparser.html">$RulesetParser</a></b> - enables to set +a specific (list of) message parsers to be used with the ruleset. <li><b>$OptimizeForUniprocessor</b> [on/<b>off</b>] - turns on optimizatons which lead to better performance on uniprocessors. If you run on multicore-machiens, turning this off lessens CPU load. The default may change as uniprocessor systems become less common. [available since 4.1.0]</li> @@ -252,19 +264,6 @@ default may change as uniprocessor systems become less common. [available since to sysklogd), the domain part from a name that is within the same domain as the receiving system is stripped. If set to on, full names are always used.</li> <li>$WorkDirectory <name> (directory for spool and other work files)</li> -<li>$UDPServerAddress <IP> (imudp) -- local IP -address (or name) the UDP listens should bind to</li> -<li>$UDPServerRun <port> (imudp) -- former --r<port> option, default 514, start UDP server on this -port, "*" means all addresses</li> -<li>$UDPServerTimeRequery <nbr-of-times> (imudp) -- this is a performance -optimization. Getting the system time is very costly. With this setting, imudp can -be instructed to obtain the precise time only once every n-times. This logic is -only activated if messages come in at a very fast rate, so doing less frequent -time calls should usually be acceptable. The default value is two, because we have -seen that even without optimization the kernel often returns twice the identical time. -You can set this value as high as you like, but do so at your own risk. The higher -the value, the less precise the timestamp. <li><a href="droppriv.html">$PrivDropToGroup</a></li> <li><a href="droppriv.html">$PrivDropToGroupID</a></li> <li><a href="droppriv.html">$PrivDropToUser</a></li> diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html index f9bdad4a..a4c8c423 100644 --- a/doc/rsyslog_conf_modules.html +++ b/doc/rsyslog_conf_modules.html @@ -11,6 +11,7 @@ what they do (list is currently not complete)</p> <li><a href="omsnmp.html">omsnmp</a> - SNMP trap output module</li> <li><a href="omstdout.html">omtdout</a> - stdout output module (mainly a test tool)</li> <li><a href="omrelp.html">omrelp</a> - RELP output module</li> +<li><a href="omruleset.html">omruleset</a> - forward message to another ruleset</li> <li>omgssapi - output module for GSS-enabled syslog</li> <li><a href="ommysql.html">ommysql</a> - output module for MySQL</li> <li>ompgsql - output module for PostgreSQL</li> @@ -25,7 +26,7 @@ permits rsyslog to alert folks by mail if something important happens</li> - input module for text files</li> <li><a href="imrelp.html">imrelp</a> - RELP input module</li> -<li>imudp - udp syslog message input</li> +<li><a href="imudp.html">imudp</a> - udp syslog message input</li> <li><a href="imtcp.html">imtcp</a> - input plugin for plain tcp syslog</li> <li><a href="imgssapi.html">imgssapi</a> - diff --git a/doc/rsyslog_secure_tls.html b/doc/rsyslog_secure_tls.html index be2811f4..b15e5a4e 100644 --- a/doc/rsyslog_secure_tls.html +++ b/doc/rsyslog_secure_tls.html @@ -51,7 +51,7 @@ google_ad_height = 125; src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> </script> </span> -I private keys have become known to third parties, the system does not provide +If private keys have become known to third parties, the system does not provide any security at all. Also, our solution bases on X.509 certificates and a (very limited) chain of trust. We have one instance (the CA) that issues all machine certificates. The machine certificate indentifies a particular machine. hile in diff --git a/doc/src/classes.dia b/doc/src/classes.dia Binary files differindex 70e91566..8cfcbd0c 100644 --- a/doc/src/classes.dia +++ b/doc/src/classes.dia diff --git a/doc/src/rfc5424layers.dia b/doc/src/rfc5424layers.dia Binary files differnew file mode 100644 index 00000000..300b7796 --- /dev/null +++ b/doc/src/rfc5424layers.dia diff --git a/doc/status.html b/doc/status.html index 02cc0d70..ff056489 100644 --- a/doc/status.html +++ b/doc/status.html @@ -2,12 +2,12 @@ <html><head><title>rsyslog status page</title></head> <body> <h2>rsyslog status page</h2> -<p>This page reflects the status as of 2009-08-21.</p> +<p>This page reflects the status as of 2009-11-05.</p> <h2>Current Releases</h2> -<p><b>v5 development:</b> 5.1.4 [2009-08-20] - -<a href="http://www.rsyslog.com/Article392.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-170.phtml">download</a> +<p><b>v5 development:</b> 5.3.4 [2009-11-04] - +<a href="http://www.rsyslog.com/Article423.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-185.phtml">download</a> <br> <!-- not at the moment! <b>v4 development:</b> 4.5.1 [2009-07-15] - @@ -15,13 +15,24 @@ <a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-167.phtml">download</a></p> --> -<br><b>v4-beta:</b> 4.5.2 [2009-08-21] - -<a href="http://www.rsyslog.com/Article395.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-172.phtml">download</a></p> +<!-- not at the moment! +<br><b>v5-beta:</b> 5.1.6 [2009-10-15] - +<a href="http://www.rsyslog.com/Article413.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-180.phtml">download</a> +--> + +<br><b>v4-beta:</b> 4.5.6 [2009-11-05] - +<a href="http://www.rsyslog.com/Article425.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-186.phtml">download</a></p> + +<p><b>v5 stable:</b> 5.2.0 [2009-11-02] (recommended to use +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-183.phtml">5.3.3</a> instead) - +<a href="http://www.rsyslog.com/Article421.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-184.phtml">download</a> -<p><b>v4 stable:</b> 4.4.0 [2009-08-21] - -<a href="http://www.rsyslog.com/Article394.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-171.phtml">download</a> +<br><b>v4 stable:</b> 4.4.2 [2009-10-09] - +<a href="http://www.rsyslog.com/Article409.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-179.phtml">download</a> <br><b>v3 stable:</b> 3.22.1 [2009-07-02] - <a href="http://www.rsyslog.com/Article381.phtml">change log</a> - diff --git a/doc/syslog_parsing.html b/doc/syslog_parsing.html index 57da6657..1ccec6f1 100644 --- a/doc/syslog_parsing.html +++ b/doc/syslog_parsing.html @@ -176,6 +176,19 @@ $template, MalfromedMsgFormater,"%timegenerated% %fromhost% %rawmsg:::drop-last- <p>This will make your log much nicer, but not look perfect. Experiment a bit with the available properties and replacer extraction options to fine-tune it to your needs. +<h2>The Ultimate Solution...</h2> +<p>Is available with rsyslog 5.3.4 and above. Here, we can define so-called custom +parsers. These are plugin modules, written in C and adapted to a specific message format +need. The big plus of custom parsers is that they offer excellent performance and unlimited +possibilities - far better than any work-around could do. Custom parsers can be +<a href="rsconf1_rulesetparser.html">bound to specific rule sets</a> +(and thus listening) ports with relative ease. The only con is that they must be written. +However, if you are lucky, a parser for your device may already exist. If not, you can +opt to write it yourself, what is not too hard if you know some C. Alternatively, +Adiscon can program one for you as part of the +<a href="http://www.rsyslog.com/professional-services">rsyslog professional services offering</a>. +In any case, you should seriously consider custom parsers as an alternative if you can not +reconfigure your device to send decent message format. <h2>Wrap-Up</h2> <p>Syslog message format is not sufficiently standardized. There exists a weak "standard" format, which is used by a good number of implementations. However, there @@ -183,14 +196,15 @@ exist many others, including mainstream vendor implementations, which have a (sometimes horribly) different format. Rsyslog tries to deal with anomalies but can not guess right in all instances. If possible, the sender should be configured to submit well-formed messages. If that is not possible, you can work around these -issues with rsyslog's property replacer and template system. +issues with rsyslog's property replacer and template system. Or you can use a suitable +message parser or write one for your needs. <p>I hope this is a useful guide. You may also have a look at the <a href="troubleshoot.html">rsyslog troubleshooting guide</a> for further help and places where to ask questions. <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body></html> diff --git a/doc/tls_cert_server.html b/doc/tls_cert_server.html index 9c68db5d..9c024bc9 100644 --- a/doc/tls_cert_server.html +++ b/doc/tls_cert_server.html @@ -37,6 +37,15 @@ src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> </script> </span> <p><center><img src="tls_cert_100.jpg"></center> +<p><i><font color="red"><b>Important:</b> Keep in mind that the order of configuration directives +is very important in rsyslog. As such, the samples given below do only work if the given +order is preserved.</font> Re-ordering the directives can break configurations and has broken them +in practice. If you intend to re-order them, please be sure that you fully understand how +the configuration language works and, most importantly, which statements form a block together. +Please also note that we understand the the current configuration file format is +ugly. However, there has been more important work in the way of enhancing it. If you would like +to contribute some time to improve the config file language, please let us know. Any help +is appreciated (be it doc or coding work!).</i> <p>Steps to do: <ul> <li>make sure you have a functional CA (<a href="tls_cert_ca.html">Setting up the CA</a>) |
