diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/Makefile.am | 1 | ||||
| -rw-r--r-- | doc/impstats.html | 62 | ||||
| -rw-r--r-- | doc/imuxsock.html | 54 | ||||
| -rw-r--r-- | doc/manual.html | 2 | ||||
| -rw-r--r-- | doc/omhdfs.html | 69 | ||||
| -rw-r--r-- | doc/rsyslog_conf_modules.html | 2 |
6 files changed, 187 insertions, 3 deletions
diff --git a/doc/Makefile.am b/doc/Makefile.am index d4df740a..a5393cbe 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -41,6 +41,7 @@ html_files = \ imfile.html \ imtcp.html \ imptcp.html \ + impstats.html \ imgssapi.html \ imrelp.html \ imsolaris.html \ diff --git a/doc/impstats.html b/doc/impstats.html new file mode 100644 index 00000000..3b4191e8 --- /dev/null +++ b/doc/impstats.html @@ -0,0 +1,62 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>Periodic Statistics of Internal Counters (impstats)</title> +</head> +<body> +<a href="rsyslog_conf_modules.html">back</a> + +<h1>Input Module to Generate Periodic Statistics of Internal Counters</h1> +<p><b>Module Name: impstats</b></p> +<p><b>Available since: </b>5.7.0+, 6.1.1+ +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>This module provides periodic output of rsyslog internal counters. +Note that the whole statistics system is currently under development. So +availabilty and format of counters may change and is not yet stable (so be +prepared to change your trending scripts when you upgrade to a newer rsyslog version). +<p>The set of available counters will be output as a set of syslog messages. This +output is periodic, with the interval being configurable (default is 5 minutes). +Be sure that your configuration records the counter messages (default is syslog.info). +<p>Note that loading this module has impact on rsyslog performance. Depending on +settings, this impact may be severe (for high-load environments). +</p> +<p><b>Configuration Directives</b>:</p> +<ul> +<li>$PStatsInterval <Seconds><br> +Sets the interval, in <b>seconds</b> at which messages are generated. Please note that the +actual interval may be a bit longer. We do not try to be precise and so the interval is +actually a sleep period which is entered after generating all messages. So the actual +interval is what is configured here plus the actual time required to generate messages. +In general, the difference should not really matter. +<li>$PStatsFacility <numerical facility><br> +The numerical syslog facility code to be used for generated messages. Default +is 5 (syslog).This is useful for filtering messages.</li> +<li>$PStatsSeverity <numerical severity><br> +The numerical syslog severity code to be used for generated messages. Default +is 6 (info).This is useful for filtering messages.</li> +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>This module MUST be loaded right at the top of rsyslog.conf, otherwise +stats may not get turned on in all places.</li> +<li>experimental code</li> +</ul> +<p><b>Sample:</b></p> +<p>This activates the module and records messages to /var/log/rsyslog-stats in 10 minute intervals:<br> +</p> +<textarea rows="8" cols="60">$ModLoad impstats +$PStatsInterval 600 +$PStatsSeverity 7 + +syslog.debug /var/log/rsyslog-stats +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2010 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/imuxsock.html b/doc/imuxsock.html index 381374d2..ee5db22d 100644 --- a/doc/imuxsock.html +++ b/doc/imuxsock.html @@ -25,6 +25,21 @@ the past four years. Alternate behaviour may be desirable if gateway-like processes send messages via the local log slot - in this case, it can be enabled via the $InputUnixListenSocketIgnoreMsgTimestamp and $SystemLogSocketIgnoreMsgTimestamp config directives</p> +<p><b>There is input rate limiting available,</b> (since 5.7.1) to guard you against +the problems of a wild running logging process. +If more than $SystemLogRateLimitInterval * $SystemLogRateLimitBurst log messages are emitted +from the same process, those messages with $SystemLogRateLimitSeverity or lower will be +dropped. It is not possible to recover anything about these messages, but imuxsock will +tell you how many it has dropped one the interval has expired AND the next message +is logged. Rate-limiting depends on SCM_CREDENTIALS. If the platform does not support +this socket option, rate limiting is turned off. If multiple sockets are configured, +rate limiting works independently on each of them (that should be what you usually expect). +The same functionality is available for additional log sockets, in which case the +config statements just use +the prefix $IMUXSockRateLimit... but otherwise works exactly the same. +When working with severities, please keep in mind that higher severity numbers mean lower +severity and configure things accordingly. +To turn off rate limiting, set the interval to zero. <p><b>Unix log sockets can be flow-controlled.</b> That is, if processing queues fill up, the unix socket reader is blocked for a short while. This may be useful to prevent overruning the queues (which may cause exessive disk-io where it actually would not be needed). However, @@ -40,12 +55,38 @@ the implications. Note that for many systems, turning on flow control does not h <br>Ignore timestamps included in the message. Applies to the next socket being added.</li> <li><b>$InputUnixListenSocketFlowControl</b> [on/<b>off</b>] - specifies if flow control should be applied to the next socket.</li> +<li><b>$IMUXSockRateLimitInterval</b> [number] - specifies the rate-limiting +interval in seconds. Default value is 5 seconds. Set it to 0 to turn rate limiting off. +</li> +<li><b>$IMUXSockRateLimitBurst</b> [number] - specifies the rate-limiting +burst in number of messages. Default is 200. +</li> +<li><b>$IMUXSockRateLimitSeverity</b> [numerical severity] - specifies the severity of +messages that shall be rate-limited. +</li> +<li><b>$InputUnixListenSocketUsePIDFromSystem</b> [on/<b>off</b>] - specifies if the pid being logged shall +be obtained from the log socket itself. If so, the TAG part of the message is rewritten. +It is recommended to turn this option on, but the default is "off" to keep compatible +with earlier versions of rsyslog. This option was introduced in 5.7.0.</li> <li><b>$SystemLogSocketIgnoreMsgTimestamp</b> [<b>on</b>/off]<br> Ignore timestamps included in the messages, applies to messages received via the system log socket.</li> <li><b>$OmitLocalLogging</b> (imuxsock) [on/<b>off</b>] -- former -o option</li> <li><b>$SystemLogSocketName</b> <name-of-socket> -- former -p option</li> <li><b>$SystemLogFlowControl</b> [on/<b>off</b>] - specifies if flow control should be applied to the system log socket.</li> +<li><b>$SystemLogUsePIDFromSystem</b> [on/<b>off</b>] - specifies if the pid being logged shall +be obtained from the log socket itself. If so, the TAG part of the message is rewritten. +It is recommended to turn this option on, but the default is "off" to keep compatible +with earlier versions of rsyslog. This option was introduced in 5.7.0.</li> +<li><b>$SystemLogRateLimitInterval</b> [number] - specifies the rate-limiting +interval in seconds. Default value is 5 seconds. Set it to 0 to turn rate limiting off. +</li> +<li><b>$SystemLogRateLimitBurst</b> [number] - specifies the rate-limiting +burst in number of messages. Default is 200. +</li> +<li><b>$SystemLogRateLimitSeverity</b> [numerical severity] - specifies the severity of +messages that shall be rate-limited. +</li> <li><b>$InputUnixListenSocketCreatePath</b> [on/<b>off</b>] - create directories in the socket path if they do not already exist. They are created with 0755 permissions with the owner being the process under which rsyslogd runs. The default is not to create directories. Keep in mind, though, that rsyslogd always @@ -66,8 +107,11 @@ will only affect the next one and then automatically be reset. This functionalit that the local hostname can be overridden in cases where that is desired.</li> </ul> <b>Caveats/Known Bugs:</b><br> -<br> -This documentation is sparse and incomplete. +<ul> +<li>There is a compile-time limit of 50 concurrent sockets. If you need more, you need to +change the array size in imuxsock.c. +<li>This documentation is sparse and incomplete. +</ul> <p><b>Sample:</b></p> <p>The following sample is the minimum setup required to accept syslog messages from applications running on the local system.<br> @@ -95,6 +139,12 @@ the $InputUnixListenSocketCreatePath and the $InputUnixListenSocketHostName.</p> $InputUnixListenSocketCreatePath on # turn on for *next* socket $InputUnixListenSocketHostName /var/run/sshd/dev/log </textarea> +<p>The following sample is used to turn off input rate limiting on the system log +socket. +<textarea rows="6" cols="70">$ModLoad imuxsock # needs to be done just once + +$SystemLogRateLimitInterval 0 # turn off rate limiting +</textarea> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the diff --git a/doc/manual.html b/doc/manual.html index 54a87cdc..7fdf3f14 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -19,7 +19,7 @@ rsyslog support</a> available directly from the source!</p> <p><b>Please visit the <a href="http://www.rsyslog.com/sponsors">rsyslog sponsor's page</a> to honor the project sponsors or become one yourself!</b> We are very grateful for any help towards the project goals.</p> -<p><b>This documentation is for version 5.6.2 (beta branch) of rsyslog.</b> +<p><b>This documentation is for version 5.7.2 (beta branch) of rsyslog.</b> Visit the <i><a href="http://www.rsyslog.com/status">rsyslog status page</a></i></b> to obtain current version information and project status. </p><p><b>If you like rsyslog, you might diff --git a/doc/omhdfs.html b/doc/omhdfs.html new file mode 100644 index 00000000..ef7e60c5 --- /dev/null +++ b/doc/omhdfs.html @@ -0,0 +1,69 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>rsyslog output module for HDFS (omhdfs)</title> +<a href="features.html">back</a> +</head> +<body> +<h1>Unix sockets Output Module (omhdfs)</h1> +<p><b>Module Name: omhdfs</b></p> +<p><b>Available since: </b> 5.7.1</p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Description</b>:</p> +<p>This module supports writing message into files on Hadoop's HDFS +file system. +<p><b>Configuration Directives</b>:</p> +<ul> +<li><b>$OMHDFSFileName</b> [name]<br> +The name of the file to which the output data shall be written. +</li> +<li><b>$OMHDFSHost</b> [name]<br> +Name or IP address of the HDFS host to connect to. +</li> +<li><b>$OMHDFSPort</b> [name]<br> +Port on which to connect to the HDFS host. +</li> +<li><b>$OMHDFSDefaultTemplate</b> [name]<br> +Default template to be used when none is specified. This saves the work of +specifying the same template ever and ever again. Of course, the default +template can be overwritten via the usual method. +</li> +</ul> +<b>Caveats/Known Bugs:</b> +<p>Building omhdfs is a challenge because we could not yet find out how +to integrate Java properly into the autotools build process. The issue is +that HDFS is written in Java and libhdfs uses JNI to talk to it. That requires +that various system-specific environment options and pathes be set correctly. At +this point, we leave this to the user. If someone know how to do it better, +please drop us a line! +<ul> +<li>In order to build, you need to set these environment variables BEFORE running +./configure: +<ul> +<li>JAVA_INCLUDES - must have all include pathes that are needed to build +JNI C programms, including the -I options necessary for gcc. An example is<br> +# export JAVA_INCLUDES="-I/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/include -I/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/include/linux" +<li>JAVA_LIBS - must have all library pathes that are needed to build +JNI C programms, including the -l/-L options necessary for gcc. An example is<br> +# export export JAVA_LIBS="-L/usr/java/jdk1.6.0_21/jre/lib/amd64 -L/usr/java/jdk1.6.0_21/jre/lib/amd64/server -ljava -ljvm -lverify" +</ul> + +<li>As of HDFS architecture, you must make sure that all relevant environment +variables (the usual Java stuff and HADOOP's home directory) are properly set. +<li>As it looks, libhdfs makes Java throw exceptions to stdout. There is no +known work-around for this (and it usually should not case any troubles. +</ul> +<p><b>Sample:</b></p> +<p> +</p> +<textarea rows="4" cols="80">$ModLoad omhdfs + +$OMHDFSFileName /var/log/logfile +*.* :omhdfs: +</textarea> +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> + +</body></html> diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html index 2a64461d..74aa319c 100644 --- a/doc/rsyslog_conf_modules.html +++ b/doc/rsyslog_conf_modules.html @@ -44,6 +44,7 @@ to message generators. <li><a href="imuxsock.html">imuxsock</a> - unix sockets, including the system log socket</li> <li><a href="imsolaris.html">imsolaris</a> - input for the Sun Solaris system log source</li> <li><a href="im3195.html">im3195</a> - accepts syslog messages via RFC 3195</li> +<li><a href="impstats.html">impstats</a> - provides periodic statistics of rsyslog internal counters</li> </ul> <a name"om"></a><h2>Output Modules</h2> @@ -65,6 +66,7 @@ permits rsyslog to alert folks by mail if something important happens</li> <li><a href="omoracle.html">omoracle</a> - output module for Oracle (native OCI interface)</li> <li><a href="omudpspoof.html">omudpspoof</a> - output module sending UDP syslog messages with a spoofed address</li> <li><a href="omuxsock.html">omuxsock</a> - output module Unix domain sockets</li> +<li><a href="omhdfs.html">omhdfs</a> - output module for Hadoop's HDFS file system</li> </ul> <a name="pm"></a><h2>Parser Modules</h2> |