diff options
| -rw-r--r-- | BUGS | 59 | ||||
| -rw-r--r-- | doc/bugs.html | 63 | ||||
| -rw-r--r-- | doc/features.html | 42 | ||||
| -rw-r--r-- | doc/history.html | 72 | ||||
| -rw-r--r-- | doc/manual.html | 48 | ||||
| -rw-r--r-- | doc/property_replacer.html | 64 | ||||
| -rw-r--r-- | doc/rsyslog_conf.html | 460 | ||||
| -rw-r--r-- | doc/rsyslog_mysql.html | 240 | ||||
| -rw-r--r-- | doc/rsyslog_stunnel.html | 248 | ||||
| -rw-r--r-- | doc/status.html | 34 |
10 files changed, 1272 insertions, 58 deletions
@@ -1,58 +1 @@ -This file contains know bugs and incompatibilities - -REPEATED LOG LINES - If multiple log lines with the exact same content are received, - the duplicates are NOT suppressed. This is done by sysklogd - in all cases. We plan to add this as an optional feature, - but as of now every line is logged. - -ERRORS ON STARTUP - Error reporting on startup is not particular good. We handle all - errors correctly, but sometimes a message is only reported if - running in debug mode. If you experience some strange behaviour - I suggest to run the syslogd interactively with debugging enabled. - As of August, 2005, the situation has much improved. Keep in mind, - thought, that debug mode is your friend ;) - -EQUALLY-NAMED TEMPLATES - If multiple templates with the SAME name are created, all but the - first definition is IGNORED. So you can NOT (yet) replace a - template definition. - -MEMORY LEAK ON HUPING - Currently, the template memory is NOT freed when SIGHUP is received. - This causes a small memory leak on each HUP. Will be fixed soon and - should not cause real trouble unless you HUP syslogd quite frequently. - -WALLMSG FORMAT (* selector) - This format is actually not 100% compatible with stock syslogd - the - date is missing. Will be fixed soon and can also be fixed just via - the proper template. Anyone up for this? ;) - -CALLS to exit() - I have removed all calls to exit() I have introduced. However, there are - still some left from the original code. Most of them are OK, being done - on initial startup when a severe error happens. There are some, I think, - that might happen during runtime when a memory shortag is detected. We - should look into this issues. Finding some smarter code would allow - rsyslog to become even more fault-tolerant (but obviously, there is always - a cost, in this case most probably the loss of at least on message). - -SIGPIPE HANDLING - Currently, SIGPIPE is ignored. This is necessary to handle broken TCP - connections. We should further look into this issue and see which other - ways exist to handle the situation. - -MULTIPLE INSTANCES - If multiple instances are running on a single machine, the one with - the -r switch must start first. Also, UDP-based syslog forwarding - between the instances does not work. Use TCP instead. - 2005-08-01 - -Updates to the bug lists (and potential fixes...) can be found at the -official home page at - -http://www.monitorware.com/rsyslog - -List as of 2005-03-08 -Rainer Gerhards +This file has been superseeded by doc/bugs.html. Check there. diff --git a/doc/bugs.html b/doc/bugs.html new file mode 100644 index 00000000..ae4fb4fd --- /dev/null +++ b/doc/bugs.html @@ -0,0 +1,63 @@ +<html>
+<head>
+<title>rsyslog bugs and annoyances</title>
+</head>
+<body>
+<h1>rsyslog bugs and annoyances</h1>
+<p><b>This page lists the known bugs rsyslog has to offer.</b> Please note that
+we also have a
+<a href="http://sourceforge.net/tracker/?atid=696552&group_id=123448&func=browse">
+bug tracker at sourceforge.net</a>. This list here contains more architectural
+things while the bug tracker most often lists things that you will actually
+experience. I am working to combine the two lists, but for now you need to visit
+both.</p>
+<p>This list has last been updated on 2005-08-02 by
+<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a>.</p>
+<h1>rsyslogd</h1>
+<h2>REPEATED LOG LINES
+ </h2>
+<p>If multiple log lines with the exact same content are received,
+ the duplicates are NOT suppressed. This is done by sysklogd
+ in all cases. We plan to add this as an optional feature,
+ but as of now every line is logged.</p>
+<h2>ERRORS ON STARTUP</h2>
+<p>Error reporting on startup is not particular good. We handle all
+ errors correctly, but sometimes a message is only reported if
+ running in debug mode. If you experience some strange behaviour
+ I suggest to run the syslogd interactively with debugging enabled.
+ As of August, 2005, the situation has much improved. Keep in mind,
+ thought, that debug mode is your friend ;)</p>
+<h2>EQUALLY-NAMED TEMPLATES</h2>
+<p>If multiple templates with the SAME name are created, all but the
+ first definition is IGNORED. So you can NOT (yet) replace a
+ template definition. I also strongly doubt I will ever support this, because
+it does not make an awful lot of sense (after all, why not use two template
+names...).</p>
+<h2>MEMORY LEAK ON HUPING</h2>
+<p>Currently, the template memory is NOT freed when SIGHUP is received.
+ This causes a small memory leak on each HUP. Will be fixed soon and
+ should not cause real trouble unless you HUP syslogd quite frequently.</p>
+<h2>WALLMSG FORMAT (* selector)
+ </h2>
+<p>This format is actually not 100% compatible with stock syslogd - the
+ date is missing. Will be fixed soon and can also be fixed just via
+ the proper template. Anyone up for this? ;)</p>
+<h2>CALLS to exit()</h2>
+<p>I have removed all calls to exit() I have introduced. However, there are
+ still some left from the original code. Most of them are OK, being done
+ on initial startup when a severe error happens. There are some, I think,
+ that might happen during runtime when a memory shortage is detected. We
+ should look into this issues. Finding some smarter code would allow
+ rsyslog to become even more fault-tolerant (but obviously, there is always
+ a cost, in this case most probably the loss of at least one message).</p>
+<h2>SIGPIPE HANDLING</h2>
+<p>Currently, SIGPIPE is ignored. This is necessary to handle broken TCP
+ connections. We should further look into this issue and see which other
+ ways exist to handle the situation.</p>
+<h2>MULTIPLE INSTANCES</h2>
+<p>If multiple instances are running on a single machine, the one with
+ the -r switch must start first. Also, UDP-based syslog forwarding between the
+instances does not work. Use TCP instead.</p>
+
+</body>
+</html>
\ No newline at end of file diff --git a/doc/features.html b/doc/features.html new file mode 100644 index 00000000..0b41afa0 --- /dev/null +++ b/doc/features.html @@ -0,0 +1,42 @@ +<html>
+<head>
+<title>rsyslog features</title>
+</head>
+<body>
+<h1>RSyslog - Features</h1>
+<p><b>This page lists both current features as well as those being considered
+for future versions of rsyslog.</b> If you think a feature is missing, drop
+<a href="mailto:rgerhards@adiscon.com">Rainer</a> a note. Rsyslog is a vital
+project. Features are added each few days. If you would like to keep up of what
+is going on, you can also subscribe to the <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">rsyslog mailing list</a>.
+</p>
+<h2>Current Features</h2>
+<ul>
+
+ <li>native support for <a href="rsyslog_mysql.html">writing to MySQL databases</a><li>support for (plain) tcp
+ based syslog - much better reliability<li>control of log output format,
+ including ability to present channel and priority as visible log data<li>good timestamp format control; at a minimum, ISO 8601/RFC 3339
+ second-resolution UTC zone<li>ability to reformat message contents and work with substrings<li>support for
+ log files larger than 2gb<li>support for file size limitation and automatic
+ rollover command execution<li>support for running multiple rsyslogd
+ instances on a single machine<li>support for <a href="rsyslog_stunnel.html">
+ ssl-protected syslog</a> (via stunnel)</ul>
+<h2>Upcoming Features</h2>
+<ul>
+ <li>support for
+ <a href="http://www.syslog.cc/ietf/drafts/draft-ietf-syslog-protocol-14.txt">
+ syslog-protocol-14</a> compliant messages
+ <li>support for native SSL enryption of plain tcp syslog sessions<li>control of whether the local hostname or the hostname of the
+ origin of the data is shown as the hostname in the output - prepared
+ for, but not yet implemented (should be easy)
+ <li>support for syslog-protocol - planned
+ <li>RFC 3195 support - planned (but late)
+ <li>pcre filtering - maybe (depending on feedback)
+ <li>multi-threaded redesign - maybe (depending on feedback; also
+ opens Pandorra's cross-platform development box ;))
+ </ul>
+<p>To see when each feature was added, see the
+<a href="http://www.rsyslog.com/Topic4.phtml">rsyslog change log</a> (online
+only).</p>
+</body>
+</html>
\ No newline at end of file diff --git a/doc/history.html b/doc/history.html new file mode 100644 index 00000000..de427ce5 --- /dev/null +++ b/doc/history.html @@ -0,0 +1,72 @@ +<html>
+<head>
+<title>rsyslog history</title>
+</head>
+<body>
+<h1>RSyslog - History</h1>
+
+<b>Rsyslog is a GPL-ed, enhanced syslogd. Among others, it offers support for
+reliable syslog over TCP, writing to
+MySQL databases and fully configurable output formats (including great timestamps).</b>
+Rsyslog was initiated by Rainer Gerhards. It has
+been forked from the <a href="http://www.infodrom.org/projects/sysklogd/">sysklogd standard package</a>.
+The goal of the
+rsyslog project is to provide a more configurable and reliable
+syslog deamon while retaining drop-in replacement capabilities for stock syslogd. By "reliable", we mean support for reliable transmission
+modes like TCP or <a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php">RFC 3195</a> (syslog-reliable).
+We do NOT imply that the sysklogd package is unreliable. In fact, the
+opposite is the case and we assume that for the time being the well-used
+sysklogd package offers better program reliability than our
+brand-new modifications to it.
+
+</p><p>The name "rsyslog" stems back to the
+planned support for syslog-reliable. Ironically, the initial release
+of rsyslog did NEITHER support syslog-reliable NOR tcp based syslog.
+Instead, it contains enhanced configurability and other enhancements
+(like database support). The reason for this is that full support for
+RFC 3195 would require even more changes and especially fundamental architectural
+changes. Also, questions asked on the loganalysis list and at other
+places indicated that RFC3195 is NOT a prime priority for users, but
+rather better control over the output format. So here we are, with
+a rsyslod that covers a lot of enhancements, but not a single one
+of these that made its name ;) Since version 0.9.2, receiving syslog messages
+via plain tcp is finally supported, bringing rsyslogd a bit closer to its name.
+</p><p>
+The next enhancement scheduled is support for the new syslog-protocol
+internet draft format, not the least to see how easy/complicated it is
+to implement. We already know that some subleties of syslog-protocol will
+require at least one considerable architectural change to the syslogd
+and this might delay things a little. Our immediate goal is to receive
+feedback and get the bugs out of the current release. Only after that
+we intend to advance the code and introduce new features.
+</p><p>
+The database support was included so that our web-based syslog interface
+can be used. This is another open source project which can be found
+under <a href="http://www.liblogging.org/">http://www.liblogging.org</a> (this
+site needs serious redesign!). We highly recommend having a look at
+it. It might not work for you if you expect thousands of messages per
+second (because your database won't be able to provide adequate performance),
+but in many cases it is a very handy analysis and troubleshooting tool.
+
+</p>
+<p>Rsyslogd supports an enhanced syslog.conf file format, and also works
+with the standard syslog.conf. In theory, it should be possible to simply replace
+the syslogd binary with the one that comes with rsyslog. Of course, in order
+to use any of the new features, you must re-write your syslog.conf. To learn
+how to do this, please review our commented <a href="sample.conf.php">sample.conf</a>
+file. It outlines the enhancements over stock syslogd.
+<p>If you are interested in the <a href="http://en.wikipedia.org/wiki/IHE">IHE</a>
+environment, you might be interested to hear that rsyslog supports message with
+sizes of 32k and more. This feature has been tested, but by default is turned off
+(as it has some memory footprint that we didn't want to put on users not
+actually requiring it). Search the file syslogd.c and search for "IHE" - you
+will find easy and precise instructions on what you need to change (it's just
+one line of code!).
+<p>Be sure to visit Rainer's <a href="http://rgerhards.blogspot.com/">syslog block</a>
+to get some more insight into the development of rsyslog and syslog in general.</p>
+<h2>Some useful links</h2>
+<ul>
+ <li><a href="http://www.rsyslog.com/Topic4.phtml">the rsyslog change log</a></li>
+</ul>
+</body>
+</html>
\ No newline at end of file diff --git a/doc/manual.html b/doc/manual.html new file mode 100644 index 00000000..48863933 --- /dev/null +++ b/doc/manual.html @@ -0,0 +1,48 @@ +<html>
+<head>
+<title>rsyslog documentation</title>
+</head>
+<body>
+<h1>RSyslog - Documentation</h1>
+<p><b><a href="http://www.rsyslog.com/">Rsyslog</a> is an enhanced syslogd
+supporting, among others, <a href="rsyslog_mysql.html">MySQL</a>, syslog/tcp
+and fine grain output format control.</b> It is quite compatible to stock
+sysklogd and can be used as a drop-in replacement. Its <a href="features.html">
+advanced features</a> make it suitable for enterprise-class,
+<a href="rsyslog_stunnel.html">encryption protected syslog</a>
+relay chains while at the same time being very easy to setup
+for the novice user.</p>
+<p>Visit the <a href="status.html">rsyslog status page</a> to obtain current
+version information and ports.</p>
+<p><b>Follow the links below for the</b></p>
+<ul>
+<li><a href="rsyslogd.man.txt">rsyslogd man page</a>
+<li><a href="rsyslog_conf.html">configuration file syntax (rsyslog.conf)</a>
+<li>a commented <a href="sample.conf.html">sample rsyslog.conf</a>
+<li><a href="bugs.html">rsyslog bug list</a></ul>
+<p><b>We have some in-depth papers on</b></p>
+<ul>
+ <li><a href="rsyslog_stunnel.html">ssl-encrypting syslog with stunnel</a></li>
+ <li><a href="rsyslog_mysql.html">writing syslog messages to MySQL</a></li>
+</ul>
+<p>Also, there is an article from Dennis Olvany on
+<a href="rsyslog084-freebsd5.4.txt">Syslog-to-SQL with rsyslog-0.8.4 on FreeBSD 5.4</a>
+(which unfortunately is a bit outdated now).</p>
+<p>Our <a href="history.html">rsyslog history</a> page is for you if you would like to learn a little more
+on why there is an rsyslog at all.</p>
+<p>Documentation is added continously. Please note that the documentation here
+matches only the current version of rsyslog. If you use an older version, be sure
+to use the doc that came with it.</p>
+<p><b>You can also browse the following online ressources:</b></p>
+<ul>
+<li><a href="http://www.rsyslog.com/module-Static_Docs-view-f-manual.html.phtml">rsyslog online documentation</a></li>
+<li><a href="http://www.rsyslog.com/Topic3.phtml">rsyslog FAQ</a></li>
+<li><a href="http://www.rsyslog.com/PNphpBB2.phtml">rsyslog discussion forum</a></li>
+<li><a href="http://www.rsyslog.com/Topic4.phtml">rsyslog change log</a></li>
+</ul>
+<p>And don't forget about the <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">rsyslog mailing list</a>.
+If you are interested in the "backstage", you may find
+<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer</a>'s
+<a href="http://rgerhards.blogspot.com/">syslog blog</a> an interesting read.</p>
+</body>
+</html>
\ No newline at end of file diff --git a/doc/property_replacer.html b/doc/property_replacer.html new file mode 100644 index 00000000..6d90fa45 --- /dev/null +++ b/doc/property_replacer.html @@ -0,0 +1,64 @@ +<html>
+<head>
+<title>The Rsyslogd Property Replacer</title>
+</head>
+<body>
+<h1>The Property Replacer</h1>
+<p><b>The property replacer is a core component in rsyslogd's output system.</b>
+A syslog message has a number of well-defined properties (see below). Each of
+this properties can be accessed <b>and</b> manipulated by the property replacer.
+With it, it is easy to use only part of a property value or manipulate the value,
+e.g. by converting all characters to lower case.</p>
+<h1>Accessing Properties</h1>
+<p>Syslog message properties are used inside templates. They are accessed by putting them between percent signs. Properties can be modified by
+the property replacer. The full syntax is as follows:</p>
+<blockquote><b><code>%propname:fromChar:toChar:options%</code></b></blockquote>
+<h2>Available Properties</h2>
+<p><b><code>propname</code></b> is the name of the property to access. It is case-sensitive.
+Currently supported are:</p>
+<table>
+<tr><td><b>msg</b></td><td>the MSG part of the message (aka "the message" ;))</td></tr>
+<tr><td><b>rawmsg</b></td><td>the message excactly as it was received from the
+socket. Should be useful for debugging.</td></tr>
+<tr><td><b>UxTradMsg</b></td><td>will disappear soon - do NOT use!</td></tr>
+<tr><td><b>HOSTNAME</b></td><td>hostname from the message</td></tr>
+<tr><td><b>source</b></td><td>alias for HOSTNAME</td></tr>
+<tr><td><b>syslogtag</b></td><td>TAG from the message</td></tr>
+<tr><td><b>PRI</b></td><td>PRI part of the message - undecoded (single value)</td></tr>
+<tr><td><b>IUT</b></td><td>the monitorware InfoUnitType - used when talking
+to a <a href="http://www.monitorware.com">MonitorWare</a> backend (also for
+ <a href="http://www.phplogcon.org/">phpLogCon</a>)</td></tr>
+<tr><td><b>syslogfacility</b></td><td>the facility from the message - in numerical form</td></tr>
+<tr><td><b>syslogpriority</b></td><td>the priority (actully severity!) from the
+ message - in numerical form</td></tr>
+<tr><td><b>timegenerated</b></td><td>timestamp when the message was RECEIVED. Always in
+ high resolution</td></tr>
+<tr><td><b>timereported</b></td><td>timestamp from the message. Resolution depends on
+what was provided in the message (in most cases,
+only seconds)</td></tr>
+<tr><td><b>TIMESTAMP</b></td><td>alias for timereported</td></tr>
+</table>
+<h2>Character Positions</h2>
+<p><b><code>FromChar</code></b> and <b><code>toChar</code></b> are used to build substrings. They specify the offset within
+the string that should be copied. Offset counting starts at 1, so if you need to
+obtain the first 2 characters of the message text, you can use this syntax:
+"%msg:1:2%". If you do not whish to specify from and to, but you want to specify
+options, you still need to include the colons. For example, if you would like to
+convert the full message text to lower case, use "%msg:::lowercase%".<br>
+<br>
+<h2>Property Options</h2>
+<b><code>property options</code></b> are case-insensitive. Currently, the following options
+are defined:</p>
+<table>
+<tr><td><b>uppercase</b></td><td>convert property to lowercase only</td></tr>
+<tr><td><b>lowercase</b></td><td>convert property text to uppercase only</td></tr>
+<tr><td><b>drop-last-lf</b></td><td>The last LF in the message (if any), is dropped.
+ Especially useful for PIX.</td></tr>
+<tr><td><b>date-mysql</b></td><td>format as mysql date</td></tr>
+<tr><td><b>date-rfc3164</b></td><td>format as RFC 3164 date</td></tr>
+<tr><td><b>date-rfc3339</b></td><td>format as RFC 3339 date</td></tr>
+<tr><td><b>escape-cc</b></td><td>NOT yet implemented</td></tr>
+</table>
+
+</body>
+</html>
\ No newline at end of file diff --git a/doc/rsyslog_conf.html b/doc/rsyslog_conf.html new file mode 100644 index 00000000..bbf97198 --- /dev/null +++ b/doc/rsyslog_conf.html @@ -0,0 +1,460 @@ +<html>
+<head>
+<title>rsyslog.conf file</title>
+</head>
+<body>
+<h1>rsyslog.conf configuration file</h1>
+<p><b>This document is currently being enhanced. Please pardon its current
+appearance.</b></p>
+<p><b>Rsyslogd is configured via the rsyslog.conf file</b>, typically found in
+/etc. By default, rsyslog reads the file /etc/rsyslog.conf.</p>
+<p>While rsyslogd contains enhancements over standard syslogd, efforts have been
+made to keep the configuration file as compatible as possible. While, for
+obvious reasons, <a href="features.html">enhanced features</a> require a
+different config file syntax, rsyslogd should be able to work with a standard
+syslog.conf file. This is especially useful while you are migrating from syslogd
+to rsyslogd.</p>
+<h2>Basic Structure</h2>
+<p>Every rule consists of two fields, a selector field and an action field.
+These two fields are separated by one or more spaces or tabs. The selector field
+specifies a pattern of facilities and priorities belonging to the specified
+action.<br>
+<br>
+Lines starting with a hash mark (``#'') and empty lines are ignored.
+<h2>Templates</h2>
+<p>Templates are a key feature of rsyslog. They allow to specify any format a user
+might want. Every output in rsyslog uses templates - this holds true for files,
+user messages and so on. The database writer expects its template to be a proper
+SQL statement - so this is highly customizable too. You might ask how does all
+of this work when no templates at all are specified. Good question ;) The answer
+is simple, though. Templates compatible with the stock syslogd formats are
+hardcoded into rsyslogd. So if no template is specified, we use one of these
+hardcoded templates. Search for "template_" in syslogd.c and you will find the
+hardcoded ones.</p>
+<p>A template consists of a template directive, a name, the actual template text
+and optional options. A sample is:</p>
+<blockquote><code>$template MyTemplateName,"\7Text %property% some more text\n",<options></code></blockquote>
+<p>The "$template" is the template directive. It tells rsyslog that this line
+contains a template. "MyTemplateName" is the template name. All
+other config lines refer to this name. The text within quotes is the actual
+template text. The backslash is an escape character, much as it is in C. It does
+all these "cool" things. For example, \7 rings the bell (this is an ASCII
+value), \n is a new line. C programmers and perl coders have the advantage of
+knowing this, but the set in rsyslog is a bit restricted currently.
+<p>
+All text in the template is used literally, except for things within percent
+signs. These are properties and allow you access to the contents of the syslog
+message. Properties are accessed via the property replacer (nice name, huh) and
+it can do cool things, too. For example, it can pick a substring or do
+date-specific formatting. More on this is below, on some lines of the property
+replacer.<br>
+<br>
+The <options> part is optional. It carries options influencing the template as
+whole. See details below. Be sure NOT to mistake template options with property
+options - the later ones are processed by the property replacer and apply to a
+SINGLE property, only (and not the whole template).<br>
+<br>
+Template options are case-insensitive. Currently defined are: </p>
+<p><b>sql</b> - format the string suitable for a SQL statement. This will
+replace single quotes ("'") by two single quotes ("''") inside each field. This
+option MUST be specified when a template is used for writing to a database,
+otherwise SQL injection might occur.<br>
+<br>
+Please note that the database writer *checks* that the sql option is present in
+the template. If it is not present, the write database action is disabled. This
+is to guard you against accidential forgetting it and then becoming vulnerable
+to SQL injection. The sql option can also be useful with files - especially if
+you want to import them into a database on another machine for performance
+reasons. However, do NOT use it if you do not have a real need for it - among
+others, it takes some toll on the processing time. Not much, but on a really
+busy system you might notice it ;)<br>
+<br>
+To escape:<br>
+% = \%<br>
+\ = \\ --> '\' is used to escape (as in C)<br>
+$template TraditionalFormat,%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"<br>
+<br>
+Properties can be accessed by the <a href="property_replacer.html">property replacer</a>
+(see there for details).</p>
+<h2>Output Channels</h2>
+<p>Output Channels are a new concept first introduced in rsyslog 0.9.0. As of this
+writing, it is still unclear if they will stay in rsyslog or go away. So if you
+use them, be prepared to change you configuration file syntax when you upgrade
+to a later release.<br>
+<br>
+The idea behind output channel definitions is that it shall provide an umbrella
+for any type of output that the user might want. In essence,<br>
+this is the "file" part of selector lines (and this is why we are not sure
+output channel syntax will stay after the next review). There is a<br>
+difference, though: selector channels both have filter conditions (currently
+facility and severity) as well as the output destination. Output channels define
+the output defintion, only. As of this build, they can only be used to write to
+files - not pipes, ttys or whatever else. If we stick with output channels, this
+will change over time.</p>
+<p>In concept, an output channel includes everything needed to know about an
+output actions. In practice, the current implementation only carries<br>
+a filename, a maximum file size and a command to be issued when this file size
+is reached. More things might be present in future version, which might also
+change the syntax of the directive.</p>
+<p>Output channels are defined via an $outchannel directive. It's syntax is as
+follows:<br>
+<br>
+$outchannel name,file-name,max-size,action-on-max-size<br>
+<br>
+name is the name of the output channel (not the file), file-name is the file
+name to be written to, max-size the maximum allowed size and action-on-max-size
+a command to be issued when the max size is reached.<br>
+<br>
+Please note that max-size is queried BEFORE writing the log message to the file.
+So be sure to set this limit reasonably low so that any message might fit. For
+the current release, setting it 1k lower than you expected is helpful. The
+max-size must always be specified in bytes - there are no special symbols (like
+1k, 1m,...) at this point of development.<br>
+<br>
+Keep in mind that $outchannel just defines a channel with "name". It does not
+activate it. To do so, you must use a selector line (see below). That selector
+line includes the channel name plus an $ sign in front of it. A sample might be:<br>
+<br>
+*.* $mychannel<br>
+<br>
+In its current form, output channels primarily provide the ability to size-limit
+an output file. To do so, specify a maximum size. When this size is reachead,
+rsyslogd will execute the action-on-max-size command and then reopen the file
+and retry. The command should be something like a log rotation script or a
+similar thing.</p>
+<blockquote>
+ <p><b>WARNING</b>
+ <p>The current command logic is a quick hack. It simply issues the command via a
+system() call, which is very dirty. Don't make rsyslogd a suid
+binary and use action-on-max-size commands - this will mess up things. Fixing
+this is on top of the todo list and the fix will hopefully
+appear soon.</p>
+</blockquote>
+<p>If there is no action-on-max-size command or the command did not resolve the
+situation, the file is closed and never reopened by rsyslogd (except, of course,
+by huping it). This logic was integrated when we first experienced severe issues
+with files larger 2gb, which could lead to rsyslogd dumping core. In such cases,
+it is more appropriate to stop writing to a single file. Meanwhile, rsyslogd has
+been fixed to support files larger 2gb, but obviously only on file systems and
+operating system versions that do so. So it can still make sense to enforce a
+2gb file size limit.</p>
+<h2>SELECTORS</h2>
+<p>The selector field itself again consists of two parts, a facility and a
+priority, separated by a period (``.''). Both parts are case insenstive and can
+also be specified as decimal numbers, but don't do that, you have been warned.
+Both facilities and priorities are described in rsyslog(3). The names mentioned
+below correspond to the similar LOG_-values in /usr/include/rsyslog.h.<br><br>The facility is one of the following keywords: auth, authpriv, cron, daemon,
+kern, lpr, mail, mark, news, security (same as auth), syslog, user, uucp and
+local0 through local7. The keyword security should not be used anymore and mark
+is only for internal use and therefore should not be used in applications.
+Anyway, you may want to specify and redirect these messages here. The facility
+specifies the subsystem that produced the message, i.e. all mail programs log
+with the mail facility (LOG_MAIL) if they log using syslog.<br><br>Please note that the upcoming next syslog-RFC specifies many more facilities.
+Support for them will be added in a future version of rsyslog, which might
+require changes to existing configuration files.<br><br>The priority is one of the following keywords, in ascending order: debug, info,
+notice, warning, warn (same as warning), err, error (same as err), crit, alert,
+emerg, panic (same as emerg). The keywords error, warn and panic are deprecated
+and should not be used anymore. The priority defines the severity of the message<br>
+<br>The behavior of the original BSD syslogd is that all messages of the specified
+priority and higher are logged according to the given action. This rsyslogd(8)
+behaves the same, but has some extensions.<br><br>In addition to the above mentioned names the rsyslogd(8) understands the
+following extensions: An asterisk (``*'') stands for all facilities or all
+priorities, depending on where it is used (before or after the period). The
+keyword none stands for no priority of the given facility.<br><br>You can specify multiple facilities with the same priority pattern in one
+statement using the comma (``,'') operator. You may specify as much facilities
+as you want. Remember that only the facility part from such a statement is
+taken, a priority part would be skipped.</p>
+<p>Multiple selectors may be specified for a single action using the semicolon
+(``;'') separator. Remember that each selector in the selector field is capable
+to overwrite the preceding ones. Using this behavior you can exclude some
+priorities from the pattern.</p>
+<p>Rsyslogd(8) has a syntax extension to the original BSD source, that makes its
+use more intuitively. You may precede every priority with an equation sign
+(``='') to specify only this single priority and not any of the above. You may
+also (both is valid, too) precede the priority with an exclamation mark (``!'')
+to ignore all that priorities, either exact this one or this and any higher
+priority. If you use both extensions than the exclamation mark must occur before
+the equation sign, just use it intuitively.</p>
+<h2>ACTIONS</h2>
+<p>The action field of a rule describes what to do with the message. In general,
+message content is written to a kind of "logfile". But also other actions might
+be done, like writing to a database table or forwarding to another host.<br>
+<br>
+Templates can be used with all actions. If used, the specified template is used
+to generate the message content (instead of the default template). To specify a
+template, write a semicolon after the action value immediately followed by the
+template name.<br>
+<br>
+Beware: templates MUST be defined BEFORE they are used. It is OK to define some
+templates, then use them in selector lines, define more templates and use use
+them in the following selector lines. But it is NOT permitted to use a template
+in a selectore line that is above its definition. If you do this, the selector
+line will be ignored.</p>
+<h3>Regular File</h3>
+<p>Typically messages are logged to real files. The file has to be specified with
+full pathname, beginning with a slash ``/''.<br>
+<br>
+You may prefix each entry with the minus ``-'' sign to omit syncing the file
+after every logging. Note that you might lose information if the system crashes
+right behind a write attempt. Nevertheless this might give you back some
+performance, especially if you run programs that use<br>
+logging in a very verbose manner.</p>
+<p>If your system is connected to a reliable UPS and you receive lots of log
+data (e.g. firewall logs), it might be a very good idea to turn of<br>
+syncing by specifying the "-" in front of the file name. </p>
+<h3>Named Pipes</h3>
+<p>This version of rsyslogd(8) has support for logging output to named pipes (fifos).
+A fifo or named pipe can be used as a destination for log messages by prepending
+a pipe symbol (``|'') to the name of the file. This is handy for debugging. Note
+that the fifo must be created with the mkfifo(1) command before rsyslogd(8) is
+started.</p>
+<h3>Terminal and Console</h3>
+<p>If the file you specified is a tty, special tty-handling is done, same with
+/dev/console.</p>
+<h3>Remote Machine</h3>
+<p>Rsyslogd provides full remote logging, i.e. is able to send messages to a
+remote host running rsyslogd(8) and to receive messages from remote hosts. The
+remote host won't forward the message again, it will just log them locally. To
+forward messages to another host, prepend the hostname with the at sign (``@'').<br>
+<br>
+Using this feature you're able to control all syslog messages on one host, if
+all other machines will log remotely to that. This tears down<br>
+administration needs.<br>
+<br>
+Please note that this version of rsyslogd by default does NOT forward messages
+it has received from the network to another host. Specify the<br>
+-h option to enable this.</p>
+<h3>List of Users</h3>
+<p>Usually critical messages are also directed to ``root'' on that machine. You can
+specify a list of users that shall get the message by simply writing the login.
+You may specify more than one user by separating them with commas (``,''). If
+they're logged in they get the message. Don't think a mail would be sent, that
+might be too late.</p>
+<h3>Everyone logged on</h3>
+<p>Emergency messages often go to all users currently online to notify them that
+something strange is happening with the system. To specify this wall(1)-feature
+use an asterisk (``*'').</p>
+<h3>Database Table</h3>
+<p>This allows logging of the message to a database table. Currently, only MySQL
+databases are supported. By default, a MonitorWare-compatible schema is required
+for this to work. You can create that schema with the createDB.SQL file that
+came with the rsyslog package. You can also<br>
+use any other schema of your liking - you just need to define a proper template
+and assign this template to the action.<br>
+<br>
+The database writer is called by specifying a greater-then sign (">") in front
+of the database connect information. Immediately after that<br>
+sign the database host name must be given, a comma, the database name, another
+comma, the database user, a comma and then the user's password. If a specific
+template is to be used, a semicolong followed by the template name can follow
+the connect information. This is as follows:<br>
+<br>
+>dbhost,dbname,dbuser,dbpassword;dbtemplate</p>
+<h3>Output Channel</h3>
+<p>Binds an output channel definition (see there for details) to this action.
+Output channel actions must start with a $-sign, e.g. if you would like to bind
+your output channel definition "mychannel" to the action, use "$mychannel".
+Output channels support template definitions like all all other actions.</p>
+<h2>TEMPLATE NAME</h2>
+<p>Every ACTION can be followed by a template name. If so, that template is used
+for message formatting. If no name is given, a hardcoded default template is
+used for the action. There can only be one template name for each given action.
+The default template is specific to each action. For a description of what a
+template is and what you can do with it, see "TEMPLATES" at the top of this
+document.</p>
+<h2>EXAMPLES</h2>
+<p>Below are example for templates and selector lines. I hope they are
+self-explanatory. If not, please see www.monitorware.com/rsyslog/ for advise.</p>
+<h3>TEMPLATES</h3>
+<p>Please note that the samples are split across multiple lines. A template MUST
+NOT actually be split across multiple lines.<br>
+<br>
+A template that resambles traditional syslogd file output:<br>
+$template TraditionalFormat,"%timegenerated% %HOSTNAME%<br>
+%syslogtag%%msg:::drop-last-lf%\n"<br>
+<br>
+A template that tells you a little more about the message:<br>
+$template precise,"%syslogpriority%,%syslogfacility%,%timegenerated%,%HOSTNAME%,<br>
+%syslogtag%,%msg%\n"<br>
+<br>
+A template for RFC 3164 format:<br>
+$template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%"<br>
+<br>
+A template for the format traditonally used for user messages:<br>
+$template usermsg," XXXX%syslogtag%%msg%\n\r"<br>
+<br>
+And a template with the traditonal wall-message format:<br>
+$template wallmsg,"\r\n\7Message from syslogd@%HOSTNAME% at %timegenerated%<br>
+<br>
+A template that can be used for the database write (please note the SQL<br>
+template option)<br>
+$template MySQLInsert,"insert iut, message, receivedat values<br>
+('%iut%', '%msg:::UPPERCASE%', '%timegenerated:::date-mysql%')<br>
+into systemevents\r\n", SQL<br>
+<br>
+The following template emulates <a href="http://www.winsyslog.com/en/">WinSyslog</a>
+format (it's an <a href="http://www.adiscon.com/en/">Adiscon</a> format, you do
+not feel bad if you don't know it ;)). It's interesting to see how it takes
+different parts out of the date stamps. What happens is that the date stamp is
+split into the actual date and time and the these two are combined with just a
+comma in between them.<br>
+<br>
+$template WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,<br>
+%timegenerated:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%,<br>
+%timegenerated:12:19:date-rfc3339%,%syslogfacility%,%syslogpriority%,<br>
+%syslogtag%%msg%\n"</p>
+<h3>SELECTOR LINES</h3>
+<p># Store critical stuff in critical<br>
+#<br>
+*.=crit;kern.none /var/adm/critical<br>
+<br>
+This will store all messages with the priority crit in the file /var/adm/critical,
+except for any kernel message.<br>
+<br>
+<br>
+# Kernel messages are first, stored in the kernel<br>
+# file, critical messages and higher ones also go<br>
+# to another host and to the console. Messages to<br>
+# the host finlandia are forwarded in RFC 3164<br>
+# format (using the template defined above).<br>
+#<br>
+kern.* /var/adm/kernel<br>
+kern.crit @finlandia;RFC3164fmt<br>
+kern.crit /dev/console<br>
+kern.info;kern.!err /var/adm/kernel-info<br>
+<br>
+The first rule direct any message that has the kernel facility to the file /var/adm/kernel.<br>
+<br>
+The second statement directs all kernel messages of the priority crit and higher
+to the remote host finlandia. This is useful, because if the host crashes and
+the disks get irreparable errors you might not be able to read the stored
+messages. If they're on a remote host, too, you still can try to find out the
+reason for the crash.<br>
+<br>
+The third rule directs these messages to the actual console, so the person who
+works on the machine will get them, too.<br>
+<br>
+The fourth line tells rsyslogd to save all kernel messages that come with
+priorities from info up to warning in the file /var/adm/kernel-info. Everything
+from err and higher is excluded.<br>
+<br>
+<br>
+# The tcp wrapper loggs with mail.info, we display<br>
+# all the connections on tty12<br>
+#<br>
+mail.=info /dev/tty12<br>
+<br>
+This directs all messages that uses mail.info (in source LOG_MAIL | LOG_INFO) to
+/dev/tty12, the 12th console. For example the tcpwrapper tcpd(8) uses this as
+it's default.<br>
+<br>
+<br>
+# Store all mail concerning stuff in a file<br>
+#<br>
+mail.*;mail.!=info /var/adm/mail<br>
+<br>
+This pattern matches all messages that come with the mail facility, except for
+the info priority. These will be stored in the file /var/adm/mail.<br>
+<br>
+<br>
+# Log all mail.info and news.info messages to info<br>
+#<br>
+mail,news.=info /var/adm/info<br>
+<br>
+This will extract all messages that come either with mail.info or with news.info
+and store them in the file /var/adm/info.<br>
+<br>
+<br>
+# Log info and notice messages to messages file<br>
+#<br>
+*.=info;*.=notice;\<br>
+mail.none /var/log/messages<br>
+<br>
+This lets rsyslogd log all messages that come with either the info or the notice
+facility into the file /var/log/messages, except for all<br>
+messages that use the mail facility.<br>
+<br>
+<br>
+# Log info messages to messages file<br>
+#<br>
+*.=info;\<br>
+mail,news.none /var/log/messages<br>
+<br>
+This statement causes rsyslogd to log all messages that come with the info
+priority to the file /var/log/messages. But any message coming either with the
+mail or the news facility will not be stored.<br>
+<br>
+<br>
+# Emergency messages will be displayed using wall<br>
+#<br>
+*.=emerg *<br>
+<br>
+This rule tells rsyslogd to write all emergency messages to all currently logged
+in users. This is the wall action.<br>
+<br>
+<br>
+# Messages of the priority alert will be directed<br>
+# to the operator<br>
+#<br>
+*.alert root,rgerhards<br>
+<br>
+This rule directs all messages with a priority of alert or higher to the
+terminals of the operator, i.e. of the users ``root'' and ``rgerhards'' if
+they're logged in.<br>
+<br>
+<br>
+*.* @finlandia<br>
+<br>
+This rule would redirect all messages to a remote host called finlandia. This is
+useful especially in a cluster of machines where all syslog messages will be
+stored on only one machine.<br>
+<br>
+In the format shown above, UDP is used for transmitting the message. The
+destination port is set to the default auf 514. Rsyslog is also capable of using
+much more secure and reliable TCP sessions for message forwarding. Also, the
+destination port can be specified. To select TCP, simply add one additional @ in
+front of the host name (that is, @host is UPD, @@host is TCP). For example:<br>
+<br>
+<br>
+*.* @@finlandia<br>
+<br>
+To specify the destination port on the remote machine, use a colon followed by
+the port number after the machine name. The following forwards to port 1514 on
+finlandia:<br>
+<br>
+<br>
+*.* @@finlandia:1514<br>
+<br>
+This syntax works both with TCP and UDP based syslog. However, you will probably
+primarily need it for TCP, as there is no well-accepted port for this transport
+(it is non-standard). For UDP, you can usually stick with the default auf 514,
+but might want to modify it for security rea-<br>
+sons. If you would like to do that, it's quite easy:<br>
+<br>
+<br>
+*.* @finlandia:1514<br>
+<br>
+<br>
+<br>
+*.* >dbhost,dbname,dbuser,dbpassword;dbtemplate<br>
+<br>
+This rule writes all message to the database "dbname" hosted on "dbhost". The
+login is done with user "dbuser" and password "dbpassword". The actual table
+that is updated is specified within the template (which contains the insert
+statement). The template is called "dbtemplate" in this case.</p>
+<h2>CONFIGURATION FILE SYNTAX DIFFERENCES</h2>
+<p>Rsyslogd uses a slightly different syntax for its configuration file than the
+original BSD sources. Originally all messages of a specific priority and above
+were forwarded to the log file. The modifiers ``='', ``!'' and ``-'' were added
+to make rsyslogd more flexible and to use it in a more intuitive manner.<br>
+<br>
+The original BSD syslogd doesn't understand spaces as separators between the
+selector and the action field.<br>
+<br>
+When compared to syslogd from sysklogd package, rsyslogd offers additional
+<a href="features.html">features</a> (like template and database support). For obvious reasons, the syntax for
+defining such features is available
+in rsyslogd, only.<br>
+ </p>
+</body>
+</html>
\ No newline at end of file diff --git a/doc/rsyslog_mysql.html b/doc/rsyslog_mysql.html new file mode 100644 index 00000000..778b4257 --- /dev/null +++ b/doc/rsyslog_mysql.html @@ -0,0 +1,240 @@ +<html><head>
+<title>Writing syslog Data to MySQL</title>
+<meta name="KEYWORDS" content="syslog, mysql, syslog to mysql, howto">
+</head>
+<body>
+<h1>Writing syslog messages to MySQL</h1>
+ <P><small><i>Written by
+ <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer
+ Gerhards</a> (2005-08-02)</i></small></P>
+<h2>Abstract</h2>
+<p><i><b>In this paper, I describe how to write
+<a href="http://www.monitorware.com/en/topics/syslog/">syslog</a>
+messages to a <a href="http://www.mysql.com">MySQL</a> database.</b> Having
+syslog messages in a database is often handy, especially when you intend to set
+up a front-end for viewing them. This paper describes an approach
+with <a href="http://www.rsyslog.com/">rsyslogd</a>, an alternative enhanced
+syslog daemon natively supporting MySQL. I describe the components needed
+to be installed and how to configure them.</i></p>
+<h2>Background</h2>
+<p>In many cases, syslog data is simply written to text files. This approach has
+some advantages, most notably it is very fast and efficient. However, data
+stored in text files is not readily accessible for real-time viewing and analysis.
+To do that, the messages need to be in a database. There are various
+ways to store syslog messages in a database. For example, some have the syslogd
+write text files which are later feed via a separate script into the database.
+Others have written scripts taking the data (via a pipe) from a
+non-database-aware syslogd and store them as they appear. Some others use
+database-aware syslogds and make them write the data directly to the database.
+In this paper, I use that "direct write" approach. I think it is superior,
+because the syslogd itself knows the status of the database connection and thus
+can handle it intelligently (well ... hopefully ;)). I use rsyslogd to acomplish
+this, simply because I have initiated the rsyslog project with
+database-awareness as one goal.</p>
+<p><b>One word of caution:</b> while message storage in the database provides an
+excellent foundation for interactive analysis, it comes at a cost. Database i/o
+is considerably slower than text file i/o. As such, directly writing to
+the database makes sense only if your message volume is low enough to allow a)
+the syslogd, b) the network, and c) the database server to catch
+up with it. Some time ago, I have written a paper on
+<a href="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php">
+optimizing syslog server performance</a>. While this paper talks about
+Window-based solutions, the ideas in it are generic enough to apply here, too.
+So it might be worth reading if you anticipate medium high to high traffic. If you
+anticipate really high traffic (or very large traffic spikes), you should
+seriously consider forgetting about direct database writes - in my opinion, such
+a situation needs either a very specialised system or a different approach (the
+text-file-to-database approach might work better for you in this case).
+</p>
+<h2>Overall System Setup</h2>
+<P>In this paper, I concentrate on the server side. If you are thinking about
+interactive syslog message review, you probably want to centralize syslog. In
+such a scenario, you have multiple machines (the so-called clients) send their
+data to a central machine (called server in this context). While I expect such a
+setup to be typical when you are interested in storing messages in the database,
+I do not describe how to set it up. This is beyond the scope of this paper. If
+you search a little, you will probably find many good descriptions on how to
+centralize syslog. If you do that, it might be a good idea to do it securely, so
+you might also be interested in my paper on <a href="rsyslog_stunnel.html">
+ssl-encrypting syslog message transfer</a>.</P>
+<P>No matter how the messages arrive at the server, their processing is always the
+same. So you can use this paper in combination with any description for
+centralized syslog reporting.</P>
+<P>As I already said, I use rsyslogd on the server. It has intrinsic support for
+talking to MySQL databases. For obvious reasons, we also need an instance of
+MySQL running. To keep us focussed, the setup of MySQL itself is also beyond the
+scope of this paper. I assume that you have successfully installed MySQL and
+also have a front-end at hand to work with it (for example,
+<a href="http://www.phpmyadmin.net/">phpMyAdmin</a>). Please make sure that this
+is installed, actually working and you have a basic understanding of how to
+handle it.</P>
+<h2>Setting up the system</h2>
+<p>You need to download and install rsyslogd first. Obtain it from the
+<a href="http://www.rsyslog.com/">rsyslog site</a>. Make sure that you disable
+stock syslogd, otherwise you will experience some difficulties.</p>
+<p>It is important to understand how rsyslogd talks to the database. In rsyslogd,
+there is the concept of "templates". Basically, a template is a string that
+includes some replacement characters, which are called "properties" in rsyslog.
+Properties are accessed via the "<a href="property_replacer.html">Property
+Replacer</a>". Simply said, you access properties by including their name
+between percent signs inside the template. For example, if the syslog message is
+"Test", the template "%msg%" would be expanded to "Test". Rsyslogd supports
+sending template text as a SQL statement to MySQL. As such, the template must be
+a valid SQL statement. There is no limit in what the statement might be, but
+there are some obvious and not so obvious choices. For example, a template "drop
+table xxx" is possible, but does not make an awful lot of sense. In practice,
+you will always use an "insert" statment inside the template.</p>
+<p>An example: if you would just like to store the msg part of the full syslog
+message, you have probably created a table "syslog" with a single column "message".
+In such a case, a good template would be "insert into syslog(message) values ('%msg%')".
+With the example above, that would be expanded to "insert into syslog(message)
+values('Test')". This expanded string is then sent to the database. It's that
+easy, no special magic. The only thing you must ensure is that your template
+expands to a proper SQL statement and that this statement matches your database
+design.</p>
+<p>Does that mean you need to create database schema yourself and also must
+fully understand rsyslogd's properties? No, that's not needed. Because we
+anticipated that folks are probably more interested in getting things going instead
+of designing them from scratch. So we have provided a default schema as well
+as build-in support for it. This schema also offers an additional
+benefit: rsyslog is part of <a href="http://www.adiscon.com/en/">Adiscon</a>'s
+<a href="http://www.monitorware.com/en/">MonitorWare product line</a> (which
+includes open source and closed source members). All of these tools share the
+same default schema and know how to operate on it. For this reason, the default
+schema is also called the "MonitorWare Schema". If you use it, you can simply
+add <a href="http://www.phplogcon.org/">phpLogCon, a GPLed syslog web interface</a>,
+to your system and have instant interactive access to your database. So there
+are some benefits in using the provided schema.</p>
+<p>The schema definition is contained in the file "createDB.sql". It comes with
+the rsyslog package. Review it to check that the database name is acceptable for
+you. Be sure to leave the table and field names unmodified, because
+otherwise you need to customize rsyslogd's default sql template, which we do not
+do
+in this paper. Then, run the script with your favourite MySQL tool. Double-check
+that the table was successfully created.</p>
+<p>Next, we need to tell rsyslogd to write data to the database. As we use
+the default schema, we do NOT need to define a template for this. We can use the
+hardcoded one (rsyslogd handles the proper template linking). So all we need to
+do is add a simple selector line to /etc/rsyslog.conf:</p>
+<blockquote>
+ <p><code>*.*
+ >database-server,database-name,database-userid,database-password</code></p>
+</blockquote>
+<p>In many cases, MySQL will run on the local machine. In this case, you can
+simply use "127.0.0.1" for <i>database-server</i>. This can be especially
+advisable, if you do not need to expose MySQL to any process outside of the
+local machine. In this case, you can simply bind it to 127.0.0.1, which provides
+a quite secure setup. Of course, also supports remote MySQL instances. In that
+case, use the remote server name (e.g. mysql.example.com) or IP-address. The <i>
+database-name</i> by default is "syslog". If you have modified the default, use
+your name here. <i>Database-userid</i> and <i>-password</i> are the credentials
+used to connect to the database. As they are stored in clear text in
+rsyslog.conf, that user should have only the least possible privileges. It is
+sufficient to grant it INSERT privileges to the systemevents table, only. As a
+side note, it is strongly advisable to make the rsyslog.conf file readable by
+root only - if you make it world-readable, everybody could obtain the password
+(and eventually other vital information from it). In our example, let's assume
+you have created a MySQL user named "syslogwriter" with a password of
+"topsecret" (just to say it bluntly: such a password is NOT a good idea...). If
+your MySQL database is on the local machine, your rsyslog.conf line might look
+like in this sample:</p>
+<blockquote>
+ <p><code>*.*
+ >127.0.0.1,syslog,syslogwriter,topsecret</code></p>
+</blockquote>
+<p>Save rsyslog.conf, restart rsyslogd - and you should see syslog messages
+being stored in the "systemevents" table!</p>
+<p>The example line stores every message to the database. Especially if you have
+a high traffic volume, you will probably limit the amount of messages being
+logged. This is easy to acomplish: the "write database" action is just a regular
+selector line. As such, you can apply normal selector-line filtering. If, for
+example, you are only interested in messages from the mail subsystem, you can
+use the following selector line:</p>
+<blockquote>
+ <p><code>mail.*
+ >127.0.0.1,syslog,syslogwriter,topsecret</code></p>
+</blockquote>
+<p>Review the <a href="rsyslog_conf.html">rsyslog.conf</a> documentation for
+details on selector lines and their filtering.</p>
+<p><b>You have now completed everything necessary to store syslog messages to
+the MySQL database.</b> If you would like to try out a front-end, you might want
+to look at <a href="http://www.phplogcon.org/">phpLogCon</a>, which displays
+syslog data in a browser. As of this writing, phpLogCon is not yet a powerful
+tool, but it's open source, so it might be a starting point for your own
+solution.</p>
+<h2>On Reliability...</h2>
+<p>Rsyslogd writes syslog messages directly to the database. This implies that
+the database must be available at the time of message arrival. If the database
+is offline, no space is left or something else goes wrong - rsyslogd can not
+write the database record. If rsyslogd is unable to store a message, it performs
+one retry. This is helpful if the database server was restarted. In this case,
+the previous connection was broken but a reconnect immediately succeeds. However,
+if the database is down for an extended period of time, an immediate retry does
+not help. While rsyslogd could retry until it finally succeeds, that would have
+negative impact. Syslog messages keep coming in. If rsyslogd would be busy
+retrying the database, it would not be able to process these messages.
+Ultimately, this would lead to loss of newly arrived messages.</p>
+<p>In most cases, rsyslogd is configured not only to write to the database but
+to perform other actions as well. In the always-retry scenario, that would mean
+no other actions would be carried out. As such, the design of rsyslogd is
+limited to a single retry. If that does not succeed, the current message is will
+not be written to the database and the MySQL database writer be suspended for a
+short period of time. Obviously, this leads to the loss of the current message
+as well as all messages received during the suspension period. But they are only
+lost in regard to the database, all other actions are correctly carried out.
+While not perfect, we consider this to be a better approach then the potential
+loss of all messages in all actions.</p>
+<p><b>In short: try to avoid database downtime if you do not want to experience
+message loss.</b></p>
+<p>Please note that this restriction is not rsyslogd specific. All approachs to
+real-time database storage share this problem area.</p>
+<h2>Conclusion</h2>
+<P>With minumal effort, you can use rsyslogd to write syslog messages to a MySQL
+database. Once the messages are arrived there, you can interactivley review and
+analyse them. In practice, the messages are also stored in text files for
+longer-term archival and the databases are cleared out after some time (to avoid
+becoming too slow). If you expect an extremely high syslog message volume,
+storing it in real-time to the database may outperform your database server. In
+such cases, either filter out some messages or think about alternate approaches
+involving non-real-time database writing (beyond the scope of this paper).</P>
+<P>The method outline in this paper provides an easy to setup and maintain
+solution for most use cases, especially with low and medium syslog message
+volume (or fast database servers).</P>
+<P>I have set up a site to <a href="http://demo.rsyslog.com/">demo web access to
+syslog data</a>. It is build using the steps outlined here and uses phpLogCon as
+the front-end. You might want to <a href="http://demo.rsyslog.com/">visit</a> it
+to get a glimpse of how such a beast might look.</P>
+<h3>Feedback Requested</h3>
+<P>I would appreciate feedback on this paper. If you have additional ideas,
+comments or find bugs, please
+<a href="mailto:rgerhards@adiscon.com">let me know</a>.</P>
+<h2>References and Additional Material</h2>
+<ul>
+ <li><a href="http://www.rsyslog.com">www.rsyslog.com</a> - the rsyslog site</li>
+ <li><a href="http://demo.rsyslog.com/">demo.rsyslog.com</a> - demo
+ site for rsyslog and the phpLogCon web interface</li>
+ <li>
+ <a href="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php">
+ Paper on Syslog Server Optimization</a></li>
+</ul>
+<h2>Revision History</h2>
+<ul>
+ <li>2005-08-02 *
+ <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> *
+ initial version created</li>
+ <li>2005-08-03 *
+ <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a>
+ * added references to demo site</li>
+</ul>
+<h2>Copyright</h2>
+<p>Copyright (c) 2005
+<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a>
+and <a href="http://www.adiscon.com/en/">Adiscon</a>.</p>
+<p>Permission is granted to copy, distribute and/or modify this document under
+the terms of the GNU Free Documentation License, Version 1.2 or any later
+version published by the Free Software Foundation; with no Invariant Sections,
+no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be
+viewed at <a href="http://www.gnu.org/copyleft/fdl.html">
+http://www.gnu.org/copyleft/fdl.html</a>.</p>
+</body>
+</html>
\ No newline at end of file diff --git a/doc/rsyslog_stunnel.html b/doc/rsyslog_stunnel.html new file mode 100644 index 00000000..71bb878d --- /dev/null +++ b/doc/rsyslog_stunnel.html @@ -0,0 +1,248 @@ +<html><head>
+<title>SSL Encrypting syslog with stunnel</title>
+<meta name="KEYWORDS" content="syslog encryption, rsyslog, stunnel, secure syslog, tcp, reliable, howto, ssl">
+</head>
+<body>
+<h1>SSL Encrypting Syslog with Stunnel</h1>
+ <P><small><i>Written by
+ <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer
+ Gerhards</a> (2005-07-22)</i></small></P>
+<h2>Abstract</h2>
+<p><i><b>In this paper, I describe how to encrypt <a href="http://www.monitorware.com/en/topics/syslog/">syslog</a>
+messages on the network.</b> Encryption
+is vital to keep the confidiental content of syslog messages secure. I describe the overall
+approach and provide an HOWTO do it with the help of
+<a href="http://www.rsyslog.com">rsyslogd</a> and <a href="http://www.stunnel.org">stunnel</a>.</i></p>
+<h2>Background</h2>
+<P><b>Syslog is a
+clear-text protocol. That means anyone with a sniffer can have
+a peek at your data.</b> In some environments, this is no problem at all. In
+others, it is a huge setback, probably even preventing deployment of syslog
+solutions. Thankfully, there is an easy way to encrypt syslog communication. I
+will describe one approach in this paper.</P>
+<P>The most straigthforward solution would be that the syslogd itself encrypts
+messages. Unfortuantely, encryption is only standardized in
+<a href="http://www.monitorware.com/Common/en/glossary/rfc3195.php">RFC 3195</a>. But there
+is currently no syslogd that implements RFC 3195's encryption features,
+so this route leads to nothing. Another approach would be to use vendor- or
+project-specific syslog extensions. There are a few around, but the problem here
+is that they have compatibility issues. However, there is one surprisingly easy
+and interoperable solution: though not standardized, many vendors and projects
+implement plain tcp syslog. In a nutshell, plain tcp syslog is a mode where
+standard syslog messages are transmitted via tcp and records are separated by
+newline characters. This mode is supported by all major syslogd's (both on Linux/Unix
+and Windows) as well as log sources (for example,
+<a href="http://www.eventreporter.com/en/">EventReporter</a> for Windows
+Event Log forwarding). Plain tcp syslog offers reliability, but it does not
+offer encryption in itself. However, since it operates on a tcp stream, it is now easy
+to add encryption. There are various ways to do that. In this paper, I will
+describe how it is done with stunnel (an
+other alternative would be <a href="http://en.wikipedia.org/wiki/IPSec">IPSec</a>, for example).</P>
+<P>Stunnel is open source and it is available both for Unix/Linux and Windows.
+It provides a way to
+ use ssl communication for any non-ssl aware client and server - in this case,
+ our syslogd.</P>
+ <P>Stunnel works much like a wrapper. Both on the client and on the server machine,
+ tunnel portals are created. The non-ssl aware client and server software is
+ configured to not directly talk to the remote partner, but to the local
+ (s)tunnel portal instead. Stunnel, in turn, takes the data received from the
+ client, encrypts it via ssl, sends it to the remote tunnel portal and that
+ remote portal sends it to the recipient process on the remote machine. The
+ transfer to the portals is done via unencrypted communication. As such,
+ it is vital that
+ the portal and the respective program that is talking to it are on the same
+ machine, otherwise data would travel partly unencrypted. Tunneling, as done by stunnel,
+ requires connection oriented communication. This is why you need to use
+ tcp-based syslog. As a side-note, you can also encrypt a plain-text RFC
+ 3195 session via stunnel, though this definitely is not what the
+ protocol designers had on their mind ;)</P>
+<P>In the rest of this document, I assume that you use rsyslog on both the
+client and the server. For the samples, I use <a href="http://www.debian.org/">Debian</a>.
+Interestingly, there are
+some annoying differences between stunnel implementations. For example, on
+Debian a comment line starts with a semicolon (';'). On
+<a href="http://www.redhat.com">Red Hat</a>, it starts with
+a hash sign ('#'). So you need to watch out for subtle issues when setting up
+your system.</P>
+<h2>Overall System Setup</h2>
+<P>In ths paper, I assume two machines, one named "client" and the other named "server".
+It is obvious that, in practice, you will probably have multiple clients but
+only one server. Syslog traffic shall be transmitted via stunnel over the
+network. Port 60514 is to be used for that purpose. The machines are set up as
+follows:</P>
+<P><b>Client</b></P>
+<ul>
+ <li>rsyslog forwards message to stunnel local portal at port 61514</li>
+ <li>local stunnel forwards data via the network to port 60514 to its remote
+ peer</li>
+</ul>
+<P><b>Server</b></P>
+<ul>
+ <li>stunnel listens on port 60514 to connections from its client peers</li>
+ <li>all connections are forwarded to the locally-running rsyslog listening
+ at port 61514</li>
+</ul>
+<h2>Setting up the system</h2>
+<P>For Debian, you need the "stunnel4" package. The "stunnel" package is the
+older 3.x release, which will not support the configuration I describe below.
+Other distributions might have other names. For example, on Red Hat it is just "stunnel".
+Make sure that you install the appropriate package on both the client and the
+server. It is also a good idea to check if there are updates for either stunnel
+or openssl (which stunnel uses) - there are often security fixes available and
+often the latest fixes are not included in the default package.</P>
+<P>In my sample setup, I use only the bare minimum of options. For example, I do
+not make the server check client cerficiates. Also, I do not talk much about
+certificates at all. If you intend to really secure your system, you should
+probably learn about certificates and how to manage and deploy them. This is
+beyond the scope of this paper. For additional information,
+<a href="http://www.stunnel.org/faq/certs.html">
+http://www.stunnel.org/faq/certs.html</a> is a good starting point.</P>
+<P>You also need to install rsyslogd on both machines. Do this before starting
+with the configuration. You should also familarize yourself with its
+configuration file syntax, so that you know which actions you can trigger with
+it. Rsyslogd can work as a drop-in replacement for stock
+<a href="http://www.infodrom.org/projects/sysklogd/">sysklogd</a>. So if you know
+the standard syslog.conf syntax, you do not need to learn any more to follow
+this paper.</P>
+<h3>Server Setup</h3>
+<P>At the server, you need to have a digital certificate. That certificate
+enables SSL operation, as it provides the necessary crypto keys being used to
+secure the connection. Many versions of stunnel come with a default certificate,
+often found in /etc/stunnel/stunnel.pem. If you have it, it is good for testing
+only. If you use it in production, it is very easy to break into your secure
+channel as everybody is able to get hold of your private key. I didn't find an
+stunnel.pem on my Debian machine. I guess the Debian folks removed it because of
+its insecurity.</P>
+<P>You can create your own certificate with a simple openssl tool - you need to
+do it if you have none and I highly recommend to create one in any case. To
+create it, cd to /etc/stunnel and type:</P>
+<p><blockquote><code>openssl req -new -x509 -days 3650 -nodes -out
+stunnel.pem -keyout stunnel.pem</code></blockquote></p>
+<P>That command will ask you a number of questions. Provide some answer for
+them. If you are unsure, read
+<a href="http://www.stunnel.org/faq/certs.html">
+http://www.stunnel.org/faq/certs.html</a>. After the command has finished, you
+should have a usable stunnel.pem in your working directory.</P>
+<P>Next is to create a configuration file for stunnel. It will direct stunnel
+what to do. You can used the following basic file:</P>
+<P><blockquote><code><pre>; Certificate/key is needed in server mode
+cert = /etc/stunnel/stunnel.pem
+
+<i>; Some debugging stuff useful for troubleshooting
+debug = 7
+foreground=yes</i>
+
+[ssyslog]
+accept = 60514
+connect = 61514</pre>
+</code></blockquote></P>
+<p>Save this file to e.g. /etc/stunnel/syslog-server.conf. Please note that the
+settings in <i>italics</i> are for debugging only. They run stunnel
+with a lot of debug information in the foreground. This is very valuable while
+you setup the system - and very useless once everything works well. So be sure
+to remove these lines when going to production.</p>
+<p>Finally, you need to start the stunnel daemon. Under Debian, this is done via
+"stunnel /etc/stunnel/syslog.server.conf". If you have enabled the debug
+settings, you will immediately see a lot of nice messages.</p>
+<p>Now you have stunnel running, but it obviously unable to talk to rsyslog -
+because it is not yet running. If not already done, configure it so that it does
+everything you want. If in doubt, you can simply copy /etc/syslog.conf to /etc/rsyslog.conf
+and you probably have what you want. The really important thing in rsyslogd
+configuration is that you must make it listen to tcp port 61514 (remember: this
+is where stunnel send the messages to). Thankfully, this is easy to achive: just
+add "-t 61514" to the rsyslogd startup options in your system startup script.
+After done so, start (or restart) rsyslogd.</p>
+<p>The server should now be fully operational.</p>
+<h3>Client Setup</h3>
+<P>The client setup is simpler. Most importantly, you do not need a certificate
+(of course, you can use one if you would like to authenticate the client, but
+this is beyond the scope of this paper). So the basic thing you need to do is
+create the stunnel configuration file.</P>
+<P><blockquote><code><pre><i>; Some debugging stuff useful for troubleshooting
+debug = 7
+foreground=yes</i>
+
+<b>client=yes</b>
+
+[ssyslog]
+accept = 127.0.0.1:61514
+connect = <font color="#FF0000">192.0.2.1</font>:60514
+</pre>
+</code></blockquote></P>
+<P>Again, the text in <i>italics</i> is for debugging purposes only. I suggest
+you leave it in during your initial testing and then remove it. The most
+important difference to the server configuration outlined above is the "client=yes"
+directive. It is what makes this stunnel behave like a client. The accept
+directive binds stunnel only to the local host, so that it is protected from
+receiving messages from the network (somebody might fake to be the local sender).
+The address "192.0.2.1" is the address of the server machine. You must change it
+to match your configuration. Save this file to /etc/stunnel/syslog-client.conf.</P>
+<P>Then, start stunnel via "stunnel4 /etc/stunnel/syslog-client.conf". Now
+you should see some startup messages. If no errors appear, you have a running
+client stunnel instance.</P>
+<P>Finally, you need to tell rsyslogd to send data to the remote host. In stock
+syslogd, you do this via the "@host" forwarding directive. The same works with
+rsyslog, but it suppports extensions to use tcp. Add the following line to your
+/etc/rsyslog.conf:</P>
+<P><blockquote><code><pre>*.* @<font color="#FF0000">@</font>127.0.0.1:61514
+</pre>
+</code></blockquote><i></P>
+
+</i>
+
+<P>Please note the double at-sign (@@). This is no typo. It tells rsyslog to use
+tcp instead of udp delivery. In this sample, all messages are forwarded to the
+remote host. Obviously, you may want to limit this via the usual rsyslog.conf
+settings (if in doubt, use man rsyslog.con).</P>
+<P>You do not need to add any special startup settings to rsyslog on the client.
+Start or restart rsyslog so that the new configuration setting takes place.</P>
+<h3>Done</h3>
+<P>After following these steps, you should have a working secure syslog
+forwarding system. To verify, you can type "logger test" or a similar smart
+command on the client. It should show up in the respective server log file. If
+you dig out you sniffer, you should see that the traffic on the wire is actually
+protected. In the configuration use above, the two stunnel endpoints should be
+quite chatty, so that you can follow the action going on on your system.</P>
+<P>If you have only basic security needs, you can probably just remove the debug
+settings and take the rest of the configuration to production. If you are
+security-sensitve, you should have a look at the various stunnel settings that
+help you further secure the system.</P>
+<h2>Preventing Systems from talking directly to the rsyslog Server</h2>
+<P>It is possible that remote systems (or attackers) talk to the rsyslog server
+by directly connecting to its port 61514. Currently (Jule of 2005), rsyslog does
+not offer the ability to bind to the local host, only. This feature is planned,
+but as long as it is missing, rsyslog must be protected via a firewall. This can
+easily be done via e.g iptables. Just be sure not to forget it.</P>
+<h2>Conclusion</h2>
+<P>With minumal effort, you can set up a secure logging infrastructure employing
+ssl encrypted syslog message transmission. As a side note, you also have the
+benefit of reliable tcp delivery which is far less prone to message loss than
+udp.</P>
+<h3>Feedback requested</h3>
+<P>I would appreciate feedback on this tutorial. If you have additional ideas,
+comments or find bugs (I *do* bugs - no way... ;)), please
+<a href="mailto:rgerhards@adiscon.com">let me know</a>.</P>
+<h2>Revision History</h2>
+<ul>
+ <li>2005-07-22 *
+ <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> * Initial Version created</li>
+ <li>2005-07-26 *
+ <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> * Some text brush-up, hyperlinks added</li>
+ <li>2005-08-03 *
+ <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a>
+ * license added</li>
+</ul>
+<h2>Copyright</h2>
+<p>Copyright (c) 2005
+<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> and
+<a href="http://www.adiscon.com/en/">Adiscon</a>.</p>
+<p> Permission is granted to copy, distribute and/or modify this document
+ under the terms of the GNU Free Documentation License, Version 1.2
+ or any later version published by the Free Software Foundation;
+ with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
+ Texts. A copy of the license can be viewed at
+<a href="http://www.gnu.org/copyleft/fdl.html">
+http://www.gnu.org/copyleft/fdl.html</a>.</p>
+
+</body>
+</html>
\ No newline at end of file diff --git a/doc/status.html b/doc/status.html new file mode 100644 index 00000000..f1217fc3 --- /dev/null +++ b/doc/status.html @@ -0,0 +1,34 @@ +<html>
+<head>
+<title>rsyslog status page</title>
+</head>
+<body>
+<h2>rsyslog status page</h2>
+<p>This page reflects the status as of 2005-08-02.</p>
+<h2>Current Release</h2>
+<p>0.9.5 - <a href="http://www.rsyslog.com/Article20.phtml">change log</a> -
+<a href="http://www.rsyslog.com/Downloads-index-req-getit-lid-11.phtml">download</a></p>
+<h2>Platforms</h2>
+<ul>
+ <li>Linux [tested on Red Hat & Debian]</li>
+ <li>BSD [tested on
+FreeBSD and a quick Test on NetBSD, 0.8.4 only]</li>
+</ul>
+<h2>Additional information</h2>
+<p><b>Currently supported features are now listed on the <a href="features.html">rsyslog features page</a>.</b></p>
+<ul>
+ <li>The rsyslog home page is <a href="http://www.rsyslog.com">www.rsyslog.com</a>.</li>
+ <li>Mailing list info can be found at
+ <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">http://lists.adiscon.com/rsyslog</a>.</li>
+ <li>The change log can be found at
+ <a href="http://www.rsyslog.com/Topic4.phtml">
+ http://www.rsyslog.com/Topic4.phtml</a>. </li>
+ <li>You may also find Rainer's <a href="http://rgerhards.blogspot.com/">
+ syslog blog</a> an interesting read.</li>
+</ul>
+<p>The project was initiated in 2004 by
+<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a>
+and is currently being maintained by him. See the <a href="history.html">history
+page</a> for more background information.</p>
+</body>
+</html>
\ No newline at end of file |