From 4568079aab0fa718b3cd0f7b0e23467de40ac3ca Mon Sep 17 00:00:00 2001 From: Rainer Gerhards Date: Thu, 4 Aug 2005 10:27:48 +0000 Subject: begun to move the documentation to html (and create better doc) --- BUGS | 59 +----- doc/bugs.html | 63 +++++++ doc/features.html | 42 +++++ doc/history.html | 72 +++++++ doc/manual.html | 48 +++++ doc/property_replacer.html | 64 +++++++ doc/rsyslog_conf.html | 460 +++++++++++++++++++++++++++++++++++++++++++++ doc/rsyslog_mysql.html | 240 +++++++++++++++++++++++ doc/rsyslog_stunnel.html | 248 ++++++++++++++++++++++++ doc/status.html | 34 ++++ 10 files changed, 1272 insertions(+), 58 deletions(-) create mode 100644 doc/bugs.html create mode 100644 doc/features.html create mode 100644 doc/history.html create mode 100644 doc/manual.html create mode 100644 doc/property_replacer.html create mode 100644 doc/rsyslog_conf.html create mode 100644 doc/rsyslog_mysql.html create mode 100644 doc/rsyslog_stunnel.html create mode 100644 doc/status.html diff --git a/BUGS b/BUGS index 00890c43..7f537a29 100644 --- a/BUGS +++ b/BUGS @@ -1,58 +1 @@ -This file contains know bugs and incompatibilities - -REPEATED LOG LINES - If multiple log lines with the exact same content are received, - the duplicates are NOT suppressed. This is done by sysklogd - in all cases. We plan to add this as an optional feature, - but as of now every line is logged. - -ERRORS ON STARTUP - Error reporting on startup is not particular good. We handle all - errors correctly, but sometimes a message is only reported if - running in debug mode. If you experience some strange behaviour - I suggest to run the syslogd interactively with debugging enabled. - As of August, 2005, the situation has much improved. Keep in mind, - thought, that debug mode is your friend ;) - -EQUALLY-NAMED TEMPLATES - If multiple templates with the SAME name are created, all but the - first definition is IGNORED. So you can NOT (yet) replace a - template definition. - -MEMORY LEAK ON HUPING - Currently, the template memory is NOT freed when SIGHUP is received. - This causes a small memory leak on each HUP. Will be fixed soon and - should not cause real trouble unless you HUP syslogd quite frequently. - -WALLMSG FORMAT (* selector) - This format is actually not 100% compatible with stock syslogd - the - date is missing. Will be fixed soon and can also be fixed just via - the proper template. Anyone up for this? ;) - -CALLS to exit() - I have removed all calls to exit() I have introduced. However, there are - still some left from the original code. Most of them are OK, being done - on initial startup when a severe error happens. There are some, I think, - that might happen during runtime when a memory shortag is detected. We - should look into this issues. Finding some smarter code would allow - rsyslog to become even more fault-tolerant (but obviously, there is always - a cost, in this case most probably the loss of at least on message). - -SIGPIPE HANDLING - Currently, SIGPIPE is ignored. This is necessary to handle broken TCP - connections. We should further look into this issue and see which other - ways exist to handle the situation. - -MULTIPLE INSTANCES - If multiple instances are running on a single machine, the one with - the -r switch must start first. Also, UDP-based syslog forwarding - between the instances does not work. Use TCP instead. - 2005-08-01 - -Updates to the bug lists (and potential fixes...) can be found at the -official home page at - -http://www.monitorware.com/rsyslog - -List as of 2005-03-08 -Rainer Gerhards +This file has been superseeded by doc/bugs.html. Check there. diff --git a/doc/bugs.html b/doc/bugs.html new file mode 100644 index 00000000..ae4fb4fd --- /dev/null +++ b/doc/bugs.html @@ -0,0 +1,63 @@ + + +rsyslog bugs and annoyances + + +

rsyslog bugs and annoyances

This page lists the known bugs rsyslog has to offer. Please note that +we also have a + +bug tracker at sourceforge.net. This list here contains more architectural +things while the bug tracker most often lists things that you will actually +experience. I am working to combine the two lists, but for now you need to visit +both.

This list has last been updated on 2005-08-02 by +Rainer Gerhards.

rsyslogd

REPEATED LOG LINES +

If multiple log lines with the exact same content are received, + the duplicates are NOT suppressed. This is done by sysklogd + in all cases. We plan to add this as an optional feature, + but as of now every line is logged.

ERRORS ON STARTUP

Error reporting on startup is not particular good. We handle all + errors correctly, but sometimes a message is only reported if + running in debug mode. If you experience some strange behaviour + I suggest to run the syslogd interactively with debugging enabled. + As of August, 2005, the situation has much improved. Keep in mind, + thought, that debug mode is your friend ;)

EQUALLY-NAMED TEMPLATES

If multiple templates with the SAME name are created, all but the + first definition is IGNORED. So you can NOT (yet) replace a + template definition. I also strongly doubt I will ever support this, because +it does not make an awful lot of sense (after all, why not use two template +names...).

MEMORY LEAK ON HUPING

Currently, the template memory is NOT freed when SIGHUP is received. + This causes a small memory leak on each HUP. Will be fixed soon and + should not cause real trouble unless you HUP syslogd quite frequently.

WALLMSG FORMAT (* selector) +

This format is actually not 100% compatible with stock syslogd - the + date is missing. Will be fixed soon and can also be fixed just via + the proper template. Anyone up for this? ;)

CALLS to exit()

I have removed all calls to exit() I have introduced. However, there are + still some left from the original code. Most of them are OK, being done + on initial startup when a severe error happens. There are some, I think, + that might happen during runtime when a memory shortage is detected. We + should look into this issues. Finding some smarter code would allow + rsyslog to become even more fault-tolerant (but obviously, there is always + a cost, in this case most probably the loss of at least one message).

SIGPIPE HANDLING

Currently, SIGPIPE is ignored. This is necessary to handle broken TCP + connections. We should further look into this issue and see which other + ways exist to handle the situation.

MULTIPLE INSTANCES

If multiple instances are running on a single machine, the one with + the -r switch must start first. Also, UDP-based syslog forwarding between the +instances does not work. Use TCP instead.

+ + + \ No newline at end of file diff --git a/doc/features.html b/doc/features.html new file mode 100644 index 00000000..0b41afa0 --- /dev/null +++ b/doc/features.html @@ -0,0 +1,42 @@ + + +rsyslog features + + +

RSyslog - Features

This page lists both current features as well as those being considered +for future versions of rsyslog. If you think a feature is missing, drop +Rainer a note. Rsyslog is a vital +project. Features are added each few days. If you would like to keep up of what +is going on, you can also subscribe to the rsyslog mailing list. +

Current Features

native support for writing to MySQL databases
support for (plain) tcp + based syslog - much better reliability
control of log output format, + including ability to present channel and priority as visible log data
good timestamp format control; at a minimum, ISO 8601/RFC 3339 + second-resolution UTC zone
ability to reformat message contents and work with substrings
support for + log files larger than 2gb
support for file size limitation and automatic + rollover command execution
support for running multiple rsyslogd + instances on a single machine
support for + ssl-protected syslog (via stunnel)

Upcoming Features

support for + + syslog-protocol-14 compliant messages +
support for native SSL enryption of plain tcp syslog sessions
control of whether the local hostname or the hostname of the + origin of the data is shown as the hostname in the output - prepared + for, but not yet implemented (should be easy) +
support for syslog-protocol - planned +
RFC 3195 support - planned (but late) +
pcre filtering - maybe (depending on feedback) +
multi-threaded redesign - maybe (depending on feedback; also + opens Pandorra's cross-platform development box ;)) +

To see when each feature was added, see the +rsyslog change log (online +only).

+ + \ No newline at end of file diff --git a/doc/history.html b/doc/history.html new file mode 100644 index 00000000..de427ce5 --- /dev/null +++ b/doc/history.html @@ -0,0 +1,72 @@ + + +rsyslog history + + +

RSyslog - History

+ +Rsyslog is a GPL-ed, enhanced syslogd. Among others, it offers support for +reliable syslog over TCP, writing to +MySQL databases and fully configurable output formats (including great timestamps). +Rsyslog was initiated by Rainer Gerhards. It has +been forked from the sysklogd standard package. +The goal of the +rsyslog project is to provide a more configurable and reliable +syslog deamon while retaining drop-in replacement capabilities for stock syslogd. By "reliable", we mean support for reliable transmission +modes like TCP or RFC 3195 (syslog-reliable). +We do NOT imply that the sysklogd package is unreliable. In fact, the +opposite is the case and we assume that for the time being the well-used +sysklogd package offers better program reliability than our +brand-new modifications to it. + +

The name "rsyslog" stems back to the +planned support for syslog-reliable. Ironically, the initial release +of rsyslog did NEITHER support syslog-reliable NOR tcp based syslog. +Instead, it contains enhanced configurability and other enhancements +(like database support). The reason for this is that full support for +RFC 3195 would require even more changes and especially fundamental architectural +changes. Also, questions asked on the loganalysis list and at other +places indicated that RFC3195 is NOT a prime priority for users, but +rather better control over the output format. So here we are, with +a rsyslod that covers a lot of enhancements, but not a single one +of these that made its name ;) Since version 0.9.2, receiving syslog messages +via plain tcp is finally supported, bringing rsyslogd a bit closer to its name. +

+The next enhancement scheduled is support for the new syslog-protocol +internet draft format, not the least to see how easy/complicated it is +to implement. We already know that some subleties of syslog-protocol will +require at least one considerable architectural change to the syslogd +and this might delay things a little. Our immediate goal is to receive +feedback and get the bugs out of the current release. Only after that +we intend to advance the code and introduce new features. +

+The database support was included so that our web-based syslog interface +can be used. This is another open source project which can be found +under http://www.liblogging.org (this +site needs serious redesign!). We highly recommend having a look at +it. It might not work for you if you expect thousands of messages per +second (because your database won't be able to provide adequate performance), +but in many cases it is a very handy analysis and troubleshooting tool. + +

Rsyslogd supports an enhanced syslog.conf file format, and also works +with the standard syslog.conf. In theory, it should be possible to simply replace +the syslogd binary with the one that comes with rsyslog. Of course, in order +to use any of the new features, you must re-write your syslog.conf. To learn +how to do this, please review our commented sample.conf +file. It outlines the enhancements over stock syslogd. +

If you are interested in the IHE +environment, you might be interested to hear that rsyslog supports message with +sizes of 32k and more. This feature has been tested, but by default is turned off +(as it has some memory footprint that we didn't want to put on users not +actually requiring it). Search the file syslogd.c and search for "IHE" - you +will find easy and precise instructions on what you need to change (it's just +one line of code!). +

Be sure to visit Rainer's syslog block +to get some more insight into the development of rsyslog and syslog in general.

Some useful links

the rsyslog change log

+ + \ No newline at end of file diff --git a/doc/manual.html b/doc/manual.html new file mode 100644 index 00000000..48863933 --- /dev/null +++ b/doc/manual.html @@ -0,0 +1,48 @@ + + +rsyslog documentation + + +

RSyslog - Documentation

Rsyslog is an enhanced syslogd +supporting, among others, MySQL, syslog/tcp +and fine grain output format control. It is quite compatible to stock +sysklogd and can be used as a drop-in replacement. Its +advanced features make it suitable for enterprise-class, +encryption protected syslog +relay chains while at the same time being very easy to setup +for the novice user.

Visit the rsyslog status page to obtain current +version information and ports.

Follow the links below for the

rsyslogd man page +
configuration file syntax (rsyslog.conf) +
a commented sample rsyslog.conf +
rsyslog bug list

We have some in-depth papers on

Also, there is an article from Dennis Olvany on +Syslog-to-SQL with rsyslog-0.8.4 on FreeBSD 5.4 +(which unfortunately is a bit outdated now).

Our rsyslog history page is for you if you would like to learn a little more +on why there is an rsyslog at all.

Documentation is added continously. Please note that the documentation here +matches only the current version of rsyslog. If you use an older version, be sure +to use the doc that came with it.

You can also browse the following online ressources:

And don't forget about the rsyslog mailing list. +If you are interested in the "backstage", you may find +Rainer's +syslog blog an interesting read.

+ + \ No newline at end of file diff --git a/doc/property_replacer.html b/doc/property_replacer.html new file mode 100644 index 00000000..6d90fa45 --- /dev/null +++ b/doc/property_replacer.html @@ -0,0 +1,64 @@ + + +The Rsyslogd Property Replacer + + +

The Property Replacer

The property replacer is a core component in rsyslogd's output system. +A syslog message has a number of well-defined properties (see below). Each of +this properties can be accessed and manipulated by the property replacer. +With it, it is easy to use only part of a property value or manipulate the value, +e.g. by converting all characters to lower case.

Accessing Properties

Syslog message properties are used inside templates. They are accessed by putting them between percent signs. Properties can be modified by +the property replacer. The full syntax is as follows:

%propname:fromChar:toChar:options%

Available Properties

propname is the name of the property to access. It is case-sensitive. +Currently supported are:

+ + + + + + + + + + + + + + +

msg	the MSG part of the message (aka "the message" ;))
rawmsg	the message excactly as it was received from the +socket. Should be useful for debugging.
UxTradMsg	will disappear soon - do NOT use!
HOSTNAME	hostname from the message
source	alias for HOSTNAME
syslogtag	TAG from the message
PRI	PRI part of the message - undecoded (single value)
IUT	the monitorware InfoUnitType - used when talking +to a MonitorWare backend (also for + phpLogCon)
syslogfacility	the facility from the message - in numerical form
syslogpriority	the priority (actully severity!) from the + message - in numerical form
timegenerated	timestamp when the message was RECEIVED. Always in + high resolution
timereported	timestamp from the message. Resolution depends on +what was provided in the message (in most cases, +only seconds)
TIMESTAMP	alias for timereported

Character Positions

FromChar and toChar are used to build substrings. They specify the offset within +the string that should be copied. Offset counting starts at 1, so if you need to +obtain the first 2 characters of the message text, you can use this syntax: +"%msg:1:2%". If you do not whish to specify from and to, but you want to specify +options, you still need to include the colons. For example, if you would like to +convert the full message text to lower case, use "%msg:::lowercase%".
+
+

Property Options

+property options are case-insensitive. Currently, the following options +are defined:

+ + + + + + + + +

uppercase	convert property to lowercase only
lowercase	convert property text to uppercase only
drop-last-lf	The last LF in the message (if any), is dropped. + Especially useful for PIX.
date-mysql	format as mysql date
date-rfc3164	format as RFC 3164 date
date-rfc3339	format as RFC 3339 date
escape-cc	NOT yet implemented

+ + + \ No newline at end of file diff --git a/doc/rsyslog_conf.html b/doc/rsyslog_conf.html new file mode 100644 index 00000000..bbf97198 --- /dev/null +++ b/doc/rsyslog_conf.html @@ -0,0 +1,460 @@ + + +rsyslog.conf file + + +

rsyslog.conf configuration file

This document is currently being enhanced. Please pardon its current +appearance.

Rsyslogd is configured via the rsyslog.conf file, typically found in +/etc. By default, rsyslog reads the file /etc/rsyslog.conf.

While rsyslogd contains enhancements over standard syslogd, efforts have been +made to keep the configuration file as compatible as possible. While, for +obvious reasons, enhanced features require a +different config file syntax, rsyslogd should be able to work with a standard +syslog.conf file. This is especially useful while you are migrating from syslogd +to rsyslogd.

Basic Structure

Every rule consists of two fields, a selector field and an action field. +These two fields are separated by one or more spaces or tabs. The selector field +specifies a pattern of facilities and priorities belonging to the specified +action.
+
+Lines starting with a hash mark (``#'') and empty lines are ignored. +

Templates

Templates are a key feature of rsyslog. They allow to specify any format a user +might want. Every output in rsyslog uses templates - this holds true for files, +user messages and so on. The database writer expects its template to be a proper +SQL statement - so this is highly customizable too. You might ask how does all +of this work when no templates at all are specified. Good question ;) The answer +is simple, though. Templates compatible with the stock syslogd formats are +hardcoded into rsyslogd. So if no template is specified, we use one of these +hardcoded templates. Search for "template_" in syslogd.c and you will find the +hardcoded ones.

A template consists of a template directive, a name, the actual template text +and optional options. A sample is:

$template MyTemplateName,"\7Text %property% some more text\n",<options>

The "$template" is the template directive. It tells rsyslog that this line +contains a template. "MyTemplateName" is the template name. All +other config lines refer to this name. The text within quotes is the actual +template text. The backslash is an escape character, much as it is in C. It does +all these "cool" things. For example, \7 rings the bell (this is an ASCII +value), \n is a new line. C programmers and perl coders have the advantage of +knowing this, but the set in rsyslog is a bit restricted currently. +

+All text in the template is used literally, except for things within percent +signs. These are properties and allow you access to the contents of the syslog +message. Properties are accessed via the property replacer (nice name, huh) and +it can do cool things, too. For example, it can pick a substring or do +date-specific formatting. More on this is below, on some lines of the property +replacer.
+
+The <options> part is optional. It carries options influencing the template as +whole. See details below. Be sure NOT to mistake template options with property +options - the later ones are processed by the property replacer and apply to a +SINGLE property, only (and not the whole template).
+
+Template options are case-insensitive. Currently defined are:

sql - format the string suitable for a SQL statement. This will +replace single quotes ("'") by two single quotes ("''") inside each field. This +option MUST be specified when a template is used for writing to a database, +otherwise SQL injection might occur.
+
+Please note that the database writer *checks* that the sql option is present in +the template. If it is not present, the write database action is disabled. This +is to guard you against accidential forgetting it and then becoming vulnerable +to SQL injection. The sql option can also be useful with files - especially if +you want to import them into a database on another machine for performance +reasons. However, do NOT use it if you do not have a real need for it - among +others, it takes some toll on the processing time. Not much, but on a really +busy system you might notice it ;)
+
+To escape:
+% = \%
+\ = \\ --> '\' is used to escape (as in C)
+$template TraditionalFormat,%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"
+
+Properties can be accessed by the property replacer +(see there for details).

Output Channels

Output Channels are a new concept first introduced in rsyslog 0.9.0. As of this +writing, it is still unclear if they will stay in rsyslog or go away. So if you +use them, be prepared to change you configuration file syntax when you upgrade +to a later release.
+
+The idea behind output channel definitions is that it shall provide an umbrella +for any type of output that the user might want. In essence,
+this is the "file" part of selector lines (and this is why we are not sure +output channel syntax will stay after the next review). There is a
+difference, though: selector channels both have filter conditions (currently +facility and severity) as well as the output destination. Output channels define +the output defintion, only. As of this build, they can only be used to write to +files - not pipes, ttys or whatever else. If we stick with output channels, this +will change over time.

In concept, an output channel includes everything needed to know about an +output actions. In practice, the current implementation only carries
+a filename, a maximum file size and a command to be issued when this file size +is reached. More things might be present in future version, which might also +change the syntax of the directive.

Output channels are defined via an $outchannel directive. It's syntax is as +follows:
+
+$outchannel name,file-name,max-size,action-on-max-size
+
+name is the name of the output channel (not the file), file-name is the file +name to be written to, max-size the maximum allowed size and action-on-max-size +a command to be issued when the max size is reached.
+
+Please note that max-size is queried BEFORE writing the log message to the file. +So be sure to set this limit reasonably low so that any message might fit. For +the current release, setting it 1k lower than you expected is helpful. The +max-size must always be specified in bytes - there are no special symbols (like +1k, 1m,...) at this point of development.
+
+Keep in mind that $outchannel just defines a channel with "name". It does not +activate it. To do so, you must use a selector line (see below). That selector +line includes the channel name plus an $ sign in front of it. A sample might be:
+
+*.* $mychannel
+
+In its current form, output channels primarily provide the ability to size-limit +an output file. To do so, specify a maximum size. When this size is reachead, +rsyslogd will execute the action-on-max-size command and then reopen the file +and retry. The command should be something like a log rotation script or a +similar thing.

+
WARNING +
The current command logic is a quick hack. It simply issues the command via a +system() call, which is very dirty. Don't make rsyslogd a suid +binary and use action-on-max-size commands - this will mess up things. Fixing +this is on top of the todo list and the fix will hopefully +appear soon.
+

If there is no action-on-max-size command or the command did not resolve the +situation, the file is closed and never reopened by rsyslogd (except, of course, +by huping it). This logic was integrated when we first experienced severe issues +with files larger 2gb, which could lead to rsyslogd dumping core. In such cases, +it is more appropriate to stop writing to a single file. Meanwhile, rsyslogd has +been fixed to support files larger 2gb, but obviously only on file systems and +operating system versions that do so. So it can still make sense to enforce a +2gb file size limit.

SELECTORS

The selector field itself again consists of two parts, a facility and a +priority, separated by a period (``.''). Both parts are case insenstive and can +also be specified as decimal numbers, but don't do that, you have been warned. +Both facilities and priorities are described in rsyslog(3). The names mentioned +below correspond to the similar LOG_-values in /usr/include/rsyslog.h.

The facility is one of the following keywords: auth, authpriv, cron, daemon, +kern, lpr, mail, mark, news, security (same as auth), syslog, user, uucp and +local0 through local7. The keyword security should not be used anymore and mark +is only for internal use and therefore should not be used in applications. +Anyway, you may want to specify and redirect these messages here. The facility +specifies the subsystem that produced the message, i.e. all mail programs log +with the mail facility (LOG_MAIL) if they log using syslog.

Please note that the upcoming next syslog-RFC specifies many more facilities. +Support for them will be added in a future version of rsyslog, which might +require changes to existing configuration files.

The priority is one of the following keywords, in ascending order: debug, info, +notice, warning, warn (same as warning), err, error (same as err), crit, alert, +emerg, panic (same as emerg). The keywords error, warn and panic are deprecated +and should not be used anymore. The priority defines the severity of the message
+
The behavior of the original BSD syslogd is that all messages of the specified +priority and higher are logged according to the given action. This rsyslogd(8) +behaves the same, but has some extensions.

In addition to the above mentioned names the rsyslogd(8) understands the +following extensions: An asterisk (``*'') stands for all facilities or all +priorities, depending on where it is used (before or after the period). The +keyword none stands for no priority of the given facility.

You can specify multiple facilities with the same priority pattern in one +statement using the comma (``,'') operator. You may specify as much facilities +as you want. Remember that only the facility part from such a statement is +taken, a priority part would be skipped.

Multiple selectors may be specified for a single action using the semicolon +(``;'') separator. Remember that each selector in the selector field is capable +to overwrite the preceding ones. Using this behavior you can exclude some +priorities from the pattern.

Rsyslogd(8) has a syntax extension to the original BSD source, that makes its +use more intuitively. You may precede every priority with an equation sign +(``='') to specify only this single priority and not any of the above. You may +also (both is valid, too) precede the priority with an exclamation mark (``!'') +to ignore all that priorities, either exact this one or this and any higher +priority. If you use both extensions than the exclamation mark must occur before +the equation sign, just use it intuitively.

ACTIONS

The action field of a rule describes what to do with the message. In general, +message content is written to a kind of "logfile". But also other actions might +be done, like writing to a database table or forwarding to another host.
+
+Templates can be used with all actions. If used, the specified template is used +to generate the message content (instead of the default template). To specify a +template, write a semicolon after the action value immediately followed by the +template name.
+
+Beware: templates MUST be defined BEFORE they are used. It is OK to define some +templates, then use them in selector lines, define more templates and use use +them in the following selector lines. But it is NOT permitted to use a template +in a selectore line that is above its definition. If you do this, the selector +line will be ignored.

Regular File

Typically messages are logged to real files. The file has to be specified with +full pathname, beginning with a slash ``/''.
+
+You may prefix each entry with the minus ``-'' sign to omit syncing the file +after every logging. Note that you might lose information if the system crashes +right behind a write attempt. Nevertheless this might give you back some +performance, especially if you run programs that use
+logging in a very verbose manner.

If your system is connected to a reliable UPS and you receive lots of log +data (e.g. firewall logs), it might be a very good idea to turn of
+syncing by specifying the "-" in front of the file name.

Named Pipes

This version of rsyslogd(8) has support for logging output to named pipes (fifos). +A fifo or named pipe can be used as a destination for log messages by prepending +a pipe symbol (``|'') to the name of the file. This is handy for debugging. Note +that the fifo must be created with the mkfifo(1) command before rsyslogd(8) is +started.

Terminal and Console

If the file you specified is a tty, special tty-handling is done, same with +/dev/console.

Remote Machine

Rsyslogd provides full remote logging, i.e. is able to send messages to a +remote host running rsyslogd(8) and to receive messages from remote hosts. The +remote host won't forward the message again, it will just log them locally. To +forward messages to another host, prepend the hostname with the at sign (``@'').
+
+Using this feature you're able to control all syslog messages on one host, if +all other machines will log remotely to that. This tears down
+administration needs.
+
+Please note that this version of rsyslogd by default does NOT forward messages +it has received from the network to another host. Specify the
+-h option to enable this.

List of Users

Usually critical messages are also directed to ``root'' on that machine. You can +specify a list of users that shall get the message by simply writing the login. +You may specify more than one user by separating them with commas (``,''). If +they're logged in they get the message. Don't think a mail would be sent, that +might be too late.

Everyone logged on

Emergency messages often go to all users currently online to notify them that +something strange is happening with the system. To specify this wall(1)-feature +use an asterisk (``*'').

Database Table

This allows logging of the message to a database table. Currently, only MySQL +databases are supported. By default, a MonitorWare-compatible schema is required +for this to work. You can create that schema with the createDB.SQL file that +came with the rsyslog package. You can also
+use any other schema of your liking - you just need to define a proper template +and assign this template to the action.
+
+The database writer is called by specifying a greater-then sign (">") in front +of the database connect information. Immediately after that
+sign the database host name must be given, a comma, the database name, another +comma, the database user, a comma and then the user's password. If a specific +template is to be used, a semicolong followed by the template name can follow +the connect information. This is as follows:
+
+>dbhost,dbname,dbuser,dbpassword;dbtemplate

Output Channel

Binds an output channel definition (see there for details) to this action. +Output channel actions must start with a $-sign, e.g. if you would like to bind +your output channel definition "mychannel" to the action, use "$mychannel". +Output channels support template definitions like all all other actions.

TEMPLATE NAME

Every ACTION can be followed by a template name. If so, that template is used +for message formatting. If no name is given, a hardcoded default template is +used for the action. There can only be one template name for each given action. +The default template is specific to each action. For a description of what a +template is and what you can do with it, see "TEMPLATES" at the top of this +document.

EXAMPLES

Below are example for templates and selector lines. I hope they are +self-explanatory. If not, please see www.monitorware.com/rsyslog/ for advise.

TEMPLATES

Please note that the samples are split across multiple lines. A template MUST +NOT actually be split across multiple lines.
+
+A template that resambles traditional syslogd file output:
+$template TraditionalFormat,"%timegenerated% %HOSTNAME%
+%syslogtag%%msg:::drop-last-lf%\n"
+
+A template that tells you a little more about the message:
+$template precise,"%syslogpriority%,%syslogfacility%,%timegenerated%,%HOSTNAME%,
+%syslogtag%,%msg%\n"
+
+A template for RFC 3164 format:
+$template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%"
+
+A template for the format traditonally used for user messages:
+$template usermsg," XXXX%syslogtag%%msg%\n\r"
+
+And a template with the traditonal wall-message format:
+$template wallmsg,"\r\n\7Message from syslogd@%HOSTNAME% at %timegenerated%
+
+A template that can be used for the database write (please note the SQL
+template option)
+$template MySQLInsert,"insert iut, message, receivedat values
+('%iut%', '%msg:::UPPERCASE%', '%timegenerated:::date-mysql%')
+into systemevents\r\n", SQL
+
+The following template emulates WinSyslog +format (it's an Adiscon format, you do +not feel bad if you don't know it ;)). It's interesting to see how it takes +different parts out of the date stamps. What happens is that the date stamp is +split into the actual date and time and the these two are combined with just a +comma in between them.
+
+$template WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,
+%timegenerated:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%,
+%timegenerated:12:19:date-rfc3339%,%syslogfacility%,%syslogpriority%,
+%syslogtag%%msg%\n"

SELECTOR LINES

# Store critical stuff in critical
+#
+*.=crit;kern.none /var/adm/critical
+
+This will store all messages with the priority crit in the file /var/adm/critical, +except for any kernel message.
+
+
+# Kernel messages are first, stored in the kernel
+# file, critical messages and higher ones also go
+# to another host and to the console. Messages to
+# the host finlandia are forwarded in RFC 3164
+# format (using the template defined above).
+#
+kern.* /var/adm/kernel
+kern.crit @finlandia;RFC3164fmt
+kern.crit /dev/console
+kern.info;kern.!err /var/adm/kernel-info
+
+The first rule direct any message that has the kernel facility to the file /var/adm/kernel.
+
+The second statement directs all kernel messages of the priority crit and higher +to the remote host finlandia. This is useful, because if the host crashes and +the disks get irreparable errors you might not be able to read the stored +messages. If they're on a remote host, too, you still can try to find out the +reason for the crash.
+
+The third rule directs these messages to the actual console, so the person who +works on the machine will get them, too.
+
+The fourth line tells rsyslogd to save all kernel messages that come with +priorities from info up to warning in the file /var/adm/kernel-info. Everything +from err and higher is excluded.
+
+
+# The tcp wrapper loggs with mail.info, we display
+# all the connections on tty12
+#
+mail.=info /dev/tty12
+
+This directs all messages that uses mail.info (in source LOG_MAIL | LOG_INFO) to +/dev/tty12, the 12th console. For example the tcpwrapper tcpd(8) uses this as +it's default.
+
+
+# Store all mail concerning stuff in a file
+#
+mail.*;mail.!=info /var/adm/mail
+
+This pattern matches all messages that come with the mail facility, except for +the info priority. These will be stored in the file /var/adm/mail.
+
+
+# Log all mail.info and news.info messages to info
+#
+mail,news.=info /var/adm/info
+
+This will extract all messages that come either with mail.info or with news.info +and store them in the file /var/adm/info.
+
+
+# Log info and notice messages to messages file
+#
+*.=info;*.=notice;\
+mail.none /var/log/messages
+
+This lets rsyslogd log all messages that come with either the info or the notice +facility into the file /var/log/messages, except for all
+messages that use the mail facility.
+
+
+# Log info messages to messages file
+#
+*.=info;\
+mail,news.none /var/log/messages
+
+This statement causes rsyslogd to log all messages that come with the info +priority to the file /var/log/messages. But any message coming either with the +mail or the news facility will not be stored.
+
+
+# Emergency messages will be displayed using wall
+#
+*.=emerg *
+
+This rule tells rsyslogd to write all emergency messages to all currently logged +in users. This is the wall action.
+
+
+# Messages of the priority alert will be directed
+# to the operator
+#
+*.alert root,rgerhards
+
+This rule directs all messages with a priority of alert or higher to the +terminals of the operator, i.e. of the users ``root'' and ``rgerhards'' if +they're logged in.
+
+
+*.* @finlandia
+
+This rule would redirect all messages to a remote host called finlandia. This is +useful especially in a cluster of machines where all syslog messages will be +stored on only one machine.
+
+In the format shown above, UDP is used for transmitting the message. The +destination port is set to the default auf 514. Rsyslog is also capable of using +much more secure and reliable TCP sessions for message forwarding. Also, the +destination port can be specified. To select TCP, simply add one additional @ in +front of the host name (that is, @host is UPD, @@host is TCP). For example:
+
+
+*.* @@finlandia
+
+To specify the destination port on the remote machine, use a colon followed by +the port number after the machine name. The following forwards to port 1514 on +finlandia:
+
+
+*.* @@finlandia:1514
+
+This syntax works both with TCP and UDP based syslog. However, you will probably +primarily need it for TCP, as there is no well-accepted port for this transport +(it is non-standard). For UDP, you can usually stick with the default auf 514, +but might want to modify it for security rea-
+sons. If you would like to do that, it's quite easy:
+
+
+*.* @finlandia:1514
+
+
+
+*.* >dbhost,dbname,dbuser,dbpassword;dbtemplate
+
+This rule writes all message to the database "dbname" hosted on "dbhost". The +login is done with user "dbuser" and password "dbpassword". The actual table +that is updated is specified within the template (which contains the insert +statement). The template is called "dbtemplate" in this case.

CONFIGURATION FILE SYNTAX DIFFERENCES

Rsyslogd uses a slightly different syntax for its configuration file than the +original BSD sources. Originally all messages of a specific priority and above +were forwarded to the log file. The modifiers ``='', ``!'' and ``-'' were added +to make rsyslogd more flexible and to use it in a more intuitive manner.
+
+The original BSD syslogd doesn't understand spaces as separators between the +selector and the action field.
+
+When compared to syslogd from sysklogd package, rsyslogd offers additional +features (like template and database support). For obvious reasons, the syntax for +defining such features is available +in rsyslogd, only.
+

+ + \ No newline at end of file diff --git a/doc/rsyslog_mysql.html b/doc/rsyslog_mysql.html new file mode 100644 index 00000000..778b4257 --- /dev/null +++ b/doc/rsyslog_mysql.html @@ -0,0 +1,240 @@ + +Writing syslog Data to MySQL + + + +

Writing syslog messages to MySQL

Written by + Rainer + Gerhards (2005-08-02)

Abstract

In this paper, I describe how to write +syslog +messages to a MySQL database. Having +syslog messages in a database is often handy, especially when you intend to set +up a front-end for viewing them. This paper describes an approach +with rsyslogd, an alternative enhanced +syslog daemon natively supporting MySQL. I describe the components needed +to be installed and how to configure them.

Background

In many cases, syslog data is simply written to text files. This approach has +some advantages, most notably it is very fast and efficient. However, data +stored in text files is not readily accessible for real-time viewing and analysis. +To do that, the messages need to be in a database. There are various +ways to store syslog messages in a database. For example, some have the syslogd +write text files which are later feed via a separate script into the database. +Others have written scripts taking the data (via a pipe) from a +non-database-aware syslogd and store them as they appear. Some others use +database-aware syslogds and make them write the data directly to the database. +In this paper, I use that "direct write" approach. I think it is superior, +because the syslogd itself knows the status of the database connection and thus +can handle it intelligently (well ... hopefully ;)). I use rsyslogd to acomplish +this, simply because I have initiated the rsyslog project with +database-awareness as one goal.

One word of caution: while message storage in the database provides an +excellent foundation for interactive analysis, it comes at a cost. Database i/o +is considerably slower than text file i/o. As such, directly writing to +the database makes sense only if your message volume is low enough to allow a) +the syslogd, b) the network, and c) the database server to catch +up with it. Some time ago, I have written a paper on + +optimizing syslog server performance. While this paper talks about +Window-based solutions, the ideas in it are generic enough to apply here, too. +So it might be worth reading if you anticipate medium high to high traffic. If you +anticipate really high traffic (or very large traffic spikes), you should +seriously consider forgetting about direct database writes - in my opinion, such +a situation needs either a very specialised system or a different approach (the +text-file-to-database approach might work better for you in this case). +

Overall System Setup

In this paper, I concentrate on the server side. If you are thinking about +interactive syslog message review, you probably want to centralize syslog. In +such a scenario, you have multiple machines (the so-called clients) send their +data to a central machine (called server in this context). While I expect such a +setup to be typical when you are interested in storing messages in the database, +I do not describe how to set it up. This is beyond the scope of this paper. If +you search a little, you will probably find many good descriptions on how to +centralize syslog. If you do that, it might be a good idea to do it securely, so +you might also be interested in my paper on +ssl-encrypting syslog message transfer.

No matter how the messages arrive at the server, their processing is always the +same. So you can use this paper in combination with any description for +centralized syslog reporting.

As I already said, I use rsyslogd on the server. It has intrinsic support for +talking to MySQL databases. For obvious reasons, we also need an instance of +MySQL running. To keep us focussed, the setup of MySQL itself is also beyond the +scope of this paper. I assume that you have successfully installed MySQL and +also have a front-end at hand to work with it (for example, +phpMyAdmin). Please make sure that this +is installed, actually working and you have a basic understanding of how to +handle it.

Setting up the system

You need to download and install rsyslogd first. Obtain it from the +rsyslog site. Make sure that you disable +stock syslogd, otherwise you will experience some difficulties.

It is important to understand how rsyslogd talks to the database. In rsyslogd, +there is the concept of "templates". Basically, a template is a string that +includes some replacement characters, which are called "properties" in rsyslog. +Properties are accessed via the "Property +Replacer". Simply said, you access properties by including their name +between percent signs inside the template. For example, if the syslog message is +"Test", the template "%msg%" would be expanded to "Test". Rsyslogd supports +sending template text as a SQL statement to MySQL. As such, the template must be +a valid SQL statement. There is no limit in what the statement might be, but +there are some obvious and not so obvious choices. For example, a template "drop +table xxx" is possible, but does not make an awful lot of sense. In practice, +you will always use an "insert" statment inside the template.

An example: if you would just like to store the msg part of the full syslog +message, you have probably created a table "syslog" with a single column "message". +In such a case, a good template would be "insert into syslog(message) values ('%msg%')". +With the example above, that would be expanded to "insert into syslog(message) +values('Test')". This expanded string is then sent to the database. It's that +easy, no special magic. The only thing you must ensure is that your template +expands to a proper SQL statement and that this statement matches your database +design.

Does that mean you need to create database schema yourself and also must +fully understand rsyslogd's properties? No, that's not needed. Because we +anticipated that folks are probably more interested in getting things going instead +of designing them from scratch. So we have provided a default schema as well +as build-in support for it. This schema also offers an additional +benefit: rsyslog is part of Adiscon's +MonitorWare product line (which +includes open source and closed source members). All of these tools share the +same default schema and know how to operate on it. For this reason, the default +schema is also called the "MonitorWare Schema". If you use it, you can simply +add phpLogCon, a GPLed syslog web interface, +to your system and have instant interactive access to your database. So there +are some benefits in using the provided schema.

The schema definition is contained in the file "createDB.sql". It comes with +the rsyslog package. Review it to check that the database name is acceptable for +you. Be sure to leave the table and field names unmodified, because +otherwise you need to customize rsyslogd's default sql template, which we do not +do +in this paper. Then, run the script with your favourite MySQL tool. Double-check +that the table was successfully created.

Next, we need to tell rsyslogd to write data to the database. As we use +the default schema, we do NOT need to define a template for this. We can use the +hardcoded one (rsyslogd handles the proper template linking). So all we need to +do is add a simple selector line to /etc/rsyslog.conf:

+
*.* + >database-server,database-name,database-userid,database-password
+

In many cases, MySQL will run on the local machine. In this case, you can +simply use "127.0.0.1" for database-server. This can be especially +advisable, if you do not need to expose MySQL to any process outside of the +local machine. In this case, you can simply bind it to 127.0.0.1, which provides +a quite secure setup. Of course, also supports remote MySQL instances. In that +case, use the remote server name (e.g. mysql.example.com) or IP-address. The +database-name by default is "syslog". If you have modified the default, use +your name here. Database-userid and -password are the credentials +used to connect to the database. As they are stored in clear text in +rsyslog.conf, that user should have only the least possible privileges. It is +sufficient to grant it INSERT privileges to the systemevents table, only. As a +side note, it is strongly advisable to make the rsyslog.conf file readable by +root only - if you make it world-readable, everybody could obtain the password +(and eventually other vital information from it). In our example, let's assume +you have created a MySQL user named "syslogwriter" with a password of +"topsecret" (just to say it bluntly: such a password is NOT a good idea...). If +your MySQL database is on the local machine, your rsyslog.conf line might look +like in this sample:

+
*.* + >127.0.0.1,syslog,syslogwriter,topsecret
+

Save rsyslog.conf, restart rsyslogd - and you should see syslog messages +being stored in the "systemevents" table!

The example line stores every message to the database. Especially if you have +a high traffic volume, you will probably limit the amount of messages being +logged. This is easy to acomplish: the "write database" action is just a regular +selector line. As such, you can apply normal selector-line filtering. If, for +example, you are only interested in messages from the mail subsystem, you can +use the following selector line:

+
mail.* + >127.0.0.1,syslog,syslogwriter,topsecret
+

Review the rsyslog.conf documentation for +details on selector lines and their filtering.

You have now completed everything necessary to store syslog messages to +the MySQL database. If you would like to try out a front-end, you might want +to look at phpLogCon, which displays +syslog data in a browser. As of this writing, phpLogCon is not yet a powerful +tool, but it's open source, so it might be a starting point for your own +solution.

On Reliability...

Rsyslogd writes syslog messages directly to the database. This implies that +the database must be available at the time of message arrival. If the database +is offline, no space is left or something else goes wrong - rsyslogd can not +write the database record. If rsyslogd is unable to store a message, it performs +one retry. This is helpful if the database server was restarted. In this case, +the previous connection was broken but a reconnect immediately succeeds. However, +if the database is down for an extended period of time, an immediate retry does +not help. While rsyslogd could retry until it finally succeeds, that would have +negative impact. Syslog messages keep coming in. If rsyslogd would be busy +retrying the database, it would not be able to process these messages. +Ultimately, this would lead to loss of newly arrived messages.

In most cases, rsyslogd is configured not only to write to the database but +to perform other actions as well. In the always-retry scenario, that would mean +no other actions would be carried out. As such, the design of rsyslogd is +limited to a single retry. If that does not succeed, the current message is will +not be written to the database and the MySQL database writer be suspended for a +short period of time. Obviously, this leads to the loss of the current message +as well as all messages received during the suspension period. But they are only +lost in regard to the database, all other actions are correctly carried out. +While not perfect, we consider this to be a better approach then the potential +loss of all messages in all actions.

In short: try to avoid database downtime if you do not want to experience +message loss.

Please note that this restriction is not rsyslogd specific. All approachs to +real-time database storage share this problem area.

Conclusion

With minumal effort, you can use rsyslogd to write syslog messages to a MySQL +database. Once the messages are arrived there, you can interactivley review and +analyse them. In practice, the messages are also stored in text files for +longer-term archival and the databases are cleared out after some time (to avoid +becoming too slow). If you expect an extremely high syslog message volume, +storing it in real-time to the database may outperform your database server. In +such cases, either filter out some messages or think about alternate approaches +involving non-real-time database writing (beyond the scope of this paper).

The method outline in this paper provides an easy to setup and maintain +solution for most use cases, especially with low and medium syslog message +volume (or fast database servers).

I have set up a site to demo web access to +syslog data. It is build using the steps outlined here and uses phpLogCon as +the front-end. You might want to visit it +to get a glimpse of how such a beast might look.

Feedback Requested

I would appreciate feedback on this paper. If you have additional ideas, +comments or find bugs, please +let me know.

References and Additional Material

www.rsyslog.com - the rsyslog site
demo.rsyslog.com - demo + site for rsyslog and the phpLogCon web interface
+ + Paper on Syslog Server Optimization

Revision History

2005-08-02 * + Rainer Gerhards * + initial version created
2005-08-03 * + Rainer Gerhards + * added references to demo site

Copyright

Permission is granted to copy, distribute and/or modify this document under +the terms of the GNU Free Documentation License, Version 1.2 or any later +version published by the Free Software Foundation; with no Invariant Sections, +no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be +viewed at +http://www.gnu.org/copyleft/fdl.html.

+ + \ No newline at end of file diff --git a/doc/rsyslog_stunnel.html b/doc/rsyslog_stunnel.html new file mode 100644 index 00000000..71bb878d --- /dev/null +++ b/doc/rsyslog_stunnel.html @@ -0,0 +1,248 @@ + +SSL Encrypting syslog with stunnel + + + +

SSL Encrypting Syslog with Stunnel

Written by + Rainer + Gerhards (2005-07-22)

Abstract

In this paper, I describe how to encrypt syslog +messages on the network. Encryption +is vital to keep the confidiental content of syslog messages secure. I describe the overall +approach and provide an HOWTO do it with the help of +rsyslogd and stunnel.

Background

Syslog is a +clear-text protocol. That means anyone with a sniffer can have +a peek at your data. In some environments, this is no problem at all. In +others, it is a huge setback, probably even preventing deployment of syslog +solutions. Thankfully, there is an easy way to encrypt syslog communication. I +will describe one approach in this paper.

The most straigthforward solution would be that the syslogd itself encrypts +messages. Unfortuantely, encryption is only standardized in +RFC 3195. But there +is currently no syslogd that implements RFC 3195's encryption features, +so this route leads to nothing. Another approach would be to use vendor- or +project-specific syslog extensions. There are a few around, but the problem here +is that they have compatibility issues. However, there is one surprisingly easy +and interoperable solution: though not standardized, many vendors and projects +implement plain tcp syslog. In a nutshell, plain tcp syslog is a mode where +standard syslog messages are transmitted via tcp and records are separated by +newline characters. This mode is supported by all major syslogd's (both on Linux/Unix +and Windows) as well as log sources (for example, +EventReporter for Windows +Event Log forwarding). Plain tcp syslog offers reliability, but it does not +offer encryption in itself. However, since it operates on a tcp stream, it is now easy +to add encryption. There are various ways to do that. In this paper, I will +describe how it is done with stunnel (an +other alternative would be IPSec, for example).

Stunnel is open source and it is available both for Unix/Linux and Windows. +It provides a way to + use ssl communication for any non-ssl aware client and server - in this case, + our syslogd.

Stunnel works much like a wrapper. Both on the client and on the server machine, + tunnel portals are created. The non-ssl aware client and server software is + configured to not directly talk to the remote partner, but to the local + (s)tunnel portal instead. Stunnel, in turn, takes the data received from the + client, encrypts it via ssl, sends it to the remote tunnel portal and that + remote portal sends it to the recipient process on the remote machine. The + transfer to the portals is done via unencrypted communication. As such, + it is vital that + the portal and the respective program that is talking to it are on the same + machine, otherwise data would travel partly unencrypted. Tunneling, as done by stunnel, + requires connection oriented communication. This is why you need to use + tcp-based syslog. As a side-note, you can also encrypt a plain-text RFC + 3195 session via stunnel, though this definitely is not what the + protocol designers had on their mind ;)

In the rest of this document, I assume that you use rsyslog on both the +client and the server. For the samples, I use Debian. +Interestingly, there are +some annoying differences between stunnel implementations. For example, on +Debian a comment line starts with a semicolon (';'). On +Red Hat, it starts with +a hash sign ('#'). So you need to watch out for subtle issues when setting up +your system.

Overall System Setup

In ths paper, I assume two machines, one named "client" and the other named "server". +It is obvious that, in practice, you will probably have multiple clients but +only one server. Syslog traffic shall be transmitted via stunnel over the +network. Port 60514 is to be used for that purpose. The machines are set up as +follows:

Client

rsyslog forwards message to stunnel local portal at port 61514
local stunnel forwards data via the network to port 60514 to its remote + peer

Server

stunnel listens on port 60514 to connections from its client peers
all connections are forwarded to the locally-running rsyslog listening + at port 61514

Setting up the system

For Debian, you need the "stunnel4" package. The "stunnel" package is the +older 3.x release, which will not support the configuration I describe below. +Other distributions might have other names. For example, on Red Hat it is just "stunnel". +Make sure that you install the appropriate package on both the client and the +server. It is also a good idea to check if there are updates for either stunnel +or openssl (which stunnel uses) - there are often security fixes available and +often the latest fixes are not included in the default package.

In my sample setup, I use only the bare minimum of options. For example, I do +not make the server check client cerficiates. Also, I do not talk much about +certificates at all. If you intend to really secure your system, you should +probably learn about certificates and how to manage and deploy them. This is +beyond the scope of this paper. For additional information, + +http://www.stunnel.org/faq/certs.html is a good starting point.

You also need to install rsyslogd on both machines. Do this before starting +with the configuration. You should also familarize yourself with its +configuration file syntax, so that you know which actions you can trigger with +it. Rsyslogd can work as a drop-in replacement for stock +sysklogd. So if you know +the standard syslog.conf syntax, you do not need to learn any more to follow +this paper.

Server Setup

At the server, you need to have a digital certificate. That certificate +enables SSL operation, as it provides the necessary crypto keys being used to +secure the connection. Many versions of stunnel come with a default certificate, +often found in /etc/stunnel/stunnel.pem. If you have it, it is good for testing +only. If you use it in production, it is very easy to break into your secure +channel as everybody is able to get hold of your private key. I didn't find an +stunnel.pem on my Debian machine. I guess the Debian folks removed it because of +its insecurity.

You can create your own certificate with a simple openssl tool - you need to +do it if you have none and I highly recommend to create one in any case. To +create it, cd to /etc/stunnel and type:

openssl req -new -x509 -days 3650 -nodes -out +stunnel.pem -keyout stunnel.pem

That command will ask you a number of questions. Provide some answer for +them. If you are unsure, read + +http://www.stunnel.org/faq/certs.html. After the command has finished, you +should have a usable stunnel.pem in your working directory.

Next is to create a configuration file for stunnel. It will direct stunnel +what to do. You can used the following basic file:

; Certificate/key is needed in server mode
+cert = /etc/stunnel/stunnel.pem
+
+; Some debugging stuff useful for troubleshooting
+debug = 7
+foreground=yes
+
+[ssyslog]
+accept  = 60514
+connect = 61514
+

Save this file to e.g. /etc/stunnel/syslog-server.conf. Please note that the +settings in italics are for debugging only. They run stunnel +with a lot of debug information in the foreground. This is very valuable while +you setup the system - and very useless once everything works well. So be sure +to remove these lines when going to production.

Finally, you need to start the stunnel daemon. Under Debian, this is done via +"stunnel /etc/stunnel/syslog.server.conf". If you have enabled the debug +settings, you will immediately see a lot of nice messages.

Now you have stunnel running, but it obviously unable to talk to rsyslog - +because it is not yet running. If not already done, configure it so that it does +everything you want. If in doubt, you can simply copy /etc/syslog.conf to /etc/rsyslog.conf +and you probably have what you want. The really important thing in rsyslogd +configuration is that you must make it listen to tcp port 61514 (remember: this +is where stunnel send the messages to). Thankfully, this is easy to achive: just +add "-t 61514" to the rsyslogd startup options in your system startup script. +After done so, start (or restart) rsyslogd.

The server should now be fully operational.

Client Setup

The client setup is simpler. Most importantly, you do not need a certificate +(of course, you can use one if you would like to authenticate the client, but +this is beyond the scope of this paper). So the basic thing you need to do is +create the stunnel configuration file.

; Some debugging stuff useful for troubleshooting
+debug = 7
+foreground=yes
+
+client=yes
+
+[ssyslog]
+accept  = 127.0.0.1:61514
+connect = 192.0.2.1:60514
+
+

Again, the text in italics is for debugging purposes only. I suggest +you leave it in during your initial testing and then remove it. The most +important difference to the server configuration outlined above is the "client=yes" +directive. It is what makes this stunnel behave like a client. The accept +directive binds stunnel only to the local host, so that it is protected from +receiving messages from the network (somebody might fake to be the local sender). +The address "192.0.2.1" is the address of the server machine. You must change it +to match your configuration. Save this file to /etc/stunnel/syslog-client.conf.

Then, start stunnel via "stunnel4 /etc/stunnel/syslog-client.conf". Now +you should see some startup messages. If no errors appear, you have a running +client stunnel instance.

Finally, you need to tell rsyslogd to send data to the remote host. In stock +syslogd, you do this via the "@host" forwarding directive. The same works with +rsyslog, but it suppports extensions to use tcp. Add the following line to your +/etc/rsyslog.conf:

*.*      @@127.0.0.1:61514
+
+

+ + + +

Please note the double at-sign (@@). This is no typo. It tells rsyslog to use +tcp instead of udp delivery. In this sample, all messages are forwarded to the +remote host. Obviously, you may want to limit this via the usual rsyslog.conf +settings (if in doubt, use man rsyslog.con).

You do not need to add any special startup settings to rsyslog on the client. +Start or restart rsyslog so that the new configuration setting takes place.

Done

After following these steps, you should have a working secure syslog +forwarding system. To verify, you can type "logger test" or a similar smart +command on the client. It should show up in the respective server log file. If +you dig out you sniffer, you should see that the traffic on the wire is actually +protected. In the configuration use above, the two stunnel endpoints should be +quite chatty, so that you can follow the action going on on your system.

If you have only basic security needs, you can probably just remove the debug +settings and take the rest of the configuration to production. If you are +security-sensitve, you should have a look at the various stunnel settings that +help you further secure the system.

Preventing Systems from talking directly to the rsyslog Server

It is possible that remote systems (or attackers) talk to the rsyslog server +by directly connecting to its port 61514. Currently (Jule of 2005), rsyslog does +not offer the ability to bind to the local host, only. This feature is planned, +but as long as it is missing, rsyslog must be protected via a firewall. This can +easily be done via e.g iptables. Just be sure not to forget it.

Conclusion

With minumal effort, you can set up a secure logging infrastructure employing +ssl encrypted syslog message transmission. As a side note, you also have the +benefit of reliable tcp delivery which is far less prone to message loss than +udp.

Feedback requested

I would appreciate feedback on this tutorial. If you have additional ideas, +comments or find bugs (I *do* bugs - no way... ;)), please +let me know.

Revision History

2005-07-22 * + Rainer Gerhards * Initial Version created
2005-07-26 * + Rainer Gerhards * Some text brush-up, hyperlinks added
2005-08-03 * + Rainer Gerhards + * license added

Copyright

Permission is granted to copy, distribute and/or modify this document + under the terms of the GNU Free Documentation License, Version 1.2 + or any later version published by the Free Software Foundation; + with no Invariant Sections, no Front-Cover Texts, and no Back-Cover + Texts. A copy of the license can be viewed at + +http://www.gnu.org/copyleft/fdl.html.

+ + + \ No newline at end of file diff --git a/doc/status.html b/doc/status.html new file mode 100644 index 00000000..f1217fc3 --- /dev/null +++ b/doc/status.html @@ -0,0 +1,34 @@ + + +rsyslog status page + + +

rsyslog status page

This page reflects the status as of 2005-08-02.

Current Release

0.9.5 - change log - +download

Platforms

Linux [tested on Red Hat & Debian]
BSD [tested on +FreeBSD and a quick Test on NetBSD, 0.8.4 only]

Additional information

Currently supported features are now listed on the rsyslog features page.

The rsyslog home page is www.rsyslog.com.
Mailing list info can be found at + http://lists.adiscon.com/rsyslog.
The change log can be found at + + http://www.rsyslog.com/Topic4.phtml.
You may also find Rainer's + syslog blog an interesting read.

The project was initiated in 2004 by +Rainer Gerhards +and is currently being maintained by him. See the history +page for more background information.

+ + \ No newline at end of file -- cgit