RulesetParser - rsyslog.conf file

From b6a92e2f23895cadf9f0ea2de64021912bd2eeb0 Mon Sep 17 00:00:00 2001 From: Rainer Gerhards Date: Wed, 4 Nov 2009 14:21:08 +0100 Subject: added $RulesetParser config directive The implementation is now almost done and works, including doc. I now need to verify shutdown, guess there are some resource leaks left... --- doc/multi_ruleset.html | 7 +++ doc/rsconf1_rulesetparser.html | 117 +++++++++++++++++++++++++++++++++++++++++ doc/rsyslog_conf_global.html | 4 +- 3 files changed, 127 insertions(+), 1 deletion(-) create mode 100644 doc/rsconf1_rulesetparser.html (limited to 'doc') diff --git a/doc/multi_ruleset.html b/doc/multi_ruleset.html index f82a73a6..da65b4ba 100644 --- a/doc/multi_ruleset.html +++ b/doc/multi_ruleset.html @@ -71,6 +71,13 @@ at the time the bind directive is given. There are many ways to make sure this h I personally think that it is best to define all rule sets at the top of rsyslog.conf and define the inputs at the bottom. This kind of reverses the traditional recommended ordering, but seems to be a really useful and straightforward way of doing things. +

Why are rulesets important for different parser configurations?

Custom message parsers, used to handle differnet (and potentially otherwise-invalid) +message formats, can be bound to rulesets. So multiple rulesets can be a very useful +way to handle devices sending messages in different malformed formats in a consistent +way. Unfortunately, this is not uncommon in the syslog world. An in-depth explanation +with configuration sample can be found at the +$RulesetParser configuration directive.

Can I use a different Ruleset as the default?

This is possible by using the diff --git a/doc/rsconf1_rulesetparser.html b/doc/rsconf1_rulesetparser.html new file mode 100644 index 00000000..03306ca9 --- /dev/null +++ b/doc/rsconf1_rulesetparser.html @@ -0,0 +1,117 @@ + + +RulesetParser - rsyslog.conf file + + +rsyslog.conf configuration directive + +

$RulesetParser

Type: ruleset-specific configuration directive

Parameter Values: string

Available since: 5.3.4+

Default: rsyslog.rfc5424;rsyslog.rfc5425

Description:

+This directive permits to specify which message parsers should be used for the ruleset +in question. It no ruleset is explicitely specified, the default ruleset is used. Message +parsers are contained in (loadable) parser modules with the most common cases +(RFC3164 and RFC5424) being build-in into rsyslogd. +

When this directive is specified the first time for a ruleset, it will not only add the +parser to the ruleset, it will also wipe out the default parsers. So if you need to have +them in addition to the custom parser, you need to specify them as well. +

Order of directives is important. Parsers are tried one after another, in the order +they are specified inside the config. As soon as a parser is able to parse the message, +it will do so and no other parsers will be executed. If no matching parser can be found, +the message will be discarded and a warning message be issued (but only for the first +1,000 instances of this problem, to prevent message generation loops). +

Note that the rfc3164 parser will always be able to parse a message - it may +just not be the format that you like. This has two important implications: 1) always place +that parser at the END of the parser list, or the other parsers after it will never +be tried and 2) if you would like to make sure no message is lost, placing the rfc3164 +parser at the end of the parser list ensures that. +

Multiple parser modules are very useful if you have various devices that emit +messages that are malformed in various ways. The route to take then is +

make sure you find a custom parser for that device; if there is no one, you +may consider writing one yourself (it is not that hard) or getting one written +as part of +Adiscon's professional services +for rsyslog. +
load your custom parsers via $ModLoad +
create a ruleset for each malformed format; assign the custom parser to it +
create a specific listening port for all devices that emit the same +malformed format +
bind the listener to the ruleset with the required parser +

Note that it may be cumbersome to add all rules to all rulesets. To avoid this, +you can either use $Include or omruleset +(what probably provides the best solution). +

More information about rulesets in general can be found in +multi-ruleset support in rsyslog. +

Caveats:

currently none known

+ +

Example:

This example assumes there are two devices emiting malformed messages via UDP. +We have two custom parsers for them, named "device1.parser" and +"device2.parser". In addition to that, we have a number of other +devices sending wellformed messages, also via UDP. +

The solution is to listen for data from the two devices on two special +ports (10514 and 10515 in this example), create a ruleset for each and +assign the custom parsers to them. The rest of the messages are received via +port 514 using the regular parsers. Processing shall be equal for all messages. +So we simply forward the malformed messages to the regular queue once they are parsed (keep +in mind that a message is never again parsed once any parser properly processed it). +

$ModLoad imudp
+$ModLoad pmdevice1 # load parser "device1.parser" for device 1
+$ModLoad pmdevice2 # load parser "device2.parser" for device 2
+
+# define ruleset for the first device sending malformed data
+$Ruleset maldev1
+$RulesetCreateMainQueue on # create ruleset-specific queue
+$RulesetParser "device1.parser" # note: this deactivates the default parsers
+# forward all messages to default ruleset:
+$ActionOmrulesetRulesetName RSYSLOG_DefaultRuleset 
+*.* :omruleset:
+
+# define ruleset for the second device sending malformed data
+$Ruleset maldev2
+$RulesetCreateMainQueue on # create ruleset-specific queue
+$RulesetParser "device2.parser" # note: this deactivates the default parsers
+# forward all messages to default ruleset:
+$ActionOmrulesetRulesetName RSYSLOG_DefaultRuleset 
+*.* :omruleset:
+
+# switch back to default ruleset
+$Ruleset RSYSLOG_DefaultRuleset
+*.*           /path/to/file
+auth.info     @authlogger.example.net
+# whatever else you usually do...
+
+
+# now define the inputs and bind them to the rulesets
+# first the default listener (utilizing the default ruleset)
+$UDPServerRun 514
+
+# now the one with the parser for device type 1:
+$InputUDPServerBindRuleset maldev1
+$UDPServerRun 10514
+
+# and finally the one for device type 2:
+$InputUDPServerBindRuleset maldev2
+$UDPServerRun 10515
+

Note the positions of the directives. With the current config language, +sequence of statements is very important. This is ugly, but unfortunately +the way it currently works. +

+ +

[rsyslog.conf overview] [manual +index] [rsyslog site]

+ + diff --git a/doc/rsyslog_conf_global.html b/doc/rsyslog_conf_global.html index 1bf02a55..beb90e02 100644 --- a/doc/rsyslog_conf_global.html +++ b/doc/rsyslog_conf_global.html @@ -249,12 +249,14 @@ large enough for the whole message. (Introduced with 4.1.5). Once set, it affect

$ResetConfigVariables

$Ruleset name - starts a new ruleset or switches back to one already defined. All following actions belong to that new rule set. -the name does not yet exist, it is created. To swith back to rsyslog's +the name does not yet exist, it is created. To switch back to rsyslog's default ruleset, specify "RSYSLOG_DefaultRuleset") as the name. All following actions belong to that new rule set. It is advised to also read our paper on using multiple rule sets in rsyslog.

$RulesetCreateMainQueue on - creates a ruleset-specific main queue. +

$RulesetParser - enables to set +a specific (list of) message parsers to be used with the ruleset.

$OptimizeForUniprocessor [on/off] - turns on optimizatons which lead to better performance on uniprocessors. If you run on multicore-machiens, turning this off lessens CPU load. The default may change as uniprocessor systems become less common. [available since 4.1.0]

-- cgit