1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head>
<title>Multiple Rulesets in rsyslog</title></head>
<body>
<h1>Multiple Rulesets in rsyslog</h1>
<p>Starting with version 4.5.0 and 5.1.1, <a href="http://www.rsyslog.com">rsyslog</a> supports
multiple rulesets within a single configuration.
This is especially useful for routing the recpetion of remote messages to a set of specific rules.
Note that the input module must support binding to non-standard rulesets, so the functionality
may not be available with all inputs.
<p>In this document, I am using <a href="imtcp.html">imtcp</a>, an input module
that supports binding to non-standard rulesets since rsyslog started to support them.
<h2>What is a Ruleset?</h2>
If you have worked with (r)syslog.conf, you know that it is made up of what I call rules (others
tend to call them selectors, a sysklogd term). Each rule consist of a filter and one or more
actions to be carried out when the filter evaluates to true. A filter may be as simple as a
traditional
syslog priority based filter (like "*.*" or "mail.info" or a as complex as a
script-like expression. Details on that are covered in the config file documentation. After the
filter come action specifiers, and an action is something that does something to a message, e.g.
write it to a file or forward it to a remote logging server.
<p>A traditional configuration file is made up of one or more of these rules. When a new
message arrives, its processing starts with the first rule (in order of appearance in
rsyslog.conf) and continues for each rule until either all rules have been processed or
a so-called "discard" action happens, in which case processing stops and the
message is thrown away (what also happens after the last rule has been processed).
<p>The <b>multi-ruleset</b> support now permits to specify more than one such rule sequence.
You can think of a traditional config file just as a single default rule set, which is
automatically bound to each of the inputs. This is even what actually happens. When
rsyslog.conf is processed, the config file parser looks for the directive
<pre>$RuleSet <name>
</pre>
<p>Where name is any name the user likes (but must not start with "RSYSLOG_", which
is the name space reserved for rsyslog use). If it finds this directive, it begins a new
rule set (if the name was not yet know) or switches to an already-existing one (if the name
was known). All rules defined between this $RuleSet directive and the next one are appended
to the named ruleset. Note that the reserved name "RSYSLOG_DefaultRuleset" is used to
specify rsyslogd's default ruleset. You can use that name whereever you can use a ruleset name,
including when binding an input to it.
<p>Inside a ruleset, messages are processed as described above: they start with the first rule
and rules are processed in the order of appearance of the configuration file until either
there are no more rules or the discard action is executed. Note that with multiple rulesets
no longer <b>all</b> rsyslog.conf rules are executed but <b>only</b> those that are
contained within the specific ruleset.
<p>Inputs must explicitely bind to rulesets. If they don't do, the default ruleset is bound.
<p>This brings up the next question:
<h2>What does "To bind to a Ruleset" mean?</h2>
<p>This term is used in the same sense as "to bind an IP address to an interface":
it means that a specific input, or part of an input (like a tcp listener) will use a specific
ruleset to "pass its messages to". So when a new message arrives, it will be processed
via the bound ruleset. Rule from all other rulesets are irrelevant and will never be processed.
<p>This makes multiple rulesets very handy to process local and remote message via
seperate means: bind the respective receivers to different rule sets, and you do not need
to seperate the messages by any other method.
<p>Binding to rulesets is input-specifc. For imtcp, this is done via the
<pre>$InputTCPServerBindRuleset <name>
</pre>
directive. Note that "name" must be the name of a ruleset that is already defined
at the time the bind directive is given. There are many ways to make sure this happens, but
I personally think that it is best to define all rule sets at the top of rsyslog.conf and
define the inputs at the bottom. This kind of reverses the traditional recommended ordering, but
seems to be a really useful and straightforward way of doing things.
<h2>Why are rulesets important for different parser configurations?</h2>
<p>Custom message parsers, used to handle differnet (and potentially otherwise-invalid)
message formats, can be bound to rulesets. So multiple rulesets can be a very useful
way to handle devices sending messages in different malformed formats in a consistent
way. Unfortunately, this is not uncommon in the syslog world. An in-depth explanation
with configuration sample can be found at the
<a href="rsconf1_rulesetparser.html">$RulesetParser</a> configuration directive.
<h2>Can I use a different Ruleset as the default?</h2>
<p>This is possible by using the
<pre>$DefaultRuleset <name>
</pre>
Directive. Please note, however, that this directive is actually global: that is, it does not
modify the ruleset to which the next input is bound but rather provides a system-wide
default rule set for those inputs that did not explicitly bind to one. As such, the directive
can not be used as a work-around to bind inputs to non-default rulesets that do not support
ruleset binding.
<h2>Examples</h2>
<h3>Split local and remote logging</h3>
<p>Let's say you have a pretty standard system that logs its local messages to the usual
bunch of files that are specified in the default rsyslog.conf. As an example, your rsyslog.conf
might look like this:
<pre>
# ... module loading ...
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
... more ...
</pre>
<p>Now, you want to add receive messages from a remote system and log these to
a special file, but you do not want to have these messages written to the files
specified above. The traditional approach is to add a rule in front of all others that
filters on the message, processes it and then discards it:
<pre>
# ... module loading ...
# process remote messages
:fromhost-ip, isequal, "192.0.2.1" /var/log/remotefile
& ~
# only messages not from 192.0.21 make it past this point
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
... more ...
</pre>
<p>Note the tilde character, which is the discard action!. Also note that we assume that
192.0.2.1 is the sole remote sender (to keep it simple).
<p>With multiple rulesets, we can simply define a dedicated ruleset for the remote reception
case and bind it to the receiver. This may be written as follows:
<pre>
# ... module loading ...
# process remote messages
# define new ruleset and add rules to it:
$RuleSet remote
*.* /var/log/remotefile
# only messages not from 192.0.21 make it past this point
# bind ruleset to tcp listener
$InputTCPServerBindRuleset remote
# and activate it:
$InputTCPServerRun 10514
# switch back to the default ruleset:
$RuleSet RSYSLOG_DefaultRuleset
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
... more ...
</pre>
<p>Here, we need to switch back to the default ruleset after we have defined our custom
one. This is why I recommend a different ordering, which I find more intuitive. The sample
below has it, and it leads to the same results:
<pre>
# ... module loading ...
# at first, this is a copy of the unmodified rsyslog.conf
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
... more ...
# end of the "regular" rsyslog.conf. Now come the new definitions:
# process remote messages
# define new ruleset and add rules to it:
$RuleSet remote
*.* /var/log/remotefile
# bind ruleset to tcp listener
$InputTCPServerBindRuleset remote
# and activate it:
$InputTCPServerRun 10514
</pre>
<p>Here, we do not switch back to the default ruleset, because this is not needed as it is
completely defined when we begin the "remote" ruleset.
<p>Now look at the examples and compare them to the single-ruleset solution. You will notice
that we do <b>not</b> need a real filter in the multi-ruleset case: we can simply use
"*.*" as all messages now means all messages that are being processed by this
rule set and all of them come in via the TCP receiver! This is what makes using multiple
rulesets so much easier.
<h3>Split local and remote logging for three different ports</h3>
<p>This example is almost like the first one, but it extends it a little bit. While it is
very similar, I hope it is different enough to provide a useful example why you may want
to have more than two rulesets.
<p>Again, we would like to use the "regular" log files for local logging, only. But
this time we set up three syslog/tcp listeners, each one listening to a different
port (in this example 10514, 10515, and 10516). Logs received from these receivers shall go into
different files. Also, logs received from 10516 (and only from that port!) with
"mail.*" priority, shall be written into a specif file and <b>not</b> be
written to 10516's general log file.
<p>This is the config:
<pre>
# ... module loading ...
# at first, this is a copy of the unmodified rsyslog.conf
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
... more ...
# end of the "regular" rsyslog.conf. Now come the new definitions:
# process remote messages
#define rulesets first
$RuleSet remote10514
*.* /var/log/remote10514
$RuleSet remote10515
*.* /var/log/remote10515
$RuleSet remote10516
mail.* /var/log/mail10516
& ~
# note that the discard-action will prevent this messag from
# being written to the remote10516 file - as usual...
*.* /var/log/remote10516
# and now define listners bound to the relevant ruleset
$InputTCPServerBindRuleset remote10514
$InputTCPServerRun 10514
$InputTCPServerBindRuleset remote10515
$InputTCPServerRun 10515
$InputTCPServerBindRuleset remote10516
$InputTCPServerRun 10516
</pre>
<p>Note that the "mail.*" rule inside the "remote10516" ruleset does
not affect processing inside any other rule set, including the default rule set.
<h2>Performance</h2>
<h3>Fewer Filters</h3>
<p>No rule processing can be faster than not processing a rule at all. As such, it is useful
for a high performance system to identify disjunct actions and try to split these off to
different rule sets. In the example section, we had a case where three different tcp listeners
need to write to three different files. This is a perfect example of where multiple rule sets
are easier to use and offer more performance. The performance is better simply because there
is no need to check the reception service - instead messages are automatically pushed to the
right rule set and can be processed by very simple rules (maybe even with
"*.*"-filters, the fastest ones available).
<h3>Partitioning of Input Data</h3>
<p>Starting with rsyslog 5.3.4, rulesets permit higher concurrency. They offer the ability
to run on their own "main" queue. What that means is that a own queue is associated
with a specific rule set. That means that inputs bound to that ruleset do no longer need
to compete with each other when they enqueue a data element into the queue. Instead, enqueue
operations can be completed in parallel.
<p>An example: let us assume we have three TCP listeners. Without rulesets, each of them
needs to insert messages into the main message queue. So if each of them wants to
submit a newly arrived message into the queue at the same time, only one can do so
while the others need to wait. With multiple rulesets, its own queue can be created for each
ruleset. If now each listener is bound to its own ruleset, concurrent message submission is
possible. On a machine with a sufficiently large number of corse, this can result in
dramatic performance improvement.
<p>It is highly advised that high-performance systems define a dedicated ruleset, with a
dedicated queue for each of the inputs.
<p>By default, rulesets do <b>not</b> have their own queue. It must be activated via the
<a href="rsconf1_rulesetcreatemainqueue.html">$RulesetCreateMainQueue</a> directive.
<h3>Future Enhancements</h3>
<p>In the long term, multiple rule sets will probably lay the foundation for even better
optimizations. So it is not a bad idea to get aquainted with them.
<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p>
<p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a>
project.<br>
Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and
<a href="http://www.adiscon.com/">Adiscon</a>.
Released under the GNU GPL version 3 or higher.</font></p>
</body></html>
|
