diff options
author | Rainer Gerhards <rgerhards@adiscon.com> | 2010-06-04 12:45:31 +0200 |
---|---|---|
committer | Rainer Gerhards <rgerhards@adiscon.com> | 2010-06-04 12:45:31 +0200 |
commit | d9e64c16e52357bae1eb00fc8403c4e63d6365ca (patch) | |
tree | 9a37b8395ff1e8996950328696054b24ddffd564 /doc | |
parent | 527bfcea5c9ca5c8414620a022f097d4e53af784 (diff) | |
download | rsyslog-d9e64c16e52357bae1eb00fc8403c4e63d6365ca.tar.gz rsyslog-d9e64c16e52357bae1eb00fc8403c4e63d6365ca.tar.xz rsyslog-d9e64c16e52357bae1eb00fc8403c4e63d6365ca.zip |
finshed implementation of strgen modules
and also provided four build-in modules for the most common use
cases, hopefully resulting in a speedup of around 5% for typical
rsyslog processing.
Diffstat (limited to 'doc')
-rw-r--r-- | doc/rsyslog_conf_modules.html | 39 | ||||
-rw-r--r-- | doc/rsyslog_conf_templates.html | 46 |
2 files changed, 84 insertions, 1 deletions
diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html index b721c935..57146e88 100644 --- a/doc/rsyslog_conf_modules.html +++ b/doc/rsyslog_conf_modules.html @@ -25,6 +25,7 @@ for you by <li><a href="rsyslog_conf_modules.html#om">Output Modules</a> <li><a href="rsyslog_conf_modules.html#pm">Parser Modules</a> <li><a href="rsyslog_conf_modules.html#mm">Message Modification Modules</a> +<li><a href="rsyslog_conf_modules.html#sm">String Generator Modules</a> <li><a href="rsyslog_conf_modules.html#lm">Library Modules</a> </ul> @@ -87,6 +88,39 @@ the methods the engine provides. They could be used, for example, to: <li>add dynamically computed content to message (fields) </ul> +<a name="lm"></a><h2>String Generator Modules</h2> +<p>String generator modules are used, as the name implies, to generate strings based +on the message content. They are currently tightly coupled with the template system. +Their primary use is to speed up template processing by providing a native C +interface to template generation. These modules exist since 5.5.6. To get an idea +of the potential speedup, the default file format, when generated by a string generator, +provides a roughly 5% speedup. For more complex strings, especially those that include +multiple regular expressions, the speedup may be considerably higher. +<p>String generator modules are written to a quite simple interface. However, a word of +caution is due: they access the rsyslog message object via a low-level interface. +That interface is not guaranteed yet to stay stable. So it may be necessary to +modify string generator modules if the interface changes. Obviously, we will not do that +without good reason, but it may happen. +<p>Rsyslog comes with a set of core, build-in string generators, which are used +to provide those default templates that we consider to be time-critical: +<ul> +<li>smfile - the default rsyslog file format +<li>smfwd - the default rsyslog (network) forwarding format +<li>smtradfile - the traditional syslog file format +<li>smfwd - the traditional syslog (network) forwarding format +</ul> +<p>Note that when you replace these defaults be some custom strings, you will +loose some performance (around 5%). For typical systems, this is not really relevant. +But for a high-performance systems, it may be very relevant. To solve that issue, create +a new string generator module for your custom format, starting out from one of the +default generators provided. If you can not do this yourself, you may want to +contact <a href="mailto:info%40adiscon.com">Adiscon</a> as we offer custom development +of string generators at a very low price. +<p>Note that string generator modules can be dynamically loaded. However, the default +ones provided are so important that they are build right into the executable. But this +does not need to be done that way (and it is straightforward to do it dynamic). + + <a name="lm"></a><h2>Library Modules</h2> <p>Library modules provide dynamically loadable functionality for parts of rsyslog, most often for other loadable modules. They can not be user-configured and are loaded @@ -106,6 +140,9 @@ parser modules, which generate the in-memory representation of the message and m also modify the message itself. The, the internal representation is passed to output modules, which may output a message and (with the interfaces newly introduced in v5) may also modify messageo object content. +<p>String generator modules are not included inside this picture, because they are +not a required part of the workflow. If used, they operate "in front of" the +output modules, because they are called during template generation. <p>Note that the actual flow is much more complex and depends a lot on queue and filter settings. This graphic above is a high-level message flow diagram. @@ -114,7 +151,7 @@ filter settings. This graphic above is a high-level message flow diagram. [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body> diff --git a/doc/rsyslog_conf_templates.html b/doc/rsyslog_conf_templates.html index baa4ce29..23a02049 100644 --- a/doc/rsyslog_conf_templates.html +++ b/doc/rsyslog_conf_templates.html @@ -16,6 +16,35 @@ compatible with the stock syslogd formats are hardcoded into rsyslogd. So if no template is specified, we use one of these hardcoded templates. Search for "template_" in syslogd.c and you will find the hardcoded ones.</p> +<p>Starting with 5.5.6, there are actually two differnt types of template: +<ul> +<li>string based +<li>string-generator module based +</ul> +<p><a href="rsyslog_conf_modules.html#sm">String-generator module</a> based templates +have been introduced in 5.5.6. They permit a string generator, actually a C "program", +the generate a format. Obviously, it is more work required to code such a generator, +but the reward is speed improvement. If you do not need the ultimate throughput, you +can forget about string generators (so most people never need to know what they are). +You may just be interested in learning that for the most important default formats, +rsyslog already contains highly optimized string generators and these are called +without any need to configure anything. But if you have written (or purchased) a +string generator module, you need to know how to call it. Each such module has a name, +which you need to know (look it up in the module doc or ask the developer). Let's assume +that "mystrgen" is the module name. Then you can define a template for that strgen +in the following way: +<blockquote><code>$template MyTemplateName,=mystrgen</code></blockquote> +(Of course, you must have first loaded the module via $ModLoad). +<p>The important part is the equal sign: it tells the rsyslog config parser that +no string follows but a strgen module name. +<p>There are no additional parameters but the module name supported. This is because +there is no way to customize anything inside such a "template" other than by +modifying the code of the string generator. + +<p>So for most use cases, string-generator module based templates are <b>not</b> +the route to take. Usually, us use <b>string based templates</b> instead. +This is what the rest of the documentation now talks about. + <p>A template consists of a template directive, a name, the actual template text and optional options. A sample is:</p> <blockquote><code>$template MyTemplateName,"\7Text @@ -136,6 +165,23 @@ out, but this may happen.</li> is meant to be written to a log file. Do <b>not</b> use for production or remote forwarding.</li> </ul> +<h3>String-based Template Samples</h3> +<p>This section provides some sample of what the default formats would +look as a text-based template. Hopefully, their description is self-explanatory. +Note that each $Template statement is on a <b>single</b> line, but probably broken +accross several lines for display purposes by your browsers. Lines are separated by +empty lines. +<p><code> +$template FileFormat,"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" +<br><br> +$template TraditionalFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" +<br><br> +$template ForwardFormat,"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" +<br><br> +$template TraditionalForwardFormat,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" +<br><br> +$template StdSQLFormat,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL +</code></p> <p>[<a href="manual.html">manual index</a>] [<a href="rsyslog_conf.html">rsyslog.conf</a>] |