From d9e64c16e52357bae1eb00fc8403c4e63d6365ca Mon Sep 17 00:00:00 2001 From: Rainer Gerhards Date: Fri, 4 Jun 2010 12:45:31 +0200 Subject: finshed implementation of strgen modules and also provided four build-in modules for the most common use cases, hopefully resulting in a speedup of around 5% for typical rsyslog processing. --- doc/rsyslog_conf_modules.html | 39 +++++++++++++++++++++++++++++++++- doc/rsyslog_conf_templates.html | 46 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html index b721c935..57146e88 100644 --- a/doc/rsyslog_conf_modules.html +++ b/doc/rsyslog_conf_modules.html @@ -25,6 +25,7 @@ for you by
  • Output Modules
  • Parser Modules
  • Message Modification Modules +
  • String Generator Modules
  • Library Modules @@ -87,6 +88,39 @@ the methods the engine provides. They could be used, for example, to:
  • add dynamically computed content to message (fields) +

    String Generator Modules

    +

    String generator modules are used, as the name implies, to generate strings based +on the message content. They are currently tightly coupled with the template system. +Their primary use is to speed up template processing by providing a native C +interface to template generation. These modules exist since 5.5.6. To get an idea +of the potential speedup, the default file format, when generated by a string generator, +provides a roughly 5% speedup. For more complex strings, especially those that include +multiple regular expressions, the speedup may be considerably higher. +

    String generator modules are written to a quite simple interface. However, a word of +caution is due: they access the rsyslog message object via a low-level interface. +That interface is not guaranteed yet to stay stable. So it may be necessary to +modify string generator modules if the interface changes. Obviously, we will not do that +without good reason, but it may happen. +

    Rsyslog comes with a set of core, build-in string generators, which are used +to provide those default templates that we consider to be time-critical: +

    +

    Note that when you replace these defaults be some custom strings, you will +loose some performance (around 5%). For typical systems, this is not really relevant. +But for a high-performance systems, it may be very relevant. To solve that issue, create +a new string generator module for your custom format, starting out from one of the +default generators provided. If you can not do this yourself, you may want to +contact Adiscon as we offer custom development +of string generators at a very low price. +

    Note that string generator modules can be dynamically loaded. However, the default +ones provided are so important that they are build right into the executable. But this +does not need to be done that way (and it is straightforward to do it dynamic). + +

    Library Modules

    Library modules provide dynamically loadable functionality for parts of rsyslog, most often for other loadable modules. They can not be user-configured and are loaded @@ -106,6 +140,9 @@ parser modules, which generate the in-memory representation of the message and m also modify the message itself. The, the internal representation is passed to output modules, which may output a message and (with the interfaces newly introduced in v5) may also modify messageo object content. +

    String generator modules are not included inside this picture, because they are +not a required part of the workflow. If used, they operate "in front of" the +output modules, because they are called during template generation.

    Note that the actual flow is much more complex and depends a lot on queue and filter settings. This graphic above is a high-level message flow diagram. @@ -114,7 +151,7 @@ filter settings. This graphic above is a high-level message flow diagram. [rsyslog site]

    This documentation is part of the rsyslog project.
    -Copyright © 2008, 2009 by Rainer Gerhards and +Copyright © 2008-2010 by Rainer Gerhards and Adiscon. Released under the GNU GPL version 3 or higher.

    diff --git a/doc/rsyslog_conf_templates.html b/doc/rsyslog_conf_templates.html index baa4ce29..23a02049 100644 --- a/doc/rsyslog_conf_templates.html +++ b/doc/rsyslog_conf_templates.html @@ -16,6 +16,35 @@ compatible with the stock syslogd formats are hardcoded into rsyslogd. So if no template is specified, we use one of these hardcoded templates. Search for "template_" in syslogd.c and you will find the hardcoded ones.

    +

    Starting with 5.5.6, there are actually two differnt types of template: +

    +

    String-generator module based templates +have been introduced in 5.5.6. They permit a string generator, actually a C "program", +the generate a format. Obviously, it is more work required to code such a generator, +but the reward is speed improvement. If you do not need the ultimate throughput, you +can forget about string generators (so most people never need to know what they are). +You may just be interested in learning that for the most important default formats, +rsyslog already contains highly optimized string generators and these are called +without any need to configure anything. But if you have written (or purchased) a +string generator module, you need to know how to call it. Each such module has a name, +which you need to know (look it up in the module doc or ask the developer). Let's assume +that "mystrgen" is the module name. Then you can define a template for that strgen +in the following way: +

    $template MyTemplateName,=mystrgen
    +(Of course, you must have first loaded the module via $ModLoad). +

    The important part is the equal sign: it tells the rsyslog config parser that +no string follows but a strgen module name. +

    There are no additional parameters but the module name supported. This is because +there is no way to customize anything inside such a "template" other than by +modifying the code of the string generator. + +

    So for most use cases, string-generator module based templates are not +the route to take. Usually, us use string based templates instead. +This is what the rest of the documentation now talks about. +

    A template consists of a template directive, a name, the actual template text and optional options. A sample is:

    $template MyTemplateName,"\7Text @@ -136,6 +165,23 @@ out, but this may happen.
    • is meant to be written to a log file. Do not use for production or remote forwarding. +

    String-based Template Samples

    +

    This section provides some sample of what the default formats would +look as a text-based template. Hopefully, their description is self-explanatory. +Note that each $Template statement is on a single line, but probably broken +accross several lines for display purposes by your browsers. Lines are separated by +empty lines. +

    +$template FileFormat,"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" +

    +$template TraditionalFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" +

    +$template ForwardFormat,"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" +

    +$template TraditionalForwardFormat,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" +

    +$template StdSQLFormat,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL +

    [manual index] [rsyslog.conf] -- cgit