finshed implementation of strgen modules

and also provided four build-in modules for the most common use
cases, hopefully resulting in a speedup of around 5% for typical
rsyslog processing.

2 files changed, 84 insertions, 1 deletions

diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html
index b721c935..57146e88 100644
--- a/doc/rsyslog_conf_modules.html
+++ b/doc/rsyslog_conf_modules.html
@@ -25,6 +25,7 @@ for you by
 <li><a href="rsyslog_conf_modules.html#om">Output Modules</a>
 <li><a href="rsyslog_conf_modules.html#pm">Parser Modules</a>
 <li><a href="rsyslog_conf_modules.html#mm">Message Modification Modules</a>
+<li><a href="rsyslog_conf_modules.html#sm">String Generator Modules</a>
 <li><a href="rsyslog_conf_modules.html#lm">Library Modules</a>
 </ul>
 
@@ -87,6 +88,39 @@ the methods the engine provides. They could be used, for example, to:
 <li>add dynamically computed content to message (fields)
 </ul>
 
+<a name="lm"></a><h2>String Generator Modules</h2>
+<p>String generator modules are used, as the name implies, to generate strings based
+on the message content. They are currently tightly coupled with the template system.
+Their primary use is to speed up template processing by providing a native C
+interface to template generation. These modules exist since 5.5.6. To get an idea
+of the potential speedup, the default file format, when generated by a string generator,
+provides a roughly 5% speedup. For more complex strings, especially those that include
+multiple regular expressions, the speedup may be considerably higher.
+<p>String generator modules are written to a quite simple interface. However, a word of
+caution is due: they access the rsyslog message object via a low-level interface.
+That interface is not guaranteed yet to stay stable. So it may be necessary to 
+modify string generator modules if the interface changes. Obviously, we will not do that
+without good reason, but it may happen.
+<p>Rsyslog comes with a set of core, build-in string generators, which are used
+to provide those default templates that we consider to be time-critical:
+<ul>
+<li>smfile - the default rsyslog file format
+<li>smfwd - the default rsyslog (network) forwarding format
+<li>smtradfile - the traditional syslog file format
+<li>smfwd - the traditional syslog (network) forwarding format
+</ul>
+<p>Note that when you replace these defaults be some custom strings, you will
+loose some performance (around 5%). For typical systems, this is not really relevant.
+But for a high-performance systems, it may be very relevant. To solve that issue, create
+a new string generator module for your custom format, starting out from one of the
+default generators provided. If you can not do this yourself, you may want to 
+contact <a href="mailto:info%40adiscon.com">Adiscon</a> as we offer custom development
+of string generators at a very low price.
+<p>Note that string generator modules can be dynamically loaded. However, the default
+ones provided are so important that they are build right into the executable. But this
+does not need to be done that way (and it is straightforward to do it dynamic).
+
+
 <a name="lm"></a><h2>Library Modules</h2>
 <p>Library modules provide dynamically loadable functionality for parts of rsyslog,
 most often for other loadable modules. They can not be user-configured and are loaded
@@ -106,6 +140,9 @@ parser modules, which generate the in-memory representation of the message and m
 also modify the message itself. The, the internal representation is passed to
 output modules, which may output a message and (with the interfaces newly introduced
 in v5) may also modify messageo object content.
+<p>String generator modules are not included inside this picture, because they are 
+not a required part of the workflow. If used, they operate "in front of" the
+output modules, because they are called during template generation.
 <p>Note that the actual flow is much more complex and depends a lot on queue and
 filter settings. This graphic above is a high-level message flow diagram.
 
@@ -114,7 +151,7 @@ filter settings. This graphic above is a high-level message flow diagram.
 [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p>
 <p><font size="2">This documentation is part of the
 <a href="http://www.rsyslog.com/">rsyslog</a> project.<br>
-Copyright &copy; 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and
+Copyright &copy; 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and
 <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL
 version 3 or higher.</font></p>
 </body>
diff --git a/doc/rsyslog_conf_templates.html b/doc/rsyslog_conf_templates.html
index baa4ce29..23a02049 100644
--- a/doc/rsyslog_conf_templates.html
+++ b/doc/rsyslog_conf_templates.html
@@ -16,6 +16,35 @@ compatible with the stock syslogd formats are hardcoded into rsyslogd.
 So if no template is specified, we use one of these hardcoded
 templates. Search for "template_" in syslogd.c and you will find the
 hardcoded ones.</p>
+<p>Starting with 5.5.6, there are actually two differnt types of template:
+<ul>
+<li>string based
+<li>string-generator module based
+</ul>
+<p><a href="rsyslog_conf_modules.html#sm">String-generator module</a> based templates
+have been introduced in 5.5.6. They permit a string generator, actually a C "program",
+the generate a format. Obviously, it is more work required to code such a generator,
+but the reward is speed improvement. If you do not need the ultimate throughput, you
+can forget about string generators (so most people never need to know what they are).
+You may just be interested in learning that for the most important default formats, 
+rsyslog already contains highly optimized string generators and these are called
+without any need to configure anything. But if you have written (or purchased) a
+string generator module, you need to know how to call it. Each such module has a name,
+which you need to know (look it up in the module doc or ask the developer). Let's assume
+that "mystrgen" is the module name. Then you can define a template for that strgen
+in the following way:
+<blockquote><code>$template MyTemplateName,=mystrgen</code></blockquote>
+(Of course, you must have first loaded the module via $ModLoad).
+<p>The important part is the equal sign: it tells the rsyslog config parser that
+no string follows but a strgen module name.
+<p>There are no additional parameters but the module name supported. This is because
+there is no way to customize anything inside such a "template" other than by
+modifying the code of the string generator.
+
+<p>So for most use cases, string-generator module based templates are <b>not</b>
+the route to take. Usually, us use <b>string based templates</b> instead.
+This is what the rest of the documentation now talks about.
+
 <p>A template consists of a template directive, a name, the
 actual template text and optional options. A sample is:</p>
 <blockquote><code>$template MyTemplateName,"\7Text
@@ -136,6 +165,23 @@ out, but this may happen.</li>
 is meant to be written to a log file. Do <b>not</b> use for production or remote
 forwarding.</li>
 </ul>
+<h3>String-based Template Samples</h3>
+<p>This section provides some sample of what the default formats would
+look as a text-based template. Hopefully, their description is self-explanatory.
+Note that each $Template statement is on a <b>single</b> line, but probably broken
+accross several lines for display purposes by your browsers. Lines are separated by
+empty lines.
+<p><code>
+$template FileFormat,"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
+<br><br>
+$template TraditionalFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
+<br><br>
+$template ForwardFormat,"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
+<br><br>
+$template TraditionalForwardFormat,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
+<br><br>
+$template StdSQLFormat,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
+</code></p>
 
 <p>[<a href="manual.html">manual index</a>]
 [<a href="rsyslog_conf.html">rsyslog.conf</a>]