9 files changed, 257 insertions, 25 deletions
diff --git a/doc/dev_oplugins.html b/doc/dev_oplugins.html
new file mode 100644
index 00000000..5bfc974c
--- /dev/null
+++ b/doc/dev_oplugins.html
@@ -0,0 +1,178 @@
+<html>
+<head>
+<title>writing rsyslog output plugins (developer's guide)</title>
+</head>
+<body>
+<h1>Writing Rsyslog Output Plugins</h1>
+<p>This page is the begin of some developer documentation for writing output
+plugins. Doing so is quite easy (and that was a design goal), but there currently
+is only sparse documentation on the process available. I was tempted NOT to 
+write this guide here because I know I will most probably not be able to
+write a complete guide.
+<p>However, I finally concluded that it may be better to have same information
+and pointers than to have nothing.
+<h2>Getting Started and Samples</h2>
+<p>The best to get started with rsyslog plugin development is by looking at
+existing plugins. All that start with "om" are <b>o</b>utput <b>m</b>odules. That
+means they are primarily thought of being message sinks. In theory, however, output
+plugins may aggergate other functionality, too. Nobody has taken this route so far
+so if you would like to do that, it is highly suggested to post your plan on the
+rsyslog mailing list, first (so that we can offer advise).
+<p>The rsyslog distribution tarball contains two plugins that are extremely well
+targeted for getting started:
+<ul>
+<li>omtemplate
+<li>omstdout
+</ul>
+Plugin omtemplate was specifically created to provide a copy template for new output
+plugins. It is bare of real functionality but has ample comments. Even if you decide
+to start from another plugin (or even from scratch), be sure to read omtemplate source
+and comments first. The omstdout is primarily a testing aide, but offers support for
+the two different parameter-passing conventions plugins can use (plus the way to 
+differentiate between the two). It also is not bare of functionaly, only mostly
+bare of it ;). But you can actually execute it and play with it.
+<p>In any case, you should also read the comments in ./runtime/module-template.h. 
+Output plugins are build based on a large set of code-generating macros. These 
+macros handle most of the plumbing needed by the interface. As long as no
+special callback to rsyslog is needed (it typically is not), an output plugin does
+not really need to be aware that it is executed by rsyslog. As a plug-in programmer,
+you can (in most cases) "code as usual". However, all macros and entry points need to be
+provided and thus reading the code comments in the files mentioned is highly suggested.
+<p>In short, the best idea is to start with a template. Let's assume you start by
+copying omtemplate. Then, the basic steps you need to do are:
+<ul>
+<li>cp ./plugins/omtemplate ./plugins/your-plugin
+<li>mv cd ./plugins/your-plugin
+<li>vi Makefile.am, adjust to your-plugin
+<li>mv omtemplate.c your-plugin.c
+<li>cd ../..
+<li>vi Makefile.am configure.ac
+<br>serach for omtemplate, copy and modify (follow comments)
+</ul>
+<p>Basically, this is all you need to do ... Well, except, of course, coding
+your plugin ;). For testing, you need rsyslog's debugging support. Some useful
+information is given in "<a href="troubleshoot.html">troubleshooting rsyslog</a>
+from the doc set.
+<h2>Special Topics</h2>
+<h3>Threading</h3>
+<p>Rsyslog uses massive parallel processing and multithreading. However, a plugin's entry
+points are guaranteed to be never called concurrently <b>for the same action</b>.
+That means your plugin must be able to be called concurrently by two or more 
+threads, but you can be sure that for the same instance no concurrent calls
+happen. This is guaranteed by the interface specification and the rsyslog core
+guards against multiple concurrent calls. An instance, in simple words, is one
+that shares a single instanceData structure.
+<p>So as long as you do not mess around with global data, you do not need
+to think about multithreading (and can apply a purely sequential programming
+methodology).
+<p>Please note that duringt the configuraton parsing stage of execution, access to
+global variables for the configuration system is safe. In that stage, the core will
+only call sequentially into the plugin.
+<h3>Getting Message Data</h3>
+<p>The doAction() entry point of your plugin is provided with messages to be processed.
+It will only be activated after filtering and all other conditions, so you do not need
+to apply any other conditional but can simply process the message.
+<p>Note that you do NOT receive the full internal representation of the message
+object. There are various (including historical) reasons for this and, among
+others, this is a design decision based on security.
+<p>Your plugin will only receive what the end user has configured in a $template
+statement. However, starting with 4.1.6, there are two ways of receiving the
+template content. The default mode, and in most cases sufficient and optimal,
+is to receive a single string with the expanded template. As I said, this is usually
+optimal, think about writing things to files, emailing content or forwarding it.
+<p>The important philosophy is that a plugin should <b>never</b> reformat any
+of such strings - that would either remove the user's ability to fully control 
+message formats or it would lead to duplicating code that is already present in the
+core. If you need some formatting that is not yet present in the core, suggest it
+to the rsyslog project, best done by sending a patch ;), and we will try hard to
+get it into the core (so far, we could accept all such suggestions - no promise, though).
+<p>If a single string seems not suitable for your application, the plugin can also
+request access to the template components. The typical use case seems to be databases, where
+you would like to access properties via specific fields. With that mode, you receive a
+char ** array, where each array element points to one field from the template (from
+left to right). Fields start at arrray index 0 and a NULL pointer means you have
+reached the end of the array (the typical Unix "poor man's linked list in an array"
+design). Note, however, that each of the individual components is a string. It is 
+not a date stamp, number or whatever, but a string. This is because rsyslog processes
+strings (from a high-level design look at it) and so this is the natural data type.
+Feel free to convert to whatever you need, but keep in mind that malformed packets
+may have lead to field contents you'd never expected...
+<p>If you like to use the array-based parameter passing method, think that it
+is only available in rsyslog 4.1.6 and above. If you can accept that your plugin
+will not be working with previous versions, you do not need to handle pre 4.1.6 cases.
+However, it would be "nice" if you shut down yourself in these cases - otherwise the
+older rsyslog core engine will pass you a string where you expect the array of pointers,
+what most probably results in a segfault. To check whether or not the core supports the
+functionality, you can use this code sequence:
+<pre>
+<code>
+BEGINmodInit()
+	rsRetVal localRet;
+	rsRetVal (*pomsrGetSupportedTplOpts)(unsigned long *pOpts);
+	unsigned long opts;
+	int bArrayPassingSupported;		/* does core support template passing as an array? */
+CODESTARTmodInit
+	*ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */
+CODEmodInit_QueryRegCFSLineHdlr
+	/* check if the rsyslog core supports parameter passing code */
+	bArrayPassingSupported = 0;
+	localRet = pHostQueryEtryPt((uchar*)"OMSRgetSupportedTplOpts", &pomsrGetSupportedTplOpts);
+	if(localRet == RS_RET_OK) {
+		/* found entry point, so let's see if core supports array passing */
+		CHKiRet((*pomsrGetSupportedTplOpts)(&opts));
+		if(opts & OMSR_TPL_AS_ARRAY)
+			bArrayPassingSupported = 1;
+	} else if(localRet != RS_RET_ENTRY_POINT_NOT_FOUND) {
+		ABORT_FINALIZE(localRet); /* Something else went wrong, what is not acceptable */
+	}
+	DBGPRINTF("omstdout: array-passing is %ssupported by rsyslog core.\n", bArrayPassingSupported ? "" : "not ");
+
+	if(!bArrayPassingSupported) {
+		DBGPRINTF("rsyslog core too old, shutting down this plug-in\n");
+		ABORT_FINALIZE(RS_RET_ERR);
+	}
+
+</code>
+</pre>
+<p>The code first checks if the core supports the OMSRgetSupportedTplOpts() API (which is
+also not present in all versions!) and, if so, queries the core if the OMSR_TPL_AS_ARRAY mode
+is supported. If either does not exits, the core is too old for this functionality. The sample
+snippet above then shuts down, but a plugin may instead just do things different. In
+omstdout, you can see how a plugin may deal with the situation.
+<p><b>In any case, it is recommended that at least a graceful shutdown is made and the
+array-passing capability not blindly be used.</b> In such cases, we can not guard the
+plugin from segfaulting and if the plugin (as currently always) is run within 
+rsyslog's process space, that results in a segfault for rsyslog. So do not do this.
+<h3>Batching of Messages</h3>
+<p>With the current plugin interface, each message is passed via a separate call to the plugin.
+This is annoying and costs performance in some uses cases (primarily for database outputs).
+However, that's the way it (currently) is, no easy way around it. There are some ideas
+to implement batching capabilities inside the rsyslog core, but without that the only
+resort is to do it inside your plugin yourself. You are not prohibited from doing so.
+There are some consequences, though: most importantly, the rsyslog core is no longer
+intersted in messages that it passed to a plugin. As such, it will not try to make sure
+the message is not lost before it was ultimately processed (because rsyslog, due to
+doAction() returning successfully, thinks the message *was* ultimately processed).
+<p>When the rsyslog core receives batching capabilities, this will be implemented in
+a way that is fully compatible to the existing plugin interface. While we have not yet
+thought about the implementation, that will probably mean that some new interfaces
+or options be used to turn on batching capabilities.
+<h3>Licensing</h3>
+<p>From the rsyslog point of view, plugins constitute separate projects. As such,
+we think plugins are not required to be compatible with GPLv3. However, this is
+no legal advise. If you intend to release something under a non-GPLV3 compatible license
+it is probably best to consult with your lawyer.
+<p>Most importantly, and this is definite, the rsyslog team does not expect
+or require you to contribute your plugin to the rsyslog project (but of course
+we are happy if you do).
+<h2>Copyright</h2>
+<p>Copyright (c) 2009 <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> 
+and <a href="http://www.adiscon.com/en/">Adiscon</a>.</p>
+<p>Permission is granted to copy, distribute and/or modify this document under 
+the terms of the GNU Free Documentation License, Version 1.2 or any later 
+version published by the Free Software Foundation; with no Invariant Sections, 
+no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be 
+viewed at <a href="http://www.gnu.org/copyleft/fdl.html">
+http://www.gnu.org/copyleft/fdl.html</a>.</p>
+</body>
+</html>
diff --git a/doc/features.html b/doc/features.html
index 336b31cc..626ff65d 100644
--- a/doc/features.html
+++ b/doc/features.html
@@ -95,6 +95,9 @@ via custom plugins</li>
 <li> an easy-to-write to plugin interface</li>
 <li> ability to send SNMP trap messages</li>
 <li> ability to filter out messages based on sequence of arrival</li>
+<li>support for comma-seperated-values (CSV) output generation
+(via the "csv" property replace option). The
+CSV format supported is that from RFC 4180.</li>
 <li>support for arbitrary complex boolean, string and
 arithmetic expressions in message filters</li>
 </ul>
diff --git a/doc/manual.html b/doc/manual.html
index 99a90594..51271701 100644
--- a/doc/manual.html
+++ b/doc/manual.html
@@ -19,7 +19,7 @@ rsyslog support</a> available directly from the source!</p>
 <p><b>Please visit the <a href="http://www.rsyslog.com/sponsors">rsyslog sponsor's page</a>
 to honor the project sponsors or become one yourself!</b> We are very grateful for any help towards the
 project goals.</p>
-<p><b>This documentation is for version 4.1.5 (devel branch) of rsyslog.</b>
+<p><b>This documentation is for version 4.1.6 (devel branch) of rsyslog.</b>
 Visit the <i> <a href="http://www.rsyslog.com/doc-status.html">rsyslog status page</a></i></b> to obtain current
 version information and project status.
 </p><p><b>If you like rsyslog, you might
@@ -64,7 +64,11 @@ the syslog priority (severity and facility) to the log file</a></li>
 syslog sender over NAT</a> (online only)</li>
 <li><a href="gssapi.html">an overview and howto of rsyslog gssapi support</a></li>
 <li><a href="debug.html">debug support in rsyslog</a></li>
-<li><a href="dev_queue.html">the rsyslog message queue object (developer's view)</a></li>
+<li>Developer Documentation
+  <ul>
+  <li><a href="dev_oplugins.html">writing rsyslog output plugins</a></li>
+  <li><a href="dev_queue.html">the rsyslog message queue object (developer's view)</a></li>
+  </ul></li>
 </ul>
 <p>Our <a href="history.html">rsyslog history</a>
 page is for you if you would like to learn a little more
@@ -83,6 +87,7 @@ wiki</a>, a community resource which&nbsp;includes <a href="http://wiki.rsyslog.
 online documentation (most current version only)</a></li>
 
 <li><a href="http://kb.monitorware.com/rsyslog-f40.html">rsyslog discussion forum - use this for technical support</a></li>
+<li><a href="http://www.rsyslog.com/Topic8.phtml">rsyslog video tutorials</a></li>
 <li><a href="http://www.rsyslog.com/Topic4.phtml">rsyslog change log</a></li>
 <li><a href="http://www.rsyslog.com/Topic3.phtml">rsyslog FAQ</a></li> 
 <li><a href="http://www.monitorware.com/en/syslog-enabled-products/">syslog device configuration guide</a> (off-site)</li>
diff --git a/doc/property_replacer.html b/doc/property_replacer.html
index baf053f2..a6e9b518 100644
--- a/doc/property_replacer.html
+++ b/doc/property_replacer.html
@@ -314,6 +314,18 @@ case-insensitive. Currently, the following options are defined:
 <td>convert property text to uppercase only</td>
 </tr>
 <tr>
+<td valign="top"><b>csv</b></td>
+<td>formats the resulting field (after all modifications) in CSV format
+as specified in <a href="http://www.ietf.org/rfc/rfc4180.txt">RFC 4180</a>.
+Rsyslog will always use double quotes. Note that in order to have full CSV-formatted
+text, you need to define a proper template. An example is this one:
+<br>$template csvline,"%syslogtag:::csv%,%msg:::csv%"
+<br>Most importantly, you need to provide the commas between the fields
+inside the template.
+<br><i>This feature was introduced in rsyslog 4.1.6.</i>
+</td>
+</tr>
+<tr>
 <td><b>drop-last-lf</b></td>
 <td>The last LF in the message (if any), is dropped.
 Especially useful for PIX.</td>
@@ -411,7 +423,7 @@ syntax</a>, this is where you actually use the property replacer.</li>
 [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p>
 <p><font size="2">This documentation is part of the
 <a href="http://www.rsyslog.com/">rsyslog</a> project.<br>
-Copyright &copy; 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and
+Copyright &copy; 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and
 <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL
 version 2 or higher.</font></p>
 
diff --git a/doc/rsconf1_maxopenfiles.html b/doc/rsconf1_maxopenfiles.html
new file mode 100644
index 00000000..b6c9cc0e
--- /dev/null
+++ b/doc/rsconf1_maxopenfiles.html
@@ -0,0 +1,35 @@
+<html>
+<head>
+<title>$MaxOpenFiles - rsyslog.conf file</title>
+</head>
+<body>
+<a href="rsyslog_conf_global.html">[rsyslog configuration directive overview]</a>
+
+<h2>$MaxOpenFiles</h2>
+<p><b>Available Since:</b> 4.3.0</p>
+<p><b>Type:</b> global configuration directive</p>
+<p><b>Default:</b> <i>operating system default</i></p>
+<p><b>Description:</b></p>
+<p>Set the maximum number of files that the rsyslog process can have open at any given
+time. Note that this includes open tcp sockets, so this setting is the upper limit for
+the number of open TCP connections as well. If you expect a large nubmer of concurrent
+connections, it is suggested that the number is set to the max number connected plus 1000.
+Please note that each dynafile also requires up to 100 open file handles.
+<p>The setting is similar to running "ulimit -n number-of-files".
+<p>Please note that depending on permissions and operating system configuration, the 
+setrlimit() request issued by rsyslog may fail, in which case the previous limit is kept
+in effect. Rsyslog will emit a warning message in this case.
+<p><b>Sample:</b></p>
+<p><code><b>$MaxOpenFiles 2000</b></code></p>
+<p><b>Bugs:</b></p>
+<p>For some reason, this settings seems not to work on all platforms. If you experience
+problems, please let us know so that we can (hopefully) narrow down the issue.
+<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual 
+index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p>
+<p><font size="2">This documentation is part of the
+<a href="http://www.rsyslog.com/">rsyslog</a> project.<br>
+Copyright &copy; 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and
+<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL 
+version 3 or higher.</font></p>
+</body>
+</html>
diff --git a/doc/rsyslog_conf.html b/doc/rsyslog_conf.html
index 852d95b5..6990c6bd 100644
--- a/doc/rsyslog_conf.html
+++ b/doc/rsyslog_conf.html
@@ -26,7 +26,7 @@ Lines can be continued by specifying a backslash ("\") as the last
 character of the line. There is a hard-coded maximum line length of 4K.
 If you need lines larger than that, you need to change compile-time
 settings inside rsyslog and recompile.
-<h2><a href="rsyslog_conf_global.html">Global Directives</a></h2>
+<h2><a href="rsyslog_conf_global.html">Configuration Directives</a></h2>
 <h2>Basic Structure</h2>
 <p>Rsyslog supports standard sysklogd's configuration file format
 and extends it. So in general, you can take a "normal" syslog.conf and
@@ -74,9 +74,9 @@ such features is available in rsyslogd, only.</p>
 [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p>
 <p><font size="2">This documentation is part of the
 <a href="http://www.rsyslog.com/">rsyslog</a> project.<br>
-Copyright &copy; 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and
+Copyright &copy; 2008,2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and
 <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL
-version 2 or higher.</font></p>
+version 3 or higher.</font></p>
 </body>
 </html>
 >
diff --git a/doc/rsyslog_conf_global.html b/doc/rsyslog_conf_global.html
index d011bd2b..3e33f0da 100644
--- a/doc/rsyslog_conf_global.html
+++ b/doc/rsyslog_conf_global.html
@@ -1,14 +1,14 @@
 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
-<html><head><title>Global Directives - rsyslog.conf</title></head>
+<html><head><title>Configuration Directives - rsyslog.conf</title></head>
 <body>
 <p>This is a part of the rsyslog.conf documentation.</p>
 <a href="rsyslog_conf.html">back</a>
-<h2>Global Directives</h2>
-<p>All global directives need to be specified on a line by their
-own and must start with a dollar-sign. Here is a list in alphabetical
-order. Follow links for a description.</p>
-<p>Please note that not all directives here are actually global. Some affect
-only the next action. This documentation will be changed soon.
+<h2>Configuration Directives</h2>
+<p>All configuration directives need to be specified on a line by their
+own and must start with a dollar-sign. Note that those starting with
+the word "Action" modify the next action and should be specified
+in front of it.
+<p>Here is a list in alphabetical order. Follow links for a description.</p>
 <p>Not all directives have an in-depth description right now.
 Default values for them are in bold. A more in-depth description will
 appear as implementation progresses.
@@ -180,6 +180,7 @@ instead of UDP (plain TCP syslog, RELP). This resolves the UDP stack size restri
 <br>Note that 2k, the current default, is the smallest size that must be
 supported in order to be compliant to the upcoming new syslog RFC series.
 </li>
+<li><a href="rsconf1_maxopenfiles.html">$MaxOpenFiles</a></li>
 <li><a href="rsconf1_moddir.html">$ModDir</a></li>
 <li><a href="rsconf1_modload.html">$ModLoad</a></li>
 <li><b>$RepeatedMsgContainsOriginalMsg</b> [on/<b>off</b>] - "last message repeated n times" messages, if generated,
@@ -214,7 +215,6 @@ the value, the less precise the timestamp.
 <li><a href="droppriv.html">$PrivDropToGroupID</a></li>
 <li><a href="droppriv.html">$PrivDropToUser</a></li>
 <li><a href="droppriv.html">$PrivDropToUserID</a></li>
-</ul>
 <li><a href="rsconf1_umask.html">$UMASK</a></li>
 </ul>
 <p><b>Where &lt;size_nbr&gt; is specified above,</b>
@@ -235,7 +235,7 @@ point of view, "1,,0.0.,.,0" also has the value 1000. </p>
 <a href="http://www.rsyslog.com/">rsyslog</a> project.<br>
 Copyright &copy; 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and
 <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL
-version 2 or higher.</font></p>
+version 3 or higher.</font></p>
 </body>
 </html>
 
diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html
index 890a55c8..a281d9e7 100644
--- a/doc/rsyslog_conf_modules.html
+++ b/doc/rsyslog_conf_modules.html
@@ -8,10 +8,9 @@
 number of modules. Here is the entry point to their documentation and
 what they do (list is currently not complete)</p>
 <ul>
-<li><a href="omsnmp.html">omsnmp</a> - SNMP
-trap output module</li>
-<li><a href="omrelp.html">omrelp</a> - RELP
-output module</li>
+<li><a href="omsnmp.html">omsnmp</a> - SNMP trap output module</li>
+<li><a href="omstdout.html">omtdout</a> - stdout output module (mainly a test tool)</li>
+<li><a href="omrelp.html">omrelp</a> - RELP output module</li>
 <li>omgssapi - output module for GSS-enabled syslog</li>
 <li><a href="ommysql.html">ommysql</a> - output module for MySQL</li>
 <li>ompgsql - output module for PostgreSQL</li>
diff --git a/doc/status.html b/doc/status.html
index 59fd0809..dae94884 100644
--- a/doc/status.html
+++ b/doc/status.html
@@ -2,19 +2,19 @@
 <html><head><title>rsyslog status page</title></head>
 <body>
 <h2>rsyslog status page</h2>
-<p>This page reflects the status as of 2009-02-02.</p>
+<p>This page reflects the status as of 2009-04-03.</p>
 <h2>Current Releases</h2>
 
 <p><b>development:</b> 4.1.5 [2009-03-11] -
 <a href="http://www.rsyslog.com/Article349.phtml">change log</a> -
 <a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-150.phtml">download</a>
 
-<br><b>beta:</b> 3.21.10 [2009-02-02] -
-<a href="http://www.rsyslog.com/Article344.phtml">change log</a> -
-<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-148.phtml">download</a></p>
+<br><b>beta:</b> 3.21.11 [2009-04-03] -
+<a href="http://www.rsyslog.com/Article358.phtml">change log</a> -
+<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-152.phtml">download</a></p>
 
-<p><b>v3 stable:</b> 3.20.3 [2009-01-19] - <a href="http://www.rsyslog.com/Article339.phtml">change log</a> -
-<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-146.phtml">download</a>
+<p><b>v3 stable:</b> 3.20.3 [2009-04-02] - <a href="http://www.rsyslog.com/Article356.phtml">change log</a> -
+<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-151.phtml">download</a>
 
 <br><b>v2 stable:</b> 2.0.6 [2008-08-07] - <a href="http://www.rsyslog.com/Article266.phtml">change log</a> -
 <a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-125.phtml">download</a>