Merge branch 'v4-beta' into beta

Conflicts: ChangeLog configure.ac doc/manual.html plugins/imuxsock/imuxsock.c template.c tests/Makefile.am tests/parsertest.sh
author: Rainer Gerhards <rgerhards@adiscon.com> 2010-02-10 16:35:12 +0100
committer: Rainer Gerhards <rgerhards@adiscon.com> 2010-02-10 16:35:12 +0100
commit: c8ebe623376bc025bf166ec333ac874c215de816 (patch)
tree: 2e1c6f6a149bce3844ed21da08e5f0695de3fecc
parent: 497bfeb287a75d1fb0f37502d3805307311a574d (diff)
parent: 76b3f2cd875dca755f4beee8dbf525c23d9730cf (diff)
download: rsyslog-c8ebe623376bc025bf166ec333ac874c215de816.tar.gz
rsyslog-c8ebe623376bc025bf166ec333ac874c215de816.tar.xz
rsyslog-c8ebe623376bc025bf166ec333ac874c215de816.zip
16 files changed, 472 insertions, 50 deletions
diff --git a/ChangeLog b/ChangeLog
index 5f647de1..e67ae0de 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -326,7 +326,18 @@ Version 4.7.0  [v4-devel] (rgerhards), 2009-09-??
   See ticket for details: http://bugzilla.adiscon.com/show_bug.cgi?id=150
 - imported changes from 4.5.6 and below
 ---------------------------------------------------------------------------
-Version 4.5.8  [v4-beta] (rgerhards), 2010-01-??
+Version 4.5.8  [v4-beta] (rgerhards), 2010-02-10
+- enhanced doc for using PostgreSQL
+  Thanks to Marc Schiffbauer for the new/updated doc
+- bugfix: property replacer returned invalid parameters under some (unusual)
+  conditions. In extreme cases, this could lead to garbled logs and/or
+  a system failure.
+- bugfix: invalid length returned (often) when using regular expressions
+  inside the property replacer
+- bugfix: submatch regex in property replacer did not honor "return 0 on
+  no match" config case
+- bugfix: imuxsock incorrectly stated inputname "imudp"
+  Thanks to Ryan Lynch for reporting this.
 - (slightly) enhanced support for FreeBSD by setting _PATH_MODDIR to
   the correct value on FreeBSD.
   Thanks to Cristiano for the patch.
diff --git a/doc/Makefile.am b/doc/Makefile.am
index 661e9c57..f1c3f871 100644
--- a/doc/Makefile.am
+++ b/doc/Makefile.am
@@ -19,6 +19,7 @@ html_files = \
 	rsyslog_conf.html \
 	rsyslog-example.conf \
 	rsyslog_mysql.html \
+	rsyslog_pgsql.html \
 	rsyslog_packages.html \
 	rsyslog_high_database_rate.html \
 	rsyslog_php_syslog_ng.html \
diff --git a/doc/manual.html b/doc/manual.html
index 166590c4..936d45b5 100644
--- a/doc/manual.html
+++ b/doc/manual.html
@@ -57,6 +57,7 @@ if you do not read the doc, but doing so will definitely improve your experience
 <li><a href="multi_ruleset.html">using multiple rule sets in rsyslog</a></li>
 <li><a href="rsyslog_stunnel.html">ssl-encrypting syslog with stunnel</a></li>
 <li><a href="rsyslog_mysql.html">writing syslog messages to MySQL (and other databases as well)</a></li>
+<li><a href="rsyslog_pgsql.html">writing syslog messages to PostgreSQL (and other databases as well)</a></li>
 <li><a href="rsyslog_high_database_rate.html">writing massive amounts of syslog messages to a database</a></li>
 <li><a href="rsyslog_reliable_forwarding.html">reliable forwarding to a remote server</a></li>
 <li><a href="rsyslog_php_syslog_ng.html">using
diff --git a/doc/rsconf1_markmessageperiod.html b/doc/rsconf1_markmessageperiod.html
index 2c833339..a6486ba1 100644
--- a/doc/rsconf1_markmessageperiod.html
+++ b/doc/rsconf1_markmessageperiod.html
@@ -7,7 +7,7 @@
 
 <h2>$MarkMessagePeriod</h2>
 <p><b>Type:</b> specific to immark input module</p>
-<p><b>Default:</b> 1800 (20 minutes)</p>
+<p><b>Default:</b> 1200 (20 minutes)</p>
 <p><b>Description:</b></p>
 <p>This specifies when mark messages are to be written to output modules. The 
 time specified is in seconds. Specifying 0 is possible and disables mark 
diff --git a/doc/rsyslog_pgsql.html b/doc/rsyslog_pgsql.html
new file mode 100644
index 00000000..dcb9dc3a
--- /dev/null
+++ b/doc/rsyslog_pgsql.html
@@ -0,0 +1,336 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
+<HTML>
+<HEAD>
+	<META HTTP-EQUIV="CONTENT-TYPE" CONTENT="text/html; charset=utf-8">
+	<TITLE></TITLE>
+	<META NAME="GENERATOR" CONTENT="OpenOffice.org 3.1  (Unix)">
+	<META NAME="AUTHOR" CONTENT="Marc Schiffbauer">
+	<META NAME="CREATED" CONTENT="20100129;15054500">
+	<META NAME="CHANGEDBY" CONTENT="Marc Schiffbauer">
+	<META NAME="CHANGED" CONTENT="20100129;16035000">
+	<META NAME="Info 1" CONTENT="">
+	<META NAME="Info 2" CONTENT="">
+	<META NAME="Info 3" CONTENT="">
+	<META NAME="Info 4" CONTENT="">
+	<STYLE TYPE="text/css">
+	<!--
+		@page { size: 8.27in 11.69in; margin: 0.79in }
+		P { margin-bottom: 0.08in }
+		P.western { font-family: "Arial", sans-serif }
+		H1 { margin-bottom: 0.08in }
+		H1.western { font-family: "Times New Roman", serif }
+		H1.cjk { font-family: "DejaVu Sans" }
+		H1.ctl { font-family: "DejaVu Sans" }
+		H2 { margin-bottom: 0.08in }
+		H2.western { font-family: "Times New Roman", serif }
+		BLOCKQUOTE.western { font-family: "Arial", sans-serif }
+		H3 { margin-bottom: 0.08in }
+		H3.western { font-family: "Times New Roman", serif }
+		A:link { so-language: zxx }
+	-->
+	</STYLE>
+</HEAD>
+<BODY LANG="de-DE" DIR="LTR">
+<H1 CLASS="western"><SPAN LANG="en-US">Writing </SPAN>syslog messages
+to MySQL, PostgreSQL or any other supported Database</H1>
+<P CLASS="western"><FONT SIZE=2><I>Written by </I></FONT><A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php"><FONT SIZE=2><I>Rainer
+Gerhards</I></FONT></A><FONT SIZE=2><I> with some additions by Marc
+Schiffbauer (2008-02-28)</I></FONT></P>
+<H2 CLASS="western">Abstract</H2>
+<P CLASS="western"><SPAN LANG="en-US"><I><B>In this paper, I describe
+how to write </B></I></SPAN><A HREF="http://www.monitorware.com/en/topics/syslog/">syslog</A><SPAN LANG="en-US"><I><B>
+messages to a </B></I></SPAN><A HREF="http://www.mysql.com/">MySQL</A><SPAN LANG="en-US"><I><B>
+or </B></I></SPAN><A HREF="http://www.postgresql.org/">PostgreSQL</A><SPAN LANG="en-US"><I><B>
+database.</B></I></SPAN><SPAN LANG="en-US"><I> Having syslog messages
+in a database is often handy, especially when you intend to set up a
+front-end for viewing them. This paper describes an approach with
+</I></SPAN><A HREF="http://www.rsyslog.com/">rsyslogd</A><SPAN LANG="en-US"><I>,
+an alternative enhanced syslog daemon natively supporting MySQL and
+PostgreSQL. I describe the components needed to be installed and how
+to configure them. Please note that as of this writing, rsyslog
+supports a variety of databases. While this guide is still MySQL- and
+PostgreSQL-focused, you can probably use it together with other ones
+too. You just need to modify a few settings.</I></SPAN></P>
+<H2 CLASS="western">Background</H2>
+<P LANG="en-US" CLASS="western">In many cases, syslog data is simply
+written to text files. This approach has some advantages, most
+notably it is very fast and efficient. However, data stored in text
+files is not readily accessible for real-time viewing and analysis.
+To do that, the messages need to be in a database. There are various
+ways to store syslog messages in a database. For example, some have
+the syslogd write text files which are later feed via a separate
+script into the database. Others have written scripts taking the data
+(via a pipe) from a non-database-aware syslogd and store them as they
+appear. Some others use database-aware syslogds and make them write
+the data directly to the database. In this paper, I use that &quot;direct
+write&quot; approach. I think it is superior, because the syslogd
+itself knows the status of the database connection and thus can
+handle it intelligently (well ... hopefully ;)). I use rsyslogd to
+acomplish this, simply because I have initiated the rsyslog project
+with database-awareness as one goal.</P>
+<P CLASS="western"><SPAN LANG="en-US"><B>One word of caution:</B></SPAN><SPAN LANG="en-US">
+while message storage in the database provides an excellent
+foundation for interactive analysis, it comes at a cost. Database i/o
+is considerably slower than text file i/o. As such, directly writing
+to the database makes sense only if your message volume is low enough
+to allow a) the syslogd, b) the network, and c) the database server
+to catch up with it. Some time ago, I have written a paper on
+</SPAN><A HREF="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php">optimizing
+syslog server performance</A><SPAN LANG="en-US">. While this paper
+talks about Window-based solutions, the ideas in it are generic
+enough to apply here, too. So it might be worth reading if you
+anticipate medium high to high traffic. If you anticipate really high
+traffic (or very large traffic spikes), you should seriously consider
+forgetting about direct database writes - in my opinion, such a
+situation needs either a very specialized system or a different
+approach (the text-file-to-database approach might work better for
+you in this case). </SPAN>
+</P>
+<H2 CLASS="western">Overall System Setup</H2>
+<P CLASS="western"><SPAN LANG="en-US">In this paper, I concentrate on
+the server side. If you are thinking about interactive syslog message
+review, you probably want to centralize syslog. In such a scenario,
+you have multiple machines (the so-called clients) send their data to
+a central machine (called server in this context). While I expect
+such a setup to be typical when you are interested in storing
+messages in the database, I do not describe how to set it up. This is
+beyond the scope of this paper. If you search a little, you will
+probably find many good descriptions on </SPAN><SPAN LANG="en-US">how
+to centralize syslog. If you do that, it might be a good idea to do
+it securely, so you might also be interested in my paper on
+</SPAN><A HREF="http://www.rsyslog.com/doc-rsyslog_stunnel.html">ssl-encrypting
+syslog message transfer</A><SPAN LANG="en-US">.</SPAN></P>
+<P LANG="en-US" CLASS="western">No matter how the messages arrive at
+the server, their processing is always the same. So you can use this
+paper in combination with any description for centralized syslog
+reporting.</P>
+<P CLASS="western"><SPAN LANG="en-US">As I already said, I use
+rsyslogd on the server. It has intrinsic support for talking to the
+supported databases. For obvious reasons, we also need an instance of
+MySQL or PostgreSQL running. To keep us focused, the setup of the
+database itself is also beyond the scope of this paper. I assume that
+you have successfully installed the database and also have a
+front-end at hand to work with it (for example, </SPAN><A HREF="http://www.phpmyadmin.net/">phpMyAdmin</A><SPAN LANG="en-US">
+or </SPAN><A HREF="http://phppgadmin.sourceforge.net/">phpPgAdmin</A><SPAN LANG="en-US">.
+Please make sure that this is installed, actually working and you
+have a basic understanding of how to handle it.</SPAN></P>
+<H2 CLASS="western">Setting up the system</H2>
+<P CLASS="western"><SPAN LANG="en-US">You need to download and
+install rsyslogd first. Obtain it from the </SPAN><A HREF="http://www.rsyslog.com/">rsyslog
+site</A><SPAN LANG="en-US">. Make sure that you disable stock
+syslogd, otherwise you will experience some difficulties. On some
+distributions &nbsp;(Fedora 8 and above, for example), rsyslog may
+already by the default syslogd, in which case you obviously do not
+need to do anything specific. For many others, there are prebuild
+packages available. If you use either, please make sure that you have
+the required database plugins for your database available. It usually
+is a separate package and typically </SPAN><SPAN LANG="en-US"><B>not</B></SPAN><SPAN LANG="en-US">
+installed by default.</SPAN></P>
+<P CLASS="western"><SPAN LANG="en-US">It is important to understand
+how rsyslogd talks to the database. In rsyslogd, there is the concept
+of &quot;templates&quot;. Basically, a template is a string that
+includes some replacement characters, which are called &quot;properties&quot;
+in rsyslog. Properties are accessed via the &quot;</SPAN><A HREF="http://www.rsyslog.com/doc-property_replacer.html">Property
+Replacer</A><SPAN LANG="en-US">&quot;. Simply said, you access
+properties by including their name between percent signs inside the
+template. For example, if the syslog message is &quot;Test&quot;, the
+template &quot;%msg%&quot; would be expanded to &quot;Test&quot;.
+Rsyslogd supports sending template text as a SQL statement to the
+database. As such, the template must be a valid SQL statement. There
+is no limit in what the statement might be, but there are some
+obvious and not so obvious choices. For example, a template &quot;drop
+table xxx&quot; is possible, but does not make an awful lot of sense.
+In practice, you will always use an &quot;insert&quot; statement
+inside the template.</SPAN></P>
+<P LANG="en-US" CLASS="western">An example: if you would just like to
+store the msg part of the full syslog message, you have probably
+created a table &quot;syslog&quot; with a single column &quot;message&quot;.
+In such a case, a good template would be &quot;insert into
+syslog(message) values ('%msg%')&quot;. With the example above, that
+would be expanded to &quot;insert into syslog(message)
+values('Test')&quot;. This expanded string is then sent to the
+database. It's that easy, no special magic. The only thing you must
+ensure is that your template expands to a proper SQL statement and
+that this statement matches your database design.</P>
+<P CLASS="western"><SPAN LANG="en-US">Does that mean you need to
+create database schema yourself and also must fully understand
+rsyslogd's properties? No, that's not needed. Because we anticipated
+that folks are probably more interested in getting things going
+instead of designing them from scratch. So we have provided a default
+schema as well as build-in support for it. This schema also offers an
+additional benefit: rsyslog is part of </SPAN><A HREF="http://www.adiscon.com/en/">Adiscon</A><SPAN LANG="en-US">'s
+</SPAN><A HREF="http://www.monitorware.com/en/">MonitorWare product
+line</A><SPAN LANG="en-US"> (which includes open source and closed
+source members). All of these tools share the same default schema and
+know how to operate on it. For this reason, the default schema is
+also called the &quot;MonitorWare Schema&quot;. If you use it, you
+can simply add </SPAN><A HREF="http://www.phplogcon.org/">phpLogCon,
+a GPLed syslog web interface</A><SPAN LANG="en-US">, to your system
+and have instant interactive access to your database. So there are
+some benefits in using the provided schema.</SPAN></P>
+<P LANG="en-US" CLASS="western">The schema definition is contained in
+the file &quot;createDB.sql&quot;. It comes with the rsyslog package
+and one can be found for each supported database type (in the plugins
+directory). Review it to check that the database name is acceptable
+for you. Be sure to leave the table and field names unmodified,
+because otherwise you need to customize rsyslogd's default sql
+template, which we do not do in this paper. Then, run the script with
+your favorite SQL client. Double-check that the table was
+successfully created.</P>
+<P LANG="en-US" CLASS="western">It is important to note that the
+correct database encoding must be used so that the database will
+accept strings independend of the string encoding. This is an
+important part because it can not be guarantied that all syslog
+messages will have a defined character encoding. This is especially
+true if the rsyslog-Server will collect messages from different
+clients and different products. 
+</P>
+<P LANG="en-US" CLASS="western">For example PostgreSQL may refuse to
+accept messages if you would set the database encoding to “UTF8”
+while a client is sending invalid byte sequences for that encoding. 
+</P>
+<P LANG="en-US" CLASS="western">Database support in rsyslog is
+integrated via loadable plugin modules. To use the database
+functionality, the database plugin must be enabled in the config file
+BEFORE the first database table action is used. This is done by
+placing the</P>
+<BLOCKQUOTE CLASS="western"><CODE>$ModLoad ommysql</CODE></BLOCKQUOTE>
+<P CLASS="western">directive at the begining of /etc/rsyslog.conf for
+MySQL and</P>
+<BLOCKQUOTE CLASS="western"><CODE>$ModLoad ompgsql</CODE></BLOCKQUOTE>
+<P CLASS="western"><CODE><FONT FACE="Arial, sans-serif">for
+PostgreSQL.</FONT></CODE></P>
+<P LANG="en-US" CLASS="western"><FONT FACE="Arial, sans-serif">For
+other databases, use their plugin name (e.g. omoracle).</FONT></P>
+<P CLASS="western">Next, we need to tell rsyslogd to write data to
+the database. As we use the default schema, we do NOT need to define
+a template for this. We can use the hardcoded one (rsyslogd handles
+the proper template linking). So all we need to do e.g. for MySQL is
+add a simple selector line to /etc/rsyslog.conf:</P>
+<BLOCKQUOTE CLASS="western"><CODE>*.*&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+:ommysql:database-server,database-name,database-userid,database-password</CODE></BLOCKQUOTE>
+<P CLASS="western">Again, other databases have other selector names,
+e.g. &quot;:ompgsql:&quot; instead of &quot;:ommysql:&quot;. See the
+output plugin's documentation for details.</P>
+<P LANG="en-US" CLASS="western">In many cases, the database will run
+on the local machine. In this case, you can simply use &quot;127.0.0.1&quot;
+for <I>database-server</I>. This can be especially advisable, if you
+do not need to expose the database to any process outside of the
+local machine. In this case, you can simply bind it to 127.0.0.1,
+which provides a quite secure setup. Of course, rsyslog also supports
+remote database instances. In that case, use the remote server name
+(e.g. mydb.example.com) or IP-address. The <I>database-name</I> by
+default is &quot;Syslog&quot;. If you have modified the default, use
+your name here. <I>Database-userid</I> and <I>-password</I> are the
+credentials used to connect to the database. As they are stored in
+clear text in rsyslog.conf, that user should have only the least
+possible privileges. It is sufficient to grant it INSERT privileges
+to the systemevents table, only. As a side note, it is strongly
+advisable to make the rsyslog.conf file readable by root only - if
+you make it world-readable, everybody could obtain the password (and
+eventually other vital information from it). In our example, let's
+assume you have created a database user named &quot;syslogwriter&quot;
+with a password of &quot;topsecret&quot; (just to say it bluntly:
+such a password is NOT a good idea...). If your database is on the
+local machine, your rsyslog.conf line might look like in this sample:</P>
+<BLOCKQUOTE CLASS="western"><CODE>*.*&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+:ommysql:127.0.0.1,Syslog,syslogwriter,topsecret</CODE></BLOCKQUOTE>
+<P CLASS="western">Save rsyslog.conf, restart rsyslogd - and you
+should see syslog messages being stored in the &quot;systemevents&quot;
+table!</P>
+<P LANG="en-US" CLASS="western">The example line stores every message
+to the database. Especially if you have a high traffic volume, you
+will probably limit the amount of messages being logged. This is easy
+to accomplish: the &quot;write database&quot; action is just a
+regular selector line. As such, you can apply normal selector-line
+filtering. If, for example, you are only interested in messages from
+the mail subsystem, you can use the following selector line:</P>
+<BLOCKQUOTE CLASS="western"><CODE>mail.*&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;:ommysql:127.0.0.1,syslog,syslogwriter,topsecret</CODE></BLOCKQUOTE>
+<P CLASS="western">Review the <A HREF="http://www.rsyslog.com/doc-rsyslog_conf.html">rsyslog.conf</A>
+documentation for details on selector lines and their filtering.</P>
+<P CLASS="western"><SPAN LANG="en-US"><B>You have now completed
+everything necessary to store syslog messages to the a database.</B></SPAN><SPAN LANG="en-US">
+If you would like to try out a front-end, you might want to look at
+</SPAN><A HREF="http://www.phplogcon.org/">phpLogCon</A><SPAN LANG="en-US">,
+which displays syslog data in a browser. As of this writing,
+phpLogCon is not yet a powerful tool, but it's open source, so it
+might be a starting point for your own solution.</SPAN></P>
+<H2 CLASS="western">On Reliability...</H2>
+<P LANG="en-US" CLASS="western">Rsyslogd writes syslog messages
+directly to the database. This implies that the database must be
+available at the time of message arrival. If the database is offline,
+no space is left or something else goes wrong - rsyslogd can not
+write the database record. If rsyslogd is unable to store a message,
+it performs one retry. This is helpful if the database server was
+restarted. In this case, the previous connection was broken but a
+reconnect immediately succeeds. However, if the database is down for
+an extended period of time, an immediate retry does not help.</P>
+<P CLASS="western"><SPAN LANG="en-US">Message loss in this scenario
+can easily be prevented with rsyslog. All you need to do is run the
+database writer in queued mode. This is now described in a generic
+way and I do not intend to duplicate it here. So please be sure to
+read &quot;</SPAN><A HREF="http://www.rsyslog.com/doc-rsyslog_high_database_rate.html">Handling
+a massive syslog database insert rate with Rsyslog</A><SPAN LANG="en-US">&quot;,
+which describes the scenario and also includes configuration
+examples.</SPAN></P>
+<H2 CLASS="western">Conclusion</H2>
+<P LANG="en-US" CLASS="western">With minimal effort, you can use
+rsyslogd to write syslog messages to a database. You can even make it
+absolutely fail-safe and protect it against database server downtime.
+Once the messages are arrived there, you can interactively review and
+analyze them. In practice, the messages are also stored in text files
+for longer-term archival and the databases are cleared out after some
+time (to avoid becoming too slow). If you expect an extremely high
+syslog message volume, storing it in real-time to the database may
+outperform your database server. In such cases, either filter out
+some messages or used queued mode (which in general is recommended
+with databases).</P>
+<P LANG="en-US" CLASS="western">The method outlined in this paper
+provides an easy to setup and maintain solution for most use cases.</P>
+<H3 CLASS="western">Feedback Requested</H3>
+<P CLASS="western">I would appreciate feedback on this paper. If you
+have additional ideas, comments or find bugs, please <A HREF="mailto:rgerhards@adiscon.com">let
+me know</A>.</P>
+<H2 CLASS="western">References and Additional Material</H2>
+<UL>
+	<LI><P CLASS="western" STYLE="margin-bottom: 0in"><A HREF="http://www.rsyslog.com/">www.rsyslog.com</A>
+	- the rsyslog site 
+	</P>
+	<LI><P CLASS="western"><A HREF="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php">Paper
+	on Syslog Server Optimization</A> 
+	</P>
+</UL>
+<H2 CLASS="western">Revision History</H2>
+<UL>
+	<LI><P CLASS="western" STYLE="margin-bottom: 0in">2005-08-02 *
+	<A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer
+	Gerhards</A> * initial version created 
+	</P>
+	<LI><P CLASS="western" STYLE="margin-bottom: 0in">2005-08-03 *
+	<A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer
+	Gerhards</A> * added references to demo site 
+	</P>
+	<LI><P CLASS="western" STYLE="margin-bottom: 0in">2007-06-13 *
+	<A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer
+	Gerhards</A> * removed demo site - was torn down because too
+	expensive for usage count 
+	</P>
+	<LI><P CLASS="western" STYLE="margin-bottom: 0in">2008-02-21 *
+	<A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer
+	Gerhards</A> * updated reliability section, can now be done with
+	on-demand disk queues</P>
+	<LI><P CLASS="western">2008-02-28 * <A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer
+	Gerhards</A> * added info on other databases, updated syntax to more
+	recent one 
+	</P>
+	<LI><P CLASS="western">2010-01-29 * Marc Schiffbauer * added some
+	PostgreSQL stuff, made wording more database generic, fixed some
+	typos</P>
+</UL>
+<H2 CLASS="western">Copyright</H2>
+<P CLASS="western">Copyright (c) 2005-2010 <A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer
+Gerhards</A>, Marc Schiffbauer and <A HREF="http://www.adiscon.com/en/">Adiscon</A>.</P>
+<P CLASS="western"><BR><BR>
+</P>
+</BODY>
+</HTML>
+\ No newline at end of file
diff --git a/doc/src/rsyslog_pgsql.odt b/doc/src/rsyslog_pgsql.odt
new file mode 100644
index 00000000..5034c5fb
--- /dev/null
+++ b/doc/src/rsyslog_pgsql.odt
diff --git a/plugins/imuxsock/imuxsock.c b/plugins/imuxsock/imuxsock.c
index cf6df56c..17c85afb 100644
--- a/plugins/imuxsock/imuxsock.c
+++ b/plugins/imuxsock/imuxsock.c
@@ -362,7 +362,7 @@ CODESTARTwillRun
 
 	/* we need to create the inputName property (only once during our lifetime) */
 	CHKiRet(prop.Construct(&pInputName));
-	CHKiRet(prop.SetString(pInputName, UCHAR_CONSTANT("imudp"), sizeof("imudp") - 1));
+	CHKiRet(prop.SetString(pInputName, UCHAR_CONSTANT("imuxsock"), sizeof("imuxsock") - 1));
 	CHKiRet(prop.ConstructFinalize(pInputName));
 
 finalize_it:
diff --git a/plugins/ompgsql/createDB.sql b/plugins/ompgsql/createDB.sql
index facdcaa0..2f72a0a6 100644
--- a/plugins/ompgsql/createDB.sql
+++ b/plugins/ompgsql/createDB.sql
@@ -1,4 +1,4 @@
-CREATE DATABASE "Syslog";
+CREATE DATABASE 'Syslog' WITH ENCODING 'SQL_ASCII';
 \c Syslog;
 CREATE TABLE SystemEvents
 (
diff --git a/runtime/msg.c b/runtime/msg.c
index cc2f0b69..2fd4cff7 100644
--- a/runtime/msg.c
+++ b/runtime/msg.c
@@ -2152,6 +2152,10 @@ static uchar *getNOW(eNOWType eNow)
  * be used in selector line processing.
  * rgerhards 2005-09-15
  */
+/* a quick helper to save some writing: */
+#define RET_OUT_OF_MEMORY { *pbMustBeFreed = 0;\
+	*pPropLen = sizeof("**OUT OF MEMORY**") - 1; \
+	return(UCHAR_CONSTANT("**OUT OF MEMORY**"));}
 uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
                  propid_t propID, size_t *pPropLen,
 		 unsigned short *pbMustBeFreed)
@@ -2214,8 +2218,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 		case PROP_PRI_TEXT:
 			pBuf = MALLOC(20 * sizeof(uchar));
 			if(pBuf == NULL) {
-				*pbMustBeFreed = 0;
-				return UCHAR_CONSTANT("**OUT OF MEMORY**");
+				RET_OUT_OF_MEMORY;
 			} else {
 				*pbMustBeFreed = 1;
 				pRes = (uchar*)textpri((char*)pBuf, 20, getPRIi(pMsg));
@@ -2260,49 +2263,49 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 			break;
 		case PROP_SYS_NOW:
 			if((pRes = getNOW(NOW_NOW)) == NULL) {
-				return UCHAR_CONSTANT("**OUT OF MEMORY**");
+				RET_OUT_OF_MEMORY;
 			} else
 				*pbMustBeFreed = 1;	/* all of these functions allocate dyn. memory */
 			break;
 		case PROP_SYS_YEAR:
 			if((pRes = getNOW(NOW_YEAR)) == NULL) {
-				return UCHAR_CONSTANT("**OUT OF MEMORY**");
+				RET_OUT_OF_MEMORY;
 			} else
 				*pbMustBeFreed = 1;	/* all of these functions allocate dyn. memory */
 			break;
 		case PROP_SYS_MONTH:
 			if((pRes = getNOW(NOW_MONTH)) == NULL) {
-				return UCHAR_CONSTANT("**OUT OF MEMORY**");
+				RET_OUT_OF_MEMORY;
 			} else
 				*pbMustBeFreed = 1;	/* all of these functions allocate dyn. memory */
 			break;
 		case PROP_SYS_DAY:
 			if((pRes = getNOW(NOW_DAY)) == NULL) {
-				return UCHAR_CONSTANT("**OUT OF MEMORY**");
+				RET_OUT_OF_MEMORY;
 			} else
 				*pbMustBeFreed = 1;	/* all of these functions allocate dyn. memory */
 			break;
 		case PROP_SYS_HOUR:
 			if((pRes = getNOW(NOW_HOUR)) == NULL) {
-				return UCHAR_CONSTANT("**OUT OF MEMORY**");
+				RET_OUT_OF_MEMORY;
 			} else
 				*pbMustBeFreed = 1;	/* all of these functions allocate dyn. memory */
 			break;
 		case PROP_SYS_HHOUR:
 			if((pRes = getNOW(NOW_HHOUR)) == NULL) {
-				return UCHAR_CONSTANT("**OUT OF MEMORY**");
+				RET_OUT_OF_MEMORY;
 			} else
 				*pbMustBeFreed = 1;	/* all of these functions allocate dyn. memory */
 			break;
 		case PROP_SYS_QHOUR:
 			if((pRes = getNOW(NOW_QHOUR)) == NULL) {
-				return UCHAR_CONSTANT("**OUT OF MEMORY**");
+				RET_OUT_OF_MEMORY;
 			} else
 				*pbMustBeFreed = 1;	/* all of these functions allocate dyn. memory */
 			break;
 		case PROP_SYS_MINUTE:
 			if((pRes = getNOW(NOW_MINUTE)) == NULL) {
-				return UCHAR_CONSTANT("**OUT OF MEMORY**");
+				RET_OUT_OF_MEMORY;
 			} else
 				*pbMustBeFreed = 1;	/* all of these functions allocate dyn. memory */
 			break;
@@ -2314,6 +2317,8 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 			 * error message unreadable. rgerhards, 2007-07-10
 			 */
 			dbgprintf("invalid property id: '%d'\n", propID);
+			*pbMustBeFreed = 0;
+			*pPropLen = sizeof("**INVALID PROPERTY NAME**") - 1;
 			return UCHAR_CONSTANT("**INVALID PROPERTY NAME**");
 	}
 
@@ -2372,8 +2377,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 			if(pBuf == NULL) {
 				if(*pbMustBeFreed == 1)
 					free(pRes);
-				*pbMustBeFreed = 0;
-				return UCHAR_CONSTANT("**OUT OF MEMORY**");
+				RET_OUT_OF_MEMORY;
 			}
 			/* now copy */
 			memcpy(pBuf, pFld, iLen);
@@ -2390,6 +2394,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 			if(*pbMustBeFreed == 1)
 				free(pRes);
 			*pbMustBeFreed = 0;
+			*pPropLen = sizeof("**FIELD NOT FOUND**") - 1;
 			return UCHAR_CONSTANT("**FIELD NOT FOUND**");
 		}
 	} else if(pTpe->data.field.iFromPos != 0 || pTpe->data.field.iToPos != 0) {
@@ -2418,8 +2423,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 			if(pBuf == NULL) {
 				if(*pbMustBeFreed == 1)
 					free(pRes);
-				*pbMustBeFreed = 0;
-				return UCHAR_CONSTANT("**OUT OF MEMORY**");
+				RET_OUT_OF_MEMORY;
 			}
 			pSb = pRes;
 			if(iFrom) {
@@ -2449,9 +2453,15 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 	} else {
 		/* Check for regular expressions */
 		if (pTpe->data.field.has_regex != 0) {
-			if (pTpe->data.field.has_regex == 2)
+			if (pTpe->data.field.has_regex == 2) {
 				/* Could not compile regex before! */
+				if (*pbMustBeFreed == 1) {
+					free(pRes);
+					*pbMustBeFreed = 0;
+				}
+				*pPropLen = sizeof("**NO MATCH** **BAD REGULAR EXPRESSION**") - 1;
 				return UCHAR_CONSTANT("**NO MATCH** **BAD REGULAR EXPRESSION**");
+			}
 
 			dbgprintf("string to match for regex is: %s\n", pRes);
 
@@ -2491,12 +2501,16 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 							free(pRes);
 							*pbMustBeFreed = 0;
 						}
-						if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_DFLTSTR)
-							return UCHAR_CONSTANT("**NO MATCH**");
-						else if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_ZERO)
-							return UCHAR_CONSTANT("0");
-						else
-							return UCHAR_CONSTANT("");
+						if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_DFLTSTR) {
+							bufLen = sizeof("**NO MATCH**") - 1;
+							pRes = UCHAR_CONSTANT("**NO MATCH**");
+						} else if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_ZERO) {
+							bufLen = 1;
+							pRes = UCHAR_CONSTANT("0");
+						} else {
+							bufLen = 0;
+							pRes = UCHAR_CONSTANT("");
+						}
 					}
 				} else {
 					/* Match- but did it match the one we wanted? */
@@ -2507,10 +2521,16 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 								free(pRes);
 								*pbMustBeFreed = 0;
 							}
-							if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_DFLTSTR)
-								return UCHAR_CONSTANT("**NO MATCH**");
-							else
-								return UCHAR_CONSTANT("");
+							if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_DFLTSTR) {
+								bufLen = sizeof("**NO MATCH**") - 1;
+								pRes = UCHAR_CONSTANT("**NO MATCH**");
+							} else if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_ZERO) {
+								bufLen = 1;
+								pRes = UCHAR_CONSTANT("0");
+							} else {
+								bufLen = 0;
+								pRes = UCHAR_CONSTANT("");
+							}
 						}
 					}
 					/* OK, we have a usable match - we now need to malloc pB */
@@ -2524,13 +2544,12 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 					if (pB == NULL) {
 						if (*pbMustBeFreed == 1)
 							free(pRes);
-						*pbMustBeFreed = 0;
-						return UCHAR_CONSTANT("**OUT OF MEMORY**");
+						RET_OUT_OF_MEMORY;
 					}
 
 					/* Lets copy the matched substring to the buffer */
 					memcpy(pB, pRes + iOffs +  pmatch[pTpe->data.field.iSubMatchToUse].rm_so, iLenBuf);
-					bufLen = iLenBuf - 1;
+					bufLen = iLenBuf;
 					pB[iLenBuf] = '\0';/* terminate string, did not happen before */
 
 					if (*pbMustBeFreed == 1)
@@ -2548,6 +2567,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 					free(pRes);
 					*pbMustBeFreed = 0;
 				}
+				*pPropLen = sizeof("***REGEXP NOT AVAILABLE***") - 1;
 				return UCHAR_CONSTANT("***REGEXP NOT AVAILABLE***");
 			}
 		}
@@ -2580,8 +2600,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 			if(pB == NULL) {
 				if(*pbMustBeFreed == 1)
 					free(pRes);
-				*pbMustBeFreed = 0;
-				return UCHAR_CONSTANT("**OUT OF MEMORY**");
+				RET_OUT_OF_MEMORY;
 			}
 			pSrc = pRes;
 			while(*pSrc) {
@@ -2627,8 +2646,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 				if(pDst == NULL) {
 					if(*pbMustBeFreed == 1)
 						free(pRes);
-					*pbMustBeFreed = 0;
-					return UCHAR_CONSTANT("**OUT OF MEMORY**");
+					RET_OUT_OF_MEMORY;
 				}
 				for(pSrc = pRes; *pSrc; pSrc++) {
 					if(!iscntrl((int) *pSrc))
@@ -2663,8 +2681,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 				if(pDst == NULL) {
 					if(*pbMustBeFreed == 1)
 						free(pRes);
-					*pbMustBeFreed = 0;
-					return UCHAR_CONSTANT("**OUT OF MEMORY**");
+					RET_OUT_OF_MEMORY;
 				}
 				for(pSrc = pRes; *pSrc; pSrc++) {
 					if(iscntrl((int) *pSrc))
@@ -2703,8 +2720,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 				if(pB == NULL) {
 					if(*pbMustBeFreed == 1)
 						free(pRes);
-					*pbMustBeFreed = 0;
-					return UCHAR_CONSTANT("**OUT OF MEMORY**");
+					RET_OUT_OF_MEMORY;
 				}
 				while(*pRes) {
 					if(iscntrl((int) *pRes)) {
@@ -2749,8 +2765,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 				if(pDst == NULL) {
 					if(*pbMustBeFreed == 1)
 						free(pRes);
-					*pbMustBeFreed = 0;
-					return UCHAR_CONSTANT("**OUT OF MEMORY**");
+					RET_OUT_OF_MEMORY;
 				}
 				for(pSrc = pRes; *pSrc; pSrc++) {
 					if(*pSrc != '/')
@@ -2785,8 +2800,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 				if(pDst == NULL) {
 					if(*pbMustBeFreed == 1)
 						free(pRes);
-					*pbMustBeFreed = 0;
-					return UCHAR_CONSTANT("**OUT OF MEMORY**");
+					RET_OUT_OF_MEMORY;
 				}
 				for(pSrc = pRes; *pSrc; pSrc++) {
 					if(*pSrc == '/')
@@ -2840,8 +2854,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 				/* ok, original copy, need a private one */
 				pB = MALLOC((iLn + 1) * sizeof(uchar));
 				if(pB == NULL) {
-					*pbMustBeFreed = 0;
-					return UCHAR_CONSTANT("**OUT OF MEMORY**");
+					RET_OUT_OF_MEMORY;
 				}
 				memcpy(pB, pRes, iLn - 1);
 				pRes = pB;
@@ -2860,6 +2873,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 	if(pTpe->data.field.options.bCSV) {
 		/* we need to obtain a private copy, as we need to at least add the double quotes */
 		int iBufLen;
+		int i;
 		uchar *pBStart;
 		uchar *pDst;
 		uchar *pSrc;
@@ -2871,10 +2885,10 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 		if(pDst == NULL) {
 			if(*pbMustBeFreed == 1)
 				free(pRes);
-			*pbMustBeFreed = 0;
-			return UCHAR_CONSTANT("**OUT OF MEMORY**");
+			RET_OUT_OF_MEMORY;
 		}
 		pSrc = pRes;
+		i = 0;
 		*pDst++ = '"'; /* starting quote */
 		while(*pSrc) {
 			if(*pSrc == '"')
diff --git a/template.c b/template.c
index 23d4b62e..d74da475 100644
--- a/template.c
+++ b/template.c
@@ -567,7 +567,7 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl)
 		cstrDestruct(&pStrB);
 		return 1;
 	}
-	cstrDestruct(&pStrB);
+	cstrDestruct(&pStrB); /* no longer needed, now use ID */
 
 	/* Check frompos, if it has an R, then topos should be a regex */
 	if(*p == ':') {
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 716207e1..cf8183a7 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -114,6 +114,10 @@ EXTRA_DIST= 1.rstest 2.rstest 3.rstest err1.rstest \
 	   testsuites/reallife.parse1 \
 	   testsuites/parse2.conf \
 	   testsuites/reallife.parse2 \
+	   testsuites/parse3.conf \
+	   testsuites/reallife.parse3 \
+	   testsuites/parse_invld_regex.conf \
+	   testsuites/samples.parse_invld_regex \
 	   testsuites/omod-if-array.conf \
 	   testsuites/1.omod-if-array \
 	   testsuites/1.field1 \
diff --git a/tests/parsertest.sh b/tests/parsertest.sh
index 470d9375..fb392db3 100755
--- a/tests/parsertest.sh
+++ b/tests/parsertest.sh
@@ -4,10 +4,18 @@ source $srcdir/diag.sh nettester parse1 udp
 source $srcdir/diag.sh nettester parse1 tcp
 source $srcdir/diag.sh nettester parse2 udp
 source $srcdir/diag.sh nettester parse2 tcp
+source $srcdir/diag.sh nettester parse3 udp
+source $srcdir/diag.sh nettester parse3 tcp
+source $srcdir/diag.sh nettester parse_invld_regex udp
+source $srcdir/diag.sh nettester parse_invld_regex tcp
 
 echo \[parsertest.sh]: redoing tests in IPv4-only mode
 source $srcdir/diag.sh nettester parse1 udp -4
 source $srcdir/diag.sh nettester parse1 tcp -4
 source $srcdir/diag.sh nettester parse2 udp -4
 source $srcdir/diag.sh nettester parse2 tcp -4
-source $srcdir/diag.sh init
+source $srcdir/diag.sh nettester parse3 udp -4
+source $srcdir/diag.sh nettester parse3 tcp -4
+source $srcdir/diag.sh nettester parse_invld_regex udp -4
+source $srcdir/diag.sh nettester parse_invld_regex tcp -4
+source $srcdir/diag.sh exit
diff --git a/tests/testsuites/parse3.conf b/tests/testsuites/parse3.conf
new file mode 100644
index 00000000..d5cf77d9
--- /dev/null
+++ b/tests/testsuites/parse3.conf
@@ -0,0 +1,8 @@
+$ModLoad ../plugins/omstdout/.libs/omstdout
+$IncludeConfig nettest.input.conf	# This picks the to be tested input from the test driver!
+
+$ErrorMessagesToStderr off
+
+# use a special format that we can easily parse in expect
+$Template output,"%timereported:1:$:date-rfc3339,csv%, %hostname:::csv%, %programname:::csv%, %syslogtag:R,ERE,0,BLANK:[0-9]+--end:csv%, %syslogseverity:::csv%, %msg:::drop-last-lf,csv%\n" 
+*.* :omstdout:;output
diff --git a/tests/testsuites/parse_invld_regex.conf b/tests/testsuites/parse_invld_regex.conf
new file mode 100644
index 00000000..736aae59
--- /dev/null
+++ b/tests/testsuites/parse_invld_regex.conf
@@ -0,0 +1,8 @@
+$ModLoad ../plugins/omstdout/.libs/omstdout
+$IncludeConfig nettest.input.conf	# This picks the to be tested input from the test driver!
+
+$ErrorMessagesToStderr off
+
+# use a special format that we can easily parse in expect
+$Template output,"%timereported:1:$:date-rfc3339,csv%, %hostname:::csv%, %programname:::csv%, %syslogtag:R,ERE,0,BLANK:[0-9+--end:csv%, %syslogseverity:::csv%, %msg:::drop-last-lf,csv%\n" 
+*.* :omstdout:;output
diff --git a/tests/testsuites/reallife.parse3 b/tests/testsuites/reallife.parse3
new file mode 100644
index 00000000..465635bd
--- /dev/null
+++ b/tests/testsuites/reallife.parse3
@@ -0,0 +1,15 @@
+# New tests should be added to this file if there is no specific
+# reason for not doing that. Initially, we could only handle one test
+# case per file, but this restriction has been removed some time ago.
+# So it is less troublesome (and easier to overlook) to have all related
+# tests in a single file.
+# This file contains a lot of real-life samples (of course mangled so
+# that they can not be traced back to the original submitter). Note
+# that IP addr 192.0.2.1 is specifically set aside for testing and
+# documentation by IANA.
+# rgerhards, 2009-10-19
+<175>Oct 16 2009 23:47:31 hostname tag This is a message
+"2009-10-16T23:47:31+01:00", "hostname", "tag", "", "7", " This is a message"
+#
+<175>Oct 16 2009 23:47:31 hostname tag[1234] This is a message
+"2009-10-16T23:47:31+01:00", "hostname", "tag", "1234", "7", " This is a message"
diff --git a/tests/testsuites/samples.parse_invld_regex b/tests/testsuites/samples.parse_invld_regex
new file mode 100644
index 00000000..9ac2c3aa
--- /dev/null
+++ b/tests/testsuites/samples.parse_invld_regex
@@ -0,0 +1,16 @@
+# New tests should be added to this file if there is no specific
+# reason for not doing that. Initially, we could only handle one test
+# case per file, but this restriction has been removed some time ago.
+# So it is less troublesome (and easier to overlook) to have all related
+# tests in a single file.
+# the actual message is not important. There is an error inside the conf
+# file, and all messages will trigger the same problem.
+# NOTE: it is correct that the "BAD REGULAR EXPRESSION" error message is
+# *NOT* run through the rest of the propety replace, in specific through
+# the CSV escaper. We do not do this because it could potentially lead
+# to an obfuscated error message, and thus making problems hard to find. As
+# this is a real error case, there is no problem in not obeying to the 
+# configured format.
+# rgerhards, 2010-02-08
+<175>Feb 08 2008 23:47:31 hostname tag This is a message
+"2008-02-08T23:47:31+01:00", "hostname", "tag", **NO MATCH** **BAD REGULAR EXPRESSION**, "7", " This is a message"
author	Rainer Gerhards <rgerhards@adiscon.com>	2010-02-10 16:35:12 +0100
committer	Rainer Gerhards <rgerhards@adiscon.com>	2010-02-10 16:35:12 +0100
commit	c8ebe623376bc025bf166ec333ac874c215de816 (patch)
tree	2e1c6f6a149bce3844ed21da08e5f0695de3fecc
parent	497bfeb287a75d1fb0f37502d3805307311a574d (diff)
parent	76b3f2cd875dca755f4beee8dbf525c23d9730cf (diff)
download	rsyslog-c8ebe623376bc025bf166ec333ac874c215de816.tar.gz rsyslog-c8ebe623376bc025bf166ec333ac874c215de816.tar.xz rsyslog-c8ebe623376bc025bf166ec333ac874c215de816.zip