Merge branch 'v3-stable' into beta

Conflicts:

	doc/property_replacer.html
	tools/syslogd.c

9 files changed, 112 insertions, 13 deletions

diff --git a/ChangeLog b/ChangeLog
index 6e5715db..fb88920b 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -171,11 +171,32 @@ Version 3.18.1 (rgerhards), 2008-07-??
   is just a simple addition of faciltity and severity). I have changed
   this to use own, consistent, code for PRI calculation. [Backport from
   3.19.10]
+- bugfix: remove PRI part from kernel message if it is present
+  Thanks to Michael Biebl for reporting this bug
 - bugfix: mark messages were not correctly written to text log files
   the markmessageinterval was not correctly propagated to all places
   where it was needed. This resulted in rsyslog using the default
   (20 minutes) in some code pathes, what looked to the user like mark
   messages were never written.
+- added a new property replacer option "sp-if-no-1st-sp" to cover
+  a problem with RFC 3164 based interpreation of tag separation. While
+  it is a generic approach, it fixes a format problem introduced in
+  3.18.0, where kernel messages no longer had a space after the tag.
+  This is done by a modifcation of the default templates.
+  Please note that this may affect some messages where there intentionally
+  is no space between the tag and the first character of the message
+  content. If so, this needs to be worked around via a specific
+  template. However, we consider this scenario to be quite remote and,
+  even if it exists, it is not expected that it will actually cause
+  problems with log parsers (instead, we assume the new default template
+  behaviour may fix previous problems with log parsers due to the 
+  missing space).
+- bugfix: imklog module was not correctly compiled for GNU/kFreeBSD.
+  Thanks to Petr Salinger for the patch
+- doc bugfix: property replacer options secpath-replace and
+  secpath-drop were not documented
+- doc bugfix: fixed some typos in rsyslog.conf man page
+- fixed typo in source comment  - thanks to Rio Fujita
 ---------------------------------------------------------------------------
 Version 3.18.0 (rgerhards), 2008-07-11
 - begun a new v3-stable based on former 3.17.4 beta plus patches to
diff --git a/configure.ac b/configure.ac
index 3e8720f4..13ef63d0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -27,6 +27,11 @@ case "${host}" in
     AC_DEFINE([OS_BSD], [1], [Indicator for a BSD OS])
     os_type="bsd"
   ;;
+  *-*-kfreebsd*)
+    # kernel is FreeBSD, but userspace is glibc - i.e. like linux
+    # do not DEFINE OS_BSD
+    os_type="bsd"
+  ;;
 esac
 
 AC_DEFINE_UNQUOTED([HOSTENV], "$host", [the host environment, can be queried via a system variable])
diff --git a/doc/property_replacer.html b/doc/property_replacer.html
index 2675e8fb..367c8add 100644
--- a/doc/property_replacer.html
+++ b/doc/property_replacer.html
@@ -288,7 +288,7 @@ Especially useful for PIX.</td>
 <td>just the subseconds of a timestamp (always 0 for a low precision timestamp)</td>
 </tr>
 <tr>
-<td><b>escape-cc</b></td>
+<td valign="top"><b>escape-cc</b></td>
 <td>replace control characters (ASCII value 127 and values
 less then 32) with an escape sequence. The sequnce is
 "#&lt;charval&gt;" where charval is the 3-digit decimal value
@@ -298,19 +298,53 @@ Note: using this option requires that <a href="rsconf1_escapecontrolcharacterson
 is set to off.</td>
 </tr>
 <tr>
-<td><b>space-cc</b></td>
+<td valign="top"><b>space-cc</b></td>
 <td>replace control characters by spaces<br>
 Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a>
 is set to off.</td>
 </tr>
 <tr>
-<td><b>drop-cc</b></td>
+<td valign="top"><b>drop-cc</b></td>
 <td>drop control characters - the resulting string will
 neither contain control characters, escape sequences nor any other
 replacement character like space.<br>
 Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a>
 is set to off.</td>
 </tr>
+<tr>
+<td valign="top"><b>sp-if-no-1st-sp</b></td>
+<td>This option looks scary and should probably not be used by a user. For any field
+given, it returns either a single space character or no character at all. Field content
+is never returned. A space is returned if (and only if) the first character of the
+field's content is NOT a space. This option is kind of a hack to solve a problem rooted
+in RFC 3164: 3164 specifies no delimiter between the syslog tag sequence and the actual
+message text. Almost all implementation in fact delemit the two by a space. As of
+RFC 3164, this space is part of the message text itself. This leads to a problem when
+building the message (e.g. when writing to disk or forwarding). Should a delimiting
+space be included if the message does not start with one? If not, the tag is immediately
+followed by another non-space character, which can lead some log parsers to misinterpret
+what is the tag and what the message. The problem finally surfaced when the klog module
+was restructured and the tag correctly written. It exists with other message sources,
+too. The solution was the introduction of this special property replacer option. Now,
+the default template can contain a conditional space, which exists only if the
+message does not start with one. While this does not solve all issues, it should
+work good enough in the far majority of all cases. If you read this text and have
+no idea of what it is talking about - relax: this is a good indication you will never
+need this option. Simply forget about it ;)
+</td>
+</tr>
+<tr>
+<td valign="top"><b>secpath-drop</b></td>
+<td>Drops slashes inside the field (e.g. "a/b" becomes "ab").
+Useful for secure pathname generation (with dynafiles).
+</td>
+</tr>
+<tr>
+<td valign="top"><b>secpath-replace</b></td>
+<td>Replace slashes inside the field by an underscore. (e.g. "a/b" becomes "a_b").
+Useful for secure pathname generation (with dynafiles).
+</td>
+</tr>
 </tbody>
 </table>
 <h2>Further Links</h2>
diff --git a/plugins/imklog/imklog.c b/plugins/imklog/imklog.c
index f6935a34..1fbc2874 100644
--- a/plugins/imklog/imklog.c
+++ b/plugins/imklog/imklog.c
@@ -144,6 +144,7 @@ parsePRI(uchar **ppSz, int *piPri)
 
 	/* OK, we have a valid PRI */
 	*piPri = i;
+	*ppSz = pSz + 1; /* update msg ptr to position after PRI */
 
 finalize_it:
 	RETiRet;
diff --git a/runtime/msg.c b/runtime/msg.c
index a5881f50..fdeae077 100644
--- a/runtime/msg.c
+++ b/runtime/msg.c
@@ -1988,6 +1988,32 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 #endif /* #ifdef FEATURE_REGEXP */
 	}
 
+	/* now check if we need to do our "SP if first char is non-space" hack logic */
+	if(*pRes && pTpe->data.field.options.bSPIffNo1stSP) {
+		char *pB;
+		uchar cFirst = *pRes;
+
+		/* here, we always destruct the buffer and return a new one */
+		pB = (char *) malloc(2 * sizeof(char));
+		if(pB == NULL) {
+			if(*pbMustBeFreed == 1)
+				free(pRes);
+			*pbMustBeFreed = 0;
+			return "**OUT OF MEMORY**";
+		}
+		pRes = pB;
+		*pbMustBeFreed = 1;
+
+		if(cFirst == ' ') {
+			/* if we have a SP, we must return an empty string */
+			*pRes = '\0'; /* empty */
+		} else {
+			/* if it is no SP, we need to return one */
+			*pRes = ' ';
+			*(pRes+1) = '\0';
+		}
+	}
+
 	if(*pRes) {
 		/* case conversations (should go after substring, because so we are able to
 		 * work on the smallest possible buffer).
diff --git a/template.c b/template.c
index b94af60e..2fe23710 100644
--- a/template.c
+++ b/template.c
@@ -446,6 +446,8 @@ static void doOptions(unsigned char **pp, struct templateEntry *pTpe)
 			pTpe->data.field.eCaseConv = tplCaseConvLower;
 		 } else if(!strcmp((char*)Buf, "uppercase")) {
 			pTpe->data.field.eCaseConv = tplCaseConvUpper;
+		 } else if(!strcmp((char*)Buf, "sp-if-no-1st-sp")) {
+			pTpe->data.field.options.bSPIffNo1stSP = 1;
 		 } else if(!strcmp((char*)Buf, "escape-cc")) {
 			pTpe->data.field.options.bEscapeCC = 1;
 		 } else if(!strcmp((char*)Buf, "drop-cc")) {
@@ -1085,6 +1087,15 @@ void tplPrintList(void)
 				if(pTpe->data.field.options.bSpaceCC) {
 				  	dbgprintf("[replace control-characters with space] ");
 				}
+				if(pTpe->data.field.options.bSecPathDrop) {
+				  	dbgprintf("[slashes are dropped] ");
+				}
+				if(pTpe->data.field.options.bSecPathReplace) {
+				  	dbgprintf("[slashes are replaced by '_'] ");
+				}
+				if(pTpe->data.field.options.bSPIffNo1stSP) {
+				  	dbgprintf("[SP iff no first SP] ");
+				}
 				if(pTpe->data.field.options.bDropLastLF) {
 				  	dbgprintf("[drop last LF in msg] ");
 				}
diff --git a/template.h b/template.h
index 3f35ebca..6e889c58 100644
--- a/template.h
+++ b/template.h
@@ -93,6 +93,7 @@ struct templateEntry {
 				unsigned bDropLastLF: 1;	/* drop last LF char in msg (PIX!) */
 				unsigned bSecPathDrop: 1;		/* drop slashes, replace dots, empty string */
 				unsigned bSecPathReplace: 1;		/* replace slashes, replace dots, empty string */
+				unsigned bSPIffNo1stSP: 1;		/* replace slashes, replace dots, empty string */
 			} options;		/* options as bit fields */
 		} field;
 	} data;
diff --git a/tools/rsyslog.conf.5 b/tools/rsyslog.conf.5
index 1c47f535..dd80fca0 100644
--- a/tools/rsyslog.conf.5
+++ b/tools/rsyslog.conf.5
@@ -28,8 +28,8 @@ file is the main configuration file for the
 which logs system messages on *nix systems.  This file specifies rules
 for logging.  For special features see the
 .BR rsyslogd (8)
-manpage. Ryslog.conf is backward-compatible with sysklogd's syslog.conf file. So if you migrate
-from syklogd you can rename it and it should work.
+manpage. Rsyslog.conf is backward-compatible with sysklogd's syslog.conf file. So if you migrate
+from sysklogd you can rename it and it should work.
 
 .B Note that this version of rsyslog ships with extensive documentation in html format.
 This is provided in the ./doc subdirectory and probably
@@ -55,7 +55,7 @@ Output module for GSS-enabled syslog
 .I ommysql
 Output module for MySQL
 .TP
-.I omprelp
+.I omrelp
 Output module for the reliable RELP protocol (prevents message loss). 
 For details, see below at imrelp and the html documentation.
 It can be used like this:
@@ -69,7 +69,7 @@ Output module for PostgreSQL
 .TP
 .I omlibdbi
 Generic database output module (Firebird/Interbase, MS SQL, Sybase,
-SQLLite, Ingres, Oracle, mSQL)
+SQLite, Ingres, Oracle, mSQL)
 .TP
 .I imfile
 Input module for text files
@@ -208,7 +208,7 @@ string ":omrelp:" in front of the hostname.
 .sp
 In the example above, messages are forwarded via UDP to the machine 192.168.0.1, the destination 
 port defaults to 514. Due to the nature of UDP, you will probably lose some messages in transit.
-If you expect high traffic volume, you can expect to lose a quite noticable number of messages
+If you expect high traffic volume, you can expect to lose a quite noticeable number of messages
 (the higher the traffic, the more likely and severe is message loss).
 
 .B If you would like to prevent message loss, use RELP:
diff --git a/tools/syslogd.c b/tools/syslogd.c
index 3f97a70f..5edf92ce 100644
--- a/tools/syslogd.c
+++ b/tools/syslogd.c
@@ -394,11 +394,11 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a
 /* hardcoded standard templates (used for defaults) */
 static uchar template_DebugFormat[] = "\"Debug line with all properties:\nFROMHOST: '%FROMHOST%', fromhost-ip: '%fromhost-ip%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nescaped msg: '%msg:::drop-cc%'\nrawmsg: '%rawmsg%'\n\n\"";
 static uchar template_SyslogProtocol23Format[] = "\"<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n\"";
-static uchar template_TraditionalFileFormat[] = "\"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n\"";
-static uchar template_FileFormat[] = "\"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n\"";
+static uchar template_TraditionalFileFormat[] = "\"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n\"";
+static uchar template_FileFormat[] = "\"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n\"";
 static uchar template_WallFmt[] = "\"\r\n\7Message from syslogd@%HOSTNAME% at %timegenerated% ...\r\n %syslogtag%%msg%\n\r\"";
-static uchar template_ForwardFormat[] = "\"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg%\"";
-static uchar template_TraditionalForwardFormat[] = "\"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg%\"";
+static uchar template_ForwardFormat[] = "\"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%\"";
+static uchar template_TraditionalForwardFormat[] = "\"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%\"";
 static uchar template_StdUsrMsgFmt[] = "\" %syslogtag%%msg%\n\r\"";
 static uchar template_StdDBFmt[] = "\"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')\",SQL";
 static uchar template_StdPgSQLFmt[] = "\"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-pgsql%', '%timegenerated:::date-pgsql%', %iut%, '%syslogtag%')\",STDSQL";
@@ -3021,7 +3021,7 @@ int realMain(int argc, char **argv)
 		case 'Q': /* dont resolve hostnames in ACL to IPs */
 		case 's':
 		case 'u': /* misc user settings */
-		case 'w': /* disable disallowed host warnigs */
+		case 'w': /* disable disallowed host warnings */
 		case 'x': /* disable dns for remote messages */
 			CHKiRet(bufOptAdd(ch, optarg));
 			break;