added new poperty replacer option, added missing documentation

- added a new property replacer option "sp-if-no-1st-sp" to cover
  a problem with RFC 3164 based interpreation of tag separation. While
  it is a generic approach, it fixes a format problem introduced in
  3.18.0, where kernel messages no longer had a space after the tag.
  This is done by a modifcation of the default templates.
  Please note that this may affect some messages where there intentionally
  is no space between the tag and the first character of the message
  content. If so, this needs to be worked around via a specific
  template. However, we consider this scenario to be quite remote and,
  even if it exists, it is not expected that it will actually cause
  problems with log parsers (instead, we assume the new default template
  behaviour may fix previous problems with log parsers due to the
  missing space).
- doc bugfix: property replacer options secpath-replace and
  secpath-drop were not documented

6 files changed, 95 insertions, 8 deletions

diff --git a/ChangeLog b/ChangeLog
index 438f79b7..ab78ab5c 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -20,8 +20,23 @@ Version 3.18.1 (rgerhards), 2008-07-??
   where it was needed. This resulted in rsyslog using the default
   (20 minutes) in some code pathes, what looked to the user like mark
   messages were never written.
+- added a new property replacer option "sp-if-no-1st-sp" to cover
+  a problem with RFC 3164 based interpreation of tag separation. While
+  it is a generic approach, it fixes a format problem introduced in
+  3.18.0, where kernel messages no longer had a space after the tag.
+  This is done by a modifcation of the default templates.
+  Please note that this may affect some messages where there intentionally
+  is no space between the tag and the first character of the message
+  content. If so, this needs to be worked around via a specific
+  template. However, we consider this scenario to be quite remote and,
+  even if it exists, it is not expected that it will actually cause
+  problems with log parsers (instead, we assume the new default template
+  behaviour may fix previous problems with log parsers due to the 
+  missing space).
 - bugfix: imklog module was not correctly compiled for GNU/kFreeBSD.
   Thanks to Petr Salinger for the patch
+- doc bugfix: property replacer options secpath-replace and
+  secpath-drop were not documented
 - doc bugfix: fixed some typos in rsyslog.conf man page
 - fixed typo in source comment  - thanks to Rio Fujita
 ---------------------------------------------------------------------------
diff --git a/doc/property_replacer.html b/doc/property_replacer.html
index a2efaede..f5fc194c 100644
--- a/doc/property_replacer.html
+++ b/doc/property_replacer.html
@@ -253,7 +253,7 @@ Especially useful for PIX.</td>
 <td>format as RFC 3339 date</td>
 </tr>
 <tr>
-<td><b>escape-cc</b></td>
+<td valign="top"><b>escape-cc</b></td>
 <td>replace control characters (ASCII value 127 and values
 less then 32) with an escape sequence. The sequnce is
 "#&lt;charval&gt;" where charval is the 3-digit decimal value
@@ -263,19 +263,53 @@ Note: using this option requires that <a href="rsconf1_escapecontrolcharacterson
 is set to off.</td>
 </tr>
 <tr>
-<td><b>space-cc</b></td>
+<td valign="top"><b>space-cc</b></td>
 <td>replace control characters by spaces<br>
 Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a>
 is set to off.</td>
 </tr>
 <tr>
-<td><b>drop-cc</b></td>
+<td valign="top"><b>drop-cc</b></td>
 <td>drop control characters - the resulting string will
 neither contain control characters, escape sequences nor any other
 replacement character like space.<br>
 Note: using this option requires that <a href="rsconf1_escapecontrolcharactersonreceive.html">$EscapeControlCharactersOnReceive</a>
 is set to off.</td>
 </tr>
+<tr>
+<td valign="top"><b>sp-if-no-1st-sp</b></td>
+<td>This option looks scary and should probably not be used by a user. For any field
+given, it returns either a single space character or no character at all. Field content
+is never returned. A space is returned if (and only if) the first character of the
+field's content is NOT a space. This option is kind of a hack to solve a problem rooted
+in RFC 3164: 3164 specifies no delimiter between the syslog tag sequence and the actual
+message text. Almost all implementation in fact delemit the two by a space. As of
+RFC 3164, this space is part of the message text itself. This leads to a problem when
+building the message (e.g. when writing to disk or forwarding). Should a delimiting
+space be included if the message does not start with one? If not, the tag is immediately
+followed by another non-space character, which can lead some log parsers to misinterpret
+what is the tag and what the message. The problem finally surfaced when the klog module
+was restructured and the tag correctly written. It exists with other message sources,
+too. The solution was the introduction of this special property replacer option. Now,
+the default template can contain a conditional space, which exists only if the
+message does not start with one. While this does not solve all issues, it should
+work good enough in the far majority of all cases. If you read this text and have
+no idea of what it is talking about - relax: this is a good indication you will never
+need this option. Simply forget about it ;)
+</td>
+</tr>
+<tr>
+<td valign="top"><b>secpath-drop</b></td>
+<td>Drops slashes inside the field (e.g. "a/b" becomes "ab").
+Useful for secure pathname generation (with dynafiles).
+</td>
+</tr>
+<tr>
+<td valign="top"><b>secpath-replace</b></td>
+<td>Replace slashes inside the field by an underscore. (e.g. "a/b" becomes "a_b").
+Useful for secure pathname generation (with dynafiles).
+</td>
+</tr>
 </tbody>
 </table>
 <h2>Further Links</h2>
@@ -286,4 +320,4 @@ to record severity and facility of a message)</li>
 <li><a href="rsyslog_conf.html">Configuration file
 syntax</a>, this is where you actually use the property replacer.</li>
 </ul>
-</body></html>
\ No newline at end of file
+</body></html>
diff --git a/msg.c b/msg.c
index 1590a7bb..bd1e425e 100644
--- a/msg.c
+++ b/msg.c
@@ -1897,6 +1897,32 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe,
 #endif /* #ifdef FEATURE_REGEXP */
 	}
 
+	/* now check if we need to do our "SP if first char is non-space" hack logic */
+	if(*pRes && pTpe->data.field.options.bSPIffNo1stSP) {
+		char *pB;
+		uchar cFirst = *pRes;
+
+		/* here, we always destruct the buffer and return a new one */
+		pB = (char *) malloc(2 * sizeof(char));
+		if(pB == NULL) {
+			if(*pbMustBeFreed == 1)
+				free(pRes);
+			*pbMustBeFreed = 0;
+			return "**OUT OF MEMORY**";
+		}
+		pRes = pB;
+		*pbMustBeFreed = 1;
+
+		if(cFirst == ' ') {
+			/* if we have a SP, we must return an empty string */
+			*pRes = '\0'; /* empty */
+		} else {
+			/* if it is no SP, we need to return one */
+			*pRes = ' ';
+			*(pRes+1) = '\0';
+		}
+	}
+
 	if(*pRes) {
 		/* case conversations (should go after substring, because so we are able to
 		 * work on the smallest possible buffer).
diff --git a/syslogd.c b/syslogd.c
index 256b2b29..35e91af0 100644
--- a/syslogd.c
+++ b/syslogd.c
@@ -424,11 +424,11 @@ int option_DisallowWarning = 1;	/* complain if message from disallowed sender is
 
 /* hardcoded standard templates (used for defaults) */
 static uchar template_SyslogProtocol23Format[] = "\"<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n\"";
-static uchar template_TraditionalFileFormat[] = "\"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n\"";
-static uchar template_FileFormat[] = "\"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n\"";
+static uchar template_TraditionalFileFormat[] = "\"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n\"";
+static uchar template_FileFormat[] = "\"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n\"";
 static uchar template_WallFmt[] = "\"\r\n\7Message from syslogd@%HOSTNAME% at %timegenerated% ...\r\n %syslogtag%%msg%\n\r\"";
-static uchar template_ForwardFormat[] = "\"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg%\"";
-static uchar template_TraditionalForwardFormat[] = "\"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg%\"";
+static uchar template_ForwardFormat[] = "\"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%\"";
+static uchar template_TraditionalForwardFormat[] = "\"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%\"";
 static uchar template_StdUsrMsgFmt[] = "\" %syslogtag%%msg%\n\r\"";
 static uchar template_StdDBFmt[] = "\"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')\",SQL";
 static uchar template_StdPgSQLFmt[] = "\"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-pgsql%', '%timegenerated:::date-pgsql%', %iut%, '%syslogtag%')\",STDSQL";
diff --git a/template.c b/template.c
index 844c5aec..627f8911 100644
--- a/template.c
+++ b/template.c
@@ -444,6 +444,8 @@ static void doOptions(unsigned char **pp, struct templateEntry *pTpe)
 			pTpe->data.field.eCaseConv = tplCaseConvLower;
 		 } else if(!strcmp((char*)Buf, "uppercase")) {
 			pTpe->data.field.eCaseConv = tplCaseConvUpper;
+		 } else if(!strcmp((char*)Buf, "sp-if-no-1st-sp")) {
+			pTpe->data.field.options.bSPIffNo1stSP = 1;
 		 } else if(!strcmp((char*)Buf, "escape-cc")) {
 			pTpe->data.field.options.bEscapeCC = 1;
 		 } else if(!strcmp((char*)Buf, "drop-cc")) {
@@ -1013,6 +1015,15 @@ void tplPrintList(void)
 				if(pTpe->data.field.options.bSpaceCC) {
 				  	dbgprintf("[replace control-characters with space] ");
 				}
+				if(pTpe->data.field.options.bSecPathDrop) {
+				  	dbgprintf("[slashes are dropped] ");
+				}
+				if(pTpe->data.field.options.bSecPathReplace) {
+				  	dbgprintf("[slashes are replaced by '_'] ");
+				}
+				if(pTpe->data.field.options.bSPIffNo1stSP) {
+				  	dbgprintf("[SP iff no first SP] ");
+				}
 				if(pTpe->data.field.options.bDropLastLF) {
 				  	dbgprintf("[drop last LF in msg] ");
 				}
diff --git a/template.h b/template.h
index 5b0bcdb4..02264201 100644
--- a/template.h
+++ b/template.h
@@ -80,6 +80,7 @@ struct templateEntry {
 				unsigned bDropLastLF: 1;	/* drop last LF char in msg (PIX!) */
 				unsigned bSecPathDrop: 1;		/* drop slashes, replace dots, empty string */
 				unsigned bSecPathReplace: 1;		/* replace slashes, replace dots, empty string */
+				unsigned bSPIffNo1stSP: 1;		/* replace slashes, replace dots, empty string */
 			} options;		/* options as bit fields */
 		} field;
 	} data;