- added support for high-precision timestamps when receiving legacy syslog

    messages
- added new $ActionForwardDefaultTemplate directive
- added new $ActionGSSForwardDefaultTemplate directive

8 files changed, 82 insertions, 17 deletions

diff --git a/ChangeLog b/ChangeLog
index 364f8fbf..6214165d 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -16,6 +16,10 @@ Version 3.12.5 (rgerhards), 2008-03-??
 - changed default file output format to include high-precision timestamps
 - added a buid-in template for previous syslogd file format
 - added new $ActionFileDefaultTemplate directive
+- added support for high-precision timestamps when receiving legacy
+  syslog messages
+- added new $ActionForwardDefaultTemplate directive
+- added new $ActionGSSForwardDefaultTemplate directive
 ---------------------------------------------------------------------------
 Version 3.12.4 (rgerhards), 2008-03-25
 - Greatly enhanced rsyslogd's file write performance by disabling
diff --git a/doc/rsyslog_conf.html b/doc/rsyslog_conf.html
index 916056a7..80c87233 100644
--- a/doc/rsyslog_conf.html
+++ b/doc/rsyslog_conf.html
@@ -64,7 +64,7 @@ unstable...). So you have been warned ;)</p>
 many parameter settings modify queue parameters. If in doubt, use the
 default, it is usually well-chosen and applicable in most cases.</p>
 <ul>
-<li><a href="rsconf1_actionexeconlywhenpreviousissuspended.html">$ActionExecOnlyWhenPreviousIsSuspended</a></li><li>$ActionFileDefaultTemplate [templateName] - sets a new default template for file actions</li><li>$ActionFileEnableSync [on/<span style="font-weight: bold;">off</span>] - enables file syncing capability of omfile</li>
+<li><a href="rsconf1_actionexeconlywhenpreviousissuspended.html">$ActionExecOnlyWhenPreviousIsSuspended</a></li><li>$ActionFileDefaultTemplate [templateName] - sets a new default template for file actions</li><li>$ActionFileEnableSync [on/<span style="font-weight: bold;">off</span>] - enables file syncing capability of omfile</li><li>$ActionForwardDefaultTemplate [templateName] - sets a new default template for UDP and plain TCP forwarding action</li><li>$ActionGSSForwardDefaultTemplate [templateName] - sets a new default template for GSS-API forwarding action</li>
 <li>$ActionQueueCheckpointInterval &lt;number&gt;</li>
 <li>$ActionQueueDequeueSlowdown &lt;number&gt; [number
 is timeout in <i> micro</i>seconds (1000000us is 1sec!),
diff --git a/doc/v3compatibility.html b/doc/v3compatibility.html
index ee7664f6..74319c38 100644
--- a/doc/v3compatibility.html
+++ b/doc/v3compatibility.html
@@ -15,7 +15,7 @@ specifying the -c option on the rsyslog command line. That will enable
 backwards-compatibility mode. However, please note that things may be
 suboptimal in backward compatibility mode, so the advise is to work
 through this document, update your rsyslog.conf, remove the no longer
-supported startup options and then add -c4 as the first option to the
+supported startup options and then add -c3 as the first option to the
 rsyslog command line. That will enable native mode.</p>
 <p>Please note that rsyslogd helps you during that process by
 logging appropriate messages about compatibility mode and
@@ -57,7 +57,7 @@ syslog.conf.</p>
 backward compatibility mode. It must always be the first option on the
 command line, as it influences processing of the other options. To use
 the rsyslog v3 native
-interface, specify -c4. To use compatibility mode&nbsp;,
+interface, specify -c3. To use compatibility mode&nbsp;,
 either do not use -c at all or use -c&lt;vers&gt; where vers is
 the
 rsyslog version that it shall be compatible to. Use -c0 to be
@@ -160,7 +160,26 @@ $ActionFileDefaultTemplate directive. Keep in mind, though, that
 templates must be defined before they are used.</p><p>Keep in mind that
 when receiving messages from remote hosts, the timestamp is just as
 precise as the remote host provided it. In most cases, this means you
-will only a receive a standard timestamp with second precision.</p><p>Please note that the default forwarding format may also change in the future. </p><h2>Queue Modes for the Main Message Queue</h2>
+will only a receive a standard timestamp with second precision. If
+rsyslog is running at the remote end, you can configure it to provide
+high-precision timestamps (see below).</p><h2>Forwarding Format</h2><p>When
+forwarding messages to remote syslog servers, rsyslogd by default uses
+the plain old syslog format with second-level resolution inside the
+timestamps. We could have made it emit high precision timestamps.
+However, that would have broken almost all receivers, including earlier
+versions of rsyslog. To avoid this hassle, high-precision timestamps
+need to be explicitely enabled. To make this as painless as possible,
+rsyslog comes with a canned template that contains everything
+necessary. &nbsp;To enable high-precision timestamps, just use:</p><p style="font-weight: bold;"><code>$ActionForwardDefaultTemplate RSYSLOG_ForwardFileFormat # for plain TCP and UDP</code></p><p style="font-weight: bold;"><code>$ActionGSSForwardDefaultTemplate RSYSLOG_ForwardFileFormat # for GSS-API</code></p><p>And, of course, you can always set different forwarding formats by just specifying the right template.</p><p>If
+you are running in a system with only rsyslog 3.12.5 and above in the
+receiver roles, it is suggested to add one (or both) of the above
+statements to the top of your rsyslog.conf - that will enable you to
+use the best in timestamp support availble. Please note that when you
+use this format with other receivers, they will probably become pretty
+confused and not detect the timestamp at all. In earlier rsyslog
+versions, for example, that leads to duplication of timestamp and
+hostname fields and disables the detection of the orignal hostname in a
+relayed/NATed environment. So use the new format with care. </p><h2>Queue Modes for the Main Message Queue</h2>
 <p>Either "FixedArray" or "LinkedList" is recommended. "Direct"
 is available, but should not be used except for a very good reason
 ("Direct" disables queueing and will potentially lead to message loss
diff --git a/omfwd.c b/omfwd.c
index e223801a..19d09379 100644
--- a/omfwd.c
+++ b/omfwd.c
@@ -90,6 +90,10 @@ typedef struct _instanceData {
 	tcpclt_t *pTCPClt;		/* our tcpclt object */
 } instanceData;
 
+/* config data */
+static uchar	*pszTplName = NULL; /* name of the default template to use */
+
+
 /* get the syslog forward port from selector_t. The passed in
  * struct must be one that is setup for forwarding.
  * rgerhards, 2007-06-28
@@ -542,9 +546,8 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1)
 			strcpy(pData->f_hname, (char*) q);
 
 		/* process template */
-		if((iRet = cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, (uchar*) " StdFwdFmt"))
-		   != RS_RET_OK)
-			goto finalize_it;
+		CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS,
+		       		(pszTplName == NULL) ? (uchar*)"RSYSLOG_TraditionalForwardFormat" : pszTplName));
 
 		/* first set the pData->eDestState */
 		memset(&hints, 0, sizeof(hints));
@@ -592,6 +595,11 @@ CODESTARTmodExit
 	objRelease(errmsg, CORE_COMPONENT);
 	objRelease(net, LM_NET_FILENAME);
 	objRelease(tcpclt, LM_TCPCLT_FILENAME);
+
+	if(pszTplName != NULL) {
+		free(pszTplName);
+		pszTplName = NULL;
+	}
 ENDmodExit
 
 
@@ -601,6 +609,20 @@ CODEqueryEtryPt_STD_OMOD_QUERIES
 ENDqueryEtryPt
 
 
+/* Reset config variables for this module to default values.
+ * rgerhards, 2008-03-28
+ */
+static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __attribute__((unused)) *pVal)
+{
+	if(pszTplName != NULL) {
+		free(pszTplName);
+		pszTplName = NULL;
+	}
+
+	return RS_RET_OK;
+}
+
+
 BEGINmodInit(Fwd)
 CODESTARTmodInit
 	*ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */
@@ -608,6 +630,9 @@ CODEmodInit_QueryRegCFSLineHdlr
 	CHKiRet(objUse(errmsg, CORE_COMPONENT));
 	CHKiRet(objUse(net, LM_NET_FILENAME));
 	CHKiRet(objUse(tcpclt, LM_TCPCLT_FILENAME));
+
+	CHKiRet(regCfSysLineHdlr((uchar *)"actionforwarddefaulttemplate", 0, eCmdHdlrGetWord, NULL, &pszTplName, NULL));
+	CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID));
 ENDmodInit
 
 #endif /* #ifdef SYSLOG_INET */
diff --git a/plugins/omgssapi/omgssapi.c b/plugins/omgssapi/omgssapi.c
index 97a7e7e7..9a7e8ab9 100644
--- a/plugins/omgssapi/omgssapi.c
+++ b/plugins/omgssapi/omgssapi.c
@@ -96,6 +96,8 @@ typedef struct _instanceData {
 	OM_uint32 gss_flags;
 } instanceData;
 
+/* config data */
+static uchar	*pszTplName = NULL; /* name of the default template to use */
 static char *gss_base_service_name = NULL;
 static enum gss_mode_t {
 	GSSMODE_MIC,
@@ -596,9 +598,8 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1)
 		strcpy(pData->f_hname, (char*) q);
 
 	/* process template */
-	if((iRet = cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, (uchar*) " StdFwdFmt"))
-	   != RS_RET_OK)
-		goto finalize_it;
+	CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS,
+			(pszTplName == NULL) ? (uchar*)"RSYSLOG_TraditionalForwardFormat" : pszTplName));
 
 	/* first set the pData->eDestState */
 	memset(&hints, 0, sizeof(hints));
@@ -635,6 +636,11 @@ CODESTARTmodExit
 	objRelease(errmsg, CORE_COMPONENT);
 	objRelease(gssutil, LM_GSSUTIL_FILENAME);
 	objRelease(tcpclt, LM_TCPCLT_FILENAME);
+
+	if(pszTplName != NULL) {
+		free(pszTplName);
+		pszTplName = NULL;
+	}
 ENDmodExit
 
 
@@ -672,6 +678,10 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a
 		free(gss_base_service_name);
 		gss_base_service_name = NULL;
 	}
+	if(pszTplName != NULL) {
+		free(pszTplName);
+		pszTplName = NULL;
+	}
 	return RS_RET_OK;
 }
 
@@ -686,6 +696,7 @@ CODEmodInit_QueryRegCFSLineHdlr
 
 	CHKiRet(omsdRegCFSLineHdlr((uchar *)"gssforwardservicename", 0, eCmdHdlrGetWord, NULL, &gss_base_service_name, STD_LOADABLE_MODULE_ID));
 	CHKiRet(omsdRegCFSLineHdlr((uchar *)"gssmode", 0, eCmdHdlrGetWord, setGSSMode, &gss_mode, STD_LOADABLE_MODULE_ID));
+	CHKiRet(regCfSysLineHdlr((uchar *)"actiongssforwarddefaulttemplate", 0, eCmdHdlrGetWord, NULL, &pszTplName, NULL));
 	CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID));
 ENDmodInit
 
diff --git a/plugins/omrelp/omrelp.c b/plugins/omrelp/omrelp.c
index 069c19d6..39d25812 100644
--- a/plugins/omrelp/omrelp.c
+++ b/plugins/omrelp/omrelp.c
@@ -290,7 +290,7 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1)
 		strcpy(pData->f_hname, (char*) q);
 
 	/* process template */
-	CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, (uchar*) " StdFwdFmt"));
+	CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, (uchar*) "RSYSLOG_ForwardFormat"));
 
 	/* create our relp client  */
 	CHKiRet(relpEngineCltConstruct(pRelpEngine, &pData->pRelpClt)); /* we use CHKiRet as librelp has a similar return value range */
diff --git a/plugins/omsnmp/omsnmp.c b/plugins/omsnmp/omsnmp.c
index d518ae3b..47afe2fd 100644
--- a/plugins/omsnmp/omsnmp.c
+++ b/plugins/omsnmp/omsnmp.c
@@ -428,7 +428,7 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1)
 	dbgprintf("SpecificType: %d\n", pData->iSpecificType);
 
 	/* process template */
-	CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, (uchar*) " StdFwdFmt"));
+	CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, (uchar*) "RSYSLOG_TraditionalForwardFormat"));
 
 	/* Init NetSNMP library and read in MIB database */
 	init_snmp("rsyslog");
diff --git a/syslogd.c b/syslogd.c
index 228b44e6..6178a4e4 100644
--- a/syslogd.c
+++ b/syslogd.c
@@ -420,7 +420,8 @@ int option_DisallowWarning = 1;	/* complain if message from disallowed sender is
 static uchar template_TraditionalFileFormat[] = "\"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n\"";
 static uchar template_FileFormat[] = "\"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n\"";
 static uchar template_WallFmt[] = "\"\r\n\7Message from syslogd@%HOSTNAME% at %timegenerated% ...\r\n %syslogtag%%msg%\n\r\"";
-static uchar template_StdFwdFmt[] = "\"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%\"";
+static uchar template_ForwardFormat[] = "\"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg%\"";
+static uchar template_TraditionalForwardFormat[] = "\"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%\"";
 static uchar template_StdUsrMsgFmt[] = "\" %syslogtag%%msg%\n\r\"";
 static uchar template_StdDBFmt[] = "\"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')\",SQL";
 static uchar template_StdPgSQLFmt[] = "\"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-pgsql%', '%timegenerated:::date-pgsql%', %iut%, '%syslogtag%')\",STDSQL";
@@ -1372,9 +1373,12 @@ static int parseLegacySyslogMsg(msg_t *pMsg, int flags)
 	assert(pMsg->pszUxTradMsg != NULL);
 	p2parse = (char*) pMsg->pszUxTradMsg;
 
-	/* Check to see if msg contains a timestamp
+	/* Check to see if msg contains a timestamp. We stary trying with a
+	 * high-precision one...
 	 */
-	if(datetime.ParseTIMESTAMP3164(&(pMsg->tTIMESTAMP), p2parse) == TRUE)
+	if(datetime.ParseTIMESTAMP3339(&(pMsg->tTIMESTAMP), &p2parse) == TRUE)
+		/* we are done - parse pointer is moved by ParseTIMESTAMP3339 */;
+	else if(datetime.ParseTIMESTAMP3164(&(pMsg->tTIMESTAMP), p2parse) == TRUE)
 		p2parse += 16;
 	else {
 		flags |= ADDDATE;
@@ -2818,8 +2822,10 @@ static void mainThread()
 	tplAddLine("RSYSLOG_TraditionalFileFormat", &pTmp);
 	pTmp = template_WallFmt;
 	tplAddLine(" WallFmt", &pTmp);
-	pTmp = template_StdFwdFmt;
-	tplAddLine(" StdFwdFmt", &pTmp);
+	pTmp = template_ForwardFormat;
+	tplAddLine("RSYSLOG_ForwardFormat", &pTmp);
+	pTmp = template_TraditionalForwardFormat;
+	tplAddLine("RSYSLOG_TraditionalForwardFormat", &pTmp);
 	pTmp = template_StdUsrMsgFmt;
 	tplAddLine(" StdUsrMsgFmt", &pTmp);
 	pTmp = template_StdDBFmt;