added $RulesetParser config directive

The implementation is now almost done and works, including doc. I now need to verify shutdown, guess there are some resource leaks left...
author: Rainer Gerhards <rgerhards@adiscon.com> 2009-11-04 14:21:08 +0100
committer: Rainer Gerhards <rgerhards@adiscon.com> 2009-11-04 14:21:08 +0100
commit: b6a92e2f23895cadf9f0ea2de64021912bd2eeb0 (patch)
tree: 734177c4783d946ea28af05dfd75b60ae6d6d2c9
parent: ef661fe13c0ab5018afedb1cb5b8cffab05ad7c4 (diff)
download: rsyslog-b6a92e2f23895cadf9f0ea2de64021912bd2eeb0.tar.gz
rsyslog-b6a92e2f23895cadf9f0ea2de64021912bd2eeb0.tar.xz
rsyslog-b6a92e2f23895cadf9f0ea2de64021912bd2eeb0.zip
6 files changed, 181 insertions, 6 deletions
diff --git a/ChangeLog b/ChangeLog
index 4d90c039..13963798 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,8 @@
 ---------------------------------------------------------------------------
 Version 5.3.4  [DEVEL] (rgerhards), 2009-10-??
+- added the ability to create custom message parsers
+- added $RulesetParser config directive that permits to bind specific
+  parsers to specific rulesets
 - added omruleset output module, which provides great flexibility in 
   action processing. THIS IS A VERY IMPORTANT ADDITION, see its doc
   for why.
diff --git a/doc/multi_ruleset.html b/doc/multi_ruleset.html
index f82a73a6..da65b4ba 100644
--- a/doc/multi_ruleset.html
+++ b/doc/multi_ruleset.html
@@ -71,6 +71,13 @@ at the time the bind directive is given. There are many ways to make sure this h
 I personally think that it is best to define all rule sets at the top of rsyslog.conf and
 define the inputs at the bottom. This kind of reverses the traditional recommended ordering, but
 seems to be a really useful and straightforward way of doing things.
+<h2>Why are rulesets important for different parser configurations?</h2>
+<p>Custom message parsers, used to handle differnet (and potentially otherwise-invalid)
+message formats, can be bound to rulesets. So multiple rulesets can be a very useful
+way to handle devices sending messages in different malformed formats in a consistent
+way. Unfortunately, this is not uncommon in the syslog world. An in-depth explanation
+with configuration sample can be found at the
+<a href="rsconf1_rulesetparser.html">$RulesetParser</a> configuration directive.
 <h2>Can I use a different Ruleset as the default?</h2>
 <p>This is possible by using the
 
diff --git a/doc/rsconf1_rulesetparser.html b/doc/rsconf1_rulesetparser.html
new file mode 100644
index 00000000..03306ca9
--- /dev/null
+++ b/doc/rsconf1_rulesetparser.html
@@ -0,0 +1,117 @@
+<html>
+<head>
+<title>RulesetParser - rsyslog.conf file</title>
+</head>
+<body>
+<a href="rsyslog_conf_global.html">rsyslog.conf configuration directive</a>
+
+<h2>$RulesetParser</h2>
+<p><b>Type:</b> ruleset-specific configuration directive</p>
+<p><b>Parameter Values:</b> string</p>
+<p><b>Available since:</b> 5.3.4+</p>
+<p><b>Default:</b> rsyslog.rfc5424;rsyslog.rfc5425</p>
+<p><b>Description:</b></p>
+<p>
+This directive permits to specify which message parsers should be used for the ruleset
+in question. It no ruleset is explicitely specified, the default ruleset is used. Message
+parsers are contained in (loadable) parser modules with the most common cases
+(RFC3164 and RFC5424) being build-in into rsyslogd.
+<p>When this directive is specified the first time for a ruleset, it will not only add the
+parser to the ruleset, it will also wipe out the default parsers. So if you need to have
+them in addition to the custom parser, you need to specify them as well.
+<p>Order of directives is important. Parsers are tried one after another, in the order
+they are specified inside the config. As soon as a parser is able to parse the message,
+it will do so and no other parsers will be executed. If no matching parser can be found,
+the message will be discarded and a warning message be issued (but only for the first
+1,000 instances of this problem, to prevent message generation loops).
+<p>Note that the rfc3164 parser will <b>always</b> be able to parse a message - it may
+just not be the format that you like. This has two important implications: 1) always place
+that parser at the END of the parser list, or the other parsers after it will never
+be tried and 2) if you would like to make sure no message is lost, placing the rfc3164
+parser at the end of the parser list ensures that.
+<p>Multiple parser modules are very useful if you have various devices that emit 
+messages that are malformed in various ways. The route to take then is
+<ul>
+<li>make sure you find a custom parser for that device; if there is no one, you
+may consider writing one yourself (it is not that hard) or getting one written
+as part of
+<a href="http://www.rsyslog.com/professional-servcies">Adiscon's professional services
+for rsyslog</a>.
+<li>load your custom parsers via $ModLoad
+<li>create a ruleset for each malformed format; assign the custom parser to it
+<li>create a specific listening port for all devices that emit the same
+malformed format
+<li>bind the listener to the ruleset with the required parser
+</ul>
+<p>Note that it may be cumbersome to add all rules to all rulesets. To avoid this,
+you can either use $Include or <a href="omruleset.html">omruleset</a>
+(what probably provides the best solution).
+<p>More information about rulesets in general can be found in
+<a href="multi_ruleset.html">multi-ruleset support in rsyslog</a>.
+<p><b>Caveats:</b></p>
+<p>currently none known</p>
+
+<p><b>Example:</b></p>
+<p>This example assumes there are two devices emiting malformed messages via UDP.
+We have two custom parsers for them, named &quot;device1.parser&quot; and
+&quot;device2.parser&quot;. In addition to that, we have a number of other
+devices sending wellformed messages, also via UDP.
+<p>The solution is to listen for data from the two devices on two special
+ports (10514 and 10515 in this example), create a ruleset for each and
+assign the custom parsers to them. The rest of the messages are received via
+port 514 using the regular parsers. Processing shall be equal for all messages.
+So we simply forward the malformed messages to the regular queue once they are parsed (keep
+in mind that a message is never again parsed once any parser properly processed it).
+</p>
+<textarea rows="40" cols="80">$ModLoad imudp
+$ModLoad pmdevice1 # load parser "device1.parser" for device 1
+$ModLoad pmdevice2 # load parser "device2.parser" for device 2
+
+# define ruleset for the first device sending malformed data
+$Ruleset maldev1
+$RulesetCreateMainQueue on # create ruleset-specific queue
+$RulesetParser "device1.parser" # note: this deactivates the default parsers
+# forward all messages to default ruleset:
+$ActionOmrulesetRulesetName RSYSLOG_DefaultRuleset 
+*.* :omruleset:
+
+# define ruleset for the second device sending malformed data
+$Ruleset maldev2
+$RulesetCreateMainQueue on # create ruleset-specific queue
+$RulesetParser "device2.parser" # note: this deactivates the default parsers
+# forward all messages to default ruleset:
+$ActionOmrulesetRulesetName RSYSLOG_DefaultRuleset 
+*.* :omruleset:
+
+# switch back to default ruleset
+$Ruleset RSYSLOG_DefaultRuleset
+*.*           /path/to/file
+auth.info     @authlogger.example.net
+# whatever else you usually do...
+
+
+# now define the inputs and bind them to the rulesets
+# first the default listener (utilizing the default ruleset)
+$UDPServerRun 514
+
+# now the one with the parser for device type 1:
+$InputUDPServerBindRuleset maldev1
+$UDPServerRun 10514
+
+# and finally the one for device type 2:
+$InputUDPServerBindRuleset maldev2
+$UDPServerRun 10515
+</textarea>
+<p>Note the positions of the directives. With the current config language,
+<b>sequence of statements is very important</b>. This is ugly, but unfortunately
+the way it currently works.
+</p>
+
+<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual 
+index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p>
+<p><font size="2">This documentation is part of the
+<a href="http://www.rsyslog.com/">rsyslog</a> project.<br>
+Copyright &copy; 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and
+<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 2 or higher.</font></p>
+</body>
+</html>
diff --git a/doc/rsyslog_conf_global.html b/doc/rsyslog_conf_global.html
index 1bf02a55..beb90e02 100644
--- a/doc/rsyslog_conf_global.html
+++ b/doc/rsyslog_conf_global.html
@@ -249,12 +249,14 @@ large enough for the whole message. (Introduced with 4.1.5). Once set, it affect
 <li><a href="rsconf1_resetconfigvariables.html">$ResetConfigVariables</a></li>
 <li><b>$Ruleset</b> <i>name</i> - starts a new ruleset or switches back to one already defined.
 All following actions belong to that new rule set.
-the <i>name</i> does not yet exist, it is created. To swith back to rsyslog's
+the <i>name</i> does not yet exist, it is created. To switch back to rsyslog's
 default ruleset, specify &quot;RSYSLOG_DefaultRuleset&quot;) as the name.
 All following actions belong to that new rule set. It is advised to also read
 our paper on <a href="multi_ruleset.html">using multiple rule sets in rsyslog</a>.</li>
 <li><b><a href="rsconf1_rulesetcreatemainqueue.html">$RulesetCreateMainQueue</a></b> on - creates
 a ruleset-specific main queue.
+<li><b><a href="rsconf1_rulesetparser.html">$RulesetParser</a></b> - enables to set
+a specific (list of) message parsers to be used with the ruleset.
 <li><b>$OptimizeForUniprocessor</b> [on/<b>off</b>] - turns on optimizatons which lead to better
 performance on uniprocessors. If you run on multicore-machiens, turning this off lessens CPU load. The
 default may change as uniprocessor systems become less common. [available since 4.1.0]</li>
diff --git a/runtime/parser.c b/runtime/parser.c
index e87068fc..0689657f 100644
--- a/runtime/parser.c
+++ b/runtime/parser.c
@@ -450,9 +450,10 @@ ParseMsg(msg_t *pMsg)
 	 * loger possible.
 	 */
 	pParserList = ruleset.GetParserList(pMsg);
-	if(pParserList == NULL)
+	if(pParserList == NULL) {
 		pParserList = pDfltParsLst;
-	DBGPRINTF("Using parser list %p%s.\n", pParserList,
+	}
+	DBGPRINTF("parse using parser list %p%s.\n", pParserList,
 		  (pParserList == pDfltParsLst) ? " (the default list)" : "");
 
 	bIsSanitized = FALSE;
@@ -467,7 +468,8 @@ ParseMsg(msg_t *pMsg)
 			}
 			bIsSanitized = TRUE;
 		}
-		localRet = pParserList->pParser->pModule->mod.pm.parse(pMsg);
+		localRet = pParser->pModule->mod.pm.parse(pMsg);
+		dbgprintf("Parser '%s' returned %d\n", pParser->pName, localRet);
 		if(localRet != RS_RET_COULD_NOT_PARSE)
 			break;
 		pParserList = pParserList->pNext;
diff --git a/runtime/ruleset.c b/runtime/ruleset.c
index 5e1bfced..bfcb520f 100644
--- a/runtime/ruleset.c
+++ b/runtime/ruleset.c
@@ -45,6 +45,7 @@
 #include "ruleset.h"
 #include "rule.h"
 #include "errmsg.h"
+#include "parser.h"
 #include "unicode-helper.h"
 #include "dirty.h" /* for main ruleset queue creation */
 
@@ -52,10 +53,11 @@
 DEFobjStaticHelpers
 DEFobjCurrIf(errmsg)
 DEFobjCurrIf(rule)
+DEFobjCurrIf(parser)
 
 linkedList_t llRulesets; /* this is NOT a pointer - no typo here ;) */
 ruleset_t *pCurrRuleset = NULL; /* currently "active" ruleset */
-ruleset_t *pDfltRuleset = NULL; /* currentl default ruleset, e.g. for binding to actions which have no other */
+ruleset_t *pDfltRuleset = NULL; /* current default ruleset, e.g. for binding to actions which have no other */
 
 /* ---------- linked-list key handling functions ---------- */
 
@@ -174,7 +176,7 @@ dbgprintf("ruleset.ProcessMsg() returns %d\n", iRet);
 static parserList_t*
 GetParserList(msg_t *pMsg)
 {
-	return (pMsg->pRuleset == NULL) ? NULL : pMsg->pRuleset->pParserLst;
+	return (pMsg->pRuleset == NULL) ? pDfltRuleset->pParserLst : pMsg->pRuleset->pParserLst;
 }
 
 
@@ -448,6 +450,46 @@ finalize_it:
 }
 
 
+/* Add a ruleset specific parser to the ruleset. Note that adding the first
+ * parser automatically disables the default parsers. If they are needed as well,
+ * the must be added via explicit config directives.
+ * Note: this is the only spot in the code that requires the parser object. In order
+ * to solve some class init bootstrap sequence problems, we get the object handle here
+ * instead of during module initialization. Note that objUse() is capable of being 
+ * called multiple times.
+ * rgerhards, 2009-11-04
+ */
+static rsRetVal
+rulesetAddParser(void __attribute__((unused)) *pVal, uchar *pName)
+{
+	parser_t *pParser;
+	DEFiRet;
+
+	assert(pCurrRuleset != NULL); 
+
+	CHKiRet(objUse(parser, CORE_COMPONENT));
+	iRet = parser.FindParser(&pParser, pName);
+	if(iRet == RS_RET_PARSER_NOT_FOUND) {
+		errmsg.LogError(0, RS_RET_PARSER_NOT_FOUND, "error: parser '%s' unknown at this time "
+			  	"(maybe defined too late in rsyslog.conf?)", pName);
+		ABORT_FINALIZE(RS_RET_NO_CURR_RULESET);
+	} else if(iRet != RS_RET_OK) {
+		errmsg.LogError(0, iRet, "error trying to find parser '%s'\n", pName);
+		FINALIZE;
+	}
+
+	CHKiRet(parser.AddParserToList(&pCurrRuleset->pParserLst, pParser));
+
+	dbgprintf("added parser '%s' to ruleset '%s'\n", pName, pCurrRuleset->pszName);
+RUNLOG_VAR("%p", pCurrRuleset->pParserLst);
+
+finalize_it:
+	d_free(pName); /* no longer needed */
+
+	RETiRet;
+}
+
+
 /* queryInterface function
  * rgerhards, 2008-02-21
  */
@@ -490,6 +532,7 @@ BEGINObjClassExit(ruleset, OBJ_IS_CORE_MODULE) /* class, version */
 	llDestroy(&llRulesets);
 	objRelease(errmsg, CORE_COMPONENT);
 	objRelease(rule, CORE_COMPONENT);
+	objRelease(parser, CORE_COMPONENT);
 ENDObjClassExit(ruleset)
 
 
@@ -510,6 +553,7 @@ BEGINObjClassInit(ruleset, 1, OBJ_IS_CORE_MODULE) /* class, version */
 	CHKiRet(llInit(&llRulesets, rulesetDestructForLinkedList, keyDestruct, strcasecmp));
 
 	/* config file handlers */
+	CHKiRet(regCfSysLineHdlr((uchar *)"rulesetparser", 0, eCmdHdlrGetWord, rulesetAddParser, NULL, NULL));
 	CHKiRet(regCfSysLineHdlr((uchar *)"rulesetcreatemainqueue", 0, eCmdHdlrBinary, rulesetCreateQueue, NULL, NULL));
 ENDObjClassInit(ruleset)
author	Rainer Gerhards <rgerhards@adiscon.com>	2009-11-04 14:21:08 +0100
committer	Rainer Gerhards <rgerhards@adiscon.com>	2009-11-04 14:21:08 +0100
commit	b6a92e2f23895cadf9f0ea2de64021912bd2eeb0 (patch)
tree	734177c4783d946ea28af05dfd75b60ae6d6d2c9
parent	ef661fe13c0ab5018afedb1cb5b8cffab05ad7c4 (diff)
download	rsyslog-b6a92e2f23895cadf9f0ea2de64021912bd2eeb0.tar.gz rsyslog-b6a92e2f23895cadf9f0ea2de64021912bd2eeb0.tar.xz rsyslog-b6a92e2f23895cadf9f0ea2de64021912bd2eeb0.zip