From 9103817f9fe811b49938036a1f9ff23672a9ec44 Mon Sep 17 00:00:00 2001 From: Rainer Gerhards Date: Fri, 3 Apr 2009 15:48:55 +0200 Subject: added (some) developer documentation for output plugin interface --- ChangeLog | 1 + doc/dev_oplugins.html | 178 ++++++++++++++++++++++++++++++++++++++++++++++++++ doc/manual.html | 6 +- doc/status.html | 12 ++-- 4 files changed, 190 insertions(+), 7 deletions(-) create mode 100644 doc/dev_oplugins.html diff --git a/ChangeLog b/ChangeLog index 7f79271f..8d4191d3 100644 --- a/ChangeLog +++ b/ChangeLog @@ -39,6 +39,7 @@ Version 4.1.6 [DEVEL] (rgerhards), 2009-03-?? - added a new way how output plugins may be passed parameters. This is more effcient for some outputs. They new can receive fields not only as a single string but rather in an array where each string is seperated. +- added (some) developer documentation for output plugin interface --------------------------------------------------------------------------- Version 4.1.5 [DEVEL] (rgerhards), 2009-03-11 - bugfix: parser did not correctly parse fields in UDP-received messages diff --git a/doc/dev_oplugins.html b/doc/dev_oplugins.html new file mode 100644 index 00000000..5bfc974c --- /dev/null +++ b/doc/dev_oplugins.html @@ -0,0 +1,178 @@ + + +writing rsyslog output plugins (developer's guide) + + +

Writing Rsyslog Output Plugins

+

This page is the begin of some developer documentation for writing output +plugins. Doing so is quite easy (and that was a design goal), but there currently +is only sparse documentation on the process available. I was tempted NOT to +write this guide here because I know I will most probably not be able to +write a complete guide. +

However, I finally concluded that it may be better to have same information +and pointers than to have nothing. +

Getting Started and Samples

+

The best to get started with rsyslog plugin development is by looking at +existing plugins. All that start with "om" are output modules. That +means they are primarily thought of being message sinks. In theory, however, output +plugins may aggergate other functionality, too. Nobody has taken this route so far +so if you would like to do that, it is highly suggested to post your plan on the +rsyslog mailing list, first (so that we can offer advise). +

The rsyslog distribution tarball contains two plugins that are extremely well +targeted for getting started: +

+Plugin omtemplate was specifically created to provide a copy template for new output +plugins. It is bare of real functionality but has ample comments. Even if you decide +to start from another plugin (or even from scratch), be sure to read omtemplate source +and comments first. The omstdout is primarily a testing aide, but offers support for +the two different parameter-passing conventions plugins can use (plus the way to +differentiate between the two). It also is not bare of functionaly, only mostly +bare of it ;). But you can actually execute it and play with it. +

In any case, you should also read the comments in ./runtime/module-template.h. +Output plugins are build based on a large set of code-generating macros. These +macros handle most of the plumbing needed by the interface. As long as no +special callback to rsyslog is needed (it typically is not), an output plugin does +not really need to be aware that it is executed by rsyslog. As a plug-in programmer, +you can (in most cases) "code as usual". However, all macros and entry points need to be +provided and thus reading the code comments in the files mentioned is highly suggested. +

In short, the best idea is to start with a template. Let's assume you start by +copying omtemplate. Then, the basic steps you need to do are: +

+

Basically, this is all you need to do ... Well, except, of course, coding +your plugin ;). For testing, you need rsyslog's debugging support. Some useful +information is given in "troubleshooting rsyslog +from the doc set. +

Special Topics

+

Threading

+

Rsyslog uses massive parallel processing and multithreading. However, a plugin's entry +points are guaranteed to be never called concurrently for the same action. +That means your plugin must be able to be called concurrently by two or more +threads, but you can be sure that for the same instance no concurrent calls +happen. This is guaranteed by the interface specification and the rsyslog core +guards against multiple concurrent calls. An instance, in simple words, is one +that shares a single instanceData structure. +

So as long as you do not mess around with global data, you do not need +to think about multithreading (and can apply a purely sequential programming +methodology). +

Please note that duringt the configuraton parsing stage of execution, access to +global variables for the configuration system is safe. In that stage, the core will +only call sequentially into the plugin. +

Getting Message Data

+

The doAction() entry point of your plugin is provided with messages to be processed. +It will only be activated after filtering and all other conditions, so you do not need +to apply any other conditional but can simply process the message. +

Note that you do NOT receive the full internal representation of the message +object. There are various (including historical) reasons for this and, among +others, this is a design decision based on security. +

Your plugin will only receive what the end user has configured in a $template +statement. However, starting with 4.1.6, there are two ways of receiving the +template content. The default mode, and in most cases sufficient and optimal, +is to receive a single string with the expanded template. As I said, this is usually +optimal, think about writing things to files, emailing content or forwarding it. +

The important philosophy is that a plugin should never reformat any +of such strings - that would either remove the user's ability to fully control +message formats or it would lead to duplicating code that is already present in the +core. If you need some formatting that is not yet present in the core, suggest it +to the rsyslog project, best done by sending a patch ;), and we will try hard to +get it into the core (so far, we could accept all such suggestions - no promise, though). +

If a single string seems not suitable for your application, the plugin can also +request access to the template components. The typical use case seems to be databases, where +you would like to access properties via specific fields. With that mode, you receive a +char ** array, where each array element points to one field from the template (from +left to right). Fields start at arrray index 0 and a NULL pointer means you have +reached the end of the array (the typical Unix "poor man's linked list in an array" +design). Note, however, that each of the individual components is a string. It is +not a date stamp, number or whatever, but a string. This is because rsyslog processes +strings (from a high-level design look at it) and so this is the natural data type. +Feel free to convert to whatever you need, but keep in mind that malformed packets +may have lead to field contents you'd never expected... +

If you like to use the array-based parameter passing method, think that it +is only available in rsyslog 4.1.6 and above. If you can accept that your plugin +will not be working with previous versions, you do not need to handle pre 4.1.6 cases. +However, it would be "nice" if you shut down yourself in these cases - otherwise the +older rsyslog core engine will pass you a string where you expect the array of pointers, +what most probably results in a segfault. To check whether or not the core supports the +functionality, you can use this code sequence: +

+
+BEGINmodInit()
+	rsRetVal localRet;
+	rsRetVal (*pomsrGetSupportedTplOpts)(unsigned long *pOpts);
+	unsigned long opts;
+	int bArrayPassingSupported;		/* does core support template passing as an array? */
+CODESTARTmodInit
+	*ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */
+CODEmodInit_QueryRegCFSLineHdlr
+	/* check if the rsyslog core supports parameter passing code */
+	bArrayPassingSupported = 0;
+	localRet = pHostQueryEtryPt((uchar*)"OMSRgetSupportedTplOpts", &pomsrGetSupportedTplOpts);
+	if(localRet == RS_RET_OK) {
+		/* found entry point, so let's see if core supports array passing */
+		CHKiRet((*pomsrGetSupportedTplOpts)(&opts));
+		if(opts & OMSR_TPL_AS_ARRAY)
+			bArrayPassingSupported = 1;
+	} else if(localRet != RS_RET_ENTRY_POINT_NOT_FOUND) {
+		ABORT_FINALIZE(localRet); /* Something else went wrong, what is not acceptable */
+	}
+	DBGPRINTF("omstdout: array-passing is %ssupported by rsyslog core.\n", bArrayPassingSupported ? "" : "not ");
+
+	if(!bArrayPassingSupported) {
+		DBGPRINTF("rsyslog core too old, shutting down this plug-in\n");
+		ABORT_FINALIZE(RS_RET_ERR);
+	}
+
+
+
+

The code first checks if the core supports the OMSRgetSupportedTplOpts() API (which is +also not present in all versions!) and, if so, queries the core if the OMSR_TPL_AS_ARRAY mode +is supported. If either does not exits, the core is too old for this functionality. The sample +snippet above then shuts down, but a plugin may instead just do things different. In +omstdout, you can see how a plugin may deal with the situation. +

In any case, it is recommended that at least a graceful shutdown is made and the +array-passing capability not blindly be used. In such cases, we can not guard the +plugin from segfaulting and if the plugin (as currently always) is run within +rsyslog's process space, that results in a segfault for rsyslog. So do not do this. +

Batching of Messages

+

With the current plugin interface, each message is passed via a separate call to the plugin. +This is annoying and costs performance in some uses cases (primarily for database outputs). +However, that's the way it (currently) is, no easy way around it. There are some ideas +to implement batching capabilities inside the rsyslog core, but without that the only +resort is to do it inside your plugin yourself. You are not prohibited from doing so. +There are some consequences, though: most importantly, the rsyslog core is no longer +intersted in messages that it passed to a plugin. As such, it will not try to make sure +the message is not lost before it was ultimately processed (because rsyslog, due to +doAction() returning successfully, thinks the message *was* ultimately processed). +

When the rsyslog core receives batching capabilities, this will be implemented in +a way that is fully compatible to the existing plugin interface. While we have not yet +thought about the implementation, that will probably mean that some new interfaces +or options be used to turn on batching capabilities. +

Licensing

+

From the rsyslog point of view, plugins constitute separate projects. As such, +we think plugins are not required to be compatible with GPLv3. However, this is +no legal advise. If you intend to release something under a non-GPLV3 compatible license +it is probably best to consult with your lawyer. +

Most importantly, and this is definite, the rsyslog team does not expect +or require you to contribute your plugin to the rsyslog project (but of course +we are happy if you do). +

Copyright

+

Copyright (c) 2009 Rainer Gerhards +and Adiscon.

+

Permission is granted to copy, distribute and/or modify this document under +the terms of the GNU Free Documentation License, Version 1.2 or any later +version published by the Free Software Foundation; with no Invariant Sections, +no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be +viewed at +http://www.gnu.org/copyleft/fdl.html.

+ + diff --git a/doc/manual.html b/doc/manual.html index 99a90594..0b03a255 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -64,7 +64,11 @@ the syslog priority (severity and facility) to the log file syslog sender over NAT (online only)
  • an overview and howto of rsyslog gssapi support
  • debug support in rsyslog
    • -
  • the rsyslog message queue object (developer's view)
    • +
  • Developer Documentation +

    • Our rsyslog history page is for you if you would like to learn a little more diff --git a/doc/status.html b/doc/status.html index 59fd0809..dae94884 100644 --- a/doc/status.html +++ b/doc/status.html @@ -2,19 +2,19 @@ rsyslog status page

    rsyslog status page

    -

    This page reflects the status as of 2009-02-02.

    +

    This page reflects the status as of 2009-04-03.

    Current Releases

    development: 4.1.5 [2009-03-11] - change log - download -
    beta: 3.21.10 [2009-02-02] - -change log - -download

    +
    beta: 3.21.11 [2009-04-03] - +change log - +download

    -

    v3 stable: 3.20.3 [2009-01-19] - change log - -download +

    v3 stable: 3.20.3 [2009-04-02] - change log - +download
    v2 stable: 2.0.6 [2008-08-07] - change log - download -- cgit