From 48480e833488c8bc571c62ad84331038edddf3c3 Mon Sep 17 00:00:00 2001 From: Rainer Gerhards Date: Fri, 21 Nov 2008 15:38:34 +0100 Subject: improved doc a big --- doc/manual.html | 3 +++ doc/queues.html | 79 +++++++++++++++++++++++++-------------------------------- 2 files changed, 38 insertions(+), 44 deletions(-) diff --git a/doc/manual.html b/doc/manual.html index e208755d..1762fcca 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -16,6 +16,9 @@ relay chains while at the same time being very easy to setup for the novice user. And as we know what enterprise users really need, there is also professional rsyslog support available directly from the source!

+

Please visit the rsyslog sponsor's page +to honor the project sponsor or become one yourself! We are very grateful for any help towards the +project goals.

This documentation is for version 4.1.0 (devel branch) of rsyslog. Visit the rsyslog status page to obtain current version information and project status. diff --git a/doc/queues.html b/doc/queues.html index 727bc26a..41c5865f 100644 --- a/doc/queues.html +++ b/doc/queues.html @@ -17,6 +17,36 @@ rule processor, which then evaluates which actions are to be carried out. In front of each action, there is also a queue, which potentially de-couples the filter processing from the actual action (e.g. writing to file, database or forwarding to another host).

+

Where are Queues Used?

+

 Currently, queues are used for the main message queue and for the +actions.

+

There is a single main message queue inside rsyslog. Each input module +delivers messages to it. The main message queue worker filters messages based on +rules specified in rsyslog.conf and dispatches them to the individual action +queues. Once a message is in an action queue, it is deleted from the main +message queue.

+

There are multiple action queues, one for each configured action. By default, +these queues operate in direct (non-queueing) mode. Action queues are fully +configurable and thus can be changed to whatever is best for the given use case.

+

Future versions of rsyslog will most probably utilize queues at other places, +too.

+

Wherever "<object>"  is used in the config file +statements, substitute "<object>" with either "MainMsg" or "Action". The +former will set main message queue +parameters, the later parameters for the next action that will be +created. Action queue parameters can not be modified once the action has been +specified. For example, to tell the main message queue to save its content on +shutdown, use $MainMsgQueueSaveOnShutdown on".

+

If the same parameter is specified multiple times before a queue is created, +the last one specified takes precedence. The main message queue is created after +parsing the config file and all of its potential includes. An action queue is +created each time an action selector is specified. Action queue parameters are +reset to default after an action queue has been created (to provide a clean +environment for the next action).

+

Not all queues necessarily support the full set of queue configuration +parameters, because not all are applicable. For example, in current output +module design, actions do not support multi-threading. Consequently, the number +of worker threads is fixed to one for action queues and can not be changed.

Queue Modes

Rsyslog supports different queue modes, some with submodes. Each of them has specific advantages and disadvantages. Selecting the right queue mode is quite @@ -114,8 +144,7 @@ only memory if in use. A FixedArray queue may have a too large static memory footprint in such cases.

In general, it is advised to use LinkedList mode if in doubt. The processing overhead compared to FixedArray is low and may be - -outweigh by the reduction in memory use. Paging in most-often-unused +outweigh by the reduction in memory use. Paging in most-often-unused pointer array pages can be much slower than dynamically allocating them.

To create an in-memory queue, use the "$<object>QueueType LinkedList" or  "$<object>QueueType FixedArray" config directive.

@@ -260,19 +289,15 @@ unavoidable and you prefer to discard less important messages first.

disk space, it is finally full. If so, rsyslogd throttles the data element submitter. If that, for example, is a reliable input (TCP, local log socket), that will slow down the message originator which is a good - -resolution for this scenario.

-

During - -throtteling, a disk-assisted queue continues to write to disk and +resolution for this scenario.

+

During throtteling, a disk-assisted queue continues to write to disk and messages are also discarded based on severity as well as regular dequeuing and processing continues. So chances are good the situation will be resolved by simply throttling. Note, though, that throtteling is highly undesirable for unreliable sources, like UDP message reception. So it is not a good thing to run into throtteling mode at all.

We can not hold processing - -infinitely, not even when throtteling. For example, throtteling the local +infinitely, not even when throtteling. For example, throtteling the local log socket too long would cause the system at whole come to a standstill. To prevent this, rsyslogd times out after a configured period ("$<object>QueueTimeoutEnqueue", specified in milliseconds) if no space becomes available. As a last resort, it @@ -303,8 +328,7 @@ There are two configuration directives, both should be used together or results are unpredictable:" $<object>QueueDequeueTimeBegin <hour>" and "$<object>QueueDequeueTimeEnd <hour>". The hour parameter must be specified in 24-hour format (so 10pm is 22). A use case for this parameter can be found in the rsyslog wiki.

Terminating Queues

Terminating a process sounds easy, but can be complex. - -Terminating a running queue is in fact the most complex operation a queue +Terminating a running queue is in fact the most complex operation a queue object can perform. You don't see that from a user's point of view, but its quite hard work for the developer to do everything in the right order.

The complexity arises when the queue has still data enqueued when it @@ -325,39 +349,6 @@ it terminates. This includes data elements there were begun being processed by workers that needed to be cancelled due to too-long processing. For a large queue, this operation may be lengthy. No timeout applies to a required shutdown save.

-

Where are Queues Used?

-

 Currently, queues are used for the main message queue and for the -actions.

-

There is a single main message queue inside rsyslog. Each input module -delivers messages to it. The main message queue worker filters messages based on -rules specified in rsyslog.conf and dispatches them to the individual action -queues. Once a message is in an action queue, it is deleted from the main -message queue.

-

There are multiple action queues, one for each configured action. By default, -these queues operate in direct (non-queueing) mode. Action queues are fully -configurable and thus can be changed to whatever is best for the given use case.

-

Future versions of rsyslog will most probably utilize queues at other places, -too.

-

- -Wherever "<object>"  was used above in the config file -statements, substitute "<object>" with either "MainMsg" or "Action". The -former will set main message queue - -parameters, the later parameters for the next action that will be -created. Action queue parameters can not be modified once the action has been -specified. For example, to tell the main message queue to save its content on -shutdown, use $MainMsgQueueSaveOnShutdown on".

-

If the same parameter is specified multiple times before a queue is created, -the last one specified takes precedence. The main message queue is created after -parsing the config file and all of its potential includes. An action queue is -created each time an action selector is specified. Action queue parameters are -reset to default after an action queue has been created (to provide a clean -environment for the next action).

-

Not all queues necessarily support the full set of queue configuration -parameters, because not all are applicable. For example, in current output -module design, actions do not support multi-threading. Consequently, the number -of worker threads is fixed to one for action queues and can not be changed.

[manual index] [rsyslog.conf] [rsyslog site]

-- cgit