From fc166fb84d027cf3f7fb54c8cdce44ec682fc3c2 Mon Sep 17 00:00:00 2001 From: Rainer Gerhards Date: Tue, 29 Jan 2008 11:38:06 +0000 Subject: added some info on queues (needs to be extended) --- doc/queues.html | 79 +++++++++++++++++++++++++++++++++++++++++++++++++++ doc/rsyslog_conf.html | 3 ++ 2 files changed, 82 insertions(+) create mode 100644 doc/queues.html diff --git a/doc/queues.html b/doc/queues.html new file mode 100644 index 00000000..7f651a4c --- /dev/null +++ b/doc/queues.html @@ -0,0 +1,79 @@ + + + + +Understanding rsyslog queues + + + + +

Understanding rsyslog Queues

Rsyslog uses queues whenever two activities need to be losely coupled. With a +queue, one part of the system "produces" something while another part "consumes" +this something. The "something" is most often syslog messages, but queues may +also be used for other purposes.

The most prominent example is the main message queue. Whenever rsyslog +receives a message (e.g. locally, via UDP, TCP or in whatever else way), it +places these messages into the main message queue. Later, it is dequeued by the +rule processor, which then evaluates which actions are to be carried out. In +front of each action, there is also a queue, which potentially de-couples the +filter processing from the actual action (e.g. writing to file, database or +forwarding to another host).

Queue Modes

Rsyslog supports different queue modes, some with submodes. Each of them has +specific advantages and disadvantages. Selecting the right queue mode is quite +important when tuning rsyslogd.

Direct Queues

Direct queues are non-queueing queues. A queue in direct mode does +neither queue nor bufer any of the queue elements but rather passes the element +directly (and immediately) from the producer to the consumer.This sounds strange, +but there is a good reason for this queue type.

Direct mode queues allow to use queues generically, even in places where +queing is not always desired. A good example is the queue in front of output +actions. While it makes perfect sense to buffer forwarding actions or database +writes, it makes only limited sense to build up a queue in front of simple local +file writes. Yet, rsyslog still has a queue in front of every action. So for +file writes, the queue mode can simply be set to "direct", in which case no +queueing happens.

Please note that a direct queue also is the only queue type that passes back +the execution return code (sucess/failure) from the consumer to the producer. +This, for example, is needed for the backup action logic. Consequently, backup +actions require the to-be-checked action to use a "direct" mode queue.

Disk Queues

Disk queues use disk drives for buffering. The important fact is that the +always use the disk and do not buffer anything in memory. Thus, the queue is +ultra-reliable, but by far the slowest mode. For regular use cases, this queue +mode is not recommended. It is useful if log data is so important that it must +not be lost, even in extreme cases.

Please note, however, that the disk queue by default does not update its +bookkeeping structures every time it writes to disk. This is for performance +reasons. In the event of failure, data will still be lost (except when manually +is mangled with the file structures). However, disk queues can be set to write +bookkeeping information on checkpoints (every n records), so that this can be +made ultra-reliable, too. If the checkpoint interval is set to one, no data can +be lost, but the queue is exceptionally slow.

In-Memory Queues

In-memory queue mode is what most people have on their mind when they think +about computing queues. Here, the enqueued data elements are held in memory. +Consequently, in-memory queues are very fast. But of course, they do not survive +any program or operating system abort (what usually is tolerable and unlikely). +Be sure to use an UPS if you use in-memory mode and your log data is important +to you. Note that even in-memory queues may hold data for an infinite amount of +time if e.g. an output destination system is down and there is no reason to move +the data out of memory (lying around in memory for an extended period of time is +NOT a reason). Pure in-memory queues can't even store queue elements anywhere +else than in core memory.

If a disk queue name is defined for in-memory queues, they automatically +become "disk-assisted" (DA). In that mode, data is written to disk (and read +back) on a per-needed basis. Actually, the regular memory queue (called the +"primary queue") and a disk queue (called the "DA queue") work in tandem in this +mode. Most importantly, the disk queue is activated if the primary queue is full +or needs to be persisted on shutdown. Disk-assisted queues combine the +advantages of pure memory queues with those from pure disk queues. Under normal +operations, they are very fast and messages will never touch the disk. But if +there is need to, an unlimited amount of messages can be buffered (actually +limited by free space) and data can be persisted between rsyslogd runs.

+ + + + diff --git a/doc/rsyslog_conf.html b/doc/rsyslog_conf.html index 929a8a0b..0f3f2a2d 100644 --- a/doc/rsyslog_conf.html +++ b/doc/rsyslog_conf.html @@ -29,6 +29,9 @@ them are in bold. A more in-depth description will appear as implementation progresses. Directives may change during that process, we will NOT try hard to maintain backwards compatibility (after all, v3 is still very early in development and quite unstable...). So you have been warned ;)

Be sure to read information about queues in rsyslog +- many parameter settings modify queue parameters. If in doubt, use the default, +it is usually well-choosen and applicable in most cases.