From fc166fb84d027cf3f7fb54c8cdce44ec682fc3c2 Mon Sep 17 00:00:00 2001 From: Rainer Gerhards Date: Tue, 29 Jan 2008 11:38:06 +0000 Subject: added some info on queues (needs to be extended) --- doc/queues.html | 79 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 doc/queues.html (limited to 'doc/queues.html') diff --git a/doc/queues.html b/doc/queues.html new file mode 100644 index 00000000..7f651a4c --- /dev/null +++ b/doc/queues.html @@ -0,0 +1,79 @@ + + + + +Understanding rsyslog queues + + + + +

Understanding rsyslog Queues

Rsyslog uses queues whenever two activities need to be losely coupled. With a +queue, one part of the system "produces" something while another part "consumes" +this something. The "something" is most often syslog messages, but queues may +also be used for other purposes.

The most prominent example is the main message queue. Whenever rsyslog +receives a message (e.g. locally, via UDP, TCP or in whatever else way), it +places these messages into the main message queue. Later, it is dequeued by the +rule processor, which then evaluates which actions are to be carried out. In +front of each action, there is also a queue, which potentially de-couples the +filter processing from the actual action (e.g. writing to file, database or +forwarding to another host).

Queue Modes

Rsyslog supports different queue modes, some with submodes. Each of them has +specific advantages and disadvantages. Selecting the right queue mode is quite +important when tuning rsyslogd.

Direct Queues

Direct queues are non-queueing queues. A queue in direct mode does +neither queue nor bufer any of the queue elements but rather passes the element +directly (and immediately) from the producer to the consumer.This sounds strange, +but there is a good reason for this queue type.

Direct mode queues allow to use queues generically, even in places where +queing is not always desired. A good example is the queue in front of output +actions. While it makes perfect sense to buffer forwarding actions or database +writes, it makes only limited sense to build up a queue in front of simple local +file writes. Yet, rsyslog still has a queue in front of every action. So for +file writes, the queue mode can simply be set to "direct", in which case no +queueing happens.

Please note that a direct queue also is the only queue type that passes back +the execution return code (sucess/failure) from the consumer to the producer. +This, for example, is needed for the backup action logic. Consequently, backup +actions require the to-be-checked action to use a "direct" mode queue.

Disk Queues

Disk queues use disk drives for buffering. The important fact is that the +always use the disk and do not buffer anything in memory. Thus, the queue is +ultra-reliable, but by far the slowest mode. For regular use cases, this queue +mode is not recommended. It is useful if log data is so important that it must +not be lost, even in extreme cases.

Please note, however, that the disk queue by default does not update its +bookkeeping structures every time it writes to disk. This is for performance +reasons. In the event of failure, data will still be lost (except when manually +is mangled with the file structures). However, disk queues can be set to write +bookkeeping information on checkpoints (every n records), so that this can be +made ultra-reliable, too. If the checkpoint interval is set to one, no data can +be lost, but the queue is exceptionally slow.

In-Memory Queues

In-memory queue mode is what most people have on their mind when they think +about computing queues. Here, the enqueued data elements are held in memory. +Consequently, in-memory queues are very fast. But of course, they do not survive +any program or operating system abort (what usually is tolerable and unlikely). +Be sure to use an UPS if you use in-memory mode and your log data is important +to you. Note that even in-memory queues may hold data for an infinite amount of +time if e.g. an output destination system is down and there is no reason to move +the data out of memory (lying around in memory for an extended period of time is +NOT a reason). Pure in-memory queues can't even store queue elements anywhere +else than in core memory.

If a disk queue name is defined for in-memory queues, they automatically +become "disk-assisted" (DA). In that mode, data is written to disk (and read +back) on a per-needed basis. Actually, the regular memory queue (called the +"primary queue") and a disk queue (called the "DA queue") work in tandem in this +mode. Most importantly, the disk queue is activated if the primary queue is full +or needs to be persisted on shutdown. Disk-assisted queues combine the +advantages of pure memory queues with those from pure disk queues. Under normal +operations, they are very fast and messages will never touch the disk. But if +there is need to, an unlimited amount of messages can be buffered (actually +limited by free space) and data can be persisted between rsyslogd runs.

+ + + + -- cgit