diff options
Diffstat (limited to 'doc')
30 files changed, 424 insertions, 32 deletions
diff --git a/doc/Makefile.am b/doc/Makefile.am index 3015d6b5..4d9d94ff 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -33,6 +33,7 @@ html_files = \ dev_queue.html \ omsnmp.html \ ommysql.html \ + omoracle.html \ omlibdbi.html \ imfile.html \ imtcp.html \ @@ -108,6 +109,20 @@ html_files = \ rsyslog_conf_output.html \ rsyslog_conf_templates.html \ rsyslog_conf_nomatch.html \ + queues_analogy.html \ src/classes.dia -EXTRA_DIST = $(html_files) +grfx_files = \ + direct_queue0.png \ + direct_queue1.png \ + direct_queue2.png \ + direct_queue3.png \ + direct_queue_rsyslog.png \ + direct_queue_rsyslog2.png \ + direct_queue_directq.png \ + dataflow.png \ + queue_analogy_tv.png \ + gssapi.png \ + rsyslog-vers.png + +EXTRA_DIST = $(html_files) $(grfx_files) diff --git a/doc/dataflow.png b/doc/dataflow.png Binary files differnew file mode 100644 index 00000000..fd614d8c --- /dev/null +++ b/doc/dataflow.png diff --git a/doc/direct_queue0.png b/doc/direct_queue0.png Binary files differnew file mode 100644 index 00000000..6d1b373f --- /dev/null +++ b/doc/direct_queue0.png diff --git a/doc/direct_queue1.png b/doc/direct_queue1.png Binary files differnew file mode 100644 index 00000000..503f8151 --- /dev/null +++ b/doc/direct_queue1.png diff --git a/doc/direct_queue2.png b/doc/direct_queue2.png Binary files differnew file mode 100644 index 00000000..cbb99af8 --- /dev/null +++ b/doc/direct_queue2.png diff --git a/doc/direct_queue3.png b/doc/direct_queue3.png Binary files differnew file mode 100644 index 00000000..cc49299f --- /dev/null +++ b/doc/direct_queue3.png diff --git a/doc/direct_queue_directq.png b/doc/direct_queue_directq.png Binary files differnew file mode 100644 index 00000000..c5d8769d --- /dev/null +++ b/doc/direct_queue_directq.png diff --git a/doc/direct_queue_rsyslog.png b/doc/direct_queue_rsyslog.png Binary files differnew file mode 100644 index 00000000..6150222d --- /dev/null +++ b/doc/direct_queue_rsyslog.png diff --git a/doc/direct_queue_rsyslog2.png b/doc/direct_queue_rsyslog2.png Binary files differnew file mode 100644 index 00000000..807b064d --- /dev/null +++ b/doc/direct_queue_rsyslog2.png diff --git a/doc/features.html b/doc/features.html index 17a995bf..626ff65d 100644 --- a/doc/features.html +++ b/doc/features.html @@ -127,7 +127,6 @@ community. Plus, it can be financially attractive: just think about how much les be to sponsor a feature instead of purchasing a commercial implementation. Also, the benefit of being recognised as a sponsor may even drive new customers to your business!</b> <ul> -<li>Finalize the DTN "planetary Internet" space ship mode output plugin <li>port it to more *nix variants (eg AIX and HP UX) - this needs volunteers with access to those machines and knowledge </li> <li>pcre filtering - maybe (depending on feedback) - diff --git a/doc/manual.html b/doc/manual.html index 51271701..ae90f780 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -19,7 +19,7 @@ rsyslog support</a> available directly from the source!</p> <p><b>Please visit the <a href="http://www.rsyslog.com/sponsors">rsyslog sponsor's page</a> to honor the project sponsors or become one yourself!</b> We are very grateful for any help towards the project goals.</p> -<p><b>This documentation is for version 4.1.6 (devel branch) of rsyslog.</b> +<p><b>This documentation is for version 4.3.0 (devel branch) of rsyslog.</b> Visit the <i> <a href="http://www.rsyslog.com/doc-status.html">rsyslog status page</a></i></b> to obtain current version information and project status. </p><p><b>If you like rsyslog, you might diff --git a/doc/modules.html b/doc/modules.html index 92887508..4eae6db3 100644 --- a/doc/modules.html +++ b/doc/modules.html @@ -4,9 +4,8 @@ </head> <body> <h1>About rsyslog Modules</h1> - <P><small><i>Written by - <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer - Gerhards</a> (2007-07-28)</i></small></P> +<P><small><i>Written by +<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> (2007-07-28)</i></small></P> <p><font color="#FF0000"><b>This document is incomplete. The module interface is also quite incomplete and under development. Do not currently use it!</b></font> You may want to visit <a href="http://rgerhards.blogspot.com/">Rainer's blog</a> diff --git a/doc/omoracle.html b/doc/omoracle.html new file mode 100644 index 00000000..40f6360f --- /dev/null +++ b/doc/omoracle.html @@ -0,0 +1,78 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>Oracle Database Output Module</title> +</head> + +<body> +<a href="rsyslog_conf_modules.html">rsyslog module reference</a> + +<h1>Oracle Database Output Module</h1> +<p><b>Module Name: omoracle</b></p> +<p><b>Author: </b>Luis Fernando Muñoz Mejías <Luis.Fernando.Munoz.Mejias@cern.ch></p> +<p><b>Available since: </b>: 4.3.0 +<p><b>Status: </b>: contributed module, not maitained by rsyslog core authors +<p><b>Description</b>:</p> +<p>This module provides native support for logging to Oracle databases. It offers +superior performance over the more generic <a href="omlibdbi.html">omlibdbi</a> module. +It also includes a number of enhancements, most importantly prepared statements and +batching, what provides a big performance improvements. +</p> +<p>Note that this module is maintained by its original author. If you need assistance with it, +it is suggested to post questions to the +<a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">rsyslog mailing list</a>. +<p>From the header comments of this module: +<p><pre> + + This is an output module feeding directly to an Oracle + database. It uses Oracle Call Interface, a propietary module + provided by Oracle. + + Selector lines to be used are of this form: + + :omoracle:;TemplateName + + The module gets its configuration via rsyslog $... directives, + namely: + + $OmoracleDBUser: user name to log in on the database. + + $OmoracleDBPassword: password to log in on the database. + + $OmoracleDB: connection string (an Oracle easy connect or a db + name as specified by tnsnames.ora) + + $OmoracleBatchSize: Number of elements to send to the DB on each + transaction. + + $OmoracleStatement: Statement to be prepared and executed in + batches. Please note that Oracle's prepared statements have their + placeholders as ':identifier', and this module uses the colon to + guess how many placeholders there will be. + + All these directives are mandatory. The dbstring can be an Oracle + easystring or a DB name, as present in the tnsnames.ora file. + + The form of the template is just a list of strings you want + inserted to the DB, for instance: + + $template TestStmt,"%hostname%%msg%" + + Will provide the arguments to a statement like + + $OmoracleStatement \ + insert into foo(hostname,message)values(:host,:message) + + Also note that identifiers to placeholders are arbitrarry. You + need to define the properties on the template in the correct order + you want them passed to the statement! +</pre> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/queue_analogy_tv.png b/doc/queue_analogy_tv.png Binary files differnew file mode 100644 index 00000000..fedcb558 --- /dev/null +++ b/doc/queue_analogy_tv.png diff --git a/doc/queues.html b/doc/queues.html index 41c5865f..4a9509a0 100644 --- a/doc/queues.html +++ b/doc/queues.html @@ -1,6 +1,5 @@ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html><head> -<meta http-equiv="Content-Language" content="de"> <title>Understanding rsyslog queues</title></head> <body> <a href="rsyslog_conf_global.html">back</a> @@ -10,6 +9,13 @@ queue, one part of the system "produces" something while another part "consumes" this something. The "something" is most often syslog messages, but queues may also be used for other purposes.</p> +<p>This document provides a good insight into technical details, operation modes +and implications. In addition to it, an +<a href="queues_analogy.html">rsyslog queue concepts overview</a> document +exists which tries to explain queues with the help of some analogies. This may +probably be a better place to start reading about queues. I assume that once you +have understood that document, the material here will be much easier to grasp +and look much more natural. <p>The most prominent example is the main message queue. Whenever rsyslog receives a message (e.g. locally, via UDP, TCP or in whatever else way), it places these messages into the main message queue. Later, it is dequeued by the @@ -18,7 +24,7 @@ front of each action, there is also a queue, which potentially de-couples the filter processing from the actual action (e.g. writing to file, database or forwarding to another host).</p> <h1>Where are Queues Used?</h1> -<p> Currently, queues are used for the main message queue and for the +<p>Currently, queues are used for the main message queue and for the actions.</p> <p>There is a single main message queue inside rsyslog. Each input module delivers messages to it. The main message queue worker filters messages based on @@ -354,8 +360,8 @@ save.</p> [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL -version 2 or higher.</font></p> +version 3 or higher.</font></p> </body></html> diff --git a/doc/queues_analogy.html b/doc/queues_analogy.html new file mode 100644 index 00000000..1584c66d --- /dev/null +++ b/doc/queues_analogy.html @@ -0,0 +1,259 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>turning lanes and rsyslog queues - an analogy</title></head> +<body> +<a href="rsyslog_conf_global.html">back</a> + +<h1>Turning Lanes and Rsyslog Queues - an Analogy</h1> +<p>If there is a single object absolutely vital to understanding the way +rsyslog works, this object is queues. Queues offer a variety of services, +including support for multithreading. While there is elaborate in-depth +documentation on the ins and outs of <a href="queues.html">rsyslog queues</a>, +some of the concepts are hard to grasp even for experienced people. I think this +is because rsyslog uses a very high layer of abstraction which includes things +that look quite unnatural, like queues that do <b>not</b> actually queue... +<p>With this document, I take a different approach: I will not describe every specific +detail of queue operation but hope to be able to provide the core idea of how +queues are used in rsyslog by using an analogy. I will compare the rsyslog data flow +with real-life traffic flowing at an intersection. +<p>But first let's set the stage for the rsyslog part. The graphic below describes +the data flow inside rsyslog: +<p align="center"><img src="dataflow.png" alt="rsyslog data flow"> +<p>Note that there is a <a href="http://www.rsyslog.com/Article350.phtml">video tutorial</a> +available on the data flow. It is not perfect, but may aid in understanding this picture. +<p>For our needs, the important fact to know is that messages enter rsyslog on "the +left side" (for example, via UDP), are being preprocessed, put into the +so-called main queue, taken off that queue, be filtered and be placed into one or +several action queues (depending on filter results). They leave rsyslog on "the +right side" where output modules (like the file or database writer) consume them. +<p>So there are always <b>two</b> stages where a message (conceptually) is queued - first +in the main queue and later on in <i>n</i> action specific queues (with <i>n</i> being the number of +actions that the message in question needs to be processed by, what is being decided +by the "Filter Engine"). As such, a message will be in at least two queues +during its lifetime (with the exeception of messages being discarded by the queue itself, +but for the purpose of this document, we will ignore that possibility). +<p>Also, it is vitally +important to understand that <b>each</b> action has a queue sitting in front of it. +If you have dug into the details of rsyslog configuration, you have probably seen that +a queue mode can be set for each action. And the default queue mode is the so-called +"direct mode", in which "the queue does not actually enqueue data". +That sounds silly, but is not. It is an important abstraction that helps keep the code clean. +<p>To understand this, we first need to look at who is the active component. In our data flow, +the active part always sits to the left of the object. For example, the "Preprocessor" +is being called by the inputs and calls itself into the main message queue. That is, the queue +receiver is called, it is passive. One might think that the "Parser & Filter Engine" +is an active component that actively pulls messages from the queue. This is wrong! Actually, +it is the queue that has a pool of worker threads, and these workers pull data from the queue +and then call the passively waiting Parser and Filter Engine with those messages. So the +main message queue is the active part, the Parser and Filter Engine is passive. +<p>Let's now try an anlogy analogy for this part: Think about a TV show. The show is produced +in some TV studio, from there sent (actively) to a radio tower. The radio tower passively +receives from the studio and then actively sends out a signal, which is passively received +by your TV set. In our simplified view, we have the following picture: +<p align="center"><img src="queue_analogy_tv.png" alt="rsyslog queues and TV analogy"> +<p>The lower part of the picture lists the equivalent rsyslog entities, in an abstracted way. +Every queue has a producer (in the above sample the input) and a consumer (in the above sample the Parser +and Filter Engine). Their active and passive functions are equivalent to the TV entities +that are listed on top of the rsyslog entity. For example, a rsyslog consumer can never +actively initate reception of a message in the same way a TV set can not actively +"initiate" a TV show - both can only "handle" (display or process) +what is sent to them. +<p>Now let's look at the action queues: here, the active part, the producer, is the +Parser and Filter Engine. The passive part is the Action Processor. The later does any +processing that is necessary to call the output plugin, in particular it processes the template +to create the plugin calling parameters (eiter a string or vector of arguments). From the +action queue's point of view, Action Processor and Output form a single entity. Again, the +TV set analogy holds. The Output <b>does not</b> actively ask the queue for data, but +rater passively waits until the queue itself pushes some data to it. + +<p>Armed with this knowledge, we can now look at the way action queue modes work. My analogy here +is a junction, as shown below (note that the colors in the pictures below are <b>not</b> related to +the colors in the pictures above!): +<p align="center"><img src="direct_queue0.png"> +<p>This is a very simple real-life traffic case: one road joins another. We look at +traffic on the straight road, here shown by blue and green arrows. Traffic in the +opposing direction is shown in blue. Traffic flows without +any delays as long as nobody takes turns. To be more precise, if the opposing traffic takes +a (right) turn, traffic still continues to flow without delay. However, if a car in the red traffic +flow intend to do a (left, then) turn, the situation changes: +<p align="center"><img src="direct_queue1.png"> +<p>The turning car is represented by the green arrow. It can not turn unless there is a gap +in the "blue traffic stream". And as this car blocks the roadway, the remaining +traffic (now shown in red, which should indicate the block condition), +must wait until the "green" car has made its turn. So +a queue will build up on that lane, waiting for the turn to be completed. +Note that in the examples below I do not care that much about the properties of the +opposing traffic. That is, because its structure is not really important for what I intend to +show. Think about the blue arrow as being a traffic stream that most of the time blocks +left-turners, but from time to time has a gap that is sufficiently large for a left-turn +to complete. +<p>Our road network designers know that this may be unfortunate, and for more important roads +and junctions, they came up with the concept of turning lanes: +<p align="center"><img src="direct_queue2.png"> +<p>Now, the car taking the turn can wait in a special area, the turning lane. As such, +the "straight" traffic is no longer blocked and can flow in parallel to the +turning lane (indicated by a now-green-again arrow). + +<p>However, the turning lane offers only finite space. So if too many cars intend to +take a left turn, and there is no gap in the "blue" traffic, we end up with +this well-known situation: +<p align="center"><img src="direct_queue3.png"> +<p>The turning lane is now filled up, resulting in a tailback of cars intending to +left turn on the main driving lane. The end result is that "straight" traffic +is again being blocked, just as in our initial problem case without the turning lane. +In essence, the turning lane has provided some relief, but only for a limited amount of +cars. Street system designers now try to weight cost vs. benefit and create (costly) +turning lanes that are sufficiently large to prevent traffic jams in most, but not all +cases. +<p><b>Now let's dig a bit into the mathematical properties of turning lanes.</b> We assume that +cars all have the same length. So, units of cars, the length is alsways one (which is nice, +as we don't need to care about that factor any longer ;)). A turning lane has finite capacity of +<i>n</i> cars. As long as the number of cars wanting to take a turn is less than or eqal +to <i>n</i>, "straigth traffic" is not blocked (or the other way round, traffic +is blocked if at least <i>n + 1</i> cars want to take a turn!). We can now find an optimal +value for <i>n</i>: it is a function of the probability that a car wants to turn +and the cost of the turning lane +(as well as the probability there is a gap in the "blue" traffic, but we ignore this +in our simple sample). +If we start from some finite upper bound of <i>n</i>, we can decrease +<i>n</i> to a point where it reaches zero. But let's first look at <i>n = 1</i>, in which case exactly +one car can wait on the turning lane. More than one car, and the rest of the traffic is blocked. +Our everyday logic indicates that this is actually the lowest boundary for <i>n</i>. +<p>In an abstract view, however, <i>n</i> can be zero and that works nicely. There still can be +<i>n</i> cars at any given time on the turning lane, it just happens that this means there can +be no car at all on it. And, as usual, if we have at least <i>n + 1</i> cars wanting to turn, +the main traffic flow is blocked. True, but <i>n + 1 = 0 + 1 = 1</i> so as soon as there is any +car wanting to take a turn, the main traffic flow is blocked (remeber, in all cases, I assume +no sufficently large gaps in the opposing trafic). +<p>This is the situation our everyday perception calls "road without turning lane". +In my math model, it is a "road with turning lane of size 0". The subtle difference +is important: my math model guarantees that, in an abstract sense, there always is a turning +lane, it may just be too short. But it exists, even though we don't see it. And now I can +claim that even in my small home village, all roads have turning lanes, which is rather +impressive, isn't it? ;) +<p><b>And now we finally have arrived at rsyslog's queues!</b> Rsyslog action queues exists for +all actions just like all roads in my village have turning lanes! And as in this real-life sample, +it may be hard to see the action queues for that reason. In rsyslog, the "direct" queue +mode is the equivalent to the 0-sized turning lane. And actions queues are the equivalent to turning +lanes in general, with our real-life <i>n</i> being the maximum queue size. +The main traffic line (which sometimes is blocked) is the equivalent to the +main message queue. And the periods without gaps in the opposing traffic are equivalent to +execution time of an action. In a rough sketch, the rsyslog main and action queues look like in the +following picture. +<p align="center"><img src="direct_queue_rsyslog.png"> +<p>We need to read this picture from right to left (otherwise I would need to redo all +the graphics ;)). In action 3, you see a 0-sized turning lane, aka an action queue in "direct" +mode. All other queues are run in non-direct modes, but with different sizes greater than 0. +<p>Let us first use our car analogy: +Assume we are in a car on the main lane that wants to take turn into the "action 4" +road. We pass action 1, where a number of cars wait in the turning lane and we pass +action 2, which has a slightly smaller, but still not filled up turning lane. So we pass that +without delay, too. Then we come to "action 4", which has no turning lane. Unfortunately, +the car in front of us wants to turn left into that road, so it blocks the main lane. So, this time +we need to wait. An observer standing on the sidewalk may see that while we need to wait, there are +still some cars in the "action 4" turning lane. As such, even though no new cars can +arrive on the main lane, cars still turn into the "action 4" lane. In other words, +an observer standing in "action 4" road is unable to see that traffic on the main lane +is blocked. +<p>Now on to rsyslog: Other than in the real-world traffic example, messages in rsyslog +can - at more or less the +same time - "take turns" into several roads at once. This is done by duplicating the message +if the road has a non-zero-sized "turning lane" - or in rsyslog terms a queue that is +running in any non-direct mode. If so, a deep copy of the message object is made, that placed into +the action queue and then the initial message proceeds on the "main lane". The action +queue then pushes the duplicates through action processing. This is also the reason why a +discard action inside a non-direct queue does not seem to have an effect. Actually, it discards the +copy that was just created, but the original message object continues to flow. +<p> +In action 1, we have some entries in the action queue, as we have in action 2 (where the queue is +slightly shorter). As we have seen, new messages pass action one and two almost instantaneously. +However, when a messages reaches action 3, its flow is blocked. Now, message processing must wait +for the action to complete. Processing flow in a direct mode queue is something like a U-turn: + +<p align="center"><img src="direct_queue_directq.png" alt="message processing in an rsyslog action queue in direct mode"> +<p>The message starts to execute the action and once this is done, processing flow continues. +In a real-life analogy, this may be the route of a delivery man who needs to drop a parcel +in a side street before he continues driving on the main route. As a side-note, think of what happens +with the rest of the delivery route, at least for today, if the delivery truck has a serious accident +in the side street. The rest of the parcels won't be delivered today, will they? This is exactly how the +discard action works. It drops the message object inside the action and thus the message will no +longer be available for further delivery - but as I said, only if the discard is done in a +direct mode queue (I am stressing this example because it often causes a lot of confusion). +<p>Back to the overall scenario. We have seen that messages need to wait for action 3 to +complete. Does this necessarily mean that at the same time no messages can be processed +in action 4? Well, it depends. As in the real-life scenario, action 4 will continue to +receive traffic as long as its action queue ("turn lane") is not drained. In +our drawing, it is not. So action 4 will be executed while messages still wait for action 3 +to be completed. +<p>Now look at the overall picture from a slightly different angle: +<p align="center"><img src="direct_queue_rsyslog2.png" alt="message processing in an rsyslog action queue in direct mode"> +<p>The number of all connected green and red arrows is four - one each for action 1, 2 and 3 +(this one is dotted as action 4 was a special case) and one for the "main lane" as +well as acton 3 (this one contains the sole red arrow). <b>This number is the lower bound for +the number of threads in rsyslog's output system ("right-hand part" of the main message +queue)!</b> Each of the connected arrows is a continous thread and each "turn lane" is +a place where processing is forked onto a new thread. Also, note that in action 3 the processing +is carried out on the main thread, but not in the non-direct queue modes. +<p>I have said this is "the lower bound for the number of threads...". This is with +good reason: the main queue may have more than one worker thread (individual action queues +currently do not support this, but could do in the future - there are good reasons for that, too +but exploring why would finally take us away from what we intend to see). Note that you +configure an upper bound for the number of main message queue worker threads. The actual number +varies depending on a lot of operational variables, most importantly the number of messages +inside the queue. The number <i>t_m</i> of actually running threads is within the integer-interval +[0,confLimit] (with confLimit being the operator configured limit, which defaults to 5). +Output plugins may have more than one thread created by themselves. It is quite unusual for an +output plugin to create such threads and so I assume we do not have any of these. +Then, the overall number of threads in rsyslog's filtering and output system is +<i>t_total = t_m + number of actions in non-direct modes</i>. Add the number of +inputs configured to that and you have the total number of threads running in rsyslog at +a given time (assuming again that inputs utilize only one thread per plugin, a not-so-safe +assumption). +<p>A quick side-note: I gave the lower bound for <i>t_m</i> as zero, which is somewhat in contrast +to what I wrote at the begin of the last paragraph. Zero is actually correct, because rsyslog +stops all worker threads when there is no work to do. This is also true for the action queues. +So the ultimate lower bound for a rsyslog output system without any work to carry out actually is zero. +But this bound will never be reached when there is continous flow of activity. And, if you are +curios: if the number of workers is zero, the worker wakeup process is actually handled within the +threading context of the "left-hand-side" (or producer) of the queue. After being +started, the worker begins to play the active queue component again. All of this, of course, +can be overridden with configuraton directives. +<p>When looking at the threading model, one can simply add n lanes to the main lane but otherwise +retain the traffic analogy. This is a very good description of the actual process (think what this +means to the "turning lanes"; hint: there still is only one per action!). +<p><b>Let's try to do a warp-up:</b> I have hopefully been able to show that in rsyslog, an action +queue "sits in front of" each output plugin. Messages are received and flow, from input +to output, over various stages and two level of queues to the outputs. Actions queues are always +present, but may not easily be visible when in direct mode (where no actual queueing takes place). +The "road junktion with turning lane" analogy well describes the way - and intent - of the various +queue levels in rsyslog. +<p>On the output side, the queue is the active component, <b>not</b> the consumer. As such, the consumer +can not ask the queue for anything (like n number of messages) but rather is activated by the queue +itself. As such, a queue somewhat resembles a "living thing" whereas the outputs are +just tools that this "living thing" uses. +<p><b>Note that I left out a couple of subtleties</b>, especially when it comes +to error handling and terminating +a queue (you hopefully have now at least a rough idea why I say "terminating <b>a queue</b>" +and not "terminating an action" - <i>who is the "living thing"?</i>). An action returns +a status to the queue, but it is the queue that ultimately decides which messages can finally be +considered processed and which not. Please note that the queue may even cancel an output right in +the middle of its action. This happens, if configured, if an output needs more than a configured +maximum processing time and is a guard condition to prevent slow outputs from defering a rsyslog +restart for too long. Especially in this case re-queueing and cleanup is not trivial. Also, note that +I did not discuss disk-assisted queue modes. The basic rules apply, but there are some additonal +constraints, especially in regard to the threading model. Transitioning between actual +disk-assisted mode and pure-in-memory-mode (which is done automatically when needed) is also far from +trivial and a real joy for an implementor to work on ;). +<p>If you have not done so before, it may be worth reading the +<a href="queues.html">rsyslog queue user's guide,</a> which most importantly lists all +the knobs you can turn to tweak queue operation. +<p>[<a href="manual.html">manual index</a>] +[<a href="rsyslog_conf.html">rsyslog.conf</a>] +[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> +</body> +</html> diff --git a/doc/rsconf1_maxopenfiles.html b/doc/rsconf1_maxopenfiles.html new file mode 100644 index 00000000..b6c9cc0e --- /dev/null +++ b/doc/rsconf1_maxopenfiles.html @@ -0,0 +1,35 @@ +<html> +<head> +<title>$MaxOpenFiles - rsyslog.conf file</title> +</head> +<body> +<a href="rsyslog_conf_global.html">[rsyslog configuration directive overview]</a> + +<h2>$MaxOpenFiles</h2> +<p><b>Available Since:</b> 4.3.0</p> +<p><b>Type:</b> global configuration directive</p> +<p><b>Default:</b> <i>operating system default</i></p> +<p><b>Description:</b></p> +<p>Set the maximum number of files that the rsyslog process can have open at any given +time. Note that this includes open tcp sockets, so this setting is the upper limit for +the number of open TCP connections as well. If you expect a large nubmer of concurrent +connections, it is suggested that the number is set to the max number connected plus 1000. +Please note that each dynafile also requires up to 100 open file handles. +<p>The setting is similar to running "ulimit -n number-of-files". +<p>Please note that depending on permissions and operating system configuration, the +setrlimit() request issued by rsyslog may fail, in which case the previous limit is kept +in effect. Rsyslog will emit a warning message in this case. +<p><b>Sample:</b></p> +<p><code><b>$MaxOpenFiles 2000</b></code></p> +<p><b>Bugs:</b></p> +<p>For some reason, this settings seems not to work on all platforms. If you experience +problems, please let us know so that we can (hopefully) narrow down the issue. +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 3 or higher.</font></p> +</body> +</html> diff --git a/doc/rsyslog_conf.html b/doc/rsyslog_conf.html index 852d95b5..6990c6bd 100644 --- a/doc/rsyslog_conf.html +++ b/doc/rsyslog_conf.html @@ -26,7 +26,7 @@ Lines can be continued by specifying a backslash ("\") as the last character of the line. There is a hard-coded maximum line length of 4K. If you need lines larger than that, you need to change compile-time settings inside rsyslog and recompile. -<h2><a href="rsyslog_conf_global.html">Global Directives</a></h2> +<h2><a href="rsyslog_conf_global.html">Configuration Directives</a></h2> <h2>Basic Structure</h2> <p>Rsyslog supports standard sysklogd's configuration file format and extends it. So in general, you can take a "normal" syslog.conf and @@ -74,9 +74,9 @@ such features is available in rsyslogd, only.</p> [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008,2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL -version 2 or higher.</font></p> +version 3 or higher.</font></p> </body> </html> > diff --git a/doc/rsyslog_conf_global.html b/doc/rsyslog_conf_global.html index d011bd2b..3e33f0da 100644 --- a/doc/rsyslog_conf_global.html +++ b/doc/rsyslog_conf_global.html @@ -1,14 +1,14 @@ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head><title>Global Directives - rsyslog.conf</title></head> +<html><head><title>Configuration Directives - rsyslog.conf</title></head> <body> <p>This is a part of the rsyslog.conf documentation.</p> <a href="rsyslog_conf.html">back</a> -<h2>Global Directives</h2> -<p>All global directives need to be specified on a line by their -own and must start with a dollar-sign. Here is a list in alphabetical -order. Follow links for a description.</p> -<p>Please note that not all directives here are actually global. Some affect -only the next action. This documentation will be changed soon. +<h2>Configuration Directives</h2> +<p>All configuration directives need to be specified on a line by their +own and must start with a dollar-sign. Note that those starting with +the word "Action" modify the next action and should be specified +in front of it. +<p>Here is a list in alphabetical order. Follow links for a description.</p> <p>Not all directives have an in-depth description right now. Default values for them are in bold. A more in-depth description will appear as implementation progresses. @@ -180,6 +180,7 @@ instead of UDP (plain TCP syslog, RELP). This resolves the UDP stack size restri <br>Note that 2k, the current default, is the smallest size that must be supported in order to be compliant to the upcoming new syslog RFC series. </li> +<li><a href="rsconf1_maxopenfiles.html">$MaxOpenFiles</a></li> <li><a href="rsconf1_moddir.html">$ModDir</a></li> <li><a href="rsconf1_modload.html">$ModLoad</a></li> <li><b>$RepeatedMsgContainsOriginalMsg</b> [on/<b>off</b>] - "last message repeated n times" messages, if generated, @@ -214,7 +215,6 @@ the value, the less precise the timestamp. <li><a href="droppriv.html">$PrivDropToGroupID</a></li> <li><a href="droppriv.html">$PrivDropToUser</a></li> <li><a href="droppriv.html">$PrivDropToUserID</a></li> -</ul> <li><a href="rsconf1_umask.html">$UMASK</a></li> </ul> <p><b>Where <size_nbr> is specified above,</b> @@ -235,7 +235,7 @@ point of view, "1,,0.0.,.,0" also has the value 1000. </p> <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL -version 2 or higher.</font></p> +version 3 or higher.</font></p> </body> </html> diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html index a281d9e7..df9abeea 100644 --- a/doc/rsyslog_conf_modules.html +++ b/doc/rsyslog_conf_modules.html @@ -19,6 +19,7 @@ generic database output module (Firebird/Interbase, MS SQL, Sybase, SQLLite, Ingres, Oracle, mSQL)</li> <li><a href="ommail.html">ommail</a> - permits rsyslog to alert folks by mail if something important happens</li> +<li><a href="omoracle.html">omoracle</a> - output module for Oracle (native OCI interface)</li> <li><a href="imfile.html">imfile</a> - input module for text files</li> <li><a href="imrelp.html">imrelp</a> - RELP @@ -44,9 +45,9 @@ only available if it has been loaded (using $ModLoad).</p> [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL -version 2 or higher.</font></p> +version 3 or higher.</font></p> </body> </html> diff --git a/doc/src/dataflow.dia b/doc/src/dataflow.dia Binary files differnew file mode 100644 index 00000000..3875fc61 --- /dev/null +++ b/doc/src/dataflow.dia diff --git a/doc/src/direct_queue0.dia b/doc/src/direct_queue0.dia Binary files differnew file mode 100644 index 00000000..4446619b --- /dev/null +++ b/doc/src/direct_queue0.dia diff --git a/doc/src/direct_queue1.dia b/doc/src/direct_queue1.dia Binary files differnew file mode 100644 index 00000000..7a64ea09 --- /dev/null +++ b/doc/src/direct_queue1.dia diff --git a/doc/src/direct_queue2.dia b/doc/src/direct_queue2.dia Binary files differnew file mode 100644 index 00000000..b0c394c0 --- /dev/null +++ b/doc/src/direct_queue2.dia diff --git a/doc/src/direct_queue3.dia b/doc/src/direct_queue3.dia Binary files differnew file mode 100644 index 00000000..bc477b25 --- /dev/null +++ b/doc/src/direct_queue3.dia diff --git a/doc/src/direct_queue_directq.dia b/doc/src/direct_queue_directq.dia Binary files differnew file mode 100644 index 00000000..37fdb44c --- /dev/null +++ b/doc/src/direct_queue_directq.dia diff --git a/doc/src/direct_queue_rsyslog.dia b/doc/src/direct_queue_rsyslog.dia Binary files differnew file mode 100644 index 00000000..9a030117 --- /dev/null +++ b/doc/src/direct_queue_rsyslog.dia diff --git a/doc/src/direct_queue_rsyslog2.dia b/doc/src/direct_queue_rsyslog2.dia Binary files differnew file mode 100644 index 00000000..c596f39f --- /dev/null +++ b/doc/src/direct_queue_rsyslog2.dia diff --git a/doc/src/queue_analogy_tv.dia b/doc/src/queue_analogy_tv.dia Binary files differnew file mode 100644 index 00000000..00fbdeb5 --- /dev/null +++ b/doc/src/queue_analogy_tv.dia diff --git a/doc/status.html b/doc/status.html index dae94884..01a9cf8f 100644 --- a/doc/status.html +++ b/doc/status.html @@ -2,22 +2,22 @@ <html><head><title>rsyslog status page</title></head> <body> <h2>rsyslog status page</h2> -<p>This page reflects the status as of 2009-04-03.</p> +<p>This page reflects the status as of 2009-04-17.</p> <h2>Current Releases</h2> -<p><b>development:</b> 4.1.5 [2009-03-11] - -<a href="http://www.rsyslog.com/Article349.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-150.phtml">download</a> +<p><b>development:</b> 4.3.0 [2009-04-17] - +<a href="http://www.rsyslog.com/Article366.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-156.phtml">download</a> <br><b>beta:</b> 3.21.11 [2009-04-03] - <a href="http://www.rsyslog.com/Article358.phtml">change log</a> - <a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-152.phtml">download</a></p> -<p><b>v3 stable:</b> 3.20.3 [2009-04-02] - <a href="http://www.rsyslog.com/Article356.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-151.phtml">download</a> +<p><b>v3 stable:</b> 3.20.6 [2009-04-16] - <a href="http://www.rsyslog.com/Article364.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-155.phtml">download</a> -<br><b>v2 stable:</b> 2.0.6 [2008-08-07] - <a href="http://www.rsyslog.com/Article266.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-125.phtml">download</a> +<br><b>v2 stable:</b> 2.0.7 [2009-04-14] - <a href="http://www.rsyslog.com/Article362.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-154.phtml">download</a> <br>v0 and v1 are deprecated and no longer supported. If you absolutely do not like to upgrade, you may consider purchasing a <a href="professional_support.html">commercial rsyslog support package</a>. Just let us point |
