diff options
| author | Rainer Gerhards <rgerhards@adiscon.com> | 2008-02-21 09:41:56 +0000 |
|---|---|---|
| committer | Rainer Gerhards <rgerhards@adiscon.com> | 2008-02-21 09:41:56 +0000 |
| commit | 227c44dbe0f3ce8ab465829c2d6114d5e6b38470 (patch) | |
| tree | a15b84c5f20af5ac42687424e71fd20c5e61ad14 | |
| parent | 4ff521048dbab0557c863c28321693e1f31c6a96 (diff) | |
| download | rsyslog-3-11-4.tar.gz rsyslog-3-11-4.tar.xz rsyslog-3-11-4.zip | |
cleanup for 3.11.4v3-11-4
| -rw-r--r-- | ChangeLog | 3 | ||||
| -rw-r--r-- | configure.ac | 3 | ||||
| -rw-r--r-- | doc/rsyslog_mysql.html | 442 | ||||
| -rw-r--r-- | doc/status.html | 12 | ||||
| -rw-r--r-- | doc/v3compatibility.html | 196 | ||||
| -rw-r--r-- | obj.c | 6 | ||||
| -rw-r--r-- | var.c | 2 | ||||
| -rw-r--r-- | vmop.c | 2 | ||||
| -rw-r--r-- | vmprg.c | 2 |
9 files changed, 349 insertions, 319 deletions
@@ -1,5 +1,5 @@ --------------------------------------------------------------------------- -Version 3.12.0 (rgerhards), 2008-02-?? +Version 3.11.4 (rgerhards), 2008-02-21 - bugfix: debug.html was missing from release tarball - thanks to Michael Biebl for bringing this to my attention - some internal cleanup on the stringbuf object calling interface @@ -8,6 +8,7 @@ Version 3.12.0 (rgerhards), 2008-02-?? (previously only integers) - bugfix: message object was not properly synchronized when the main queue had a single thread and non-direct action queues were used +- some documentation improvements --------------------------------------------------------------------------- Version 3.11.3 (rgerhards), 2008-02-18 - fixed a bug in imklog which lead to duplicate message content in diff --git a/configure.ac b/configure.ac index fcccde10..e427232d 100644 --- a/configure.ac +++ b/configure.ac @@ -2,7 +2,8 @@ # Process this file with autoconf to produce a configure script. AC_PREREQ(2.61) -AC_INIT([rsyslog],[3.12.0],[rsyslog@lists.adiscon.com.]) +#AC_INIT([rsyslog],[3.12.0],[rsyslog@lists.adiscon.com.]) +AC_INIT([rsyslog],[3.11.4],[rsyslog@lists.adiscon.com.]) AM_INIT_AUTOMAKE AC_CONFIG_SRCDIR([syslogd.c]) AC_CONFIG_HEADERS([config.h]) diff --git a/doc/rsyslog_mysql.html b/doc/rsyslog_mysql.html index 53ee30cc..0ba6191e 100644 --- a/doc/rsyslog_mysql.html +++ b/doc/rsyslog_mysql.html @@ -1,249 +1,253 @@ -<html><head> -<title>Writing syslog Data to MySQL</title> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Writing syslog Data to MySQL</title> + <meta name="KEYWORDS" content="syslog, mysql, syslog to mysql, howto"> -</head> -<body> +</head><body> <h1>Writing syslog messages to MySQL</h1> - <P><small><i>Written by - <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer - Gerhards</a> (2005-08-02)</i></small></P> +<p><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> (2005-08-02)</i></small></p> <h2>Abstract</h2> <p><i><b>In this paper, I describe how to write <a href="http://www.monitorware.com/en/topics/syslog/">syslog</a> -messages to a <a href="http://www.mysql.com">MySQL</a> database.</b> Having -syslog messages in a database is often handy, especially when you intend to set -up a front-end for viewing them. This paper describes an approach -with <a href="http://www.rsyslog.com/">rsyslogd</a>, an alternative enhanced -syslog daemon natively supporting MySQL. I describe the components needed -to be installed and how to configure them.</i></p> +messages to a <a href="http://www.mysql.com">MySQL</a> +database.</b> Having syslog messages in a database is often +handy, especially when you intend to set up a front-end for viewing +them. This paper describes an approach with <a href="http://www.rsyslog.com/">rsyslogd</a>, an +alternative enhanced syslog daemon natively supporting MySQL. I +describe the components needed to be installed and how to configure +them.</i></p> <h2>Background</h2> -<p>In many cases, syslog data is simply written to text files. This approach has -some advantages, most notably it is very fast and efficient. However, data -stored in text files is not readily accessible for real-time viewing and analysis. -To do that, the messages need to be in a database. There are various -ways to store syslog messages in a database. For example, some have the syslogd -write text files which are later feed via a separate script into the database. -Others have written scripts taking the data (via a pipe) from a -non-database-aware syslogd and store them as they appear. Some others use -database-aware syslogds and make them write the data directly to the database. -In this paper, I use that "direct write" approach. I think it is superior, -because the syslogd itself knows the status of the database connection and thus -can handle it intelligently (well ... hopefully ;)). I use rsyslogd to acomplish -this, simply because I have initiated the rsyslog project with -database-awareness as one goal.</p> -<p><b>One word of caution:</b> while message storage in the database provides an -excellent foundation for interactive analysis, it comes at a cost. Database i/o -is considerably slower than text file i/o. As such, directly writing to -the database makes sense only if your message volume is low enough to allow a) -the syslogd, b) the network, and c) the database server to catch -up with it. Some time ago, I have written a paper on -<a href="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php"> -optimizing syslog server performance</a>. While this paper talks about -Window-based solutions, the ideas in it are generic enough to apply here, too. -So it might be worth reading if you anticipate medium high to high traffic. If you -anticipate really high traffic (or very large traffic spikes), you should -seriously consider forgetting about direct database writes - in my opinion, such -a situation needs either a very specialised system or a different approach (the -text-file-to-database approach might work better for you in this case). +<p>In many cases, syslog data is simply written to text files. +This approach has some advantages, most notably it is very fast and +efficient. However, data stored in text files is not readily accessible +for real-time viewing and analysis. To do that, the messages need to be +in a database. There are various ways to store syslog messages in a +database. For example, some have the syslogd write text files which are +later feed via a separate script into the database. Others have written +scripts taking the data (via a pipe) from a non-database-aware syslogd +and store them as they appear. Some others use database-aware syslogds +and make them write the data directly to the database. In this paper, I +use that "direct write" approach. I think it is superior, because the +syslogd itself knows the status of the database connection and thus can +handle it intelligently (well ... hopefully ;)). I use rsyslogd to +acomplish this, simply because I have initiated the rsyslog project +with database-awareness as one goal.</p> +<p><b>One word of caution:</b> while message storage +in the database provides an excellent foundation for interactive +analysis, it comes at a cost. Database i/o is considerably slower than +text file i/o. As such, directly writing to the database makes sense +only if your message volume is low enough to allow a) the syslogd, b) +the network, and c) the database server to catch up with it. Some time +ago, I have written a paper on +<a href="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php">optimizing +syslog server performance</a>. While this paper talks about +Window-based solutions, the ideas in it are generic enough to apply +here, too. So it might be worth reading if you anticipate medium high +to high traffic. If you anticipate really high traffic (or very large +traffic spikes), you should seriously consider forgetting about direct +database writes - in my opinion, such a situation needs either a very +specialised system or a different approach (the text-file-to-database +approach might work better for you in this case). </p> <h2>Overall System Setup</h2> -<P>In this paper, I concentrate on the server side. If you are thinking about -interactive syslog message review, you probably want to centralize syslog. In -such a scenario, you have multiple machines (the so-called clients) send their -data to a central machine (called server in this context). While I expect such a -setup to be typical when you are interested in storing messages in the database, -I do not describe how to set it up. This is beyond the scope of this paper. If -you search a little, you will probably find many good descriptions on how to -centralize syslog. If you do that, it might be a good idea to do it securely, so -you might also be interested in my paper on <a href="rsyslog_stunnel.html"> -ssl-encrypting syslog message transfer</a>.</P> -<P>No matter how the messages arrive at the server, their processing is always the -same. So you can use this paper in combination with any description for -centralized syslog reporting.</P> -<P>As I already said, I use rsyslogd on the server. It has intrinsic support for -talking to MySQL databases. For obvious reasons, we also need an instance of -MySQL running. To keep us focussed, the setup of MySQL itself is also beyond the -scope of this paper. I assume that you have successfully installed MySQL and -also have a front-end at hand to work with it (for example, -<a href="http://www.phpmyadmin.net/">phpMyAdmin</a>). Please make sure that this -is installed, actually working and you have a basic understanding of how to -handle it.</P> +<p>In this paper, I concentrate on the server side. If you are +thinking about interactive syslog message review, you probably want to +centralize syslog. In such a scenario, you have multiple machines (the +so-called clients) send their data to a central machine (called server +in this context). While I expect such a setup to be typical when you +are interested in storing messages in the database, I do not describe +how to set it up. This is beyond the scope of this paper. If you search +a little, you will probably find many good descriptions on how to +centralize syslog. If you do that, it might be a good idea to do it +securely, so you might also be interested in my paper on <a href="rsyslog_stunnel.html"> +ssl-encrypting syslog message transfer</a>.</p> +<p>No matter how the messages arrive at the server, their +processing is always the same. So you can use this paper in combination +with any description for centralized syslog reporting.</p> +<p>As I already said, I use rsyslogd on the server. It has +intrinsic support for talking to MySQL databases. For obvious reasons, +we also need an instance of MySQL running. To keep us focussed, the +setup of MySQL itself is also beyond the scope of this paper. I assume +that you have successfully installed MySQL and also have a front-end at +hand to work with it (for example, +<a href="http://www.phpmyadmin.net/">phpMyAdmin</a>). +Please make sure that this is installed, actually working and you have +a basic understanding of how to handle it.</p> <h2>Setting up the system</h2> -<p>You need to download and install rsyslogd first. Obtain it from the -<a href="http://www.rsyslog.com/">rsyslog site</a>. Make sure that you disable -stock syslogd, otherwise you will experience some difficulties.</p> -<p>It is important to understand how rsyslogd talks to the database. In rsyslogd, -there is the concept of "templates". Basically, a template is a string that -includes some replacement characters, which are called "properties" in rsyslog. -Properties are accessed via the "<a href="property_replacer.html">Property -Replacer</a>". Simply said, you access properties by including their name -between percent signs inside the template. For example, if the syslog message is -"Test", the template "%msg%" would be expanded to "Test". Rsyslogd supports -sending template text as a SQL statement to MySQL. As such, the template must be -a valid SQL statement. There is no limit in what the statement might be, but -there are some obvious and not so obvious choices. For example, a template "drop -table xxx" is possible, but does not make an awful lot of sense. In practice, -you will always use an "insert" statment inside the template.</p> -<p>An example: if you would just like to store the msg part of the full syslog -message, you have probably created a table "syslog" with a single column "message". -In such a case, a good template would be "insert into syslog(message) values ('%msg%')". -With the example above, that would be expanded to "insert into syslog(message) -values('Test')". This expanded string is then sent to the database. It's that -easy, no special magic. The only thing you must ensure is that your template -expands to a proper SQL statement and that this statement matches your database -design.</p> -<p>Does that mean you need to create database schema yourself and also must -fully understand rsyslogd's properties? No, that's not needed. Because we -anticipated that folks are probably more interested in getting things going instead -of designing them from scratch. So we have provided a default schema as well -as build-in support for it. This schema also offers an additional -benefit: rsyslog is part of <a href="http://www.adiscon.com/en/">Adiscon</a>'s -<a href="http://www.monitorware.com/en/">MonitorWare product line</a> (which -includes open source and closed source members). All of these tools share the -same default schema and know how to operate on it. For this reason, the default -schema is also called the "MonitorWare Schema". If you use it, you can simply -add <a href="http://www.phplogcon.org/">phpLogCon, a GPLed syslog web interface</a>, -to your system and have instant interactive access to your database. So there -are some benefits in using the provided schema.</p> -<p>The schema definition is contained in the file "createDB.sql". It comes with -the rsyslog package. Review it to check that the database name is acceptable for -you. Be sure to leave the table and field names unmodified, because -otherwise you need to customize rsyslogd's default sql template, which we do not -do -in this paper. Then, run the script with your favourite MySQL tool. Double-check -that the table was successfully created.</p> - -<p>MySQL support in rsyslog is integrated via a loadable plug-in module. To use the database -functionality, MySQL must be enabled in the config file BEFORE the first database table action is +<p>You need to download and install rsyslogd first. Obtain it +from the +<a href="http://www.rsyslog.com/">rsyslog site</a>. +Make sure that you disable stock syslogd, otherwise you will experience +some difficulties.</p> +<p>It is important to understand how rsyslogd talks to the +database. In rsyslogd, there is the concept of "templates". Basically, +a template is a string that includes some replacement characters, which +are called "properties" in rsyslog. Properties are accessed via the "<a href="property_replacer.html">Property Replacer</a>". +Simply said, you access properties by including their name between +percent signs inside the template. For example, if the syslog message +is "Test", the template "%msg%" would be expanded to "Test". Rsyslogd +supports sending template text as a SQL statement to MySQL. As such, +the template must be a valid SQL statement. There is no limit in what +the statement might be, but there are some obvious and not so obvious +choices. For example, a template "drop table xxx" is possible, but does +not make an awful lot of sense. In practice, you will always use an +"insert" statment inside the template.</p> +<p>An example: if you would just like to store the msg part of +the full syslog message, you have probably created a table "syslog" +with a single column "message". In such a case, a good template would +be "insert into syslog(message) values ('%msg%')". With the example +above, that would be expanded to "insert into syslog(message) +values('Test')". This expanded string is then sent to the database. +It's that easy, no special magic. The only thing you must ensure is +that your template expands to a proper SQL statement and that this +statement matches your database design.</p> +<p>Does that mean you need to create database schema yourself and +also must fully understand rsyslogd's properties? No, that's not +needed. Because we anticipated that folks are probably more interested +in getting things going instead of designing them from scratch. So we +have provided a default schema as well as build-in support for it. This +schema also offers an additional benefit: rsyslog is part of <a href="http://www.adiscon.com/en/">Adiscon</a>'s +<a href="http://www.monitorware.com/en/">MonitorWare +product line</a> (which includes open source and closed source +members). All of these tools share the same default schema and know how +to operate on it. For this reason, the default schema is also called +the "MonitorWare Schema". If you use it, you can simply add <a href="http://www.phplogcon.org/">phpLogCon, a GPLed syslog +web interface</a>, to your system and have instant interactive +access to your database. So there are some benefits in using the +provided schema.</p> +<p>The schema definition is contained in the file "createDB.sql". +It comes with the rsyslog package. Review it to check that the database +name is acceptable for you. Be sure to leave the table and field names +unmodified, because otherwise you need to customize rsyslogd's default +sql template, which we do not do in this paper. Then, run the script +with your favourite MySQL tool. Double-check that the table was +successfully created.</p> +<p>MySQL support in rsyslog is integrated via a loadable plug-in +module. To use the database +functionality, MySQL must be enabled in the config file BEFORE the +first database table action is used. This is done by placing the</p> <blockquote> - <p><code>$ModLoad MySQL</code></p> +<p><code>$ModLoad MySQL</code></p> </blockquote> <p>directive at the begining of /etc/rsyslog.conf</p> - -<p>Next, we need to tell rsyslogd to write data to the database. As we use -the default schema, we do NOT need to define a template for this. We can use the -hardcoded one (rsyslogd handles the proper template linking). So all we need to -do is add a simple selector line to /etc/rsyslog.conf:</p> +<p>Next, we need to tell rsyslogd to write data to the database. +As we use the default schema, we do NOT need to define a template for +this. We can use the hardcoded one (rsyslogd handles the proper +template linking). So all we need to do is add a simple selector line +to /etc/rsyslog.conf:</p> <blockquote> - <p><code>*.* - >database-server,database-name,database-userid,database-password</code></p> +<p><code>*.* +>database-server,database-name,database-userid,database-password</code></p> </blockquote> -<p>In many cases, MySQL will run on the local machine. In this case, you can -simply use "127.0.0.1" for <i>database-server</i>. This can be especially -advisable, if you do not need to expose MySQL to any process outside of the -local machine. In this case, you can simply bind it to 127.0.0.1, which provides -a quite secure setup. Of course, also supports remote MySQL instances. In that -case, use the remote server name (e.g. mysql.example.com) or IP-address. The <i> -database-name</i> by default is "syslog". If you have modified the default, use -your name here. <i>Database-userid</i> and <i>-password</i> are the credentials -used to connect to the database. As they are stored in clear text in -rsyslog.conf, that user should have only the least possible privileges. It is -sufficient to grant it INSERT privileges to the systemevents table, only. As a -side note, it is strongly advisable to make the rsyslog.conf file readable by -root only - if you make it world-readable, everybody could obtain the password -(and eventually other vital information from it). In our example, let's assume -you have created a MySQL user named "syslogwriter" with a password of -"topsecret" (just to say it bluntly: such a password is NOT a good idea...). If -your MySQL database is on the local machine, your rsyslog.conf line might look -like in this sample:</p> +<p>In many cases, MySQL will run on the local machine. In this +case, you can simply use "127.0.0.1" for <i>database-server</i>. +This can be especially advisable, if you do not need to expose MySQL to +any process outside of the local machine. In this case, you can simply +bind it to 127.0.0.1, which provides a quite secure setup. Of course, +also supports remote MySQL instances. In that case, use the remote +server name (e.g. mysql.example.com) or IP-address. The <i> +database-name</i> by default is "syslog". If you have modified +the default, use your name here. <i>Database-userid</i> +and <i>-password</i> are the credentials used to connect +to the database. As they are stored in clear text in rsyslog.conf, that +user should have only the least possible privileges. It is sufficient +to grant it INSERT privileges to the systemevents table, only. As a +side note, it is strongly advisable to make the rsyslog.conf file +readable by root only - if you make it world-readable, everybody could +obtain the password (and eventually other vital information from it). +In our example, let's assume you have created a MySQL user named +"syslogwriter" with a password of "topsecret" (just to say it bluntly: +such a password is NOT a good idea...). If your MySQL database is on +the local machine, your rsyslog.conf line might look like in this +sample:</p> <blockquote> - <p><code>*.* - >127.0.0.1,syslog,syslogwriter,topsecret</code></p> +<p><code>*.* +>127.0.0.1,syslog,syslogwriter,topsecret</code></p> </blockquote> -<p>Save rsyslog.conf, restart rsyslogd - and you should see syslog messages -being stored in the "systemevents" table!</p> -<p>The example line stores every message to the database. Especially if you have -a high traffic volume, you will probably limit the amount of messages being -logged. This is easy to acomplish: the "write database" action is just a regular -selector line. As such, you can apply normal selector-line filtering. If, for -example, you are only interested in messages from the mail subsystem, you can -use the following selector line:</p> +<p>Save rsyslog.conf, restart rsyslogd - and you should see +syslog messages being stored in the "systemevents" table!</p> +<p>The example line stores every message to the database. +Especially if you have a high traffic volume, you will probably limit +the amount of messages being logged. This is easy to acomplish: the +"write database" action is just a regular selector line. As such, you +can apply normal selector-line filtering. If, for example, you are only +interested in messages from the mail subsystem, you can use the +following selector line:</p> <blockquote> - <p><code>mail.* - >127.0.0.1,syslog,syslogwriter,topsecret</code></p> +<p><code>mail.* +>127.0.0.1,syslog,syslogwriter,topsecret</code></p> </blockquote> -<p>Review the <a href="rsyslog_conf.html">rsyslog.conf</a> documentation for -details on selector lines and their filtering.</p> -<p><b>You have now completed everything necessary to store syslog messages to -the MySQL database.</b> If you would like to try out a front-end, you might want -to look at <a href="http://www.phplogcon.org/">phpLogCon</a>, which displays -syslog data in a browser. As of this writing, phpLogCon is not yet a powerful -tool, but it's open source, so it might be a starting point for your own -solution.</p> +<p>Review the <a href="rsyslog_conf.html">rsyslog.conf</a> +documentation for details on selector lines and their filtering.</p> +<p><b>You have now completed everything necessary to store +syslog messages to the MySQL database.</b> If you would like to +try out a front-end, you might want to look at <a href="http://www.phplogcon.org/">phpLogCon</a>, which +displays syslog data in a browser. As of this writing, phpLogCon is not +yet a powerful tool, but it's open source, so it might be a starting +point for your own solution.</p> <h2>On Reliability...</h2> -<p><b>This section needs updating. You can now solve the issue with failover -database servers. Read the <a href="rsyslog_conf.html">rsyslog.conf </a>doc on -that</b>.</p> -<p>Rsyslogd writes syslog messages directly to the database. This implies that -the database must be available at the time of message arrival. If the database -is offline, no space is left or something else goes wrong - rsyslogd can not -write the database record. If rsyslogd is unable to store a message, it performs -one retry. This is helpful if the database server was restarted. In this case, -the previous connection was broken but a reconnect immediately succeeds. However, -if the database is down for an extended period of time, an immediate retry does -not help. While rsyslogd could retry until it finally succeeds, that would have -negative impact. Syslog messages keep coming in. If rsyslogd would be busy -retrying the database, it would not be able to process these messages. -Ultimately, this would lead to loss of newly arrived messages.</p> -<p>In most cases, rsyslogd is configured not only to write to the database but -to perform other actions as well. In the always-retry scenario, that would mean -no other actions would be carried out. As such, the design of rsyslogd is -limited to a single retry. If that does not succeed, the current message is will -not be written to the database and the MySQL database writer be suspended for a -short period of time. Obviously, this leads to the loss of the current message -as well as all messages received during the suspension period. But they are only -lost in regard to the database, all other actions are correctly carried out. -While not perfect, we consider this to be a better approach then the potential -loss of all messages in all actions.</p> -<p><b>In short: try to avoid database downtime if you do not want to experience -message loss.</b></p> -<p>Please note that this restriction is not rsyslogd specific. All approaches to -real-time database storage share this problem area.</p> +<p>Rsyslogd writes syslog messages directly to the database. This +implies that the database must be available at the time of message +arrival. If the database is offline, no space is left or something else +goes wrong - rsyslogd can not write the database record. If rsyslogd is +unable to store a message, it performs one retry. This is helpful if +the database server was restarted. In this case, the previous +connection was broken but a reconnect immediately succeeds. However, if +the database is down for an extended period of time, an immediate retry +does not help.</p> +<p>Message loss in this scenario can easily be prevented with +rsyslog. All you need to do is run the database writer in queued mode. +This is now described in a generic way and I do not intend to duplicate +it here. So please be sure to read "<a href="rsyslog_high_database_rate.html">Handling a massive +syslog database insert rate with Rsyslog</a>", which describes +the scenario and also includes configuration examples.</p> <h2>Conclusion</h2> -<P>With minimal effort, you can use rsyslogd to write syslog messages to a MySQL -database. Once the messages are arrived there, you can interactivley review and -analyse them. In practice, the messages are also stored in text files for -longer-term archival and the databases are cleared out after some time (to avoid -becoming too slow). If you expect an extremely high syslog message volume, -storing it in real-time to the database may outperform your database server. In -such cases, either filter out some messages or think about alternate approaches -involving non-real-time database writing (beyond the scope of this paper).</P> -<P>The method outlined in this paper provides an easy to setup and maintain -solution for most use cases, especially with low and medium syslog message -volume (or fast database servers).</P> +<p>With minimal effort, you can use rsyslogd to write syslog +messages to a MySQL database. You can even make it absolutely fail-safe +and protect it against database server downtime. Once the messages are +arrived there, you +can interactivley review and analyse them. In practice, the messages +are also stored in text files for longer-term archival and the +databases are cleared out after some time (to avoid becoming too slow). +If you expect an extremely high syslog message volume, storing it in +real-time to the database may outperform your database server. In such +cases, either filter out some messages or used queued mode (which in +general is recommended with databases).</p> +<p>The method outlined in this paper provides an easy to setup +and maintain solution for most use cases.</p> <h3>Feedback Requested</h3> -<P>I would appreciate feedback on this paper. If you have additional ideas, -comments or find bugs, please -<a href="mailto:rgerhards@adiscon.com">let me know</a>.</P> +<p>I would appreciate feedback on this paper. If you have +additional ideas, comments or find bugs, please +<a href="mailto:rgerhards@adiscon.com">let me know</a>.</p> <h2>References and Additional Material</h2> <ul> - <li><a href="http://www.rsyslog.com">www.rsyslog.com</a> - the rsyslog site</li> - <li> - <a href="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php"> - Paper on Syslog Server Optimization</a></li> +<li><a href="http://www.rsyslog.com">www.rsyslog.com</a> +- the rsyslog site</li> +<li> <a href="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php"> +Paper on Syslog Server Optimization</a></li> </ul> <h2>Revision History</h2> <ul> - <li>2005-08-02 * - <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> * - initial version created</li> - <li>2005-08-03 * - <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> - * added references to demo site</li> - <li>2007-06-13 * - <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> - * removed demo site - was torn down because too expensive for usage count</li> +<li>2005-08-02 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> * initial version created</li> +<li>2005-08-03 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> * added references to demo site</li> +<li>2007-06-13 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> * removed demo site - was torn down because too +expensive for usage count</li> +<li>2008-02-21 * <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> * updated reliability section, can now be done with +on-demand disk queues</li> </ul> <h2>Copyright</h2> -<p>Copyright (c) 2005-2007 -<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> -and <a href="http://www.adiscon.com/en/">Adiscon</a>.</p> -<p>Permission is granted to copy, distribute and/or modify this document under -the terms of the GNU Free Documentation License, Version 1.2 or any later -version published by the Free Software Foundation; with no Invariant Sections, -no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be -viewed at <a href="http://www.gnu.org/copyleft/fdl.html"> +<p>Copyright (c) 2005-2008 +<a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</a> and <a href="http://www.adiscon.com/en/">Adiscon</a>.</p> +<p>Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version +1.2 or any later version published by the Free Software Foundation; +with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +Texts. A copy of the license can be viewed at <a href="http://www.gnu.org/copyleft/fdl.html"> http://www.gnu.org/copyleft/fdl.html</a>.</p> -</body> -</html> +</body></html>
\ No newline at end of file diff --git a/doc/status.html b/doc/status.html index c620802e..5747d1a7 100644 --- a/doc/status.html +++ b/doc/status.html @@ -1,15 +1,13 @@ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head><title>rsyslog status page</title> - -</head> +<html><head><title>rsyslog status page</title></head> <body> <h2>rsyslog status page</h2> -<p>This page reflects the status as of 2008-02-18.</p> +<p>This page reflects the status as of 2008-02-21.</p> <h2>Current Releases</h2> -<p><b>development:</b> 3.11.3 - -<a href="http://www.rsyslog.com/Article177.phtml">change +<p><b>development:</b> 3.11.4 - +<a href="http://www.rsyslog.com/Article179.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-index-req-getit-lid-80.phtml">download</a></p> +<a href="http://www.rsyslog.com/Downloads-index-req-getit-lid-81.phtml">download</a></p> <p><b><font color="#ff0000"><a href="v3compatibility.html">If you used version 2, be sure to read the rsyslog v3 compatibility document!</a></font></b><br> Documentation for 3.x is currently partly sparse. If you need diff --git a/doc/v3compatibility.html b/doc/v3compatibility.html index 4a391f97..221d8302 100644 --- a/doc/v3compatibility.html +++ b/doc/v3compatibility.html @@ -1,119 +1,139 @@ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head> +<html><head><title>Compatibility notes for rsyslog v3</title> -<title>Writing syslog Data to MySQL</title><meta name="KEYWORDS" content="syslog, mysql, syslog to mysql, howto"></head><body> +<meta name="KEYWORDS" content="syslog, mysql, syslog to mysql, howto"> +</head><body> <h1>Compatibility notes for rsyslog v3</h1> - <p><small><i>Written by - <a href="http://www.gerhards.net/rainer">Rainer - Gerhards</a> (2007-12-17)</i></small></p> -<p>Rsyslog aims to be a drop-in replacement for sysklogd. However, version 3 has -some considerable enhancements, which lead to some backward compatibility issues -both in regard to sysklogd and rsyslog v1 and v2.</p> -<p>Rsyslog v3 is currently under initial development. Compatibility issues may -be resolved, so be sure to check back often. Also, comments and suggestions are -appreciated. Feedback right in time can now have a big impact on the route we -take ;)</p> +<p><small><i>Written by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> +(2007-12-17)</i></small></p> +<p>Rsyslog aims to be a drop-in replacement for sysklogd. +However, version 3 has some considerable enhancements, which lead to +some backward compatibility issues both in regard to sysklogd and +rsyslog v1 and v2.</p> +<p>Rsyslog v3 is currently under initial development. +Compatibility issues may be resolved, so be sure to check back often. +Also, comments and suggestions are appreciated. Feedback right in time +can now have a big impact on the route we take ;)</p> <h2>inputs</h2> -<p>With v2 and below, inputs were automatically started together with rsyslog. -In v3, inputs are optional! They come in the form of plug-in modules. -<font color="#ff0000"><b>At least one input module must be loaded to make -rsyslog do any useful work.</b></font> The config file directives doc briefly -lists which config statements are available by which modules.</p> -<p>It is suggested that input modules be loaded in the top part of the config -file. Here is an example, also highlighting the most important modules:</p> -<p><b>$ModLoad immark.so # provides --MARK-- message capability<br> +<p>With v2 and below, inputs were automatically started together +with rsyslog. In v3, inputs are optional! They come in the form of +plug-in modules. +<font color="#ff0000"><b>At least one input module +must be loaded to make rsyslog do any useful work.</b></font> +The config file directives doc briefly lists which config statements +are available by which modules.</p> +<p>It is suggested that input modules be loaded in the top part +of the config file. Here is an example, also highlighting the most +important modules:</p> +<p><b>$ModLoad immark.so # provides --MARK-- +message capability<br> $ModLoad imudp.so # provides UDP syslog reception<br> -$ModLoad imtcp.so # provides TCP syslog reception and GSS-API (if compiled -to support it)<br> -$ModLoad imuxsock.so # provides support for local system logging (e.g. via -logger command)<br> -$ModLoad imklog.so # provides kernel logging support (previously done by rklogd)</b></p> +$ModLoad imtcp.so # provides TCP syslog reception and GSS-API +(if compiled to support it)<br> +$ModLoad imuxsock.so # provides support for local system logging (e.g. +via logger command)<br> +$ModLoad imklog.so # provides kernel logging support (previously done +by rklogd)</b></p> <h2>command line options</h2> -<p>A number of command line options have been removed. New config file -directives have been added for them. Once we implement compatibiltiy mode, these -options will return, but only if running in non-v3-native mode.</p> +<p>A number of command line options have been removed. New config +file directives have been added for them. Once we implement +compatibiltiy mode, these options will return, but only if running in +non-v3-native mode.</p> <h2>-m command line option</h2> -<p>The -m command line option is ignored for the time being. There is no default -mark period. If you need a 20 minute mark period you need to</p> +<p>The -m command line option is ignored for the time being. +There is no default mark period. If you need a 20 minute mark period +you need to</p> <p><b>$ModLoad immark.so # wherever this is<br> $MarkMessageInterval 1800 # 30 minutes</b></p> <h2>-r command line option</h2> -<p>Is also no longer available. Use the <b>$UDPSeverRun <port></b> config file -directives. You can now also set the local address the server should listen to -via <b>$UDPServerAddress <ip></b> config directive.</p> -<p>The following example configures an UDP syslog server at the local address -192.0.2.1 on port 514:</p> +<p>Is also no longer available. Use the <b>$UDPSeverRun +<port></b> config file directives. You can now also +set the local address the server should listen to via <b>$UDPServerAddress +<ip></b> config directive.</p> +<p>The following example configures an UDP syslog server at the +local address 192.0.2.1 on port 514:</p> <p><b>$ModLoad imudp.so<br> -$UDPSeverAddress 192.0.2.1 # this MUST be before the $UDPServerRun directive!<br> +$UDPSeverAddress 192.0.2.1 # this MUST be before the $UDPServerRun +directive!<br> $UDPServerRun 514</b></p> -<p>"$UDPServerAddress *" means listen on all local interfaces. This is the -default if no directive is specified.</p> -<p>Please note that now multiple listeners are supported. For example, you can -do the following:</p> +<p>"$UDPServerAddress *" means listen on all local interfaces. +This is the default if no directive is specified.</p> +<p>Please note that now multiple listeners are supported. For +example, you can do the following:</p> <p><b>$ModLoad imudp.so<br> -$UDPSeverAddress 192.0.2.1 # this MUST be before the $UDPServerRun directive!<br> +$UDPSeverAddress 192.0.2.1 # this MUST be before the $UDPServerRun +directive!<br> $UDPServerRun 514<br> $UDPSeverAddress * # all local interfaces<br> $UDPServerRun 1514</b></p> -<p>These config file settings run two listeners: one at192.0.2.1:514 and one on -port 1514, which listens on all local interfaces.</p> +<p>These config file settings run two listeners: one +at192.0.2.1:514 and one on port 1514, which listens on all local +interfaces.</p> <h2>Default port for UDP (and TCP) Servers</h2> -<p>Please note that with pre-v3 rsyslogd, a service database lookup was made -when a UDP server was started and no port was configured. Only if that failed, -the IANA default of 514 was used. For TCP serves, this lookup was never done and -514 always used if no specific port was configured. For consitency, both TCP and -UDP now use port 514 as default. If a lookup is desired, you need to specify it -in the "Run" directive, e.g. "<i>$UDPServerRun syslog</i>".</p> +<p>Please note that with pre-v3 rsyslogd, a service database +lookup was made when a UDP server was started and no port was +configured. Only if that failed, the IANA default of 514 was used. For +TCP serves, this lookup was never done and 514 always used if no +specific port was configured. For consitency, both TCP and UDP now use +port 514 as default. If a lookup is desired, you need to specify it in +the "Run" directive, e.g. "<i>$UDPServerRun syslog</i>".</p> <h2>klogd</h2> -<p>klogd has (finally) been replaced by a loadable input module. To enable klogd -functionality, do</p> +<p>klogd has (finally) been replaced by a loadable input module. +To enable klogd functionality, do</p> <p><b>$ModLoad imklog.so</b></p> -<p>A limited set of klogd command line settings is now supported via -rsyslog.conf. That set of configuration directives is to be expanded. So far, we -support:</p> +<p>A limited set of klogd command line settings is now supported +via rsyslog.conf. That set of configuration directives is to be +expanded. So far, we support:</p> <p>$klogSymbolsTwice [on/off]<br> $DebugPrintKernelSymbols [on/off] # spits *a lot* of messages at startup</p> <h2>Queue Modes for the Main Message Queue</h2> -<p>Either "FixedArray" or "LinkedList" is recommended. "Direct" is available, -but should not be used except for a very good reason ("Direct" disables queueing -and will potentially lead to message loss on the input side).</p> +<p>Either "FixedArray" or "LinkedList" is recommended. "Direct" +is available, but should not be used except for a very good reason +("Direct" disables queueing and will potentially lead to message loss +on the input side).</p> <h1>TODO List</h1> -<p>I have decided to move the todo list for v3 also into this document. After -all, it is kind of a compatibility issue ;) But, more seriously, I think this is -a good place to make sure nothing is forgotten. Remember that as of now, rsyslog -v3 is in its early stage of development and so there definitely are some nits -that will give you grief. This section here tells you what it is.</p> +<p>I have decided to move the todo list for v3 also into this +document. After all, it is kind of a compatibility issue ;) But, more +seriously, I think this is a good place to make sure nothing is +forgotten. Remember that as of now, rsyslog v3 is in its early stage of +development and so there definitely are some nits that will give you +grief. This section here tells you what it is.</p> <h2>-c option</h2> -<p>needs to be implemented. So far, only -c3 is supported. Everything else does -not change anything but just provides a warning message. In short: there is no -backwards compatibility yet.</p> +<p>needs to be implemented. So far, only -c3 is supported. +Everything else does not change anything but just provides a warning +message. In short: there is no backwards compatibility yet.</p> <h2>Thread Termination</h2> -<p>Thread termination in mode 1 needs to be looked at. I begin to think that it -is OK if we simply cancel the thread, because we already have a different -cleanup function. Otherwise, there is a potential for a race condition after we -unblocked the TermOK mutex but before select() is entered. If the signal then -occurs, we may have a problem. However, I would like to create the other input -modules first so that I have more working experience with termination -requirements. Again, I think this will lead to changes in thread termination.</p> -<p>So far, this issue is resolved, but there is still some code present that -needs to go away (right now disabled). I keep it in because I will make the -final decision based on experience gained during creation of input modules.</p> +<p>Thread termination in mode 1 needs to be looked at. I begin to +think that it is OK if we simply cancel the thread, because we already +have a different cleanup function. Otherwise, there is a potential for +a race condition after we unblocked the TermOK mutex but before +select() is entered. If the signal then occurs, we may have a problem. +However, I would like to create the other input modules first so that I +have more working experience with termination requirements. Again, I +think this will lead to changes in thread termination.</p> +<p>So far, this issue is resolved, but there is still some code +present that needs to go away (right now disabled). I keep it in +because I will make the final decision based on experience gained +during creation of input modules.</p> <h2>Call Encapsulation</h2> -<p>In an ideal world, the modules should call back only to some well-defined and -specifically exported interface functions. Right now, we do not have an ideal -world and the input modules make some calls back into the core. This needs to be -re-thought but is accepted for the time being.</p> +<p>In an ideal world, the modules should call back only to some +well-defined and specifically exported interface functions. Right now, +we do not have an ideal world and the input modules make some calls +back into the core. This needs to be re-thought but is accepted for the +time being.</p> <h2>No real Modules yet</h2> -<p>As already said in "Call Encapsulation", the modules look like such, but are -not really able to work stand alone. The worst in this regard are</p> +<p>As already said in "Call Encapsulation", the modules look like +such, but are not really able to work stand alone. The worst in this +regard are</p> <ul> - <li>imudp/omfwd/syslogd.c</li> - <li>imtcp</li> +<li>imudp/omfwd/syslogd.c</li> +<li>imtcp</li> </ul> -<p>But the others are not necessarily better. I have moved the code to the -modules, because now at least the majority of code is where it belongs to. But -there are still a large number of global variables, cross-function calls and the -like. This needs to be solved when we try to claim a real modular design. That's -not an easy task, but, hey, aren't we up for that...? ;)</p> +<p>But the others are not necessarily better. I have moved the +code to the modules, because now at least the majority of code is where +it belongs to. But there are still a large number of global variables, +cross-function calls and the like. This needs to be solved when we try +to claim a real modular design. That's not an easy task, but, hey, +aren't we up for that...? ;)</p> <p> </p> </body></html>
\ No newline at end of file @@ -278,6 +278,9 @@ rsRetVal objSerializeProp(strm_t *pStrm, uchar *pszPropName, varType_t propType, ABORT_FINALIZE(RS_RET_PROVIDED_BUFFER_TOO_SMALL); pszBuf = szBuf; break; + default: + dbgprintf("invalid VARTYPE %d\n", propType); + break; } /* cookie */ @@ -519,6 +522,9 @@ static rsRetVal objDeserializeProperty(var_t *pProp, strm_t *pStrm) case VARTYPE_SYSLOGTIME: CHKiRet(objDeserializeSyslogTime(&pProp->val.vSyslogTime, pStrm)); break; + default: + dbgprintf("invalid VARTYPE %d\n", pProp->varType); + break; } /* we should now be at the end of the line. So the next char must be \n */ @@ -48,7 +48,7 @@ ENDobjConstruct(var) /* ConstructionFinalizer * rgerhards, 2008-01-09 */ -rsRetVal varConstructFinalize(var_t *pThis) +rsRetVal varConstructFinalize(var_t __attribute__((unused)) *pThis) { DEFiRet; @@ -44,7 +44,7 @@ ENDobjConstruct(vmop) /* ConstructionFinalizer * rgerhards, 2008-01-09 */ -rsRetVal vmopConstructFinalize(vmop_t *pThis) +rsRetVal vmopConstructFinalize(vmop_t __attribute__((unused)) *pThis) { DEFiRet; ISOBJ_TYPE_assert(pThis, vmop); @@ -43,7 +43,7 @@ ENDobjConstruct(vmprg) /* ConstructionFinalizer * rgerhards, 2008-01-09 */ -rsRetVal vmprgConstructFinalize(vmprg_t *pThis) +rsRetVal vmprgConstructFinalize(vmprg_t __attribute__((unused)) *pThis) { DEFiRet; ISOBJ_TYPE_assert(pThis, vmprg); |