diff options
251 files changed, 11563 insertions, 4086 deletions
@@ -1,11 +1,388 @@ --------------------------------------------------------------------------- -Version 4.4.2a [private build] (rgerhards), 2010-10-15 +Version 4.6.6 [v4-stable] (rgerhards), 2010-11-?? - bugfix: a couple of problems that imfile had on some platforms, namely Ubuntu (not their fault, but occured there) - bugfix: imfile utilizes 32 bit to track offset. Most importantly, this problem can not experienced on Fedora 64 bit OS (which has 64 bit long's!) --------------------------------------------------------------------------- +Version 4.6.5 [v4-stable] (rgerhards), 2010-11-24 +- bugfix(important): problem in TLS handling could cause rsyslog to loop + in a tight loop, effectively disabling functionality and bearing the + risk of unresponsiveness of the whole system. + Bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=194 +--------------------------------------------------------------------------- +Version 4.6.4 [v4-stable] (rgerhards), 2010-08-05 +- bugfix: zero-sized (empty) messages were processed by imtcp + they are now dropped as they always should have been +- bugfix: programname filter in ! configuration can not be reset + Thanks to Kiss Gabor for the patch. +--------------------------------------------------------------------------- +Version 4.6.3 [v4-stable] (rgerhards), 2010-07-07 +- improvded testbench + - added test with truly random data received via syslog to test + robustness + - added new configure option that permits to disable and enable an + extended testbench +- bugfix: segfault on HUP when "HUPIsRestart" was set to "on" + thanks varmojfekoj for the patch +- bugfix: default for $OMFileFlushOnTXEnd was wrong ("off"). + This, in default mode, caused buffered writing to be used, what + means that it looked like no output were written or partial + lines. Thanks to Michael Biebl for pointing out this bug. +- bugfix: testbench failed when not executed in UTC+1 timezone + accidently, the time zone information was kept inside some + to-be-checked-for responses +- temporary bugfix replaced by permanent one for + message-induced off-by-one error (potential segfault) (see 4.6.2) + The analysis has been completed and a better fix been crafted and + integrated. +- some doc fixes; incorrect config samples could cause confusion + thanks to Anthony Edwards for pointing the problems out +--------------------------------------------------------------------------- +Version 4.6.2 [v4-stable] (rgerhards), 2010-03-26 +- new feature: "." action type added to support writing files to relative + pathes (this is primarily meant as a debug aid) +- new feature: $OMFileAsyncWriting directive added + it permits to specifiy if asynchronous writing should be done or not +- bugfix(temporary): message-induced off-by-one error (potential segfault) + Some types of malformed messages could trigger an off-by-one error + (for example, \0 or \n as the last character, and generally control + character escaption is questionable). This is due to not strictly + following a the \0 or string counted string paradigm (during the last + optimization on the cstring class). As a temporary fix, we have + introduced a proper recalculation of the size. However, a final + patch is expected in the future. See bug tracker for further details + and when the final patch will be available: + http://bugzilla.adiscon.com/show_bug.cgi?id=184 + Note that the current patch is considered sufficient to solve the + situation, but it requires a bit more runtime than desirable. +- bugfix: potential segfault in dynafile cache + This bug was triggered by an open failure. The the cache was full and + a new entry needed to be placed inside it, a victim for eviction was + selected. That victim was freed, then the open of the new file tried. If + the open failed, the victim entry was still freed, and the function + exited. However, on next invocation and cache search, the victim entry + was used as if it were populated, most probably resulting in a segfault. +- bugfix: race condition during directory creation + If multiple files try to create a directory at (almost) the same time, + some of them may fail. This is a data race and also exists with other + processes that may create the same directory. We do now check for this + condition and gracefully handle it. +- bugfix: potential re-use of free()ed file stream object in omfile + when dynaCache is enabled, the cache is full, a new entry needs to + be allocated, thus the LRU discarded, then a new entry is opend and that + fails. In that case, it looks like the discarded stream may be reused + improperly (based on code analysis, test case and confirmation pending) +- added new property replacer option "date-rfc3164-buggyday" primarily + to ease migration from syslog-ng. See property replacer doc for + details. [backport from 5.5.3 because urgently needed by some] +- improved testbench +- bugfix: invalid buffer write in (file) stream class + currently being accessed buffer could be overwritten with new data. + While this probably did not cause access violations, it could case loss + and/or duplication of some data (definitely a race with no deterministic + outcome) +- bugfix: potential hang condition during filestream close + predicate was not properly checked when waiting for the background file + writer +- bugfix: improper synchronization when "$OMFileFlushOnTXEnd on" was used + Internal data structures were not properly protected due to missing + mutex calls. +- bugfix: potential data loss during file stream shutdown +- bugfix: potential problems during file stream shutdown + The shutdown/close sequence was not clean, what potentially (but + unlikely) could lead to some issues. We have not been able to describe + any fatal cases, but there was some bug potential. Sequence has now + been straighted out. +- bugfix: potential problem (loop, abort) when file write error occured + When a write error occured in stream.c, variable iWritten had the error + code but this was handled as if it were the actual number of bytes + written. That was used in pointer arithmetic later on, and thus could + lead to all sorts of problems. However, this could only happen if the + error was EINTR or the file in question was a tty. All other cases were + handled properly. Now, iWritten is reset to zero in such cases, resulting + in proper retries. +- bugfix: $omfileFlushOnTXEnd was turned on when set to off and vice + versa due to an invalid check +- bugfix: recent patch to fix small memory leak could cause invalid free. + This could only happen during config file parsing. +- bugfix(minor): handling of extremely large strings in dbgprintf() fixed + Previously, it could lead to garbagge output and, in extreme cases, also + to segfaults. Note: this was a problem only when debug output was + actually enabled, so it caused no problem in production use. +- bugfix(minor): BSD_SO_COMPAT query function had some global vars not + properly initialized. However, in practice the loader initializes them + with zero, the desired value, so there were no actual issue in almost + all cases. +--------------------------------------------------------------------------- +Version 4.6.1 [v4-stable] (rgerhards), 2010-03-04 +- re-enabled old pipe output (using new module ompipe, built-in) after + some problems with pipes (and especially in regard to xconsole) were + discovered. Thanks to Michael Biebl for reporting the issues. +- bugfix: potential problems with large file support could cause segfault + ... and other weird problems. This seemed to affect 32bit-platforms + only, but I can not totally outrule there were issues on other + platforms as well. The previous code could cause system data types + to be defined inconsistently, and that could lead to various + troubles. Special thanks go to the Mandriva team for identifying + an initial problem, help discussing it and ultimately a fix they + contributed. +- bugfix: fixed problem that caused compilation on FreeBSD 9.0 to fail. + bugtracker: http://bugzilla.adiscon.com/show_bug.cgi?id=181 + Thanks to Christiano for reporting. +- bugfix: potential segfault in omfile when a dynafile open failed + In that case, a partial cache entry was written, and some internal + pointers (iCurrElt) not correctly updated. In the next iteration, that + could lead to a segfault, especially if iCurrElt then points to the + then-partial record. Not very likely, but could happen in practice. +- bugfix (theoretical): potential segfault in omfile under low memory + condition. This is only a theoretical bug, because it would only + happen when strdup() fails to allocate memory - which is highly + unlikely and will probably lead to all other sorts of errors. +- bugfix: comment char ('#') in literal terminated script parsing + and thus could not be used. + but tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=119 + [merged in from v3.22.2] +--------------------------------------------------------------------------- +Version 4.6.0 [v4-stable] (rgerhards), 2010-02-24 +*************************************************************************** +* This is a new stable v4 version. It contains all fixes and enhancements * +* made during the 4.5.x phase as well as those listed below. * +* Note: this version is scheduled to conclude the v4 development process. * +* Do not expect any more new developments in v4. The focus is now * +* on v5 (what also means we have a single devel branch again). * +* ("development" means new feature development, bug fixes are of * +* course provided for v4-stable) * +*************************************************************************** +- improved testbench to contain samples for totally malformed messages + which miss parts of the message content +- bugfix: some malformed messages could lead to a missing LF inside files + or some other missing parts of the template content. +- bugfix: if a message ended immediately with a hostname, the hostname + was mistakenly interpreted as TAG, and localhost be used as hostname +- bugfix: message without MSG part could case a segfault + [backported from v5 commit 98d1ed504ec001728955a5bcd7916f64cd85f39f] + This actually was a "recent" regression, but I did not realize that it + was introduced by the performance optimization in v4-devel. Shame on + me for having two devel versions at the same time... +--------------------------------------------------------------------------- +Version 4.5.8 [v4-beta] (rgerhards), 2010-02-10 +- enhanced doc for using PostgreSQL + Thanks to Marc Schiffbauer for the new/updated doc +- bugfix: property replacer returned invalid parameters under some (unusual) + conditions. In extreme cases, this could lead to garbled logs and/or + a system failure. +- bugfix: invalid length returned (often) when using regular expressions + inside the property replacer +- bugfix: submatch regex in property replacer did not honor "return 0 on + no match" config case +- bugfix: imuxsock incorrectly stated inputname "imudp" + Thanks to Ryan Lynch for reporting this. +- (slightly) enhanced support for FreeBSD by setting _PATH_MODDIR to + the correct value on FreeBSD. + Thanks to Cristiano for the patch. +- bugfix: -d did not enable display of debug messages + regression from introduction of "debug on demand" mode + Thanks to Michael Biebl for reporting this bug +- bugfix: blanks inside file names did not terminate file name parsing. + This could reslult in the whole rest of a line (including comments) + to be treated as file name in "write to file" actions. + Thanks to Jack for reporting this issue. +- bugfix: rsyslog hang when writing to a named pipe which nobody was + reading. Thanks to Michael Biebl for reporting this bug. +- bugfix: memory leak when sending messages in zip-compressed format + Thanks to Naoya Nakazawa for analyzing this issue and providing a patch. +- bugfix: potential segfaults during queue shutdown + (bugs require certain non-standard settings to appear) + Thanks to varmojfekoj for the patch +--------------------------------------------------------------------------- +Version 4.5.7 [v4-beta] (rgerhards), 2009-11-18 +- added a so-called "On Demand Debug" mode, in which debug output can + be generated only after the process has started, but not right from + the beginning. This is assumed to be useful for hard-to-find bugs. + Also improved the doc on the debug system. +- bugfix (kind of): check if TCP connection is still alive if using TLS + Thanks to Jonathan Bond-Caron for the patch. +- bugfix: hostname accidently set to IP address for some message sources, + for example imudp. Thanks to Anton for reporting this bug. +- bugfix [imported from 4.4.3]: $ActionExecOnlyOnceEveryInterval did + not work. +--------------------------------------------------------------------------- +Version 4.5.6 [v4-beta] (rgerhards), 2009-11-05 +- bugfix: named pipes did no longer work (they always got an open error) + this was a regression from the omfile rewrite in 4.5.0 +- bugfix(minor): diag function returned wrong queue memeber count + for the main queue if an active DA queue existed. This had no relevance + to real deployments (assuming they are not running the debug/diagnostic + module...), but sometimes caused grief and false alerts in the + testbench. +- included some important fixes from v4-stable: + * bugfix: invalid handling of zero-sized messages + * bugfix: zero-sized UDP messages are no longer processed + * bugfix: random data could be appended to message + * bugfix: reverse lookup reduction logic in imudp do DNS queries too often +- bugfix(testbench): testcase did not properly wait for rsyslod shutdown + thus some unpredictable behavior and a false negative test result + could occur. [BACKPORTED from v5] +- bugfix(testbench): sequence check was not always performed correctly, + that could result in tests reporting success when they actually failed +--------------------------------------------------------------------------- +Version 4.5.5 [v4-beta] (rgerhards), 2009-10-21 +- added $InputTCPServerNotifyOnConnectionClose config directive + see doc for details +- bugfix: debug string larger than 1K were improperly displayed. Max size + is now 32K +- bugfix: invalid storage class selected for some size config parameters. + This resulted in wrong values. The most prominent victim was the + directory creation mode, which was set to zero in some cases. For + details, see related blog post: + http://blog.gerhards.net/2009/10/another-note-on-hard-to-find-bugs.html +--------------------------------------------------------------------------- +Version 4.5.4 [v4-beta] (rgerhards), 2009-09-29 +- bugfix: potential segfault in stream writer on destruction + Most severely affected omfile. The problem was that some buffers were + freed before the asynchronous writer thread was shut down. So the + writer thread accessed invalid data, which may even already be + overwritten. Symptoms (with omfile) were segfaults, grabled data + and files with random names placed around the file system (most + prominently into the root directory). Special thanks to Aaron for + helping to track this down. +- bugfix: potential race in object loader (obj.c) during use/release + of object interface +- bugfixes: potential problems in out file zip writer. Problems could + lead to abort and/or memory leak. The module is now hardened in a very + conservative way, which is sub-optimal from a performance point of view. + This should be improved if it has proven reliable in practice. +--------------------------------------------------------------------------- +Version 4.5.3 [v4-beta] (rgerhards), 2009-09-17 +- bugfix: repeated messages were incorrectly processed + this could lead to loss of the repeated message content. As a side- + effect, it could probably also be possible that some segfault occurs + (quite unlikely). The root cause was that some counters introduced + during the malloc optimizations were not properly duplicated in + MsgDup(). Note that repeated message processing is not enabled + by default. +- bugfix: message sanitation had some issues: + - control character DEL was not properly escaped + - NUL and LF characters were not properly stripped if no control + character replacement was to be done + - NUL characters in the message body were silently dropped (this was + a regeression introduced by some of the recent optimizations) +- bugfix: strings improperly reused, resulting in some message properties + be populated with strings from previous messages. This was caused by + an improper predicate check. [backported from v5] +- fixed some minor portability issues +- bugfix: reverse lookup reduction logic in imudp do DNS queries too often + [imported from 4.4.2] +--------------------------------------------------------------------------- +Version 4.5.2 [v4-beta] (rgerhards), 2009-08-21 +- legacy syslog parser changed so that it now accepts date stamps in + wrong case. Some devices seem to create them and I do not see any harm + in supporting that. +- added $InputTCPMaxListeners directive - permits to specify how many + TCP servers shall be possible (default is 20). +- bugfix: memory leak with some input modules. Those inputs that + use parseAndSubmitMsg() leak two small memory blocks with every message. + Typically, those process only relatively few messages, so the issue + does most probably not have any effect in practice. +- bugfix: if tcp listen port could not be created, no error message was + emitted +- bugfix: potential segfault in output file writer (omfile) + In async write mode, we use modular arithmetic to index the output + buffer array. However, the counter variables accidently were signed, + thus resulting in negative indizes after integer overflow. That in turn + could lead to segfaults, but was depending on the memory layout of + the instance in question (which in turn depended on a number of + variables, like compile settings but also configuration). The counters + are now unsigned (as they always should have been) and so the dangling + mis-indexing does no longer happen. This bug potentially affected all + installations, even if only some may actually have seen a segfault. +- bugfix: hostnames with dashes in them were incorrectly treated as + malformed, thus causing them to be treated as TAG (this was a regression + introduced from the "rfc3164 strict" change in 4.5.0). +--------------------------------------------------------------------------- +Version 4.5.1 [DEVEL] (rgerhards), 2009-07-15 +- CONFIG CHANGE: $HUPisRestart default is now "off". We are doing this + to support removal of restart-type HUP in v5. +- bugfix: fromhost-ip was sometimes truncated +- bugfix: potential segfault when zip-compressed syslog records were + received (double free) +- bugfix: properties inputname, fromhost, fromhost-ip, msg were lost when + working with disk queues +- performance enhancement: much faster, up to twice as fast (depending + on configuration) +- bugfix: abort condition when RecvFrom was not set and message reduction + was on. Happend e.g. with imuxsock. +- added $klogConsoleLogLevel directive which permits to set a new + console log level while rsyslog is active +- bugfix: message could be truncated after TAG, often when forwarding + This was a result of an internal processing error if maximum field + sizes had been specified in the property replacer. +- added ability for the TCP output action to "rebind" its send socket after + sending n messages (actually, it re-opens the connection, the name is + used because this is a concept very similiar to $ActionUDPRebindInterval). + New config directive $ActionSendTCPRebindInterval added for the purpose. + By default, rebinding is disabled. This is considered useful for load + balancers. +- testbench improvements +--------------------------------------------------------------------------- +Version 4.5.0 [DEVEL] (rgerhards), 2009-07-02 +- activation order of inputs changed, they are now activated only after + privileges are dropped. Thanks to Michael Terry for the patch. +- greatly improved performance +- greatly reduced memory requirements of msg object + to around half of the previous demand. This means that more messages can + be stored in core! Due to fewer cache misses, this also means some + performance improvement. +- improved config error messages: now contain a copy of the config line + that (most likely) caused the error +- reduced max value for $DynaFileCacheSize to 1,000 (the former maximum + of 10,000 really made no sense, even 1,000 is very high, but we like + to keep the user in control ;)). +- added capability to fsync() queue disk files for enhanced reliability + (also add's speed, because you do no longer need to run the whole file + system in sync mode) +- more strict parsing of the hostname in rfc3164 mode, hopefully + removes false positives (but may cause some trouble with hostname + parsing). For details, see this bug tracker: + http://bugzilla.adiscon.com/show_bug.cgi?id=126 +- omfile rewrite to natively support zip files (includes large extension + of the stream class) +- added configuration commands (see doc for explanations) + * $OMFileZipLevel + * $OMFileIOBufferSize + * $OMFileFlushOnTXEnd + * $MainMsgQueueSyncQueueFiles + * $ActionQueueSyncQueueFiles +- done some memory accesses explicitely atomic +- bugfix: subtle (and usually irrelevant) issue in timout processing + timeout could be one second too early if nanoseconds wrapped +- set a more sensible timeout for shutdow, now 1.5 seconds to complete + processing (this also removes those cases where the shutdown message + was not written because the termination happened before it) +- internal bugfix: object pointer was only reset to NULL when an object + was actually destructed. This most likely had no effect to existing code, + but it may also have caused trouble in remote cases. Similarly, the fix + may also cause trouble... +- bugfix: missing initialization during timestamp creation + This could lead to timestamps written in the wrong format, but not to + an abort +--------------------------------------------------------------------------- +Version 4.4.3 [v4-stable] (rgerhards), 2009-10-?? +- bugfix: several smaller bugs resolved after flexelint review + Thanks to varmojfekoj for the patch. +- bugfix: $ActionExecOnlyOnceEveryInterval did not work. + This was a regression from the time() optimizations done in v4. + Bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=143 + Thanks to Klaus Tachtler for reporting this bug. +- bugfix: potential segfault on queue shutdown + Thanks to varmojfekoj for the patch. +- bugfix: potential hang condition on queue shutdown + [imported from v3-stable] +- bugfix: segfault on startup when -q or -Q option was given + [imported from v3-stable] +--------------------------------------------------------------------------- Version 4.4.2 [v4-stable] (rgerhards), 2009-10-09 - bugfix: invalid handling of zero-sized messages, could lead to mis- addressing and potential memory corruption/segfault @@ -256,10 +633,31 @@ version before switching to this one. - bugfix: memory leak in ompgsql Thanks to Ken for providing the patch --------------------------------------------------------------------------- -Version 3.22.2 [v3-stable] (rgerhards), 2009-07-?? +Version 3.22.3 [v3-stable] (rgerhards), 2010-11-24 +- bugfix(important): problem in TLS handling could cause rsyslog to loop + in a tight loop, effectively disabling functionality and bearing the + risk of unresponsiveness of the whole system. + Bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=194 +--------------------------------------------------------------------------- +Version 3.22.2 [v3-stable] (rgerhards), 2010-08-05 +- bugfix: comment char ('#') in literal terminated script parsing + and thus could not be used. + but tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=119 +- enhance: imrelp now also provides remote peer's IP address + [if librelp != 1.0.0 is used] - bugfix: sending syslog messages with zip compression did not work +- bugfix: potential hang condition on queue shutdown +- bugfix: segfault on startup when -q or -Q option was given + bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=157 + Thanks to Jonas Nogueira for reporting this bug. - clarified use of $ActionsSendStreamDriver[AuthMode/PermittedPeers] in doc set (require TLS drivers) +- bugfix: $CreateDirs variable not properly initialized, default thus + was random (but most often "on") +- bugfix: potential segfault when -p command line option was used + thanks to varmojfekoj for pointing me at this bug +- bugfix: programname filter in ! configuration can not be reset + Thanks to Kiss Gabor for the patch. --------------------------------------------------------------------------- Version 3.22.1 [v3-stable] (rgerhards), 2009-07-02 - bugfix: invalid error message issued if $inlcudeConfig was on an empty diff --git a/Makefile.am b/Makefile.am index a050e95e..f5f9a6ed 100644 --- a/Makefile.am +++ b/Makefile.am @@ -134,5 +134,5 @@ SUBDIRS += tests # temporarily be removed below. The intent behind forcing everthing to compile # in a make distcheck is so that we detect code that accidently was not updated # when some global update happened. -DISTCHECK_CONFIGURE_FLAGS=--enable-gssapi_krb5 --enable-imfile --enable-snmp --enable-pgsql --enable-libdbi --enable-mysql --enable-omtemplate --enable-imtemplate --enable-relp --enable-rsyslogd --enable-mail --enable-klog --enable-diagtools --enable-gnutls --enable-omstdout --enable-omprog --enable-imdiag --enable-shave +DISTCHECK_CONFIGURE_FLAGS=--enable-gssapi_krb5 --enable-imfile --enable-snmp --enable-pgsql --enable-libdbi --enable-mysql --enable-omtemplate --enable-imtemplate --enable-relp --enable-rsyslogd --enable-mail --enable-klog --enable-diagtools --enable-gnutls --enable-omstdout --enable-omprog --enable-imdiag --enable-shave --enable-extended-tests ACLOCAL_AMFLAGS = -I m4 @@ -4,7 +4,7 @@ * * File begun on 2007-08-06 by RGerhards (extracted from syslogd.c) * - * Copyright 2007 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2010 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -43,6 +43,7 @@ #include "srUtils.h" #include "errmsg.h" #include "datetime.h" +#include "unicode-helper.h" /* forward definitions */ rsRetVal actionCallDoAction(action_t *pAction, msg_t *pMsg); @@ -72,6 +73,7 @@ static int iActionQueueNumWorkers = 1; /* number of worker threads for the mm static uchar *pszActionQFName = NULL; /* prefix for the main message queue file */ static int64 iActionQueMaxFileSize = 1024*1024; static int iActionQPersistUpdCnt = 0; /* persist queue info every n updates */ +static int bActionQSyncQeueFiles = 0; /* sync queue files */ static int iActionQtoQShutdown = 0; /* queue shutdown */ static int iActionQtoActShutdown = 1000; /* action shutdown (in phase 2) */ static int iActionQtoEnq = 2000; /* timeout for queue enque */ @@ -151,6 +153,7 @@ actionResetQueueParams(void) iActionQueueNumWorkers = 1; /* number of worker threads for the mm queue above */ iActionQueMaxFileSize = 1024*1024; iActionQPersistUpdCnt = 0; /* persist queue info every n updates */ + bActionQSyncQeueFiles = 0; iActionQtoQShutdown = 0; /* queue shutdown */ iActionQtoActShutdown = 1000; /* action shutdown (in phase 2) */ iActionQtoEnq = 2000; /* timeout for queue enque */ @@ -176,6 +179,7 @@ actionResetQueueParams(void) */ rsRetVal actionDestruct(action_t *pThis) { + int i; DEFiRet; ASSERT(pThis != NULL); @@ -193,6 +197,33 @@ rsRetVal actionDestruct(action_t *pThis) pthread_mutex_destroy(&pThis->mutActExec); d_free(pThis->pszName); d_free(pThis->ppTpl); + + /* message ptr cleanup */ + for(i = 0 ; i < pThis->iNumTpls ; ++i) { + if(pThis->ppMsgs[i] != NULL) { + switch(pThis->eParamPassing) { + case ACT_ARRAY_PASSING: +#if 0 /* later! */ + iArr = 0; + while(((char **)pThis->ppMsgs[i])[iArr] != NULL) { + d_free(((char **)pThis->ppMsgs[i])[iArr++]); + ((char **)pThis->ppMsgs[i])[iArr++] = NULL; + } + d_free(pThis->ppMsgs[i]); + pThis->ppMsgs[i] = NULL; +#endif + break; + case ACT_STRING_PASSING: + d_free(pThis->ppMsgs[i]); + break; + default: + assert(0); + } + } + } + d_free(pThis->ppMsgs); + d_free(pThis->lenMsgs); + d_free(pThis); RETiRet; @@ -209,10 +240,7 @@ rsRetVal actionConstruct(action_t **ppThis) ASSERT(ppThis != NULL); - if((pThis = (action_t*) calloc(1, sizeof(action_t))) == NULL) { - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } - + CHKmalloc(pThis = (action_t*) calloc(1, sizeof(action_t))); pThis->iResumeInterval = glbliActionResumeInterval; pThis->iResumeRetryCount = glbliActionResumeRetryCount; pThis->tLastOccur = time(NULL); /* done once per action on startup only */ @@ -273,6 +301,7 @@ actionConstructFinalize(action_t *pThis) setQPROP(qqueueSetMaxFileSize, "$ActionQueueFileSize", iActionQueMaxFileSize); setQPROPstr(qqueueSetFilePrefix, "$ActionQueueFileName", pszActionQFName); setQPROP(qqueueSetiPersistUpdCnt, "$ActionQueueCheckpointInterval", iActionQPersistUpdCnt); + setQPROP(qqueueSetbSyncQueueFiles, "$ActionQueueSyncQueueFiles", bActionQSyncQeueFiles); setQPROP(qqueueSettoQShutdown, "$ActionQueueTimeoutShutdown", iActionQtoQShutdown ); setQPROP(qqueueSettoActShutdown, "$ActionQueueTimeoutActionCompletion", iActionQtoActShutdown); setQPROP(qqueueSettoWrkShutdown, "$ActionQueueWorkerTimeoutThreadShutdown", iActionQtoWrkShutdown); @@ -295,7 +324,7 @@ actionConstructFinalize(action_t *pThis) CHKiRet(qqueueStart(pThis->pQueue)); - dbgprintf("Action %p: queue %p created\n", pThis, pThis->pQueue); + DBGPRINTF("Action %p: queue %p created\n", pThis, pThis->pQueue); /* and now reset the queue params (see comment in its function header!) */ actionResetQueueParams(); @@ -383,7 +412,7 @@ static rsRetVal actionTryResume(action_t *pThis) if(iRet == RS_RET_OK) actionResume(pThis); - dbgprintf("actionTryResume: iRet: %d, next retry (if applicable): %u [now %u]\n", + DBGPRINTF("actionTryResume: iRet: %d, next retry (if applicable): %u [now %u]\n", iRet, (unsigned) pThis->ttResumeRtry, (unsigned) ttNow); RETiRet; @@ -429,37 +458,33 @@ actionCallDoAction(action_t *pAction, msg_t *pMsg) int iSleepPeriod; int bCallAction; int iCancelStateSave; - uchar **ppMsgs; /* array of message pointers for doAction */ ASSERT(pAction != NULL); - /* create the array for doAction() message pointers */ - if((ppMsgs = calloc(pAction->iNumTpls, sizeof(uchar *))) == NULL) { - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } + /* We now must guard the output module against execution by multiple threads. The + * plugin interface specifies that output modules must not be thread-safe (except + * if they notify us they are - functionality not yet implemented...). + * rgerhards, 2008-01-30 + */ + pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &iCancelStateSave); + d_pthread_mutex_lock(&pAction->mutActExec); + pthread_cleanup_push(mutexCancelCleanup, &pAction->mutActExec); + pthread_setcancelstate(iCancelStateSave, NULL); /* here we must loop to process all requested strings */ for(i = 0 ; i < pAction->iNumTpls ; ++i) { switch(pAction->eParamPassing) { case ACT_STRING_PASSING: - CHKiRet(tplToString(pAction->ppTpl[i], pMsg, &(ppMsgs[i]))); + CHKiRet(tplToString(pAction->ppTpl[i], pMsg, &(pAction->ppMsgs[i]), &(pAction->lenMsgs[i]))); break; case ACT_ARRAY_PASSING: - CHKiRet(tplToArray(pAction->ppTpl[i], pMsg, (uchar***) &(ppMsgs[i]))); + CHKiRet(tplToArray(pAction->ppTpl[i], pMsg, (uchar***) &(pAction->ppMsgs[i]))); break; default:assert(0); /* software bug if this happens! */ } } + iRetries = 0; - /* We now must guard the output module against execution by multiple threads. The - * plugin interface specifies that output modules must not be thread-safe (except - * if they notify us they are - functionality not yet implemented...). - * rgerhards, 2008-01-30 - */ - pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &iCancelStateSave); - d_pthread_mutex_lock(&pAction->mutActExec); - pthread_cleanup_push(mutexCancelCleanup, &pAction->mutActExec); - pthread_setcancelstate(iCancelStateSave, NULL); do { /* on first invocation, this if should never be true. We just put it at the top * of the loop so that processing (and code) is simplified. This code is actually @@ -484,9 +509,9 @@ actionCallDoAction(action_t *pAction, msg_t *pMsg) if(bCallAction) { /* call configured action */ - iRet = pAction->pMod->mod.om.doAction(ppMsgs, pMsg->msgFlags, pAction->pModData); + iRet = pAction->pMod->mod.om.doAction(pAction->ppMsgs, pMsg->msgFlags, pAction->pModData); if(iRet == RS_RET_SUSPENDED) { - dbgprintf("Action requested to be suspended, done that.\n"); + DBGPRINTF("Action requested to be suspended, done that.\n"); actionSuspend(pAction, getActNow(pAction)); } } @@ -494,34 +519,35 @@ actionCallDoAction(action_t *pAction, msg_t *pMsg) } while(iRet == RS_RET_SUSPENDED && (pAction->iResumeRetryCount == -1 || iRetries < pAction->iResumeRetryCount)); /* do...while! */ if(iRet == RS_RET_DISABLE_ACTION) { - dbgprintf("Action requested to be disabled, done that.\n"); + DBGPRINTF("Action requested to be disabled, done that.\n"); pAction->bEnabled = 0; /* that's it... */ } - pthread_cleanup_pop(1); /* unlock mutex */ - finalize_it: /* cleanup */ for(i = 0 ; i < pAction->iNumTpls ; ++i) { - if(ppMsgs[i] != NULL) { + if(pAction->ppMsgs[i] != NULL) { switch(pAction->eParamPassing) { case ACT_ARRAY_PASSING: iArr = 0; - while(((char **)ppMsgs[i])[iArr] != NULL) - d_free(((char **)ppMsgs[i])[iArr++]); - d_free(ppMsgs[i]); + while(((char **)pAction->ppMsgs[i])[iArr] != NULL) { + d_free(((char **)pAction->ppMsgs[i])[iArr++]); + ((char **)pAction->ppMsgs[i])[iArr++] = NULL; + } + d_free(pAction->ppMsgs[i]); + pAction->ppMsgs[i] = NULL; break; case ACT_STRING_PASSING: - d_free(ppMsgs[i]); break; default: assert(0); } } } - d_free(ppMsgs); - msgDestruct(&pMsg); /* we are now finished with the message */ + pthread_cleanup_pop(1); /* unlock mutex */ + + msgDestruct(&pMsg); /* we are now finished with the message */ RETiRet; } #pragma GCC diagnostic warning "-Wempty-body" @@ -569,16 +595,16 @@ static rsRetVal setActionQueType(void __attribute__((unused)) *pVal, uchar *pszT if (!strcasecmp((char *) pszType, "fixedarray")) { ActionQueType = QUEUETYPE_FIXED_ARRAY; - dbgprintf("action queue type set to FIXED_ARRAY\n"); + DBGPRINTF("action queue type set to FIXED_ARRAY\n"); } else if (!strcasecmp((char *) pszType, "linkedlist")) { ActionQueType = QUEUETYPE_LINKEDLIST; - dbgprintf("action queue type set to LINKEDLIST\n"); + DBGPRINTF("action queue type set to LINKEDLIST\n"); } else if (!strcasecmp((char *) pszType, "disk")) { ActionQueType = QUEUETYPE_DISK; - dbgprintf("action queue type set to DISK\n"); + DBGPRINTF("action queue type set to DISK\n"); } else if (!strcasecmp((char *) pszType, "direct")) { ActionQueType = QUEUETYPE_DIRECT; - dbgprintf("action queue type set to DIRECT (no queueing at all)\n"); + DBGPRINTF("action queue type set to DIRECT (no queueing at all)\n"); } else { errmsg.LogError(0, RS_RET_INVALID_PARAMS, "unknown actionqueue parameter: %s", (char *) pszType); iRet = RS_RET_INVALID_PARAMS; @@ -624,15 +650,15 @@ actionWriteToAction(action_t *pAction) /* we need to care about multiple occurences */ if( pAction->iExecEveryNthOccurTO > 0 && (getActNow(pAction) - pAction->tLastOccur) > pAction->iExecEveryNthOccurTO) { - dbgprintf("n-th occurence handling timed out (%d sec), restarting from 0\n", + DBGPRINTF("n-th occurence handling timed out (%d sec), restarting from 0\n", (int) (getActNow(pAction) - pAction->tLastOccur)); pAction->iNbrNoExec = 0; pAction->tLastOccur = getActNow(pAction); } if(pAction->iNbrNoExec < pAction->iExecEveryNthOccur - 1) { ++pAction->iNbrNoExec; - dbgprintf("action %p passed %d times to execution - less than neded - discarding\n", - pAction, pAction->iNbrNoExec); + DBGPRINTF("action %p passed %d times to execution - less than neded - discarding\n", + pAction, pAction->iNbrNoExec); FINALIZE; } else { pAction->iNbrNoExec = 0; /* we execute the action now, so the number of no execs is down to */ @@ -649,36 +675,34 @@ actionWriteToAction(action_t *pAction) */ if(pAction->f_prevcount > 1) { msg_t *pMsg; + size_t lenRepMsg; uchar szRepMsg[1024]; if((pMsg = MsgDup(pAction->f_pMsg)) == NULL) { /* it failed - nothing we can do against it... */ - dbgprintf("Message duplication failed, dropping repeat message.\n"); + DBGPRINTF("Message duplication failed, dropping repeat message.\n"); ABORT_FINALIZE(RS_RET_ERR); } if(pAction->bRepMsgHasMsg == 0) { /* old format repeat message? */ - snprintf((char*)szRepMsg, sizeof(szRepMsg), "last message repeated %d times", + lenRepMsg = snprintf((char*)szRepMsg, sizeof(szRepMsg), " last message repeated %d times", pAction->f_prevcount); } else { - snprintf((char*)szRepMsg, sizeof(szRepMsg), "message repeated %d times: [%.800s]", + lenRepMsg = snprintf((char*)szRepMsg, sizeof(szRepMsg), " message repeated %d times: [%.800s]", pAction->f_prevcount, getMSG(pAction->f_pMsg)); } - /* We now need to update the other message properties. - * ... RAWMSG is a problem ... Please note that digital + /* We now need to update the other message properties. Please note that digital * signatures inside the message are also invalidated. */ datetime.getCurrTime(&(pMsg->tRcvdAt), &(pMsg->ttGenTime)); memcpy(&pMsg->tTIMESTAMP, &pMsg->tRcvdAt, sizeof(struct syslogTime)); - MsgSetMSG(pMsg, (char*)szRepMsg); - MsgSetRawMsg(pMsg, (char*)szRepMsg); - + MsgReplaceMSG(pMsg, szRepMsg, lenRepMsg); pMsgSave = pAction->f_pMsg; /* save message pointer for later restoration */ pAction->f_pMsg = pMsg; /* use the new msg (pointer will be restored below) */ } - dbgprintf("Called action, logging to %s\n", module.GetStateName(pAction->pMod)); + DBGPRINTF("Called action, logging to %s\n", module.GetStateName(pAction->pMod)); /* now check if we need to drop the message because otherwise the action would be too * frequently called. -- rgerhards, 2008-04-08 @@ -686,10 +710,10 @@ actionWriteToAction(action_t *pAction) * a purely logical point of view. However, if safes us to check the system time in * (those common) cases where ExecOnceInterval is not used. -- rgerhards, 2008-09-16 */ - if(pAction->f_time != 0 && pAction->iSecsExecOnceInterval > 0 && + if(pAction->iSecsExecOnceInterval > 0 && pAction->iSecsExecOnceInterval + pAction->tLastExec > getActNow(pAction)) { /* in this case we need to discard the message - its not yet time to exec the action */ - dbgprintf("action not yet ready again to be executed, onceInterval %d, tCurr %d, tNext %d\n", + DBGPRINTF("action not yet ready again to be executed, onceInterval %d, tCurr %d, tNext %d\n", (int) pAction->iSecsExecOnceInterval, (int) getActNow(pAction), (int) (pAction->iSecsExecOnceInterval + pAction->tLastExec)); pAction->tLastExec = getActNow(pAction); /* re-init time flags */ @@ -697,6 +721,7 @@ actionWriteToAction(action_t *pAction) } /* we use reception time, not dequeue time - this is considered more appropriate and also faster ;) -- rgerhards, 2008-09-17 */ + pAction->tLastExec = getActNow(pAction); /* re-init time flags */ pAction->f_time = pAction->f_pMsg->ttGenTime; /* When we reach this point, we have a valid, non-disabled action. @@ -726,32 +751,13 @@ finalize_it: } -/* call the configured action. Does all necessary housekeeping. - * rgerhards, 2007-08-01 - * FYI: currently, this function is only called from the queue - * consumer. So we (conceptually) run detached from the input - * threads (which also means we may run much later than when the - * message was generated). +/* helper to actonCallAction, mostly needed because of this damn + * pthread_cleanup_push() POSIX macro... */ -#pragma GCC diagnostic ignored "-Wempty-body" -rsRetVal -actionCallAction(action_t *pAction, msg_t *pMsg) +static rsRetVal +doActionCallAction(action_t *pAction, msg_t *pMsg) { DEFiRet; - int iCancelStateSave; - - ISOBJ_TYPE_assert(pMsg, msg); - ASSERT(pAction != NULL); - - /* Make sure nodbody else modifies/uses this action object. Right now, this - * is important because of "message repeated n times" processing and potentially - * multiple worker threads. -- rgerhards, 2007-12-11 - */ - pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &iCancelStateSave); - LockObj(pAction); - pthread_cleanup_push(mutexCancelCleanup, pAction->Sync_mut); - pthread_setcancelstate(iCancelStateSave, NULL); - /* first, we need to check if this is a disabled * entry. If so, we must not further process it. * rgerhards 2005-09-26 @@ -776,12 +782,12 @@ actionCallAction(action_t *pAction, msg_t *pMsg) /* suppress duplicate messages */ if ((pAction->f_ReduceRepeated == 1) && pAction->f_pMsg != NULL && (pMsg->msgFlags & MARK) == 0 && getMSGLen(pMsg) == getMSGLen(pAction->f_pMsg) && - !strcmp(getMSG(pMsg), getMSG(pAction->f_pMsg)) && + !ustrcmp(getMSG(pMsg), getMSG(pAction->f_pMsg)) && !strcmp(getHOSTNAME(pMsg), getHOSTNAME(pAction->f_pMsg)) && - !strcmp(getPROCID(pMsg), getPROCID(pAction->f_pMsg)) && - !strcmp(getAPPNAME(pMsg), getAPPNAME(pAction->f_pMsg))) { + !strcmp(getPROCID(pMsg, LOCK_MUTEX), getPROCID(pAction->f_pMsg, LOCK_MUTEX)) && + !strcmp(getAPPNAME(pMsg, LOCK_MUTEX), getAPPNAME(pAction->f_pMsg, LOCK_MUTEX))) { pAction->f_prevcount++; - dbgprintf("msg repeated %d times, %ld sec of %d.\n", + DBGPRINTF("msg repeated %d times, %ld sec of %d.\n", pAction->f_prevcount, (long) getActNow(pAction) - pAction->f_time, repeatinterval[pAction->f_repeatcount]); /* use current message, so we have the new timestamp (means we need to discard previous one) */ @@ -812,10 +818,36 @@ actionCallAction(action_t *pAction, msg_t *pMsg) } finalize_it: + RETiRet; +} + +/* call the configured action. Does all necessary housekeeping. + * rgerhards, 2007-08-01 + * FYI: currently, this function is only called from the queue + * consumer. So we (conceptually) run detached from the input + * threads (which also means we may run much later than when the + * message was generated). + */ +#pragma GCC diagnostic ignored "-Wempty-body" +rsRetVal +actionCallAction(action_t *pAction, msg_t *pMsg) +{ + DEFiRet; + int iCancelStateSave; + + ISOBJ_TYPE_assert(pMsg, msg); + ASSERT(pAction != NULL); + + pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &iCancelStateSave); + LockObj(pAction); + pthread_cleanup_push(mutexCancelCleanup, pAction->Sync_mut); + pthread_setcancelstate(iCancelStateSave, NULL); + iRet = doActionCallAction(pAction, pMsg); pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &iCancelStateSave); UnlockObj(pAction); pthread_cleanup_pop(0); /* remove mutex cleanup handler */ pthread_setcancelstate(iCancelStateSave, NULL); + RETiRet; } #pragma GCC diagnostic warning "-Wempty-body" @@ -838,6 +870,7 @@ actionAddCfSysLineHdrl(void) CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuediscardmark", 0, eCmdHdlrInt, NULL, &iActionQDiscardMark, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuediscardseverity", 0, eCmdHdlrInt, NULL, &iActionQDiscardSeverity, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuecheckpointinterval", 0, eCmdHdlrInt, NULL, &iActionQPersistUpdCnt, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuesyncqueuefiles", 0, eCmdHdlrBinary, NULL, &bActionQSyncQeueFiles, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuetype", 0, eCmdHdlrGetWord, setActionQueType, NULL, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"actionqueueworkerthreads", 0, eCmdHdlrInt, NULL, &iActionQueueNumWorkers, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuetimeoutshutdown", 0, eCmdHdlrInt, NULL, &iActionQtoQShutdown, NULL)); @@ -877,7 +910,7 @@ addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, omodStringReques assert(ppAction != NULL); assert(pMod != NULL); assert(pOMSR != NULL); - dbgprintf("Module %s processed this config line.\n", module.GetName(pMod)); + DBGPRINTF("Module %s processed this config line.\n", module.GetName(pMod)); CHKiRet(actionConstruct(&pAction)); /* create action object first */ pAction->pMod = pMod; @@ -901,9 +934,9 @@ addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, omodStringReques */ if(pAction->iNumTpls > 0) { /* we first need to create the template pointer array */ - if((pAction->ppTpl = calloc(pAction->iNumTpls, sizeof(struct template *))) == NULL) { - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } + CHKmalloc(pAction->ppTpl = (struct template **)calloc(pAction->iNumTpls, sizeof(struct template *))); + CHKmalloc(pAction->ppMsgs = (uchar**) calloc(pAction->iNumTpls, sizeof(uchar *))); + CHKmalloc(pAction->lenMsgs = (size_t*) calloc(pAction->iNumTpls, sizeof(size_t))); } for(i = 0 ; i < pAction->iNumTpls ; ++i) { @@ -936,7 +969,7 @@ addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, omodStringReques pAction->eParamPassing = ACT_STRING_PASSING; } - dbgprintf("template: '%s' assigned\n", pTplName); + DBGPRINTF("template: '%s' assigned\n", pTplName); } pAction->pMod = pMod; @@ -945,7 +978,7 @@ addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, omodStringReques if(pMod->isCompatibleWithFeature(sFEATURERepeatedMsgReduction) == RS_RET_OK) pAction->f_ReduceRepeated = bReduceRepeatMsgs; else { - dbgprintf("module is incompatible with RepeatedMsgReduction - turned off\n"); + DBGPRINTF("module is incompatible with RepeatedMsgReduction - turned off\n"); pAction->f_ReduceRepeated = 0; } pAction->bEnabled = 1; /* action is enabled */ @@ -43,10 +43,10 @@ struct action_s { time_t tActNow; /* the current time for an action execution. Initially set to -1 and populated on an as-needed basis. This is a performance optimization. */ time_t tLastExec; /* time this action was last executed */ - int bExecWhenPrevSusp;/* execute only when previous action is suspended? */ + bool bExecWhenPrevSusp;/* execute only when previous action is suspended? */ int iSecsExecOnceInterval; /* if non-zero, minimum seconds to wait until action is executed again */ short bEnabled; /* is the related action enabled (1) or disabled (0)? */ - short bSuspended; /* is the related action temporarily suspended? */ + bool bSuspended; /* is the related action temporarily suspended? */ time_t ttResumeRtry; /* when is it time to retry the resume? */ int iResumeInterval;/* resume interval for this action */ int iResumeRetryCount;/* how often shall we retry a suspended action? (-1 --> eternal) */ @@ -57,7 +57,7 @@ struct action_s { time_t tLastOccur; /* time last occurence was seen (for timing them out) */ struct modInfo_s *pMod;/* pointer to output module handling this selector */ void *pModData; /* pointer to module data - content is module-specific */ - short bRepMsgHasMsg; /* "message repeated..." has msg fragment in it (0-no, 1-yes) */ + bool bRepMsgHasMsg; /* "message repeated..." has msg fragment in it (0-no, 1-yes) */ short f_ReduceRepeated;/* reduce repeated lines 0 - no, 1 - yes */ int f_prevcount; /* repetition cnt of prevline */ int f_repeatcount; /* number of "repeated" msgs */ @@ -66,14 +66,16 @@ struct action_s { int iNumTpls; /* number of array entries for template element below */ struct template **ppTpl;/* array of template to use - strings must be passed to doAction * in this order. */ - struct msg* f_pMsg; /* pointer to the message (this will replace the other vars with msg + msg_t *f_pMsg; /* pointer to the message (this will replace the other vars with msg * content later). This is preserved after the message has been * processed - it is also used to detect duplicates. */ qqueue_t *pQueue; /* action queue */ SYNC_OBJ_TOOL; /* required for mutex support */ - uchar *pszName; /* action name (for documentation) */ pthread_mutex_t mutActExec; /* mutex to guard actual execution of doAction for single-threaded modules */ + uchar *pszName; /* action name (for documentation) */ + uchar **ppMsgs; /* pointer to action-calling parameters (kept in structure to save alloc() time!) */ + size_t *lenMsgs; /* length of message in ppMsgs */ }; typedef struct action_s action_t; diff --git a/configure.ac b/configure.ac index 7b4797aa..0e5544b1 100644 --- a/configure.ac +++ b/configure.ac @@ -2,7 +2,7 @@ # Process this file with autoconf to produce a configure script. AC_PREREQ(2.61) -AC_INIT([rsyslog],[4.4.2a],[rsyslog@lists.adiscon.com]) +AC_INIT([rsyslog],[4.6.5],[rsyslog@lists.adiscon.com]) AM_INIT_AUTOMAKE AC_CONFIG_SRCDIR([ChangeLog]) AC_CONFIG_MACRO_DIR([m4]) @@ -69,7 +69,7 @@ AC_SUBST(DL_LIBS) AC_HEADER_RESOLV AC_HEADER_STDC AC_HEADER_SYS_WAIT -AC_CHECK_HEADERS([arpa/inet.h libgen.h fcntl.h locale.h netdb.h netinet/in.h paths.h stddef.h stdlib.h string.h sys/file.h sys/ioctl.h sys/param.h sys/socket.h sys/time.h sys/stat.h syslog.h unistd.h utmp.h sys/epoll.h]) +AC_CHECK_HEADERS([arpa/inet.h libgen.h malloc.h fcntl.h locale.h netdb.h netinet/in.h paths.h stddef.h stdlib.h string.h sys/file.h sys/ioctl.h sys/param.h sys/socket.h sys/time.h sys/stat.h syslog.h unistd.h utmp.h utmpx.h sys/epoll.h sys/prctl.h]) # Checks for typedefs, structures, and compiler characteristics. AC_C_CONST @@ -104,7 +104,7 @@ AC_TYPE_SIGNAL AC_FUNC_STAT AC_FUNC_STRERROR_R AC_FUNC_VPRINTF -AC_CHECK_FUNCS([flock basename alarm clock_gettime gethostbyname gethostname gettimeofday localtime_r memset mkdir regcomp select setid socket strcasecmp strchr strdup strerror strndup strnlen strrchr strstr strtol strtoul uname ttyname_r epoll_wait getline]) +AC_CHECK_FUNCS([flock basename alarm clock_gettime gethostbyname gethostname gettimeofday localtime_r memset mkdir regcomp select setid socket strcasecmp strchr strdup strerror strndup strnlen strrchr strstr strtol strtoul uname ttyname_r epoll_wait getline malloc_trim prctl fdatasync lseek64]) # Check for MAXHOSTNAMELEN AC_MSG_CHECKING(for MAXHOSTNAMELEN) @@ -152,19 +152,12 @@ AC_SUBST(moddirs) # Large file support -AC_ARG_ENABLE(largefile, - [AS_HELP_STRING([--enable-largefile],[Enable large file support @<:@default=yes@:>@])], - [case "${enableval}" in - yes) enable_largefile="yes" ;; - no) enable_largefile="no" ;; - *) AC_MSG_ERROR(bad value ${enableval} for --enable-largefile) ;; - esac], - [enable_largefile="yes"] -) -if test "$enable_largefile" = "no"; then - AC_DEFINE(NOLARGEFILE, 1, [Defined when large file support is disabled.]) -fi - +# http://www.gnu.org/software/autoconf/manual/html_node/System-Services.html#index-AC_005fSYS_005fLARGEFILE-1028 +AC_SYS_LARGEFILE +case "${enable_largefile}" in + no) ;; + *) enable_largefile="yes" ;; +esac # Regular expressions AC_ARG_ENABLE(regexp, @@ -192,6 +185,7 @@ AC_ARG_ENABLE(zlib, esac], [enable_zlib=yes] ) +AM_CONDITIONAL(ENABLE_ZLIB, test x$enable_zlib = xyes) if test "$enable_zlib" = "yes"; then AC_CHECK_HEADER(zlib.h, [zlib_header="yes"], [zlib_header="no" enable_zlib="false"]) if test "$zlib_header" = "yes"; then @@ -618,6 +612,23 @@ AC_ARG_ENABLE(rsyslogd, AM_CONDITIONAL(ENABLE_RSYSLOGD, test x$enable_rsyslogd = xyes) +# capability to enable an extended testbench. By default, this is off. The reason +# for this switch is that some test simply take too long to execute them on a regular +# basis. So we enable to skip them, while the majority of tests can still be used. The +# idea is that at least "make distcheck" executes the extended testbench, and also +# developers should explicitely enable it after important changes. -- rgerhards, 2010-04-12 +AC_ARG_ENABLE(extended_tests, + [AS_HELP_STRING([--enable-extended-tests],[extended testbench @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_rsyslogd="yes" ;; + no) enable_rsyslogd="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-extended-tests) ;; + esac], + [enable_extended_tests=no] +) +AM_CONDITIONAL(ENABLE_EXTENDED_TESTS, test x$enable_extended_tests = xyes) + + # Mail support (so far we do not need a library, but we need to turn this on and off) AC_ARG_ENABLE(mail, [AS_HELP_STRING([--enable-mail],[Enable mail support @<:@default=no@:>@])], @@ -870,6 +881,7 @@ echo " SNMP support enabled: $enable_snmp" echo echo "---{ debugging support }---" echo " Testbench enabled: $enable_testbench" +echo " Extended Testbench enabled: $enable_extended_tests" echo " Debug mode enabled: $enable_debug" echo " Runtime Instrumentation enabled: $enable_rtinst" echo " Diagnostic tools enabled: $enable_diagtools" @@ -27,16 +27,14 @@ #ifndef DIRTY_H_INCLUDED #define DIRTY_H_INCLUDED 1 +rsRetVal multiSubmitMsg(multi_submit_t *pMultiSub); rsRetVal submitMsg(msg_t *pMsg); rsRetVal logmsgInternal(int iErr, int pri, uchar *msg, int flags); -rsRetVal parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int flags, flowControl_t flowCtlTypeu, uchar *pszInputName, struct syslogTime *stTime, time_t ttGenTime); +rsRetVal parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int flags, flowControl_t flowCtlTypeu, prop_t *pInputName, struct syslogTime *stTime, time_t ttGenTime); int parseRFCSyslogMsg(msg_t *pMsg, int flags); int parseLegacySyslogMsg(msg_t *pMsg, int flags); rsRetVal diagGetMainMsgQSize(int *piSize); /* for imdiag */ - -/* TODO: the following 2 need to go in conf obj interface... */ -rsRetVal cflineParseTemplateName(uchar** pp, omodStringRequest_t *pOMSR, int iEntry, int iTplOpts, uchar *dfltTplName); -rsRetVal cflineParseFileName(uchar* p, uchar *pFileName, omodStringRequest_t *pOMSR, int iEntry, int iTplOpts, uchar *pszTpl); +char* getFIOPName(unsigned iFIOP); /* Intervals at which we flush out "message repeated" messages, * in seconds after previous message is logged. After each flush, diff --git a/doc/Makefile.am b/doc/Makefile.am index 0703b8fc..ca2ee71c 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -13,13 +13,13 @@ html_files = \ ipv6.html \ log_rotation_fix_size.html \ manual.html \ - man_rsyslogd.html \ modules.html \ property_replacer.html \ rsyslog_ng_comparison.html \ rsyslog_conf.html \ rsyslog-example.conf \ rsyslog_mysql.html \ + rsyslog_pgsql.html \ rsyslog_packages.html \ rsyslog_high_database_rate.html \ rsyslog_php_syslog_ng.html \ @@ -41,7 +41,6 @@ html_files = \ imrelp.html \ imuxsock.html \ imklog.html \ - professional_support.html \ queues.html \ src/queueWorkerLogic.dia \ queueWorkerLogic.jpg \ @@ -91,6 +90,7 @@ html_files = \ rsconf1_resetconfigvariables.html \ rsconf1_umask.html \ v3compatibility.html \ + v4compatibility.html \ im3195.html \ netstream.html \ ns_gtls.html \ @@ -111,6 +111,7 @@ html_files = \ rsyslog_conf_templates.html \ rsyslog_conf_nomatch.html \ queues_analogy.html \ + multi_ruleset.html \ src/classes.dia grfx_files = \ diff --git a/doc/debug.html b/doc/debug.html index de77f04a..46759986 100644 --- a/doc/debug.html +++ b/doc/debug.html @@ -1,41 +1,145 @@ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head> -<meta http-equiv="Content-Language" content="en"><title>Debug Support</title></head> +<html> +<head> +<meta http-equiv="Content-Language" content="en"> +<title>Rsyslog Debug Support</title></head> <body> -<h1>Debug Support</h1> +<h1>Rsyslog Debug Support</h1> <p> -Rsyslog provides a number of debug aids. Some of them are activated by +Rsyslog provides a number of debug aides. Some of them are activated by adding the --enable-rtinst ./configure option ("rtinst" means runtime instrumentation). Turning debugging on obviously costs some performance (in some cases considerable). </p> <p>This is document is just being created and thus terse.</p> -<p style="font-weight: bold;">Signals supported</p> -<p>SIGUSR1 - turns debug messages on and off (expect this signal -to go away over time)</p> -<p>SIGUSR2 - outputs debug information (including active threads +<h2>Signals supported</h2> +<p><b>SIGUSR1</b> - turns debug messages on and off. Note that for this +signal to work, rsyslogd must be running with debugging enabled, either +via the -d command line switch or the environment options specified below. +It is <b>not</b> required that rsyslog was compiled with debugging enabled +(but depending on the settings this may lead to better debug info). +<p><b>SIGUSR2</b> - outputs debug information (including active threads and a call stack) for the state when SIGUSR2 was received. This is a -one-time output. Can be sent as often as the user likes.</p> -<p style="font-weight: bold;">Environment Variables</p> -<p>There are two environment variables that set several debug settings. The "RSYSLOG_DEBUGLOG" (sample: RSYSLOG_DEBUGLOG="/path/to/debuglog/") +one-time output. Can be sent as often as the user likes. +<p><b>Note:</b> this signal <b>may go away</b> in later releases and may +be replaced by something else.</p> +<h2>Environment Variables</h2> +<p>There are two environment variables that set several debug settings: +<ul> +<li>The "RSYSLOG_DEBUGLOG" (sample: RSYSLOG_DEBUGLOG="/path/to/debuglog/") writes (allmost) all debug message to the specified log file in addition to stdout. Some system messages (e.g. segfault or abort message) are not written to the -file as we can not capture them. Runtime debug support is controlled by -"RSYSLOG_DEBUG". It contains an option string with the following -options possible (all are case insensitive):</p><ul><li><span style="font-weight: bold;">LogFuncFlow</span> - print out the logical flow of functions (entering and exiting them)</li><li><span style="font-weight: bold;">FileTrace</span> - specifies which files to trace LogFuncFlow. If <span style="font-weight: bold;">not</span> +file as we can not capture them. +<li>Runtime debug support is controlled by "RSYSLOG_DEBUG". +<p>The "RSYSLOG_DEBUG" environment variable contains an option string with the following +options possible (all are case insensitive):</p> +<ul> +<li><b>LogFuncFlow</b> - print out the logical flow of functions (entering and exiting them)</li> +<li><b>FileTrace</b> - specifies which files to trace LogFuncFlow. If <b>not</b> set (the default), a LogFuncFlow trace is provided for all files. Set to limit it to the files specified. FileTrace may be specified multiple times, one file each (e.g. export RSYSLOG_DEBUG="LogFuncFlow -FileTrace=vm.c FileTrace=expr.c"</li><li><span style="font-weight: bold;">PrintFuncDB</span> - print the content of the debug function database whenever debug information is printed (e.g. abort case)!</li><li><span style="font-weight: bold;">PrintAllDebugInfoOnExit</span> - print all debug information immediately before rsyslogd exits (<span style="font-weight: bold; font-style: italic;">currently not implemented!</span>)</li><li><span style="font-weight: bold;">PrintMutexAction</span> - print mutex action as it happens. Useful for finding deadlocks and such.</li><li><span style="font-weight: bold;">NoLogTimeStamp</span> - do not prefix log lines with a timestamp (default is to do that).</li><li><span style="font-weight: bold;">NoStdOut</span> - do not emit debug messages to stdout. If RSYSLOG_DEBUGLOG is not set, this means no messages will be displayed at all.</li><li><span style="font-weight: bold;">help</span> - display a very short list of commands - hopefully a life saver if you can't access the documentation...</li></ul> +FileTrace=vm.c FileTrace=expr.c"</li> +<li><b>PrintFuncDB</b> - print the content of the debug function database whenever debug information is printed (e.g. abort case)!</li> +<li><b>PrintAllDebugInfoOnExit</b> - print all debug information immediately before rsyslogd exits (<span style="font-weight: bold; font-style: italic;">currently not implemented!</span>)</li> +<li><b>PrintMutexAction</b> - print mutex action as it happens. Useful for finding deadlocks and such.</li> +<li><b>NoLogTimeStamp</b> - do not prefix log lines with a timestamp (default is to do that).</li> +<li><b>NoStdOut</b> - do not emit debug messages to stdout. If RSYSLOG_DEBUGLOG is not set, this means no messages will be displayed at all.</li> +<li><b>Debug</b> - if present, turns on the debug system and enables debug output +<li><b>DebugOnDemand</b> - if present, turns on the debug system but does not enable +debug output itself. You need to send SIGUSR1 to turn it on when desired. +<li><b>help</b> - display a very short list of commands - hopefully a life saver if you can't access the documentation...</li> +</ul> +</ul> +<h3>Why Environment Variables?</h3> +<p>You may ask why we use environment variables for debug-system parameters and not +the usual rsyslog.conf configuration commands. After all, environment variables force one +to change distro-specific configuration files, whereas regular configuration directives +would fit nicely into the one central rsyslog.conf. +<p>The problem here is that many settings of the debug system must be initialized +before the full rsyslog engine starts up. At that point, there is no such thing like +rsyslog.conf or the objects needed to process it present in an running instance. +And even if we would enable to change settings some time later, that would mean that +we can not correctly monitor (and debug) the initial startup phase of rsyslogd. What +makes matters worse is that during this startup phase (and never again later!) some +of the base debug structure needs to be created, at least if the build is +configured for that (many of these things only happen in --enable-rtinst mode). So +if we do not initialize the debug system <b>before</b> actually startig up the +rsyslog core, we get a number of data structures wrong. +<p>For these reasons, we utilize environment variables to initialize and configure +the debugging system. We understand this may be somewhat painful, but now you know +there are at least some good reasons for doing so. +<h2>Getting debug information from a running Instance</h2> +<p>It is possible to obtain debugging information from a running instance, but this requires +some setup. We assume that the instance runs in the background, so debug output to +stdout is not desired. As such, all debug information needs to go into a log file. +<p>To create this setup, you need to <ul> +<li>point the RSYSLOG_DEBUGLOG environment variable to a file that is accessible +during the while runtime (we strongly suggest a file in the local file system!) +<li>set RSYSLOG_DEBUG at least to "DebugOnDeman NoStdOut" +<li>make sure these environment variables are set in the correct (distro-specifc) +startup script if you do not run rsyslogd interactively </ul> +<p>These settings enable the capability to react to SIGUSR1. The signal will toggle +debug status when received. So send it one to turn debug loggin on, and send it again +to turn debug logging off again. The third time it will be turned on again ... and so on. +<p>On a typical system, you can signal rsyslogd as follows: +<pre> +kill -USR1 `cat /var/run/rsyslogd.pid` +</pre> +Important: there are backticks around the "cat"-command. If you use the regular +quote it won't work. The debug log will show whether debug logging has been turned +on or off. There is no other indication of the status. +<p>Note: running with DebugOnDemand by itself does in practice not have any performance +toll. However, switching debug logging on has a severe performance toll. Also, debug +logging synchronizes much of the code, removing a lot of concurrency and thus +potential race conditions. As such, the very same running instance may behave +very differently with debug logging turned on vs. off. The on-demand debug log +functionality is considered to be very valuable to analyze hard-to-find bugs that +only manifest after a long runtime. Turning debug logging on a failing instance +may reveal the cause of the failure. However, depending on the failure, debug logging +may not even be successfully be turned on. Also note that with this rsyslog version we cannot +obtain any debug information on events that happened <i>before</i> debug logging was +turned on. +<p>If an instance hangs, it is possible to obtain some useful information about the current +threads and their calling stack by sending SIGUSR2. However, the usefulness of that +information is very much depending on rsyslog compile-time settings, must importantly +the --enable-rtinst configure flag. Note that activating this option causes additional overhead +and slows down rsyslgod considerable. So if you do that, you need to check if it is +capable to handle the workload. Also, threading behavior is modified by the +runtime instrumentation. +<p>Sending SIGUSR2 writes new process state information to the log file each time +it is sent. So it may be useful to do that from time to time. It probably is most +useful if the process seems to hang, in which case it may (may!) be able to output +some diagnostic information on the current processing state. In that case, turning +on the mutex debugging options (see above) is probably useful. +<h2>Interpreting the Logs</h2> +<p>Debug logs are primarily meant for rsyslog developers. But they may still provide valuable +information to users. Just be warned that logs sometimes contains informaton the looks like +an error, but actually is none. We put a lot of extra information into the logs, and there +are some cases where it is OK for an error to happen, we just wanted to record it inside +the log. The code handles many cases automatically. So, in short, the log may not make sense to +you, but it (hopefully) makes sense to a developer. Note that we developers often need +many lines of the log file, it is relatively rare that a problem can be diagnosed by +looking at just a couple of (hundered) log records. +<h2>Security Risks</h2> +<p>The debug log will reveal potentially sensible information, including user accounts and +passwords, to anyone able to read the log file. As such, it is recommended to properly +guard access to the log file. Also, an instance running with debug log enabled runs much +slower than one without. An attacker may use this to place carry out a denial-of-service +attack or try to hide some information from the log file. As such, it is suggested to +enable DebugOnDemand mode only for a reason. Note that when no debug mode is enabled, +SIGUSR1 and SIGUSR2 are completely ignored. +<p>When running in any of the debug modes (including on demand mode), an interactive +instance of rsyslogd can be aborted by pressing ctl-c. +<p> <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> -project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer -Gerhards</a> and +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> -</body></html>
\ No newline at end of file +</body> +</html> diff --git a/doc/imklog.html b/doc/imklog.html index 9166bae6..5bfab5ce 100644 --- a/doc/imklog.html +++ b/doc/imklog.html @@ -34,9 +34,9 @@ handled. The default is "off", in which case these messages are ignored. Switch it to on to submit non-kernel messages to rsyslog processing.<span style="font-weight: bold;"></span></li> <li><span style="font-weight: bold;"></span>$DebugPrintKernelSymbols -(imklog) [on/<b>off</b>]<br> +[on/<b>off</b>]<br> Linux only, ignored on other platforms (but may be specified)</li> -<li>$klogSymbolLookup (imklog) [on/<b>off</b>] -- +<li>$klogSymbolLookup [on/<b>off</b>] -- disables imklog kernel symbol translation (former klogd -x option). NOTE that this option is counter-productive on recent kernels (>= 2.6) because the kernel already does the symbol translation and this option breaks the information.<br> @@ -44,10 +44,19 @@ kernel already does the symbol translation and this option breaks the informatio it except if you have a very good reason. If you have one, let us know because otherwise new versions will no longer support it.<br> Linux only, ignored on other platforms (but may be specified)</li> -<li>$klogUseSyscallInterface (imklog) [on/<b>off</b>] +<li><b>$klogConsoleLogLevel</b> [<i>number</i>] +(former klogd -c option) -- sets the console log level. If specified, only messages with +up to the specified level are printed to the console. The default is -1, which means that +the current settings are not modified. To get this behavior, do not specify +$klogConsoleLogLevel in the configuration file. Note that this is a global parameter. Each time +it is changed, the previous definition is re-set. The one activate will be that one that is +active when imklog actually starts processing. In short words: do not specify this +directive more than once! +<br><b>Linux only</b>, ignored on other platforms (but may be specified)</li> +<li><b>$klogUseSyscallInterface</b> [on/<b>off</b>] -- former klogd -s option<br> Linux only, ignored on other platforms (but may be specified)</li> -<li>$klogSymbolsTwice (imklog) [on/<b>off</b>] -- +<li>$klogSymbolsTwice [on/<b>off</b>] -- former klogd -2 option<br> Linux only, ignored on other platforms (but may be specified)<br style="font-weight: bold;"> </li> @@ -69,7 +78,7 @@ is needed to start pulling kernel messages.<br> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2008-2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> diff --git a/doc/imrelp.html b/doc/imrelp.html index 53826ac2..2cf9c1f7 100644 --- a/doc/imrelp.html +++ b/doc/imrelp.html @@ -35,6 +35,9 @@ Starts a RELP server on selected port</li> <b>Caveats/Known Bugs:</b> <ul> <li>see description</li> +<li>To obtain the remote system's IP address, you need to have at least +librelp 1.0.0 installed. Versions below it return the hostname instead +of the IP address.</li> </ul> <p><b>Sample:</b></p> <p>This sets up a RELP server on port 20514.<br> diff --git a/doc/imtcp.html b/doc/imtcp.html index 9ea7efa1..0ccdecc7 100644 --- a/doc/imtcp.html +++ b/doc/imtcp.html @@ -41,17 +41,25 @@ very limited interest in fixing this issue. This directive <b>can not</b> fix th That would require much more code changes, which I was unable to do so far. Full details can be found at the <a href="http://www.rsyslog.com/Article321.phtml">Cisco tcp syslog anomaly</a> page. +<li>$InputTCPServerNotifyOnConnectionClose [on/<b>off</b>] (available since 4.5.5)<br> +instructs imtcp to emit a message if the remote peer closes a connection.<br> +<b>Important:</b> This directive is global to all listeners and must be given right +after loading imtcp, otherwise it may have no effect.</li> <li>$InputTCPServerRun <port><br> Starts a TCP server on selected port</li> -<li><ul><li>$InputTCPMaxSessions <number></li></ul> -Sets the maximum number of sessions supported</li><li>$InputTCPServerStreamDriverMode <number><br> +<li>$InputTCPMaxListeners <number><br> +Sets the maximum number of listeners (server ports) supported. Default is 20. This must be set before the first $InputTCPServerRun directive.</li> +<li>$InputTCPMaxSessions <number><br> +Sets the maximum number of sessions supported. Default is 200. This must be set before the first $InputTCPServerRun directive</li> +<li>$InputTCPServerStreamDriverMode <number><br> Sets the driver mode for the currently selected <a href="netstream.html">network stream driver</a>. <number> is driver specifc.</li> <li>$InputTCPServerInputName <name><br> Sets a name for the inputname property. If no name is set "imtcp" is used by default. Setting a name is not strictly necessary, but can be useful to apply filtering based on which input the message was received from. <li>$InputTCPServerStreamDriverAuthMode <mode-string><br> -Sets the authentication mode for the currently selected <a href="netstream.html">network stream driver</a>. <mode-string> is driver specifc.</li><li>$InputTCPServerStreamDriverPermittedPeer <id-string><br> +Sets the authentication mode for the currently selected <a href="netstream.html">network stream driver</a>. <mode-string> is driver specifc.</li> +<li>$InputTCPServerStreamDriverPermittedPeer <id-string><br> Sets permitted peer IDs. Only these peers are able to connect to the listener. <id-string> semantics depend on the currently selected AuthMode and <a href="netstream.html">network stream driver</a>. PermittedPeers may not be set in anonymous modes.</li> diff --git a/doc/man_rsyslogd.html b/doc/man_rsyslogd.html deleted file mode 100644 index d18fd88a..00000000 --- a/doc/man_rsyslogd.html +++ /dev/null @@ -1,438 +0,0 @@ -<BODY><PRE> -RSYSLOGD(8) Linux System Administration RSYSLOGD(8) - - - -<B>NAME</B> - rsyslogd - reliable and extended syslogd - -<B>SYNOPSIS</B> - <B>rsyslogd </B>[ <B>-4 </B>] [ <B>-6 </B>] [ <B>-A </B>] [ <B>-a </B><I>socket </I>] [ <B>-d </B>] [ <B>-e </B>] - [ <B>-f </B><I>config file </I>] [ <B>-h </B>] [ <B>-i </B><I>pid file </I>] [ <B>-l </B><I>hostlist </I>] - [ <B>-m </B><I>interval </I>] [ <B>-n </B>] [ <B>-o </B>] [ <B>-p </B><I>socket </I>] - [ <B>-r </B><I>[port] </I>] [ <B>-s </B><I>domainlist </I>] [ <B>-t </B><I>port,max-nbr-of-sessions </I>] - [ <B>-v </B>] [ <B>-w </B>] [ <B>-x </B>] - - -<B>DESCRIPTION</B> - <B>Rsyslogd </B>is a system utility providing support for message logging. - Support of both internet and unix domain sockets enables this utility - to support both local and remote logging (via UDP and TCP). - - <B>Rsyslogd</B>(8) is derived from the sysklogd package which in turn is - derived from the stock BSD sources. - - <B>Rsyslogd </B>provides a kind of logging that many modern programs use. - Every logged message contains at least a time and a hostname field, - normally a program name field, too, but that depends on how trusty the - logging program is. The rsyslog package supports free definition of - output formats via templates. It also supports precise timestamps and - writing directly to MySQL databases. If the database option is used, - tools like phpLogCon can be used to view the log data. - - While the <B>rsyslogd </B>sources have been heavily modified a couple of notes - are in order. First of all there has been a systematic attempt to - insure that rsyslogd follows its default, standard BSD behavior. Of - course, some configuration file changes are necessary in order to sup- - port the template system. However, rsyslogd should be able to use a - standard syslog.conf and act like the original syslogd. However, an - original syslogd will not work correctly with a rsyslog-enhanced con- - figuration file. At best, it will generate funny looking file names. - The second important concept to note is that this version of rsyslogd - interacts transparently with the version of syslog found in the stan- - dard libraries. If a binary linked to the standard shared libraries - fails to function correctly we would like an example of the anomalous - behavior. - - The main configuration file <I>/etc/rsyslog.conf </I>or an alternative file, - given with the <B>-f </B>option, is read at startup. Any lines that begin - with the hash mark (‘‘#’’) and empty lines are ignored. If an error - occurs during parsing the error element is ignored. It is tried to - parse the rest of the line. - - For details and configuration examples, see the <B>rsyslog.conf (5) </B>man - page. - - - -<B>OPTIONS</B> - <B>-A </B>When sending UDP messages, there are potentially multiple paths - to the target destination. By default, <B>rsyslogd </B>only sends to - the first target it can successfully send to. If -A is given, - messages are sent to all targets. This may improve reliability, - but may also cause message duplication. This option should - enabled only if it is fully understood. - - <B>-4 </B>Causes <B>rsyslogd </B>to listen to IPv4 addresses only. If neither -4 - nor -6 is given, <B>rsyslogd </B>listens to all configured addresses of - the system. - - <B>-6 </B>Causes <B>rsyslogd </B>to listen to IPv6 addresses only. If neither -4 - nor -6 is given, <B>rsyslogd </B>listens to all configured addresses of - the system. - - <B>-a </B><I>socket</I> - Using this argument you can specify additional sockets from that - <B>rsyslogd </B>has to listen to. This is needed if you’re going to - let some daemon run within a chroot() environment. You can use - up to 19 additional sockets. If your environment needs even - more, you have to increase the symbol <B>MAXFUNIX </B>within the sys- - logd.c source file. An example for a chroot() daemon is - described by the people from OpenBSD at - http://www.psionic.com/papers/dns.html. - - <B>-d </B>Turns on debug mode. Using this the daemon will not proceed a - <B>fork</B>(2) to set itself in the background, but opposite to that - stay in the foreground and write much debug information on the - current tty. See the DEBUGGING section for more information. - - <B>-e </B>Set the default of $RepeatedMsgReduction config option to "off". - Hine: "e" like "every message". For further information, see - there. - - <B>-f </B><I>config file</I> - Specify an alternative configuration file instead of <I>/etc/rsys-</I> - <I>log.conf</I>, which is the default. - - <B>-h </B>By default rsyslogd will not forward messages it receives from - remote hosts. Specifying this switch on the command line will - cause the log daemon to forward any remote messages it receives - to forwarding hosts which have been defined. - - <B>-i </B><I>pid file</I> - Specify an alternative pid file instead of the default one. - This option must be used if multiple instances of rsyslogd - should run on a single machine. - - <B>-l </B><I>hostlist</I> - Specify a hostname that should be logged only with its simple - hostname and not the fqdn. Multiple hosts may be specified - using the colon (‘‘:’’) separator. - - <B>-m </B><I>interval</I> - The <B>rsyslogd </B>logs a mark timestamp regularly. The default - <I>interval </I>between two <I>-- MARK -- </I>lines is 20 minutes. This can - be changed with this option. Setting the <I>interval </I>to zero turns - it off entirely. - - <B>-n </B>Avoid auto-backgrounding. This is needed especially if the - <B>rsyslogd </B>is started and controlled by <B>init</B>(8). - - <B>-o </B>Omit reading the standard local log socket. This option is most - useful for running multiple instances of rsyslogd on a single - machine. When specified, no local log socket is opened at all. - - <B>-p </B><I>socket</I> - You can specify an alternative unix domain socket instead of - <I>/dev/log</I>. - - <B>-r </B><I>["port"]</I> - Activates the syslog/udp listener service. The listener will - listen to the specified port. If no port is specified, 0 is - used as port number, which in turn will lead to a lookup of the - system default syslog port. If there is no system default, 514 - is used. Please note that the port must immediately follow the - -r option. Thus "-r514" is valid while "-r 514" is invalid (note - the space). - - <B>-s </B><I>domainlist</I> - Specify a domainname that should be stripped off before logging. - Multiple domains may be specified using the colon (‘‘:’’) sepa- - rator. Please be advised that no sub-domains may be specified - but only entire domains. For example if <B>-s north.de </B>is speci- - fied and the host logging resolves to satu.infodrom.north.de no - domain would be cut, you will have to specify two domains like: - <B>-s north.de:infodrom.north.de</B>. - - <B>-t </B><I>port,max-nbr-of-sessions</I> - Activates the syslog/tcp listener service. The listener will - listen to the specified port. If max-nbr-of-sessions is speci- - fied, that becomes the maximum number of concurrent tcp ses- - sions. If not specified, the default is 200. Please note that - syslog/tcp is not standardized, but the implementation in rsys- - logd follows common practice and is compatible with e.g. Cisco - PIX, syslog-ng and MonitorWare (Windows). Please note that the - port must immediately follow the -t option. Thus "-t514" is - valid while "-t 514" is invalid (note the space). - - <B>-v </B>Print version and exit. - - <B>-w </B>Supress warnings issued when messages are received from non- - authorized machines (those, that are in no AllowedSender list). - - <B>-x </B>Disable DNS for remote messages. - - -<B>SIGNALS</B> - <B>Rsyslogd </B>reacts to a set of signals. You may easily send a signal to - <B>rsyslogd </B>using the following: - - kill -SIGNAL ‘cat /var/run/rsyslogd.pid‘ - - - <B>SIGHUP </B>This lets <B>rsyslogd </B>perform a re-initialization. All open files - are closed, the configuration file (default is <I>/etc/rsys-</I> - <I>log.conf</I>) will be reread and the <B>rsyslog</B>(3) facility is started - again. - - <B>SIGTERM</B> - <B>Rsyslogd </B>will die. - - <B>SIGINT</B>, <B>SIGQUIT</B> - If debugging is enabled these are ignored, otherwise <B>rsyslogd</B> - will die. - - <B>SIGUSR1</B> - Switch debugging on/off. This option can only be used if <B>rsys-</B> - <B>logd </B>is started with the <B>-d </B>debug option. - - <B>SIGCHLD</B> - Wait for childs if some were born, because of wall’ing messages. - - -<B>SUPPORT FOR REMOTE LOGGING</B> - <B>Rsyslogd </B>provides network support to the syslogd facility. Network - support means that messages can be forwarded from one node running - rsyslogd to another node running rsyslogd (or a compatible syslog - implementation) where they will be actually logged to a disk file. - - To enable this you have to specify either the <B>-r </B>or <B>-t </B>option on the - command line. The default behavior is that <B>rsyslogd </B>won’t listen to - the network. You can also combine these two options if you want rsys- - logd to listen to both TCP and UDP messages. - - The strategy is to have rsyslogd listen on a unix domain socket for - locally generated log messages. This behavior will allow rsyslogd to - inter-operate with the syslog found in the standard C library. At the - same time rsyslogd listens on the standard syslog port for messages - forwarded from other hosts. To have this work correctly the <B>ser-</B> - <B>vices</B>(5) files (typically found in <I>/etc</I>) must have the following entry: - - syslog 514/udp - - If this entry is missing <B>rsyslogd </B>will use the well known port of 514 - (so in most cases, it’s not really needed). - - To cause messages to be forwarded to another host replace the normal - file line in the <I>rsyslog.conf </I>file with the name of the host to which - the messages is to be sent prepended with an @ (for UDP delivery) or - the sequence @@ (for TCP delivery). The host name can also be followed - by a colon and a port number, in which case the message is sent to the - specified port on the remote host. - - For example, to forward <B>ALL </B>messages to a remote host use the - following <I>rsyslog.conf </I>entry: - - # Sample rsyslogd configuration file to - # messages to a remote host forward all. - *.* @hostname - More samples can be found in sample.conf. - - If the remote hostname cannot be resolved at startup, because - the name-server might not be accessible (it may be started after - rsyslogd) you don’t have to worry. <B>Rsyslogd </B>will retry to - resolve the name ten times and then complain. Another possibil- - ity to avoid this is to place the hostname in <I>/etc/hosts</I>. - - With normal <B>syslogd</B>s you would get syslog-loops if you send out - messages that were received from a remote host to the same host - (or more complicated to a third host that sends it back to the - first one, and so on). - - To avoid this no messages that were received from a remote host - are sent out to another (or the same) remote host. You can dis- - able this feature by the <B>-h </B>option. - - If the remote host is located in the same domain as the host, - <B>rsyslogd </B>is running on, only the simple hostname will be logged - instead of the whole fqdn. - - In a local network you may provide a central log server to have - all the important information kept on one machine. If the net- - work consists of different domains you don’t have to complain - about logging fully qualified names instead of simple hostnames. - You may want to use the strip-domain feature <B>-s </B>of this server. - You can tell <B>rsyslogd </B>to strip off several domains other than - the one the server is located in and only log simple hostnames. - - Using the <B>-l </B>option there’s also a possibility to define single - hosts as local machines. This, too, results in logging only - their simple hostnames and not the fqdns. - - -<B>OUTPUT TO DATABASES</B> - <B>Rsyslogd </B>has support for writing data to MySQL database tables. The - exact specifics are described in the <B>rsyslog.conf (5) </B>man page. Be sure - to read it if you plan to use database logging. - - While it is often handy to have the data in a database, you must be - aware of the implications. Most importantly, database logging takes far - longer than logging to a text file. A system that can handle a large - log volume when writing to text files can most likely not handle a sim- - ilar large volume when writing to a database table. - - -<B>OUTPUT TO NAMED PIPES (FIFOs)</B> - <B>Rsyslogd </B>has support for logging output to named pipes (fifos). A fifo - or named pipe can be used as a destination for log messages by prepend- - ing a pipy symbol (‘‘|’’) to the name of the file. This is handy for - debugging. Note that the fifo must be created with the mkfifo command - before <B>rsyslogd </B>is started. - - The following configuration file routes debug messages from the - kernel to a fifo: - - # Sample configuration to route kernel debugging - # messages ONLY to /usr/adm/debug which is a - # named pipe. - kern.=debug |/usr/adm/debug - - -<B>INSTALLATION CONCERNS</B> - There is probably one important consideration when installing rsyslogd. - It is dependent on proper formatting of messages by the syslog func- - tion. The functioning of the syslog function in the shared libraries - changed somewhere in the region of libc.so.4.[2-4].n. The specific - change was to null-terminate the message before transmitting it to the - <I>/dev/log </I>socket. Proper functioning of this version of rsyslogd is - dependent on null-termination of the message. - - This problem will typically manifest itself if old statically linked - binaries are being used on the system. Binaries using old versions of - the syslog function will cause empty lines to be logged followed by the - message with the first character in the message removed. Relinking - these binaries to newer versions of the shared libraries will correct - this problem. - - The <B>rsyslogd</B>(8) can be run from <B>init</B>(8) or started as part of the rc.* - sequence. If it is started from init the option <I>-n </I>must be set, other- - wise you’ll get tons of syslog daemons started. This is because - <B>init</B>(8) depends on the process ID. - - -<B>SECURITY THREATS</B> - There is the potential for the rsyslogd daemon to be used as a conduit - for a denial of service attack. A rogue program(mer) could very easily - flood the rsyslogd daemon with syslog messages resulting in the log - files consuming all the remaining space on the filesystem. Activating - logging over the inet domain sockets will of course expose a system to - risks outside of programs or individuals on the local machine. - - There are a number of methods of protecting a machine: - - 1. Implement kernel firewalling to limit which hosts or networks - have access to the 514/UDP socket. - - 2. Logging can be directed to an isolated or non-root filesystem - which, if filled, will not impair the machine. - - 3. The ext2 filesystem can be used which can be configured to limit - a certain percentage of a filesystem to usage by root only. - <B>NOTE </B>that this will require rsyslogd to be run as a non-root - process. <B>ALSO NOTE </B>that this will prevent usage of remote log- - ging since rsyslogd will be unable to bind to the 514/UDP - socket. - - 4. Disabling inet domain sockets will limit risk to the local - machine. - - 5. Use step 4 and if the problem persists and is not secondary to a - rogue program/daemon get a 3.5 ft (approx. 1 meter) length of - sucker rod* and have a chat with the user in question. - - Sucker rod def. — 3/4, 7/8 or 1in. hardened steel rod, male - threaded on each end. Primary use in the oil industry in West- - ern North Dakota and other locations to pump ’suck’ oil from oil - wells. Secondary uses are for the construction of cattle feed - lots and for dealing with the occasional recalcitrant or bel- - ligerent individual. - - <B>Message replay and spoofing</B> - If remote logging is enabled, messages can easily be spoofed and - replayed. As the messages are transmitted in clear-text, an attacker - might use the information obtained from the packets for malicious - things. Also, an attacker might reply recorded messages or spoof a - sender’s IP address, which could lead to a wrong perception of system - activity. Be sure to think about syslog network security before - enabling it. - - -<B>DEBUGGING</B> - When debugging is turned on using <B>-d </B>option then <B>rsyslogd </B>will be very - verbose by writing much of what it does on stdout. Whenever the con- - figuration file is reread and re-parsed you’ll see a tabular, corre- - sponding to the internal data structure. This tabular consists of four - fields: - - <I>number </I>This field contains a serial number starting by zero. This num- - ber represents the position in the internal data structure (i.e. - the array). If one number is left out then there might be an - error in the corresponding line in <I>/etc/rsyslog.conf</I>. - - <I>pattern</I> - This field is tricky and represents the internal structure - exactly. Every column stands for a facility (refer to <B>sys-</B> - <B>log</B>(3)). As you can see, there are still some facilities left - free for former use, only the left most are used. Every field - in a column represents the priorities (refer to <B>syslog</B>(3)). - - <I>action </I>This field describes the particular action that takes place - whenever a message is received that matches the pattern. Refer - to the <B>syslog.conf</B>(5) manpage for all possible actions. - - <I>arguments</I> - This field shows additional arguments to the actions in the last - field. For file-logging this is the filename for the logfile; - for user-logging this is a list of users; for remote logging - this is the hostname of the machine to log to; for console-log- - ging this is the used console; for tty-logging this is the spec- - ified tty; wall has no additional arguments. - - - <B>templates</B> - There will also be a second internal structure which lists all - defined templates and there contents. This also enables you to - see the internally-defined, hardcoded templates. - -<B>FILES</B> - <I>/etc/rsyslog.conf</I> - Configuration file for <B>rsyslogd</B>. See <B>rsyslog.conf</B>(5) for exact - information. - <I>/dev/log</I> - The Unix domain socket to from where local syslog messages are - read. - <I>/var/run/rsyslogd.pid</I> - The file containing the process id of <B>rsyslogd</B>. - -<B>BUGS</B> - Please review the file BUGS for up-to-date information on known bugs - and annoyances. - -<B>Further Information</B> - Please visit <B>http://www.rsyslog.com/doc </B>for additional information, - tutorials and a support forum. - -<B>SEE ALSO</B> - <B>rsyslog.conf</B>(5), <B>logger</B>(1), <B>syslog</B>(2), <B>syslog</B>(3), <B>services</B>(5), - <B>savelog</B>(8) - - -<B>COLLABORATORS</B> - <B>rsyslogd </B>is derived from sysklogd sources, which in turn was taken from - the BSD sources. Special thanks to Greg Wettstein (greg@wind.enjel- - lic.com) and Martin Schulze (joey@linux.de) for the fine sysklogd pack- - age. - - Rainer Gerhards - Adiscon GmbH - Grossrinderfeld, Germany - rgerhards@adiscon.com - - Michael Meckelein - Adiscon GmbH - mmeckelein@adiscon.com - - - -Version 1.16.1 (devel) 17 July 2007 RSYSLOGD(8) -</PRE></BODY> diff --git a/doc/manual.html b/doc/manual.html index 5d43cb25..aff23c8b 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -14,13 +14,13 @@ replacement. Its <a href="features.html"> advanced features</a> make it suitable for enterprise-class, <a href="rsyslog_tls.html">encryption protected syslog</a> relay chains while at the same time being very easy to setup for the novice user. And as we know what enterprise users really need, there is -also <a href="professional_support.html">professional +also <a href="http://www.rsyslog.com/professional-services">professional rsyslog support</a> available directly from the source!</p> <p><b>Please visit the <a href="http://www.rsyslog.com/sponsors">rsyslog sponsor's page</a> to honor the project sponsors or become one yourself!</b> We are very grateful for any help towards the project goals.</p> -<p><b>This documentation is for version 4.4.2 (v4-stable) of rsyslog.</b> -Visit the <i> <a href="http://www.rsyslog.com/doc-status.html">rsyslog status page</a></i></b> to obtain current +<p><b>This documentation is for version 4.6.4 (v4-stable branch) of rsyslog.</b> +Visit the <i> <a href="http://www.rsyslog.com/status">rsyslog status page</a></i></b> to obtain current version information and project status. </p><p><b>If you like rsyslog, you might want to lend us a helping hand. </b>It doesn't require a lot of @@ -28,12 +28,13 @@ time - even a single mouse click helps. Learn <a href="how2help.html">how to hel Due to popular demand, there is now a <a href="rsyslog_ng_comparison.html">side-by-side comparison between rsyslog and syslog-ng</a>.</p> <p>If you are upgrading from rsyslog v2 or stock sysklogd, -<a href="v3compatibility.html">be -sure to read the rsyslog v3 compatibility document!</a> It will work even +<a href="v3compatibility.html">be sure to read the rsyslog v3 compatibility document</a>, +and if you are upgrading from v3, read the +<a href="v4compatibility.html">rsyslog v4 compatibility document</a>. +<p>Rsyslog will work even if you do not read the doc, but doing so will definitely improve your experience.</p> -<p><span style="font-weight: bold;"></span><b>Follow -the links below for the</b><br></p><ul> - +<p><b>Follow the links below for the</b></p> +<ul> <li><a href="troubleshoot.html">troubleshooting rsyslog problems</a></li> <li><a href="rsyslog_conf.html">configuration file syntax (rsyslog.conf)</a></li> <li><a href="http://www.rsyslog.com/tool-regex">a regular expression checker/generator tool for rsyslog</a></li> @@ -42,8 +43,9 @@ the links below for the</b><br></p><ul> <li><a href="bugs.html">rsyslog bug list</a></li> <li><a href="rsyslog_packages.html"> rsyslog packages</a></li> <li><a href="generic_design.html">backgrounder on -generic syslog application design</a><!-- not good as it currently is ;) <li><a href="contributors.html">contributor "Hall of Fame"</a>--></li> +generic syslog application design</a> <li><a href="modules.html">description of rsyslog modules</a></li> +<li><a href="http://cookbook.rsyslog.com">the rsyslog "cookbook"</a> - a set of configurations ready to use</li> </ul> <p><b>We have some in-depth papers on</b></p> <ul> @@ -51,8 +53,10 @@ generic syslog application design</a><!-- not good as it currently is ;) <li><a <li><a href="build_from_repo.html">obtaining rsyslog from the source repository</a></li> <li><a href="ipv6.html">rsyslog and IPv6</a> (which is fully supported)</li> <li><a href="rsyslog_secure_tls.html">native TLS encryption for syslog</a></li> +<li><a href="multi_ruleset.html">using multiple rule sets in rsyslog</a></li> <li><a href="rsyslog_stunnel.html">ssl-encrypting syslog with stunnel</a></li> <li><a href="rsyslog_mysql.html">writing syslog messages to MySQL (and other databases as well)</a></li> +<li><a href="rsyslog_pgsql.html">writing syslog messages to PostgreSQL (and other databases as well)</a></li> <li><a href="rsyslog_high_database_rate.html">writing massive amounts of syslog messages to a database</a></li> <li><a href="rsyslog_reliable_forwarding.html">reliable forwarding to a remote server</a></li> <li><a href="rsyslog_php_syslog_ng.html">using diff --git a/doc/multi_ruleset.html b/doc/multi_ruleset.html new file mode 100644 index 00000000..8d8c614f --- /dev/null +++ b/doc/multi_ruleset.html @@ -0,0 +1,275 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<title>Multiple Rulesets in rsyslog</title></head> +<body> +<h1>Multiple Rulesets in rsyslog</h1> +<p>Starting with version 4.5.0 and 5.1.1, <a href="http://www.rsyslog.com">rsyslog</a> supports +multiple rulesets within a single configuration. +This is especially useful for routing the recpetion of remote messages to a set of specific rules. +Note that the input module must support binding to non-standard rulesets, so the functionality +may not be available with all inputs. +<p>In this document, I am using <a href="imtcp.html">imtcp</a>, an input module +that supports binding to non-standard rulesets since rsyslog started to support them. +<h2>What is a Ruleset?</h2> +If you have worked with (r)syslog.conf, you know that it is made up of what I call rules (others +tend to call them selectors, a sysklogd term). Each rule consist of a filter and one or more +actions to be carried out when the filter evaluates to true. A filter may be as simple as a +traditional +syslog priority based filter (like "*.*" or "mail.info" or a as complex as a +script-like expression. Details on that are covered in the config file documentation. After the +filter come action specifiers, and an action is something that does something to a message, e.g. +write it to a file or forward it to a remote logging server. + +<p>A traditional configuration file is made up of one or more of these rules. When a new +message arrives, its processing starts with the first rule (in order of appearance in +rsyslog.conf) and continues for each rule until either all rules have been processed or +a so-called "e;discard" action happens, in which case processing stops and the +message is thrown away (what also happens after the last rule has been processed). + +<p>The <b>multi-ruleset</b> support now permits to specify more than one such rule sequence. +You can think of a traditional config file just as a single default rule set, which is +automatically bound to each of the inputs. This is even what actually happens. When +rsyslog.conf is processed, the config file parser looks for the directive + +<pre>$RuleSet <name> +</pre> + +<p>Where name is any name the user likes (but must not start with "RSYSLOG_", which +is the name space reserved for rsyslog use). If it finds this directive, it begins a new +rule set (if the name was not yet know) or switches to an already-existing one (if the name +was known). All rules defined between this $RuleSet directive and the next one are appended +to the named ruleset. Note that the reserved name "RSYSLOG_DefaultRuleset" is used to +specify rsyslogd's default ruleset. You can use that name whereever you can use a ruleset name, +including when binding an input to it. + +<p>Inside a ruleset, messages are processed as described above: they start with the first rule +and rules are processed in the order of appearance of the configuration file until either +there are no more rules or the discard action is executed. Note that with multiple rulesets +no longer <b>all</b> rsyslog.conf rules are executed but <b>only</b> those that are +contained within the specific ruleset. + +<p>Inputs must explicitely bind to rulesets. If they don't do, the default ruleset is bound. + +<p>This brings up the next question: + +<h2>What does "To bind to a Ruleset" mean?</h2> +<p>This term is used in the same sense as "to bind an IP address to an interface": +it means that a specific input, or part of an input (like a tcp listener) will use a specific +ruleset to "pass its messages to". So when a new message arrives, it will be processed +via the bound ruleset. Rule from all other rulesets are irrelevant and will never be processed. +<p>This makes multiple rulesets very handy to process local and remote message via +seperate means: bind the respective receivers to different rule sets, and you do not need +to seperate the messages by any other method. + +<p>Binding to rulesets is input-specifc. For imtcp, this is done via the + +<pre>$InputTCPServerBindRuleset <name> +</pre> + +directive. Note that "name"e; must be the name of a ruleset that is already defined +at the time the bind directive is given. There are many ways to make sure this happens, but +I personally think that it is best to define all rule sets at the top of rsyslog.conf and +define the inputs at the bottom. This kind of reverses the traditional recommended ordering, but +seems to be a really useful and straightforward way of doing things. +<h2>Can I use a different Ruleset as the default?</h2> +<p>This is possible by using the + +<pre>$DefaultRuleset <name> +</pre> + +Directive. Please note, however, that this directive is actually global: that is, it does not +modify the ruleset to which the next input is bound but rather provides a system-wide +default rule set for those inputs that did not explicitly bind to one. As such, the directive +can not be used as a work-around to bind inputs to non-default rulesets that do not support +ruleset binding. +<h2>Examples</h2> +<h3>Split local and remote logging</h3> +<p>Let's say you have a pretty standard system that logs its local messages to the usual +bunch of files that are specified in the default rsyslog.conf. As an example, your rsyslog.conf +might look like this: + +<pre> +# ... module loading ... +# The authpriv file has restricted access. +authpriv.* /var/log/secure +# Log all the mail messages in one place. +mail.* /var/log/maillog +# Log cron stuff +cron.* /var/log/cron +# Everybody gets emergency messages +*.emerg * +... more ... +</pre> + +<p>Now, you want to add receive messages from a remote system and log these to +a special file, but you do not want to have these messages written to the files +specified above. The traditional approach is to add a rule in front of all others that +filters on the message, processes it and then discards it: + +<pre> +# ... module loading ... +# process remote messages +:fromhost-ip, isequal, "192.0.2.1" /var/log/remotefile +& ~ +# only messages not from 192.0.21 make it past this point + +# The authpriv file has restricted access. +authpriv.* /var/log/secure +# Log all the mail messages in one place. +mail.* /var/log/maillog +# Log cron stuff +cron.* /var/log/cron +# Everybody gets emergency messages +*.emerg * +... more ... +</pre> + +<p>Note the tilde character, which is the discard action!. Also note that we assume that +192.0.2.1 is the sole remote sender (to keep it simple). + +<p>With multiple rulesets, we can simply define a dedicated ruleset for the remote reception +case and bind it to the receiver. This may be written as follows: + +<pre> +# ... module loading ... +# process remote messages +# define new ruleset and add rules to it: +$RuleSet remote +*.* /var/log/remotefile +# only messages not from 192.0.21 make it past this point + +# bind ruleset to tcp listener +$InputTCPServerBindRuleset remote +# and activate it: +$InputTCPServerRun 10514 + +# switch back to the default ruleset: +$RuleSet RSYSLOG_DefaultRuleset +# The authpriv file has restricted access. +authpriv.* /var/log/secure +# Log all the mail messages in one place. +mail.* /var/log/maillog +# Log cron stuff +cron.* /var/log/cron +# Everybody gets emergency messages +*.emerg * +... more ... +</pre> + +<p>Here, we need to switch back to the default ruleset after we have defined our custom +one. This is why I recommend a different ordering, which I find more intuitive. The sample +below has it, and it leads to the same results: + +<pre> +# ... module loading ... +# at first, this is a copy of the unmodified rsyslog.conf +# The authpriv file has restricted access. +authpriv.* /var/log/secure +# Log all the mail messages in one place. +mail.* /var/log/maillog +# Log cron stuff +cron.* /var/log/cron +# Everybody gets emergency messages +*.emerg * +... more ... +# end of the "regular" rsyslog.conf. Now come the new definitions: + +# process remote messages +# define new ruleset and add rules to it: +$RuleSet remote +*.* /var/log/remotefile + +# bind ruleset to tcp listener +$InputTCPServerBindRuleset remote +# and activate it: +$InputTCPServerRun 10514 +</pre> + +<p>Here, we do not switch back to the default ruleset, because this is not needed as it is +completely defined when we begin the "remote" ruleset. + +<p>Now look at the examples and compare them to the single-ruleset solution. You will notice +that we do <b>not</b> need a real filter in the multi-ruleset case: we can simply use +"*.*" as all messages now means all messages that are being processed by this +rule set and all of them come in via the TCP receiver! This is what makes using multiple +rulesets so much easier. + +<h3>Split local and remote logging for three different ports</h3> +<p>This example is almost like the first one, but it extends it a little bit. While it is +very similar, I hope it is different enough to provide a useful example why you may want +to have more than two rulesets. + +<p>Again, we would like to use the "regular" log files for local logging, only. But +this time we set up three syslog/tcp listeners, each one listening to a different +port (in this example 10514, 10515, and 10516). Logs received from these receivers shall go into +different files. Also, logs received from 10516 (and only from that port!) with +"mail.*" priority, shall be written into a specif file and <b>not</b> be +written to 10516's general log file. + +<p>This is the config: + +<pre> +# ... module loading ... +# at first, this is a copy of the unmodified rsyslog.conf +# The authpriv file has restricted access. +authpriv.* /var/log/secure +# Log all the mail messages in one place. +mail.* /var/log/maillog +# Log cron stuff +cron.* /var/log/cron +# Everybody gets emergency messages +*.emerg * +... more ... +# end of the "regular" rsyslog.conf. Now come the new definitions: + +# process remote messages + +#define rulesets first +$RuleSet remote10514 +*.* /var/log/remote10514 + +$RuleSet remote10515 +*.* /var/log/remote10515 + +$RuleSet remote10516 +mail.* /var/log/mail10516 +& ~ +# note that the discard-action will prevent this messag from +# being written to the remote10516 file - as usual... +*.* /var/log/remote10516 + +# and now define listners bound to the relevant ruleset +$InputTCPServerBindRuleset remote10514 +$InputTCPServerRun 10514 + +$InputTCPServerBindRuleset remote10515 +$InputTCPServerRun 10515 + +$InputTCPServerBindRuleset remote10516 +$InputTCPServerRun 10516 +</pre> + +<p>Note that the "mail.*" rule inside the "remote10516"e; ruleset does +not affect processing inside any other rule set, including the default rule set. + + +<h2>Performance</h2> +<p>No rule processing can be faster than not processing a rule at all. As such, it is useful +for a high performance system to identify disjunct actions and try to split these off to +different rule sets. In the example section, we had a case where three different tcp listeners +need to write to three different files. This is a perfect example of where multiple rule sets +are easier to use and offer more performance. The performance is better simply because there +is no need to check the reception service - instead messages are automatically pushed to the +right rule set and can be processed by very simple rules (maybe even with +"*.*"-filters, the fastest ones available). + +<p>In the long term, multiple rule sets will probably lay the foundation for even better +optimizations. So it is not a bad idea to get aquainted with them. + +<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/omoracle.html b/doc/omoracle.html index 40f6360f..cfcf277f 100644 --- a/doc/omoracle.html +++ b/doc/omoracle.html @@ -67,6 +67,128 @@ it is suggested to post questions to the need to define the properties on the template in the correct order you want them passed to the statement! </pre> +<p>Some additional documentation contributed by Ronny Egner: +<pre> +REQUIREMENTS: +-------------- + +- Oracle Instantclient 10g (NOT 11g) Base + Devel + (if you´re on 64-bit linux you should choose the 64-bit libs!) +- JDK 1.6 (not neccessary for oracle plugin but "make" didd not finsished successfully without it) + +- "oracle-instantclient-config" script + (seems to shipped with instantclient 10g Release 1 but i was unable to find it for 10g Release 2 so here it is) + + +====================== /usr/local/bin/oracle-instantclient-config ===================== +#!/bin/sh +# +# Oracle InstantClient SDK config file +# Jean-Christophe Duberga - Bordeaux 2 University +# + +# just adapt it to your environment +incdirs="-I/usr/include/oracle/10.2.0.4/client64" +libdirs="-L/usr/lib/oracle/10.2.0.4/client64/lib" + +usage="\ +Usage: oracle-instantclient-config [--prefix[=DIR]] [--exec-prefix[=DIR]] [--version] [--cflags] [--libs] [--static-libs]" + +if test $# -eq 0; then + echo "${usage}" 1>&2 + exit 1 +fi + +while test $# -gt 0; do + case "$1" in + -*=*) optarg=`echo "$1" | sed 's/[-_a-zA-Z0-9]*=//'` ;; + *) optarg= ;; + esac + + case $1 in + --prefix=*) + prefix=$optarg + if test $exec_prefix_set = no ; then + exec_prefix=$optarg + fi + ;; + --prefix) + echo $prefix + ;; + --exec-prefix=*) + exec_prefix=$optarg + exec_prefix_set=yes + ;; + --exec-prefix) + echo ${exec_prefix} + ;; + --version) + echo ${version} + ;; + --cflags) + echo ${incdirs} + ;; + --libs) + echo $libdirs -lclntsh -lnnz10 -locci -lociei -locijdbc10 + ;; + --static-libs) + echo "No static libs" 1>&2 + exit 1 + ;; + *) + echo "${usage}" 1>&2 + exit 1 + ;; + esac + shift +done + +=============== END ============== + + + + +COMPILING RSYSLOGD +------------------- + + +./configure --enable-oracle + + + + +RUNNING +------- + +- make sure rsyslogd is able to locate the oracle libs (either via LD_LIBRARY_PATH or /etc/ld.so.conf) +- set TNS_ADMIN to point to your tnsnames.ora +- create a tnsnames.ora and test you are able to connect to the database + +- create user in oracle as shown in the following example: + create user syslog identified by syslog default tablespace users quota unlimited on users; + grant create session to syslog; + create role syslog_role; + grant syslog_role to syslog; + grant create table to syslog_role; + grant create sequence to syslog_role; + +- create tables as needed + +- configure rsyslog as shown in the following example + $ModLoad omoracle + + $OmoracleDBUser syslog + $OmoracleDBPassword syslog + $OmoracleDB syslog + $OmoracleBatchSize 1 + $OmoracleBatchItemSize 4096 + + $OmoracleStatementTemplate OmoracleStatement + $template OmoracleStatement,"insert into foo(hostname,message) values (:host,:message)" + $template TestStmt,"%hostname%%msg%" + *.* :omoracle:;TestStmt + (you guess it: username = password = database = "syslog".... see $rsyslogd_source/plugins/omoracle/omoracle.c for me info) +</pre> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the diff --git a/doc/professional_support.html b/doc/professional_support.html deleted file mode 100644 index 7724ede8..00000000 --- a/doc/professional_support.html +++ /dev/null @@ -1,112 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head> -<meta http-equiv="Content-Language" content="en"><title>Professional Support for Rsyslog</title> - -</head> -<body> -<h1>Professional Services for Rsyslog</h1> -<p>Professional services are being offered by <a href="http://www.adiscon.com">Adiscon</a>, the company -that sponsors rsyslog development. For details, please contact <a href="mailto:info%40adiscon.com">Adiscon Sales</a>. </p> - -<h3>EMail Support Service</h3> -Price: 99.00 EURO <br> -Duration: 180 days -<br> -Support level: 8x5 -<p>Purchase rsyslog support directly from the source. This -contract provides priority email support. It is a great option if you -need to provide proof of software support in your organization. This -contract provides</p> -<ul> -<li>unlimited email support tickets during validity -</li> -<li><span style="font-weight: bold;">fixes for</span> -current and <span style="font-weight: bold;">past rsyslog -releases</span> -</li> -<li>advise on how to implement rsyslog in the best possible -way. -</li> -</ul> -<p>Under this contract, fixes for old rsyslog releases will be -provided / created, provided that it is possible to do that with the -code base in question. Phone support is not included.</p> -<h3>Custom-Written Config File</h3> -Price: 29.00 EURO -<br> -Duration: N/A -<br> -Support level: 8x5 -<p>Creating rsyslog config files is easy - but if you would like -to have that extra feature and have no time to do it, this service is -for you. Important: BEFORE you purchase this service, contact us and -inquire (via <a href="mailto:info%40adiscon.com">info@adiscon.com</a>) -whether or not your desired result can be achieved via rsyslog. Once -this is clear, order the service and we will ship a custom-made -configuration file within 5 working days (at latest, most often much -faster). For security reasons, we will not put passwords into the -configuration file, but will place easy to read comments in the places -where you need to put them in. The agreement is governed under German -law. You may also purchase this service if you would like to have your -own configuration file reviewed, e.g. for auditing purposes.</p> -<h3>Local Installation Support</h3> -<p>If you intend to install rsyslog on your system but would like -to do so with minimal effort and according to your specification, you -can ask us to perform the installation for you. You get a perfect -installation, exactly like you needed, but without a need to -touch the system. This is a perfect choice for the busy administrator! -<p>In order to perform this work, we just need ssh access to your -system and the proper permissions. -<p>We charge a low one-time fee for this service. For details, please -contact <a href="mailto:info@adiscon.com">info@adiscon.com</a>. -<h3>Local Installation Maintenance</h3> -<p>If you used our services to set up the system, why not keep it -running perfectly with maintenance support? Under this contract, we -assure you run a recent build that does not interfere with your -environment and we even carry out change requests you may have. So this -is a hassle-free, everything cared about solution. -<p>Again, all we need to have is ssh access and the proper permissions -to your machine. Of course, work will only be carried out when you -expect us to do so. You are always in control of what happens. This -is a perfect outsourcing solution for those who would like to run -a great logging system but can not afford the time to keep it -in perfect shape! -<p>We charge a low monthly fee for this service. For details, please -contact <a href="mailto:info@adiscon.com">info@adiscon.com</a>. -<h3>Custom Development</h3> -<p>Do you need an exotic feature that otherwise would not be implemented? -Do you need something really quick, quicker than it is available via -the regular development schedule? Then, you may consider funding -development for a specific functionality. We are always looking for -interesting projects. If you hire us to to do the job, you can be sure -to get the best possible and probably quickest solution, because we are -obviously at the heart of the source code. No need to get aquainted to -anything, no risk of misunderstanding program concepts. Benefit from -our vast syslog experience.</p> -<p>Please note that custom development is not limited to rsyslog. We offer -a number of logging solutions and can also work as part of your time -for specific requirements. The opportunities are endless, just ask. We -will work with you on your requirements and provide a quote on the -estimated cost. Just write to <a href="mailto:sales@adiscon.com">sales@adiscon.com</a> for details.</p><h3>Consulting Services</h3> -<p>Do you have demanding logging requirements? Why not talk to a -real logging professional? Instead of trying to find the solution -like a needle in the haystack, talk to the team that brought rsyslog, -phpLogCon, the Windows MonitorWare products and other logging -solutions. We sweat logging for over 15 years now and can help quickly. -Depending on your needs, consulting can be carried out via email, the -phone or on your premises (for larger or local projects). Everything is -possible, it just depends on your needs. Consulting services are -available in English and German. Just mail <a href="mailto:sales@adiscon.com">sales@adiscon.com</a> what you are interested in and we will work with you on a proposal that fits your needs. -</p><p></p><p>All agreements are -governed under German law. -</p> - -<p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> -<p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> -project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer -Gerhards</a> and -<a href="http://www.adiscon.com/">Adiscon</a>. -Released under the GNU GPL version 3 or higher.</font></p> -</body></html> diff --git a/doc/property_replacer.html b/doc/property_replacer.html index 7b604ea0..4d242a34 100644 --- a/doc/property_replacer.html +++ b/doc/property_replacer.html @@ -335,6 +335,19 @@ Especially useful for PIX.</td> <td>format as RFC 3164 date</td> </tr> <tr> +<tr> +<td valign="top"><b>date-rfc3164-buggyday</b></td> +<td>similar to date-rfc3164, but emulates a common coding error: RFC 3164 demands +that a space is written for single-digit days. With this option, a zero is +written instead. This format seems to be used by syslog-ng and the +date-rfc3164-buggyday option can be used in migration scenarios where otherwise +lots of scripts would need to be adjusted. It is recommended <i>not</i> to use this +option when forwarding to remote hosts - they may treat the date as invalid +(especially when parsing strictly according to RFC 3164).</td> +<br><i>This feature was introduced in rsyslog 4.6.2 and v4 versions above and +5.5.3 and all versions above.</i> +</tr> +<tr> <td><b>date-rfc3339</b></td> <td>format as RFC 3339 date</td> </tr> diff --git a/doc/queues.html b/doc/queues.html index 4a9509a0..45ce1bd1 100644 --- a/doc/queues.html +++ b/doc/queues.html @@ -115,7 +115,11 @@ isolation. This is currently selected by specifying different <i>$WorkDirectory< config directives before the queue creation statement.</p> <p>To create a disk queue, use the "<i>$<object>QueueType Disk</i>" config directive. Checkpoint intervals can be specified via "<i>$<object>QueueCheckpointInterval</i>", -with 0 meaning no checkpoints. </p> +with 0 meaning no checkpoints. Note that disk-based queues can be made very reliable +by issuing a (f)sync after each write operation. Starting with version 4.3.2, this can +be requested via "<i><object>QueueSyncQueueFiles on/off</i> with the +default being off. Activating this option has a performance penalty, so it should +not be turned on without reason.</p> <h2>In-Memory Queues</h2> <p>In-memory queue mode is what most people have on their mind when they think about computing queues. Here, the enqueued data elements are held in memory. diff --git a/doc/rsconf1_markmessageperiod.html b/doc/rsconf1_markmessageperiod.html index 2c833339..a6486ba1 100644 --- a/doc/rsconf1_markmessageperiod.html +++ b/doc/rsconf1_markmessageperiod.html @@ -7,7 +7,7 @@ <h2>$MarkMessagePeriod</h2> <p><b>Type:</b> specific to immark input module</p> -<p><b>Default:</b> 1800 (20 minutes)</p> +<p><b>Default:</b> 1200 (20 minutes)</p> <p><b>Description:</b></p> <p>This specifies when mark messages are to be written to output modules. The time specified is in seconds. Specifying 0 is possible and disables mark diff --git a/doc/rsyslog_conf_actions.html b/doc/rsyslog_conf_actions.html index 2ef3f4b0..6020dd88 100644 --- a/doc/rsyslog_conf_actions.html +++ b/doc/rsyslog_conf_actions.html @@ -37,8 +37,15 @@ compared to the otherwise-equal config directives below:</p> *.=crit /var/log/critmsgs</b></code></p> <p> </p> <h3>Regular File</h3> -<p>Typically messages are logged to real files. The file has to -be specified with full pathname, beginning with a slash "/''.<br> +<p>Typically messages are logged to real files. The file usually is +specified by full pathname, beginning with a slash "/". +Starting with version 4.6.2 and 5.4.1 (previous v5 version do NOT support this) +relative file names can also be specified. To do so, these must begin with a +dot. For example, use "./file-in-current-dir.log" to specify a file in the +current directory. Please note that rsyslogd usually changes its working +directory to the root, so relative file names must be tested with care (they +were introduced primarily as a debugging vehicle, but may have useful other applications +as well).<br> <br> <br> You may prefix each entry with the minus "-'' sign to omit syncing the @@ -98,18 +105,14 @@ done, same with /dev/console.</p> messages to a remote host running rsyslogd(8) and to receive messages from remote hosts. Using this feature you're able to control all syslog messages on one host, if all other machines will log remotely to that. -This tears down<br> -administration needs.<br> -<br> -<b>Please note that this version of rsyslogd by default does NOT -forward messages it has received from the network to another host. -Specify the "-h" option to enable this.</b></p> +This tears down administration needs.</p> <p>To forward messages to another host, prepend the hostname with the at sign ("@"). A single at sign means that messages will be forwarded via UDP protocol (the standard for syslog). If you prepend two at signs ("@@"), the messages will be transmitted via TCP. Please note that plain TCP based syslog is not officially standardized, but -most major syslogds support it (e.g. syslog-ng or WinSyslog). The +most major syslogds support it (e.g. syslog-ng or +<a href="http://www.winsyslog.com/">WinSyslog</a>). The forwarding action indicator (at-sign) can be followed by one or more options. If they are given, they must be immediately (without a space) following the final at sign and be enclosed in parenthesis. The diff --git a/doc/rsyslog_conf_filter.html b/doc/rsyslog_conf_filter.html index 1d30d8ae..63c29817 100644 --- a/doc/rsyslog_conf_filter.html +++ b/doc/rsyslog_conf_filter.html @@ -48,8 +48,8 @@ in rsyslog and offer the best performance for this job.</p> facility and a priority, separated by a period (".''). Both parts are case insensitive and can also be specified as decimal numbers, but don't do that, you have been warned. Both facilities and priorities are -described in rsyslog(3). The names mentioned below correspond to the -similar LOG_-values in /usr/include/rsyslog.h.<br> +described in syslog(3). The names mentioned below correspond to the +similar LOG_-values in /usr/include/syslog.h.<br> <br> The facility is one of the following keywords: auth, authpriv, cron, daemon, kern, lpr, mail, mark, news, security (same as auth), syslog, @@ -231,7 +231,7 @@ A few quick samples:<br> <br> <code> *.* /var/log/file1 # the traditional way<br> -if $msg contains 'error' /var/log/errlog # the expression-based way<br> +if $msg contains 'error' then /var/log/errlog # the expression-based way<br> </code> <br> Right now, you need to specify numerical values if you would like to diff --git a/doc/rsyslog_conf_global.html b/doc/rsyslog_conf_global.html index 7dda046f..ce46bac2 100644 --- a/doc/rsyslog_conf_global.html +++ b/doc/rsyslog_conf_global.html @@ -99,6 +99,13 @@ netstream drivers. For all others, it will be ignored. <li>$ActionSendStreamDriverPermittedPeer <ID>, accepted fingerprint (SHA1) or name of remote peer. Note that this directive requires TLS netstream drivers. For all others, it will be ignored. (driver-specific) -<span style="font-weight: bold;"> directive may go away</span>!</li> +<li><b>$ActionSendTCPRebindInterval</b> nbr</a>- [available since 4.5.1] - instructs the TCP send +action to close and re-open the connection to the remote host every nbr of messages sent. +Zero, the default, means that no such processing is done. This directive is useful for +use with load-balancers. Note that there is some performance overhead associated with it, +so it is advisable to not too often "rebind" the connection (what +"too often" actually means depends on your configuration, a rule of thumb is +that it should be not be much more often than once per second).</li> <li><b>$ActionSendUDPRebindInterval</b> nbr</a>- [available since 4.3.2] - instructs the UDP send action to rebind the send socket every nbr of messages sent. Zero, the default, means that no rebind is done. This directive is useful for use with load-balancers.</li> @@ -111,6 +118,10 @@ that no rebind is done. This directive is useful for use with load-balancers.</l <li>$DefaultNetstreamDriver <drivername>, the default <a href="netstream.html">network stream driver</a> to use. Defaults to ptcp.$DefaultNetstreamDriverCAFile </path/to/cafile.pem></li> <li>$DefaultNetstreamDriverCertFile </path/to/certfile.pem></li> <li>$DefaultNetstreamDriverKeyFile </path/to/keyfile.pem></li> +<li><b>$DefaultRuleset</b> <i>name</i> - changes the default ruleset for unbound inputs to +the provided <i>name</i> (the default default ruleset is named +"RSYSLOG_DefaultRuleset"). It is advised to also read +our paper on <a href="multi_ruleset.html">using multiple rule sets in rsyslog</a>.</li> <li><b>$CreateDirs</b> [<b>on</b>/off] - create directories on an as-needed basis</li> <li><a href="rsconf1_dircreatemode.html">$DirCreateMode</a></li> <li><a href="rsconf1_dirgroup.html">$DirGroup</a></li> @@ -128,11 +139,15 @@ that no rebind is done. This directive is useful for use with load-balancers.</l <li><a href="rsconf1_gssforwardservicename.html">$GssForwardServiceName</a></li> <li><a href="rsconf1_gsslistenservicename.html">$GssListenServiceName</a></li> <li><a href="rsconf1_gssmode.html">$GssMode</a></li> -<li>$HUPisRestart [<b>on</b>/off] - if set to on, a HUP is a full daemon restart. This means any queued messages are discarded (depending +<li>$HUPisRestart [on/<b>off</b>] - if set to on, a HUP is a full daemon restart. This means any queued messages are discarded (depending on queue configuration, of course) all modules are unloaded and reloaded. This mode keeps compatible with sysklogd, but is -not recommended for use with rsyslog. To do a full restart, simply stop and start the daemon. The default is "on" for -compatibility reasons. If it is set to "off", a HUP will only close open files. This is a much quicker action and usually -the only one that is needed e.g. for log rotation. <b>It is recommended to set the setting to "off".</b></li> +not recommended for use with rsyslog. To do a full restart, simply stop and start the daemon. The default (since 4.5.1) is "off". +If it is set to "off", a HUP will only close open files. This is a much quicker action and usually +the only one that is needed e.g. for log rotation. <b>Restart-type HUPs (value "on") are depricated</b> +and will go away in rsyslog v5. So it is a good idea to change anything that needs it, now. +Usually that should not be a big issue, as the restart-type HUP can easily be replaced by +something along the lines of "/etc/init.d/rsyslog restart". +</li> <li><a href="rsconf1_includeconfig.html">$IncludeConfig</a></li><li>MainMsgQueueCheckpointInterval <number></li> <li>$MainMsgQueueDequeueSlowdown <number> [number is timeout in <i> micro</i>seconds (1000000us is 1sec!), @@ -192,6 +207,28 @@ supported in order to be compliant to the upcoming new syslog RFC series. <li><a href="rsconf1_maxopenfiles.html">$MaxOpenFiles</a></li> <li><a href="rsconf1_moddir.html">$ModDir</a></li> <li><a href="rsconf1_modload.html">$ModLoad</a></li> +<li><b>$OMFileAsyncWriting</b> [on/<b>off</b>], if turned on, the files will be written +in asynchronous mode via a separate thread. In that case, double buffers will be used so +that one buffer can be filled while the other buffer is being written. Note that in order +to enable $OMFileFlushInterval, $OMFileAsyncWriting must be set to "on". Otherwise, the flush +interval will be ignored. Also note that when $OMFileFlushOnTXEnd is "on" but +$OMFileAsyncWriting is off, output will only be written when the buffer is full. This may take +several hours, or even require a rsyslog shutdown. However, a buffer flush can be forced +in that case by sending rsyslogd a HUP signal. +<li><b>$OMFileZipLevel</b> 0..9 [default 0] - if greater 0, turns on gzip compression +of the output file. The higher the number, the better the compression, but also the +more CPU is required for zipping.</li> +<li><b>$OMFileIOBufferSize</b> <size_nbr>, default 4k, size of the buffer used to writing output data. The larger the buffer, the potentially better performance is. The default of 4k is quite conservative, it is useful to go up to 64k, and 128K if you used gzip compression (then, even higher sizes may make sense)</li> +<li><b>$OMFileFlushOnTXEnd</b> <[<b>on</b>/off]>, default on. Omfile has the +capability to +write output using a buffered writer. Disk writes are only done when the buffer is +full. So if an error happens during that write, data is potentially lost. In cases where +this is unacceptable, set $OMFileFlushOnTXEnd to on. Then, data is written at the end +of each transaction (for pre-v5 this means after <b>each</b> log message) and the usual +error recovery thus can handle write errors without data loss. Note that this option +severely reduces the effect of zip compression and should be switched to off +for that use case. Note that the default -on- is primarily an aid to preserve +the traditional syslogd behaviour.</li> <li><b>$RepeatedMsgContainsOriginalMsg</b> [on/<b>off</b>] - "last message repeated n times" messages, if generated, have a different format that contains the message that is being repeated. Note that only the first "n" characters are included, with n to be at least 80 characters, most @@ -200,6 +237,12 @@ line is that n is large enough to get a good idea which message was repeated but large enough for the whole message. (Introduced with 4.1.5). Once set, it affects all following actions.</li> <li><a href="rsconf1_repeatedmsgreduction.html">$RepeatedMsgReduction</a></li> <li><a href="rsconf1_resetconfigvariables.html">$ResetConfigVariables</a></li> +<li><b>$Ruleset</b> <i>name</i> - starts a new ruleset or switches back to one already defined. +All following actions belong to that new rule set. +the <i>name</i> does not yet exist, it is created. To swith back to rsyslog's +default ruleset, specify "RSYSLOG_DefaultRuleset") as the name. +All following actions belong to that new rule set. It is advised to also read +our paper on <a href="multi_ruleset.html">using multiple rule sets in rsyslog</a>.</li> <li><b>$OptimizeForUniprocessor</b> [on/<b>off</b>] - turns on optimizatons which lead to better performance on uniprocessors. If you run on multicore-machiens, turning this off lessens CPU load. The default may change as uniprocessor systems become less common. [available since 4.1.0]</li> diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html index df9abeea..675b8bb3 100644 --- a/doc/rsyslog_conf_modules.html +++ b/doc/rsyslog_conf_modules.html @@ -2,11 +2,42 @@ <html><head><title>Modules - rsyslog.conf</title></head> <body> <p>This is a part of the rsyslog.conf documentation.</p> -<a href="rsyslog_conf.html">back</a> -<h2>Modules</h2> -<p>Rsyslog has a modular design. Consequently, there is a growing +<a href="rsyslog_conf.html">Back to rsyslog.conf manual</a> +<h1>Modules</h1> +<p>Rsyslog has a modular design. This enables functionality to be +dynamically loaded from modules, which may also be written by any +third party. Rsyslog itself offers all non-core functionality as +modules. Consequently, there is a growing number of modules. Here is the entry point to their documentation and what they do (list is currently not complete)</p> +<p>Please note that each module provides configuration +directives, which are NOT necessarily being listed below. Also +remember, that a modules configuration directive (and functionality) is +only available if it has been loaded (using $ModLoad).</p> +<p>It is relatively easy to write a rsyslog module. <b>If none of the provided +modules solve your need, you may consider writing one or have one written +for you by +<a href="http://www.rsyslog.com/professional-services">Adiscon's professional services for rsyslog</a> +</b>(this often is a very cost-effective and efficient way of getting what you need). + +<h2>Input Modules</h2> +<p>Input modules are used to gather messages from various sources. They interface +to message generators. +<ul> +<li><a href="imfile.html">imfile</a> - input module for text files</li> +<li><a href="imrelp.html">imrelp</a> - RELP input module</li> +<li>imudp - udp syslog message input</li> +<li><a href="imtcp.html">imtcp</a> - input plugin for plain tcp syslog</li> +<li><a href="imgssapi.html">imgssapi</a> - input plugin for plain tcp and GSS-enabled syslog</li> +<li>immark - support for mark messages</li> +<li><a href="imklog.html">imklog</a> - kernel logging</li> +<li><a href="imuxsock.html">imuxsock</a> - unix sockets, including the system log socket</li> +<li><a href="im3195.html">im3195</a> - accepts syslog messages via RFC 3195</li> +</ul> + +<h2>Output Modules</h2> +<p>Output modules process messages. With them, message formats can be transformed +and messages be transmitted to various different targets. <ul> <li><a href="omsnmp.html">omsnmp</a> - SNMP trap output module</li> <li><a href="omstdout.html">omtdout</a> - stdout output module (mainly a test tool)</li> @@ -20,26 +51,14 @@ SQLLite, Ingres, Oracle, mSQL)</li> <li><a href="ommail.html">ommail</a> - permits rsyslog to alert folks by mail if something important happens</li> <li><a href="omoracle.html">omoracle</a> - output module for Oracle (native OCI interface)</li> -<li><a href="imfile.html">imfile</a> -- input module for text files</li> -<li><a href="imrelp.html">imrelp</a> - RELP -input module</li> -<li>imudp - udp syslog message input</li> -<li><a href="imtcp.html">imtcp</a> - input -plugin for plain tcp syslog</li> -<li><a href="imgssapi.html">imgssapi</a> - -input plugin for plain tcp and GSS-enabled syslog</li> -<li>immark - support for mark messages</li> -<li><a href="imklog.html">imklog</a> - kernel logging</li> -<li><a href="imuxsock.html">imuxsock</a> - -unix sockets, including the system log socket</li> -<li><a href="im3195.html">im3195</a> - -accepts syslog messages via RFC 3195</li> </ul> -<p>Please note that each module provides configuration -directives, which are NOT necessarily being listed below. Also -remember, that a modules configuration directive (and functionality) is -only available if it has been loaded (using $ModLoad).</p> + +<h2>Library Modules</h2> +<p>Library modules provide dynamically loadable functionality for parts of rsyslog, +most often for other loadable modules. They can not be user-configured and are loaded +automatically by some components. They are just mentioned so that error messages that +point to library moduls can be understood. No module list is provided. + <p>[<a href="manual.html">manual index</a>] [<a href="rsyslog_conf.html">rsyslog.conf</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> diff --git a/doc/rsyslog_conf_templates.html b/doc/rsyslog_conf_templates.html index 6c68b801..baa4ce29 100644 --- a/doc/rsyslog_conf_templates.html +++ b/doc/rsyslog_conf_templates.html @@ -87,8 +87,7 @@ option. Otherwise you will become vulnerable to SQL injection. <br> To escape:<br> % = \%<br> \ = \\ --> '\' is used to escape (as in C)<br> -$template TraditionalFormat,%timegenerated% %HOSTNAME% -%syslogtag%%msg%\n"<br> +$template TraditionalFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"<br> <br> Properties can be accessed by the <a href="property_replacer.html">property replacer</a> (see there for details).</p> diff --git a/doc/rsyslog_pgsql.html b/doc/rsyslog_pgsql.html new file mode 100644 index 00000000..dcb9dc3a --- /dev/null +++ b/doc/rsyslog_pgsql.html @@ -0,0 +1,336 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> +<HTML> +<HEAD> + <META HTTP-EQUIV="CONTENT-TYPE" CONTENT="text/html; charset=utf-8"> + <TITLE></TITLE> + <META NAME="GENERATOR" CONTENT="OpenOffice.org 3.1 (Unix)"> + <META NAME="AUTHOR" CONTENT="Marc Schiffbauer"> + <META NAME="CREATED" CONTENT="20100129;15054500"> + <META NAME="CHANGEDBY" CONTENT="Marc Schiffbauer"> + <META NAME="CHANGED" CONTENT="20100129;16035000"> + <META NAME="Info 1" CONTENT=""> + <META NAME="Info 2" CONTENT=""> + <META NAME="Info 3" CONTENT=""> + <META NAME="Info 4" CONTENT=""> + <STYLE TYPE="text/css"> + <!-- + @page { size: 8.27in 11.69in; margin: 0.79in } + P { margin-bottom: 0.08in } + P.western { font-family: "Arial", sans-serif } + H1 { margin-bottom: 0.08in } + H1.western { font-family: "Times New Roman", serif } + H1.cjk { font-family: "DejaVu Sans" } + H1.ctl { font-family: "DejaVu Sans" } + H2 { margin-bottom: 0.08in } + H2.western { font-family: "Times New Roman", serif } + BLOCKQUOTE.western { font-family: "Arial", sans-serif } + H3 { margin-bottom: 0.08in } + H3.western { font-family: "Times New Roman", serif } + A:link { so-language: zxx } + --> + </STYLE> +</HEAD> +<BODY LANG="de-DE" DIR="LTR"> +<H1 CLASS="western"><SPAN LANG="en-US">Writing </SPAN>syslog messages +to MySQL, PostgreSQL or any other supported Database</H1> +<P CLASS="western"><FONT SIZE=2><I>Written by </I></FONT><A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php"><FONT SIZE=2><I>Rainer +Gerhards</I></FONT></A><FONT SIZE=2><I> with some additions by Marc +Schiffbauer (2008-02-28)</I></FONT></P> +<H2 CLASS="western">Abstract</H2> +<P CLASS="western"><SPAN LANG="en-US"><I><B>In this paper, I describe +how to write </B></I></SPAN><A HREF="http://www.monitorware.com/en/topics/syslog/">syslog</A><SPAN LANG="en-US"><I><B> +messages to a </B></I></SPAN><A HREF="http://www.mysql.com/">MySQL</A><SPAN LANG="en-US"><I><B> +or </B></I></SPAN><A HREF="http://www.postgresql.org/">PostgreSQL</A><SPAN LANG="en-US"><I><B> +database.</B></I></SPAN><SPAN LANG="en-US"><I> Having syslog messages +in a database is often handy, especially when you intend to set up a +front-end for viewing them. This paper describes an approach with +</I></SPAN><A HREF="http://www.rsyslog.com/">rsyslogd</A><SPAN LANG="en-US"><I>, +an alternative enhanced syslog daemon natively supporting MySQL and +PostgreSQL. I describe the components needed to be installed and how +to configure them. Please note that as of this writing, rsyslog +supports a variety of databases. While this guide is still MySQL- and +PostgreSQL-focused, you can probably use it together with other ones +too. You just need to modify a few settings.</I></SPAN></P> +<H2 CLASS="western">Background</H2> +<P LANG="en-US" CLASS="western">In many cases, syslog data is simply +written to text files. This approach has some advantages, most +notably it is very fast and efficient. However, data stored in text +files is not readily accessible for real-time viewing and analysis. +To do that, the messages need to be in a database. There are various +ways to store syslog messages in a database. For example, some have +the syslogd write text files which are later feed via a separate +script into the database. Others have written scripts taking the data +(via a pipe) from a non-database-aware syslogd and store them as they +appear. Some others use database-aware syslogds and make them write +the data directly to the database. In this paper, I use that "direct +write" approach. I think it is superior, because the syslogd +itself knows the status of the database connection and thus can +handle it intelligently (well ... hopefully ;)). I use rsyslogd to +acomplish this, simply because I have initiated the rsyslog project +with database-awareness as one goal.</P> +<P CLASS="western"><SPAN LANG="en-US"><B>One word of caution:</B></SPAN><SPAN LANG="en-US"> +while message storage in the database provides an excellent +foundation for interactive analysis, it comes at a cost. Database i/o +is considerably slower than text file i/o. As such, directly writing +to the database makes sense only if your message volume is low enough +to allow a) the syslogd, b) the network, and c) the database server +to catch up with it. Some time ago, I have written a paper on +</SPAN><A HREF="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php">optimizing +syslog server performance</A><SPAN LANG="en-US">. While this paper +talks about Window-based solutions, the ideas in it are generic +enough to apply here, too. So it might be worth reading if you +anticipate medium high to high traffic. If you anticipate really high +traffic (or very large traffic spikes), you should seriously consider +forgetting about direct database writes - in my opinion, such a +situation needs either a very specialized system or a different +approach (the text-file-to-database approach might work better for +you in this case). </SPAN> +</P> +<H2 CLASS="western">Overall System Setup</H2> +<P CLASS="western"><SPAN LANG="en-US">In this paper, I concentrate on +the server side. If you are thinking about interactive syslog message +review, you probably want to centralize syslog. In such a scenario, +you have multiple machines (the so-called clients) send their data to +a central machine (called server in this context). While I expect +such a setup to be typical when you are interested in storing +messages in the database, I do not describe how to set it up. This is +beyond the scope of this paper. If you search a little, you will +probably find many good descriptions on </SPAN><SPAN LANG="en-US">how +to centralize syslog. If you do that, it might be a good idea to do +it securely, so you might also be interested in my paper on +</SPAN><A HREF="http://www.rsyslog.com/doc-rsyslog_stunnel.html">ssl-encrypting +syslog message transfer</A><SPAN LANG="en-US">.</SPAN></P> +<P LANG="en-US" CLASS="western">No matter how the messages arrive at +the server, their processing is always the same. So you can use this +paper in combination with any description for centralized syslog +reporting.</P> +<P CLASS="western"><SPAN LANG="en-US">As I already said, I use +rsyslogd on the server. It has intrinsic support for talking to the +supported databases. For obvious reasons, we also need an instance of +MySQL or PostgreSQL running. To keep us focused, the setup of the +database itself is also beyond the scope of this paper. I assume that +you have successfully installed the database and also have a +front-end at hand to work with it (for example, </SPAN><A HREF="http://www.phpmyadmin.net/">phpMyAdmin</A><SPAN LANG="en-US"> +or </SPAN><A HREF="http://phppgadmin.sourceforge.net/">phpPgAdmin</A><SPAN LANG="en-US">. +Please make sure that this is installed, actually working and you +have a basic understanding of how to handle it.</SPAN></P> +<H2 CLASS="western">Setting up the system</H2> +<P CLASS="western"><SPAN LANG="en-US">You need to download and +install rsyslogd first. Obtain it from the </SPAN><A HREF="http://www.rsyslog.com/">rsyslog +site</A><SPAN LANG="en-US">. Make sure that you disable stock +syslogd, otherwise you will experience some difficulties. On some +distributions (Fedora 8 and above, for example), rsyslog may +already by the default syslogd, in which case you obviously do not +need to do anything specific. For many others, there are prebuild +packages available. If you use either, please make sure that you have +the required database plugins for your database available. It usually +is a separate package and typically </SPAN><SPAN LANG="en-US"><B>not</B></SPAN><SPAN LANG="en-US"> +installed by default.</SPAN></P> +<P CLASS="western"><SPAN LANG="en-US">It is important to understand +how rsyslogd talks to the database. In rsyslogd, there is the concept +of "templates". Basically, a template is a string that +includes some replacement characters, which are called "properties" +in rsyslog. Properties are accessed via the "</SPAN><A HREF="http://www.rsyslog.com/doc-property_replacer.html">Property +Replacer</A><SPAN LANG="en-US">". Simply said, you access +properties by including their name between percent signs inside the +template. For example, if the syslog message is "Test", the +template "%msg%" would be expanded to "Test". +Rsyslogd supports sending template text as a SQL statement to the +database. As such, the template must be a valid SQL statement. There +is no limit in what the statement might be, but there are some +obvious and not so obvious choices. For example, a template "drop +table xxx" is possible, but does not make an awful lot of sense. +In practice, you will always use an "insert" statement +inside the template.</SPAN></P> +<P LANG="en-US" CLASS="western">An example: if you would just like to +store the msg part of the full syslog message, you have probably +created a table "syslog" with a single column "message". +In such a case, a good template would be "insert into +syslog(message) values ('%msg%')". With the example above, that +would be expanded to "insert into syslog(message) +values('Test')". This expanded string is then sent to the +database. It's that easy, no special magic. The only thing you must +ensure is that your template expands to a proper SQL statement and +that this statement matches your database design.</P> +<P CLASS="western"><SPAN LANG="en-US">Does that mean you need to +create database schema yourself and also must fully understand +rsyslogd's properties? No, that's not needed. Because we anticipated +that folks are probably more interested in getting things going +instead of designing them from scratch. So we have provided a default +schema as well as build-in support for it. This schema also offers an +additional benefit: rsyslog is part of </SPAN><A HREF="http://www.adiscon.com/en/">Adiscon</A><SPAN LANG="en-US">'s +</SPAN><A HREF="http://www.monitorware.com/en/">MonitorWare product +line</A><SPAN LANG="en-US"> (which includes open source and closed +source members). All of these tools share the same default schema and +know how to operate on it. For this reason, the default schema is +also called the "MonitorWare Schema". If you use it, you +can simply add </SPAN><A HREF="http://www.phplogcon.org/">phpLogCon, +a GPLed syslog web interface</A><SPAN LANG="en-US">, to your system +and have instant interactive access to your database. So there are +some benefits in using the provided schema.</SPAN></P> +<P LANG="en-US" CLASS="western">The schema definition is contained in +the file "createDB.sql". It comes with the rsyslog package +and one can be found for each supported database type (in the plugins +directory). Review it to check that the database name is acceptable +for you. Be sure to leave the table and field names unmodified, +because otherwise you need to customize rsyslogd's default sql +template, which we do not do in this paper. Then, run the script with +your favorite SQL client. Double-check that the table was +successfully created.</P> +<P LANG="en-US" CLASS="western">It is important to note that the +correct database encoding must be used so that the database will +accept strings independend of the string encoding. This is an +important part because it can not be guarantied that all syslog +messages will have a defined character encoding. This is especially +true if the rsyslog-Server will collect messages from different +clients and different products. +</P> +<P LANG="en-US" CLASS="western">For example PostgreSQL may refuse to +accept messages if you would set the database encoding to “UTF8†+while a client is sending invalid byte sequences for that encoding. +</P> +<P LANG="en-US" CLASS="western">Database support in rsyslog is +integrated via loadable plugin modules. To use the database +functionality, the database plugin must be enabled in the config file +BEFORE the first database table action is used. This is done by +placing the</P> +<BLOCKQUOTE CLASS="western"><CODE>$ModLoad ommysql</CODE></BLOCKQUOTE> +<P CLASS="western">directive at the begining of /etc/rsyslog.conf for +MySQL and</P> +<BLOCKQUOTE CLASS="western"><CODE>$ModLoad ompgsql</CODE></BLOCKQUOTE> +<P CLASS="western"><CODE><FONT FACE="Arial, sans-serif">for +PostgreSQL.</FONT></CODE></P> +<P LANG="en-US" CLASS="western"><FONT FACE="Arial, sans-serif">For +other databases, use their plugin name (e.g. omoracle).</FONT></P> +<P CLASS="western">Next, we need to tell rsyslogd to write data to +the database. As we use the default schema, we do NOT need to define +a template for this. We can use the hardcoded one (rsyslogd handles +the proper template linking). So all we need to do e.g. for MySQL is +add a simple selector line to /etc/rsyslog.conf:</P> +<BLOCKQUOTE CLASS="western"><CODE>*.* +:ommysql:database-server,database-name,database-userid,database-password</CODE></BLOCKQUOTE> +<P CLASS="western">Again, other databases have other selector names, +e.g. ":ompgsql:" instead of ":ommysql:". See the +output plugin's documentation for details.</P> +<P LANG="en-US" CLASS="western">In many cases, the database will run +on the local machine. In this case, you can simply use "127.0.0.1" +for <I>database-server</I>. This can be especially advisable, if you +do not need to expose the database to any process outside of the +local machine. In this case, you can simply bind it to 127.0.0.1, +which provides a quite secure setup. Of course, rsyslog also supports +remote database instances. In that case, use the remote server name +(e.g. mydb.example.com) or IP-address. The <I>database-name</I> by +default is "Syslog". If you have modified the default, use +your name here. <I>Database-userid</I> and <I>-password</I> are the +credentials used to connect to the database. As they are stored in +clear text in rsyslog.conf, that user should have only the least +possible privileges. It is sufficient to grant it INSERT privileges +to the systemevents table, only. As a side note, it is strongly +advisable to make the rsyslog.conf file readable by root only - if +you make it world-readable, everybody could obtain the password (and +eventually other vital information from it). In our example, let's +assume you have created a database user named "syslogwriter" +with a password of "topsecret" (just to say it bluntly: +such a password is NOT a good idea...). If your database is on the +local machine, your rsyslog.conf line might look like in this sample:</P> +<BLOCKQUOTE CLASS="western"><CODE>*.* +:ommysql:127.0.0.1,Syslog,syslogwriter,topsecret</CODE></BLOCKQUOTE> +<P CLASS="western">Save rsyslog.conf, restart rsyslogd - and you +should see syslog messages being stored in the "systemevents" +table!</P> +<P LANG="en-US" CLASS="western">The example line stores every message +to the database. Especially if you have a high traffic volume, you +will probably limit the amount of messages being logged. This is easy +to accomplish: the "write database" action is just a +regular selector line. As such, you can apply normal selector-line +filtering. If, for example, you are only interested in messages from +the mail subsystem, you can use the following selector line:</P> +<BLOCKQUOTE CLASS="western"><CODE>mail.* :ommysql:127.0.0.1,syslog,syslogwriter,topsecret</CODE></BLOCKQUOTE> +<P CLASS="western">Review the <A HREF="http://www.rsyslog.com/doc-rsyslog_conf.html">rsyslog.conf</A> +documentation for details on selector lines and their filtering.</P> +<P CLASS="western"><SPAN LANG="en-US"><B>You have now completed +everything necessary to store syslog messages to the a database.</B></SPAN><SPAN LANG="en-US"> +If you would like to try out a front-end, you might want to look at +</SPAN><A HREF="http://www.phplogcon.org/">phpLogCon</A><SPAN LANG="en-US">, +which displays syslog data in a browser. As of this writing, +phpLogCon is not yet a powerful tool, but it's open source, so it +might be a starting point for your own solution.</SPAN></P> +<H2 CLASS="western">On Reliability...</H2> +<P LANG="en-US" CLASS="western">Rsyslogd writes syslog messages +directly to the database. This implies that the database must be +available at the time of message arrival. If the database is offline, +no space is left or something else goes wrong - rsyslogd can not +write the database record. If rsyslogd is unable to store a message, +it performs one retry. This is helpful if the database server was +restarted. In this case, the previous connection was broken but a +reconnect immediately succeeds. However, if the database is down for +an extended period of time, an immediate retry does not help.</P> +<P CLASS="western"><SPAN LANG="en-US">Message loss in this scenario +can easily be prevented with rsyslog. All you need to do is run the +database writer in queued mode. This is now described in a generic +way and I do not intend to duplicate it here. So please be sure to +read "</SPAN><A HREF="http://www.rsyslog.com/doc-rsyslog_high_database_rate.html">Handling +a massive syslog database insert rate with Rsyslog</A><SPAN LANG="en-US">", +which describes the scenario and also includes configuration +examples.</SPAN></P> +<H2 CLASS="western">Conclusion</H2> +<P LANG="en-US" CLASS="western">With minimal effort, you can use +rsyslogd to write syslog messages to a database. You can even make it +absolutely fail-safe and protect it against database server downtime. +Once the messages are arrived there, you can interactively review and +analyze them. In practice, the messages are also stored in text files +for longer-term archival and the databases are cleared out after some +time (to avoid becoming too slow). If you expect an extremely high +syslog message volume, storing it in real-time to the database may +outperform your database server. In such cases, either filter out +some messages or used queued mode (which in general is recommended +with databases).</P> +<P LANG="en-US" CLASS="western">The method outlined in this paper +provides an easy to setup and maintain solution for most use cases.</P> +<H3 CLASS="western">Feedback Requested</H3> +<P CLASS="western">I would appreciate feedback on this paper. If you +have additional ideas, comments or find bugs, please <A HREF="mailto:rgerhards@adiscon.com">let +me know</A>.</P> +<H2 CLASS="western">References and Additional Material</H2> +<UL> + <LI><P CLASS="western" STYLE="margin-bottom: 0in"><A HREF="http://www.rsyslog.com/">www.rsyslog.com</A> + - the rsyslog site + </P> + <LI><P CLASS="western"><A HREF="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php">Paper + on Syslog Server Optimization</A> + </P> +</UL> +<H2 CLASS="western">Revision History</H2> +<UL> + <LI><P CLASS="western" STYLE="margin-bottom: 0in">2005-08-02 * + <A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer + Gerhards</A> * initial version created + </P> + <LI><P CLASS="western" STYLE="margin-bottom: 0in">2005-08-03 * + <A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer + Gerhards</A> * added references to demo site + </P> + <LI><P CLASS="western" STYLE="margin-bottom: 0in">2007-06-13 * + <A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer + Gerhards</A> * removed demo site - was torn down because too + expensive for usage count + </P> + <LI><P CLASS="western" STYLE="margin-bottom: 0in">2008-02-21 * + <A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer + Gerhards</A> * updated reliability section, can now be done with + on-demand disk queues</P> + <LI><P CLASS="western">2008-02-28 * <A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer + Gerhards</A> * added info on other databases, updated syntax to more + recent one + </P> + <LI><P CLASS="western">2010-01-29 * Marc Schiffbauer * added some + PostgreSQL stuff, made wording more database generic, fixed some + typos</P> +</UL> +<H2 CLASS="western">Copyright</H2> +<P CLASS="western">Copyright (c) 2005-2010 <A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</A>, Marc Schiffbauer and <A HREF="http://www.adiscon.com/en/">Adiscon</A>.</P> +<P CLASS="western"><BR><BR> +</P> +</BODY> +</HTML>
\ No newline at end of file diff --git a/doc/rsyslog_php_syslog_ng.html b/doc/rsyslog_php_syslog_ng.html index bf48a1eb..ed4d72fc 100644 --- a/doc/rsyslog_php_syslog_ng.html +++ b/doc/rsyslog_php_syslog_ng.html @@ -7,8 +7,10 @@ <P><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> (2005-08-04)</i></small></P> -<p><b>Note: it has been reported that this guide is somewhat outdated. Please -use with care. </b></p> +<p>Note: it has been reported that this guide is somewhat outdated. Please +use with care. Also, please note that <b>rsyslog's "native" web frontend is +<a href="http://www.phplogcon.org">phpLogCon</a></b>, which provides best integration +and a lot of extra functionality.</p> <h2>Abstract</h2> <p><i><b>In this paper, I describe how to use <a href="http://www.vermeer.org/projects/php-syslog-ng">php-syslog-ng</a> with @@ -116,11 +118,11 @@ those unfamiliar with syslog-ng, this configuration is probably easier to set up then switching to syslog-ng. For existing rsyslogd users, php-syslog-ng might be a nice add-on to their logging infrastructure.</P> <P>Please note that the <a href="http://www.monitorware.com/en/">MonitorWare family</a> (to which rsyslog belongs) also -offers a web-interface: <a href="http://www.phplogcon.org/">phpLogCon</a>. At the time of this writing, phpLogCon's code -is by far not as clean as I would like it to be. Also the user-interface is -definitely not as intutive as pp-syslog-ng. From a functionality point of view, -however, I think it already is a bit ahead. So you might -consider using it. I have set up a <a href="http://demo.rsyslog.com/">demo server</a>., +offers a web-interface: <a href="http://www.phplogcon.org/">phpLogCon</a>. +From my point of view, obviously, <b>phpLogCon is the more natural choice for a web interface +to be used together with rsyslog</b>. It also offers superb functionality and provides, +for example,native display of Windows event log entries. +I have set up a <a href="http://demo.phplogcon.org/">demo server</a>., You can have a peek at it without installing anything.</P> <h2>Feedback Requested</h2> diff --git a/doc/src/rsyslog_pgsql.odt b/doc/src/rsyslog_pgsql.odt Binary files differnew file mode 100644 index 00000000..5034c5fb --- /dev/null +++ b/doc/src/rsyslog_pgsql.odt diff --git a/doc/tls_cert_machine.html b/doc/tls_cert_machine.html index 5ecde0d1..095e15c2 100644 --- a/doc/tls_cert_machine.html +++ b/doc/tls_cert_machine.html @@ -75,7 +75,15 @@ Locality name: <font color="red">Somewhere</font> State or province name: <font color="red">CA</font> Common name: <font color="red">machine.example.net</font> UID: -Enter a challenge password: +Enter a dnsName of the subject of the certificate: +Enter the IP address of the subject of the certificate: +Enter the e-mail of the subject of the certificate: +Enter a challange password: +Does the certificate belong to an authority? (y/N): <font color="red">n</font> +Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N): +Will the certificate be used for encryption (RSA ciphersuites)? (y/N): +Is this a TLS web client certificate? (y/N): <font color="red">y</font> +Is this also a TLS web server certificate? (y/N): <font color="red">y</font> [root@rgf9dev sample]# <font color="red">certtool --generate-certificate --load-request request.pem --outfile cert.pem --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem</font> Generating a signed certificate... Enter the certificate's serial number (decimal): @@ -86,10 +94,12 @@ The certificate will expire in (days): 1000 Extensions. +Do you want to honour the extensions from the request? (y/N): Does the certificate belong to an authority? (Y/N): <font color="red">n</font> Is this a TLS web client certificate? (Y/N): <font color="red">y</font> Is this also a TLS web server certificate? (Y/N): <font color="red">y</font> Enter the dnsName of the subject of the certificate: <font color="red">machine.example.net</font> <i>{This is the name of the machine that will use the certificate}</i> +Enter the IP address of the subject of certificate: Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (Y/N): Will the certificate be used for encryption (RSA ciphersuites)? (Y/N): X.509 Certificate Information: diff --git a/doc/v4compatibility.html b/doc/v4compatibility.html new file mode 100644 index 00000000..5d877af1 --- /dev/null +++ b/doc/v4compatibility.html @@ -0,0 +1,77 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head><title>Compatibility notes for rsyslog v4</title> +</head> +<body> +<h1>Compatibility Notes for rsyslog v4</h1> +<p><small><i>Written by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> +(2009-07-15)</i></small></p> +<p>The changes introduced in rsyslog v4 are numerous, but not very intrusive. +This document describes things to keep in mind when moving from v3 to v4. It +does not list enhancements nor does it talk about compatibility concerns introduced +by v3 (for this, see the <a href="v3compatibility.html">rsyslog v3 compatibility notes</a>). +<h2>HUP processing</h2> +<p>With v3 and below, rsyslog used the traditional HUP behaviour. That meant that +all output files are closed and the configuration file is re-read and the new configuration +applied. +<p>With a program as simple and static as sysklogd, this was not much of an issue. The +most important config settings (like udp reception) of a traditional syslogd can not be +modified via the configuration file. So a config file reload only meant setting up a new set of filters. It also didn't account as problem that while doing so messages may be lost - without +any threading and queuing model, a traditional syslogd will potentially always loose +messages, so it is irrelevant if this happens, too, during the short config re-read +phase. +<p>In rsyslog, things are quite different: the program is more or less a framework into +which loadable modules are loaded as needed for a particular configuration. The software +that will acutally be running is taylored via the config file. Thus, a re-read of +the config file requires a full, very heavy restart, because the software acutally +running with the new config can be totally different from what ran with the old config. +<p>Consequently, the traditional HUP is a very heavy operation and may even cause some +data loss because queues must be shut down, listeners stopped and so on. Some of these +operations (depending on their configuration) involve intentional message loss. The operation +also takes up a lot of system resources and needs quite some time (maybe seconds) to be +completed. During this restart period, the syslog subsytem is not fully available. +<p>From the software developer's point of view, the full restart done by a HUP is rather complex, +especially if user-timeout limits set on action completion are taken into consideration (for +those in the know: at the extreme ends this means we need to cancel threads as a last resort, +but than we need to make sure that such cancellation does not happen at points where it +would be fatal for a restart). A regular restart, where the process is actually terminated, is +much less complex, because the operating system does a full cleanup after process termination, +so rsyslogd does not need to take care for exotic cleanup cases and leave that to the OS. +In the end result, restart-type HUPs clutter the code, increase complexity (read: add bugs) +and cost performance. +<p>On the contrary, a HUP is typically needed for log rotation, and the real desire is +to close files. This is a non-disruptive and very lightweigth operation. +<p>Many people have said that they are used to HUP the syslogd to apply configuration +changes. This is true, but it is questionable if that really justifies all the cost that +comes with it. After all, it is the difference between typing +<pre> +$ kill -HUP `cat /var/run/rsyslogd.pid` +</pre> +versus +<pre> +$ /etc/init.d/rsyslog restart +</pre> +Semantically, both is mostly the same thing. The only difference is that with the restart +command rsyslogd can spit config error message to stderr, so that the user is able to see +any problems and fix them. With a HUP, we do not have access to stderr and thus can log +error messages only to their configured destinations; exprience tells that most users +will never find them there. What, by the way, is another strong argument against +restarting rsyslogd by HUPing it. +<p>So a restart via HUP is not strictly necessary +and most other deamons require that a restart command is typed in if a restart is required. +<p>Rsyslog will follow this paradigm in the next versions, resulting in many benefits. In v4, +we provide some support for the old-style semantics. We introduced a setting $HUPisRestart +which may be set to "on" (tradional, heavy operationg) +or "off" (new, lightweight "file close only" operation). +The initial versions had the default set to traditional behavior, but starting with 4.5.1 +we are now using the new behavior as the default. +<p>Most importantly, <b>this may break some scripts</b>, but my sincere belief is that +there are very few scripts that automatically <b>change</b> rsyslog's config and then do a +HUP to reload it. Anyhow, if you have some of these, it may be a good idea to change +them now instead of turning restart-type HUPs on. Other than that, one mainly needs +to change the habit of how to restart rsyslog after a configuration change. +<p><b>Please note that restart-type HUP is depricated and will go away in rsyslog v5.</b> +So it is a good idea to become ready for the new version now and also enjoy some of the +benefits of the "real restart", like the better error-reporting capability. +<p>Note that code complexity reduction (and thus performance improvement) needs the restart-type +HUP code to be removed, so these changes can (and will) only happen in version 5. +</body></html> diff --git a/m4/ax_check_off64_t.m4 b/m4/ax_check_off64_t.m4 new file mode 100644 index 00000000..05d6f45d --- /dev/null +++ b/m4/ax_check_off64_t.m4 @@ -0,0 +1,69 @@ +# =========================================================================== +# http://www.gnu.org/software/autoconf-archive/ax_check_off64_t.html +# =========================================================================== +# +# SYNOPSIS +# +# AX_CHECK_OFF64_T +# +# DESCRIPTION +# +# Check if off64_t is defined. On true define HAVE_OFF64_T, also define +# __LARGEFILE64_SOURCE where one is needed. (Note that an appropriative +# entry must be in config.h.in.) +# +# LICENSE +# +# Copyright (c) 2008 Ruslan Shevchenko <Ruslan@Shevchenko.Kiev.UA> +# +# Copying and distribution of this file, with or without modification, are +# permitted in any medium without royalty provided the copyright notice +# and this notice are preserved. This file is offered as-is, without any +# warranty. + +#serial 6 + +AU_ALIAS([RSSH_CHECK_OFF64_T], [AX_CHECK_OFF64_T]) +AC_DEFUN([AX_CHECK_OFF64_T], [ +AC_REQUIRE([AC_SYS_LARGEFILE])dnl +AC_CHECK_HEADER(unistd.h) +AC_CACHE_CHECK([whether type off64_t support], + [ax_cv_check_off64_t], + [ + AC_COMPILE_IFELSE( +AC_LANG_SOURCE([ +#ifdef HAVE_UNISTD_H +#include <unistd.h> +#endif +extern off64_t x1; +]) +,ax_have_off64t=1) + if test "x$ax_have_off64t" = "x" + then + AC_COMPILE_IFELSE( +AC_LANG_SOURCE([ +#define _LARGEFILE64_SOURCE +#ifdef HAVE_UNISTD_H +#include <unistd.h> +#endif +extern off64_t x1; +]), + ax_cv_check_off64_t="_LARGEFILE64_SOURCE", + ax_cv_check_off64_t="no" +)dnl + + else + ax_cv_check_off64_t=yes + fi + ])dnl + +if test "x$ax_cv_check_off64_t" = "x_LARGEFILE64_SOURCE" +then + AC_DEFINE(_LARGEFILE64_SOURCE) + AC_DEFINE(HAVE_OFF64_T) +elif test "x$ax_cv_check_off64_t" = "xyes" +then + AC_DEFINE(HAVE_OFF64_T) +fi +])dnl + diff --git a/outchannel.c b/outchannel.c index 4f8abb32..74c18218 100644 --- a/outchannel.c +++ b/outchannel.c @@ -106,7 +106,6 @@ static rsRetVal get_Field(uchar **pp, uchar **pField) p = *pp; CHKiRet(cstrConstruct(&pStrB)); - rsCStrSetAllocIncrement(pStrB, 32); /* copy the field */ while(*p && *p != ' ' && *p != ',') { @@ -175,7 +174,6 @@ static inline rsRetVal get_restOfLine(uchar **pp, uchar **pBuf) p = *pp; CHKiRet(cstrConstruct(&pStrB)); - rsCStrSetAllocIncrement(pStrB, 32); /* copy the field */ while(*p) { @@ -256,7 +256,7 @@ rsRetVal parsDelimCStr(rsParsObj *pThis, cstr_t **ppCStr, char cDelim, int bTrim pC = rsCStrGetBufBeg(pThis->pCStr) + pThis->iCurrPos; while(pThis->iCurrPos < rsCStrLen(pThis->pCStr) && *pC != cDelim) { - CHKiRet(rsCStrAppendChar(pCStr, bConvLower ? tolower(*pC) : *pC)); + CHKiRet(cstrAppendChar(pCStr, bConvLower ? tolower(*pC) : *pC)); ++pThis->iCurrPos; ++pC; } @@ -271,7 +271,7 @@ rsRetVal parsDelimCStr(rsParsObj *pThis, cstr_t **ppCStr, char cDelim, int bTrim CHKiRet(cstrFinalize(pCStr)); if(bTrimTrailing) { - CHKiRet(rsCStrTrimTrailingWhiteSpace(pCStr)); + CHKiRet(cstrTrimTrailingWhiteSpace(pCStr)); } /* done! */ @@ -313,23 +313,23 @@ rsRetVal parsQuotedCStr(rsParsObj *pThis, cstr_t **ppCStr) pC = rsCStrGetBufBeg(pThis->pCStr) + pThis->iCurrPos; /* OK, we most probably can obtain a value... */ - CHKiRet(rsCStrConstruct(&pCStr)); + CHKiRet(cstrConstruct(&pCStr)); - while(pThis->iCurrPos < rsCStrLen(pThis->pCStr)) { + while(pThis->iCurrPos < cstrLen(pThis->pCStr)) { if(*pC == '"') { break; /* we are done! */ } else if(*pC == '\\') { ++pThis->iCurrPos; ++pC; - if(pThis->iCurrPos < rsCStrLen(pThis->pCStr)) { + if(pThis->iCurrPos < cstrLen(pThis->pCStr)) { /* in this case, we copy the escaped character * to the output buffer (but do not rely on this, * we might later introduce other things, like \007! */ - CHKiRet(rsCStrAppendChar(pCStr, *pC)); + CHKiRet(cstrAppendChar(pCStr, *pC)); } } else { /* regular character */ - CHKiRet(rsCStrAppendChar(pCStr, *pC)); + CHKiRet(cstrAppendChar(pCStr, *pC)); } ++pThis->iCurrPos; ++pC; @@ -339,7 +339,7 @@ rsRetVal parsQuotedCStr(rsParsObj *pThis, cstr_t **ppCStr) ++pThis->iCurrPos; /* 'eat' trailing quote */ } else { /* error - improperly quoted string! */ - rsCStrDestruct(&pCStr); + cstrDestruct(&pCStr); ABORT_FINALIZE(RS_RET_MISSING_TRAIL_QUOTE); } @@ -352,7 +352,7 @@ rsRetVal parsQuotedCStr(rsParsObj *pThis, cstr_t **ppCStr) finalize_it: if(iRet != RS_RET_OK) { if(pCStr != NULL) - rsCStrDestruct(&pCStr); + cstrDestruct(&pCStr); } RETiRet; diff --git a/plugins/imdiag/imdiag.c b/plugins/imdiag/imdiag.c index 51f319ca..bf972191 100644 --- a/plugins/imdiag/imdiag.c +++ b/plugins/imdiag/imdiag.c @@ -66,10 +66,14 @@ DEFobjCurrIf(net) DEFobjCurrIf(netstrm) DEFobjCurrIf(errmsg) DEFobjCurrIf(datetime) +DEFobjCurrIf(prop) /* Module static data */ static tcpsrv_t *pOurTcpsrv = NULL; /* our TCP server(listener) TODO: change for multiple instances */ static permittedPeers_t *pPermPeersRoot = NULL; +static prop_t *pInputName = NULL; /* there is only one global inputName for all messages generated by this input */ +static prop_t *pRcvDummy = NULL; +static prop_t *pRcvIPDummy = NULL; /* config settings */ @@ -205,14 +209,13 @@ doInjectMsg(int iNum) datetime.getCurrTime(&stTime, &ttGenTime); /* we now create our own message object and submit it to the queue */ CHKiRet(msgConstructWithTime(&pMsg, &stTime, ttGenTime)); - CHKmalloc(pMsg->pszRawMsg = ustrdup(szMsg)); - pMsg->iLenRawMsg = ustrlen(szMsg); - MsgSetInputName(pMsg, UCHAR_CONSTANT("imdiag"), sizeof("imdiag")-1); + MsgSetRawMsg(pMsg, (char*) szMsg, ustrlen(szMsg)); + MsgSetInputName(pMsg, pInputName); MsgSetFlowControlType(pMsg, eFLOWCTL_NO_DELAY); pMsg->msgFlags = NEEDS_PARSING | PARSE_HOSTNAME; pMsg->bParseHOSTNAME = 1; - MsgSetRcvFrom(pMsg, UCHAR_CONSTANT("127.0.0.1")); /* TODO: way may use the real sender here... */ - CHKiRet(MsgSetRcvFromIP(pMsg, UCHAR_CONSTANT("127.0.0.1"))); + MsgSetRcvFrom(pMsg, pRcvDummy); + CHKiRet(MsgSetRcvFromIP(pMsg, pRcvIPDummy)); CHKiRet(submitMsg(pMsg)); finalize_it: @@ -377,13 +380,31 @@ CODESTARTwillRun /* first apply some config settings */ if(pOurTcpsrv == NULL) ABORT_FINALIZE(RS_RET_NO_RUN); + /* we need to create the inputName property (only once during our lifetime) */ + CHKiRet(prop.Construct(&pInputName)); + CHKiRet(prop.SetString(pInputName, UCHAR_CONSTANT("imdiag"), sizeof("imdiag") - 1)); + CHKiRet(prop.ConstructFinalize(pInputName)); + + CHKiRet(prop.Construct(&pRcvDummy)); + CHKiRet(prop.SetString(pRcvDummy, UCHAR_CONSTANT("127.0.0.1"), sizeof("127.0.0.1") - 1)); + CHKiRet(prop.ConstructFinalize(pRcvDummy)); + + CHKiRet(prop.Construct(&pRcvIPDummy)); + CHKiRet(prop.SetString(pRcvIPDummy, UCHAR_CONSTANT("127.0.0.1"), sizeof("127.0.0.1") - 1)); + CHKiRet(prop.ConstructFinalize(pRcvIPDummy)); + finalize_it: ENDwillRun BEGINafterRun CODESTARTafterRun - /* do cleanup here */ + if(pInputName != NULL) + prop.Destruct(&pInputName); + if(pRcvDummy != NULL) + prop.Destruct(&pRcvDummy); + if(pRcvIPDummy != NULL) + prop.Destruct(&pRcvIPDummy); ENDafterRun @@ -403,6 +424,7 @@ CODESTARTmodExit objRelease(tcpsrv, LM_TCPSRV_FILENAME); objRelease(errmsg, CORE_COMPONENT); objRelease(datetime, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); ENDmodExit @@ -440,6 +462,7 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(tcpsrv, LM_TCPSRV_FILENAME)); CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(datetime, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); /* register config file handlers */ CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("imdiagserverrun"), 0, eCmdHdlrGetWord, diff --git a/plugins/imfile/imfile.c b/plugins/imfile/imfile.c index 1ae6e69a..3981f9f7 100644 --- a/plugins/imfile/imfile.c +++ b/plugins/imfile/imfile.c @@ -46,6 +46,7 @@ #include "glbl.h" #include "datetime.h" #include "unicode-helper.h" +#include "prop.h" MODULE_TYPE_INPUT /* must be present for input modules, do not remove */ @@ -56,10 +57,13 @@ DEF_IMOD_STATIC_DATA /* must be present, starts static data */ DEFobjCurrIf(errmsg) DEFobjCurrIf(glbl) DEFobjCurrIf(datetime) +DEFobjCurrIf(strm) +DEFobjCurrIf(prop) typedef struct fileInfo_s { uchar *pszFileName; uchar *pszTag; + size_t lenTag; uchar *pszStateFile; /* file in which state between runs is to be stored */ int iFacility; int iSeverity; @@ -79,6 +83,7 @@ static int iFilPtr = 0; /* number of files to be monitored; pointer to next fre #define MAX_INPUT_FILES 100 static fileInfo_t files[MAX_INPUT_FILES]; +static prop_t *pInputName = NULL; /* there is only one global inputName for all messages generated by this input */ /* enqueue the read file line as a message. The provided string is * not freed - thuis must be done by the caller. @@ -95,11 +100,11 @@ static rsRetVal enqLine(fileInfo_t *pInfo, cstr_t *cstrLine) CHKiRet(msgConstruct(&pMsg)); MsgSetFlowControlType(pMsg, eFLOWCTL_FULL_DELAY); - MsgSetInputName(pMsg, UCHAR_CONSTANT("imfile"), sizeof("imfile")-1); - MsgSetRawMsg(pMsg, (char*)rsCStrGetSzStr(cstrLine)); - MsgSetMSG(pMsg, (char*)rsCStrGetSzStr(cstrLine)); - MsgSetHOSTNAME(pMsg, glbl.GetLocalHostName()); - MsgSetTAG(pMsg, (char*)pInfo->pszTag); + MsgSetInputName(pMsg, pInputName); + MsgSetRawMsg(pMsg, (char*)rsCStrGetSzStr(cstrLine), cstrLen(cstrLine)); + MsgSetMSGoffs(pMsg, 0); /* we do not have a header... */ + MsgSetHOSTNAME(pMsg, glbl.GetLocalHostName(), ustrlen(glbl.GetLocalHostName())); + MsgSetTAG(pMsg, pInfo->pszTag, pInfo->lenTag); pMsg->iFacility = LOG_FAC(pInfo->iFacility); pMsg->iSeverity = LOG_PRI(pInfo->iSeverity); pMsg->bParseHOSTNAME = 0; @@ -138,16 +143,16 @@ openFile(fileInfo_t *pThis) /* If we reach this point, we have a .si file */ - CHKiRet(strmConstruct(&psSF)); - CHKiRet(strmSettOperationsMode(psSF, STREAMMODE_READ)); - CHKiRet(strmSetsType(psSF, STREAMTYPE_FILE_SINGLE)); - CHKiRet(strmSetFName(psSF, pszSFNam, lenSFNam)); - CHKiRet(strmConstructFinalize(psSF)); + CHKiRet(strm.Construct(&psSF)); + CHKiRet(strm.SettOperationsMode(psSF, STREAMMODE_READ)); + CHKiRet(strm.SetsType(psSF, STREAMTYPE_FILE_SINGLE)); + CHKiRet(strm.SetFName(psSF, pszSFNam, lenSFNam)); + CHKiRet(strm.ConstructFinalize(psSF)); /* read back in the object */ CHKiRet(obj.Deserialize(&pThis->pStrm, (uchar*) "strm", psSF, NULL, pThis)); - CHKiRet(strmSeekCurrOffs(pThis->pStrm)); + CHKiRet(strm.SeekCurrOffs(pThis->pStrm)); /* OK, we could successfully read the file, so we now can request that it be deleted. * If we need it again, it will be written on the next shutdown. @@ -156,14 +161,14 @@ openFile(fileInfo_t *pThis) finalize_it: if(psSF != NULL) - strmDestruct(&psSF); + strm.Destruct(&psSF); if(iRet != RS_RET_OK) { - CHKiRet(strmConstruct(&pThis->pStrm)); - CHKiRet(strmSettOperationsMode(pThis->pStrm, STREAMMODE_READ)); - CHKiRet(strmSetsType(pThis->pStrm, STREAMTYPE_FILE_MONITOR)); - CHKiRet(strmSetFName(pThis->pStrm, pThis->pszFileName, strlen((char*) pThis->pszFileName))); - CHKiRet(strmConstructFinalize(pThis->pStrm)); + CHKiRet(strm.Construct(&pThis->pStrm)); + CHKiRet(strm.SettOperationsMode(pThis->pStrm, STREAMMODE_READ)); + CHKiRet(strm.SetsType(pThis->pStrm, STREAMTYPE_FILE_MONITOR)); + CHKiRet(strm.SetFName(pThis->pStrm, pThis->pszFileName, strlen((char*) pThis->pszFileName))); + CHKiRet(strm.ConstructFinalize(pThis->pStrm)); } RETiRet; @@ -202,7 +207,7 @@ static rsRetVal pollFile(fileInfo_t *pThis, int *pbHadFileData) /* loop below will be exited when strmReadLine() returns EOF */ while(1) { - CHKiRet(strmReadLine(pThis->pStrm, &pCStr)); + CHKiRet(strm.ReadLine(pThis->pStrm, &pCStr)); *pbHadFileData = 1; /* this is just a flag, so set it and forget it */ CHKiRet(enqLine(pThis, pCStr)); /* process line */ rsCStrDestruct(&pCStr); /* discard string (must be done by us!) */ @@ -334,6 +339,11 @@ CODESTARTwillRun ABORT_FINALIZE(RS_RET_NO_RUN); } + /* we need to create the inputName property (only once during our lifetime) */ + CHKiRet(prop.Construct(&pInputName)); + CHKiRet(prop.SetString(pInputName, UCHAR_CONSTANT("imfile"), sizeof("imfile") - 1)); + CHKiRet(prop.ConstructFinalize(pInputName)); + finalize_it: ENDwillRun @@ -354,23 +364,22 @@ persistStrmState(fileInfo_t *pInfo) ASSERT(pInfo != NULL); /* TODO: create a function persistObj in obj.c? */ - CHKiRet(strmConstruct(&psSF)); + CHKiRet(strm.Construct(&psSF)); lenDir = strlen((char*)glbl.GetWorkDir()); if(lenDir > 0) - CHKiRet(strmSetDir(psSF, glbl.GetWorkDir(), lenDir)); - CHKiRet(strmSettOperationsMode(psSF, STREAMMODE_WRITE)); - CHKiRet(strmSetiAddtlOpenFlags(psSF, O_TRUNC)); - CHKiRet(strmSetsType(psSF, STREAMTYPE_FILE_SINGLE)); - CHKiRet(strmSetFName(psSF, pInfo->pszStateFile, strlen((char*) pInfo->pszStateFile))); - CHKiRet(strmConstructFinalize(psSF)); + CHKiRet(strm.SetDir(psSF, glbl.GetWorkDir(), lenDir)); + CHKiRet(strm.SettOperationsMode(psSF, STREAMMODE_WRITE_TRUNC)); + CHKiRet(strm.SetsType(psSF, STREAMTYPE_FILE_SINGLE)); + CHKiRet(strm.SetFName(psSF, pInfo->pszStateFile, strlen((char*) pInfo->pszStateFile))); + CHKiRet(strm.ConstructFinalize(psSF)); - CHKiRet(strmSerialize(pInfo->pStrm, psSF)); + CHKiRet(strm.Serialize(pInfo->pStrm, psSF)); - CHKiRet(strmDestruct(&psSF)); + CHKiRet(strm.Destruct(&psSF)); finalize_it: if(psSF != NULL) - strmDestruct(&psSF); + strm.Destruct(&psSF); RETiRet; } @@ -390,9 +399,12 @@ CODESTARTafterRun for(i = 0 ; i < iFilPtr ; ++i) { if(files[i].pStrm != NULL) { /* stream open? */ persistStrmState(&files[i]); - strmDestruct(&(files[i].pStrm)); + strm.Destruct(&(files[i].pStrm)); } } + + if(pInputName != NULL) + prop.Destruct(&pInputName); ENDafterRun @@ -403,9 +415,11 @@ ENDafterRun BEGINmodExit CODESTARTmodExit /* release objects we used */ + objRelease(strm, CORE_COMPONENT); objRelease(datetime, CORE_COMPONENT); objRelease(glbl, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); ENDmodExit @@ -473,6 +487,7 @@ static rsRetVal addMonitor(void __attribute__((unused)) *pVal, uchar *pNewVal) ABORT_FINALIZE(RS_RET_CONFIG_ERROR); } else { pThis->pszTag = (uchar*) strdup((char*) pszFileTag); + pThis->lenTag = ustrlen(pThis->pszTag); } if(pszStateFile == NULL) { @@ -514,6 +529,8 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); CHKiRet(objUse(datetime, CORE_COMPONENT)); + CHKiRet(objUse(strm, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputfilename", 0, eCmdHdlrGetWord, NULL, &pszFileName, STD_LOADABLE_MODULE_ID)); diff --git a/plugins/imklog/bsd.c b/plugins/imklog/bsd.c index 090c4e9b..cecf21c4 100644 --- a/plugins/imklog/bsd.c +++ b/plugins/imklog/bsd.c @@ -83,6 +83,11 @@ static int fklog = -1; /* /dev/klog */ # define _PATH_KLOG "/dev/klog" #endif +static uchar *GetPath(void) +{ + return pszPath ? pszPath : (uchar*) _PATH_KLOG; +} + /* open the kernel log - will be called inside the willRun() imklog * entry point. -- rgerhards, 2008-04-09 */ @@ -91,9 +96,9 @@ klogWillRun(void) { DEFiRet; - fklog = open(_PATH_KLOG, O_RDONLY, 0); + fklog = open((char*)GetPath(), O_RDONLY, 0); if (fklog < 0) { - dbgprintf("can't open %s (%d)\n", _PATH_KLOG, errno); + dbgprintf("can't open %s (%d)\n", GetPath(), errno); iRet = RS_RET_ERR; // TODO: better error code } @@ -147,7 +152,7 @@ readklog(void) break; } - for (p = pRcv; (q = strchr(p, '\n')) != NULL; p = q + 1) { + for (p = (char*)pRcv; (q = strchr(p, '\n')) != NULL; p = q + 1) { *q = '\0'; Syslog(LOG_INFO, (uchar*) p); } diff --git a/plugins/imklog/imklog.c b/plugins/imklog/imklog.c index 420ebbf1..7994c5eb 100644 --- a/plugins/imklog/imklog.c +++ b/plugins/imklog/imklog.c @@ -18,7 +18,7 @@ * Please note that this file replaces the klogd daemon that was * also present in pre-v3 versions of rsyslog. * - * Copyright (C) 2008 by Rainer Gerhards and Adiscon GmbH + * Copyright (C) 2008, 2009 by Rainer Gerhards and Adiscon GmbH * * This file is part of rsyslog. * @@ -44,6 +44,7 @@ #include <string.h> #include <stdarg.h> #include <ctype.h> +#include <stdlib.h> #include "dirty.h" #include "cfsysline.h" @@ -53,6 +54,7 @@ #include "datetime.h" #include "imklog.h" #include "glbl.h" +#include "prop.h" #include "unicode-helper.h" MODULE_TYPE_INPUT @@ -61,6 +63,7 @@ MODULE_TYPE_INPUT DEF_IMOD_STATIC_DATA DEFobjCurrIf(datetime) DEFobjCurrIf(glbl) +DEFobjCurrIf(prop) /* configuration settings */ int dbgPrintSymbols = 0; /* this one is extern so the helpers can access it! */ @@ -69,6 +72,8 @@ int use_syscall = 0; int symbol_lookup = 0; /* on recent kernels > 2.6, the kernel does this */ int bPermitNonKernel = 0; /* permit logging of messages not having LOG_KERN facility */ int iFacilIntMsg; /* the facility to use for internal messages (set by driver) */ +uchar *pszPath = NULL; +int console_log_level = -1; /* TODO: configuration for the following directives must be implemented. It * was not done yet because we either do not yet have a config handler for * that type or I thought it was acceptable to push it to a later stage when @@ -76,9 +81,11 @@ int iFacilIntMsg; /* the facility to use for internal messages (set by driver) * * changes resulting from that). -- rgerhards, 2007-12-20 */ char *symfile = NULL; -int console_log_level = -1; +static prop_t *pInputName = NULL; /* there is only one global inputName for all messages generated by this module */ +static prop_t *pLocalHostIP = NULL; /* a pseudo-constant propterty for 127.0.0.1 */ + /* enqueue the the kernel message into the message queue. * The provided msg string is not freed - thus must be done * by the caller. @@ -95,13 +102,13 @@ enqMsg(uchar *msg, uchar* pszTag, int iFacility, int iSeverity) CHKiRet(msgConstruct(&pMsg)); MsgSetFlowControlType(pMsg, eFLOWCTL_LIGHT_DELAY); - MsgSetInputName(pMsg, UCHAR_CONSTANT("imklog"), sizeof("imklog")-1); - MsgSetRawMsg(pMsg, (char*)msg); - MsgSetMSG(pMsg, (char*)msg); - MsgSetRcvFrom(pMsg, glbl.GetLocalHostName()); - MsgSetRcvFromIP(pMsg, (uchar*)"127.0.0.1"); - MsgSetHOSTNAME(pMsg, glbl.GetLocalHostName()); - MsgSetTAG(pMsg, (char*)pszTag); + MsgSetInputName(pMsg, pInputName); + MsgSetRawMsgWOSize(pMsg, (char*)msg); + MsgSetMSGoffs(pMsg, 0); /* we do not have a header... */ + MsgSetRcvFrom(pMsg, glbl.GetLocalHostNameProp()); + MsgSetRcvFromIP(pMsg, pLocalHostIP); + MsgSetHOSTNAME(pMsg, glbl.GetLocalHostName(), ustrlen(glbl.GetLocalHostName())); + MsgSetTAG(pMsg, pszTag, ustrlen(pszTag)); pMsg->iFacility = LOG_FAC(iFacility); pMsg->iSeverity = LOG_PRI(iSeverity); pMsg->bParseHOSTNAME = 0; @@ -228,13 +235,23 @@ ENDrunInput BEGINwillRun CODESTARTwillRun + /* we need to create the inputName property (only once during our lifetime) */ + CHKiRet(prop.CreateStringProp(&pInputName, UCHAR_CONSTANT("imklog"), sizeof("imklog") - 1)); + CHKiRet(prop.CreateStringProp(&pLocalHostIP, UCHAR_CONSTANT("127.0.0.1"), sizeof("127.0.0.1") - 1)); + iRet = klogWillRun(); +finalize_it: ENDwillRun BEGINafterRun CODESTARTafterRun iRet = klogAfterRun(); + + if(pInputName != NULL) + prop.Destruct(&pInputName); + if(pLocalHostIP != NULL) + prop.Destruct(&pLocalHostIP); ENDafterRun @@ -243,6 +260,9 @@ CODESTARTmodExit /* release objects we used */ objRelease(glbl, CORE_COMPONENT); objRelease(datetime, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); + if(pszPath != NULL) + free(pszPath); ENDmodExit @@ -259,6 +279,10 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a symfile = NULL; symbol_lookup = 0; bPermitNonKernel = 0; + if(pszPath != NULL) { + free(pszPath); + pszPath = NULL; + } iFacilIntMsg = klogFacilIntMsg(); return RS_RET_OK; } @@ -269,14 +293,17 @@ CODESTARTmodInit CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(datetime, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); iFacilIntMsg = klogFacilIntMsg(); CHKiRet(omsdRegCFSLineHdlr((uchar *)"debugprintkernelsymbols", 0, eCmdHdlrBinary, NULL, &dbgPrintSymbols, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"klogpath", 0, eCmdHdlrGetWord, NULL, &pszPath, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"klogsymbollookup", 0, eCmdHdlrBinary, NULL, &symbol_lookup, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"klogsymbolstwice", 0, eCmdHdlrBinary, NULL, &symbols_twice, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"klogusesyscallinterface", 0, eCmdHdlrBinary, NULL, &use_syscall, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"klogpermitnonkernelfacility", 0, eCmdHdlrBinary, NULL, &bPermitNonKernel, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"klogconsoleloglevel", 0, eCmdHdlrInt, NULL, &console_log_level, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"kloginternalmsgfacility", 0, eCmdHdlrFacility, NULL, &iFacilIntMsg, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID)); ENDmodInit diff --git a/plugins/imklog/imklog.h b/plugins/imklog/imklog.h index 37bd58b0..c183026d 100644 --- a/plugins/imklog/imklog.h +++ b/plugins/imklog/imklog.h @@ -52,6 +52,7 @@ extern int symbol_lookup; extern char *symfile; extern int console_log_level; extern int dbgPrintSymbols; +extern uchar *pszPath; /* the functions below may be called by the drivers */ rsRetVal imklogLogIntMsg(int priority, char *fmt, ...) __attribute__((format(printf,2, 3))); diff --git a/plugins/imklog/linux.c b/plugins/imklog/linux.c index 0dd4320d..727708a5 100644 --- a/plugins/imklog/linux.c +++ b/plugins/imklog/linux.c @@ -37,6 +37,7 @@ #include "msg.h" #include "module-template.h" #include "imklog.h" +#include "unicode-helper.h" /* Includes. */ @@ -84,19 +85,21 @@ static enum LOGSRC {none, proc, kernel} logsrc; extern int ksyslog(int type, char *buf, int len); +static uchar *GetPath(void) +{ + return pszPath ? pszPath : UCHAR_CONSTANT(_PATH_KLOG); +} + static void CloseLogSrc(void) { - /* Turn on logging of messages to console, but only if we had the -c - * option -- rgerhards, 2007-08-01 - */ - if (console_log_level != -1) + /* Turn on logging of messages to console, but only if a log level was speficied */ + if(console_log_level != -1) ksyslog(7, NULL, 0); /* Shutdown the log sources. */ - switch ( logsrc ) - { + switch(logsrc) { case kernel: - ksyslog(0, 0, 0); + ksyslog(0, NULL, 0); imklogLogIntMsg(LOG_INFO, "Kernel logging (ksyslog) stopped."); break; case proc: @@ -135,7 +138,7 @@ static enum LOGSRC GetKernelLogSrc(void) * file system is available to get kernel messages from. */ if ( use_syscall || - ((stat(_PATH_KLOG, &sb) < 0) && (errno == ENOENT)) ) + ((stat((char*)GetPath(), &sb) < 0) && (errno == ENOENT)) ) { /* Initialize kernel logging. */ ksyslog(1, NULL, 0); @@ -144,14 +147,14 @@ static enum LOGSRC GetKernelLogSrc(void) return(kernel); } - if ( (kmsg = open(_PATH_KLOG, O_RDONLY|O_CLOEXEC)) < 0 ) + if ( (kmsg = open((char*)GetPath(), O_RDONLY|O_CLOEXEC)) < 0 ) { imklogLogIntMsg(LOG_ERR, "imklog: Cannot open proc file system, %d.\n", errno); - ksyslog(7, NULL, 0); /* TODO: check this, implement more */ + ksyslog(7, NULL, 0); return(none); } - imklogLogIntMsg(LOG_INFO, "imklog %s, log source = %s started.", VERSION, _PATH_KLOG); + imklogLogIntMsg(LOG_INFO, "imklog %s, log source = %s started.", VERSION, GetPath()); return(proc); } diff --git a/plugins/imrelp/imrelp.c b/plugins/imrelp/imrelp.c index 2255e643..9be38f8f 100644 --- a/plugins/imrelp/imrelp.c +++ b/plugins/imrelp/imrelp.c @@ -4,7 +4,7 @@ * * File begun on 2008-03-13 by RGerhards * - * Copyright 2008 Rainer Gerhards and Adiscon GmbH. + * Copyright 2008, 2009 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -43,15 +43,19 @@ #include "module-template.h" #include "net.h" #include "msg.h" +#include "unicode-helper.h" +#include "prop.h" MODULE_TYPE_INPUT /* static data */ DEF_IMOD_STATIC_DATA DEFobjCurrIf(net) +DEFobjCurrIf(prop) /* Module static data */ static relpEngine_t *pRelpEngine; /* our relp engine */ +static prop_t *pInputName = NULL; /* there is only one global inputName for all messages generated by this module */ /* config settings */ @@ -77,15 +81,15 @@ isPermittedHost(struct sockaddr *addr, char *fromHostFQDN, void __attribute__((u * are different from our rsRetVal. So we can simply use our own iRet system * to fulfill the requirement. * rgerhards, 2008-03-21 - * TODO: we currently do not receive the remote hosts's IP. As a work-around, we - * use "???" for the time being. -- rgerhards, 2008-05-16 + * Note: librelp 1.0.0 is required in order to receive the IP address, otherwise + * we will only see the hostname (twice). -- rgerhards, 2009-10-14 */ static relpRetVal -onSyslogRcv(uchar *pHostname, uchar __attribute__((unused)) *pIP, uchar *pMsg, size_t lenMsg) +onSyslogRcv(uchar *pHostname, uchar *pIP, uchar *pMsg, size_t lenMsg) { DEFiRet; - parseAndSubmitMessage(pHostname, (uchar*) "[unset]", pMsg, lenMsg, PARSE_HOSTNAME, - eFLOWCTL_LIGHT_DELAY, (uchar*)"imrelp", NULL, 0); + parseAndSubmitMessage(pHostname, pIP, pMsg, lenMsg, PARSE_HOSTNAME, + eFLOWCTL_LIGHT_DELAY, pInputName, NULL, 0); RETiRet; } @@ -130,6 +134,11 @@ CODESTARTwillRun //net.PrintAllowedSenders(2); /* TCP */ if(pRelpEngine == NULL) ABORT_FINALIZE(RS_RET_NO_RUN); + + /* we need to create the inputName property (only once during our lifetime) */ + CHKiRet(prop.Construct(&pInputName)); + CHKiRet(prop.SetString(pInputName, UCHAR_CONSTANT("imrelp"), sizeof("imrelp") - 1)); + CHKiRet(prop.ConstructFinalize(pInputName)); finalize_it: ENDwillRun @@ -143,6 +152,9 @@ CODESTARTafterRun net.pAllowedSenders_TCP = NULL; } #endif + + if(pInputName != NULL) + prop.Destruct(&pInputName); ENDafterRun @@ -152,6 +164,7 @@ CODESTARTmodExit iRet = relpEngineDestruct(&pRelpEngine); /* release objects we used */ + objRelease(prop, CORE_COMPONENT); objRelease(net, LM_NET_FILENAME); ENDmodExit @@ -177,6 +190,7 @@ CODESTARTmodInit CODEmodInit_QueryRegCFSLineHdlr pRelpEngine = NULL; /* request objects we use */ + CHKiRet(objUse(prop, CORE_COMPONENT)); CHKiRet(objUse(net, LM_NET_FILENAME)); /* register config file handlers */ diff --git a/plugins/imtcp/imtcp.c b/plugins/imtcp/imtcp.c index 84e660bc..d122e976 100644 --- a/plugins/imtcp/imtcp.c +++ b/plugins/imtcp/imtcp.c @@ -61,6 +61,7 @@ #include "netstrm.h" #include "errmsg.h" #include "tcpsrv.h" +#include "ruleset.h" #include "net.h" /* for permittedPeers, may be removed when this is removed */ MODULE_TYPE_INPUT @@ -72,6 +73,7 @@ DEFobjCurrIf(tcps_sess) DEFobjCurrIf(net) DEFobjCurrIf(netstrm) DEFobjCurrIf(errmsg) +DEFobjCurrIf(ruleset) /* Module static data */ static tcpsrv_t *pOurTcpsrv = NULL; /* our TCP server(listener) TODO: change for multiple instances */ @@ -80,10 +82,13 @@ static permittedPeers_t *pPermPeersRoot = NULL; /* config settings */ static int iTCPSessMax = 200; /* max number of sessions */ +static int iTCPLstnMax = 20; /* max number of sessions */ static int iStrmDrvrMode = 0; /* mode for stream driver, driver-dependent (0 mostly means plain tcp) */ +static int bEmitMsgOnClose = 0; /* emit an informational message on close by remote peer */ static int iAddtlFrameDelim = TCPSRV_NO_ADDTL_DELIMITER; /* addtl frame delimiter, e.g. for netscreen, default none */ static uchar *pszStrmDrvrAuthMode = NULL; /* authentication mode to use */ static uchar *pszInputName = NULL; /* value for inputname property, NULL is OK and handled by core engine */ +static ruleset_t *pBindRuleset = NULL; /* ruleset to bind listener to (use system default if unspecified) */ /* callbacks */ @@ -157,6 +162,27 @@ finalize_it: } +/* accept a new ruleset to bind. Checks if it exists and complains, if not */ +static rsRetVal setRuleset(void __attribute__((unused)) *pVal, uchar *pszName) +{ + ruleset_t *pRuleset; + rsRetVal localRet; + DEFiRet; + + localRet = ruleset.GetRuleset(&pRuleset, pszName); + if(localRet == RS_RET_NOT_FOUND) { + errmsg.LogError(0, NO_ERRCODE, "error: ruleset '%s' not found - ignored", pszName); + } + CHKiRet(localRet); + pBindRuleset = pRuleset; + DBGPRINTF("imtcp current bind ruleset %p: '%s'\n", pRuleset, pszName); + +finalize_it: + free(pszName); /* no longer needed */ + RETiRet; +} + + static rsRetVal addTCPListener(void __attribute__((unused)) *pVal, uchar *pNewVal) { DEFiRet; @@ -164,6 +190,7 @@ static rsRetVal addTCPListener(void __attribute__((unused)) *pVal, uchar *pNewVa if(pOurTcpsrv == NULL) { CHKiRet(tcpsrv.Construct(&pOurTcpsrv)); CHKiRet(tcpsrv.SetSessMax(pOurTcpsrv, iTCPSessMax)); + CHKiRet(tcpsrv.SetLstnMax(pOurTcpsrv, iTCPLstnMax)); CHKiRet(tcpsrv.SetCBIsPermittedHost(pOurTcpsrv, isPermittedHost)); CHKiRet(tcpsrv.SetCBRcvData(pOurTcpsrv, doRcvData)); CHKiRet(tcpsrv.SetCBOpenLstnSocks(pOurTcpsrv, doOpenLstnSocks)); @@ -171,6 +198,7 @@ static rsRetVal addTCPListener(void __attribute__((unused)) *pVal, uchar *pNewVa CHKiRet(tcpsrv.SetCBOnErrClose(pOurTcpsrv, onErrClose)); CHKiRet(tcpsrv.SetDrvrMode(pOurTcpsrv, iStrmDrvrMode)); CHKiRet(tcpsrv.SetAddtlFrameDelim(pOurTcpsrv, iAddtlFrameDelim)); + CHKiRet(tcpsrv.SetNotificationOnRemoteClose(pOurTcpsrv, bEmitMsgOnClose)); /* now set optional params, but only if they were actually configured */ if(pszStrmDrvrAuthMode != NULL) { CHKiRet(tcpsrv.SetDrvrAuthMode(pOurTcpsrv, pszStrmDrvrAuthMode)); @@ -180,7 +208,8 @@ static rsRetVal addTCPListener(void __attribute__((unused)) *pVal, uchar *pNewVa } } - /* initialized, now add socket */ + /* initialized, now add socket and listener params */ + CHKiRet(tcpsrv.SetRuleset(pOurTcpsrv, pBindRuleset)); CHKiRet(tcpsrv.SetInputName(pOurTcpsrv, pszInputName == NULL ? UCHAR_CONSTANT("imtcp") : pszInputName)); tcpsrv.configureTCPListen(pOurTcpsrv, pNewVal); @@ -240,6 +269,7 @@ CODESTARTmodExit objRelease(tcps_sess, LM_TCPSRV_FILENAME); objRelease(tcpsrv, LM_TCPSRV_FILENAME); objRelease(errmsg, CORE_COMPONENT); + objRelease(ruleset, CORE_COMPONENT); ENDmodExit @@ -247,16 +277,14 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __attribute__((unused)) *pVal) { iTCPSessMax = 200; + iTCPLstnMax = 20; iStrmDrvrMode = 0; + bEmitMsgOnClose = 0; iAddtlFrameDelim = TCPSRV_NO_ADDTL_DELIMITER; - if(pszInputName != NULL) { - free(pszInputName); - pszInputName = NULL; - } - if(pszStrmDrvrAuthMode != NULL) { - free(pszStrmDrvrAuthMode); - pszStrmDrvrAuthMode = NULL; - } + free(pszInputName); + pszInputName = NULL; + free(pszStrmDrvrAuthMode); + pszStrmDrvrAuthMode = NULL; return RS_RET_OK; } @@ -279,12 +307,17 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(tcps_sess, LM_TCPSRV_FILENAME)); CHKiRet(objUse(tcpsrv, LM_TCPSRV_FILENAME)); CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(ruleset, CORE_COMPONENT)); /* register config file handlers */ CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpserverrun"), 0, eCmdHdlrGetWord, addTCPListener, NULL, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpmaxsessions"), 0, eCmdHdlrInt, NULL, &iTCPSessMax, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpmaxlisteners"), 0, eCmdHdlrInt, + NULL, &iTCPLstnMax, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpservernotifyonconnectionclose"), 0, + eCmdHdlrBinary, NULL, &bEmitMsgOnClose, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpserverstreamdrivermode"), 0, eCmdHdlrInt, NULL, &iStrmDrvrMode, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpserverstreamdriverauthmode"), 0, @@ -295,6 +328,8 @@ CODEmodInit_QueryRegCFSLineHdlr NULL, &iAddtlFrameDelim, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpserverinputname"), 0, eCmdHdlrGetWord, NULL, &pszInputName, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("inputtcpserverbindruleset"), 0, + eCmdHdlrGetWord, setRuleset, NULL, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr(UCHAR_CONSTANT("resetconfigvariables"), 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID)); ENDmodInit diff --git a/plugins/imudp/imudp.c b/plugins/imudp/imudp.c index 6f4a6384..f8555f00 100644 --- a/plugins/imudp/imudp.c +++ b/plugins/imudp/imudp.c @@ -43,6 +43,7 @@ #include "msg.h" #include "parser.h" #include "datetime.h" +#include "prop.h" #include "unicode-helper.h" MODULE_TYPE_INPUT @@ -55,6 +56,7 @@ DEFobjCurrIf(errmsg) DEFobjCurrIf(glbl) DEFobjCurrIf(net) DEFobjCurrIf(datetime) +DEFobjCurrIf(prop) static int iMaxLine; /* maximum UDP message size supported */ static time_t ttLastDiscard = 0; /* timestamp when a message from a non-permitted sender was last discarded @@ -68,6 +70,8 @@ static uchar *pRcvBuf = NULL; /* receive buffer (for a single packet). We use a * it so that we can check available memory in willRun() and request * termination if we can not get it. -- rgerhards, 2007-12-27 */ +static prop_t *pInputName = NULL; /* our inputName currently is always "imudp", and this will hold it */ +// TODO: static ruleset_t *pBindRuleset = NULL; /* ruleset to bind listener to (use system default if unspecified) */ #define TIME_REQUERY_DFLT 2 static int iTimeRequery = TIME_REQUERY_DFLT;/* how often is time to be queried inside tight recv loop? 0=always */ @@ -97,7 +101,7 @@ static rsRetVal addListner(void __attribute__((unused)) *pVal, uchar *pNewVal) else bindAddr = pszBindAddr; - dbgprintf("Trying to open syslog UDP ports at %s:%s.\n", + DBGPRINTF("Trying to open syslog UDP ports at %s:%s.\n", (bindAddr == NULL) ? (uchar*)"*" : bindAddr, pNewVal); newSocks = net.create_udp_socket(bindAddr, (pNewVal == NULL || *pNewVal == '\0') ? (uchar*) "514" : pNewVal, 1); @@ -137,6 +141,30 @@ finalize_it: } +#if 0 /* TODO: implement when tehre is time, requires restructure of socket array! */ +/* accept a new ruleset to bind. Checks if it exists and complains, if not */ +static rsRetVal +setRuleset(void __attribute__((unused)) *pVal, uchar *pszName) +{ + ruleset_t *pRuleset; + rsRetVal localRet; + DEFiRet; + + localRet = ruleset.GetRuleset(&pRuleset, pszName); + if(localRet == RS_RET_NOT_FOUND) { + errmsg.LogError(0, NO_ERRCODE, "error: ruleset '%s' not found - ignored", pszName); + } + CHKiRet(localRet); + pBindRuleset = pRuleset; + DBGPRINTF("imudp current bind ruleset %p: '%s'\n", pRuleset, pszName); + +finalize_it: + free(pszName); /* no longer needed */ + RETiRet; +} +#endif + + /* This function is a helper to runInput. I have extracted it * from the main loop just so that we do not have that large amount of code * in a single place. This function takes a socket and pulls messages from @@ -163,6 +191,8 @@ processSocket(int fd, struct sockaddr_storage *frominetPrev, int *pbIsPermitted, ssize_t lenRcvBuf; struct sockaddr_storage frominet; msg_t *pMsg; + prop_t *propFromHost = NULL; + prop_t *propFromHostIP = NULL; char errStr[1024]; iNbrTimeUsed = 0; @@ -211,7 +241,7 @@ processSocket(int fd, struct sockaddr_storage *frominetPrev, int *pbIsPermitted, } } - DBGPRINTF("recv(%d,%d)/%s,acl:%d,msg:%.80s\n", fd, (int) lenRcvBuf, fromHost, *pbIsPermitted, pRcvBuf); + DBGPRINTF("recv(%d,%d)/%s,acl:%d,msg:%s\n", fd, (int) lenRcvBuf, fromHost, *pbIsPermitted, pRcvBuf); if(*pbIsPermitted) { if((iTimeRequery == 0) || (iNbrTimeUsed++ % iTimeRequery) == 0) { @@ -219,22 +249,23 @@ processSocket(int fd, struct sockaddr_storage *frominetPrev, int *pbIsPermitted, } /* we now create our own message object and submit it to the queue */ CHKiRet(msgConstructWithTime(&pMsg, &stTime, ttGenTime)); - /* first trim the buffer to what we have actually received */ - CHKmalloc(pMsg->pszRawMsg = malloc(sizeof(uchar)* lenRcvBuf)); - memcpy(pMsg->pszRawMsg, pRcvBuf, lenRcvBuf); - pMsg->iLenRawMsg = lenRcvBuf; - MsgSetInputName(pMsg, UCHAR_CONSTANT("imudp"), sizeof("imudp")-1); + MsgSetRawMsg(pMsg, (char*)pRcvBuf, lenRcvBuf); + MsgSetInputName(pMsg, pInputName); MsgSetFlowControlType(pMsg, eFLOWCTL_NO_DELAY); pMsg->msgFlags = NEEDS_PARSING | PARSE_HOSTNAME; pMsg->bParseHOSTNAME = 1; - MsgSetRcvFrom(pMsg, fromHost); - CHKiRet(MsgSetRcvFromIP(pMsg, fromHostIP)); + MsgSetRcvFromStr(pMsg, fromHost, ustrlen(fromHost), &propFromHost); + CHKiRet(MsgSetRcvFromIPStr(pMsg, fromHostIP, ustrlen(fromHostIP), &propFromHostIP)); CHKiRet(submitMsg(pMsg)); } } - finalize_it: + if(propFromHost != NULL) + prop.Destruct(&propFromHost); + if(propFromHostIP != NULL) + prop.Destruct(&propFromHostIP); + RETiRet; } @@ -319,6 +350,11 @@ ENDrunInput /* initialize and return if will run or not */ BEGINwillRun CODESTARTwillRun + /* we need to create the inputName property (only once during our lifetime) */ + CHKiRet(prop.Construct(&pInputName)); + CHKiRet(prop.SetString(pInputName, UCHAR_CONSTANT("imudp"), sizeof("imudp") - 1)); + CHKiRet(prop.ConstructFinalize(pInputName)); + net.PrintAllowedSenders(1); /* UDP */ /* if we could not set up any listners, there is no point in running... */ @@ -346,6 +382,8 @@ CODESTARTafterRun free(pRcvBuf); pRcvBuf = NULL; } + if(pInputName != NULL) + prop.Destruct(&pInputName); ENDafterRun @@ -355,6 +393,7 @@ CODESTARTmodExit objRelease(errmsg, CORE_COMPONENT); objRelease(glbl, CORE_COMPONENT); objRelease(datetime, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); objRelease(net, LM_NET_FILENAME); ENDmodExit @@ -386,9 +425,14 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); CHKiRet(objUse(datetime, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); CHKiRet(objUse(net, LM_NET_FILENAME)); /* register config file handlers */ + /* TODO: add - but this requires more changes, no time right now... + CHKiRet(omsdRegCFSLineHdlr((uchar *)"udpserverbindruleset", 0, eCmdHdlrGetWord, + setRuleset, NULL, STD_LOADABLE_MODULE_ID)); + */ CHKiRet(omsdRegCFSLineHdlr((uchar *)"udpserverrun", 0, eCmdHdlrGetWord, addListner, NULL, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"udpserveraddress", 0, eCmdHdlrGetWord, diff --git a/plugins/imuxsock/imuxsock.c b/plugins/imuxsock/imuxsock.c index 1d88a2b5..5567a405 100644 --- a/plugins/imuxsock/imuxsock.c +++ b/plugins/imuxsock/imuxsock.c @@ -37,12 +37,14 @@ #include <sys/un.h> #include "dirty.h" #include "cfsysline.h" +#include "unicode-helper.h" #include "module-template.h" #include "srUtils.h" #include "errmsg.h" #include "net.h" #include "glbl.h" #include "msg.h" +#include "prop.h" MODULE_TYPE_INPUT @@ -66,7 +68,9 @@ MODULE_TYPE_INPUT DEF_IMOD_STATIC_DATA DEFobjCurrIf(errmsg) DEFobjCurrIf(glbl) +DEFobjCurrIf(prop) +static prop_t *pInputName = NULL; /* our inputName currently is always "imuxsock", and this will hold it */ static int startIndexUxLocalSockets; /* process funix from that index on (used to * suppress local logging. rgerhards 2005-08-01 * read-only after startup @@ -223,7 +227,7 @@ static rsRetVal readSocket(int fd, int iSock) parseAndSubmitMessage(funixHName[iSock] == NULL ? glbl.GetLocalHostName() : funixHName[iSock], (uchar*)"127.0.0.1", pRcv, iRcvd, funixParseHost[iSock] ? (funixFlags[iSock] | PARSE_HOSTNAME) : funixFlags[iSock], - funixFlowCtl[iSock], (uchar*)"imuxsock", NULL, 0); + funixFlowCtl[iSock], pInputName, NULL, 0); } else if (iRcvd < 0 && errno != EINTR) { char errStr[1024]; rs_strerror_r(errno, errStr, sizeof(errStr)); @@ -306,7 +310,12 @@ CODESTARTwillRun dbgprintf("Opened UNIX socket '%s' (fd %d).\n", funixn[i], funix[i]); } - RETiRet; + /* we need to create the inputName property (only once during our lifetime) */ + CHKiRet(prop.Construct(&pInputName)); + CHKiRet(prop.SetString(pInputName, UCHAR_CONSTANT("imuxsock"), sizeof("imuxsock") - 1)); + CHKiRet(prop.ConstructFinalize(pInputName)); + +finalize_it: ENDwillRun @@ -332,6 +341,9 @@ CODESTARTafterRun discardFunixn(); nfunix = 1; + + if(pInputName != NULL) + prop.Destruct(&pInputName); ENDafterRun @@ -339,6 +351,7 @@ BEGINmodExit CODESTARTmodExit objRelease(glbl, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); ENDmodExit @@ -375,6 +388,7 @@ CODESTARTmodInit CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); dbgprintf("imuxsock version %s initializing\n", PACKAGE_VERSION); diff --git a/plugins/omgssapi/omgssapi.c b/plugins/omgssapi/omgssapi.c index 361f657f..782ac22f 100644 --- a/plugins/omgssapi/omgssapi.c +++ b/plugins/omgssapi/omgssapi.c @@ -44,6 +44,7 @@ #include <pthread.h> #include <gssapi/gssapi.h> #include "dirty.h" +#include "conf.h" #include "syslogd-types.h" #include "srUtils.h" #include "net.h" @@ -376,7 +377,7 @@ CODESTARTtryResume ENDtryResume BEGINdoAction - char *psz; /* temporary buffering */ + char *psz = NULL; /* temporary buffering */ register unsigned l; int iMaxLine; CODESTARTdoAction @@ -430,11 +431,14 @@ CODESTARTdoAction * rgerhards, 2006-11-30 */ dbgprintf("Compression failed, sending uncompressed message\n"); + free(out); } else if(destLen+1 < l) { /* only use compression if there is a gain in using it! */ dbgprintf("there is gain in compression, so we do it\n"); psz = (char*) out; l = destLen + 1; /* take care for the "z" at message start! */ + } else { + free(out); } ++destLen; } @@ -449,6 +453,12 @@ CODESTARTdoAction break; } finalize_it: +# ifdef USE_NETZIP + if((psz != NULL) && (psz != (char*) ppString[0])) { + /* we need to free temporary buffer, alloced above - Naoya Nakazawa, 2010-01-11 */ + free(psz); + } +# endif ENDdoAction diff --git a/plugins/ommail/ommail.c b/plugins/ommail/ommail.c index 5faadce3..3a7669c9 100644 --- a/plugins/ommail/ommail.c +++ b/plugins/ommail/ommail.c @@ -44,7 +44,7 @@ #include <netdb.h> #include <time.h> #include <sys/socket.h> -#include "dirty.h" +#include "conf.h" #include "syslogd-types.h" #include "srUtils.h" #include "cfsysline.h" diff --git a/plugins/ommysql/ommysql.c b/plugins/ommysql/ommysql.c index ecf738a9..d6870a7b 100644 --- a/plugins/ommysql/ommysql.c +++ b/plugins/ommysql/ommysql.c @@ -36,7 +36,7 @@ #include <errno.h> #include <time.h> #include <mysql.h> -#include "dirty.h" +#include "conf.h" #include "syslogd-types.h" #include "srUtils.h" #include "template.h" diff --git a/plugins/ompgsql/createDB.sql b/plugins/ompgsql/createDB.sql index facdcaa0..2f72a0a6 100644 --- a/plugins/ompgsql/createDB.sql +++ b/plugins/ompgsql/createDB.sql @@ -1,4 +1,4 @@ -CREATE DATABASE "Syslog"; +CREATE DATABASE 'Syslog' WITH ENCODING 'SQL_ASCII'; \c Syslog; CREATE TABLE SystemEvents ( diff --git a/plugins/ompgsql/ompgsql.c b/plugins/ompgsql/ompgsql.c index 6daac1c7..eb774835 100644 --- a/plugins/ompgsql/ompgsql.c +++ b/plugins/ompgsql/ompgsql.c @@ -40,7 +40,7 @@ #include <errno.h> #include <time.h> #include <libpq-fe.h> -#include "dirty.h" +#include "conf.h" #include "syslogd-types.h" #include "srUtils.h" #include "template.h" diff --git a/plugins/omprog/omprog.c b/plugins/omprog/omprog.c index 2a078a6d..01fa7cea 100644 --- a/plugins/omprog/omprog.c +++ b/plugins/omprog/omprog.c @@ -36,7 +36,7 @@ #include <errno.h> #include <unistd.h> #include <wait.h> -#include "dirty.h" +#include "conf.h" #include "syslogd-types.h" #include "srUtils.h" #include "template.h" diff --git a/plugins/omrelp/omrelp.c b/plugins/omrelp/omrelp.c index 8d74c82f..d5ef8b4f 100644 --- a/plugins/omrelp/omrelp.c +++ b/plugins/omrelp/omrelp.c @@ -36,7 +36,7 @@ #include <errno.h> #include <ctype.h> #include <librelp.h> -#include "dirty.h" +#include "conf.h" #include "syslogd-types.h" #include "srUtils.h" #include "cfsysline.h" diff --git a/plugins/omsnmp/omsnmp.c b/plugins/omsnmp/omsnmp.c index 72fa8d64..4db60e62 100644 --- a/plugins/omsnmp/omsnmp.c +++ b/plugins/omsnmp/omsnmp.c @@ -36,7 +36,7 @@ #include <netdb.h> #include <ctype.h> #include <assert.h> -#include "dirty.h" +#include "conf.h" #include "syslogd-types.h" #include "cfsysline.h" #include "module-template.h" diff --git a/plugins/omstdout/omstdout.c b/plugins/omstdout/omstdout.c index 181895a4..b3ec6287 100644 --- a/plugins/omstdout/omstdout.c +++ b/plugins/omstdout/omstdout.c @@ -35,7 +35,7 @@ #include <signal.h> #include <errno.h> #include <unistd.h> -#include "dirty.h" +#include "conf.h" #include "syslogd-types.h" #include "srUtils.h" #include "template.h" @@ -50,11 +50,13 @@ MODULE_TYPE_OUTPUT DEF_OMOD_STATIC_DATA /* config variables */ -static int bUseArrayInterface; /* shall action use array instead of string template interface? */ +static int bUseArrayInterface = 0; /* shall action use array instead of string template interface? */ +static int bEnsureLFEnding = 1; /* shall action use array instead of string template interface? */ typedef struct _instanceData { int bUseArrayInterface; /* uses action use array instead of string template interface? */ + int bEnsureLFEnding; /* ensure that a linefeed is written at the end of EACH record (test aid for nettester) */ } instanceData; BEGINcreateInstance @@ -90,6 +92,7 @@ BEGINdoAction int iParam; int iBuf; char szBuf[65564]; + size_t len; CODESTARTdoAction if(pData->bUseArrayInterface) { /* if we use array passing, we need to put together a string @@ -120,7 +123,11 @@ CODESTARTdoAction } else { toWrite = (char*) ppString[0]; } - write(1, toWrite, strlen(toWrite)); /* 1 is stdout! */ + len = strlen(toWrite); + write(1, toWrite, len); /* 1 is stdout! */ + if(pData->bEnsureLFEnding && toWrite[len-1] != '\n') { + write(1, "\n", 1); /* write missing LF */ + } ENDdoAction @@ -143,6 +150,7 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) iTplOpts = (bUseArrayInterface == 0) ? 0 : OMSR_TPL_AS_ARRAY; CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, iTplOpts, (uchar*) "RSYSLOG_FileFormat")); pData->bUseArrayInterface = bUseArrayInterface; + pData->bEnsureLFEnding = bEnsureLFEnding; CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct @@ -165,6 +173,7 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a { DEFiRet; bUseArrayInterface = 0; + bEnsureLFEnding = 1; RETiRet; } @@ -195,6 +204,8 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionomstdoutarrayinterface", 0, eCmdHdlrBinary, NULL, &bUseArrayInterface, STD_LOADABLE_MODULE_ID)); } + CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionomstdoutensurelfending", 0, eCmdHdlrBinary, NULL, + &bEnsureLFEnding, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID)); ENDmodInit diff --git a/plugins/omtemplate/omtemplate.c b/plugins/omtemplate/omtemplate.c index e35968ad..5577f8c6 100644 --- a/plugins/omtemplate/omtemplate.c +++ b/plugins/omtemplate/omtemplate.c @@ -36,7 +36,7 @@ #include <signal.h> #include <errno.h> #include <time.h> -#include "dirty.h" +#include "conf.h" #include "syslogd-types.h" #include "srUtils.h" #include "template.h" diff --git a/runtime/Makefile.am b/runtime/Makefile.am index eeb656d6..14abe722 100644 --- a/runtime/Makefile.am +++ b/runtime/Makefile.am @@ -39,6 +39,8 @@ librsyslog_la_SOURCES = \ obj.h \ modules.c \ modules.h \ + apc.c \ + apc.h \ sync.c \ sync.h \ expr.c \ @@ -67,6 +69,12 @@ librsyslog_la_SOURCES = \ vmop.h \ queue.c \ queue.h \ + ruleset.c \ + ruleset.h \ + rule.c \ + rule.h \ + prop.c \ + prop.h \ cfsysline.c \ cfsysline.h \ \ @@ -105,6 +113,17 @@ lmregexp_la_LDFLAGS = -module -avoid-version lmregexp_la_LIBADD = endif +# +# zlib support +# +if ENABLE_ZLIB +pkglib_LTLIBRARIES += lmzlibw.la +lmzlibw_la_SOURCES = zlibw.c zlibw.h +lmzlibw_la_CPPFLAGS = $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) +lmzlibw_la_LDFLAGS = -module -avoid-version +lmzlibw_la_LIBADD = +endif + if ENABLE_INET pkglib_LTLIBRARIES += lmnet.la lmnetstrms.la # diff --git a/runtime/apc.c b/runtime/apc.c new file mode 100644 index 00000000..bc330e39 --- /dev/null +++ b/runtime/apc.c @@ -0,0 +1,405 @@ +/* apc.c - asynchronous procedure call support + * + * An asynchronous procedure call (APC) is a procedure call (guess what) that is potentially run + * asynchronously to its main thread. It can be scheduled to occur at a caller-provided time. + * As long as the procedure has not been called, the APC entry may be modified by the caller + * or deleted. It is the caller's purpose to make sure proper synchronization is in place. + * The APC object only case about APC's own control structures (which *are* properly + * guarded by synchronization primitives). + * + * Module begun 2009-06-15 by Rainer Gerhards + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * The rsyslog runtime library is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * The rsyslog runtime library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with the rsyslog runtime library. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. + */ + +#include "config.h" +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <pthread.h> + +#include "rsyslog.h" +#include "obj.h" +#include "apc.h" +#include "srUtils.h" + +/* static data */ +DEFobjStaticHelpers + +/* following is a used to implement a monotonically increasing id for the apcs. That + * ID can be used to cancel an apc request. Note that the ID is generated with modulo + * arithmetic, so at some point, it will wrap. Howerver, this happens at 2^32-1 at + * earliest, so this is not considered a problem. + */ +apc_id_t apcID = 0; + +/* private data structures */ + +/* the apc list and its entries + * This is a doubly-linked list as we need to be able to do inserts + * and deletes right in the middle of the list. It is inspired by the + * Unix callout mechanism. + * Note that we support two generic caller-provided parameters as + * experience shows that at most two are often used. This causes very + * little overhead, but simplifies caller code in cases where exactly + * two parameters are needed. We hope this is a useful optimizaton. + * rgerhards, 2009-06-15 + */ +typedef struct apc_list_s { + struct apc_list_s *pNext; + struct apc_list_s *pPrev; + apc_id_t id; + apc_t *pApc; /* pointer to the APC object to be scheduled */ +} apc_list_t; + +apc_list_t *apcListRoot = NULL; +apc_list_t *apcListTail = NULL; +pthread_mutex_t listMutex; /* needs to be locked for all list operations */ + + +/* destructor for the apc object */ +BEGINobjDestruct(apc) /* be sure to specify the object type also in END and CODESTART macros! */ +CODESTARTobjDestruct(apc) +ENDobjDestruct(apc) + + +/* ------------------------------ APC list handling functions ------------------------------ */ + +/* Function that handles changes to the list root. Most importantly, this function + * needs to schedule a new timer. It is OK to call this function with an empty list. + */ +static rsRetVal +listRootChanged(void) +{ + DEFiRet; + + if(apcListRoot == NULL) + FINALIZE; + + // TODO: implement! + +finalize_it: + RETiRet; +} + + +/* insert an apc entry into the APC list. The same entry MUST NOT already be present! + */ +static rsRetVal +insertApc(apc_t *pThis, apc_id_t *pID) +{ + apc_list_t *pCurr; + apc_list_t *pNew; + DEFiRet; + + CHKmalloc(pNew = (apc_list_t*) calloc(1, sizeof(apc_list_t))); + pNew->pApc = pThis; + pNew->id = *pID = apcID++; +dbgprintf("insert apc %p, id %ld\n", pThis, pNew->id); + + /* find right list location */ + if(apcListRoot == NULL) { + /* no need to search, list is empty */ + apcListRoot = pNew; + apcListTail = pNew; + CHKiRet(listRootChanged()); + } else { + for(pCurr = apcListRoot ; pCurr != NULL ; pCurr = pCurr->pNext) { + if(pCurr->pApc->ttExec > pThis->ttExec) + break; + } + + if(pCurr == NULL) { + /* insert at tail */ + pNew->pPrev = apcListTail; + apcListTail->pNext = pNew; + apcListTail = pNew; + } else { + if(pCurr == apcListRoot) { + /* new first entry */ + pCurr->pPrev = pNew; + pNew->pNext = pCurr; + apcListRoot = pNew; + CHKiRet(listRootChanged()); + } else { + /* in the middle of the list */ + pCurr->pPrev = pNew; + pNew->pNext = pCurr; + } + } + } + + +finalize_it: + RETiRet; +} + + +/* Delete an apc entry from the APC list. It is OK if the entry is not found, + * in this case we assume it already has been processed. + */ +static rsRetVal +deleteApc(apc_id_t id) +{ + apc_list_t *pCurr; + DEFiRet; + +dbgprintf("trying to delete apc %ld\n", id); + for(pCurr = apcListRoot ; pCurr != NULL ; pCurr = pCurr->pNext) { + if(pCurr->id == id) { +RUNLOG_STR("apc id found, now deleting!\n"); + if(pCurr == apcListRoot) { + apcListRoot = pCurr->pNext; + CHKiRet(listRootChanged()); + } else { + pCurr->pPrev->pNext = pCurr->pNext; + } + if(pCurr->pNext == NULL) { + apcListTail = pCurr->pPrev; + } else { + pCurr->pNext->pPrev = pCurr->pPrev; + } + free(pCurr); + pCurr = NULL; + break; + } + } + +finalize_it: + RETiRet; +} + + +/* unlist all elements up to the current timestamp. Return this as a seperate list + * to the caller. Returns an empty (NULL ptr) list if there are no such elements. + * The caller must handle that gracefully. The list is returned in the parameter. + */ +static rsRetVal +unlistCurrent(apc_list_t **ppList) +{ + apc_list_t *pCurr; + time_t tCurr; + DEFiRet; + assert(ppList != NULL); + + time(&tCurr); + + if(apcListRoot == NULL || apcListRoot->pApc->ttExec > tCurr) { + *ppList = NULL; + FINALIZE; + } + + *ppList = apcListRoot; + /* now search up to which entry we need to execute */ + for(pCurr = apcListRoot ; pCurr != NULL && pCurr->pApc->ttExec <= tCurr ; pCurr = pCurr->pNext) { + /*JUST SKIP TO LAST ELEMENT*/; + } + + if(pCurr == NULL) { + /* all elements can be unlisted */ + apcListRoot = NULL; + apcListTail = NULL; + } else { + /* need to set a new root */ + pCurr->pPrev->pNext = NULL; /* terminate newly unlisted list */ + pCurr->pPrev = NULL; /* we are the new root */ + apcListRoot = pCurr; + } + +finalize_it: + RETiRet; +} + + +/* ------------------------------ END APC list handling functions ------------------------------ */ + + +/* execute all list elements that are currently scheduled for execution. We do this in two phases. + * In the first phase, we look the list mutex and move everything from the head of the queue to + * the current timestamp to a new to-be-executed list. Then we unlock the mutex and do the actual + * exec (which may take some time). + * Note that the caller is responsible for proper + * caller-level synchronization. The caller may schedule another Apc, this module must + * ensure that (and it does so by not locking the list mutex while we call the Apc). + * Note: this function "consumes" the apc_t, so it is no longer existing after this + * function returns. + */ +// TODO make static and associated with our own pthread-based timer +rsRetVal +execScheduled(void) +{ + apc_list_t *pExecList; + apc_list_t *pCurr; + apc_list_t *pNext; + DEFVARS_mutexProtection_uncond; + DEFiRet; + + BEGIN_MTX_PROTECTED_OPERATIONS_UNCOND(&listMutex); + iRet = unlistCurrent(&pExecList); + END_MTX_PROTECTED_OPERATIONS_UNCOND(&listMutex); + CHKiRet(iRet); + + if(pExecList != NULL) { + DBGPRINTF("running apc scheduler - we have %s to execute\n", + pExecList == NULL ? "nothing" : "something"); + } + + for(pCurr = pExecList ; pCurr != NULL ; pCurr = pNext) { +dbgprintf("executing apc list entry %p, apc %p\n", pCurr, pCurr->pApc); + pNext = pCurr->pNext; + pCurr->pApc->pProc(pCurr->pApc->param1, pCurr->pApc->param2); + apcDestruct(&pCurr->pApc); + free(pCurr); + } + +finalize_it: + RETiRet; +} + + +/* Standard-Constructor + */ +BEGINobjConstruct(apc) /* be sure to specify the object type also in END macro! */ +ENDobjConstruct(apc) + + +/* ConstructionFinalizer + * Note that we use a non-standard calling interface: pID returns the current APC + * id. This is the only way to handle the situation without the need for extra + * locking. + * rgerhards, 2008-01-09 + */ +static rsRetVal +apcConstructFinalize(apc_t *pThis, apc_id_t *pID) +{ + DEFVARS_mutexProtection_uncond; + DEFiRet; + ISOBJ_TYPE_assert(pThis, apc); + assert(pID != NULL); + BEGIN_MTX_PROTECTED_OPERATIONS_UNCOND(&listMutex); + insertApc(pThis, pID); + END_MTX_PROTECTED_OPERATIONS_UNCOND(&listMutex); +RUNLOG_STR("apcConstructFinalize post mutex unlock\n"); + RETiRet; +} + + +/* some set methods */ +static rsRetVal +SetProcedure(apc_t *pThis, void (*pProc)(void*, void*)) +{ + ISOBJ_TYPE_assert(pThis, apc); + pThis->pProc = pProc; + return RS_RET_OK; +} +static rsRetVal +SetParam1(apc_t *pThis, void *param1) +{ + ISOBJ_TYPE_assert(pThis, apc); + pThis->param1 = param1; + return RS_RET_OK; +} +static rsRetVal +SetParam2(apc_t *pThis, void *param2) +{ + ISOBJ_TYPE_assert(pThis, apc); + pThis->param1 = param2; + return RS_RET_OK; +} + + +/* cancel an Apc request, ID is provided. It is OK if the ID can not be found, this may + * happen if the Apc was executed in the mean time. So it is safe to call CancelApc() at + * any time. + */ +static rsRetVal +CancelApc(apc_id_t id) +{ + DEFVARS_mutexProtection_uncond; + + BEGINfunc + BEGIN_MTX_PROTECTED_OPERATIONS_UNCOND(&listMutex); + deleteApc(id); + END_MTX_PROTECTED_OPERATIONS_UNCOND(&listMutex); + ENDfunc + return RS_RET_OK; +} + + +/* debugprint for the apc object */ +BEGINobjDebugPrint(apc) /* be sure to specify the object type also in END and CODESTART macros! */ +CODESTARTobjDebugPrint(apc) + dbgoprint((obj_t*) pThis, "APC module, currently no state info available\n"); +ENDobjDebugPrint(apc) + + +/* queryInterface function + */ +BEGINobjQueryInterface(apc) +CODESTARTobjQueryInterface(apc) + if(pIf->ifVersion != apcCURR_IF_VERSION) { /* check for current version, increment on each change */ + ABORT_FINALIZE(RS_RET_INTERFACE_NOT_SUPPORTED); + } + + /* ok, we have the right interface, so let's fill it + * Please note that we may also do some backwards-compatibility + * work here (if we can support an older interface version - that, + * of course, also affects the "if" above). + */ + pIf->Construct = apcConstruct; + pIf->ConstructFinalize = apcConstructFinalize; + pIf->Destruct = apcDestruct; + pIf->DebugPrint = apcDebugPrint; + pIf->CancelApc = CancelApc; + pIf->SetProcedure = SetProcedure; + pIf->SetParam1 = SetParam1; + pIf->SetParam2 = SetParam2; +finalize_it: +ENDobjQueryInterface(apc) + + +/* Exit the apc class. + * rgerhards, 2009-04-06 + */ +BEGINObjClassExit(apc, OBJ_IS_CORE_MODULE) /* class, version */ + //objRelease(apcstk, CORE_COMPONENT); + pthread_mutex_destroy(&listMutex); +ENDObjClassExit(apc) + + +/* Initialize the apc class. Must be called as the very first method + * before anything else is called inside this class. + * rgerhards, 2008-02-19 + */ +BEGINObjClassInit(apc, 1, OBJ_IS_CORE_MODULE) /* class, version */ + /* request objects we use */ + //CHKiRet(objUse(apcstk, CORE_COMPONENT)); + + /* set our own handlers */ + OBJSetMethodHandler(objMethod_DEBUGPRINT, apcDebugPrint); + OBJSetMethodHandler(objMethod_CONSTRUCTION_FINALIZER, apcConstructFinalize); + + /* do other initializations */ + pthread_mutex_init(&listMutex, NULL); +ENDObjClassInit(apc) + +/* vi:set ai: + */ diff --git a/runtime/apc.h b/runtime/apc.h new file mode 100644 index 00000000..7c679b97 --- /dev/null +++ b/runtime/apc.h @@ -0,0 +1,56 @@ +/* The apc object. + * + * See apc.c for more information. + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * The rsyslog runtime library is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * The rsyslog runtime library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with the rsyslog runtime library. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. + */ +#ifndef INCLUDED_APC_H +#define INCLUDED_APC_H + +/* the apc object */ +typedef struct apc_s { + BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ + time_t ttExec; /* when to call procedure (so far seconds...) */ + void (*pProc)(void*, void*); /* which procedure to call */ + void *param1; /* user-supplied parameters */ + void *param2; /* user-supplied parameters */ +} apc_t; + +typedef unsigned long apc_id_t; /* monotonically incrementing apc ID */ + +/* interfaces */ +BEGINinterface(apc) /* name must also be changed in ENDinterface macro! */ + INTERFACEObjDebugPrint(apc); + rsRetVal (*Construct)(apc_t **ppThis); + rsRetVal (*ConstructFinalize)(apc_t *pThis, apc_id_t *); + rsRetVal (*Destruct)(apc_t **ppThis); + rsRetVal (*SetProcedure)(apc_t *pThis, void (*pProc)(void*, void*)); + rsRetVal (*SetParam1)(apc_t *pThis, void *); + rsRetVal (*SetParam2)(apc_t *pThis, void *); + rsRetVal (*CancelApc)(apc_id_t); +ENDinterface(apc) +#define apcCURR_IF_VERSION 1 /* increment whenever you change the interface structure! */ + + +/* prototypes */ +PROTOTYPEObj(apc); + +#endif /* #ifndef INCLUDED_APC_H */ diff --git a/runtime/atomic.h b/runtime/atomic.h index fdf64214..d5aaf56b 100644 --- a/runtime/atomic.h +++ b/runtime/atomic.h @@ -42,10 +42,16 @@ */ #ifdef HAVE_ATOMIC_BUILTINS # define ATOMIC_INC(data) ((void) __sync_fetch_and_add(&(data), 1)) +# define ATOMIC_INC_AND_FETCH(data) __sync_fetch_and_add(&(data), 1) # define ATOMIC_DEC(data) ((void) __sync_sub_and_fetch(&(data), 1)) # define ATOMIC_DEC_AND_FETCH(data) __sync_sub_and_fetch(&(data), 1) # define ATOMIC_FETCH_32BIT(data) ((unsigned) __sync_fetch_and_and(&(data), 0xffffffff)) # define ATOMIC_STORE_1_TO_32BIT(data) __sync_lock_test_and_set(&(data), 1) +# define ATOMIC_STORE_0_TO_INT(data) __sync_fetch_and_and(&(data), 0) +# define ATOMIC_STORE_1_TO_INT(data) __sync_fetch_and_or(&(data), 1) +# define ATOMIC_STORE_INT_TO_INT(data, val) __sync_fetch_and_or(&(data), (val)) +# define ATOMIC_CAS(data, oldVal, newVal) __sync_bool_compare_and_swap(&(data), (oldVal), (newVal)); +# define ATOMIC_CAS_VAL(data, oldVal, newVal) __sync_val_compare_and_swap(&(data), (oldVal), (newVal)); #else /* note that we gained parctical proof that theoretical problems DO occur * if we do not properly address them. See this blog post for details: diff --git a/runtime/cfsysline.c b/runtime/cfsysline.c index 0fb4247d..469fbfc2 100644 --- a/runtime/cfsysline.c +++ b/runtime/cfsysline.c @@ -217,9 +217,9 @@ static rsRetVal doGetSize(uchar **pp, rsRetVal (*pSetHdlr)(void*, uid_t), void * case 'K': i *= 1000; ++(*pp); break; case 'M': i *= 1000000; ++(*pp); break; case 'G': i *= 1000000000; ++(*pp); break; - case 'T': i *= 1000000000000; ++(*pp); break; /* tera */ - case 'P': i *= 1000000000000000; ++(*pp); break; /* peta */ - case 'E': i *= 1000000000000000000; ++(*pp); break; /* exa */ + case 'T': i *= (long long) 1000000000000; ++(*pp); break; /* tera */ + case 'P': i *= (long long) 1000000000000000; ++(*pp); break; /* peta */ + case 'E': i *= (long long) 1000000000000000000; ++(*pp); break; /* exa */ } /* done */ @@ -814,7 +814,7 @@ rsRetVal regCfSysLineHdlr(uchar *pCmdName, int bChainingPermitted, ecslCmdHdrlTy CHKiRet(cslcConstruct(&pThis, bChainingPermitted)); CHKiRet_Hdlr(cslcAddHdlr(pThis, eType, pHdlr, pData, pOwnerCookie)) { cslcDestruct(pThis); - goto finalize_it; + FINALIZE; } /* important: add to list, AFTER everything else is OK. Else * we mess up things in the error case. @@ -825,7 +825,7 @@ rsRetVal regCfSysLineHdlr(uchar *pCmdName, int bChainingPermitted, ecslCmdHdrlTy } CHKiRet_Hdlr(llAppend(&llCmdList, pMyCmdName, (void*) pThis)) { cslcDestruct(pThis); - goto finalize_it; + FINALIZE; } } else { /* command already exists, are we allowed to chain? */ @@ -834,7 +834,7 @@ rsRetVal regCfSysLineHdlr(uchar *pCmdName, int bChainingPermitted, ecslCmdHdrlTy } CHKiRet_Hdlr(cslcAddHdlr(pThis, eType, pHdlr, pData, pOwnerCookie)) { cslcDestruct(pThis); - goto finalize_it; + FINALIZE; } } diff --git a/runtime/conf.c b/runtime/conf.c index a1d74b93..e2d3a889 100644 --- a/runtime/conf.c +++ b/runtime/conf.c @@ -69,13 +69,16 @@ #include "expr.h" #include "ctok.h" #include "ctok_token.h" +#include "rule.h" +#include "ruleset.h" +#include "unicode-helper.h" #ifdef OS_SOLARIS # define NAME_MAX MAXNAMELEN #endif /* forward definitions */ -static rsRetVal cfline(uchar *line, selector_t **pfCurr); +static rsRetVal cfline(uchar *line, rule_t **pfCurr); static rsRetVal processConfFile(uchar *pConfFile); @@ -87,6 +90,8 @@ DEFobjCurrIf(ctok_token) DEFobjCurrIf(module) DEFobjCurrIf(errmsg) DEFobjCurrIf(net) +DEFobjCurrIf(rule) +DEFobjCurrIf(ruleset) static int iNbrActions; /* number of actions the running config has. Needs to be init on ReInitConf() */ @@ -392,15 +397,17 @@ finalize_it: static rsRetVal processConfFile(uchar *pConfFile) { - DEFiRet; int iLnNbr = 0; FILE *cf; - selector_t *fCurr = NULL; + rule_t *pCurrRule = NULL; uchar *p; uchar cbuf[CFGLNSIZ]; uchar *cline; int i; int bHadAnError = 0; + uchar *pszOrgLine = NULL; + size_t lenLine; + DEFiRet; ASSERT(pConfFile != NULL); if((cf = fopen((char*)pConfFile, "r")) == NULL) { @@ -413,9 +420,12 @@ processConfFile(uchar *pConfFile) while (fgets((char*)cline, sizeof(cbuf) - (cline - cbuf), cf) != NULL) { ++iLnNbr; /* drop LF - TODO: make it better, replace fgets(), but its clean as it is */ - if(cline[strlen((char*)cline)-1] == '\n') { - cline[strlen((char*)cline) -1] = '\0'; + lenLine = ustrlen(cline); + if(cline[lenLine-1] == '\n') { + cline[lenLine-1] = '\0'; } + free(pszOrgLine); + pszOrgLine = ustrdup(cline); /* save if needed for errmsg, NULL ptr is OK */ /* check for end-of-section, comments, strip off trailing * spaces and newline character. */ @@ -429,7 +439,6 @@ processConfFile(uchar *pConfFile) * TODO: review the code at whole - this is highly suspect (but will go away * once we do the rest of RainerScript). */ - /* was: strcpy((char*)cline, (char*)p); */ for( i = 0 ; p[i] != '\0' ; ++i) { cline[i] = p[i]; } @@ -453,7 +462,7 @@ processConfFile(uchar *pConfFile) /* we now have the complete line, and are positioned at the first non-whitespace * character. So let's process it */ - if(cfline(cbuf, &fCurr) != RS_RET_OK) { + if(cfline(cbuf, &pCurrRule) != RS_RET_OK) { /* we log a message, but otherwise ignore the error. After all, the next * line can be correct. -- rgerhards, 2007-08-02 */ @@ -461,28 +470,32 @@ processConfFile(uchar *pConfFile) dbgprintf("config line NOT successfully processed\n"); snprintf((char*)szErrLoc, sizeof(szErrLoc) / sizeof(uchar), "%s, line %d", pConfFile, iLnNbr); - errmsg.LogError(0, NO_ERRCODE, "the last error occured in %s", (char*)szErrLoc); + errmsg.LogError(0, NO_ERRCODE, "the last error occured in %s:\"%s\"", (char*)szErrLoc, (char*)pszOrgLine); bHadAnError = 1; } } /* we probably have one selector left to be added - so let's do that now */ - CHKiRet(selectorAddList(fCurr)); + if(pCurrRule != NULL) { + CHKiRet(ruleset.AddRule(rule.GetAssRuleset(pCurrRule), &pCurrRule)); + } /* close the configuration file */ - (void) fclose(cf); + fclose(cf); finalize_it: if(iRet != RS_RET_OK) { char errStr[1024]; - if(fCurr != NULL) - selectorDestruct(fCurr); + if(pCurrRule != NULL) + rule.Destruct(&pCurrRule); rs_strerror_r(errno, errStr, sizeof(errStr)); dbgprintf("error %d processing config file '%s'; os error (if any): %s\n", iRet, pConfFile, errStr); } + free(pszOrgLine); + if(bHadAnError && (iRet == RS_RET_OK)) { /* a bit dirty, enhance in future releases */ iRet = RS_RET_NONFATAL_CONFIG_ERR; } @@ -500,7 +513,7 @@ finalize_it: rsRetVal cflineParseTemplateName(uchar** pp, omodStringRequest_t *pOMSR, int iEntry, int iTplOpts, uchar *dfltTplName) { uchar *p; - uchar *tplName; + uchar *tplName = NULL; cstr_t *pStrB; DEFiRet; @@ -537,10 +550,12 @@ rsRetVal cflineParseTemplateName(uchar** pp, omodStringRequest_t *pOMSR, int iEn CHKiRet(cstrConvSzStrAndDestruct(pStrB, &tplName, 0)); } - iRet = OMSRsetEntry(pOMSR, iEntry, tplName, iTplOpts); - if(iRet != RS_RET_OK) goto finalize_it; + CHKiRet(OMSRsetEntry(pOMSR, iEntry, tplName, iTplOpts)); finalize_it: + if(iRet != RS_RET_OK) + free(tplName); + *pp = p; RETiRet; @@ -555,6 +570,7 @@ finalize_it: * rgerhards, 2007-07-25 * updated to include OMSR pointer -- rgerhards, 2007-07-27 * updated to include template name -- rgerhards, 2008-03-28 + * rgerhards, 2010-01-19: file names end at the first space */ rsRetVal cflineParseFileName(uchar* p, uchar *pFileName, omodStringRequest_t *pOMSR, int iEntry, int iTplOpts, uchar *pszTpl) @@ -567,7 +583,7 @@ cflineParseFileName(uchar* p, uchar *pFileName, omodStringRequest_t *pOMSR, int pName = pFileName; i = 1; /* we start at 1 so that we reseve space for the '\0'! */ - while(*p && *p != ';' && i < MAXFNAME) { + while(*p && *p != ';' && *p != ' ' && i < MAXFNAME) { *pName++ = *p++; ++i; } @@ -586,7 +602,7 @@ cflineParseFileName(uchar* p, uchar *pFileName, omodStringRequest_t *pOMSR, int * rgerhards 2005-09-15 */ /* GPLv3 - stems back to sysklogd */ -static rsRetVal cflineProcessTradPRIFilter(uchar **pline, register selector_t *f) +static rsRetVal cflineProcessTradPRIFilter(uchar **pline, register rule_t *pRule) { uchar *p; register uchar *q; @@ -601,17 +617,17 @@ static rsRetVal cflineProcessTradPRIFilter(uchar **pline, register selector_t *f ASSERT(pline != NULL); ASSERT(*pline != NULL); - ASSERT(f != NULL); + ISOBJ_TYPE_assert(pRule, rule); dbgprintf(" - traditional PRI filter\n"); errno = 0; /* keep strerror_r() stuff out of logerror messages */ - f->f_filter_type = FILTER_PRI; + pRule->f_filter_type = FILTER_PRI; /* Note: file structure is pre-initialized to zero because it was * created with calloc()! */ for (i = 0; i <= LOG_NFACILITIES; i++) { - f->f_filterData.f_pmask[i] = TABLE_NOPRI; + pRule->f_filterData.f_pmask[i] = TABLE_NOPRI; } /* scan through the list of selectors */ @@ -666,32 +682,32 @@ static rsRetVal cflineProcessTradPRIFilter(uchar **pline, register selector_t *f for (i = 0; i <= LOG_NFACILITIES; i++) { if ( pri == INTERNAL_NOPRI ) { if ( ignorepri ) - f->f_filterData.f_pmask[i] = TABLE_ALLPRI; + pRule->f_filterData.f_pmask[i] = TABLE_ALLPRI; else - f->f_filterData.f_pmask[i] = TABLE_NOPRI; + pRule->f_filterData.f_pmask[i] = TABLE_NOPRI; } else if ( singlpri ) { if ( ignorepri ) - f->f_filterData.f_pmask[i] &= ~(1<<pri); + pRule->f_filterData.f_pmask[i] &= ~(1<<pri); else - f->f_filterData.f_pmask[i] |= (1<<pri); + pRule->f_filterData.f_pmask[i] |= (1<<pri); } else { if ( pri == TABLE_ALLPRI ) { if ( ignorepri ) - f->f_filterData.f_pmask[i] = TABLE_NOPRI; + pRule->f_filterData.f_pmask[i] = TABLE_NOPRI; else - f->f_filterData.f_pmask[i] = TABLE_ALLPRI; + pRule->f_filterData.f_pmask[i] = TABLE_ALLPRI; } else { if ( ignorepri ) for (i2= 0; i2 <= pri; ++i2) - f->f_filterData.f_pmask[i] &= ~(1<<i2); + pRule->f_filterData.f_pmask[i] &= ~(1<<i2); else for (i2= 0; i2 <= pri; ++i2) - f->f_filterData.f_pmask[i] |= (1<<i2); + pRule->f_filterData.f_pmask[i] |= (1<<i2); } } } @@ -706,27 +722,27 @@ static rsRetVal cflineProcessTradPRIFilter(uchar **pline, register selector_t *f if ( pri == INTERNAL_NOPRI ) { if ( ignorepri ) - f->f_filterData.f_pmask[i >> 3] = TABLE_ALLPRI; + pRule->f_filterData.f_pmask[i >> 3] = TABLE_ALLPRI; else - f->f_filterData.f_pmask[i >> 3] = TABLE_NOPRI; + pRule->f_filterData.f_pmask[i >> 3] = TABLE_NOPRI; } else if ( singlpri ) { if ( ignorepri ) - f->f_filterData.f_pmask[i >> 3] &= ~(1<<pri); + pRule->f_filterData.f_pmask[i >> 3] &= ~(1<<pri); else - f->f_filterData.f_pmask[i >> 3] |= (1<<pri); + pRule->f_filterData.f_pmask[i >> 3] |= (1<<pri); } else { if ( pri == TABLE_ALLPRI ) { if ( ignorepri ) - f->f_filterData.f_pmask[i >> 3] = TABLE_NOPRI; + pRule->f_filterData.f_pmask[i >> 3] = TABLE_NOPRI; else - f->f_filterData.f_pmask[i >> 3] = TABLE_ALLPRI; + pRule->f_filterData.f_pmask[i >> 3] = TABLE_ALLPRI; } else { if ( ignorepri ) for (i2= 0; i2 <= pri; ++i2) - f->f_filterData.f_pmask[i >> 3] &= ~(1<<i2); + pRule->f_filterData.f_pmask[i >> 3] &= ~(1<<i2); else for (i2= 0; i2 <= pri; ++i2) - f->f_filterData.f_pmask[i >> 3] |= (1<<i2); + pRule->f_filterData.f_pmask[i >> 3] |= (1<<i2); } } } @@ -752,7 +768,7 @@ static rsRetVal cflineProcessTradPRIFilter(uchar **pline, register selector_t *f * A pointer to that beginnig is passed back to the caller. * rgerhards 2008-01-19 */ -static rsRetVal cflineProcessIfFilter(uchar **pline, register selector_t *f) +static rsRetVal cflineProcessIfFilter(uchar **pline, register rule_t *f) { DEFiRet; ctok_t *tok; @@ -765,7 +781,6 @@ static rsRetVal cflineProcessIfFilter(uchar **pline, register selector_t *f) dbgprintf(" - general expression-based filter\n"); errno = 0; /* keep strerror_r() stuff out of logerror messages */ -dbgprintf("calling expression parser, pp %p ('%s')\n", *pline, *pline); f->f_filter_type = FILTER_EXPR; /* if we come to over here, pline starts with "if ". We just skip that part. */ @@ -821,10 +836,11 @@ finalize_it: * of the action part. A pointer to that beginnig is passed back to the caller. * rgerhards 2005-09-15 */ -static rsRetVal cflineProcessPropFilter(uchar **pline, register selector_t *f) +static rsRetVal cflineProcessPropFilter(uchar **pline, register rule_t *f) { rsParsObj *pPars; cstr_t *pCSCompOp; + cstr_t *pCSPropName; rsRetVal iRet; int iOffset; /* for compare operations */ @@ -844,12 +860,19 @@ static rsRetVal cflineProcessPropFilter(uchar **pline, register selector_t *f) } /* read property */ - iRet = parsDelimCStr(pPars, &f->f_filterData.prop.pCSPropName, ',', 1, 1, 1); + iRet = parsDelimCStr(pPars, &pCSPropName, ',', 1, 1, 1); if(iRet != RS_RET_OK) { errmsg.LogError(0, iRet, "error %d parsing filter property - ignoring selector", iRet); rsParsDestruct(pPars); return(iRet); } + iRet = propNameToID(pCSPropName, &f->f_filterData.prop.propID); + if(iRet != RS_RET_OK) { + errmsg.LogError(0, iRet, "error %d parsing filter property - ignoring selector", iRet); + rsParsDestruct(pPars); + return(iRet); + } + cstrDestruct(&pCSPropName); /* read operation */ iRet = parsDelimCStr(pPars, &pCSCompOp, ',', 1, 1, 1); @@ -992,7 +1015,7 @@ static rsRetVal cflineProcessTagSelector(uchar **pline) if(**pline != '\0' && **pline == '*' && *(*pline+1) == '\0') { dbgprintf("resetting programname filter\n"); if(pDfltProgNameCmp != NULL) { - CHKiRet(rsCStrSetSzStr(pDfltProgNameCmp, NULL)); + rsCStrDestruct(&pDfltProgNameCmp); } } else { dbgprintf("setting programname filter to '%s'\n", *pline); @@ -1010,10 +1033,10 @@ finalize_it: /* read the filter part of a configuration line and store the filter - * in the supplied selector_t + * in the supplied rule_t * rgerhards, 2007-08-01 */ -static rsRetVal cflineDoFilter(uchar **pp, selector_t *f) +static rsRetVal cflineDoFilter(uchar **pp, rule_t *f) { DEFiRet; @@ -1106,17 +1129,15 @@ static rsRetVal cflineDoAction(uchar **p, action_t **ppAction) /* Process a configuration file line in traditional "filter selector" format - * or one that builds upon this format. + * or one that builds upon this format. Note that ppRule may be a NULL pointer, + * which is valid and happens if there is no previous line (right at the start + * of the master config file!). */ -static rsRetVal cflineClassic(uchar *p, selector_t **pfCurr) +static rsRetVal +cflineClassic(uchar *p, rule_t **ppRule) { DEFiRet; action_t *pAction; - selector_t *fCurr; - - ASSERT(pfCurr != NULL); - - fCurr = *pfCurr; /* lines starting with '&' have no new filters and just add * new actions to the currently processed selector. @@ -1134,16 +1155,19 @@ static rsRetVal cflineClassic(uchar *p, selector_t **pfCurr) * selector is NULL, which means we do not need to care about it at * all. -- rgerhards, 2007-08-01 */ - CHKiRet(selectorAddList(fCurr)); - CHKiRet(selectorConstruct(&fCurr)); /* create "fresh" selector */ - CHKiRet(cflineDoFilter(&p, fCurr)); /* pull filters */ + if(*ppRule != NULL) { + CHKiRet(ruleset.AddRule(rule.GetAssRuleset(*ppRule), ppRule)); + } + CHKiRet(rule.Construct(ppRule)); /* create "fresh" selector */ + CHKiRet(rule.SetAssRuleset(*ppRule, ruleset.GetCurrent())); /* create "fresh" selector */ + CHKiRet(rule.ConstructFinalize(*ppRule)); /* create "fresh" selector */ + CHKiRet(cflineDoFilter(&p, *ppRule)); /* pull filters */ } CHKiRet(cflineDoAction(&p, &pAction)); - CHKiRet(llAppend(&fCurr->llActList, NULL, (void*) pAction)); + CHKiRet(llAppend(&(*ppRule)->llActList, NULL, (void*) pAction)); finalize_it: - *pfCurr = fCurr; RETiRet; } @@ -1153,7 +1177,7 @@ finalize_it: * rgerhards, 2007-08-01 */ static rsRetVal -cfline(uchar *line, selector_t **pfCurr) +cfline(uchar *line, rule_t **pfCurr) { DEFiRet; @@ -1250,6 +1274,8 @@ CODESTARTObjClassExit(conf) objRelease(module, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); objRelease(net, LM_NET_FILENAME); + objRelease(rule, CORE_COMPONENT); + objRelease(ruleset, CORE_COMPONENT); ENDObjClassExit(conf) @@ -1265,6 +1291,8 @@ BEGINAbstractObjClassInit(conf, 1, OBJ_IS_CORE_MODULE) /* class, version - CHANG CHKiRet(objUse(module, CORE_COMPONENT)); CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(net, LM_NET_FILENAME)); /* TODO: make this dependcy go away! */ + CHKiRet(objUse(rule, CORE_COMPONENT)); + CHKiRet(objUse(ruleset, CORE_COMPONENT)); ENDObjClassInit(conf) /* vi:set ai: diff --git a/runtime/conf.h b/runtime/conf.h index 2494d4dc..25b887be 100644 --- a/runtime/conf.h +++ b/runtime/conf.h @@ -35,7 +35,7 @@ BEGINinterface(conf) /* name must also be changed in ENDinterface macro! */ rsRetVal (*cfsysline)(uchar *p); rsRetVal (*doModLoad)(uchar **pp, __attribute__((unused)) void* pVal); rsRetVal (*doIncludeLine)(uchar **pp, __attribute__((unused)) void* pVal); - rsRetVal (*cfline)(uchar *line, selector_t **pfCurr); + rsRetVal (*cfline)(uchar *line, rule_t **pfCurr); rsRetVal (*processConfFile)(uchar *pConfFile); rsRetVal (*ReInitConf)(void); rsRetVal (*GetNbrActActions)(int *); @@ -51,5 +51,9 @@ PROTOTYPEObj(conf); extern EHostnameCmpMode eDfltHostnameCmpMode; extern cstr_t *pDfltHostnameCmp; extern cstr_t *pDfltProgNameCmp; +/* TODO: the following 2 need to go in conf obj interface... */ +rsRetVal cflineParseTemplateName(uchar** pp, omodStringRequest_t *pOMSR, int iEntry, int iTplOpts, uchar *dfltTplName); +rsRetVal cflineParseFileName(uchar* p, uchar *pFileName, omodStringRequest_t *pOMSR, int iEntry, int iTplOpts, uchar *pszTpl); + #endif /* #ifndef INCLUDED_CONF_H */ diff --git a/runtime/ctok.c b/runtime/ctok.c index 263e656c..18ddaed2 100644 --- a/runtime/ctok.c +++ b/runtime/ctok.c @@ -87,11 +87,12 @@ ctokUngetCharFromStream(ctok_t *pThis, uchar __attribute__((unused)) c) } -/* get the next character from the input "stream" (currently just a in-memory - * string...) -- rgerhards, 2008-02-19 +/* get the next character from the input "stream". Note that this version + * does NOT look for comment characters as end-of-stream, so it is suitable + * when building constant strings! -- rgerhards, 2010-03-01 */ -static rsRetVal -ctokGetCharFromStream(ctok_t *pThis, uchar *pc) +static inline rsRetVal +ctokGetCharFromStreamNoComment(ctok_t *pThis, uchar *pc) { DEFiRet; @@ -99,7 +100,7 @@ ctokGetCharFromStream(ctok_t *pThis, uchar *pc) ASSERT(pc != NULL); /* end of string or begin of comment terminates the "stream" */ - if(*pThis->pp == '\0' || *pThis->pp == '#') { + if(*pThis->pp == '\0') { ABORT_FINALIZE(RS_RET_EOS); } else { *pc = *pThis->pp; @@ -111,6 +112,28 @@ finalize_it: } +/* get the next character from the input "stream" (currently just a in-memory + * string...) -- rgerhards, 2008-02-19 + */ +static rsRetVal +ctokGetCharFromStream(ctok_t *pThis, uchar *pc) +{ + DEFiRet; + + ISOBJ_TYPE_assert(pThis, ctok); + ASSERT(pc != NULL); + + CHKiRet(ctokGetCharFromStreamNoComment(pThis, pc)); + /* begin of comment terminates the "stream"! */ + if(*pc == '#') { + ABORT_FINALIZE(RS_RET_EOS); + } + +finalize_it: + RETiRet; +} + + /* skip whitespace in the input "stream". * rgerhards, 2008-02-19 */ @@ -258,13 +281,13 @@ ctokGetVar(ctok_t *pThis, ctok_token_t *pToken) pToken->tok = ctok_MSGVAR; } - CHKiRet(rsCStrConstruct(&pstrVal)); + CHKiRet(cstrConstruct(&pstrVal)); /* this loop is quite simple, a variable name is terminated when a non-supported * character is detected. Note that we currently permit a numerical digit as the * first char, which is not permitted by ABNF. -- rgerhards, 2009-03-10 */ while(isalpha(c) || isdigit(c) || (c == '_') || (c == '-')) { - CHKiRet(rsCStrAppendChar(pstrVal, tolower(c))); + CHKiRet(cstrAppendChar(pstrVal, tolower(c))); CHKiRet(ctokGetCharFromStream(pThis, &c)); } CHKiRet(ctokUngetCharFromStream(pThis, c)); /* put not processed char back */ @@ -277,7 +300,7 @@ ctokGetVar(ctok_t *pThis, ctok_token_t *pToken) finalize_it: if(iRet != RS_RET_OK) { if(pstrVal != NULL) { - rsCStrDestruct(&pstrVal); + cstrDestruct(&pstrVal); } } @@ -301,23 +324,23 @@ ctokGetSimpStr(ctok_t *pThis, ctok_token_t *pToken) pToken->tok = ctok_SIMPSTR; - CHKiRet(rsCStrConstruct(&pstrVal)); - CHKiRet(ctokGetCharFromStream(pThis, &c)); + CHKiRet(cstrConstruct(&pstrVal)); + CHKiRet(ctokGetCharFromStreamNoComment(pThis, &c)); /* while we are in escape mode (had a backslash), no sequence * terminates the loop. If outside, it is terminated by a single quote. */ while(bInEsc || c != '\'') { if(bInEsc) { - CHKiRet(rsCStrAppendChar(pstrVal, c)); + CHKiRet(cstrAppendChar(pstrVal, c)); bInEsc = 0; } else { if(c == '\\') { bInEsc = 1; } else { - CHKiRet(rsCStrAppendChar(pstrVal, c)); + CHKiRet(cstrAppendChar(pstrVal, c)); } } - CHKiRet(ctokGetCharFromStream(pThis, &c)); + CHKiRet(ctokGetCharFromStreamNoComment(pThis, &c)); } CHKiRet(cstrFinalize(pstrVal)); @@ -327,7 +350,7 @@ ctokGetSimpStr(ctok_t *pThis, ctok_token_t *pToken) finalize_it: if(iRet != RS_RET_OK) { if(pstrVal != NULL) { - rsCStrDestruct(&pstrVal); + cstrDestruct(&pstrVal); } } @@ -519,8 +542,9 @@ ctokGetToken(ctok_t *pThis, ctok_token_t **ppToken) CHKiRet(ctokUngetCharFromStream(pThis, c)); pToken->tok = ctok_FUNCTION; /* fill function name */ - CHKiRet(rsCStrConstruct(&pstrVal)); + CHKiRet(cstrConstruct(&pstrVal)); CHKiRet(rsCStrSetSzStr(pstrVal, szWord)); + CHKiRet(cstrFinalize(pstrVal)); CHKiRet(var.SetString(pToken->pVar, pstrVal)); } else { /* give up... */ dbgprintf("parser has an invalid word (token) '%s'\n", szWord); diff --git a/runtime/datetime.c b/runtime/datetime.c index 99caaf97..eff72f91 100644 --- a/runtime/datetime.c +++ b/runtime/datetime.c @@ -49,6 +49,8 @@ DEFobjStaticHelpers DEFobjCurrIf(errmsg) +/* the following table of ten powers saves us some computation */ +static const int tenPowers[6] = { 1, 10, 100, 1000, 10000, 100000 }; /* ------------------------------ methods ------------------------------ */ @@ -121,6 +123,7 @@ static void getCurrTime(struct syslogTime *t, time_t *ttSeconds) t->OffsetMode = '+'; t->OffsetHour = lBias / 3600; t->OffsetMinute = lBias % 3600; + t->timeType = TIME_TYPE_RFC5424; /* we have a high precision timestamp */ } @@ -288,11 +291,11 @@ ParseTIMESTAMP3339(struct syslogTime *pTime, uchar** ppszTS, int *pLenStr) } /* OK, we actually have a 3339 timestamp, so let's indicated this */ - if(lenStr > 0 && *pszTS == ' ') { + if(lenStr > 0) { + if(*pszTS != ' ') /* if it is not a space, it can not be a "good" time - 2010-02-22 rgerhards */ + ABORT_FINALIZE(RS_RET_INVLD_TIME); + ++pszTS; /* just skip past it */ --lenStr; - ++pszTS; - } else { - ABORT_FINALIZE(RS_RET_INVLD_TIME); } /* we had success, so update parse pointer and caller-provided timestamp */ @@ -363,6 +366,10 @@ ParseTIMESTAMP3164(struct syslogTime *pTime, uchar** ppszTS, int *pLenStr) * We will use this for parsing, as it probably is the * fastest way to parse it. * + * 2009-08-17: we now do case-insensitive comparisons, as some devices obviously do not + * obey to the RFC-specified case. As we need to guess in any case, we can ignore case + * in the first place -- rgerhards + * * 2005-07-18, well sometimes it pays to be a bit more verbose, even in C... * Fixed a bug that lead to invalid detection of the data. The issue was that * we had an if(++pszTS == 'x') inside of some of the consturcts below. However, @@ -377,20 +384,21 @@ ParseTIMESTAMP3164(struct syslogTime *pTime, uchar** ppszTS, int *pLenStr) switch(*pszTS++) { + case 'j': case 'J': - if(*pszTS == 'a') { + if(*pszTS == 'a' || *pszTS == 'A') { ++pszTS; - if(*pszTS == 'n') { + if(*pszTS == 'n' || *pszTS == 'N') { ++pszTS; month = 1; } else ABORT_FINALIZE(RS_RET_INVLD_TIME); - } else if(*pszTS == 'u') { + } else if(*pszTS == 'u' || *pszTS == 'U') { ++pszTS; - if(*pszTS == 'n') { + if(*pszTS == 'n' || *pszTS == 'N') { ++pszTS; month = 6; - } else if(*pszTS == 'l') { + } else if(*pszTS == 'l' || *pszTS == 'L') { ++pszTS; month = 7; } else @@ -398,10 +406,11 @@ ParseTIMESTAMP3164(struct syslogTime *pTime, uchar** ppszTS, int *pLenStr) } else ABORT_FINALIZE(RS_RET_INVLD_TIME); break; + case 'f': case 'F': - if(*pszTS == 'e') { + if(*pszTS == 'e' || *pszTS == 'E') { ++pszTS; - if(*pszTS == 'b') { + if(*pszTS == 'b' || *pszTS == 'B') { ++pszTS; month = 2; } else @@ -409,13 +418,14 @@ ParseTIMESTAMP3164(struct syslogTime *pTime, uchar** ppszTS, int *pLenStr) } else ABORT_FINALIZE(RS_RET_INVLD_TIME); break; + case 'm': case 'M': - if(*pszTS == 'a') { + if(*pszTS == 'a' || *pszTS == 'A') { ++pszTS; - if(*pszTS == 'r') { + if(*pszTS == 'r' || *pszTS == 'R') { ++pszTS; month = 3; - } else if(*pszTS == 'y') { + } else if(*pszTS == 'y' || *pszTS == 'Y') { ++pszTS; month = 5; } else @@ -423,17 +433,18 @@ ParseTIMESTAMP3164(struct syslogTime *pTime, uchar** ppszTS, int *pLenStr) } else ABORT_FINALIZE(RS_RET_INVLD_TIME); break; + case 'a': case 'A': - if(*pszTS == 'p') { + if(*pszTS == 'p' || *pszTS == 'P') { ++pszTS; - if(*pszTS == 'r') { + if(*pszTS == 'r' || *pszTS == 'R') { ++pszTS; month = 4; } else ABORT_FINALIZE(RS_RET_INVLD_TIME); - } else if(*pszTS == 'u') { + } else if(*pszTS == 'u' || *pszTS == 'U') { ++pszTS; - if(*pszTS == 'g') { + if(*pszTS == 'g' || *pszTS == 'G') { ++pszTS; month = 8; } else @@ -441,10 +452,11 @@ ParseTIMESTAMP3164(struct syslogTime *pTime, uchar** ppszTS, int *pLenStr) } else ABORT_FINALIZE(RS_RET_INVLD_TIME); break; + case 's': case 'S': - if(*pszTS == 'e') { + if(*pszTS == 'e' || *pszTS == 'E') { ++pszTS; - if(*pszTS == 'p') { + if(*pszTS == 'p' || *pszTS == 'P') { ++pszTS; month = 9; } else @@ -452,10 +464,11 @@ ParseTIMESTAMP3164(struct syslogTime *pTime, uchar** ppszTS, int *pLenStr) } else ABORT_FINALIZE(RS_RET_INVLD_TIME); break; + case 'o': case 'O': - if(*pszTS == 'c') { + if(*pszTS == 'c' || *pszTS == 'C') { ++pszTS; - if(*pszTS == 't') { + if(*pszTS == 't' || *pszTS == 'T') { ++pszTS; month = 10; } else @@ -463,10 +476,11 @@ ParseTIMESTAMP3164(struct syslogTime *pTime, uchar** ppszTS, int *pLenStr) } else ABORT_FINALIZE(RS_RET_INVLD_TIME); break; + case 'n': case 'N': - if(*pszTS == 'o') { + if(*pszTS == 'o' || *pszTS == 'O') { ++pszTS; - if(*pszTS == 'v') { + if(*pszTS == 'v' || *pszTS == 'V') { ++pszTS; month = 11; } else @@ -474,10 +488,11 @@ ParseTIMESTAMP3164(struct syslogTime *pTime, uchar** ppszTS, int *pLenStr) } else ABORT_FINALIZE(RS_RET_INVLD_TIME); break; + case 'd': case 'D': - if(*pszTS == 'e') { + if(*pszTS == 'e' || *pszTS == 'E') { ++pszTS; - if(*pszTS == 'c') { + if(*pszTS == 'c' || *pszTS == 'C') { ++pszTS; month = 12; } else @@ -495,6 +510,7 @@ ParseTIMESTAMP3164(struct syslogTime *pTime, uchar** ppszTS, int *pLenStr) if(lenStr == 0 || *pszTS++ != ' ') ABORT_FINALIZE(RS_RET_INVLD_TIME); + --lenStr; /* we accept a slightly malformed timestamp when receiving. This is * we accept one-digit days @@ -550,7 +566,13 @@ ParseTIMESTAMP3164(struct syslogTime *pTime, uchar** ppszTS, int *pLenStr) * invalid format, it occurs frequently enough (e.g. with Cisco devices) * to permit it as a valid case. -- rgerhards, 2008-09-12 */ - if(lenStr == 0 || *pszTS++ == ':') { + if(lenStr > 0 && *pszTS == ':') { + ++pszTS; /* just skip past it */ + --lenStr; + } + if(lenStr > 0) { + if(*pszTS != ' ') /* if it is not a space, it can not be a "good" time - 2010-02-22 rgerhards */ + ABORT_FINALIZE(RS_RET_INVLD_TIME); ++pszTS; /* just skip past it */ --lenStr; } @@ -589,7 +611,7 @@ finalize_it: * returns the size of the timestamp written in bytes (without * the string terminator). If 0 is returend, an error occured. */ -int formatTimestampToMySQL(struct syslogTime *ts, char* pDst, size_t iLenDst) +int formatTimestampToMySQL(struct syslogTime *ts, char* pBuf) { /* currently we do not consider localtime/utc. This may later be * added. If so, I recommend using a property replacer option @@ -598,28 +620,54 @@ int formatTimestampToMySQL(struct syslogTime *ts, char* pDst, size_t iLenDst) * rgerhards, 2007-06-26 */ assert(ts != NULL); - assert(pDst != NULL); - - if (iLenDst < 15) /* we need at least 14 bytes - 14 digits for timestamp + '\n' */ - return(0); + assert(pBuf != NULL); - return(snprintf(pDst, iLenDst, "%4.4d%2.2d%2.2d%2.2d%2.2d%2.2d", - ts->year, ts->month, ts->day, ts->hour, ts->minute, ts->second)); + pBuf[0] = (ts->year / 1000) % 10 + '0'; + pBuf[1] = (ts->year / 100) % 10 + '0'; + pBuf[2] = (ts->year / 10) % 10 + '0'; + pBuf[3] = ts->year % 10 + '0'; + pBuf[4] = (ts->month / 10) % 10 + '0'; + pBuf[5] = ts->month % 10 + '0'; + pBuf[6] = (ts->day / 10) % 10 + '0'; + pBuf[7] = ts->day % 10 + '0'; + pBuf[8] = (ts->hour / 10) % 10 + '0'; + pBuf[9] = ts->hour % 10 + '0'; + pBuf[10] = (ts->minute / 10) % 10 + '0'; + pBuf[11] = ts->minute % 10 + '0'; + pBuf[12] = (ts->second / 10) % 10 + '0'; + pBuf[13] = ts->second % 10 + '0'; + pBuf[14] = '\0'; + return 15; } -int formatTimestampToPgSQL(struct syslogTime *ts, char *pDst, size_t iLenDst) +int formatTimestampToPgSQL(struct syslogTime *ts, char *pBuf) { - /* see note in formatTimestampToMySQL, applies here as well */ - assert(ts != NULL); - assert(pDst != NULL); - - if (iLenDst < 21) /* we need 20 bytes + '\n' */ - return(0); + /* see note in formatTimestampToMySQL, applies here as well */ + assert(ts != NULL); + assert(pBuf != NULL); - return(snprintf(pDst, iLenDst, "%4.4d-%2.2d-%2.2d %2.2d:%2.2d:%2.2d", - ts->year, ts->month, ts->day, ts->hour, ts->minute, ts->second)); + pBuf[0] = (ts->year / 1000) % 10 + '0'; + pBuf[1] = (ts->year / 100) % 10 + '0'; + pBuf[2] = (ts->year / 10) % 10 + '0'; + pBuf[3] = ts->year % 10 + '0'; + pBuf[4] = '-'; + pBuf[5] = (ts->month / 10) % 10 + '0'; + pBuf[6] = ts->month % 10 + '0'; + pBuf[7] = '-'; + pBuf[8] = (ts->day / 10) % 10 + '0'; + pBuf[9] = ts->day % 10 + '0'; + pBuf[10] = ' '; + pBuf[11] = (ts->hour / 10) % 10 + '0'; + pBuf[12] = ts->hour % 10 + '0'; + pBuf[13] = ':'; + pBuf[14] = (ts->minute / 10) % 10 + '0'; + pBuf[15] = ts->minute % 10 + '0'; + pBuf[16] = ':'; + pBuf[17] = (ts->second / 10) % 10 + '0'; + pBuf[18] = ts->second % 10 + '0'; + pBuf[19] = '\0'; + return 19; } @@ -629,35 +677,36 @@ int formatTimestampToPgSQL(struct syslogTime *ts, char *pDst, size_t iLenDst) * buffer that will receive the resulting string. The function * returns the size of the timestamp written in bytes (without * the string terminator). If 0 is returend, an error occured. - * The buffer must be at least 10 bytes large. + * The buffer must be at least 7 bytes large. * rgerhards, 2008-06-06 */ -int formatTimestampSecFrac(struct syslogTime *ts, char* pBuf, size_t iLenBuf) +int formatTimestampSecFrac(struct syslogTime *ts, char* pBuf) { - int lenRet; - char szFmtStr[64]; + int iBuf; + int power; + int secfrac; + short digit; assert(ts != NULL); assert(pBuf != NULL); - assert(iLenBuf >= 10); + iBuf = 0; if(ts->secfracPrecision > 0) - { /* We must look at - * the precision specified. For example, if we have millisec precision (3 digits), a - * secFrac value of 12 is not equivalent to ".12" but ".012". Obviously, this - * is a huge difference ;). To avoid this, we first create a format string with - * the specific precision and *then* use that format string to do the actual formating. - */ - /* be careful: there is ONE actual %d in the format string below ;) */ - snprintf(szFmtStr, sizeof(szFmtStr), "%%0%dd", ts->secfracPrecision); - lenRet = snprintf(pBuf, iLenBuf, szFmtStr, ts->secfrac); + { + power = tenPowers[(ts->secfracPrecision - 1) % 6]; + secfrac = ts->secfrac; + while(power > 0) { + digit = secfrac / power; + secfrac -= digit * power; + power /= 10; + pBuf[iBuf++] = digit + '0'; + } } else { - pBuf[0] = '0'; - pBuf[1] = '\0'; - lenRet = 1; + pBuf[iBuf++] = '0'; } + pBuf[iBuf] = '\0'; - return(lenRet); + return iBuf; } @@ -669,48 +718,73 @@ int formatTimestampSecFrac(struct syslogTime *ts, char* pBuf, size_t iLenBuf) * returns the size of the timestamp written in bytes (without * the string terminator). If 0 is returend, an error occured. */ -int formatTimestamp3339(struct syslogTime *ts, char* pBuf, size_t iLenBuf) +int formatTimestamp3339(struct syslogTime *ts, char* pBuf) { - int iRet; - char szTZ[7]; /* buffer for TZ information */ + int iBuf; + int power; + int secfrac; + short digit; + BEGINfunc assert(ts != NULL); assert(pBuf != NULL); - - if(iLenBuf < 20) - return(0); /* we NEED at least 20 bytes */ - /* do TZ information first, this is easier to take care of "Z" zone in rfc3339 */ + /* start with fixed parts */ + /* year yyyy */ + pBuf[0] = (ts->year / 1000) % 10 + '0'; + pBuf[1] = (ts->year / 100) % 10 + '0'; + pBuf[2] = (ts->year / 10) % 10 + '0'; + pBuf[3] = ts->year % 10 + '0'; + pBuf[4] = '-'; + /* month */ + pBuf[5] = (ts->month / 10) % 10 + '0'; + pBuf[6] = ts->month % 10 + '0'; + pBuf[7] = '-'; + /* day */ + pBuf[8] = (ts->day / 10) % 10 + '0'; + pBuf[9] = ts->day % 10 + '0'; + pBuf[10] = 'T'; + /* hour */ + pBuf[11] = (ts->hour / 10) % 10 + '0'; + pBuf[12] = ts->hour % 10 + '0'; + pBuf[13] = ':'; + /* minute */ + pBuf[14] = (ts->minute / 10) % 10 + '0'; + pBuf[15] = ts->minute % 10 + '0'; + pBuf[16] = ':'; + /* second */ + pBuf[17] = (ts->second / 10) % 10 + '0'; + pBuf[18] = ts->second % 10 + '0'; + + iBuf = 19; /* points to next free entry, now it becomes dynamic! */ + + if(ts->secfracPrecision > 0) { + pBuf[iBuf++] = '.'; + power = tenPowers[(ts->secfracPrecision - 1) % 6]; + secfrac = ts->secfrac; + while(power > 0) { + digit = secfrac / power; + secfrac -= digit * power; + power /= 10; + pBuf[iBuf++] = digit + '0'; + } + } + if(ts->OffsetMode == 'Z') { - szTZ[0] = 'Z'; - szTZ[1] = '\0'; + pBuf[iBuf++] = 'Z'; } else { - snprintf(szTZ, sizeof(szTZ) / sizeof(char), "%c%2.2d:%2.2d", - ts->OffsetMode, ts->OffsetHour, ts->OffsetMinute); + pBuf[iBuf++] = ts->OffsetMode; + pBuf[iBuf++] = (ts->OffsetHour / 10) % 10 + '0'; + pBuf[iBuf++] = ts->OffsetHour % 10 + '0'; + pBuf[iBuf++] = ':'; + pBuf[iBuf++] = (ts->OffsetMinute / 10) % 10 + '0'; + pBuf[iBuf++] = ts->OffsetMinute % 10 + '0'; } - if(ts->secfracPrecision > 0) - { /* we now need to include fractional seconds. While doing so, we must look at - * the precision specified. For example, if we have millisec precision (3 digits), a - * secFrac value of 12 is not equivalent to ".12" but ".012". Obviously, this - * is a huge difference ;). To avoid this, we first create a format string with - * the specific precision and *then* use that format string to do the actual - * formating (mmmmhhh... kind of self-modifying code... ;)). - */ - char szFmtStr[64]; - /* be careful: there is ONE actual %d in the format string below ;) */ - snprintf(szFmtStr, sizeof(szFmtStr), - "%%04d-%%02d-%%02dT%%02d:%%02d:%%02d.%%0%dd%%s", - ts->secfracPrecision); - iRet = snprintf(pBuf, iLenBuf, szFmtStr, ts->year, ts->month, ts->day, - ts->hour, ts->minute, ts->second, ts->secfrac, szTZ); - } - else - iRet = snprintf(pBuf, iLenBuf, - "%4.4d-%2.2d-%2.2dT%2.2d:%2.2d:%2.2d%s", - ts->year, ts->month, ts->day, - ts->hour, ts->minute, ts->second, szTZ); - return(iRet); + pBuf[iBuf] = '\0'; + + ENDfunc + return iBuf; } /** @@ -719,47 +793,40 @@ int formatTimestamp3339(struct syslogTime *ts, char* pBuf, size_t iLenBuf) * buffer that will receive the resulting string. The function * returns the size of the timestamp written in bytes (without * the string termnator). If 0 is returend, an error occured. + * rgerhards, 2010-03-05: Added support to for buggy 3164 dates, + * where a zero-digit is written instead of a space for the first + * day character if day < 10. syslog-ng seems to do that, and some + * parsing scripts (in migration cases) rely on that. */ -int formatTimestamp3164(struct syslogTime *ts, char* pBuf, size_t iLenBuf) +int formatTimestamp3164(struct syslogTime *ts, char* pBuf, int bBuggyDay) { - static char* monthNames[13] = {"ERR", "Jan", "Feb", "Mar", - "Apr", "May", "Jun", "Jul", - "Aug", "Sep", "Oct", "Nov", "Dec"}; + static char* monthNames[12] = { "Jan", "Feb", "Mar", "Apr", "May", "Jun", + "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" }; + int iDay; assert(ts != NULL); assert(pBuf != NULL); - if(iLenBuf < 16) - return(0); /* we NEED 16 bytes */ - return(snprintf(pBuf, iLenBuf, "%s %2d %2.2d:%2.2d:%2.2d", - monthNames[ts->month], ts->day, ts->hour, - ts->minute, ts->second - )); + pBuf[0] = monthNames[(ts->month - 1)% 12][0]; + pBuf[1] = monthNames[(ts->month - 1) % 12][1]; + pBuf[2] = monthNames[(ts->month - 1) % 12][2]; + pBuf[3] = ' '; + iDay = (ts->day / 10) % 10; /* we need to write a space if the first digit is 0 */ + pBuf[4] = (bBuggyDay || iDay > 0) ? iDay + '0' : ' '; + pBuf[5] = ts->day % 10 + '0'; + pBuf[6] = ' '; + pBuf[7] = (ts->hour / 10) % 10 + '0'; + pBuf[8] = ts->hour % 10 + '0'; + pBuf[9] = ':'; + pBuf[10] = (ts->minute / 10) % 10 + '0'; + pBuf[11] = ts->minute % 10 + '0'; + pBuf[12] = ':'; + pBuf[13] = (ts->second / 10) % 10 + '0'; + pBuf[14] = ts->second % 10 + '0'; + pBuf[15] = '\0'; + return 16; /* traditional: number of bytes written */ } -/** - * Format a syslogTimestamp to a text format. - * The caller must provide the timestamp as well as a character - * buffer that will receive the resulting string. The function - * returns the size of the timestamp written in bytes (without - * the string termnator). If 0 is returend, an error occured. - */ -#if 0 /* This method is currently not called, be we like to preserve it */ -static int formatTimestamp(struct syslogTime *ts, char* pBuf, size_t iLenBuf) -{ - assert(ts != NULL); - assert(pBuf != NULL); - - if(ts->timeType == 1) { - return(formatTimestamp3164(ts, pBuf, iLenBuf)); - } - - if(ts->timeType == 2) { - return(formatTimestamp3339(ts, pBuf, iLenBuf)); - } - return(0); -} -#endif /* queryInterface function * rgerhards, 2008-03-05 */ @@ -793,7 +860,6 @@ ENDobjQueryInterface(datetime) BEGINAbstractObjClassInit(datetime, 1, OBJ_IS_CORE_MODULE) /* class, version */ /* request objects we use */ CHKiRet(objUse(errmsg, CORE_COMPONENT)); - ENDObjClassInit(datetime) /* vi:set ai: diff --git a/runtime/datetime.h b/runtime/datetime.h index 6377a4a4..9dcce3c5 100644 --- a/runtime/datetime.h +++ b/runtime/datetime.h @@ -23,8 +23,6 @@ #ifndef INCLUDED_DATETIME_H #define INCLUDED_DATETIME_H -#include "datetime.h" - /* TODO: define error codes */ #define NO_ERRCODE -1 @@ -36,20 +34,22 @@ typedef struct datetime_s { /* interfaces */ BEGINinterface(datetime) /* name must also be changed in ENDinterface macro! */ void (*getCurrTime)(struct syslogTime *t, time_t *ttSeconds); - rsRetVal (*ParseTIMESTAMP3339)(struct syslogTime *pTime, uchar** ppszTS, int *); - rsRetVal (*ParseTIMESTAMP3164)(struct syslogTime *pTime, uchar** pszTS, int *); - int (*formatTimestampToMySQL)(struct syslogTime *ts, char* pDst, size_t iLenDst); - int (*formatTimestampToPgSQL)(struct syslogTime *ts, char *pDst, size_t iLenDst); - int (*formatTimestamp3339)(struct syslogTime *ts, char* pBuf, size_t iLenBuf); - int (*formatTimestamp3164)(struct syslogTime *ts, char* pBuf, size_t iLenBuf); - int (*formatTimestampSecFrac)(struct syslogTime *ts, char* pBuf, size_t iLenBuf); + rsRetVal (*ParseTIMESTAMP3339)(struct syslogTime *pTime, uchar** ppszTS, int*); + rsRetVal (*ParseTIMESTAMP3164)(struct syslogTime *pTime, uchar** pszTS, int*); + int (*formatTimestampToMySQL)(struct syslogTime *ts, char* pDst); + int (*formatTimestampToPgSQL)(struct syslogTime *ts, char *pDst); + int (*formatTimestamp3339)(struct syslogTime *ts, char* pBuf); + int (*formatTimestamp3164)(struct syslogTime *ts, char* pBuf, int); + int (*formatTimestampSecFrac)(struct syslogTime *ts, char* pBuf); ENDinterface(datetime) -#define datetimeCURR_IF_VERSION 2 /* increment whenever you change the interface structure! */ +#define datetimeCURR_IF_VERSION 4 /* increment whenever you change the interface structure! */ /* interface changes: * 1 - initial version * 2 - not compatible to 1 - bugfix required ParseTIMESTAMP3164 to accept char ** as * last parameter. Did not try to remain compatible as this is not something any * third-party module should call. -- rgerhards, 2008.-09-12 + * 3 - taken by v5 branch! + * 4 - formatTimestamp3164 takes a third int parameter */ /* prototypes */ diff --git a/runtime/debug.c b/runtime/debug.c index 4ee90226..9b7c2952 100644 --- a/runtime/debug.c +++ b/runtime/debug.c @@ -1,4 +1,3 @@ -#include <sys/syscall.h> /* debug.c * * This file proides debug and run time error analysis support. Some of the @@ -51,6 +50,7 @@ #include "rsyslog.h" #include "debug.h" #include "atomic.h" +#include "cfsysline.h" #include "obj.h" @@ -549,6 +549,7 @@ if(pLog == NULL) { return; /* if we don't know it yet, we can not clean up... */ } #endif +#include <sys/syscall.h> /* we found the last lock entry. We now need to see from which FuncDB we need to * remove it. This is recorded inside the mutex log entry. @@ -732,6 +733,8 @@ static void dbgGetThrdName(char *pszBuf, size_t lenBuf, pthread_t thrd, int bInc */ void dbgSetThrdName(uchar *pszName) { +return; + dbgThrdInfo_t *pThrd = dbgGetThrdInfo(); if(pThrd->pszThrdName != NULL) free(pThrd->pszThrdName); @@ -776,7 +779,7 @@ static void dbgCallStackPrint(dbgThrdInfo_t *pThrd) /* print all threads call stacks */ -static void dbgCallStackPrintAll(void) +void dbgCallStackPrintAll(void) { dbgThrdInfo_t *pThrd; /* stack info */ @@ -839,7 +842,7 @@ dbgprint(obj_t *pObj, char *pszMsg, size_t lenMsg) static pthread_t ptLastThrdID = 0; static int bWasNL = 0; char pszThrdName[64]; /* 64 is to be on the safe side, anything over 20 is bad... */ - char pszWriteBuf[1024]; + char pszWriteBuf[32*1024]; size_t lenWriteBuf; struct timespec t; uchar *pszObjName = NULL; @@ -919,7 +922,7 @@ void dbgoprint(obj_t *pObj, char *fmt, ...) { va_list ap; - char pszWriteBuf[1024]; + char pszWriteBuf[32*1024]; size_t lenWriteBuf; if(!(Debug && debugging_on)) @@ -949,7 +952,7 @@ void dbgprintf(char *fmt, ...) { va_list ap; - char pszWriteBuf[1024]; + char pszWriteBuf[20480]; size_t lenWriteBuf; if(!(Debug && debugging_on)) @@ -958,6 +961,15 @@ dbgprintf(char *fmt, ...) va_start(ap, fmt); lenWriteBuf = vsnprintf(pszWriteBuf, sizeof(pszWriteBuf), fmt, ap); va_end(ap); + if(lenWriteBuf >= sizeof(pszWriteBuf)) { + /* prevent buffer overrruns and garbagge display */ + pszWriteBuf[sizeof(pszWriteBuf) - 5] = '.'; + pszWriteBuf[sizeof(pszWriteBuf) - 4] = '.'; + pszWriteBuf[sizeof(pszWriteBuf) - 3] = '.'; + pszWriteBuf[sizeof(pszWriteBuf) - 2] = '\n'; + pszWriteBuf[sizeof(pszWriteBuf) - 1] = '\0'; + lenWriteBuf = sizeof(pszWriteBuf); + } dbgprint(NULL, pszWriteBuf, lenWriteBuf); } @@ -1261,6 +1273,7 @@ dbgGetRuntimeOptions(void) "NoLogTimestamp\n" "Nostdoout\n" "filetrace=file (may be provided multiple times)\n" + "DebugOnDemand - enables debugging on USR1, but does not turn on output\n" "\nSee debug.html in your doc set or http://www.rsyslog.com for details\n"); exit(1); } else if(!strcasecmp((char*)optname, "debug")) { @@ -1269,6 +1282,13 @@ dbgGetRuntimeOptions(void) */ Debug = 1; debugging_on = 1; + } else if(!strcasecmp((char*)optname, "debugondemand")) { + /* Enables debugging, but turns off debug output */ + Debug = 1; + debugging_on = 1; + dbgprintf("Note: debug on demand turned on via configuraton file, " + "use USR1 signal to activate.\n"); + debugging_on = 0; } else if(!strcasecmp((char*)optname, "logfuncflow")) { bLogFuncFlow = 1; } else if(!strcasecmp((char*)optname, "logallocfree")) { diff --git a/runtime/debug.h b/runtime/debug.h index 1375493d..dcbfb930 100644 --- a/runtime/debug.h +++ b/runtime/debug.h @@ -134,8 +134,7 @@ void dbgPrintAllDebugInfo(void); /* debug aides */ -//#ifdef RTINST -#if 0 // temporarily removed for helgrind +#ifdef RTINST #define d_pthread_mutex_lock(x) dbgMutexLock(x, pdbgFuncDB, __LINE__, dbgCALLStaCK_POP_POINT ) #define d_pthread_mutex_trylock(x) dbgMutexTryLock(x, pdbgFuncDB, __LINE__, dbgCALLStaCK_POP_POINT ) #define d_pthread_mutex_unlock(x) dbgMutexUnlock(x, pdbgFuncDB, __LINE__, dbgCALLStaCK_POP_POINT ) diff --git a/runtime/glbl.c b/runtime/glbl.c index 28f14320..7fa61963 100644 --- a/runtime/glbl.c +++ b/runtime/glbl.c @@ -35,8 +35,10 @@ #include "rsyslog.h" #include "obj.h" +#include "unicode-helper.h" #include "cfsysline.h" #include "glbl.h" +#include "prop.h" /* some defaults */ #ifndef DFLT_NETSTRM_DRVR @@ -45,6 +47,7 @@ /* static data */ DEFobjStaticHelpers +DEFobjCurrIf(prop) /* static data * For this object, these variables are obviously what makes the "meat" of the @@ -52,13 +55,14 @@ DEFobjStaticHelpers */ static uchar *pszWorkDir = NULL; static int bOptimizeUniProc = 1; /* enable uniprocessor optimizations */ -static int bHUPisRestart = 1; /* should SIGHUP cause a full system restart? */ +static int bHUPisRestart = 0; /* should SIGHUP cause a full system restart? */ static int bPreserveFQDN = 0; /* should FQDNs always be preserved? */ static int iMaxLine = 2048; /* maximum length of a syslog message */ static int iDefPFFamily = PF_UNSPEC; /* protocol family (IPv4, IPv6 or both) */ static int bDropMalPTRMsgs = 0;/* Drop messages which have malicious PTR records during DNS lookup */ static int option_DisallowWarning = 1; /* complain if message from disallowed sender is received */ static int bDisableDNS = 0; /* don't look up IP addresses of remote messages */ +static prop_t *propLocalHostName = NULL;/* our hostname as FQDN - read-only after startup */ static uchar *LocalHostName = NULL;/* our hostname - read-only after startup */ static uchar *LocalFQDNName = NULL;/* our hostname as FQDN - read-only after startup */ static uchar *LocalDomain; /* our local domain name - read-only after startup */ @@ -132,6 +136,44 @@ GetLocalHostName(void) } +/* generate the local hostname property. This must be done after the hostname info + * has been set as well as PreserveFQDN. + * rgerhards, 2009-06-30 + */ +static rsRetVal +GenerateLocalHostNameProperty(void) +{ + DEFiRet; + uchar *pszName; + + if(propLocalHostName != NULL) + prop.Destruct(&propLocalHostName); + + CHKiRet(prop.Construct(&propLocalHostName)); + if(LocalHostName == NULL) + pszName = (uchar*) "[localhost]"; + else { + if(GetPreserveFQDN() == 1) + pszName = LocalFQDNName; + else + pszName = LocalHostName; + } + CHKiRet(prop.SetString(propLocalHostName, pszName, ustrlen(pszName))); + CHKiRet(prop.ConstructFinalize(propLocalHostName)); + +finalize_it: + RETiRet; +} + +/* return our local hostname as a string property + */ +static prop_t* +GetLocalHostNameProp(void) +{ + return(propLocalHostName); +} + + /* return the current localhost name as FQDN (requires FQDN to be set) * TODO: we should set the FQDN ourselfs in here! */ @@ -197,6 +239,8 @@ CODESTARTobjQueryInterface(glbl) * of course, also affects the "if" above). */ pIf->GetWorkDir = GetWorkDir; + pIf->GenerateLocalHostNameProperty = GenerateLocalHostNameProperty; + pIf->GetLocalHostNameProp = GetLocalHostNameProp; #define SIMP_PROP(name) \ pIf->Get##name = Get##name; \ pIf->Set##name = Set##name; @@ -249,7 +293,7 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a } bDropMalPTRMsgs = 0; bOptimizeUniProc = 1; - bHUPisRestart = 1; + bHUPisRestart = 0; bPreserveFQDN = 0; return RS_RET_OK; } @@ -262,6 +306,7 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a */ BEGINAbstractObjClassInit(glbl, 1, OBJ_IS_CORE_MODULE) /* class, version */ /* request objects we use */ + CHKiRet(objUse(prop, CORE_COMPONENT)); /* register config handlers (TODO: we need to implement a way to unregister them) */ CHKiRet(regCfSysLineHdlr((uchar *)"workdirectory", 0, eCmdHdlrGetWord, NULL, &pszWorkDir, NULL)); @@ -295,6 +340,7 @@ BEGINObjClassExit(glbl, OBJ_IS_CORE_MODULE) /* class, version */ free(LocalHostName); if(LocalFQDNName != NULL) free(LocalFQDNName); + objRelease(prop, CORE_COMPONENT); ENDObjClassExit(glbl) /* vi:set ai: diff --git a/runtime/glbl.h b/runtime/glbl.h index 5bdf4f57..dcfb6d5f 100644 --- a/runtime/glbl.h +++ b/runtime/glbl.h @@ -32,6 +32,8 @@ #ifndef GLBL_H_INCLUDED #define GLBL_H_INCLUDED +#include "prop.h" + #define glblGetIOBufSize() 4096 /* size of the IO buffer, e.g. for strm class */ /* interfaces */ @@ -57,9 +59,12 @@ BEGINinterface(glbl) /* name must also be changed in ENDinterface macro! */ SIMP_PROP(DfltNetstrmDrvrCAF, uchar*) SIMP_PROP(DfltNetstrmDrvrKeyFile, uchar*) SIMP_PROP(DfltNetstrmDrvrCertFile, uchar*) + /* added v3, 2009-06-30 */ + rsRetVal (*GenerateLocalHostNameProperty)(void); + prop_t* (*GetLocalHostNameProp)(void); #undef SIMP_PROP ENDinterface(glbl) -#define glblCURR_IF_VERSION 2 /* increment whenever you change the interface structure! */ +#define glblCURR_IF_VERSION 3 /* increment whenever you change the interface structure! */ /* version 2 had PreserveFQDN added - rgerhards, 2008-12-08 */ /* the remaining prototypes */ diff --git a/runtime/linkedlist.c b/runtime/linkedlist.c index 8f842e43..cc095f6e 100644 --- a/runtime/linkedlist.c +++ b/runtime/linkedlist.c @@ -398,7 +398,7 @@ rsRetVal llExecFunc(linkedList_t *pThis, rsRetVal (*pFunc)(void*, void*), void* */ llCookie = llCookiePrev; } else if (iRet != RS_RET_OK) { - goto finalize_it; + FINALIZE; } llCookiePrev = llCookie; } diff --git a/runtime/modules.c b/runtime/modules.c index 32ae659f..871f356a 100644 --- a/runtime/modules.c +++ b/runtime/modules.c @@ -383,7 +383,7 @@ doModInit(rsRetVal (*modInit)(int, int*, rsRetVal(**)(), rsRetVal(*)(), modInfo_ * can never change in the lifetime of an module. -- rgerhards, 2007-12-14 */ CHKiRet((*pNew->modQueryEtryPt)((uchar*)"getType", &modGetType)); - CHKiRet((iRet = (*modGetType)(&pNew->eType)) != RS_RET_OK); + CHKiRet((*modGetType)(&pNew->eType)); dbgprintf("module of type %d being loaded.\n", pNew->eType); /* OK, we know we can successfully work with the module. So we now fill the @@ -399,6 +399,7 @@ doModInit(rsRetVal (*modInit)(int, int*, rsRetVal(**)(), rsRetVal(*)(), modInfo_ CHKiRet((*pNew->modQueryEtryPt)((uchar*)"runInput", &pNew->mod.im.runInput)); CHKiRet((*pNew->modQueryEtryPt)((uchar*)"willRun", &pNew->mod.im.willRun)); CHKiRet((*pNew->modQueryEtryPt)((uchar*)"afterRun", &pNew->mod.im.afterRun)); + pNew->mod.im.bCanRun = 0; break; case eMOD_OUT: CHKiRet((*pNew->modQueryEtryPt)((uchar*)"freeInstance", &pNew->freeInstance)); diff --git a/runtime/modules.h b/runtime/modules.h index 372529ee..4d874019 100644 --- a/runtime/modules.h +++ b/runtime/modules.h @@ -106,6 +106,7 @@ typedef struct modInfo_s { rsRetVal (*runInput)(thrdInfo_t*); /* function to gather input and submit to queue */ rsRetVal (*willRun)(void); /* function to gather input and submit to queue */ rsRetVal (*afterRun)(thrdInfo_t*); /* function to gather input and submit to queue */ + int bCanRun; /* cached value of whether willRun() succeeded */ } im; struct {/* data for output modules */ /* below: perform the configured action diff --git a/runtime/msg.c b/runtime/msg.c index e26fe5ee..f5041b85 100644 --- a/runtime/msg.c +++ b/runtime/msg.c @@ -35,6 +35,9 @@ #include <string.h> #include <assert.h> #include <ctype.h> +#if HAVE_MALLOC_H +# include <malloc.h> +#endif #include "rsyslog.h" #include "srUtils.h" #include "stringbuf.h" @@ -46,6 +49,8 @@ #include "regexp.h" #include "atomic.h" #include "unicode-helper.h" +#include "ruleset.h" +#include "prop.h" /* static data */ DEFobjStaticHelpers @@ -53,58 +58,433 @@ DEFobjCurrIf(var) DEFobjCurrIf(datetime) DEFobjCurrIf(glbl) DEFobjCurrIf(regexp) +DEFobjCurrIf(prop) + +static struct { + uchar *pszName; + short lenName; +} syslog_pri_names[192] = { + { UCHAR_CONSTANT("0"), 3}, + { UCHAR_CONSTANT("1"), 3}, + { UCHAR_CONSTANT("2"), 3}, + { UCHAR_CONSTANT("3"), 3}, + { UCHAR_CONSTANT("4"), 3}, + { UCHAR_CONSTANT("5"), 3}, + { UCHAR_CONSTANT("6"), 3}, + { UCHAR_CONSTANT("7"), 3}, + { UCHAR_CONSTANT("8"), 3}, + { UCHAR_CONSTANT("9"), 3}, + { UCHAR_CONSTANT("10"), 4}, + { UCHAR_CONSTANT("11"), 4}, + { UCHAR_CONSTANT("12"), 4}, + { UCHAR_CONSTANT("13"), 4}, + { UCHAR_CONSTANT("14"), 4}, + { UCHAR_CONSTANT("15"), 4}, + { UCHAR_CONSTANT("16"), 4}, + { UCHAR_CONSTANT("17"), 4}, + { UCHAR_CONSTANT("18"), 4}, + { UCHAR_CONSTANT("19"), 4}, + { UCHAR_CONSTANT("20"), 4}, + { UCHAR_CONSTANT("21"), 4}, + { UCHAR_CONSTANT("22"), 4}, + { UCHAR_CONSTANT("23"), 4}, + { UCHAR_CONSTANT("24"), 4}, + { UCHAR_CONSTANT("25"), 4}, + { UCHAR_CONSTANT("26"), 4}, + { UCHAR_CONSTANT("27"), 4}, + { UCHAR_CONSTANT("28"), 4}, + { UCHAR_CONSTANT("29"), 4}, + { UCHAR_CONSTANT("30"), 4}, + { UCHAR_CONSTANT("31"), 4}, + { UCHAR_CONSTANT("32"), 4}, + { UCHAR_CONSTANT("33"), 4}, + { UCHAR_CONSTANT("34"), 4}, + { UCHAR_CONSTANT("35"), 4}, + { UCHAR_CONSTANT("36"), 4}, + { UCHAR_CONSTANT("37"), 4}, + { UCHAR_CONSTANT("38"), 4}, + { UCHAR_CONSTANT("39"), 4}, + { UCHAR_CONSTANT("40"), 4}, + { UCHAR_CONSTANT("41"), 4}, + { UCHAR_CONSTANT("42"), 4}, + { UCHAR_CONSTANT("43"), 4}, + { UCHAR_CONSTANT("44"), 4}, + { UCHAR_CONSTANT("45"), 4}, + { UCHAR_CONSTANT("46"), 4}, + { UCHAR_CONSTANT("47"), 4}, + { UCHAR_CONSTANT("48"), 4}, + { UCHAR_CONSTANT("49"), 4}, + { UCHAR_CONSTANT("50"), 4}, + { UCHAR_CONSTANT("51"), 4}, + { UCHAR_CONSTANT("52"), 4}, + { UCHAR_CONSTANT("53"), 4}, + { UCHAR_CONSTANT("54"), 4}, + { UCHAR_CONSTANT("55"), 4}, + { UCHAR_CONSTANT("56"), 4}, + { UCHAR_CONSTANT("57"), 4}, + { UCHAR_CONSTANT("58"), 4}, + { UCHAR_CONSTANT("59"), 4}, + { UCHAR_CONSTANT("60"), 4}, + { UCHAR_CONSTANT("61"), 4}, + { UCHAR_CONSTANT("62"), 4}, + { UCHAR_CONSTANT("63"), 4}, + { UCHAR_CONSTANT("64"), 4}, + { UCHAR_CONSTANT("65"), 4}, + { UCHAR_CONSTANT("66"), 4}, + { UCHAR_CONSTANT("67"), 4}, + { UCHAR_CONSTANT("68"), 4}, + { UCHAR_CONSTANT("69"), 4}, + { UCHAR_CONSTANT("70"), 4}, + { UCHAR_CONSTANT("71"), 4}, + { UCHAR_CONSTANT("72"), 4}, + { UCHAR_CONSTANT("73"), 4}, + { UCHAR_CONSTANT("74"), 4}, + { UCHAR_CONSTANT("75"), 4}, + { UCHAR_CONSTANT("76"), 4}, + { UCHAR_CONSTANT("77"), 4}, + { UCHAR_CONSTANT("78"), 4}, + { UCHAR_CONSTANT("79"), 4}, + { UCHAR_CONSTANT("80"), 4}, + { UCHAR_CONSTANT("81"), 4}, + { UCHAR_CONSTANT("82"), 4}, + { UCHAR_CONSTANT("83"), 4}, + { UCHAR_CONSTANT("84"), 4}, + { UCHAR_CONSTANT("85"), 4}, + { UCHAR_CONSTANT("86"), 4}, + { UCHAR_CONSTANT("87"), 4}, + { UCHAR_CONSTANT("88"), 4}, + { UCHAR_CONSTANT("89"), 4}, + { UCHAR_CONSTANT("90"), 4}, + { UCHAR_CONSTANT("91"), 4}, + { UCHAR_CONSTANT("92"), 4}, + { UCHAR_CONSTANT("93"), 4}, + { UCHAR_CONSTANT("94"), 4}, + { UCHAR_CONSTANT("95"), 4}, + { UCHAR_CONSTANT("96"), 4}, + { UCHAR_CONSTANT("97"), 4}, + { UCHAR_CONSTANT("98"), 4}, + { UCHAR_CONSTANT("99"), 4}, + { UCHAR_CONSTANT("100"), 5}, + { UCHAR_CONSTANT("101"), 5}, + { UCHAR_CONSTANT("102"), 5}, + { UCHAR_CONSTANT("103"), 5}, + { UCHAR_CONSTANT("104"), 5}, + { UCHAR_CONSTANT("105"), 5}, + { UCHAR_CONSTANT("106"), 5}, + { UCHAR_CONSTANT("107"), 5}, + { UCHAR_CONSTANT("108"), 5}, + { UCHAR_CONSTANT("109"), 5}, + { UCHAR_CONSTANT("110"), 5}, + { UCHAR_CONSTANT("111"), 5}, + { UCHAR_CONSTANT("112"), 5}, + { UCHAR_CONSTANT("113"), 5}, + { UCHAR_CONSTANT("114"), 5}, + { UCHAR_CONSTANT("115"), 5}, + { UCHAR_CONSTANT("116"), 5}, + { UCHAR_CONSTANT("117"), 5}, + { UCHAR_CONSTANT("118"), 5}, + { UCHAR_CONSTANT("119"), 5}, + { UCHAR_CONSTANT("120"), 5}, + { UCHAR_CONSTANT("121"), 5}, + { UCHAR_CONSTANT("122"), 5}, + { UCHAR_CONSTANT("123"), 5}, + { UCHAR_CONSTANT("124"), 5}, + { UCHAR_CONSTANT("125"), 5}, + { UCHAR_CONSTANT("126"), 5}, + { UCHAR_CONSTANT("127"), 5}, + { UCHAR_CONSTANT("128"), 5}, + { UCHAR_CONSTANT("129"), 5}, + { UCHAR_CONSTANT("130"), 5}, + { UCHAR_CONSTANT("131"), 5}, + { UCHAR_CONSTANT("132"), 5}, + { UCHAR_CONSTANT("133"), 5}, + { UCHAR_CONSTANT("134"), 5}, + { UCHAR_CONSTANT("135"), 5}, + { UCHAR_CONSTANT("136"), 5}, + { UCHAR_CONSTANT("137"), 5}, + { UCHAR_CONSTANT("138"), 5}, + { UCHAR_CONSTANT("139"), 5}, + { UCHAR_CONSTANT("140"), 5}, + { UCHAR_CONSTANT("141"), 5}, + { UCHAR_CONSTANT("142"), 5}, + { UCHAR_CONSTANT("143"), 5}, + { UCHAR_CONSTANT("144"), 5}, + { UCHAR_CONSTANT("145"), 5}, + { UCHAR_CONSTANT("146"), 5}, + { UCHAR_CONSTANT("147"), 5}, + { UCHAR_CONSTANT("148"), 5}, + { UCHAR_CONSTANT("149"), 5}, + { UCHAR_CONSTANT("150"), 5}, + { UCHAR_CONSTANT("151"), 5}, + { UCHAR_CONSTANT("152"), 5}, + { UCHAR_CONSTANT("153"), 5}, + { UCHAR_CONSTANT("154"), 5}, + { UCHAR_CONSTANT("155"), 5}, + { UCHAR_CONSTANT("156"), 5}, + { UCHAR_CONSTANT("157"), 5}, + { UCHAR_CONSTANT("158"), 5}, + { UCHAR_CONSTANT("159"), 5}, + { UCHAR_CONSTANT("160"), 5}, + { UCHAR_CONSTANT("161"), 5}, + { UCHAR_CONSTANT("162"), 5}, + { UCHAR_CONSTANT("163"), 5}, + { UCHAR_CONSTANT("164"), 5}, + { UCHAR_CONSTANT("165"), 5}, + { UCHAR_CONSTANT("166"), 5}, + { UCHAR_CONSTANT("167"), 5}, + { UCHAR_CONSTANT("168"), 5}, + { UCHAR_CONSTANT("169"), 5}, + { UCHAR_CONSTANT("170"), 5}, + { UCHAR_CONSTANT("171"), 5}, + { UCHAR_CONSTANT("172"), 5}, + { UCHAR_CONSTANT("173"), 5}, + { UCHAR_CONSTANT("174"), 5}, + { UCHAR_CONSTANT("175"), 5}, + { UCHAR_CONSTANT("176"), 5}, + { UCHAR_CONSTANT("177"), 5}, + { UCHAR_CONSTANT("178"), 5}, + { UCHAR_CONSTANT("179"), 5}, + { UCHAR_CONSTANT("180"), 5}, + { UCHAR_CONSTANT("181"), 5}, + { UCHAR_CONSTANT("182"), 5}, + { UCHAR_CONSTANT("183"), 5}, + { UCHAR_CONSTANT("184"), 5}, + { UCHAR_CONSTANT("185"), 5}, + { UCHAR_CONSTANT("186"), 5}, + { UCHAR_CONSTANT("187"), 5}, + { UCHAR_CONSTANT("188"), 5}, + { UCHAR_CONSTANT("189"), 5}, + { UCHAR_CONSTANT("190"), 5}, + { UCHAR_CONSTANT("191"), 5} + }; + +/*syslog facility names (as of RFC5424) */ +static char *syslog_fac_names[24] = { "kern", "user", "mail", "daemon", "auth", "syslog", "lpr", + "news", "uucp", "cron", "authpriv", "ftp", "ntp", "audit", + "alert", "clock", "local0", "local1", "local2", "local3", + "local4", "local5", "local6", "local7" }; + +/* table of severity names (in numerical order)*/ +static char *syslog_severity_names[8] = { "emerg", "alert", "crit", "err", "warning", "notice", "info", "debug" }; + +/* numerical values as string - this is the most efficient approach to convert severity + * and facility values to a numerical string... -- rgerhars, 2009-06-17 + */ -static syslogCODE rs_prioritynames[] = - { - { "alert", LOG_ALERT }, - { "crit", LOG_CRIT }, - { "debug", LOG_DEBUG }, - { "emerg", LOG_EMERG }, - { "err", LOG_ERR }, - { "error", LOG_ERR }, /* DEPRECATED */ - { "info", LOG_INFO }, - { "none", INTERNAL_NOPRI }, /* INTERNAL */ - { "notice", LOG_NOTICE }, - { "panic", LOG_EMERG }, /* DEPRECATED */ - { "warn", LOG_WARNING }, /* DEPRECATED */ - { "warning", LOG_WARNING }, - { NULL, -1 } - }; - -#ifndef LOG_AUTHPRIV -# define LOG_AUTHPRIV LOG_AUTH -#endif -static syslogCODE rs_facilitynames[] = - { - { "auth", LOG_AUTH }, - { "authpriv", LOG_AUTHPRIV }, - { "cron", LOG_CRON }, - { "daemon", LOG_DAEMON }, -#if defined(LOG_FTP) - {"ftp", LOG_FTP}, -#endif - { "kern", LOG_KERN }, - { "lpr", LOG_LPR }, - { "mail", LOG_MAIL }, - { "news", LOG_NEWS }, - { "security", LOG_AUTH }, /* DEPRECATED */ - { "syslog", LOG_SYSLOG }, - { "user", LOG_USER }, - { "uucp", LOG_UUCP }, - { "local0", LOG_LOCAL0 }, - { "local1", LOG_LOCAL1 }, - { "local2", LOG_LOCAL2 }, - { "local3", LOG_LOCAL3 }, - { "local4", LOG_LOCAL4 }, - { "local5", LOG_LOCAL5 }, - { "local6", LOG_LOCAL6 }, - { "local7", LOG_LOCAL7 }, - { NULL, -1 } - }; +static char *syslog_number_names[24] = { "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", + "15", "16", "17", "18", "19", "20", "21", "22", "23" }; /* some forward declarations */ -static int getAPPNAMELen(msg_t *pM); -static int getProtocolVersion(msg_t *pM); +static int getAPPNAMELen(msg_t *pM, bool bLockMutex); + + +static inline int getProtocolVersion(msg_t *pM) +{ + return(pM->iProtocolVersion); +} + + +static inline void +getInputName(msg_t *pM, uchar **ppsz, int *plen) +{ + BEGINfunc + if(pM == NULL) { + *ppsz = UCHAR_CONSTANT(""); + *plen = 0; + } else { + prop.GetString(pM->pInputName, ppsz, plen); + } + ENDfunc +} + + +static inline uchar* +getRcvFromIP(msg_t *pM) +{ + uchar *psz; + int len; + BEGINfunc + if(pM == NULL) { + psz = UCHAR_CONSTANT(""); + } else { + if(pM->pRcvFromIP == NULL) + psz = UCHAR_CONSTANT(""); + else + prop.GetString(pM->pRcvFromIP, &psz, &len); + } + ENDfunc + return psz; +} + + + +/* map a property name (string) to a property ID */ +rsRetVal propNameToID(cstr_t *pCSPropName, propid_t *pPropID) +{ + uchar *pName; + DEFiRet; + + assert(pCSPropName != NULL); + assert(pPropID != NULL); + pName = rsCStrGetSzStrNoNULL(pCSPropName); + + /* sometimes there are aliases to the original MonitoWare + * property names. These come after || in the ifs below. */ + if(!strcmp((char*) pName, "msg")) { + *pPropID = PROP_MSG; + } else if(!strcmp((char*) pName, "timestamp") + || !strcmp((char*) pName, "timereported")) { + *pPropID = PROP_TIMESTAMP; + } else if(!strcmp((char*) pName, "hostname") || !strcmp((char*) pName, "source")) { + *pPropID = PROP_HOSTNAME; + } else if(!strcmp((char*) pName, "syslogtag")) { + *pPropID = PROP_SYSLOGTAG; + } else if(!strcmp((char*) pName, "rawmsg")) { + *pPropID = PROP_RAWMSG; + /* enable this, if someone actually uses UxTradMsg, delete after some time has + * passed and nobody complained -- rgerhards, 2009-06-16 + } else if(!strcmp((char*) pName, "uxtradmsg")) { + pRes = getUxTradMsg(pMsg); + */ + } else if(!strcmp((char*) pName, "inputname")) { + *pPropID = PROP_INPUTNAME; + } else if(!strcmp((char*) pName, "fromhost")) { + *pPropID = PROP_FROMHOST; + } else if(!strcmp((char*) pName, "fromhost-ip")) { + *pPropID = PROP_FROMHOST_IP; + } else if(!strcmp((char*) pName, "pri")) { + *pPropID = PROP_PRI; + } else if(!strcmp((char*) pName, "pri-text")) { + *pPropID = PROP_PRI_TEXT; + } else if(!strcmp((char*) pName, "iut")) { + *pPropID = PROP_IUT; + } else if(!strcmp((char*) pName, "syslogfacility")) { + *pPropID = PROP_SYSLOGFACILITY; + } else if(!strcmp((char*) pName, "syslogfacility-text")) { + *pPropID = PROP_SYSLOGFACILITY_TEXT; + } else if(!strcmp((char*) pName, "syslogseverity") || !strcmp((char*) pName, "syslogpriority")) { + *pPropID = PROP_SYSLOGSEVERITY; + } else if(!strcmp((char*) pName, "syslogseverity-text") || !strcmp((char*) pName, "syslogpriority-text")) { + *pPropID = PROP_SYSLOGSEVERITY_TEXT; + } else if(!strcmp((char*) pName, "timegenerated")) { + *pPropID = PROP_TIMEGENERATED; + } else if(!strcmp((char*) pName, "programname")) { + *pPropID = PROP_PROGRAMNAME; + } else if(!strcmp((char*) pName, "protocol-version")) { + *pPropID = PROP_PROTOCOL_VERSION; + } else if(!strcmp((char*) pName, "structured-data")) { + *pPropID = PROP_STRUCTURED_DATA; + } else if(!strcmp((char*) pName, "app-name")) { + *pPropID = PROP_APP_NAME; + } else if(!strcmp((char*) pName, "procid")) { + *pPropID = PROP_PROCID; + } else if(!strcmp((char*) pName, "msgid")) { + *pPropID = PROP_MSGID; + /* here start system properties (those, that do not relate to the message itself */ + } else if(!strcmp((char*) pName, "$now")) { + *pPropID = PROP_SYS_NOW; + } else if(!strcmp((char*) pName, "$year")) { + *pPropID = PROP_SYS_YEAR; + } else if(!strcmp((char*) pName, "$month")) { + *pPropID = PROP_SYS_MONTH; + } else if(!strcmp((char*) pName, "$day")) { + *pPropID = PROP_SYS_DAY; + } else if(!strcmp((char*) pName, "$hour")) { + *pPropID = PROP_SYS_HOUR; + } else if(!strcmp((char*) pName, "$hhour")) { + *pPropID = PROP_SYS_HHOUR; + } else if(!strcmp((char*) pName, "$qhour")) { + *pPropID = PROP_SYS_QHOUR; + } else if(!strcmp((char*) pName, "$minute")) { + *pPropID = PROP_SYS_MINUTE; + } else if(!strcmp((char*) pName, "$myhostname")) { + *pPropID = PROP_SYS_MYHOSTNAME; + } else { + *pPropID = PROP_INVALID; + iRet = RS_RET_VAR_NOT_FOUND; + } + + RETiRet; +} + + +/* map a property ID to a name string (useful for displaying) */ +uchar *propIDToName(propid_t propID) +{ + switch(propID) { + case PROP_MSG: + return UCHAR_CONSTANT("msg"); + case PROP_TIMESTAMP: + return UCHAR_CONSTANT("timestamp"); + case PROP_HOSTNAME: + return UCHAR_CONSTANT("hostname"); + case PROP_SYSLOGTAG: + return UCHAR_CONSTANT("syslogtag"); + case PROP_RAWMSG: + return UCHAR_CONSTANT("rawmsg"); + /* enable this, if someone actually uses UxTradMsg, delete after some time has + * passed and nobody complained -- rgerhards, 2009-06-16 + case PROP_UXTRADMSG: + pRes = getUxTradMsg(pMsg); + break; + */ + case PROP_INPUTNAME: + return UCHAR_CONSTANT("inputname"); + case PROP_FROMHOST: + return UCHAR_CONSTANT("fromhost"); + case PROP_FROMHOST_IP: + return UCHAR_CONSTANT("fromhost-ip"); + case PROP_PRI: + return UCHAR_CONSTANT("pri"); + case PROP_PRI_TEXT: + return UCHAR_CONSTANT("pri-text"); + case PROP_IUT: + return UCHAR_CONSTANT("iut"); + case PROP_SYSLOGFACILITY: + return UCHAR_CONSTANT("syslogfacility"); + case PROP_SYSLOGFACILITY_TEXT: + return UCHAR_CONSTANT("syslogfacility-text"); + case PROP_SYSLOGSEVERITY: + return UCHAR_CONSTANT("syslogseverity"); + case PROP_SYSLOGSEVERITY_TEXT: + return UCHAR_CONSTANT("syslogseverity-text"); + case PROP_TIMEGENERATED: + return UCHAR_CONSTANT("timegenerated"); + case PROP_PROGRAMNAME: + return UCHAR_CONSTANT("programname"); + case PROP_PROTOCOL_VERSION: + return UCHAR_CONSTANT("protocol-version"); + case PROP_STRUCTURED_DATA: + return UCHAR_CONSTANT("structured-data"); + case PROP_APP_NAME: + return UCHAR_CONSTANT("app-name"); + case PROP_PROCID: + return UCHAR_CONSTANT("procid"); + case PROP_MSGID: + return UCHAR_CONSTANT("msgid"); + case PROP_SYS_NOW: + return UCHAR_CONSTANT("$NOW"); + case PROP_SYS_YEAR: + return UCHAR_CONSTANT("$YEAR"); + case PROP_SYS_MONTH: + return UCHAR_CONSTANT("$MONTH"); + case PROP_SYS_DAY: + return UCHAR_CONSTANT("$DAY"); + case PROP_SYS_HOUR: + return UCHAR_CONSTANT("$HOUR"); + case PROP_SYS_HHOUR: + return UCHAR_CONSTANT("$HHOUR"); + case PROP_SYS_QHOUR: + return UCHAR_CONSTANT("$QHOUR"); + case PROP_SYS_MINUTE: + return UCHAR_CONSTANT("$MINUTE"); + case PROP_SYS_MYHOSTNAME: + return UCHAR_CONSTANT("$MYHOSTNAME"); + default: + return UCHAR_CONSTANT("*invalid property id*"); + } +} + /* The following functions will support advanced output module * multithreading, once this is implemented. Currently, we @@ -166,32 +546,9 @@ static void MsgLockingDummy(msg_t __attribute__((unused)) *pMsg) */ static void MsgPrepareEnqueueLockingCase(msg_t *pThis) { - int iErr; BEGINfunc assert(pThis != NULL); - iErr = pthread_mutexattr_init(&pThis->mutAttr); - if(iErr != 0) { - dbgprintf("error initializing mutex attribute in %s:%d, trying to continue\n", - __FILE__, __LINE__); - } - iErr = pthread_mutexattr_settype(&pThis->mutAttr, PTHREAD_MUTEX_RECURSIVE); - if(iErr != 0) { - dbgprintf("ERROR setting mutex attribute to recursive in %s:%d, trying to continue " - "but we will probably either abort or hang soon\n", - __FILE__, __LINE__); - /* TODO: it makes very little sense to continue here, - * but it requires an iRet interface to gracefully shut - * down. We should do that over time. -- rgerhards, 2008-07-14 - */ - } - pthread_mutex_init(&pThis->mut, &pThis->mutAttr); - - /* we do no longer need the attribute. According to the - * POSIX spec, we can destroy it without affecting the - * initialized mutex (that used the attribute). - * rgerhards, 2008-07-14 - */ - pthread_mutexattr_destroy(&pThis->mutAttr); + pthread_mutex_init(&pThis->mut, NULL); pThis->bDoLock = 1; ENDfunc } @@ -256,6 +613,11 @@ rsRetVal MsgEnableThreadSafety(void) * itself but rather uses a user-supplied value. This enables the caller * to do some tricks to save processing time (done, for example, in the * udp input). + * NOTE: this constructor does NOT call calloc(), as we have many bytes + * inside the structure which do not need to be cleared. bzero() will + * heavily thrash the cache, so we do the init manually (which also + * is the right thing to do with pointers, as they are not neccessarily + * a binary 0 on all machines [but today almost always...]). * rgerhards, 2008-10-06 */ static inline rsRetVal msgBaseConstruct(msg_t **ppThis) @@ -264,14 +626,50 @@ static inline rsRetVal msgBaseConstruct(msg_t **ppThis) msg_t *pM; assert(ppThis != NULL); - if((pM = calloc(1, sizeof(msg_t))) == NULL) - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + CHKmalloc(pM = malloc(sizeof(msg_t))); + objConstructSetObjInfo(pM); /* intialize object helper entities */ - /* initialize members that are non-zero */ + /* initialize members in ORDER they appear in structure (think "cache line"!) */ + pM->flowCtlType = 0; + pM->bDoLock = 0; + pM->bParseHOSTNAME = 0; pM->iRefCount = 1; pM->iSeverity = -1; pM->iFacility = -1; - objConstructSetObjInfo(pM); + pM->offAfterPRI = 0; + pM->offMSG = -1; + pM->iProtocolVersion = 0; + pM->msgFlags = 0; + pM->iLenRawMsg = 0; + pM->iLenMSG = 0; + pM->iLenTAG = 0; + pM->iLenHOSTNAME = 0; + pM->pszRawMsg = NULL; + pM->pszHOSTNAME = NULL; + pM->pszRcvdAt3164 = NULL; + pM->pszRcvdAt3339 = NULL; + pM->pszRcvdAt_MySQL = NULL; + pM->pszRcvdAt_PgSQL = NULL; + pM->pszTIMESTAMP3164 = NULL; + pM->pszTIMESTAMP3339 = NULL; + pM->pszTIMESTAMP_MySQL = NULL; + pM->pszTIMESTAMP_PgSQL = NULL; + pM->pCSProgName = NULL; + pM->pCSStrucData = NULL; + pM->pCSAPPNAME = NULL; + pM->pCSPROCID = NULL; + pM->pCSMSGID = NULL; + pM->pInputName = NULL; + pM->pRcvFromIP = NULL; + pM->pRcvFrom = NULL; + pM->pRuleset = NULL; + memset(&pM->tRcvdAt, 0, sizeof(pM->tRcvdAt)); + memset(&pM->tTIMESTAMP, 0, sizeof(pM->tTIMESTAMP)); + pM->TAG.pszTAG = NULL; + pM->pszTimestamp3164[0] = '\0'; + pM->pszTimestamp3339[0] = '\0'; + pM->pszTIMESTAMP_SecFrac[0] = '\0'; + pM->pszRcvdAt_SecFrac[0] = '\0'; /* DEV debugging only! dbgprintf("msgConstruct\t0x%x, ref 1\n", (int)pM);*/ @@ -330,6 +728,21 @@ finalize_it: } +/* some free handlers for (slightly) complicated cases... All of them may be called + * with an empty element. + */ +static inline void freeTAG(msg_t *pThis) +{ + if(pThis->iLenTAG >= CONF_TAG_BUFSIZE) + free(pThis->TAG.pszTAG); +} +static inline void freeHOSTNAME(msg_t *pThis) +{ + if(pThis->iLenHOSTNAME >= CONF_HOSTNAME_BUFSIZE) + free(pThis->pszHOSTNAME); +} + + BEGINobjDestruct(msg) /* be sure to specify the object type also in END and CODESTART macros! */ int currRefCount; CODESTARTobjDestruct(msg) @@ -343,25 +756,20 @@ CODESTARTobjDestruct(msg) if(currRefCount == 0) { /* DEV Debugging Only! dbgprintf("msgDestruct\t0x%lx, RefCount now 0, doing DESTROY\n", (unsigned long)pThis); */ - free(pThis->pszRawMsg); - free(pThis->pszTAG); - free(pThis->pszHOSTNAME); - free(pThis->pszInputName); - free(pThis->pszRcvFrom); - free(pThis->pszRcvFromIP); - free(pThis->pszMSG); - free(pThis->pszFacility); - free(pThis->pszFacilityStr); - free(pThis->pszSeverity); - free(pThis->pszSeverityStr); + if(pThis->pszRawMsg != pThis->szRawMsg) + free(pThis->pszRawMsg); + freeTAG(pThis); + freeHOSTNAME(pThis); + if(pThis->pInputName != NULL) + prop.Destruct(&pThis->pInputName); + if(pThis->pRcvFrom != NULL) + prop.Destruct(&pThis->pRcvFrom); + if(pThis->pRcvFromIP != NULL) + prop.Destruct(&pThis->pRcvFromIP); free(pThis->pszRcvdAt3164); free(pThis->pszRcvdAt3339); - free(pThis->pszRcvdAt_SecFrac); free(pThis->pszRcvdAt_MySQL); free(pThis->pszRcvdAt_PgSQL); - free(pThis->pszTIMESTAMP3164); - free(pThis->pszTIMESTAMP3339); - free(pThis->pszTIMESTAMP_SecFrac); free(pThis->pszTIMESTAMP_MySQL); free(pThis->pszTIMESTAMP_PgSQL); if(pThis->pCSProgName != NULL) @@ -378,6 +786,25 @@ CODESTARTobjDestruct(msg) MsgUnlock(pThis); # endif funcDeleteMutex(pThis); + /* now we need to do our own optimization. Testing has shown that at least the glibc + * malloc() subsystem returns memory to the OS far too late in our case. So we need + * to help it a bit, by calling malloc_trim(), which will tell the alloc subsystem + * to consolidate and return to the OS. We keep 128K for our use, as a safeguard + * to too-frequent reallocs. But more importantly, we call this hook only every + * 100,000 messages (which is an approximation, as we do not work with atomic + * operations on the counter. --- rgerhards, 2009-06-22. + */ +# if HAVE_MALLOC_TRIM + { /* standard C requires a new block for a new variable definition! + * To simplify matters, we use modulo arithmetic and live with the fact + * that we trim too often when the counter wraps. + */ + static unsigned iTrimCtr = 1; + if(ATOMIC_INC_AND_FETCH(iTrimCtr) % 100000 == 0) { + malloc_trim(128*1024); + } + } +# endif } else { # ifndef HAVE_ATOMIC_BUILTINS MsgUnlock(pThis); @@ -438,21 +865,50 @@ msg_t* MsgDup(msg_t* pOld) pNew->msgFlags = pOld->msgFlags; pNew->iProtocolVersion = pOld->iProtocolVersion; pNew->ttGenTime = pOld->ttGenTime; - /* enable this, if someone actually uses UxTradMsg, delete after some time has + pNew->offMSG = pOld->offMSG; + pNew->iLenRawMsg = pOld->iLenRawMsg; + pNew->iLenMSG = pOld->iLenMSG; + pNew->iLenTAG = pOld->iLenTAG; + pNew->iLenHOSTNAME = pOld->iLenHOSTNAME; + if(pOld->pRcvFrom != NULL) { + pNew->pRcvFrom = pOld->pRcvFrom; + prop.AddRef(pNew->pRcvFrom); + } + if(pOld->pRcvFromIP != NULL) { + pNew->pRcvFromIP = pOld->pRcvFromIP; + prop.AddRef(pNew->pRcvFromIP); + } + if(pOld->pInputName != NULL) { + pNew->pInputName = pOld->pInputName; + prop.AddRef(pNew->pInputName); + } + /* enable this, if someone actually uses UxTradMsg, delete after some time has * passed and nobody complained -- rgerhards, 2009-06-16 pNew->offAfterPRI = pOld->offAfterPRI; */ - memcpy(pNew->bufPRI, pOld->bufPRI, pOld->iLenPRI); - pNew->iLenPRI = pOld->iLenPRI; - tmpCOPYSZ(Severity); - tmpCOPYSZ(SeverityStr); - tmpCOPYSZ(Facility); - tmpCOPYSZ(FacilityStr); - tmpCOPYSZ(RawMsg); - tmpCOPYSZ(MSG); - tmpCOPYSZ(TAG); - tmpCOPYSZ(HOSTNAME); - tmpCOPYSZ(RcvFrom); + if(pOld->iLenTAG > 0) { + if(pOld->iLenTAG < CONF_TAG_BUFSIZE) { + memcpy(pNew->TAG.szBuf, pOld->TAG.szBuf, pOld->iLenTAG); + } else { + if((pNew->TAG.pszTAG = srUtilStrDup(pOld->TAG.pszTAG, pOld->iLenTAG)) == NULL) { + msgDestruct(&pNew); + return NULL; + } + pNew->iLenTAG = pOld->iLenTAG; + } + } + if(pOld->iLenRawMsg < CONF_RAWMSG_BUFSIZE) { + memcpy(pNew->szRawMsg, pOld->szRawMsg, pOld->iLenRawMsg + 1); + pNew->pszRawMsg = pNew->szRawMsg; + } else { + tmpCOPYSZ(RawMsg); + } + if(pOld->iLenHOSTNAME < CONF_HOSTNAME_BUFSIZE) { + memcpy(pNew->szHOSTNAME, pOld->szHOSTNAME, pOld->iLenHOSTNAME + 1); + pNew->pszHOSTNAME = pNew->szHOSTNAME; + } else { + tmpCOPYSZ(HOSTNAME); + } tmpCOPYCSTR(ProgName); tmpCOPYCSTR(StrucData); @@ -485,11 +941,14 @@ msg_t* MsgDup(msg_t* pOld) */ static rsRetVal MsgSerialize(msg_t *pThis, strm_t *pStrm) { + uchar *psz; + int len; DEFiRet; assert(pThis != NULL); assert(pStrm != NULL); + /* then serialize elements */ CHKiRet(obj.BeginSerialize(pStrm, (obj_t*) pThis)); objSerializeSCALAR(pStrm, iProtocolVersion, SHORT); objSerializeSCALAR(pStrm, iSeverity, SHORT); @@ -503,19 +962,28 @@ static rsRetVal MsgSerialize(msg_t *pThis, strm_t *pStrm) objSerializeSCALAR(pStrm, offsAfterPRI, SHORT); */ + CHKiRet(obj.SerializeProp(pStrm, UCHAR_CONSTANT("pszTAG"), PROPTYPE_PSZ, (void*) + ((pThis->iLenTAG < CONF_TAG_BUFSIZE) ? pThis->TAG.szBuf : pThis->TAG.pszTAG))); + objSerializePTR(pStrm, pszRawMsg, PSZ); - objSerializePTR(pStrm, pszMSG, PSZ); - objSerializePTR(pStrm, pszTAG, PSZ); objSerializePTR(pStrm, pszHOSTNAME, PSZ); - objSerializePTR(pStrm, pszInputName, PSZ); - objSerializePTR(pStrm, pszRcvFrom, PSZ); - objSerializePTR(pStrm, pszRcvFromIP, PSZ); + getInputName(pThis, &psz, &len); + CHKiRet(obj.SerializeProp(pStrm, UCHAR_CONSTANT("pszInputName"), PROPTYPE_PSZ, (void*) psz)); + psz = getRcvFrom(pThis); + CHKiRet(obj.SerializeProp(pStrm, UCHAR_CONSTANT("pszRcvFrom"), PROPTYPE_PSZ, (void*) psz)); + psz = getRcvFromIP(pThis); + CHKiRet(obj.SerializeProp(pStrm, UCHAR_CONSTANT("pszRcvFromIP"), PROPTYPE_PSZ, (void*) psz)); objSerializePTR(pStrm, pCSStrucData, CSTR); objSerializePTR(pStrm, pCSAPPNAME, CSTR); objSerializePTR(pStrm, pCSPROCID, CSTR); objSerializePTR(pStrm, pCSMSGID, CSTR); + /* offset must be serialized after pszRawMsg, because we need that to obtain the correct + * MSG size. + */ + objSerializeSCALAR(pStrm, offMSG, SHORT); + CHKiRet(obj.EndSerialize(pStrm)); finalize_it: @@ -554,22 +1022,27 @@ msg_t *MsgAddRef(msg_t *pM) * can obtain a PROCID. Take in mind that not every legacy syslog message * actually has a PROCID. * rgerhards, 2005-11-24 + * THIS MUST be called with the message lock locked. */ static rsRetVal aquirePROCIDFromTAG(msg_t *pM) { register int i; + uchar *pszTag; DEFiRet; assert(pM != NULL); + if(pM->pCSPROCID != NULL) return RS_RET_OK; /* we are already done ;) */ if(getProtocolVersion(pM) != 0) return RS_RET_OK; /* we can only emulate if we have legacy format */ + pszTag = (uchar*) ((pM->iLenTAG < CONF_TAG_BUFSIZE) ? pM->TAG.szBuf : pM->TAG.pszTAG); + /* find first '['... */ i = 0; - while((i < pM->iLenTAG) && (pM->pszTAG[i] != '[')) + while((i < pM->iLenTAG) && (pszTag[i] != '[')) ++i; if(!(i < pM->iLenTAG)) return RS_RET_OK; /* no [, so can not emulate... */ @@ -577,10 +1050,9 @@ static rsRetVal aquirePROCIDFromTAG(msg_t *pM) ++i; /* skip '[' */ /* now obtain the PROCID string... */ - CHKiRet(rsCStrConstruct(&pM->pCSPROCID)); - rsCStrSetAllocIncrement(pM->pCSPROCID, 16); - while((i < pM->iLenTAG) && (pM->pszTAG[i] != ']')) { - CHKiRet(rsCStrAppendChar(pM->pCSPROCID, pM->pszTAG[i])); + CHKiRet(cstrConstruct(&pM->pCSPROCID)); + while((i < pM->iLenTAG) && (pszTag[i] != ']')) { + CHKiRet(cstrAppendChar(pM->pCSPROCID, pszTag[i])); ++i; } @@ -590,7 +1062,7 @@ static rsRetVal aquirePROCIDFromTAG(msg_t *pM) * the buffer and simply return. Note that this is NOT an error * case! */ - rsCStrDestruct(&pM->pCSPROCID); + cstrDestruct(&pM->pCSPROCID); FINALIZE; } @@ -615,27 +1087,27 @@ finalize_it: * The program name is not parsed by default, because it is infrequently-used. * If it is needed, this function should be called first. It checks if it is * already set and extracts it, if not. - * A message object must be provided, else a crash will occur. + * + * IMPORTANT: A locked message object must be provided, else a crash will occur. * rgerhards, 2005-10-19 */ static rsRetVal aquireProgramName(msg_t *pM) { - DEFiRet; register int i; + uchar *pszTag; + DEFiRet; assert(pM != NULL); if(pM->pCSProgName == NULL) { - /* ok, we do not yet have it. So let's parse the TAG - * to obtain it. - */ - CHKiRet(rsCStrConstruct(&pM->pCSProgName)); - rsCStrSetAllocIncrement(pM->pCSProgName, 33); + /* ok, we do not yet have it. So let's parse the TAG to obtain it. */ + pszTag = (uchar*) ((pM->iLenTAG < CONF_TAG_BUFSIZE) ? pM->TAG.szBuf : pM->TAG.pszTAG); + CHKiRet(cstrConstruct(&pM->pCSProgName)); for( i = 0 - ; (i < pM->iLenTAG) && isprint((int) pM->pszTAG[i]) - && (pM->pszTAG[i] != '\0') && (pM->pszTAG[i] != ':') - && (pM->pszTAG[i] != '[') && (pM->pszTAG[i] != '/') + ; (i < pM->iLenTAG) && isprint((int) pszTag[i]) + && (pszTag[i] != '\0') && (pszTag[i] != ':') + && (pszTag[i] != '[') && (pszTag[i] != '/') ; ++i) { - CHKiRet(rsCStrAppendChar(pM->pCSProgName, pM->pszTAG[i])); + CHKiRet(cstrAppendChar(pM->pCSProgName, pszTag[i])); } CHKiRet(cstrFinalize(pM->pCSProgName)); } @@ -644,28 +1116,6 @@ finalize_it: } -/* This function moves the HOSTNAME inside the message object to the - * TAG. It is a specialised function used to handle the condition when - * a message without HOSTNAME is being processed. The missing HOSTNAME - * is only detected at a later stage, during TAG processing, so that - * we already had set the HOSTNAME property and now need to move it to - * the TAG. Of course, we could do this via a couple of get/set methods, - * but it is far more efficient to do it via this specialised method. - * This is especially important as this can be a very common case, e.g. - * when BSD syslog is acting as a sender. - * rgerhards, 2005-11-10. - */ -void moveHOSTNAMEtoTAG(msg_t *pM) -{ - assert(pM != NULL); - if(pM->pszTAG != NULL) - free(pM->pszTAG); - pM->pszTAG = pM->pszHOSTNAME; - pM->iLenTAG = pM->iLenHOSTNAME; - pM->pszHOSTNAME = NULL; - pM->iLenHOSTNAME = 0; -} - /* Access methods - dumb & easy, not a comment for each ;) */ void setProtocolVersion(msg_t *pM, int iNewVersion) @@ -678,12 +1128,6 @@ void setProtocolVersion(msg_t *pM, int iNewVersion) pM->iProtocolVersion = iNewVersion; } -static int getProtocolVersion(msg_t *pM) -{ - assert(pM != NULL); - return(pM->iProtocolVersion); -} - /* note: string is taken from constant pool, do NOT free */ char *getProtocolVersionString(msg_t *pM) { @@ -691,11 +1135,6 @@ char *getProtocolVersionString(msg_t *pM) return(pM->iProtocolVersion ? "1" : "0"); } -int getMSGLen(msg_t *pM) -{ - return((pM == NULL) ? 0 : pM->iLenMSG); -} - static char *getRawMsg(msg_t *pM) { @@ -720,42 +1159,48 @@ char *getUxTradMsg(msg_t *pM) } */ -char *getMSG(msg_t *pM) + +int getMSGLen(msg_t *pM) +{ + return((pM == NULL) ? 0 : pM->iLenMSG); +} + +uchar *getMSG(msg_t *pM) { + uchar *ret; if(pM == NULL) - return ""; - else - if(pM->pszMSG == NULL) - return ""; + ret = UCHAR_CONSTANT(""); + else { + if(pM->iLenMSG == 0) + ret = UCHAR_CONSTANT(""); else - return (char*)pM->pszMSG; + ret = pM->pszRawMsg + pM->offMSG; + } + return ret; } /* Get PRI value as integer */ static int getPRIi(msg_t *pM) { - assert(pM != NULL); return (pM->iFacility << 3) + (pM->iSeverity); } -/* Get PRI value in text form */ +/* Get PRI value in text form + */ static inline char *getPRI(msg_t *pM) { + /* PRI is a number in the range 0..191. Thus, we use a simple lookup table to obtain the + * string value. It looks a bit clumpsy here in code ;) + */ + int iPRI; + if(pM == NULL) return ""; - /* there are some cases where bufPRI may not contain a valid string, - * and then we need to build it. - */ - MsgLock(pM); - if(pM->bufPRI[0] == '\0') { - snprintf((char*)pM->bufPRI, sizeof(pM->bufPRI), "%d", getPRIi(pM)); - } - MsgUnlock(pM); - - return (char*)pM->bufPRI; + iPRI = getPRIi(pM); + return (iPRI > 191) ? "invld" : (char*)syslog_pri_names[iPRI].pszName; } @@ -767,13 +1212,13 @@ static inline char *getTimeReported(msg_t *pM, enum tplFormatTypes eFmt) switch(eFmt) { case tplFmtDefault: + case tplFmtRFC3164Date: + case tplFmtRFC3164BuggyDate: MsgLock(pM); if(pM->pszTIMESTAMP3164 == NULL) { - if((pM->pszTIMESTAMP3164 = malloc(16)) == NULL) { - MsgUnlock(pM); - return ""; - } - datetime.formatTimestamp3164(&pM->tTIMESTAMP, pM->pszTIMESTAMP3164, 16); + pM->pszTIMESTAMP3164 = pM->pszTimestamp3164; + datetime.formatTimestamp3164(&pM->tTIMESTAMP, pM->pszTIMESTAMP3164, + (eFmt == tplFmtRFC3164BuggyDate)); } MsgUnlock(pM); return(pM->pszTIMESTAMP3164); @@ -784,7 +1229,7 @@ static inline char *getTimeReported(msg_t *pM, enum tplFormatTypes eFmt) MsgUnlock(pM); return ""; } - datetime.formatTimestampToMySQL(&pM->tTIMESTAMP, pM->pszTIMESTAMP_MySQL, 15); + datetime.formatTimestampToMySQL(&pM->tTIMESTAMP, pM->pszTIMESTAMP_MySQL); } MsgUnlock(pM); return(pM->pszTIMESTAMP_MySQL); @@ -795,42 +1240,27 @@ static inline char *getTimeReported(msg_t *pM, enum tplFormatTypes eFmt) MsgUnlock(pM); return ""; } - datetime.formatTimestampToPgSQL(&pM->tTIMESTAMP, pM->pszTIMESTAMP_PgSQL, 21); + datetime.formatTimestampToPgSQL(&pM->tTIMESTAMP, pM->pszTIMESTAMP_PgSQL); } MsgUnlock(pM); return(pM->pszTIMESTAMP_PgSQL); - case tplFmtRFC3164Date: - MsgLock(pM); - if(pM->pszTIMESTAMP3164 == NULL) { - if((pM->pszTIMESTAMP3164 = malloc(16)) == NULL) { - MsgUnlock(pM); - return ""; - } - datetime.formatTimestamp3164(&pM->tTIMESTAMP, pM->pszTIMESTAMP3164, 16); - } - MsgUnlock(pM); - return(pM->pszTIMESTAMP3164); case tplFmtRFC3339Date: MsgLock(pM); if(pM->pszTIMESTAMP3339 == NULL) { - if((pM->pszTIMESTAMP3339 = malloc(33)) == NULL) { - MsgUnlock(pM); - return ""; /* TODO: check this: can it cause a free() of constant memory?) */ - } - datetime.formatTimestamp3339(&pM->tTIMESTAMP, pM->pszTIMESTAMP3339, 33); + pM->pszTIMESTAMP3339 = pM->pszTimestamp3339; + datetime.formatTimestamp3339(&pM->tTIMESTAMP, pM->pszTIMESTAMP3339); } MsgUnlock(pM); return(pM->pszTIMESTAMP3339); case tplFmtSecFrac: - MsgLock(pM); - if(pM->pszTIMESTAMP_SecFrac == NULL) { - if((pM->pszTIMESTAMP_SecFrac = malloc(10)) == NULL) { - MsgUnlock(pM); - return ""; /* TODO: check this: can it cause a free() of constant memory?) */ + if(pM->pszTIMESTAMP_SecFrac[0] == '\0') { + MsgLock(pM); + /* re-check, may have changed while we did not hold lock */ + if(pM->pszTIMESTAMP_SecFrac[0] == '\0') { + datetime.formatTimestampSecFrac(&pM->tTIMESTAMP, pM->pszTIMESTAMP_SecFrac); } - datetime.formatTimestampSecFrac(&pM->tTIMESTAMP, pM->pszTIMESTAMP_SecFrac, 10); + MsgUnlock(pM); } - MsgUnlock(pM); return(pM->pszTIMESTAMP_SecFrac); } ENDfunc @@ -851,7 +1281,7 @@ static inline char *getTimeGenerated(msg_t *pM, enum tplFormatTypes eFmt) MsgUnlock(pM); return ""; } - datetime.formatTimestamp3164(&pM->tRcvdAt, pM->pszRcvdAt3164, 16); + datetime.formatTimestamp3164(&pM->tRcvdAt, pM->pszRcvdAt3164, 0); } MsgUnlock(pM); return(pM->pszRcvdAt3164); @@ -862,7 +1292,7 @@ static inline char *getTimeGenerated(msg_t *pM, enum tplFormatTypes eFmt) MsgUnlock(pM); return ""; } - datetime.formatTimestampToMySQL(&pM->tRcvdAt, pM->pszRcvdAt_MySQL, 15); + datetime.formatTimestampToMySQL(&pM->tRcvdAt, pM->pszRcvdAt_MySQL); } MsgUnlock(pM); return(pM->pszRcvdAt_MySQL); @@ -873,18 +1303,20 @@ static inline char *getTimeGenerated(msg_t *pM, enum tplFormatTypes eFmt) MsgUnlock(pM); return ""; } - datetime.formatTimestampToPgSQL(&pM->tRcvdAt, pM->pszRcvdAt_PgSQL, 21); + datetime.formatTimestampToPgSQL(&pM->tRcvdAt, pM->pszRcvdAt_PgSQL); } MsgUnlock(pM); return(pM->pszRcvdAt_PgSQL); case tplFmtRFC3164Date: + case tplFmtRFC3164BuggyDate: MsgLock(pM); if(pM->pszRcvdAt3164 == NULL) { if((pM->pszRcvdAt3164 = malloc(16)) == NULL) { MsgUnlock(pM); return ""; } - datetime.formatTimestamp3164(&pM->tRcvdAt, pM->pszRcvdAt3164, 16); + datetime.formatTimestamp3164(&pM->tRcvdAt, pM->pszRcvdAt3164, + (eFmt == tplFmtRFC3164BuggyDate)); } MsgUnlock(pM); return(pM->pszRcvdAt3164); @@ -895,20 +1327,19 @@ static inline char *getTimeGenerated(msg_t *pM, enum tplFormatTypes eFmt) MsgUnlock(pM); return ""; } - datetime.formatTimestamp3339(&pM->tRcvdAt, pM->pszRcvdAt3339, 33); + datetime.formatTimestamp3339(&pM->tRcvdAt, pM->pszRcvdAt3339); } MsgUnlock(pM); return(pM->pszRcvdAt3339); case tplFmtSecFrac: - MsgLock(pM); - if(pM->pszRcvdAt_SecFrac == NULL) { - if((pM->pszRcvdAt_SecFrac = malloc(10)) == NULL) { - MsgUnlock(pM); - return ""; /* TODO: check this: can it cause a free() of constant memory?) */ + if(pM->pszRcvdAt_SecFrac[0] == '\0') { + MsgLock(pM); + /* re-check, may have changed while we did not hold lock */ + if(pM->pszRcvdAt_SecFrac[0] == '\0') { + datetime.formatTimestampSecFrac(&pM->tRcvdAt, pM->pszRcvdAt_SecFrac); } - datetime.formatTimestampSecFrac(&pM->tRcvdAt, pM->pszRcvdAt_SecFrac, 10); + MsgUnlock(pM); } - MsgUnlock(pM); return(pM->pszRcvdAt_SecFrac); } ENDfunc @@ -918,101 +1349,67 @@ static inline char *getTimeGenerated(msg_t *pM, enum tplFormatTypes eFmt) static inline char *getSeverity(msg_t *pM) { + char *name = NULL; + if(pM == NULL) return ""; - MsgLock(pM); - if(pM->pszSeverity == NULL) { - /* we use a 2 byte buffer - can only be one digit */ - if((pM->pszSeverity = malloc(2)) == NULL) { MsgUnlock(pM) ; return ""; } - pM->iLenSeverity = - snprintf((char*)pM->pszSeverity, 2, "%d", pM->iSeverity); + if(pM->iSeverity < 0 || pM->iSeverity > 7) { + name = "invld"; + } else { + name = syslog_number_names[pM->iSeverity]; } - MsgUnlock(pM); - return((char*)pM->pszSeverity); + + return name; } static inline char *getSeverityStr(msg_t *pM) { - syslogCODE *c; - int val; char *name = NULL; if(pM == NULL) return ""; - MsgLock(pM); - if(pM->pszSeverityStr == NULL) { - for(c = rs_prioritynames, val = pM->iSeverity; c->c_name; c++) - if(c->c_val == val) { - name = c->c_name; - break; - } - if(name == NULL) { - /* we use a 2 byte buffer - can only be one digit */ - if((pM->pszSeverityStr = malloc(2)) == NULL) { MsgUnlock(pM) ; return ""; } - pM->iLenSeverityStr = - snprintf((char*)pM->pszSeverityStr, 2, "%d", pM->iSeverity); - } else { - if((pM->pszSeverityStr = (uchar*) strdup(name)) == NULL) { MsgUnlock(pM) ; return ""; } - pM->iLenSeverityStr = strlen((char*)name); - } + if(pM->iSeverity < 0 || pM->iSeverity > 7) { + name = "invld"; + } else { + name = syslog_severity_names[pM->iSeverity]; } - MsgUnlock(pM); - return((char*)pM->pszSeverityStr); + + return name; } static inline char *getFacility(msg_t *pM) { + char *name = NULL; + if(pM == NULL) return ""; - MsgLock(pM); - if(pM->pszFacility == NULL) { - /* we use a 12 byte buffer - as of - * syslog-protocol, facility can go - * up to 2^32 -1 - */ - if((pM->pszFacility = malloc(12)) == NULL) { MsgUnlock(pM) ; return ""; } - pM->iLenFacility = - snprintf((char*)pM->pszFacility, 12, "%d", pM->iFacility); + if(pM->iFacility < 0 || pM->iFacility > 23) { + name = "invld"; + } else { + name = syslog_number_names[pM->iFacility]; } - MsgUnlock(pM); - return((char*)pM->pszFacility); + + return name; } static inline char *getFacilityStr(msg_t *pM) { - syslogCODE *c; - int val; char *name = NULL; if(pM == NULL) return ""; - MsgLock(pM); - if(pM->pszFacilityStr == NULL) { - for(c = rs_facilitynames, val = pM->iFacility << 3; c->c_name; c++) - if(c->c_val == val) { - name = c->c_name; - break; - } - if(name == NULL) { - /* we use a 12 byte buffer - as of - * syslog-protocol, facility can go - * up to 2^32 -1 - */ - if((pM->pszFacilityStr = malloc(12)) == NULL) { MsgUnlock(pM) ; return ""; } - pM->iLenFacilityStr = - snprintf((char*)pM->pszFacilityStr, 12, "%d", val >> 3); - } else { - if((pM->pszFacilityStr = (uchar*)strdup(name)) == NULL) { MsgUnlock(pM) ; return ""; } - pM->iLenFacilityStr = strlen((char*)name); - } - } - MsgUnlock(pM); - return((char*)pM->pszFacilityStr); + if(pM->iFacility < 0 || pM->iFacility > 23) { + name = "invld"; + } else { + name = syslog_fac_names[pM->iFacility]; + } + + return name; } @@ -1047,7 +1444,10 @@ MsgSetAfterPRIOffs(msg_t *pMsg, short offs) /* rgerhards 2004-11-24: set APP-NAME in msg object - * TODO: revisit msg locking code! + * This is not locked, because it either is called during message + * construction (where we need no locking) or later as part of a function + * which already obtained the lock. So in general, this function here must + * only be called when it it safe to do so without it aquiring a lock. */ rsRetVal MsgSetAPPNAME(msg_t *pMsg, char* pszAPPNAME) { @@ -1056,7 +1456,6 @@ rsRetVal MsgSetAPPNAME(msg_t *pMsg, char* pszAPPNAME) if(pMsg->pCSAPPNAME == NULL) { /* we need to obtain the object first */ CHKiRet(rsCStrConstruct(&pMsg->pCSAPPNAME)); - rsCStrSetAllocIncrement(pMsg->pCSAPPNAME, 128); } /* if we reach this point, we have the object */ iRet = rsCStrSetSzStr(pMsg->pCSAPPNAME, (uchar*) pszAPPNAME); @@ -1066,20 +1465,6 @@ finalize_it: } -static void tryEmulateAPPNAME(msg_t *pM); /* forward reference */ -/* rgerhards, 2005-11-24 - */ -char *getAPPNAME(msg_t *pM) -{ - assert(pM != NULL); - MsgLock(pM); - if(pM->pCSAPPNAME == NULL) - tryEmulateAPPNAME(pM); - MsgUnlock(pM); - return (pM->pCSAPPNAME == NULL) ? "" : (char*) rsCStrGetSzStrNoNULL(pM->pCSAPPNAME); -} - - /* rgerhards 2004-11-24: set PROCID in msg object */ rsRetVal MsgSetPROCID(msg_t *pMsg, char* pszPROCID) @@ -1088,42 +1473,54 @@ rsRetVal MsgSetPROCID(msg_t *pMsg, char* pszPROCID) ISOBJ_TYPE_assert(pMsg, msg); if(pMsg->pCSPROCID == NULL) { /* we need to obtain the object first */ - CHKiRet(rsCStrConstruct(&pMsg->pCSPROCID)); - rsCStrSetAllocIncrement(pMsg->pCSPROCID, 128); + CHKiRet(cstrConstruct(&pMsg->pCSPROCID)); } /* if we reach this point, we have the object */ iRet = rsCStrSetSzStr(pMsg->pCSPROCID, (uchar*) pszPROCID); + CHKiRet(cstrFinalize(pMsg->pCSPROCID)); finalize_it: RETiRet; } + +/* check if we have a procid, and, if not, try to aquire/emulate it. + * This must be called WITHOUT the message lock being held. + * rgerhards, 2009-06-26 + */ +static inline void preparePROCID(msg_t *pM, bool bLockMutex) +{ + if(pM->pCSPROCID == NULL) { + if(bLockMutex == LOCK_MUTEX) + MsgLock(pM); + /* re-query, things may have changed in the mean time... */ + if(pM->pCSPROCID == NULL) + aquirePROCIDFromTAG(pM); + if(bLockMutex == LOCK_MUTEX) + MsgUnlock(pM); + } +} + + +#if 0 /* rgerhards, 2005-11-24 */ -static inline int getPROCIDLen(msg_t *pM) +static inline int getPROCIDLen(msg_t *pM, bool bLockMutex) { assert(pM != NULL); - MsgLock(pM); - if(pM->pCSPROCID == NULL) - aquirePROCIDFromTAG(pM); - MsgUnlock(pM); + preparePROCID(pM, bLockMutex); return (pM->pCSPROCID == NULL) ? 1 : rsCStrLen(pM->pCSPROCID); } +#endif /* rgerhards, 2005-11-24 */ -char *getPROCID(msg_t *pM) +char *getPROCID(msg_t *pM, bool bLockMutex) { - char* pszRet; - ISOBJ_TYPE_assert(pM, msg); - MsgLock(pM); - if(pM->pCSPROCID == NULL) - aquirePROCIDFromTAG(pM); - pszRet = (pM->pCSPROCID == NULL) ? "-" : (char*) rsCStrGetSzStrNoNULL(pM->pCSPROCID); - MsgUnlock(pM); - return pszRet; + preparePROCID(pM, bLockMutex); + return (pM->pCSPROCID == NULL) ? "-" : (char*) cstrGetSzStrNoNULL(pM->pCSPROCID); } @@ -1136,7 +1533,6 @@ rsRetVal MsgSetMSGID(msg_t *pMsg, char* pszMSGID) if(pMsg->pCSMSGID == NULL) { /* we need to obtain the object first */ CHKiRet(rsCStrConstruct(&pMsg->pCSMSGID)); - rsCStrSetAllocIncrement(pMsg->pCSMSGID, 128); } /* if we reach this point, we have the object */ iRet = rsCStrSetSzStr(pMsg->pCSMSGID, (uchar*) pszMSGID); @@ -1154,30 +1550,41 @@ static inline char *getMSGID(msg_t *pM) } -/* Set the TAG to a caller-provided string. This is thought - * to be a heap buffer that the caller will no longer use. This - * function is a performance optimization over MsgSetTAG(). - * rgerhards 2004-11-19 +/* rgerhards 2009-06-12: set associated ruleset */ -void MsgAssignTAG(msg_t *pMsg, uchar *pBuf) +void MsgSetRuleset(msg_t *pMsg, ruleset_t *pRuleset) { assert(pMsg != NULL); - pMsg->iLenTAG = (pBuf == NULL) ? 0 : strlen((char*)pBuf); - pMsg->pszTAG = (uchar*) pBuf; + pMsg->pRuleset = pRuleset; } -/* rgerhards 2004-11-16: set TAG in msg object +/* set TAG in msg object + * (rewritten 2009-06-18 rgerhards) */ -void MsgSetTAG(msg_t *pMsg, char* pszTAG) +void MsgSetTAG(msg_t *pMsg, uchar* pszBuf, size_t lenBuf) { + uchar *pBuf; assert(pMsg != NULL); - free(pMsg->pszTAG); - pMsg->iLenTAG = strlen(pszTAG); - if((pMsg->pszTAG = malloc(pMsg->iLenTAG + 1)) != NULL) - memcpy(pMsg->pszTAG, pszTAG, pMsg->iLenTAG + 1); - else - dbgprintf("Could not allocate memory in MsgSetTAG()\n"); + + freeTAG(pMsg); + + pMsg->iLenTAG = lenBuf; + if(pMsg->iLenTAG < CONF_TAG_BUFSIZE) { + /* small enough: use fixed buffer (faster!) */ + pBuf = pMsg->TAG.szBuf; + } else { + if((pBuf = (uchar*) malloc(pMsg->iLenTAG + 1)) == NULL) { + /* truncate message, better than completely loosing it... */ + pBuf = pMsg->TAG.szBuf; + pMsg->iLenTAG = CONF_TAG_BUFSIZE - 1; + } else { + pMsg->TAG.pszTAG = pBuf; + } + } + + memcpy(pBuf, pszBuf, pMsg->iLenTAG); + pBuf[pMsg->iLenTAG] = '\0'; /* this also works with truncation! */ } @@ -1188,63 +1595,51 @@ void MsgSetTAG(msg_t *pMsg, char* pszTAG) * if there is a TAG and, if not, if it can emulate it. * rgerhards, 2005-11-24 */ -static void tryEmulateTAG(msg_t *pM) +static inline void tryEmulateTAG(msg_t *pM, bool bLockMutex) { - int iTAGLen; - uchar *pBuf; + size_t lenTAG; + uchar bufTAG[CONF_TAG_MAXSIZE]; assert(pM != NULL); - if(pM->pszTAG != NULL) + if(bLockMutex == LOCK_MUTEX) + MsgLock(pM); + if(pM->iLenTAG > 0) return; /* done, no need to emulate */ if(getProtocolVersion(pM) == 1) { - if(!strcmp(getPROCID(pM), "-")) { + if(!strcmp(getPROCID(pM, MUTEX_ALREADY_LOCKED), "-")) { /* no process ID, use APP-NAME only */ - MsgSetTAG(pM, getAPPNAME(pM)); + MsgSetTAG(pM, (uchar*) getAPPNAME(pM, MUTEX_ALREADY_LOCKED), getAPPNAMELen(pM, MUTEX_ALREADY_LOCKED)); } else { /* now we can try to emulate */ - iTAGLen = getAPPNAMELen(pM) + getPROCIDLen(pM) + 3; - if((pBuf = malloc(iTAGLen * sizeof(char))) == NULL) - return; /* nothing we can do */ - snprintf((char*)pBuf, iTAGLen, "%s[%s]", getAPPNAME(pM), getPROCID(pM)); - MsgAssignTAG(pM, pBuf); + lenTAG = snprintf((char*)bufTAG, CONF_TAG_MAXSIZE, "%s[%s]", + getAPPNAME(pM, MUTEX_ALREADY_LOCKED), getPROCID(pM, MUTEX_ALREADY_LOCKED)); + bufTAG[32] = '\0'; /* just to make sure... */ + MsgSetTAG(pM, bufTAG, lenTAG); } } + if(bLockMutex == LOCK_MUTEX) + MsgUnlock(pM); } -#if 0 /* This method is currently not called, be we like to preserve it */ -static int getTAGLen(msg_t *pM) -{ - if(pM == NULL) - return 0; - else { - tryEmulateTAG(pM); - if(pM->pszTAG == NULL) - return 0; - else - return pM->iLenTAG; - } -} -#endif - - -static inline char *getTAG(msg_t *pM) +static inline void +getTAG(msg_t *pM, uchar **ppBuf, int *piLen) { - char *ret; - - if(pM == NULL) - ret = ""; - else { - MsgLock(pM); - tryEmulateTAG(pM); - if(pM->pszTAG == NULL) - ret = ""; - else - ret = (char*) pM->pszTAG; - MsgUnlock(pM); + if(pM == NULL) { + *ppBuf = UCHAR_CONSTANT(""); + *piLen = 0; + } else { + if(pM->iLenTAG == 0) + tryEmulateTAG(pM, LOCK_MUTEX); + if(pM->iLenTAG == 0) { + *ppBuf = UCHAR_CONSTANT(""); + *piLen = 0; + } else { + *ppBuf = (pM->iLenTAG < CONF_TAG_BUFSIZE) ? pM->TAG.szBuf : pM->TAG.pszTAG; + *piLen = pM->iLenTAG; + } } - return(ret); } @@ -1254,7 +1649,10 @@ int getHOSTNAMELen(msg_t *pM) return 0; else if(pM->pszHOSTNAME == NULL) - return 0; + if(pM->pRcvFrom == NULL) + return 0; + else + return prop.GetStringLen(pM->pRcvFrom); else return pM->iLenHOSTNAME; } @@ -1265,48 +1663,39 @@ char *getHOSTNAME(msg_t *pM) if(pM == NULL) return ""; else - if(pM->pszHOSTNAME == NULL) - return ""; - else + if(pM->pszHOSTNAME == NULL) { + if(pM->pRcvFrom == NULL) { + return ""; + } else { + uchar *psz; + int len; + prop.GetString(pM->pRcvFrom, &psz, &len); + return (char*) psz; + } + } else { return (char*) pM->pszHOSTNAME; -} - - -static uchar *getInputName(msg_t *pM) -{ - if(pM == NULL) - return (uchar*) ""; - else - if(pM->pszInputName == NULL) - return (uchar*) ""; - else - return pM->pszInputName; + } } uchar *getRcvFrom(msg_t *pM) { - if(pM == NULL) - return UCHAR_CONSTANT(""); - else - if(pM->pszRcvFrom == NULL) - return UCHAR_CONSTANT(""); + uchar *psz; + int len; + BEGINfunc + if(pM == NULL) { + psz = UCHAR_CONSTANT(""); + } else { + if(pM->pRcvFrom == NULL) + psz = UCHAR_CONSTANT(""); else - return pM->pszRcvFrom; + prop.GetString(pM->pRcvFrom, &psz, &len); + } + ENDfunc + return psz; } -uchar *getRcvFromIP(msg_t *pM) -{ - if(pM == NULL) - return (uchar*) ""; - else - if(pM->pszRcvFromIP == NULL) - return (uchar*) ""; - else - return pM->pszRcvFromIP; -} - /* rgerhards 2004-11-24: set STRUCTURED DATA in msg object */ rsRetVal MsgSetStructuredData(msg_t *pMsg, char* pszStrucData) @@ -1316,7 +1705,6 @@ rsRetVal MsgSetStructuredData(msg_t *pMsg, char* pszStrucData) if(pMsg->pCSStrucData == NULL) { /* we need to obtain the object first */ CHKiRet(rsCStrConstruct(&pMsg->pCSStrucData)); - rsCStrSetAllocIncrement(pMsg->pCSStrucData, 128); } /* if we reach this point, we have the object */ iRet = rsCStrSetSzStr(pMsg->pCSStrucData, (uchar*) pszStrucData); @@ -1345,101 +1733,50 @@ static inline char *getStructuredData(msg_t *pM) } - -/* get the length of the "programname" sz string - * rgerhards, 2005-10-19 +/* check if we have a ProgramName, and, if not, try to aquire/emulate it. + * rgerhards, 2009-06-26 */ -int getProgramNameLen(msg_t *pM) +static inline void prepareProgramName(msg_t *pM, bool bLockMutex) { - int iRet; + if(pM->pCSProgName == NULL) { + if(bLockMutex == LOCK_MUTEX) + MsgLock(pM); - assert(pM != NULL); - MsgLock(pM); - if((iRet = aquireProgramName(pM)) != RS_RET_OK) { - dbgprintf("error %d returned by aquireProgramName() in getProgramNameLen()\n", iRet); - MsgUnlock(pM); - return 0; /* best we can do (consistent wiht what getProgramName() returns) */ - } - MsgUnlock(pM); + /* re-query as things might have changed during locking */ + if(pM->pCSProgName == NULL) + aquireProgramName(pM); - return (pM->pCSProgName == NULL) ? 0 : rsCStrLen(pM->pCSProgName); + if(bLockMutex == LOCK_MUTEX) + MsgUnlock(pM); + } } -/* get the "programname" as sz string +/* get the length of the "programname" sz string * rgerhards, 2005-10-19 */ -char *getProgramName(msg_t *pM) /* this is the non-locking version for internal use */ +int getProgramNameLen(msg_t *pM, bool bLockMutex) { - int iRet; - char *pszRet; - assert(pM != NULL); - MsgLock(pM); - if((iRet = aquireProgramName(pM)) != RS_RET_OK) { - dbgprintf("error %d returned by aquireProgramName() in getProgramName()\n", iRet); - pszRet = ""; /* best we can do */ - } else { - pszRet = (pM->pCSProgName == NULL) ? "" : (char*) rsCStrGetSzStrNoNULL(pM->pCSProgName); - } - - MsgUnlock(pM); - return pszRet; + prepareProgramName(pM, bLockMutex); + return (pM->pCSProgName == NULL) ? 0 : rsCStrLen(pM->pCSProgName); } -/* The code below was an approach without PTHREAD_MUTEX_RECURSIVE - * However, it turned out to be quite complex. So far, we use recursive - * locking, which is OK from a performance point of view, especially as - * we do not anticipate that multithreading msg objects is used often. - * However, we may re-think about using non-recursive locking and I leave this - * code in here to conserve the idea. -- rgerhards, 2008-01-05 - */ -#if 0 -static char *getProgramNameNoLock(msg_t *pM) /* this is the non-locking version for internal use */ -{ - int iRet; - assert(pM != NULL); - if((iRet = aquireProgramName(pM)) != RS_RET_OK) { - dbgprintf("error %d returned by aquireProgramName() in getProgramName()\n", iRet); - return ""; /* best we can do */ - } - return (pM->pCSProgName == NULL) ? "" : (char*) rsCStrGetSzStrNoNULL(pM->pCSProgName); -} -char *getProgramName(msg_t *pM) /* this is the external callable version */ +/* get the "programname" as sz string + * rgerhards, 2005-10-19 + */ +char *getProgramName(msg_t *pM, bool bLockMutex) { - char *pszRet; - - MsgLock(pM); - pszRet = getProgramNameNoLock(pM); - MsgUnlock(pM); - return pszRet; + prepareProgramName(pM, bLockMutex); + return (pM->pCSProgName == NULL) ? "" : (char*) rsCStrGetSzStrNoNULL(pM->pCSProgName); } -/* an alternative approach has been: */ -/* The macro below is used to generate external function definitions - * for such functions that may also be called internally (and thus have - * both a locking and non-locking implementation. Over time, we could - * reconsider how we handle that. -- rgerhards, 2008-01-05 - */ -#define EXT_LOCKED_FUNC(fName, ret) \ -ret fName(msg_t *pM) \ -{ \ - ret valRet; \ - MsgLock(pM); \ - valRet = fName##NoLock(pM); \ - MsgUnlock(pM); \ - return(valRet); \ -} -EXT_LOCKED_FUNC(getProgramName, char*) -/* in this approach, the external function is provided by the macro and - * needs not to be writen. - */ -#endif /* #if 0 -- saved code */ /* This function tries to emulate APPNAME if it is not present. Its * main use is when we have received a log record via legacy syslog and * now would like to send out the same one via syslog-protocol. + * MUST be called with the Msg Lock locked! */ static void tryEmulateAPPNAME(msg_t *pM) { @@ -1449,79 +1786,136 @@ static void tryEmulateAPPNAME(msg_t *pM) if(getProtocolVersion(pM) == 0) { /* only then it makes sense to emulate */ - MsgSetAPPNAME(pM, getProgramName(pM)); + MsgSetAPPNAME(pM, getProgramName(pM, MUTEX_ALREADY_LOCKED)); } } + +/* check if we have a APPNAME, and, if not, try to aquire/emulate it. + * This must be called WITHOUT the message lock being held. + * rgerhards, 2009-06-26 + */ +static inline void prepareAPPNAME(msg_t *pM, bool bLockMutex) +{ + if(pM->pCSAPPNAME == NULL) { + if(bLockMutex == LOCK_MUTEX) + MsgLock(pM); + + /* re-query as things might have changed during locking */ + if(pM->pCSAPPNAME == NULL) + tryEmulateAPPNAME(pM); + + if(bLockMutex == LOCK_MUTEX) + MsgUnlock(pM); + } +} + /* rgerhards, 2005-11-24 */ -static int getAPPNAMELen(msg_t *pM) +char *getAPPNAME(msg_t *pM, bool bLockMutex) { assert(pM != NULL); - if(pM->pCSAPPNAME == NULL) - tryEmulateAPPNAME(pM); + prepareAPPNAME(pM, bLockMutex); + return (pM->pCSAPPNAME == NULL) ? "" : (char*) rsCStrGetSzStrNoNULL(pM->pCSAPPNAME); +} + +/* rgerhards, 2005-11-24 + */ +static int getAPPNAMELen(msg_t *pM, bool bLockMutex) +{ + assert(pM != NULL); + prepareAPPNAME(pM, bLockMutex); return (pM->pCSAPPNAME == NULL) ? 0 : rsCStrLen(pM->pCSAPPNAME); } -/* rgerhards 2008-09-10: set pszInputName in msg object +/* rgerhards 2008-09-10: set pszInputName in msg object. This calls AddRef() + * on the property, because this must be done in all current cases and there + * is no case expected where this may not be necessary. * rgerhards, 2009-06-16 */ -void MsgSetInputName(msg_t *pMsg, uchar* pszInputName, size_t lenInputName) +void MsgSetInputName(msg_t *pThis, prop_t *inputName) { - assert(pMsg != NULL); - free(pMsg->pszInputName); - pMsg->iLenInputName = lenInputName; - if((pMsg->pszInputName = malloc(pMsg->iLenInputName + 1)) != NULL) { - memcpy(pMsg->pszInputName, pszInputName, pMsg->iLenInputName + 1); - } + assert(pThis != NULL); + + prop.AddRef(inputName); + if(pThis->pInputName != NULL) + prop.Destruct(&pThis->pInputName); + pThis->pInputName = inputName; } -/* rgerhards 2004-11-16: set pszRcvFrom in msg object + +/* rgerhards 2008-09-10: set RcvFrom name in msg object. This calls AddRef() + * on the property, because this must be done in all current cases and there + * is no case expected where this may not be necessary. + * rgerhards, 2009-06-30 */ -void MsgSetRcvFrom(msg_t *pMsg, uchar* pszRcvFrom) +void MsgSetRcvFrom(msg_t *pThis, prop_t *new) { - assert(pMsg != NULL); - free(pMsg->pszRcvFrom); + assert(pThis != NULL); - pMsg->iLenRcvFrom = ustrlen(pszRcvFrom); - if((pMsg->pszRcvFrom = malloc(pMsg->iLenRcvFrom + 1)) != NULL) { - memcpy(pMsg->pszRcvFrom, pszRcvFrom, pMsg->iLenRcvFrom + 1); - } + prop.AddRef(new); + if(pThis->pRcvFrom != NULL) + prop.Destruct(&pThis->pRcvFrom); + pThis->pRcvFrom = new; } -/* rgerhards 2005-05-16: set pszRcvFromIP in msg object */ -rsRetVal -MsgSetRcvFromIP(msg_t *pMsg, uchar* pszRcvFromIP) +/* This is used to set the property via a string. This function should not be + * called if there is a reliable way for a caller to make sure that the + * same name can be used across multiple messages. However, if it can not + * ensure that, calling this function is the second best thing, because it + * will re-use the previously created property if it contained the same + * name (but it works only for the immediate previous). + * rgerhards, 2009-06-31 + */ +void MsgSetRcvFromStr(msg_t *pThis, uchar *psz, int len, prop_t **ppProp) { - DEFiRet; - assert(pMsg != NULL); - if(pMsg->pszRcvFromIP != NULL) { - free(pMsg->pszRcvFromIP); - pMsg->iLenRcvFromIP = 0; - } + assert(pThis != NULL); + assert(ppProp != NULL); - CHKmalloc(pMsg->pszRcvFromIP = (uchar*)strdup((char*)pszRcvFromIP)); - pMsg->iLenRcvFromIP = strlen((char*)pszRcvFromIP); -finalize_it: - RETiRet; + prop.CreateOrReuseStringProp(ppProp, psz, len); + MsgSetRcvFrom(pThis, *ppProp); } -/* Set the HOSTNAME to a caller-provided string. This is thought - * to be a heap buffer that the caller will no longer use. This - * function is a performance optimization over MsgSetHOSTNAME(). - * rgerhards 2004-11-19 +/* set RcvFromIP name in msg object. This calls AddRef() + * on the property, because this must be done in all current cases and there + * is no case expected where this may not be necessary. + * rgerhards, 2009-06-30 */ -void MsgAssignHOSTNAME(msg_t *pMsg, char *pBuf) +rsRetVal MsgSetRcvFromIP(msg_t *pThis, prop_t *new) { - assert(pMsg != NULL); - assert(pBuf != NULL); - if(pMsg->pszHOSTNAME != NULL) - free(pMsg->pszHOSTNAME); - pMsg->iLenHOSTNAME = strlen(pBuf); - pMsg->pszHOSTNAME = (uchar*) pBuf; + assert(pThis != NULL); + + BEGINfunc + prop.AddRef(new); + if(pThis->pRcvFromIP != NULL) + prop.Destruct(&pThis->pRcvFromIP); + pThis->pRcvFromIP = new; + ENDfunc + return RS_RET_OK; +} + + +/* This is used to set the property via a string. This function should not be + * called if there is a reliable way for a caller to make sure that the + * same name can be used across multiple messages. However, if it can not + * ensure that, calling this function is the second best thing, because it + * will re-use the previously created property if it contained the same + * name (but it works only for the immediate previous). + * rgerhards, 2009-06-31 + */ +rsRetVal MsgSetRcvFromIPStr(msg_t *pThis, uchar *psz, int len, prop_t **ppProp) +{ + DEFiRet; + assert(pThis != NULL); + + CHKiRet(prop.CreateOrReuseStringProp(ppProp, psz, len)); + MsgSetRcvFromIP(pThis, *ppProp); + +finalize_it: + RETiRet; } @@ -1535,49 +1929,120 @@ void MsgAssignHOSTNAME(msg_t *pMsg, char *pBuf) * we need it. The rest of the code already knows how to handle an * unset HOSTNAME. */ -void MsgSetHOSTNAME(msg_t *pMsg, uchar* pszHOSTNAME) +void MsgSetHOSTNAME(msg_t *pThis, uchar* pszHOSTNAME, int lenHOSTNAME) { - assert(pMsg != NULL); - free(pMsg->pszHOSTNAME); + assert(pThis != NULL); - pMsg->iLenHOSTNAME = ustrlen(pszHOSTNAME); - if((pMsg->pszHOSTNAME = malloc(pMsg->iLenHOSTNAME + 1)) != NULL) - memcpy(pMsg->pszHOSTNAME, pszHOSTNAME, pMsg->iLenHOSTNAME + 1); - else - DBGPRINTF("Could not allocate memory in MsgSetHOSTNAME()\n"); + freeHOSTNAME(pThis); + + pThis->iLenHOSTNAME = lenHOSTNAME; + if(pThis->iLenHOSTNAME < CONF_HOSTNAME_BUFSIZE) { + /* small enough: use fixed buffer (faster!) */ + pThis->pszHOSTNAME = pThis->szHOSTNAME; + } else if((pThis->pszHOSTNAME = (uchar*) malloc(pThis->iLenHOSTNAME + 1)) == NULL) { + /* truncate message, better than completely loosing it... */ + pThis->pszHOSTNAME = pThis->szHOSTNAME; + pThis->iLenHOSTNAME = CONF_HOSTNAME_BUFSIZE - 1; + } + + memcpy(pThis->pszHOSTNAME, pszHOSTNAME, pThis->iLenHOSTNAME); + pThis->pszHOSTNAME[pThis->iLenHOSTNAME] = '\0'; /* this also works with truncation! */ } -/* rgerhards 2004-11-09: set MSG in msg object +/* set the offset of the MSG part into the raw msg buffer + * Note that the offset may be higher than the length of the raw message + * (exactly by one). This can happen if we have a message that does not + * contain any MSG part. */ -void MsgSetMSG(msg_t *pMsg, char* pszMSG) +void MsgSetMSGoffs(msg_t *pMsg, short offs) { - assert(pMsg != NULL); + ISOBJ_TYPE_assert(pMsg, msg); + pMsg->offMSG = offs; + if(offs > pMsg->iLenRawMsg) { + assert(offs - 1 == pMsg->iLenRawMsg); + pMsg->iLenMSG = 0; + } else { + pMsg->iLenMSG = pMsg->iLenRawMsg - offs; + } +} + + +/* replace the MSG part of a message. The update actually takes place inside + * rawmsg. + * There are two cases: either the new message will be larger than the new msg + * or it will be less than or equal. If it is less than or equal, we can utilize + * the previous message buffer. If it is larger, we can utilize the msg_t-included + * message buffer if it fits in there. If this is not the case, we need to alloc + * a new, larger, chunk and copy over the data to it. Note that this function is + * (hopefully) relatively seldom being called, so some performance impact is + * uncritical. In any case, pszMSG is copied, so if it was dynamically allocated, + * the caller is responsible for freeing it. + * rgerhards, 2009-06-23 + */ +rsRetVal MsgReplaceMSG(msg_t *pThis, uchar* pszMSG, int lenMSG) +{ + int lenNew; + uchar *bufNew; + DEFiRet; + ISOBJ_TYPE_assert(pThis, msg); assert(pszMSG != NULL); - if(pMsg->pszMSG != NULL) - free(pMsg->pszMSG); + lenNew = pThis->iLenRawMsg + lenMSG - pThis->iLenMSG; + if(lenMSG > pThis->iLenMSG && lenNew >= CONF_RAWMSG_BUFSIZE) { + /* we have lost our "bet" and need to alloc a new buffer ;) */ + CHKmalloc(bufNew = malloc(lenNew + 1)); + memcpy(bufNew, pThis->pszRawMsg, pThis->offMSG); + if(pThis->pszRawMsg != pThis->szRawMsg) + free(pThis->pszRawMsg); + pThis->pszRawMsg = bufNew; + } + + if(lenMSG > 0) + memcpy(pThis->pszRawMsg + pThis->offMSG, pszMSG, lenMSG); + pThis->pszRawMsg[lenNew] = '\0'; /* this also works with truncation! */ + pThis->iLenRawMsg = lenNew; + pThis->iLenMSG = lenMSG; - pMsg->iLenMSG = strlen(pszMSG); - if((pMsg->pszMSG = (uchar*) malloc(pMsg->iLenMSG + 1)) != NULL) - memcpy(pMsg->pszMSG, pszMSG, pMsg->iLenMSG + 1); - else - dbgprintf("MsgSetMSG could not allocate memory for pszMSG buffer."); +finalize_it: + RETiRet; } -/* rgerhards 2004-11-11: set RawMsg in msg object + +/* set raw message in message object. Size of message is provided. + * The function makes sure that the stored rawmsg is properly + * terminated by '\0'. + * rgerhards, 2009-06-16 */ -void MsgSetRawMsg(msg_t *pMsg, char* pszRawMsg) +void MsgSetRawMsg(msg_t *pThis, char* pszRawMsg, size_t lenMsg) { - assert(pMsg != NULL); - if(pMsg->pszRawMsg != NULL) - free(pMsg->pszRawMsg); + assert(pThis != NULL); + if(pThis->pszRawMsg != pThis->szRawMsg) + free(pThis->pszRawMsg); - pMsg->iLenRawMsg = strlen(pszRawMsg); - if((pMsg->pszRawMsg = (uchar*) malloc(pMsg->iLenRawMsg + 1)) != NULL) - memcpy(pMsg->pszRawMsg, pszRawMsg, pMsg->iLenRawMsg + 1); - else - dbgprintf("Could not allocate memory for pszRawMsg buffer."); + pThis->iLenRawMsg = lenMsg; + if(pThis->iLenRawMsg < CONF_RAWMSG_BUFSIZE) { + /* small enough: use fixed buffer (faster!) */ + pThis->pszRawMsg = pThis->szRawMsg; + } else if((pThis->pszRawMsg = (uchar*) malloc(pThis->iLenRawMsg + 1)) == NULL) { + /* truncate message, better than completely loosing it... */ + pThis->pszRawMsg = pThis->szRawMsg; + pThis->iLenRawMsg = CONF_RAWMSG_BUFSIZE - 1; + } + + memcpy(pThis->pszRawMsg, pszRawMsg, pThis->iLenRawMsg); + pThis->pszRawMsg[pThis->iLenRawMsg] = '\0'; /* this also works with truncation! */ +} + + +/* set raw message in message object. Size of message is not provided. This + * function should only be used when it is unavoidable (and over time we should + * try to remove it altogether). + * rgerhards, 2009-06-16 + */ +void MsgSetRawMsgWOSize(msg_t *pMsg, char* pszRawMsg) +{ + MsgSetRawMsg(pMsg, pszRawMsg, strlen(pszRawMsg)); } @@ -1591,15 +2056,11 @@ void MsgSetRawMsg(msg_t *pMsg, char* pszRawMsg) */ char *textpri(char *pRes, size_t pResLen, int pri) { - syslogCODE *c_pri, *c_fac; - assert(pRes != NULL); assert(pResLen > 0); - for (c_fac = rs_facilitynames; c_fac->c_name && !(c_fac->c_val == LOG_FAC(pri)<<3); c_fac++); - for (c_pri = rs_prioritynames; c_pri->c_name && !(c_pri->c_val == LOG_PRI(pri)); c_pri++); - - snprintf (pRes, pResLen, "%s.%s<%d>", c_fac->c_name, c_pri->c_name, pri); + snprintf(pRes, pResLen, "%s.%s<%d>", syslog_fac_names[LOG_FAC(pri)], + syslog_severity_names[LOG_PRI(pri)], pri); return pRes; } @@ -1693,141 +2154,172 @@ static uchar *getNOW(eNOWType eNow) * be used in selector line processing. * rgerhards 2005-09-15 */ -char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, - cstr_t *pCSPropName, unsigned short *pbMustBeFreed) -{ - uchar *pName; - char *pRes; /* result pointer */ - char *pBufStart; - char *pBuf; +/* a quick helper to save some writing: */ +#define RET_OUT_OF_MEMORY { *pbMustBeFreed = 0;\ + *pPropLen = sizeof("**OUT OF MEMORY**") - 1; \ + return(UCHAR_CONSTANT("**OUT OF MEMORY**"));} +uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, + propid_t propID, size_t *pPropLen, + unsigned short *pbMustBeFreed) +{ + uchar *pRes; /* result pointer */ + int bufLen = -1; /* length of string or -1, if not known */ + uchar *pBufStart; + uchar *pBuf; int iLen; short iOffs; + BEGINfunc + assert(pMsg != NULL); + assert(pbMustBeFreed != NULL); + #ifdef FEATURE_REGEXP /* Variables necessary for regular expression matching */ size_t nmatch = 10; regmatch_t pmatch[10]; #endif - assert(pMsg != NULL); - assert(pbMustBeFreed != NULL); - - if(pCSPropName == NULL) { - assert(pTpe != NULL); - pName = pTpe->data.field.pPropRepl; - } else { - pName = rsCStrGetSzStrNoNULL(pCSPropName); - } *pbMustBeFreed = 0; - /* sometimes there are aliases to the original MonitoWare - * property names. These come after || in the ifs below. */ - if(!strcmp((char*) pName, "msg")) { - pRes = getMSG(pMsg); - } else if(!strcmp((char*) pName, "rawmsg")) { - pRes = getRawMsg(pMsg); - /* enable this, if someone actually uses UxTradMsg, delete after some time has - * passed and nobody complained -- rgerhards, 2009-06-16 - } else if(!strcmp((char*) pName, "uxtradmsg")) { - pRes = getUxTradMsg(pMsg); - */ - } else if(!strcmp((char*) pName, "inputname")) { - pRes = (char*) getInputName(pMsg); - } else if(!strcmp((char*) pName, "fromhost")) { - pRes = (char*) getRcvFrom(pMsg); - } else if(!strcmp((char*) pName, "fromhost-ip")) { - pRes = (char*) getRcvFromIP(pMsg); - } else if(!strcmp((char*) pName, "source") || !strcmp((char*) pName, "hostname")) { - pRes = getHOSTNAME(pMsg); - } else if(!strcmp((char*) pName, "syslogtag")) { - pRes = getTAG(pMsg); - } else if(!strcmp((char*) pName, "pri")) { - pRes = getPRI(pMsg); - } else if(!strcmp((char*) pName, "pri-text")) { - pBuf = malloc(20 * sizeof(char)); - if(pBuf == NULL) { + switch(propID) { + case PROP_MSG: + pRes = getMSG(pMsg); + bufLen = getMSGLen(pMsg); + break; + case PROP_TIMESTAMP: + pRes = (uchar*)getTimeReported(pMsg, pTpe->data.field.eDateFormat); + break; + case PROP_HOSTNAME: + pRes = (uchar*)getHOSTNAME(pMsg); + break; + case PROP_SYSLOGTAG: + getTAG(pMsg, &pRes, &bufLen); + break; + case PROP_RAWMSG: + pRes = (uchar*)getRawMsg(pMsg); + break; + /* enable this, if someone actually uses UxTradMsg, delete after some time has + * passed and nobody complained -- rgerhards, 2009-06-16 + case PROP_UXTRADMSG: + pRes = getUxTradMsg(pMsg); + break; + */ + case PROP_INPUTNAME: + getInputName(pMsg, &pRes, &bufLen); + break; + case PROP_FROMHOST: + pRes = getRcvFrom(pMsg); + break; + case PROP_FROMHOST_IP: + pRes = getRcvFromIP(pMsg); + break; + case PROP_PRI: + pRes = (uchar*)getPRI(pMsg); + break; + case PROP_PRI_TEXT: + pBuf = malloc(20 * sizeof(uchar)); + if(pBuf == NULL) { + RET_OUT_OF_MEMORY; + } else { + *pbMustBeFreed = 1; + pRes = (uchar*)textpri((char*)pBuf, 20, getPRIi(pMsg)); + } + break; + case PROP_IUT: + pRes = UCHAR_CONSTANT("1"); /* always 1 for syslog messages (a MonitorWare thing;)) */ + break; + case PROP_SYSLOGFACILITY: + pRes = (uchar*)getFacility(pMsg); + break; + case PROP_SYSLOGFACILITY_TEXT: + pRes = (uchar*)getFacilityStr(pMsg); + break; + case PROP_SYSLOGSEVERITY: + pRes = (uchar*)getSeverity(pMsg); + break; + case PROP_SYSLOGSEVERITY_TEXT: + pRes = (uchar*)getSeverityStr(pMsg); + break; + case PROP_TIMEGENERATED: + pRes = (uchar*)getTimeGenerated(pMsg, pTpe->data.field.eDateFormat); + break; + case PROP_PROGRAMNAME: + pRes = (uchar*)getProgramName(pMsg, LOCK_MUTEX); + break; + case PROP_PROTOCOL_VERSION: + pRes = (uchar*)getProtocolVersionString(pMsg); + break; + case PROP_STRUCTURED_DATA: + pRes = (uchar*)getStructuredData(pMsg); + break; + case PROP_APP_NAME: + pRes = (uchar*)getAPPNAME(pMsg, LOCK_MUTEX); + break; + case PROP_PROCID: + pRes = (uchar*)getPROCID(pMsg, LOCK_MUTEX); + break; + case PROP_MSGID: + pRes = (uchar*)getMSGID(pMsg); + break; + case PROP_SYS_NOW: + if((pRes = getNOW(NOW_NOW)) == NULL) { + RET_OUT_OF_MEMORY; + } else + *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + break; + case PROP_SYS_YEAR: + if((pRes = getNOW(NOW_YEAR)) == NULL) { + RET_OUT_OF_MEMORY; + } else + *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + break; + case PROP_SYS_MONTH: + if((pRes = getNOW(NOW_MONTH)) == NULL) { + RET_OUT_OF_MEMORY; + } else + *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + break; + case PROP_SYS_DAY: + if((pRes = getNOW(NOW_DAY)) == NULL) { + RET_OUT_OF_MEMORY; + } else + *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + break; + case PROP_SYS_HOUR: + if((pRes = getNOW(NOW_HOUR)) == NULL) { + RET_OUT_OF_MEMORY; + } else + *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + break; + case PROP_SYS_HHOUR: + if((pRes = getNOW(NOW_HHOUR)) == NULL) { + RET_OUT_OF_MEMORY; + } else + *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + break; + case PROP_SYS_QHOUR: + if((pRes = getNOW(NOW_QHOUR)) == NULL) { + RET_OUT_OF_MEMORY; + } else + *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + break; + case PROP_SYS_MINUTE: + if((pRes = getNOW(NOW_MINUTE)) == NULL) { + RET_OUT_OF_MEMORY; + } else + *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ + break; + case PROP_SYS_MYHOSTNAME: + pRes = glbl.GetLocalHostName(); + break; + default: + /* there is no point in continuing, we may even otherwise render the + * error message unreadable. rgerhards, 2007-07-10 + */ + dbgprintf("invalid property id: '%d'\n", propID); *pbMustBeFreed = 0; - return "**OUT OF MEMORY**"; - } else { - *pbMustBeFreed = 1; - pRes = textpri(pBuf, 20, getPRIi(pMsg)); - } - } else if(!strcmp((char*) pName, "iut")) { - pRes = "1"; /* always 1 for syslog messages (a MonitorWare thing;)) */ - } else if(!strcmp((char*) pName, "syslogfacility")) { - pRes = getFacility(pMsg); - } else if(!strcmp((char*) pName, "syslogfacility-text")) { - pRes = getFacilityStr(pMsg); - } else if(!strcmp((char*) pName, "syslogseverity") || !strcmp((char*) pName, "syslogpriority")) { - pRes = getSeverity(pMsg); - } else if(!strcmp((char*) pName, "syslogseverity-text") || !strcmp((char*) pName, "syslogpriority-text")) { - pRes = getSeverityStr(pMsg); - } else if(!strcmp((char*) pName, "timegenerated")) { - pRes = getTimeGenerated(pMsg, pTpe->data.field.eDateFormat); - } else if(!strcmp((char*) pName, "timereported") - || !strcmp((char*) pName, "timestamp")) { - pRes = getTimeReported(pMsg, pTpe->data.field.eDateFormat); - } else if(!strcmp((char*) pName, "programname")) { - pRes = getProgramName(pMsg); - } else if(!strcmp((char*) pName, "protocol-version")) { - pRes = getProtocolVersionString(pMsg); - } else if(!strcmp((char*) pName, "structured-data")) { - pRes = getStructuredData(pMsg); - } else if(!strcmp((char*) pName, "app-name")) { - pRes = getAPPNAME(pMsg); - } else if(!strcmp((char*) pName, "procid")) { - pRes = getPROCID(pMsg); - } else if(!strcmp((char*) pName, "msgid")) { - pRes = getMSGID(pMsg); - /* here start system properties (those, that do not relate to the message itself */ - } else if(!strcmp((char*) pName, "$now")) { - if((pRes = (char*) getNOW(NOW_NOW)) == NULL) { - return "***OUT OF MEMORY***"; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ - } else if(!strcmp((char*) pName, "$year")) { - if((pRes = (char*) getNOW(NOW_YEAR)) == NULL) { - return "***OUT OF MEMORY***"; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ - } else if(!strcmp((char*) pName, "$month")) { - if((pRes = (char*) getNOW(NOW_MONTH)) == NULL) { - return "***OUT OF MEMORY***"; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ - } else if(!strcmp((char*) pName, "$day")) { - if((pRes = (char*) getNOW(NOW_DAY)) == NULL) { - return "***OUT OF MEMORY***"; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ - } else if(!strcmp((char*) pName, "$hour")) { - if((pRes = (char*) getNOW(NOW_HOUR)) == NULL) { - return "***OUT OF MEMORY***"; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ - } else if(!strcmp((char*) pName, "$hhour")) { - if((pRes = (char*) getNOW(NOW_HHOUR)) == NULL) { - return "***OUT OF MEMORY***"; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ - } else if(!strcmp((char*) pName, "$qhour")) { - if((pRes = (char*) getNOW(NOW_QHOUR)) == NULL) { - return "***OUT OF MEMORY***"; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ - } else if(!strcmp((char*) pName, "$minute")) { - if((pRes = (char*) getNOW(NOW_MINUTE)) == NULL) { - return "***OUT OF MEMORY***"; - } else - *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ - } else if(!strcmp((char*) pName, "$myhostname")) { - pRes = (char*) glbl.GetLocalHostName(); - } else { - /* there is no point in continuing, we may even otherwise render the - * error message unreadable. rgerhards, 2007-07-10 - */ - dbgprintf("invalid property name: '%s'\n", pName); - return "**INVALID PROPERTY NAME**"; + *pPropLen = sizeof("**INVALID PROPERTY NAME**") - 1; + return UCHAR_CONSTANT("**INVALID PROPERTY NAME**"); } /* If we did not receive a template pointer, we are already done... */ @@ -1846,8 +2338,8 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, */ if(pTpe->data.field.has_fields == 1) { size_t iCurrFld; - char *pFld; - char *pFldEnd; + uchar *pFld; + uchar *pFldEnd; /* first, skip to the field in question. The field separator * is always one character and is stored in the template entry. */ @@ -1884,11 +2376,11 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(pBuf == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return "**OUT OF MEMORY**"; + RET_OUT_OF_MEMORY; } /* now copy */ memcpy(pBuf, pFld, iLen); + bufLen = iLen; pBuf[iLen] = '\0'; /* terminate it */ if(*pbMustBeFreed == 1) free(pRes); @@ -1901,12 +2393,13 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(*pbMustBeFreed == 1) free(pRes); *pbMustBeFreed = 0; - return "**FIELD NOT FOUND**"; + *pPropLen = sizeof("**FIELD NOT FOUND**") - 1; + return UCHAR_CONSTANT("**FIELD NOT FOUND**"); } } else if(pTpe->data.field.iFromPos != 0 || pTpe->data.field.iToPos != 0) { /* we need to obtain a private copy */ int iFrom, iTo; - char *pSb; + uchar *pSb; iFrom = pTpe->data.field.iFromPos; iTo = pTpe->data.field.iToPos; /* need to zero-base to and from (they are 1-based!) */ @@ -1914,42 +2407,60 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, --iFrom; if(iTo > 0) --iTo; - iLen = iTo - iFrom + 1; /* the +1 is for an actual char, NOT \0! */ - pBufStart = pBuf = malloc((iLen + 1) * sizeof(char)); - if(pBuf == NULL) { - if(*pbMustBeFreed == 1) - free(pRes); - *pbMustBeFreed = 0; - return "**OUT OF MEMORY**"; - } - pSb = pRes; - if(iFrom) { - /* skip to the start of the substring (can't do pointer arithmetic - * because the whole string might be smaller!!) - */ - while(*pSb && iFrom) { - --iFrom; + if(bufLen == -1) + bufLen = ustrlen(pRes); + if(iFrom == 0 && iTo >= bufLen) { + /* in this case, the requested string is a superset of what we already have, + * so there is no need to do any processing. This is a frequent case for size-limited + * fields like TAG in the default forwarding template (so it is a useful optimization + * to check for this condition ;)). -- rgerhards, 2009-07-09 + */ + ; /*DO NOTHING*/ + } else { + iLen = iTo - iFrom + 1; /* the +1 is for an actual char, NOT \0! */ + pBufStart = pBuf = malloc((iLen + 1) * sizeof(char)); + if(pBuf == NULL) { + if(*pbMustBeFreed == 1) + free(pRes); + RET_OUT_OF_MEMORY; + } + pSb = pRes; + if(iFrom) { + /* skip to the start of the substring (can't do pointer arithmetic + * because the whole string might be smaller!!) + */ + while(*pSb && iFrom) { + --iFrom; + ++pSb; + } + } + /* OK, we are at the begin - now let's copy... */ + bufLen = iLen; + while(*pSb && iLen) { + *pBuf++ = *pSb; ++pSb; + --iLen; } + *pBuf = '\0'; + bufLen -= iLen; /* subtract remaining length if the string was smaller! */ + if(*pbMustBeFreed == 1) + free(pRes); + pRes = pBufStart; + *pbMustBeFreed = 1; } - /* OK, we are at the begin - now let's copy... */ - while(*pSb && iLen) { - *pBuf++ = *pSb; - ++pSb; - --iLen; - } - *pBuf = '\0'; - if(*pbMustBeFreed == 1) - free(pRes); - pRes = pBufStart; - *pbMustBeFreed = 1; #ifdef FEATURE_REGEXP } else { /* Check for regular expressions */ if (pTpe->data.field.has_regex != 0) { - if (pTpe->data.field.has_regex == 2) + if (pTpe->data.field.has_regex == 2) { /* Could not compile regex before! */ - return "**NO MATCH** **BAD REGULAR EXPRESSION**"; + if (*pbMustBeFreed == 1) { + free(pRes); + *pbMustBeFreed = 0; + } + *pPropLen = sizeof("**NO MATCH** **BAD REGULAR EXPRESSION**") - 1; + return UCHAR_CONSTANT("**NO MATCH** **BAD REGULAR EXPRESSION**"); + } dbgprintf("string to match for regex is: %s\n", pRes); @@ -1962,7 +2473,7 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, */ while(!bFound) { int iREstat; - iREstat = regexp.regexec(&pTpe->data.field.re, pRes + iOffs, nmatch, pmatch, 0); + iREstat = regexp.regexec(&pTpe->data.field.re, (char*)(pRes + iOffs), nmatch, pmatch, 0); dbgprintf("regexec return is %d\n", iREstat); if(iREstat == 0) { if(pmatch[0].rm_so == -1) { @@ -1973,7 +2484,7 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, bFound = 1; } else { dbgprintf("regex found at offset %d, new offset %d, tries %d\n", - iOffs, iOffs + pmatch[0].rm_eo, iTry); + iOffs, (int) (iOffs + pmatch[0].rm_eo), iTry); iOffs += pmatch[0].rm_eo; ++iTry; } @@ -1989,12 +2500,16 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, free(pRes); *pbMustBeFreed = 0; } - if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_DFLTSTR) - return "**NO MATCH**"; - else if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_ZERO) - return "0"; - else - return ""; + if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_DFLTSTR) { + bufLen = sizeof("**NO MATCH**") - 1; + pRes = UCHAR_CONSTANT("**NO MATCH**"); + } else if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_ZERO) { + bufLen = 1; + pRes = UCHAR_CONSTANT("0"); + } else { + bufLen = 0; + pRes = UCHAR_CONSTANT(""); + } } } else { /* Match- but did it match the one we wanted? */ @@ -2005,29 +2520,35 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, free(pRes); *pbMustBeFreed = 0; } - if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_DFLTSTR) - return "**NO MATCH**"; - else - return ""; + if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_DFLTSTR) { + bufLen = sizeof("**NO MATCH**") - 1; + pRes = UCHAR_CONSTANT("**NO MATCH**"); + } else if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_ZERO) { + bufLen = 1; + pRes = UCHAR_CONSTANT("0"); + } else { + bufLen = 0; + pRes = UCHAR_CONSTANT(""); + } } } /* OK, we have a usable match - we now need to malloc pB */ int iLenBuf; - char *pB; + uchar *pB; iLenBuf = pmatch[pTpe->data.field.iSubMatchToUse].rm_eo - pmatch[pTpe->data.field.iSubMatchToUse].rm_so; - pB = (char *) malloc((iLenBuf + 1) * sizeof(char)); + pB = malloc((iLenBuf + 1) * sizeof(uchar)); if (pB == NULL) { if (*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return "**OUT OF MEMORY ALLOCATING pBuf**"; + RET_OUT_OF_MEMORY; } /* Lets copy the matched substring to the buffer */ memcpy(pB, pRes + iOffs + pmatch[pTpe->data.field.iSubMatchToUse].rm_so, iLenBuf); + bufLen = iLenBuf; pB[iLenBuf] = '\0';/* terminate string, did not happen before */ if (*pbMustBeFreed == 1) @@ -2045,7 +2566,8 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, free(pRes); *pbMustBeFreed = 0; } - return "***REGEXP NOT AVAILABLE***"; + *pPropLen = sizeof("***REGEXP NOT AVAILABLE***") - 1; + return UCHAR_CONSTANT("***REGEXP NOT AVAILABLE***"); } } #endif /* #ifdef FEATURE_REGEXP */ @@ -2053,28 +2575,13 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, /* now check if we need to do our "SP if first char is non-space" hack logic */ if(*pRes && pTpe->data.field.options.bSPIffNo1stSP) { - char *pB; - uchar cFirst = *pRes; - /* here, we always destruct the buffer and return a new one */ - pB = (char *) malloc(2 * sizeof(char)); - if(pB == NULL) { - if(*pbMustBeFreed == 1) - free(pRes); - *pbMustBeFreed = 0; - return "**OUT OF MEMORY**"; - } - pRes = pB; - *pbMustBeFreed = 1; - - if(cFirst == ' ') { - /* if we have a SP, we must return an empty string */ - *pRes = '\0'; /* empty */ - } else { - /* if it is no SP, we need to return one */ - *pRes = ' '; - *(pRes+1) = '\0'; - } + uchar cFirst = *pRes; /* save first char */ + if(*pbMustBeFreed == 1) + free(pRes); + pRes = (cFirst == ' ') ? UCHAR_CONSTANT("") : UCHAR_CONSTANT(" "); + bufLen = (cFirst == ' ') ? 0 : 1; + *pbMustBeFreed = 0; } if(*pRes) { @@ -2083,21 +2590,21 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, */ if(pTpe->data.field.eCaseConv != tplCaseConvNo) { /* we need to obtain a private copy */ - int iBufLen = strlen(pRes); - char *pBStart; - char *pB; - char *pSrc; - pBStart = pB = malloc((iBufLen + 1) * sizeof(char)); + if(bufLen == -1) + bufLen = ustrlen(pRes); + uchar *pBStart; + uchar *pB; + uchar *pSrc; + pBStart = pB = malloc((bufLen + 1) * sizeof(char)); if(pB == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return "**OUT OF MEMORY**"; + RET_OUT_OF_MEMORY; } pSrc = pRes; while(*pSrc) { *pB++ = (pTpe->data.field.eCaseConv == tplCaseConvUpper) ? - (char)toupper((int)*pSrc) : (char)tolower((int)*pSrc); + (uchar)toupper((int)*pSrc) : (uchar)tolower((int)*pSrc); /* currently only these two exist */ ++pSrc; } @@ -2121,10 +2628,10 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, */ if(pTpe->data.field.options.bDropCC) { int iLenBuf = 0; - char *pSrc = pRes; - char *pDstStart; - char *pDst; - char bDropped = 0; + uchar *pSrc = pRes; + uchar *pDstStart; + uchar *pDst; + uchar bDropped = 0; while(*pSrc) { if(!iscntrl((int) *pSrc++)) @@ -2138,8 +2645,7 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(pDst == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return "**OUT OF MEMORY**"; + RET_OUT_OF_MEMORY; } for(pSrc = pRes; *pSrc; pSrc++) { if(!iscntrl((int) *pSrc)) @@ -2149,12 +2655,13 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(*pbMustBeFreed == 1) free(pRes); pRes = pDstStart; + bufLen = iLenBuf; *pbMustBeFreed = 1; } } else if(pTpe->data.field.options.bSpaceCC) { - char *pSrc; - char *pDstStart; - char *pDst; + uchar *pSrc; + uchar *pDstStart; + uchar *pDst; if(*pbMustBeFreed == 1) { /* in this case, we already work on dynamic @@ -2167,12 +2674,13 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, *pDst = ' '; } } else { - pDst = pDstStart = malloc(strlen(pRes) + 1); + if(bufLen == -1) + bufLen = ustrlen(pRes); + pDst = pDstStart = malloc(bufLen + 1); if(pDst == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return "**OUT OF MEMORY**"; + RET_OUT_OF_MEMORY; } for(pSrc = pRes; *pSrc; pSrc++) { if(iscntrl((int) *pSrc)) @@ -2192,7 +2700,7 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, */ int iNumCC = 0; int iLenBuf = 0; - char *pB; + uchar *pB; for(pB = pRes ; *pB ; ++pB) { ++iLenBuf; @@ -2202,21 +2710,20 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(iNumCC > 0) { /* if 0, there is nothing to escape, so we are done */ /* OK, let's do the escaping... */ - char *pBStart; - char szCCEsc[8]; /* buffer for escape sequence */ + uchar *pBStart; + uchar szCCEsc[8]; /* buffer for escape sequence */ int i; iLenBuf += iNumCC * 4; - pBStart = pB = malloc((iLenBuf + 1) * sizeof(char)); + pBStart = pB = malloc((iLenBuf + 1) * sizeof(uchar)); if(pB == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return "**OUT OF MEMORY**"; + RET_OUT_OF_MEMORY; } while(*pRes) { if(iscntrl((int) *pRes)) { - snprintf(szCCEsc, sizeof(szCCEsc), "#%3.3d", *pRes); + snprintf((char*)szCCEsc, sizeof(szCCEsc), "#%3.3d", *pRes); for(i = 0 ; i < 4 ; ++i) *pB++ = szCCEsc[i]; } else { @@ -2228,6 +2735,7 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(*pbMustBeFreed == 1) free(pRes); pRes = pBStart; + bufLen = -1; *pbMustBeFreed = 1; } } @@ -2239,10 +2747,10 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(pTpe->data.field.options.bSecPathDrop || pTpe->data.field.options.bSecPathReplace) { if(pTpe->data.field.options.bSecPathDrop) { int iLenBuf = 0; - char *pSrc = pRes; - char *pDstStart; - char *pDst; - char bDropped = 0; + uchar *pSrc = pRes; + uchar *pDstStart; + uchar *pDst; + uchar bDropped = 0; while(*pSrc) { if(*pSrc++ != '/') @@ -2256,8 +2764,7 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(pDst == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return "**OUT OF MEMORY**"; + RET_OUT_OF_MEMORY; } for(pSrc = pRes; *pSrc; pSrc++) { if(*pSrc != '/') @@ -2267,12 +2774,13 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(*pbMustBeFreed == 1) free(pRes); pRes = pDstStart; + bufLen = -1; /* TODO: can we do better? */ *pbMustBeFreed = 1; } } else { - char *pSrc; - char *pDstStart; - char *pDst; + uchar *pSrc; + uchar *pDstStart; + uchar *pDst; if(*pbMustBeFreed == 1) { /* here, again, we can modify the string as we already obtained @@ -2285,12 +2793,13 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, *pDst++ = '_'; } } else { - pDst = pDstStart = malloc(strlen(pRes) + 1); + if(bufLen == -1) + bufLen = ustrlen(pRes); + pDst = pDstStart = malloc(bufLen + 1); if(pDst == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return "**OUT OF MEMORY**"; + RET_OUT_OF_MEMORY; } for(pSrc = pRes; *pSrc; pSrc++) { if(*pSrc == '/') @@ -2310,44 +2819,48 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, /* check for "." and ".." (note the parenthesis in the if condition!) */ if((*pRes == '.') && (*(pRes + 1) == '\0' || (*(pRes + 1) == '.' && *(pRes + 2) == '\0'))) { - char *pTmp = pRes; + uchar *pTmp = pRes; if(*(pRes + 1) == '\0') - pRes = "_"; + pRes = UCHAR_CONSTANT("_"); else - pRes = "_.";; + pRes = UCHAR_CONSTANT("_.");; if(*pbMustBeFreed == 1) free(pTmp); *pbMustBeFreed = 0; } else if(*pRes == '\0') { if(*pbMustBeFreed == 1) free(pRes); - pRes = "_"; + pRes = UCHAR_CONSTANT("_"); + bufLen = 1; *pbMustBeFreed = 0; } } /* Now drop last LF if present (pls note that this must not be done - * if bEscapeCC was set! + * if bEscapeCC was set)! */ if(pTpe->data.field.options.bDropLastLF && !pTpe->data.field.options.bEscapeCC) { - int iLn = strlen(pRes); - char *pB; + int iLn; + uchar *pB; + if(bufLen == -1) + bufLen = ustrlen(pRes); + iLn = bufLen; if(iLn > 0 && *(pRes + iLn - 1) == '\n') { /* we have a LF! */ /* check if we need to obtain a private copy */ if(*pbMustBeFreed == 0) { /* ok, original copy, need a private one */ - pB = malloc((iLn + 1) * sizeof(char)); + pB = malloc((iLn + 1) * sizeof(uchar)); if(pB == NULL) { - *pbMustBeFreed = 0; - return "**OUT OF MEMORY**"; + RET_OUT_OF_MEMORY; } memcpy(pB, pRes, iLn - 1); pRes = pB; *pbMustBeFreed = 1; } *(pRes + iLn - 1) = '\0'; /* drop LF ;) */ + --bufLen; } } @@ -2358,19 +2871,23 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, */ if(pTpe->data.field.options.bCSV) { /* we need to obtain a private copy, as we need to at least add the double quotes */ - int iBufLen = strlen(pRes); - char *pBStart; - char *pDst; - char *pSrc; + int iBufLen; + int i; + uchar *pBStart; + uchar *pDst; + uchar *pSrc; + if(bufLen == -1) + bufLen = ustrlen(pRes); + iBufLen = bufLen; /* the malloc may be optimized, we currently use the worst case... */ - pBStart = pDst = malloc((2 * iBufLen + 3) * sizeof(char)); + pBStart = pDst = malloc((2 * iBufLen + 3) * sizeof(uchar)); if(pDst == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return "**OUT OF MEMORY**"; + RET_OUT_OF_MEMORY; } pSrc = pRes; + i = 0; *pDst++ = '"'; /* starting quote */ while(*pSrc) { if(*pSrc == '"') @@ -2382,10 +2899,15 @@ char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(*pbMustBeFreed == 1) free(pRes); pRes = pBStart; + bufLen = -1; *pbMustBeFreed = 1; } - /*dbgprintf("MsgGetProp(\"%s\"): \"%s\"\n", pName, pRes); only for verbose debug logging */ + if(bufLen == -1) + bufLen = ustrlen(pRes); + *pPropLen = bufLen; + + ENDfunc return(pRes); } @@ -2400,8 +2922,10 @@ msgGetMsgVar(msg_t *pThis, cstr_t *pstrPropName, var_t **ppVar) { DEFiRet; var_t *pVar; + size_t propLen; uchar *pszProp = NULL; cstr_t *pstrProp; + propid_t propid; unsigned short bMustBeFreed = 0; ISOBJ_TYPE_assert(pThis, msg); @@ -2413,7 +2937,9 @@ msgGetMsgVar(msg_t *pThis, cstr_t *pstrPropName, var_t **ppVar) CHKiRet(var.ConstructFinalize(pVar)); /* always call MsgGetProp() without a template specifier */ - pszProp = (uchar*) MsgGetProp(pThis, NULL, pstrPropName, &bMustBeFreed); + /* TODO: optimize propNameToID() call -- rgerhards, 2009-06-26 */ + propNameToID(pstrPropName, &propid); + pszProp = (uchar*) MsgGetProp(pThis, NULL, propid, &propLen, &bMustBeFreed); /* now create a string object out of it and hand that over to the var */ CHKiRet(rsCStrConstructFromszStr(&pstrProp, pszProp)); @@ -2428,8 +2954,6 @@ finalize_it: RETiRet; } - - /* This function can be used as a generic way to set properties. * We have to handle a lot of legacy, so our return value is not always * 100% correct (called functions do not always provide one, should @@ -2439,6 +2963,9 @@ finalize_it: #define isProp(name) !rsCStrSzStrCmp(pProp->pcsName, (uchar*) name, sizeof(name) - 1) rsRetVal MsgSetProperty(msg_t *pThis, var_t *pProp) { + prop_t *myProp; + prop_t *propRcvFrom = NULL; + prop_t *propRcvFromIP = NULL; DEFiRet; ISOBJ_TYPE_assert(pThis, msg); @@ -2452,27 +2979,34 @@ rsRetVal MsgSetProperty(msg_t *pThis, var_t *pProp) pThis->iFacility = pProp->val.num; } else if(isProp("msgFlags")) { pThis->msgFlags = pProp->val.num; + } else if(isProp("offMSG")) { + MsgSetMSGoffs(pThis, pProp->val.num); } else if(isProp("pszRawMsg")) { - MsgSetRawMsg(pThis, (char*) rsCStrGetSzStrNoNULL(pProp->val.pStr)); + MsgSetRawMsg(pThis, (char*) rsCStrGetSzStrNoNULL(pProp->val.pStr), cstrLen(pProp->val.pStr)); /* enable this, if someone actually uses UxTradMsg, delete after some time has * passed and nobody complained -- rgerhards, 2009-06-16 } else if(isProp("offAfterPRI")) { pThis->offAfterPRI = pProp->val.num; */ - } else if(isProp("pszMSG")) { - MsgSetMSG(pThis, (char*) rsCStrGetSzStrNoNULL(pProp->val.pStr)); } else if(isProp("pszUxTradMsg")) { /*IGNORE*/; /* this *was* a property, but does no longer exist */ } else if(isProp("pszTAG")) { - MsgSetTAG(pThis, (char*) rsCStrGetSzStrNoNULL(pProp->val.pStr)); + MsgSetTAG(pThis, rsCStrGetSzStrNoNULL(pProp->val.pStr), cstrLen(pProp->val.pStr)); } else if(isProp("pszInputName")) { - MsgSetInputName(pThis, rsCStrGetSzStrNoNULL(pProp->val.pStr), rsCStrLen(pProp->val.pStr)); + /* we need to create a property */ + CHKiRet(prop.Construct(&myProp)); + CHKiRet(prop.SetString(myProp, rsCStrGetSzStrNoNULL(pProp->val.pStr), rsCStrLen(pProp->val.pStr))); + CHKiRet(prop.ConstructFinalize(myProp)); + MsgSetInputName(pThis, myProp); + prop.Destruct(&myProp); } else if(isProp("pszRcvFromIP")) { - MsgSetRcvFromIP(pThis, rsCStrGetSzStrNoNULL(pProp->val.pStr)); + MsgSetRcvFromIPStr(pThis, rsCStrGetSzStrNoNULL(pProp->val.pStr), rsCStrLen(pProp->val.pStr), &propRcvFromIP); + prop.Destruct(&propRcvFromIP); } else if(isProp("pszRcvFrom")) { - MsgSetRcvFrom(pThis, rsCStrGetSzStrNoNULL(pProp->val.pStr)); + MsgSetRcvFromStr(pThis, rsCStrGetSzStrNoNULL(pProp->val.pStr), rsCStrLen(pProp->val.pStr), &propRcvFrom); + prop.Destruct(&propRcvFrom); } else if(isProp("pszHOSTNAME")) { - MsgSetHOSTNAME(pThis, rsCStrGetSzStrNoNULL(pProp->val.pStr)); + MsgSetHOSTNAME(pThis, rsCStrGetSzStrNoNULL(pProp->val.pStr), rsCStrLen(pProp->val.pStr)); } else if(isProp("pCSStrucData")) { MsgSetStructuredData(pThis, (char*) rsCStrGetSzStrNoNULL(pProp->val.pStr)); } else if(isProp("pCSAPPNAME")) { @@ -2487,8 +3021,11 @@ rsRetVal MsgSetProperty(msg_t *pThis, var_t *pProp) memcpy(&pThis->tRcvdAt, &pProp->val.vSyslogTime, sizeof(struct syslogTime)); } else if(isProp("tTIMESTAMP")) { memcpy(&pThis->tTIMESTAMP, &pProp->val.vSyslogTime, sizeof(struct syslogTime)); + } else if(isProp("pszMSG")) { + dbgprintf("no longer supported property pszMSG silently ignored\n"); } +finalize_it: RETiRet; } #undef isProp @@ -2532,6 +3069,7 @@ BEGINObjClassInit(msg, 1, OBJ_IS_CORE_MODULE) CHKiRet(objUse(var, CORE_COMPONENT)); CHKiRet(objUse(datetime, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); /* set our own handlers */ OBJSetMethodHandler(objMethod_SERIALIZE, MsgSerialize); diff --git a/runtime/msg.h b/runtime/msg.h index fe9f87fa..3a02365b 100644 --- a/runtime/msg.h +++ b/runtime/msg.h @@ -3,7 +3,7 @@ * * File begun on 2007-07-13 by RGerhards (extracted from syslogd.c) * - * Copyright 2007 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2009 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -33,6 +33,7 @@ #include "syslogd-types.h" #include "template.h" + /* rgerhards 2004-11-08: The following structure represents a * syslog message. * @@ -47,16 +48,20 @@ * will be decremented. If it is 1, however, the object is actually * destroyed. To make this work, it is vital that MsgAddRef() is * called each time a "copy" is stored somewhere. + * + * WARNING: this structure is not calloc()ed, so be careful when + * adding new fields. You need to initialize them in + * msgBaseConstruct(). That function header comment also describes + * why this is the case. */ struct msg { BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ - pthread_mutexattr_t mutAttr; - bool bDoLock; /* use the mutex? */ - pthread_mutex_t mut; flowControl_t flowCtlType; /**< type of flow control we can apply, for enqueueing, needs not to be persisted because once data has entered the queue, this property is no longer needed. */ + pthread_mutex_t mut; + bool bDoLock; /* use the mutex? */ + bool bParseHOSTNAME; /* should the hostname be parsed from the message? */ short iRefCount; /* reference counter (0 = unused) */ - short bParseHOSTNAME; /* should the hostname be parsed from the message? */ /* background: the hostname is not present on "regular" messages * received via UNIX domain sockets from the same machine. However, * it is available when we have a forwarder (e.g. rfc3195d) using local @@ -64,43 +69,35 @@ struct msg { * resolve all these issues... rgerhards, 2005-10-06 */ short iSeverity; /* the severity 0..7 */ - uchar *pszSeverity; /* severity as string... */ - int iLenSeverity; /* ... and its length. */ - uchar *pszSeverityStr; /* severity name... */ - int iLenSeverityStr; /* ... and its length. */ short iFacility; /* Facility code 0 .. 23*/ - uchar *pszFacility; /* Facility as string... */ - int iLenFacility; /* ... and its length. */ - uchar *pszFacilityStr; /* facility name... */ - int iLenFacilityStr; /* ... and its length. */ - uchar bufPRI[5]; /* PRI as string */ - int iLenPRI; /* and its length */ - uchar *pszRawMsg; /* message as it was received on the - * wire. This is important in case we - * need to preserve cryptographic verifiers. - */ short offAfterPRI; /* offset, at which raw message WITHOUT PRI part starts in pszRawMsg */ + short offMSG; /* offset at which the MSG part starts in pszRawMsg */ + short iProtocolVersion;/* protocol version of message received 0 - legacy, 1 syslog-protocol) */ + int msgFlags; /* flags associated with this message */ int iLenRawMsg; /* length of raw message */ - uchar *pszMSG; /* the MSG part itself */ int iLenMSG; /* Length of the MSG part */ - uchar *pszUxTradMsg; /* the traditional UNIX message */ - int iLenUxTradMsg;/* Length of the traditional UNIX message */ - uchar *pszTAG; /* pointer to tag value */ int iLenTAG; /* Length of the TAG part */ - uchar *pszHOSTNAME; /* HOSTNAME from syslog message */ int iLenHOSTNAME; /* Length of HOSTNAME */ - uchar *pszRcvFrom; /* System message was received from */ - int iLenRcvFrom; /* Length of pszRcvFrom */ - uchar *pszRcvFromIP; /* IP of system message was received from */ - int iLenRcvFromIP; /* Length of pszRcvFromIP */ - uchar *pszInputName; /* name of the input module that submitted this message */ - int iLenInputName; /* Length of pszInputName */ - short iProtocolVersion;/* protocol version of message received 0 - legacy, 1 syslog-protocol) */ + uchar *pszRawMsg; /* message as it was received on the wire. This is important in case we + * need to preserve cryptographic verifiers. */ + uchar *pszHOSTNAME; /* HOSTNAME from syslog message */ + char *pszRcvdAt3164; /* time as RFC3164 formatted string (always 15 charcters) */ + char *pszRcvdAt3339; /* time as RFC3164 formatted string (32 charcters at most) */ + char *pszRcvdAt_MySQL; /* rcvdAt as MySQL formatted string (always 14 charcters) */ + char *pszRcvdAt_PgSQL; /* rcvdAt as PgSQL formatted string (always 21 characters) */ + char *pszTIMESTAMP3164; /* TIMESTAMP as RFC3164 formatted string (always 15 charcters) */ + char *pszTIMESTAMP3339; /* TIMESTAMP as RFC3339 formatted string (32 charcters at most) */ + char *pszTIMESTAMP_MySQL;/* TIMESTAMP as MySQL formatted string (always 14 charcters) */ + char *pszTIMESTAMP_PgSQL;/* TIMESTAMP as PgSQL formatted string (always 21 characters) */ cstr_t *pCSProgName; /* the (BSD) program name */ cstr_t *pCSStrucData; /* STRUCTURED-DATA */ cstr_t *pCSAPPNAME; /* APP-NAME */ cstr_t *pCSPROCID; /* PROCID */ cstr_t *pCSMSGID; /* MSGID */ + prop_t *pInputName; /* input name property */ + prop_t *pRcvFrom; /* name of system message was received from */ + prop_t *pRcvFromIP; /* IP of system message was received from */ + ruleset_t *pRuleset; /* ruleset to be used for processing this message */ time_t ttGenTime; /* time msg object was generated, same as tRcvdAt, but a Unix timestamp. While this field looks redundant, it is required because a Unix timestamp is used at later processing stages (namely in the output arena). Thanks to @@ -109,20 +106,18 @@ struct msg { enough to reliable, but I prefer to leave the subtle things to the OS, where it obviously is solved in way or another...). */ struct syslogTime tRcvdAt;/* time the message entered this program */ - char *pszRcvdAt3164; /* time as RFC3164 formatted string (always 15 charcters) */ - char *pszRcvdAt3339; /* time as RFC3164 formatted string (32 charcters at most) */ - char *pszRcvdAt_SecFrac;/* time just as fractional seconds (6 charcters) */ - char *pszRcvdAt_MySQL; /* rcvdAt as MySQL formatted string (always 14 charcters) */ - char *pszRcvdAt_PgSQL; /* rcvdAt as PgSQL formatted string (always 21 characters) */ struct syslogTime tTIMESTAMP;/* (parsed) value of the timestamp */ - char *pszTIMESTAMP3164; /* TIMESTAMP as RFC3164 formatted string (always 15 charcters) */ - char *pszTIMESTAMP3339; /* TIMESTAMP as RFC3339 formatted string (32 charcters at most) */ - char *pszTIMESTAMP_MySQL;/* TIMESTAMP as MySQL formatted string (always 14 charcters) */ - char *pszTIMESTAMP_PgSQL;/* TIMESTAMP as PgSQL formatted string (always 21 characters) */ - char *pszTIMESTAMP_SecFrac;/* TIMESTAMP fractional seconds (always 6 characters) */ - int msgFlags; /* flags associated with this message */ - /* now follow fixed-size buffers to safe some time otherwise used for allocs */ - + /* some fixed-size buffers to save malloc()/free() for frequently used fields (from the default templates) */ + uchar szRawMsg[CONF_RAWMSG_BUFSIZE]; /* most messages are small, and these are stored here (without malloc/free!) */ + uchar szHOSTNAME[CONF_HOSTNAME_BUFSIZE]; + union { + uchar *pszTAG; /* pointer to tag value */ + uchar szBuf[CONF_TAG_BUFSIZE]; + } TAG; + char pszTimestamp3164[16]; + char pszTimestamp3339[33]; + char pszTIMESTAMP_SecFrac[7]; /* Note: a pointer is 64 bits/8 char, so this is actually fewer than a pointer! */ + char pszRcvdAt_SecFrac[7]; /* same as above. Both are fractional seconds for their respective timestamp */ }; @@ -146,45 +141,47 @@ rsRetVal msgDestruct(msg_t **ppM); msg_t* MsgDup(msg_t* pOld); msg_t *MsgAddRef(msg_t *pM); void setProtocolVersion(msg_t *pM, int iNewVersion); -void MsgSetInputName(msg_t *pMsg, uchar*, size_t); +void MsgSetInputName(msg_t *pMsg, prop_t*); rsRetVal MsgSetAPPNAME(msg_t *pMsg, char* pszAPPNAME); rsRetVal MsgSetPROCID(msg_t *pMsg, char* pszPROCID); rsRetVal MsgSetMSGID(msg_t *pMsg, char* pszMSGID); -void MsgAssignTAG(msg_t *pMsg, uchar *pBuf); -void MsgSetTAG(msg_t *pMsg, char* pszTAG); +void MsgSetTAG(msg_t *pMsg, uchar* pszBuf, size_t lenBuf); +void MsgSetRuleset(msg_t *pMsg, ruleset_t*); rsRetVal MsgSetFlowControlType(msg_t *pMsg, flowControl_t eFlowCtl); rsRetVal MsgSetStructuredData(msg_t *pMsg, char* pszStrucData); -void MsgSetRcvFrom(msg_t *pMsg, uchar* pszRcvFrom); -rsRetVal MsgSetRcvFromIP(msg_t *pMsg, uchar* pszRcvFromIP); -void MsgAssignHOSTNAME(msg_t *pMsg, char *pBuf); -void MsgSetHOSTNAME(msg_t *pMsg, uchar* pszHOSTNAME); +void MsgSetRcvFrom(msg_t *pMsg, prop_t*); +void MsgSetRcvFromStr(msg_t *pMsg, uchar* pszRcvFrom, int, prop_t **); +rsRetVal MsgSetRcvFromIP(msg_t *pMsg, prop_t*); +rsRetVal MsgSetRcvFromIPStr(msg_t *pThis, uchar *psz, int len, prop_t **ppProp); +void MsgSetHOSTNAME(msg_t *pMsg, uchar* pszHOSTNAME, int lenHOSTNAME); rsRetVal MsgSetAfterPRIOffs(msg_t *pMsg, short offs); -void MsgSetMSG(msg_t *pMsg, char* pszMSG); -void MsgSetRawMsg(msg_t *pMsg, char* pszRawMsg); -void moveHOSTNAMEtoTAG(msg_t *pM); -char *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, - cstr_t *pCSPropName, unsigned short *pbMustBeFreed); +void MsgSetMSGoffs(msg_t *pMsg, short offs); +void MsgSetRawMsgWOSize(msg_t *pMsg, char* pszRawMsg); +void MsgSetRawMsg(msg_t *pMsg, char* pszRawMsg, size_t lenMsg); +rsRetVal MsgReplaceMSG(msg_t *pThis, uchar* pszMSG, int lenMSG); +uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, + propid_t propID, size_t *pPropLen, unsigned short *pbMustBeFreed); char *textpri(char *pRes, size_t pResLen, int pri); rsRetVal msgGetMsgVar(msg_t *pThis, cstr_t *pstrPropName, var_t **ppVar); rsRetVal MsgEnableThreadSafety(void); +uchar *getRcvFrom(msg_t *pM); + /* TODO: remove these five (so far used in action.c) */ -char *getMSG(msg_t *pM); +uchar *getMSG(msg_t *pM); char *getHOSTNAME(msg_t *pM); -char *getPROCID(msg_t *pM); -char *getAPPNAME(msg_t *pM); +char *getPROCID(msg_t *pM, bool bLockMutex); +char *getAPPNAME(msg_t *pM, bool bLockMutex); int getMSGLen(msg_t *pM); char *getHOSTNAME(msg_t *pM); int getHOSTNAMELen(msg_t *pM); -char *getProgramName(msg_t *pM); -int getProgramNameLen(msg_t *pM); +char *getProgramName(msg_t *pM, bool bLockMutex); +int getProgramNameLen(msg_t *pM, bool bLockMutex); uchar *getRcvFrom(msg_t *pM); +rsRetVal propNameToID(cstr_t *pCSPropName, propid_t *pPropID); +uchar *propIDToName(propid_t propID); -#if 0 -char *getUxTradMsg(msg_t *pM); -int MsgSetUxTradMsg(msg_t *pMsg, char* pszUxTradMsg); -#endif /* The MsgPrepareEnqueue() function is a macro for performance reasons. * It needs one global variable to work. This is acceptable, as it gains @@ -195,6 +192,23 @@ int MsgSetUxTradMsg(msg_t *pMsg, char* pszUxTradMsg); extern void (*funcMsgPrepareEnqueue)(msg_t *pMsg); #define MsgPrepareEnqueue(pMsg) funcMsgPrepareEnqueue(pMsg) + +/* ------------------------------ some inline functions ------------------------------ */ + +/* set raw message size. This is needed in some cases where a trunctation is necessary + * but the raw message must not be newly set. The most important (and currently only) + * use case is if we remove trailing LF or NUL characters. Note that the size can NOT + * be extended, only shrunk! + * rgerhards, 2009-08-26 + */ +static inline void +MsgSetRawMsgSize(msg_t *pMsg, size_t newLen) +{ + assert(newLen <= (size_t) pMsg->iLenRawMsg); + pMsg->iLenRawMsg = newLen; +} + + #endif /* #ifndef MSG_H_INCLUDED */ /* vim:set ai: */ diff --git a/runtime/net.c b/runtime/net.c index 5cafe522..fe6eef5b 100644 --- a/runtime/net.c +++ b/runtime/net.c @@ -1010,8 +1010,8 @@ static int should_use_so_bsdcompat(void) { #ifndef OS_BSD - static int init_done; - static int so_bsdcompat_is_obsolete; + static int init_done = 0; + static int so_bsdcompat_is_obsolete = 0; if (!init_done) { struct utsname myutsname; @@ -1568,6 +1568,9 @@ CODESTARTobjQueryInterface(net) pIf->DestructPermittedPeers = DestructPermittedPeers; pIf->PermittedPeerWildcardMatch = PermittedPeerWildcardMatch; pIf->CmpHost = CmpHost; + /* data members */ + pIf->pACLAddHostnameOnFail = &ACLAddHostnameOnFail; + pIf->pACLDontResolve = &ACLDontResolve; finalize_it: ENDobjQueryInterface(net) diff --git a/runtime/nsd_gtls.c b/runtime/nsd_gtls.c index 19dc8678..fb2e219d 100644 --- a/runtime/nsd_gtls.c +++ b/runtime/nsd_gtls.c @@ -710,16 +710,16 @@ gtlsGetCN(nsd_gtls_t *pThis, gnutls_x509_crt *pCert, cstr_t **ppstrCN) } /* we found a common name, now extract it */ - CHKiRet(rsCStrConstruct(&pstrCN)); + CHKiRet(cstrConstruct(&pstrCN)); while(szDN[i] != '\0' && szDN[i] != ',') { if(szDN[i] == '\\') { /* hex escapes are not implemented */ ++i; /* escape char processed */ if(szDN[i] == '\0') ABORT_FINALIZE(RS_RET_CERT_INVALID_DN); - CHKiRet(rsCStrAppendChar(pstrCN, szDN[i])); + CHKiRet(cstrAppendChar(pstrCN, szDN[i])); } else { - CHKiRet(rsCStrAppendChar(pstrCN, szDN[i])); + CHKiRet(cstrAppendChar(pstrCN, szDN[i])); } ++i; /* char processed */ } @@ -734,7 +734,7 @@ gtlsGetCN(nsd_gtls_t *pThis, gnutls_x509_crt *pCert, cstr_t **ppstrCN) finalize_it: if(iRet != RS_RET_OK) { if(pstrCN != NULL) - rsCStrDestruct(&pstrCN); + cstrDestruct(&pstrCN); } RETiRet; @@ -761,7 +761,7 @@ gtlsChkPeerFingerprint(nsd_gtls_t *pThis, gnutls_x509_crt *pCert) size = sizeof(fingerprint); CHKgnutls(gnutls_x509_crt_get_fingerprint(*pCert, GNUTLS_DIG_SHA1, fingerprint, &size)); CHKiRet(GenFingerprintStr(fingerprint, size, &pstrFingerprint)); - dbgprintf("peer's certificate SHA1 fingerprint: %s\n", rsCStrGetSzStr(pstrFingerprint)); + dbgprintf("peer's certificate SHA1 fingerprint: %s\n", cstrGetSzStr(pstrFingerprint)); /* now search through the permitted peers to see if we can find a permitted one */ bFoundPositiveMatch = 0; @@ -779,7 +779,7 @@ gtlsChkPeerFingerprint(nsd_gtls_t *pThis, gnutls_x509_crt *pCert) if(pThis->bReportAuthErr == 1) { errno = 0; errmsg.LogError(0, RS_RET_INVALID_FINGERPRINT, "error: peer fingerprint '%s' unknown - we are " - "not permitted to talk to it", rsCStrGetSzStr(pstrFingerprint)); + "not permitted to talk to it", cstrGetSzStr(pstrFingerprint)); pThis->bReportAuthErr = 0; } ABORT_FINALIZE(RS_RET_INVALID_FINGERPRINT); @@ -787,7 +787,7 @@ gtlsChkPeerFingerprint(nsd_gtls_t *pThis, gnutls_x509_crt *pCert) finalize_it: if(pstrFingerprint != NULL) - rsCStrDestruct(&pstrFingerprint); + cstrDestruct(&pstrFingerprint); RETiRet; } @@ -874,10 +874,10 @@ gtlsChkPeerName(nsd_gtls_t *pThis, gnutls_x509_crt *pCert) /* if we did not succeed so far, we try the CN part of the DN... */ CHKiRet(gtlsGetCN(pThis, pCert, &pstrCN)); if(pstrCN != NULL) { /* NULL if there was no CN present */ - dbgprintf("gtls now checking auth for CN '%s'\n", rsCStrGetSzStr(pstrCN)); - snprintf((char*)lnBuf, sizeof(lnBuf), "CN: %s; ", rsCStrGetSzStr(pstrCN)); + dbgprintf("gtls now checking auth for CN '%s'\n", cstrGetSzStr(pstrCN)); + snprintf((char*)lnBuf, sizeof(lnBuf), "CN: %s; ", cstrGetSzStr(pstrCN)); CHKiRet(rsCStrAppendStr(pStr, lnBuf)); - CHKiRet(gtlsChkOnePeerName(pThis, rsCStrGetSzStr(pstrCN), &bFoundPositiveMatch)); + CHKiRet(gtlsChkOnePeerName(pThis, cstrGetSzStr(pstrCN), &bFoundPositiveMatch)); } } @@ -888,7 +888,7 @@ gtlsChkPeerName(nsd_gtls_t *pThis, gnutls_x509_crt *pCert) errno = 0; errmsg.LogError(0, RS_RET_INVALID_FINGERPRINT, "error: peer name not authorized - " "not permitted to talk to it. Names: %s", - rsCStrGetSzStr(pStr)); + cstrGetSzStr(pStr)); pThis->bReportAuthErr = 0; } ABORT_FINALIZE(RS_RET_INVALID_FINGERPRINT); @@ -1010,8 +1010,8 @@ gtlsChkPeerCertValidity(nsd_gtls_t *pThis) errmsg.LogError(0, NO_ERRCODE, "not permitted to talk to peer, certificate invalid: %s", pszErrCause); gtlsGetCertInfo(pThis, &pStr); - errmsg.LogError(0, NO_ERRCODE, "invalid cert info: %s", rsCStrGetSzStr(pStr)); - rsCStrDestruct(&pStr); + errmsg.LogError(0, NO_ERRCODE, "invalid cert info: %s", cstrGetSzStr(pStr)); + cstrDestruct(&pStr); ABORT_FINALIZE(RS_RET_CERT_INVALID); } @@ -1032,8 +1032,8 @@ gtlsChkPeerCertValidity(nsd_gtls_t *pThis) else if(ttCert > ttNow) { errmsg.LogError(0, RS_RET_CERT_NOT_YET_ACTIVE, "not permitted to talk to peer: certificate %d not yet active", i); gtlsGetCertInfo(pThis, &pStr); - errmsg.LogError(0, RS_RET_CERT_NOT_YET_ACTIVE, "invalid cert info: %s", rsCStrGetSzStr(pStr)); - rsCStrDestruct(&pStr); + errmsg.LogError(0, RS_RET_CERT_NOT_YET_ACTIVE, "invalid cert info: %s", cstrGetSzStr(pStr)); + cstrDestruct(&pStr); ABORT_FINALIZE(RS_RET_CERT_NOT_YET_ACTIVE); } @@ -1043,8 +1043,8 @@ gtlsChkPeerCertValidity(nsd_gtls_t *pThis) else if(ttCert < ttNow) { errmsg.LogError(0, RS_RET_CERT_EXPIRED, "not permitted to talk to peer: certificate %d expired", i); gtlsGetCertInfo(pThis, &pStr); - errmsg.LogError(0, RS_RET_CERT_EXPIRED, "invalid cert info: %s", rsCStrGetSzStr(pStr)); - rsCStrDestruct(&pStr); + errmsg.LogError(0, RS_RET_CERT_EXPIRED, "invalid cert info: %s", cstrGetSzStr(pStr)); + cstrDestruct(&pStr); ABORT_FINALIZE(RS_RET_CERT_EXPIRED); } gnutls_x509_crt_deinit(cert); @@ -1325,7 +1325,10 @@ finalize_it: static void CheckConnection(nsd_t __attribute__((unused)) *pNsd) { - /* dummy, do nothing */ + nsd_gtls_t *pThis = (nsd_gtls_t*) pNsd; + ISOBJ_TYPE_assert(pThis, nsd_gtls); + + nsd_ptcp.CheckConnection(pThis->pTcp); } diff --git a/runtime/nsdsel_gtls.c b/runtime/nsdsel_gtls.c index c3a93bee..1a389a00 100644 --- a/runtime/nsdsel_gtls.c +++ b/runtime/nsdsel_gtls.c @@ -76,6 +76,9 @@ Add(nsdsel_t *pNsdsel, nsd_t *pNsd, nsdsel_waitOp_t waitOp) if(pNsdGTLS->iMode == 1) { if(waitOp == NSDSEL_RD && gtlsHasRcvInBuffer(pNsdGTLS)) { ++pThis->iBufferRcvReady; + dbgprintf("nsdsel_gtls: data already present in buffer, initiating " + "dummy select %p->iBufferRcvReady=%d\n", + pThis, pThis->iBufferRcvReady); FINALIZE; } if(pNsdGTLS->rtryCall != gtlsRtry_None) { @@ -109,6 +112,7 @@ Select(nsdsel_t *pNsdsel, int *piNumReady) if(pThis->iBufferRcvReady > 0) { /* we still have data ready! */ *piNumReady = pThis->iBufferRcvReady; + dbgprintf("nsdsel_gtls: doing dummy select, data present\n"); } else { iRet = nsdsel_ptcp.Select(pThis->pTcp, piNumReady); } @@ -190,6 +194,9 @@ IsReady(nsdsel_t *pNsdsel, nsd_t *pNsd, nsdsel_waitOp_t waitOp, int *pbIsReady) if(pNsdGTLS->iMode == 1) { if(waitOp == NSDSEL_RD && gtlsHasRcvInBuffer(pNsdGTLS)) { *pbIsReady = 1; + --pThis->iBufferRcvReady; /* one "pseudo-read" less */ + dbgprintf("nsdl_gtls: dummy read, decermenting %p->iBufRcvReady, now %d\n", + pThis, pThis->iBufferRcvReady); FINALIZE; } if(pNsdGTLS->rtryCall != gtlsRtry_None) { @@ -200,6 +207,16 @@ IsReady(nsdsel_t *pNsdsel, nsd_t *pNsd, nsdsel_waitOp_t waitOp, int *pbIsReady) *pbIsReady = 0; FINALIZE; } + /* now we must ensure that we do not fall back to PTCP if we have + * done a "dummy" select. In that case, we know when the predicate + * is not matched here, we do not have data available for this + * socket. -- rgerhards, 2010-11-20 + */ + if(pThis->iBufferRcvReady) { + dbgprintf("nsd_gtls: dummy read, buffer not available for this FD\n"); + *pbIsReady = 0; + FINALIZE; + } } CHKiRet(nsdsel_ptcp.IsReady(pThis->pTcp, pNsdGTLS->pTcp, waitOp, pbIsReady)); diff --git a/runtime/obj-types.h b/runtime/obj-types.h index 78829f94..e1b54d4f 100644 --- a/runtime/obj-types.h +++ b/runtime/obj-types.h @@ -105,12 +105,13 @@ struct obj_s { /* the dummy struct that each derived class can be casted to */ # define ISOBJ_TYPE_assert(pObj, objType) \ do { \ ASSERT(pObj != NULL); \ - ASSERT((unsigned) ((obj_t*) (pObj))->iObjCooCKiE == (unsigned) 0xBADEFEE); \ if(strcmp((char*)(((obj_t*)pObj)->pObjInfo->pszID), #objType)) { \ dbgprintf("%s:%d ISOBJ assert failure: invalid object type, expected '%s' " \ - "actual '%s'\n", __FILE__, __LINE__, #objType, (((obj_t*)pObj)->pObjInfo->pszID)); \ + "actual '%s', cookie: %X\n", __FILE__, __LINE__, #objType, \ + (((obj_t*)pObj)->pObjInfo->pszID), ((obj_t*)(pObj))->iObjCooCKiE); \ assert(0); /* trigger assertion, messge we already have */ \ } \ + ASSERT((unsigned) ((obj_t*)(pObj))->iObjCooCKiE == (unsigned) 0xBADEFEE); \ } while(0) #else /* non-debug mode, no checks but much faster */ # define BEGINobjInstance obj_t objData @@ -292,6 +293,15 @@ rsRetVal objName##ClassExit(void) \ ISOBJ_TYPE_assert(pThis, OBJ); \ pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &iCancelStateSave); +/* note: there was a long-time bug in the macro below that lead to *ppThis = NULL + * only when the object was actually destructed. I discovered this issue during + * introduction of the pRcvFrom property in msg_t, but it potentially had other + * effects, too. I am not sure if some experienced instability resulted from this + * bug OR if its fix will cause harm to so-far "correctly" running code. The later + * may very well be. Thus I will change it only for the current branch and also + * the beta, but not in all old builds. Let's see how things evolve. + * rgerhards, 2009-06-30 + */ #define ENDobjDestruct(OBJ) \ goto finalize_it; /* prevent compiler warning ;) */ \ /* no more code here! */ \ @@ -299,8 +309,8 @@ rsRetVal objName##ClassExit(void) \ if(pThis != NULL) { \ obj.DestructObjSelf((obj_t*) pThis); \ free(pThis); \ - *ppThis = NULL; \ } \ + *ppThis = NULL; \ pthread_setcancelstate(iCancelStateSave, NULL); \ RETiRet; \ } diff --git a/runtime/obj.c b/runtime/obj.c index 355c0f97..aebea332 100644 --- a/runtime/obj.c +++ b/runtime/obj.c @@ -75,6 +75,7 @@ #include <string.h> #include <ctype.h> #include <assert.h> +#include <pthread.h> /* how many objects are supported by rsyslogd? */ #define OBJ_NUM_IDS 100 /* TODO change to a linked list? info: 16 were currently in use 2008-02-29 */ @@ -87,13 +88,17 @@ #include "modules.h" #include "errmsg.h" #include "cfsysline.h" +#include "unicode-helper.h" +#include "apc.h" /* static data */ DEFobjCurrIf(obj) /* we define our own interface, as this is expected by some macros! */ DEFobjCurrIf(var) DEFobjCurrIf(module) DEFobjCurrIf(errmsg) +DEFobjCurrIf(strm) static objInfo_t *arrObjInfo[OBJ_NUM_IDS]; /* array with object information pointers */ +static pthread_mutex_t mutObjGlobalOp; /* mutex to guard global operations of the object system */ /* cookies for serialized lines */ @@ -144,8 +149,8 @@ InfoConstruct(objInfo_t **ppThis, uchar *pszID, int iObjVers, ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); pThis->pszID = pszID; - pThis->lenID = strlen((char*)pszID); - pThis->pszName = (uchar*)strdup((char*)pszID); /* it's OK if we have NULL ptr, GetName() will deal with that! */ + pThis->lenID = ustrlen(pszID); + pThis->pszName = ustrdup(pszID); /* it's OK if we have NULL ptr, GetName() will deal with that! */ pThis->iObjVers = iObjVers; pThis->QueryIF = pQueryIF; pThis->pModInfo = pModInfo; @@ -176,8 +181,7 @@ InfoDestruct(objInfo_t **ppThis) pThis = *ppThis; assert(pThis != NULL); - if(pThis->pszName != NULL) - free(pThis->pszName); + free(pThis->pszName); free(pThis); *ppThis = NULL; @@ -205,9 +209,7 @@ DestructObjSelf(obj_t *pThis) DEFiRet; ISOBJ_assert(pThis); - if(pThis->pszName != NULL) { - free(pThis->pszName); - } + free(pThis->pszName); RETiRet; } @@ -228,20 +230,20 @@ static rsRetVal objSerializeHeader(strm_t *pStrm, obj_t *pObj, uchar *pszRecType assert(!strcmp((char*) pszRecType, "Obj") || !strcmp((char*) pszRecType, "OPB")); /* object cookie and serializer version (so far always 1) */ - CHKiRet(strmWriteChar(pStrm, COOKIE_OBJLINE)); - CHKiRet(strmWrite(pStrm, (uchar*) pszRecType, 3)); /* record types are always 3 octets */ - CHKiRet(strmWriteChar(pStrm, ':')); - CHKiRet(strmWriteChar(pStrm, '1')); + CHKiRet(strm.WriteChar(pStrm, COOKIE_OBJLINE)); + CHKiRet(strm.Write(pStrm, (uchar*) pszRecType, 3)); /* record types are always 3 octets */ + CHKiRet(strm.WriteChar(pStrm, ':')); + CHKiRet(strm.WriteChar(pStrm, '1')); /* object type, version and string length */ - CHKiRet(strmWriteChar(pStrm, ':')); - CHKiRet(strmWrite(pStrm, pObj->pObjInfo->pszID, pObj->pObjInfo->lenID)); - CHKiRet(strmWriteChar(pStrm, ':')); - CHKiRet(strmWriteLong(pStrm, objGetVersion(pObj))); + CHKiRet(strm.WriteChar(pStrm, ':')); + CHKiRet(strm.Write(pStrm, pObj->pObjInfo->pszID, pObj->pObjInfo->lenID)); + CHKiRet(strm.WriteChar(pStrm, ':')); + CHKiRet(strm.WriteLong(pStrm, objGetVersion(pObj))); /* record trailer */ - CHKiRet(strmWriteChar(pStrm, ':')); - CHKiRet(strmWriteChar(pStrm, '\n')); + CHKiRet(strm.WriteChar(pStrm, ':')); + CHKiRet(strm.WriteChar(pStrm, '\n')); finalize_it: RETiRet; @@ -259,7 +261,7 @@ BeginSerialize(strm_t *pStrm, obj_t *pObj) ISOBJ_TYPE_assert(pStrm, strm); ISOBJ_assert(pObj); - CHKiRet(strmRecordBegin(pStrm)); + CHKiRet(strm.RecordBegin(pStrm)); CHKiRet(objSerializeHeader(pStrm, pObj, (uchar*) "Obj")); finalize_it: @@ -284,7 +286,7 @@ BeginSerializePropBag(strm_t *pStrm, obj_t *pObj) ISOBJ_TYPE_assert(pStrm, strm); ISOBJ_assert(pObj); - CHKiRet(strmRecordBegin(pStrm)); + CHKiRet(strm.RecordBegin(pStrm)); CHKiRet(objSerializeHeader(pStrm, pObj, (uchar*) "OPB")); finalize_it: @@ -320,31 +322,31 @@ SerializeProp(strm_t *pStrm, uchar *pszPropName, propType_t propType, void *pUsr switch(propType) { case PROPTYPE_PSZ: pszBuf = (uchar*) pUsr; - lenBuf = strlen((char*) pszBuf); + lenBuf = ustrlen(pszBuf); vType = VARTYPE_STR; break; case PROPTYPE_SHORT: CHKiRet(srUtilItoA((char*) szBuf, sizeof(szBuf), (long) *((short*) pUsr))); pszBuf = szBuf; - lenBuf = strlen((char*) szBuf); + lenBuf = ustrlen(szBuf); vType = VARTYPE_NUMBER; break; case PROPTYPE_INT: CHKiRet(srUtilItoA((char*) szBuf, sizeof(szBuf), (long) *((int*) pUsr))); pszBuf = szBuf; - lenBuf = strlen((char*) szBuf); + lenBuf = ustrlen(szBuf); vType = VARTYPE_NUMBER; break; case PROPTYPE_LONG: CHKiRet(srUtilItoA((char*) szBuf, sizeof(szBuf), *((long*) pUsr))); pszBuf = szBuf; - lenBuf = strlen((char*) szBuf); + lenBuf = ustrlen(szBuf); vType = VARTYPE_NUMBER; break; case PROPTYPE_INT64: CHKiRet(srUtilItoA((char*) szBuf, sizeof(szBuf), *((int64*) pUsr))); pszBuf = szBuf; - lenBuf = strlen((char*) szBuf); + lenBuf = ustrlen(szBuf); vType = VARTYPE_NUMBER; break; case PROPTYPE_CSTR: @@ -377,23 +379,23 @@ SerializeProp(strm_t *pStrm, uchar *pszPropName, propType_t propType, void *pUsr } /* cookie */ - CHKiRet(strmWriteChar(pStrm, COOKIE_PROPLINE)); + CHKiRet(strm.WriteChar(pStrm, COOKIE_PROPLINE)); /* name */ - CHKiRet(strmWrite(pStrm, pszPropName, strlen((char*)pszPropName))); - CHKiRet(strmWriteChar(pStrm, ':')); + CHKiRet(strm.Write(pStrm, pszPropName, ustrlen(pszPropName))); + CHKiRet(strm.WriteChar(pStrm, ':')); /* type */ - CHKiRet(strmWriteLong(pStrm, (int) vType)); - CHKiRet(strmWriteChar(pStrm, ':')); + CHKiRet(strm.WriteLong(pStrm, (int) vType)); + CHKiRet(strm.WriteChar(pStrm, ':')); /* length */ - CHKiRet(strmWriteLong(pStrm, lenBuf)); - CHKiRet(strmWriteChar(pStrm, ':')); + CHKiRet(strm.WriteLong(pStrm, lenBuf)); + CHKiRet(strm.WriteChar(pStrm, ':')); /* data */ - CHKiRet(strmWrite(pStrm, (uchar*) pszBuf, lenBuf)); + CHKiRet(strm.Write(pStrm, (uchar*) pszBuf, lenBuf)); /* trailer */ - CHKiRet(strmWriteChar(pStrm, ':')); - CHKiRet(strmWriteChar(pStrm, '\n')); + CHKiRet(strm.WriteChar(pStrm, ':')); + CHKiRet(strm.WriteChar(pStrm, '\n')); finalize_it: RETiRet; @@ -410,12 +412,12 @@ EndSerialize(strm_t *pStrm) assert(pStrm != NULL); - CHKiRet(strmWriteChar(pStrm, COOKIE_ENDLINE)); - CHKiRet(strmWrite(pStrm, (uchar*) "End\n", sizeof("END\n") - 1)); - CHKiRet(strmWriteChar(pStrm, COOKIE_BLANKLINE)); - CHKiRet(strmWriteChar(pStrm, '\n')); + CHKiRet(strm.WriteChar(pStrm, COOKIE_ENDLINE)); + CHKiRet(strm.Write(pStrm, (uchar*) "End\n", sizeof("END\n") - 1)); + CHKiRet(strm.WriteChar(pStrm, COOKIE_BLANKLINE)); + CHKiRet(strm.WriteChar(pStrm, '\n')); - CHKiRet(strmRecordEnd(pStrm)); + CHKiRet(strm.RecordEnd(pStrm)); finalize_it: RETiRet; @@ -423,7 +425,7 @@ finalize_it: /* define a helper to make code below a bit cleaner (and quicker to write) */ -#define NEXTC CHKiRet(strmReadChar(pStrm, &c))/*;dbgprintf("c: %c\n", c)*/ +#define NEXTC CHKiRet(strm.ReadChar(pStrm, &c))/*;dbgprintf("c: %c\n", c)*/ /* de-serialize an embedded, non-octect-counted string. This is useful @@ -440,11 +442,11 @@ objDeserializeEmbedStr(cstr_t **ppStr, strm_t *pStrm) assert(ppStr != NULL); - CHKiRet(rsCStrConstruct(&pStr)); + CHKiRet(cstrConstruct(&pStr)); NEXTC; while(c != ':') { - CHKiRet(rsCStrAppendChar(pStr, c)); + CHKiRet(cstrAppendChar(pStr, c)); NEXTC; } CHKiRet(cstrFinalize(pStr)); @@ -453,7 +455,7 @@ objDeserializeEmbedStr(cstr_t **ppStr, strm_t *pStrm) finalize_it: if(iRet != RS_RET_OK && pStr != NULL) - rsCStrDestruct(&pStr); + cstrDestruct(&pStr); RETiRet; } @@ -508,11 +510,11 @@ static rsRetVal objDeserializeStr(cstr_t **ppCStr, int iLen, strm_t *pStrm) assert(ppCStr != NULL); assert(iLen >= 0); - CHKiRet(rsCStrConstruct(&pCStr)); + CHKiRet(cstrConstruct(&pCStr)); NEXTC; for(i = 0 ; i < iLen ; ++i) { - CHKiRet(rsCStrAppendChar(pCStr, c)); + CHKiRet(cstrAppendChar(pCStr, c)); NEXTC; } CHKiRet(cstrFinalize(pCStr)); @@ -524,7 +526,7 @@ static rsRetVal objDeserializeStr(cstr_t **ppCStr, int iLen, strm_t *pStrm) finalize_it: if(iRet != RS_RET_OK && pCStr != NULL) - rsCStrDestruct(&pCStr); + cstrDestruct(&pCStr); RETiRet; } @@ -617,16 +619,16 @@ static rsRetVal objDeserializeProperty(var_t *pProp, strm_t *pStrm) NEXTC; if(c != COOKIE_PROPLINE) { /* oops, we've read one char that does not belong to use - unget it first */ - CHKiRet(strmUnreadChar(pStrm, c)); + CHKiRet(strm.UnreadChar(pStrm, c)); ABORT_FINALIZE(RS_RET_NO_PROPLINE); } /* get the property name first */ - CHKiRet(rsCStrConstruct(&pProp->pcsName)); + CHKiRet(cstrConstruct(&pProp->pcsName)); NEXTC; while(c != ':') { - CHKiRet(rsCStrAppendChar(pProp->pcsName, c)); + CHKiRet(cstrAppendChar(pProp->pcsName, c)); NEXTC; } CHKiRet(cstrFinalize(pProp->pcsName)); @@ -718,7 +720,7 @@ static rsRetVal objDeserializeTryRecover(strm_t *pStrm) } } - CHKiRet(strmUnreadChar(pStrm, c)); + CHKiRet(strm.UnreadChar(pStrm, c)); finalize_it: dbgprintf("deserializer has possibly been able to re-sync and recover, state %d\n", iRet); @@ -803,7 +805,7 @@ Deserialize(void *ppObj, uchar *pszTypeExpected, strm_t *pStrm, rsRetVal (*fFixu } } while(iRetLocal != RS_RET_OK); - if(rsCStrSzStrCmp(pstrID, pszTypeExpected, strlen((char*)pszTypeExpected))) /* TODO: optimize strlen() - caller shall provide */ + if(rsCStrSzStrCmp(pstrID, pszTypeExpected, ustrlen(pszTypeExpected))) /* TODO: optimize strlen() - caller shall provide */ ABORT_FINALIZE(RS_RET_INVALID_OID); CHKiRet(FindObjInfo(pstrID, &pObjInfo)); @@ -948,13 +950,8 @@ SetName(obj_t *pThis, uchar *pszName) { DEFiRet; - if(pThis->pszName != NULL) - free(pThis->pszName); - - pThis->pszName = (uchar*) strdup((char*) pszName); - - if(pThis->pszName == NULL) - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + free(pThis->pszName); + CHKmalloc(pThis->pszName = ustrdup(pszName)); finalize_it: RETiRet; @@ -1057,7 +1054,7 @@ RegisterObj(uchar *pszObjName, objInfo_t *pInfo) i = 0; while(!bFound && i < OBJ_NUM_IDS && arrObjInfo[i] != NULL) { if( arrObjInfo[i] != NULL - && !strcmp((char*)arrObjInfo[i]->pszID, (char*)pszObjName)) { + && !ustrcmp(arrObjInfo[i]->pszID, pszObjName)) { bFound = 1; break; } @@ -1096,7 +1093,7 @@ UnregisterObj(uchar *pszObjName) i = 0; while(!bFound && i < OBJ_NUM_IDS) { if( arrObjInfo[i] != NULL - && !strcmp((char*)arrObjInfo[i]->pszID, (char*)pszObjName)) { + && !ustrcmp(arrObjInfo[i]->pszID, pszObjName)) { bFound = 1; break; } @@ -1132,6 +1129,7 @@ UseObj(char *srcFile, uchar *pObjName, uchar *pObjFile, interface_t *pIf) /* DEV debug only: dbgprintf("source file %s requests object '%s', ifIsLoaded %d\n", srcFile, pObjName, pIf->ifIsLoaded); */ + d_pthread_mutex_lock(&mutObjGlobalOp); if(pIf->ifIsLoaded == 1) { ABORT_FINALIZE(RS_RET_OK); /* we are already set */ @@ -1172,6 +1170,8 @@ UseObj(char *srcFile, uchar *pObjName, uchar *pObjFile, interface_t *pIf) pIf->ifIsLoaded = 1; /* we are happy */ finalize_it: + d_pthread_mutex_unlock(&mutObjGlobalOp); + if(pStr != NULL) rsCStrDestruct(&pStr); @@ -1193,15 +1193,16 @@ ReleaseObj(char *srcFile, uchar *pObjName, uchar *pObjFile, interface_t *pIf) /* dev debug only dbgprintf("source file %s releasing object '%s', ifIsLoaded %d\n", srcFile, pObjName, pIf->ifIsLoaded); */ + d_pthread_mutex_lock(&mutObjGlobalOp); if(pObjFile == NULL) FINALIZE; /* if it is not a lodable module, we do not need to do anything... */ if(pIf->ifIsLoaded == 0) { - ABORT_FINALIZE(RS_RET_OK); /* we are not loaded - this is perfectly OK... */ + FINALIZE; /* we are not loaded - this is perfectly OK... */ } else if(pIf->ifIsLoaded == 2) { pIf->ifIsLoaded = 0; /* clean up */ - ABORT_FINALIZE(RS_RET_OK); /* we had a load error and can not continue */ + FINALIZE; /* we had a load error and can not/must not continue */ } CHKiRet(rsCStrConstructFromszStr(&pStr, pObjName)); @@ -1213,6 +1214,8 @@ ReleaseObj(char *srcFile, uchar *pObjName, uchar *pObjFile, interface_t *pIf) pIf->ifIsLoaded = 0; /* indicated "no longer valid" */ finalize_it: + d_pthread_mutex_unlock(&mutObjGlobalOp); + if(pStr != NULL) rsCStrDestruct(&pStr); @@ -1278,14 +1281,15 @@ objClassExit(void) { DEFiRet; /* release objects we no longer need */ + objRelease(strm, CORE_COMPONENT); objRelease(var, CORE_COMPONENT); objRelease(module, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); /* TODO: implement the class exits! */ #if 0 - cfsyslineInit(pModInfo); - varClassInit(pModInfo); + cfsyslineExit(pModInfo); + varClassExit(pModInfo); #endif errmsgClassExit(); moduleClassExit(); @@ -1304,8 +1308,9 @@ objClassExit(void) rsRetVal objClassInit(modInfo_t *pModInfo) { - DEFiRet; + pthread_mutexattr_t mutAttr; int i; + DEFiRet; /* first, initialize the object system itself. This must be done * before any other object is created. @@ -1314,17 +1319,27 @@ objClassInit(modInfo_t *pModInfo) arrObjInfo[i] = NULL; } + /* the mutex must be recursive, because objects may call into other + * object identifieres recursively. + */ + pthread_mutexattr_init(&mutAttr); + pthread_mutexattr_settype(&mutAttr, PTHREAD_MUTEX_RECURSIVE); + pthread_mutex_init(&mutObjGlobalOp, &mutAttr); + /* request objects we use */ CHKiRet(objGetObjInterface(&obj)); /* get ourselves ;) */ /* init classes we use (limit to as few as possible!) */ + CHKiRet(apcClassInit(pModInfo)); CHKiRet(errmsgClassInit(pModInfo)); CHKiRet(cfsyslineInit()); CHKiRet(varClassInit(pModInfo)); CHKiRet(moduleClassInit(pModInfo)); + CHKiRet(strmClassInit(pModInfo)); CHKiRet(objUse(var, CORE_COMPONENT)); CHKiRet(objUse(module, CORE_COMPONENT)); CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(strm, CORE_COMPONENT)); finalize_it: RETiRet; diff --git a/runtime/obj.h b/runtime/obj.h index dc04203b..419d29cc 100644 --- a/runtime/obj.h +++ b/runtime/obj.h @@ -68,7 +68,7 @@ #define objSerializePTR(strm, propName, propType) \ CHKiRet(obj.SerializeProp(strm, (uchar*) #propName, PROPTYPE_##propType, (void*) pThis->propName)); #define DEFobjStaticHelpers \ - static objInfo_t *pObjInfoOBJ = NULL; \ + static objInfo_t __attribute__((unused)) *pObjInfoOBJ = NULL; \ DEFobjCurrIf(obj) @@ -77,11 +77,13 @@ /* the next macro MUST be called in Constructors: */ #ifndef NDEBUG /* this means if debug... */ # define objConstructSetObjInfo(pThis) \ - ASSERT(((obj_t*) (pThis))->pObjInfo == NULL); \ ((obj_t*) (pThis))->pObjInfo = pObjInfoOBJ; \ + ((obj_t*) (pThis))->pszName = NULL; \ ((obj_t*) (pThis))->iObjCooCKiE = 0xBADEFEE #else -# define objConstructSetObjInfo(pThis) ((obj_t*) (pThis))->pObjInfo = pObjInfoOBJ +# define objConstructSetObjInfo(pThis) \ + ((obj_t*) (pThis))->pObjInfo = pObjInfoOBJ; \ + ((obj_t*) (pThis))->pszName = NULL #endif #define objDestruct(pThis) (((obj_t*) (pThis))->pObjInfo->objMethods[objMethod_DESTRUCT])(&pThis) #define objSerialize(pThis) (((obj_t*) (pThis))->pObjInfo->objMethods[objMethod_SERIALIZE]) diff --git a/runtime/parser.c b/runtime/parser.c index 7eff0801..36e88ebd 100644 --- a/runtime/parser.c +++ b/runtime/parser.c @@ -114,10 +114,7 @@ static inline rsRetVal uncompressMessage(msg_t *pMsg) "Message ignored.", ret); FINALIZE; /* unconditional exit, nothing left to do... */ } - free(pMsg->pszRawMsg); - pMsg->pszRawMsg = deflateBuf; - pMsg->iLenRawMsg = iLenDefBuf; - deflateBuf = NULL; /* logically "freed" - caller is now responsible */ + MsgSetRawMsg(pMsg, (char*)deflateBuf, iLenDefBuf); } finalize_it: if(deflateBuf != NULL) @@ -165,6 +162,9 @@ sanitizeMessage(msg_t *pMsg) size_t iSrc; size_t iDst; size_t iMaxLine; + size_t maxDest; + bool bUpdatedLen = FALSE; + uchar szSanBuf[32*1024]; /* buffer used for sanitizing a string */ assert(pMsg != NULL); assert(pMsg->iLenRawMsg > 0); @@ -176,9 +176,13 @@ sanitizeMessage(msg_t *pMsg) pszMsg = pMsg->pszRawMsg; lenMsg = pMsg->iLenRawMsg; - /* remove NUL character at end of message (see comment in function header) */ + /* remove NUL character at end of message (see comment in function header) + * Note that we do not need to add a NUL character in this case, because it + * is already present ;) + */ if(pszMsg[lenMsg-1] == '\0') { DBGPRINTF("dropped NUL at very end of message\n"); + bUpdatedLen = TRUE; lenMsg--; } @@ -190,6 +194,8 @@ sanitizeMessage(msg_t *pMsg) if(bDropTrailingLF && pszMsg[lenMsg-1] == '\n') { DBGPRINTF("dropped LF at very end of message (DropTrailingLF is set)\n"); lenMsg--; + pszMsg[lenMsg] = '\0'; + bUpdatedLen = TRUE; } /* it is much quicker to sweep over the message and see if it actually @@ -199,77 +205,56 @@ sanitizeMessage(msg_t *pMsg) */ int bNeedSanitize = 0; for(iSrc = 0 ; iSrc < lenMsg ; iSrc++) { - if(pszMsg[iSrc] < 32) { + if(iscntrl(pszMsg[iSrc])) { if(pszMsg[iSrc] == '\0' || bEscapeCCOnRcv) { bNeedSanitize = 1; break; } } } - if(bNeedSanitize == 0) { - /* what a shame - we do not have a \0 byte... - * TODO: think about adding it or otherwise be able to use it... - */ - uchar *pRaw; - CHKmalloc(pRaw = realloc(pMsg->pszRawMsg, pMsg->iLenRawMsg + 1)); - pRaw[pMsg->iLenRawMsg] = '\0'; - pMsg->pszRawMsg = pRaw; + + if(!bNeedSanitize) { + if(bUpdatedLen == TRUE) + MsgSetRawMsgSize(pMsg, lenMsg); FINALIZE; } /* now copy over the message and sanitize it */ - /* TODO: can we get cheaper memory alloc? {alloca()?}*/ iMaxLine = glbl.GetMaxLine(); - CHKmalloc(pDst = malloc(sizeof(uchar) * (iMaxLine + 1))); + maxDest = lenMsg * 4; /* message can grow at most four-fold */ + if(maxDest > iMaxLine) + maxDest = iMaxLine; /* but not more than the max size! */ + if(maxDest < sizeof(szSanBuf)) + pDst = szSanBuf; + else + CHKmalloc(pDst = malloc(sizeof(uchar) * (iMaxLine + 1))); iSrc = iDst = 0; - while(iSrc < lenMsg && iDst < iMaxLine) { - if(pszMsg[iSrc] == '\0') { /* guard against \0 characters... */ - /* changed to the sequence (somewhat) proposed in - * draft-ietf-syslog-protocol-19. rgerhards, 2006-11-30 + while(iSrc < lenMsg && iDst < maxDest - 3) { /* leave some space if last char must be escaped */ + if(iscntrl((int) pszMsg[iSrc])) { + /* note: \0 must always be escaped, the rest of the code currently + * can not handle it! -- rgerhards, 2009-08-26 */ - if(iDst + 3 < iMaxLine) { /* do we have space? */ - pDst[iDst++] = cCCEscapeChar; - pDst[iDst++] = '0'; - pDst[iDst++] = '0'; - pDst[iDst++] = '0'; - } /* if we do not have space, we simply ignore the '\0'... */ - /* log an error? Very questionable... rgerhards, 2006-11-30 */ - /* decided: we do not log an error, it won't help... rger, 2007-06-21 */ - } else if(bEscapeCCOnRcv && iscntrl((int) pszMsg[iSrc])) { + if(pszMsg[iSrc] == '\0' || bEscapeCCOnRcv) { /* we are configured to escape control characters. Please note * that this most probably break non-western character sets like * Japanese, Korean or Chinese. rgerhards, 2007-07-17 - * Note: sysklogd logs octal values only for DEL and CCs above 127. - * For others, it logs ^n where n is the control char converted to an - * alphabet character. We like consistency and thus escape it to octal - * in all cases. If someone complains, we may change the mode. At least - * we known now what's going on. - * rgerhards, 2007-07-17 */ - if(iDst + 3 < iMaxLine) { /* do we have space? */ - pDst[iDst++] = cCCEscapeChar; - pDst[iDst++] = '0' + ((pszMsg[iSrc] & 0300) >> 6); - pDst[iDst++] = '0' + ((pszMsg[iSrc] & 0070) >> 3); - pDst[iDst++] = '0' + ((pszMsg[iSrc] & 0007)); - } /* again, if we do not have space, we ignore the char - see comment at '\0' */ + pDst[iDst++] = cCCEscapeChar; + pDst[iDst++] = '0' + ((pszMsg[iSrc] & 0300) >> 6); + pDst[iDst++] = '0' + ((pszMsg[iSrc] & 0070) >> 3); + pDst[iDst++] = '0' + ((pszMsg[iSrc] & 0007)); + } } else { pDst[iDst++] = pszMsg[iSrc]; } ++iSrc; } - pDst[iDst] = '\0'; /* space *is* reserved for this! */ - - /* we have a sanitized string. Let's save it now */ - free(pMsg->pszRawMsg); - if((pMsg->pszRawMsg = malloc((iDst+1) * sizeof(uchar))) == NULL) { - /* when we get no new buffer, we use what we already have ;) */ - pMsg->pszRawMsg = pDst; - } else { - /* trim buffer */ - memcpy(pMsg->pszRawMsg, pDst, iDst+1); - free(pDst); /* too big! */ - pMsg->iLenRawMsg = iDst; - } + pDst[iDst] = '\0'; + + MsgSetRawMsg(pMsg, (char*)pDst, iDst); /* save sanitized string */ + + if(pDst != szSanBuf) + free(pDst); finalize_it: RETiRet; @@ -295,7 +280,7 @@ rsRetVal parseMsg(msg_t *pMsg) CHKiRet(sanitizeMessage(pMsg)); /* we needed to sanitize first, because we otherwise do not have a C-string we can print... */ - DBGPRINTF("msg parser: flags %x, from '%s', msg '%s'\n", pMsg->msgFlags, pMsg->pszRcvFrom, pMsg->pszRawMsg); + DBGPRINTF("msg parser: flags %x, from '%s', msg '%s'\n", pMsg->msgFlags, getRcvFrom(pMsg), pMsg->pszRawMsg); /* pull PRI */ lenMsg = pMsg->iLenRawMsg; @@ -309,11 +294,8 @@ rsRetVal parseMsg(msg_t *pMsg) */ pri = 0; while(--lenMsg > 0 && isdigit((int) *++msg)) { - pMsg->bufPRI[iPriText++ % 4] = *msg; /* mod 4 to guard against malformed messages! */ pri = 10 * pri + (*msg - '0'); } - pMsg->bufPRI[iPriText % 4] = '\0'; - pMsg->iLenPRI = iPriText % 4; if(*msg == '>') ++msg; if(pri & ~(LOG_FACMASK|LOG_PRIMASK)) @@ -323,9 +305,6 @@ rsRetVal parseMsg(msg_t *pMsg) pMsg->iSeverity = LOG_PRI(pri); MsgSetAfterPRIOffs(pMsg, msg - pMsg->pszRawMsg); - if(pMsg->bParseHOSTNAME == 0) - MsgSetHOSTNAME(pMsg, pMsg->pszRcvFrom); - /* rger 2005-11-24 (happy thanksgiving!): we now need to check if we have * a traditional syslog message or one formatted according to syslog-protocol. * We need to apply different parsers depending on that. We use the @@ -349,7 +328,6 @@ rsRetVal parseMsg(msg_t *pMsg) /* finalize message object */ pMsg->msgFlags &= ~NEEDS_PARSING; /* this message is now parsed */ - MsgPrepareEnqueue(pMsg); /* "historical" name - prepare for multi-threading */ finalize_it: RETiRet; diff --git a/runtime/prop.c b/runtime/prop.c new file mode 100644 index 00000000..d188b2ed --- /dev/null +++ b/runtime/prop.c @@ -0,0 +1,247 @@ +/* prop.c - rsyslog's prop object + * + * This object is meant to support message properties that are stored + * seperately from the message. The main intent is to support properties + * that are "constant" during a period of time, so that many messages may + * contain a reference to the same property. It is important, though, that + * properties are destroyed when they are no longer needed. + * + * Please note that this is a performance-critical part of the software and + * as such we may use some methods in here which do not look elegant, but + * which are fast... + * + * Module begun 2009-06-17 by Rainer Gerhards + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * The rsyslog runtime library is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * The rsyslog runtime library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with the rsyslog runtime library. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. + */ + +#include "config.h" +#include <stdlib.h> +#include <assert.h> +#include <string.h> + +#include "rsyslog.h" +#include "obj.h" +#include "obj-types.h" +#include "unicode-helper.h" +#include "atomic.h" +#include "prop.h" + +/* static data */ +DEFobjStaticHelpers + + +/* Standard-Constructor + */ +BEGINobjConstruct(prop) /* be sure to specify the object type also in END macro! */ + pThis->iRefCount = 1; +ENDobjConstruct(prop) + + +/* destructor for the prop object */ +BEGINobjDestruct(prop) /* be sure to specify the object type also in END and CODESTART macros! */ + int currRefCount; +CODESTARTobjDestruct(prop) + currRefCount = ATOMIC_DEC_AND_FETCH(pThis->iRefCount); + if(currRefCount == 0) { + /* (only) in this case we need to actually destruct the object */ + if(pThis->len >= CONF_PROP_BUFSIZE) + free(pThis->szVal.psz); + } else { + pThis = NULL; /* tell framework NOT to destructing the object! */ + } +ENDobjDestruct(prop) + +/* set string, we make our own private copy! This MUST only be called BEFORE + * ConstructFinalize()! + */ +static rsRetVal SetString(prop_t *pThis, uchar *psz, int len) +{ + DEFiRet; + ISOBJ_TYPE_assert(pThis, prop); + if(pThis->len >= CONF_PROP_BUFSIZE) + free(pThis->szVal.psz); + pThis->len = len; + if(len < CONF_PROP_BUFSIZE) { + memcpy(pThis->szVal.sz, psz, len + 1); + } else { + CHKmalloc(pThis->szVal.psz = malloc(len + 1)); + memcpy(pThis->szVal.psz, psz, len + 1); + } + +finalize_it: + RETiRet; +} + + +/* get string length */ +static int GetStringLen(prop_t *pThis) +{ + return pThis->len; +} + + +/* get string */ +static rsRetVal GetString(prop_t *pThis, uchar **ppsz, int *plen) +{ + BEGINfunc + ISOBJ_TYPE_assert(pThis, prop); + if(pThis->len < CONF_PROP_BUFSIZE) { + *ppsz = pThis->szVal.sz; + } else { + *ppsz = pThis->szVal.psz; + } + *plen = pThis->len; + ENDfunc + return RS_RET_OK; +} + + +/* ConstructionFinalizer + * rgerhards, 2008-01-09 + */ +static rsRetVal +propConstructFinalize(prop_t __attribute__((unused)) *pThis) +{ + DEFiRet; + ISOBJ_TYPE_assert(pThis, prop); + RETiRet; +} + + +/* add a new reference. It is VERY IMPORTANT to call this function whenever + * the property is handed over to some entitiy that later call Destruct() on it. + */ +static rsRetVal AddRef(prop_t *pThis) +{ + ATOMIC_INC(pThis->iRefCount); + return RS_RET_OK; +} + + +/* this is a "do it all in one shot" function that creates a new property, + * assigns the provided string to it and finalizes the property. Among the + * convenience, it is alos (very, very) slightly faster. + * rgerhards, 2009-07-01 + */ +static rsRetVal CreateStringProp(prop_t **ppThis, uchar* psz, int len) +{ + DEFiRet; + propConstruct(ppThis); + SetString(*ppThis, psz, len); + propConstructFinalize(*ppThis); + RETiRet; +} + +/* another one-stop function, quite useful: it takes a property pointer and + * a string. If the string is already contained in the property, nothing happens. + * If the string is different (or the pointer NULL), the current property + * is destructed and a new one created. This can be used to get a specific + * name in those cases where there is a good chance that the property + * immediatly previously processed already contained the value we need - in + * which case we save us all the creation overhead by just reusing the already + * existing property). + * rgerhards, 2009-07-01 + */ +rsRetVal CreateOrReuseStringProp(prop_t **ppThis, uchar *psz, int len) +{ + uchar *pszPrev; + int lenPrev; + DEFiRet; + assert(ppThis != NULL); + + if(*ppThis == NULL) { + /* we need to create a property */ + CHKiRet(CreateStringProp(ppThis, psz, len)); + } else { + /* already exists, check if we can re-use it */ + GetString(*ppThis, &pszPrev, &lenPrev); + if(len != lenPrev || ustrcmp(psz, pszPrev)) { + /* different, need to discard old & create new one */ + propDestruct(ppThis); + CHKiRet(CreateStringProp(ppThis, psz, len)); + } /* else we can re-use the existing one! */ + } + +finalize_it: + RETiRet; +} + + +/* debugprint for the prop object */ +BEGINobjDebugPrint(prop) /* be sure to specify the object type also in END and CODESTART macros! */ +CODESTARTobjDebugPrint(prop) + dbgprintf("prop object %p - no further debug info implemented\n", pThis); +ENDobjDebugPrint(prop) + + +/* queryInterface function + * rgerhards, 2008-02-21 + */ +BEGINobjQueryInterface(prop) +CODESTARTobjQueryInterface(prop) + if(pIf->ifVersion != propCURR_IF_VERSION) { /* check for current version, increment on each change */ + ABORT_FINALIZE(RS_RET_INTERFACE_NOT_SUPPORTED); + } + + /* ok, we have the right interface, so let's fill it + * Please note that we may also do some backwards-compatibility + * work here (if we can support an older interface version - that, + * of course, also affects the "if" above). + */ + pIf->Construct = propConstruct; + pIf->ConstructFinalize = propConstructFinalize; + pIf->Destruct = propDestruct; + pIf->DebugPrint = propDebugPrint; + pIf->SetString = SetString; + pIf->GetString = GetString; + pIf->GetStringLen = GetStringLen; + pIf->AddRef = AddRef; + pIf->CreateStringProp = CreateStringProp; + pIf->CreateOrReuseStringProp = CreateOrReuseStringProp; + +finalize_it: +ENDobjQueryInterface(prop) + + +/* Exit the prop class. + * rgerhards, 2009-04-06 + */ +BEGINObjClassExit(prop, OBJ_IS_CORE_MODULE) /* class, version */ +// objRelease(errmsg, CORE_COMPONENT); +ENDObjClassExit(prop) + + +/* Initialize the prop class. Must be called as the very first method + * before anything else is called inside this class. + * rgerhards, 2008-02-19 + */ +BEGINObjClassInit(prop, 1, OBJ_IS_CORE_MODULE) /* class, version */ + /* request objects we use */ +// CHKiRet(objUse(errmsg, CORE_COMPONENT)); + + /* set our own handlers */ + OBJSetMethodHandler(objMethod_DEBUGPRINT, propDebugPrint); + OBJSetMethodHandler(objMethod_CONSTRUCTION_FINALIZER, propConstructFinalize); +ENDObjClassInit(prop) + +/* vi:set ai: + */ diff --git a/runtime/prop.h b/runtime/prop.h new file mode 100644 index 00000000..e3519664 --- /dev/null +++ b/runtime/prop.h @@ -0,0 +1,58 @@ +/* The prop object. + * + * This implements props within rsyslog. + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * The rsyslog runtime library is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * The rsyslog runtime library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with the rsyslog runtime library. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. + */ +#ifndef INCLUDED_PROP_H +#define INCLUDED_PROP_H + +/* the prop object */ +struct prop_s { + BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ + int iRefCount; /* reference counter */ + union { + uchar *psz; /* stored string */ + uchar sz[CONF_PROP_BUFSIZE]; + } szVal; + int len; /* we use int intentionally, otherwise we may get some troubles... */ +}; + +/* interfaces */ +BEGINinterface(prop) /* name must also be changed in ENDinterface macro! */ + INTERFACEObjDebugPrint(prop); + rsRetVal (*Construct)(prop_t **ppThis); + rsRetVal (*ConstructFinalize)(prop_t *pThis); + rsRetVal (*Destruct)(prop_t **ppThis); + rsRetVal (*SetString)(prop_t *pThis, uchar* psz, int len); + rsRetVal (*GetString)(prop_t *pThis, uchar** ppsz, int *plen); + int (*GetStringLen)(prop_t *pThis); + rsRetVal (*AddRef)(prop_t *pThis); + rsRetVal (*CreateStringProp)(prop_t **ppThis, uchar* psz, int len); + rsRetVal (*CreateOrReuseStringProp)(prop_t **ppThis, uchar *psz, int len); +ENDinterface(prop) +#define propCURR_IF_VERSION 1 /* increment whenever you change the interface structure! */ + + +/* prototypes */ +PROTOTYPEObj(prop); + +#endif /* #ifndef INCLUDED_PROP_H */ diff --git a/runtime/queue.c b/runtime/queue.c index 4e017e84..9d7a9058 100644 --- a/runtime/queue.c +++ b/runtime/queue.c @@ -49,6 +49,7 @@ #include "obj.h" #include "wtp.h" #include "wti.h" +#include "msg.h" #include "atomic.h" #ifdef OS_SOLARIS @@ -59,9 +60,10 @@ /* static data */ DEFobjStaticHelpers DEFobjCurrIf(glbl) +DEFobjCurrIf(strm) /* forward-definitions */ -rsRetVal qqueueChkPersist(qqueue_t *pThis); +static rsRetVal qqueueChkPersist(qqueue_t *pThis); static rsRetVal qqueueSetEnqOnly(qqueue_t *pThis, int bEnqOnly, int bLockMutex); static rsRetVal qqueueRateLimiter(qqueue_t *pThis); static int qqueueChkStopWrkrDA(qqueue_t *pThis); @@ -108,7 +110,7 @@ static inline void queueDrain(qqueue_t *pThis) ASSERT(pThis != NULL); /* iQueueSize is not decremented by qDel(), so we need to do it ourselves */ - while(pThis->iQueueSize-- > 0) { + while(ATOMIC_DEC_AND_FETCH(pThis->iQueueSize) > 0) { pThis->qDel(pThis, &pUsr); if(pUsr != NULL) { objDestruct(pUsr); @@ -293,6 +295,7 @@ qqueueStartDA(qqueue_t *pThis) CHKiRet(qqueueSetMaxFileSize(pThis->pqDA, pThis->iMaxFileSize)); CHKiRet(qqueueSetFilePrefix(pThis->pqDA, pThis->pszFilePrefix, pThis->lenFilePrefix)); CHKiRet(qqueueSetiPersistUpdCnt(pThis->pqDA, pThis->iPersistUpdCnt)); + CHKiRet(qqueueSetbSyncQueueFiles(pThis->pqDA, pThis->bSyncQueueFiles)); CHKiRet(qqueueSettoActShutdown(pThis->pqDA, pThis->toActShutdown)); CHKiRet(qqueueSettoEnq(pThis->pqDA, pThis->toEnq)); CHKiRet(qqueueSetEnqOnly(pThis->pqDA, pThis->bDAEnqOnly, MUTEX_ALREADY_LOCKED)); @@ -667,7 +670,7 @@ qqueueLoadPersStrmInfoFixup(strm_t *pStrm, qqueue_t __attribute__((unused)) *pTh DEFiRet; ISOBJ_TYPE_assert(pStrm, strm); ISOBJ_TYPE_assert(pThis, qqueue); - CHKiRet(strmSetDir(pStrm, glbl.GetWorkDir(), strlen((char*)glbl.GetWorkDir()))); + CHKiRet(strm.SetDir(pStrm, glbl.GetWorkDir(), strlen((char*)glbl.GetWorkDir()))); finalize_it: RETiRet; } @@ -744,11 +747,11 @@ qqueueTryLoadPersistedInfo(qqueue_t *pThis) /* If we reach this point, we have a .qi file */ - CHKiRet(strmConstruct(&psQIF)); - CHKiRet(strmSettOperationsMode(psQIF, STREAMMODE_READ)); - CHKiRet(strmSetsType(psQIF, STREAMTYPE_FILE_SINGLE)); - CHKiRet(strmSetFName(psQIF, pszQIFNam, lenQIFNam)); - CHKiRet(strmConstructFinalize(psQIF)); + CHKiRet(strm.Construct(&psQIF)); + CHKiRet(strm.SettOperationsMode(psQIF, STREAMMODE_READ)); + CHKiRet(strm.SetsType(psQIF, STREAMTYPE_FILE_SINGLE)); + CHKiRet(strm.SetFName(psQIF, pszQIFNam, lenQIFNam)); + CHKiRet(strm.ConstructFinalize(psQIF)); /* first, we try to read the property bag for ourselfs */ CHKiRet(obj.DeserializePropBag((obj_t*) pThis, psQIF)); @@ -770,8 +773,8 @@ qqueueTryLoadPersistedInfo(qqueue_t *pThis) CHKiRet(obj.Deserialize(&pThis->tVars.disk.pRead, (uchar*) "strm", psQIF, (rsRetVal(*)(obj_t*,void*))qqueueLoadPersStrmInfoFixup, pThis)); - CHKiRet(strmSeekCurrOffs(pThis->tVars.disk.pWrite)); - CHKiRet(strmSeekCurrOffs(pThis->tVars.disk.pRead)); + CHKiRet(strm.SeekCurrOffs(pThis->tVars.disk.pWrite)); + CHKiRet(strm.SeekCurrOffs(pThis->tVars.disk.pRead)); /* OK, we could successfully read the file, so we now can request that it be * deleted when we are done with the persisted information. @@ -780,7 +783,7 @@ qqueueTryLoadPersistedInfo(qqueue_t *pThis) finalize_it: if(psQIF != NULL) - strmDestruct(&psQIF); + strm.Destruct(&psQIF); if(iRet != RS_RET_OK) { dbgoprint((obj_t*) pThis, "error %d reading .qi file - can not read persisted info (if any)\n", @@ -815,24 +818,26 @@ static rsRetVal qConstructDisk(qqueue_t *pThis) if(bRestarted == 1) { ; } else { - CHKiRet(strmConstruct(&pThis->tVars.disk.pWrite)); - CHKiRet(strmSetDir(pThis->tVars.disk.pWrite, glbl.GetWorkDir(), strlen((char*)glbl.GetWorkDir()))); - CHKiRet(strmSetiMaxFiles(pThis->tVars.disk.pWrite, 10000000)); - CHKiRet(strmSettOperationsMode(pThis->tVars.disk.pWrite, STREAMMODE_WRITE)); - CHKiRet(strmSetsType(pThis->tVars.disk.pWrite, STREAMTYPE_FILE_CIRCULAR)); - CHKiRet(strmConstructFinalize(pThis->tVars.disk.pWrite)); + CHKiRet(strm.Construct(&pThis->tVars.disk.pWrite)); + CHKiRet(strm.SetbSync(pThis->tVars.disk.pWrite, pThis->bSyncQueueFiles)); + CHKiRet(strm.SetDir(pThis->tVars.disk.pWrite, glbl.GetWorkDir(), strlen((char*)glbl.GetWorkDir()))); + CHKiRet(strm.SetiMaxFiles(pThis->tVars.disk.pWrite, 10000000)); + CHKiRet(strm.SettOperationsMode(pThis->tVars.disk.pWrite, STREAMMODE_WRITE)); + CHKiRet(strm.SetsType(pThis->tVars.disk.pWrite, STREAMTYPE_FILE_CIRCULAR)); + CHKiRet(strm.ConstructFinalize(pThis->tVars.disk.pWrite)); - CHKiRet(strmConstruct(&pThis->tVars.disk.pRead)); - CHKiRet(strmSetbDeleteOnClose(pThis->tVars.disk.pRead, 1)); - CHKiRet(strmSetDir(pThis->tVars.disk.pRead, glbl.GetWorkDir(), strlen((char*)glbl.GetWorkDir()))); - CHKiRet(strmSetiMaxFiles(pThis->tVars.disk.pRead, 10000000)); - CHKiRet(strmSettOperationsMode(pThis->tVars.disk.pRead, STREAMMODE_READ)); - CHKiRet(strmSetsType(pThis->tVars.disk.pRead, STREAMTYPE_FILE_CIRCULAR)); - CHKiRet(strmConstructFinalize(pThis->tVars.disk.pRead)); + CHKiRet(strm.Construct(&pThis->tVars.disk.pRead)); + CHKiRet(strm.SetbSync(pThis->tVars.disk.pRead, pThis->bSyncQueueFiles)); + CHKiRet(strm.SetbDeleteOnClose(pThis->tVars.disk.pRead, 1)); + CHKiRet(strm.SetDir(pThis->tVars.disk.pRead, glbl.GetWorkDir(), strlen((char*)glbl.GetWorkDir()))); + CHKiRet(strm.SetiMaxFiles(pThis->tVars.disk.pRead, 10000000)); + CHKiRet(strm.SettOperationsMode(pThis->tVars.disk.pRead, STREAMMODE_READ)); + CHKiRet(strm.SetsType(pThis->tVars.disk.pRead, STREAMTYPE_FILE_CIRCULAR)); + CHKiRet(strm.ConstructFinalize(pThis->tVars.disk.pRead)); - CHKiRet(strmSetFName(pThis->tVars.disk.pWrite, pThis->pszFilePrefix, pThis->lenFilePrefix)); - CHKiRet(strmSetFName(pThis->tVars.disk.pRead, pThis->pszFilePrefix, pThis->lenFilePrefix)); + CHKiRet(strm.SetFName(pThis->tVars.disk.pWrite, pThis->pszFilePrefix, pThis->lenFilePrefix)); + CHKiRet(strm.SetFName(pThis->tVars.disk.pRead, pThis->pszFilePrefix, pThis->lenFilePrefix)); } /* now we set (and overwrite in case of a persisted restart) some parameters which @@ -840,8 +845,8 @@ static rsRetVal qConstructDisk(qqueue_t *pThis) * for example file name generation must not be changed as that would break the * ability to read existing queue files. -- rgerhards, 2008-01-12 */ - CHKiRet(strmSetiMaxFileSize(pThis->tVars.disk.pWrite, pThis->iMaxFileSize)); - CHKiRet(strmSetiMaxFileSize(pThis->tVars.disk.pRead, pThis->iMaxFileSize)); + CHKiRet(strm.SetiMaxFileSize(pThis->tVars.disk.pWrite, pThis->iMaxFileSize)); + CHKiRet(strm.SetiMaxFileSize(pThis->tVars.disk.pRead, pThis->iMaxFileSize)); finalize_it: RETiRet; @@ -853,9 +858,11 @@ static rsRetVal qDestructDisk(qqueue_t *pThis) DEFiRet; ASSERT(pThis != NULL); - - strmDestruct(&pThis->tVars.disk.pWrite); - strmDestruct(&pThis->tVars.disk.pRead); + + if (pThis->tVars.disk.pWrite != NULL) + strm.Destruct(&pThis->tVars.disk.pWrite); + if (pThis->tVars.disk.pRead != NULL) + strm.Destruct(&pThis->tVars.disk.pRead); RETiRet; } @@ -867,10 +874,10 @@ static rsRetVal qAddDisk(qqueue_t *pThis, void* pUsr) ASSERT(pThis != NULL); - CHKiRet(strmSetWCntr(pThis->tVars.disk.pWrite, &nWriteCount)); + CHKiRet(strm.SetWCntr(pThis->tVars.disk.pWrite, &nWriteCount)); CHKiRet((objSerialize(pUsr))(pUsr, pThis->tVars.disk.pWrite)); - CHKiRet(strmFlush(pThis->tVars.disk.pWrite)); - CHKiRet(strmSetWCntr(pThis->tVars.disk.pWrite, NULL)); /* no more counting for now... */ + CHKiRet(strm.Flush(pThis->tVars.disk.pWrite)); + CHKiRet(strm.SetWCntr(pThis->tVars.disk.pWrite, NULL)); /* no more counting for now... */ pThis->tVars.disk.sizeOnDisk += nWriteCount; @@ -894,9 +901,9 @@ static rsRetVal qDelDisk(qqueue_t *pThis, void **ppUsr) int64 offsIn; int64 offsOut; - CHKiRet(strmGetCurrOffset(pThis->tVars.disk.pRead, &offsIn)); + CHKiRet(strm.GetCurrOffset(pThis->tVars.disk.pRead, &offsIn)); CHKiRet(obj.Deserialize(ppUsr, (uchar*) "msg", pThis->tVars.disk.pRead, NULL, NULL)); - CHKiRet(strmGetCurrOffset(pThis->tVars.disk.pRead, &offsOut)); + CHKiRet(strm.GetCurrOffset(pThis->tVars.disk.pRead, &offsOut)); /* This time it is a bit tricky: we free disk space only upon file deletion. So we need * to keep track of what we have read until we get an out-offset that is lower than the @@ -1209,7 +1216,7 @@ static rsRetVal qqueueShutdownWorkers(qqueue_t *pThis) /* we need to re-aquire the mutex for the next check in this case! */ BEGIN_MTX_PROTECTED_OPERATIONS(pThis->mut, LOCK_MUTEX); /* some workers may be running in parallel! */ } - if(pThis->bIsDA && wtpGetCurNumWrkr(pThis->pWtpDA, LOCK_MUTEX) > 0) { + if(pThis->bRunsDA && wtpGetCurNumWrkr(pThis->pWtpDA, LOCK_MUTEX) > 0) { /* and now the same for the DA queue */ END_MTX_PROTECTED_OPERATIONS(pThis->mut); dbgoprint((obj_t*) pThis, "trying immediate shutdown of DA workers\n"); @@ -1290,7 +1297,6 @@ rsRetVal qqueueConstruct(qqueue_t **ppThis, queueType_t qType, int iWorkerThread /* we have an object, so let's fill the properties */ objConstructSetObjInfo(pThis); - pThis->bOptimizeUniProc = glbl.GetOptimizeUniProc(); if((pThis->pszSpoolDir = (uchar*) strdup((char*)glbl.GetWorkDir())) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); @@ -1448,11 +1454,11 @@ qqueueDequeueConsumable(qqueue_t *pThis, wti_t *pWti, int iCancelStateSave) * we have someone waiting for the condition (or only when we hit the watermark right * on the nail [exact value]) -- rgerhards, 2008-03-14 */ - if(iQueueSize < pThis->iFullDlyMrk) { + if(iQueueSize < pThis->iFullDlyMrk / 2) { pthread_cond_broadcast(&pThis->belowFullDlyWtrMrk); } - if(iQueueSize < pThis->iLightDlyMrk) { + if(iQueueSize < pThis->iLightDlyMrk / 2) { pthread_cond_broadcast(&pThis->belowLightDlyWtrMrk); } @@ -1662,7 +1668,7 @@ qqueueChkStopWrkrDA(qqueue_t *pThis) if(pThis->bEnqOnly) { bStopWrkr = 1; } else { - if(pThis->bRunsDA) { + if(pThis->bRunsDA == 2) { ASSERT(pThis->pqDA != NULL); if( pThis->pqDA->bEnqOnly && pThis->pqDA->sizeOnDiskMax > 0 @@ -1917,16 +1923,17 @@ static rsRetVal qqueuePersist(qqueue_t *pThis, int bIsCheckpoint) pThis->bNeedDelQIF = 0; } /* indicate spool file needs to be deleted */ - CHKiRet(strmSetbDeleteOnClose(pThis->tVars.disk.pRead, 1)); + if (pThis->tVars.disk.pRead != NULL) + CHKiRet(strm.SetbDeleteOnClose(pThis->tVars.disk.pRead, 1)); FINALIZE; /* nothing left to do, so be happy */ } - CHKiRet(strmConstruct(&psQIF)); - CHKiRet(strmSettOperationsMode(psQIF, STREAMMODE_WRITE)); - CHKiRet(strmSetiAddtlOpenFlags(psQIF, O_TRUNC)); - CHKiRet(strmSetsType(psQIF, STREAMTYPE_FILE_SINGLE)); - CHKiRet(strmSetFName(psQIF, pszQIFNam, lenQIFNam)); - CHKiRet(strmConstructFinalize(psQIF)); + CHKiRet(strm.Construct(&psQIF)); + CHKiRet(strm.SettOperationsMode(psQIF, STREAMMODE_WRITE_TRUNC)); + CHKiRet(strm.SetbSync(psQIF, pThis->bSyncQueueFiles)); + CHKiRet(strm.SetsType(psQIF, STREAMTYPE_FILE_SINGLE)); + CHKiRet(strm.SetFName(psQIF, pszQIFNam, lenQIFNam)); + CHKiRet(strm.ConstructFinalize(psQIF)); /* first, write the property bag for ourselfs * And, surprisingly enough, we currently need to persist only the size of the @@ -1951,14 +1958,16 @@ static rsRetVal qqueuePersist(qqueue_t *pThis, int bIsCheckpoint) } /* now persist the stream info */ - CHKiRet(strmSerialize(pThis->tVars.disk.pWrite, psQIF)); - CHKiRet(strmSerialize(pThis->tVars.disk.pRead, psQIF)); + if (pThis->tVars.disk.pWrite != NULL) + CHKiRet(strm.Serialize(pThis->tVars.disk.pWrite, psQIF)); + if (pThis->tVars.disk.pRead != NULL) + CHKiRet(strm.Serialize(pThis->tVars.disk.pRead, psQIF)); /* tell the input file object that it must not delete the file on close if the queue * is non-empty - but only if we are not during a simple checkpoint */ - if(bIsCheckpoint != QUEUE_CHECKPOINT) { - CHKiRet(strmSetbDeleteOnClose(pThis->tVars.disk.pRead, 0)); + if(bIsCheckpoint != QUEUE_CHECKPOINT && pThis->tVars.disk.pRead != NULL) { + CHKiRet(strm.SetbDeleteOnClose(pThis->tVars.disk.pRead, 0)); } /* we have persisted the queue object. So whenever it comes to an empty queue, @@ -1968,7 +1977,7 @@ static rsRetVal qqueuePersist(qqueue_t *pThis, int bIsCheckpoint) finalize_it: if(psQIF != NULL) - strmDestruct(&psQIF); + strm.Destruct(&psQIF); RETiRet; } @@ -1979,10 +1988,8 @@ finalize_it: * abide to our regular call interface)... * rgerhards, 2008-01-13 */ -rsRetVal qqueueChkPersist(qqueue_t *pThis) +static rsRetVal qqueueChkPersist(qqueue_t *pThis) { - DEFiRet; - ISOBJ_TYPE_assert(pThis, qqueue); if(pThis->iPersistUpdCnt && ++pThis->iUpdsSincePersist >= pThis->iPersistUpdCnt) { @@ -1990,7 +1997,7 @@ rsRetVal qqueueChkPersist(qqueue_t *pThis) pThis->iUpdsSincePersist = 0; } - RETiRet; + return RS_RET_OK; } @@ -2213,13 +2220,124 @@ finalize_it: d_pthread_mutex_unlock(pThis->mut); pthread_setcancelstate(iCancelStateSave, NULL); dbgoprint((obj_t*) pThis, "EnqueueMsg advised worker start\n"); - /* the following pthread_yield is experimental, but brought us performance - * benefit. For details, please see http://kb.monitorware.com/post14216.html#p14216 - * rgerhards, 2008-10-09 - * but this is only true for uniprocessors, so we guard it with an optimize flag -- rgerhards, 2008-10-22 - */ - if(pThis->bOptimizeUniProc) - pthread_yield(); + } + + RETiRet; +} + + +/* enqueue a single data object. This currently is a helper to qqueueMultiEnqObj. + * Note that the queue mutex MUST already be locked when this function is called. + * rgerhards, 2009-06-16 + */ +static inline rsRetVal +doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) +{ + DEFiRet; + struct timespec t; + + /* first check if we need to discard this message (which will cause CHKiRet() to exit) + */ + CHKiRet(qqueueChkDiscardMsg(pThis, pThis->iQueueSize, pThis->bRunsDA, pUsr)); + + /* then check if we need to add an assistance disk queue */ + if(pThis->bIsDA) + CHKiRet(qqueueChkStrtDA(pThis)); + + /* handle flow control + * There are two different flow control mechanisms: basic and advanced flow control. + * Basic flow control has always been implemented and protects the queue structures + * in that it makes sure no more data is enqueued than the queue is configured to + * support. Enhanced flow control is being added today. There are some sources which + * can easily be stopped, e.g. a file reader. This is the case because it is unlikely + * that blocking those sources will have negative effects (after all, the file is + * continued to be written). Other sources can somewhat be blocked (e.g. the kernel + * log reader or the local log stream reader): in general, nothing is lost if messages + * from these sources are not picked up immediately. HOWEVER, they can not block for + * an extended period of time, as this either causes message loss or - even worse - some + * other bad effects (e.g. unresponsive system in respect to the main system log socket). + * Finally, there are some (few) sources which can not be blocked at all. UDP syslog is + * a prime example. If a UDP message is not received, it is simply lost. So we can't + * do anything against UDP sockets that come in too fast. The core idea of advanced + * flow control is that we take into account the different natures of the sources and + * select flow control mechanisms that fit these needs. This also means, in the end + * result, that non-blockable sources like UDP syslog receive priority in the system. + * It's a side effect, but a good one ;) -- rgerhards, 2008-03-14 + */ + if(flowCtlType == eFLOWCTL_FULL_DELAY) { + while(pThis->iQueueSize >= pThis->iFullDlyMrk) { + dbgoprint((obj_t*) pThis, "enqueueMsg: FullDelay mark reached for full delayable message - blocking.\n"); + pthread_cond_wait(&pThis->belowFullDlyWtrMrk, pThis->mut); /* TODO error check? But what do then? */ + } + } else if(flowCtlType == eFLOWCTL_LIGHT_DELAY) { + if(pThis->iQueueSize >= pThis->iLightDlyMrk) { + dbgoprint((obj_t*) pThis, "enqueueMsg: LightDelay mark reached for light delayable message - blocking a bit.\n"); + timeoutComp(&t, 1000); /* 1000 millisconds = 1 second TODO: make configurable */ + pthread_cond_timedwait(&pThis->belowLightDlyWtrMrk, pThis->mut, &t); /* TODO error check? But what do then? */ + } + } + + /* from our regular flow control settings, we are now ready to enqueue the object. + * However, we now need to do a check if the queue permits to add more data. If that + * is not the case, basic flow control enters the field, which means we wait for + * the queue to become ready or drop the new message. -- rgerhards, 2008-03-14 + */ + while( (pThis->iMaxQueueSize > 0 && pThis->iQueueSize >= pThis->iMaxQueueSize) + || (pThis->qType == QUEUETYPE_DISK && pThis->sizeOnDiskMax != 0 + && pThis->tVars.disk.sizeOnDisk > pThis->sizeOnDiskMax)) { + dbgoprint((obj_t*) pThis, "enqueueMsg: queue FULL - waiting to drain.\n"); + timeoutComp(&t, pThis->toEnq); + if(pthread_cond_timedwait(&pThis->notFull, pThis->mut, &t) != 0) { + dbgoprint((obj_t*) pThis, "enqueueMsg: cond timeout, dropping message!\n"); + objDestruct(pUsr); + ABORT_FINALIZE(RS_RET_QUEUE_FULL); + } + } + + /* and finally enqueue the message */ + CHKiRet(qqueueAdd(pThis, pUsr)); + qqueueChkPersist(pThis); // TODO: optimize, do in outer function! (but we need parts from v5?) + +finalize_it: + RETiRet; +} + +/* enqueue multiple user data elements at once. The aim is to provide a faster interface + * for object submission. Uses the multi_submit_t helper object. + * Please note that this function is not cancel-safe and consequently + * sets the calling thread's cancelibility state to PTHREAD_CANCEL_DISABLE + * during its execution. If that is not done, race conditions occur if the + * thread is canceled (most important use case is input module termination). + * rgerhards, 2009-06-16 + */ +rsRetVal +qqueueMultiEnqObj(qqueue_t *pThis, multi_submit_t *pMultiSub) +{ + int iCancelStateSave; + int i; + DEFiRet; + + ISOBJ_TYPE_assert(pThis, qqueue); + assert(pMultiSub != NULL); + + if(pThis->qType != QUEUETYPE_DIRECT) { + pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &iCancelStateSave); + d_pthread_mutex_lock(pThis->mut); + } + + for(i = 0 ; i < pMultiSub->nElem ; ++i) { +dbgprintf("queueMultiEnq: %d\n", i); + CHKiRet(doEnqSingleObj(pThis, pMultiSub->ppMsgs[i]->flowCtlType, (void*)pMultiSub->ppMsgs[i])); + } + +finalize_it: + if(pThis->qType != QUEUETYPE_DIRECT) { + /* make sure at least one worker is running. */ + qqueueAdviseMaxWorkers(pThis); + /* and release the mutex */ + d_pthread_mutex_unlock(pThis->mut); + pthread_setcancelstate(iCancelStateSave, NULL); + dbgoprint((obj_t*) pThis, "MultiEnqObj advised worker start\n"); } RETiRet; @@ -2279,6 +2397,7 @@ finalize_it: /* some simple object access methods */ +DEFpropSetMeth(qqueue, bSyncQueueFiles, int) DEFpropSetMeth(qqueue, iPersistUpdCnt, int) DEFpropSetMeth(qqueue, iDeqtWinFromHr, int) DEFpropSetMeth(qqueue, iDeqtWinToHr, int) @@ -2340,6 +2459,7 @@ rsRetVal qqueueQueryInterface(void) { return RS_RET_NOT_IMPLEMENTED; } BEGINObjClassInit(qqueue, 1, OBJ_IS_CORE_MODULE) /* request objects we use */ CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(strm, CORE_COMPONENT)); /* now set our own handlers */ OBJSetMethodHandler(objMethod_SETPROPERTY, qqueueSetProperty); diff --git a/runtime/queue.h b/runtime/queue.h index a267862d..1d82d8d9 100644 --- a/runtime/queue.h +++ b/runtime/queue.h @@ -58,11 +58,10 @@ typedef struct qWrkThrd_s { typedef struct queue_s { BEGINobjInstance; queueType_t qType; - int bOptimizeUniProc; /* cache for the equally-named global setting, pulled at time of queue creation */ - int bEnqOnly; /* does queue run in enqueue-only mode (1) or not (0)? */ - int bSaveOnShutdown;/* persists everthing on shutdown (if DA!)? 1-yes, 0-no */ - int bQueueStarted; /* has queueStart() been called on this queue? 1-yes, 0-no */ - int bQueueInDestruction;/* 1 if queue is in destruction process, 0 otherwise */ + bool bEnqOnly; /* does queue run in enqueue-only mode (1) or not (0)? */ + bool bSaveOnShutdown;/* persists everthing on shutdown (if DA!)? 1-yes, 0-no */ + bool bQueueStarted; /* has queueStart() been called on this queue? 1-yes, 0-no */ + bool bQueueInDestruction;/* 1 if queue is in destruction process, 0 otherwise */ int iQueueSize; /* Current number of elements in the queue */ int iMaxQueueSize; /* how large can the queue grow? */ int iNumWorkerThreads;/* number of worker threads to use */ @@ -73,13 +72,14 @@ typedef struct queue_s { void *pUsr; /* a global, user-supplied pointer. Is passed back to consumer. */ int iUpdsSincePersist;/* nbr of queue updates since the last persist call */ int iPersistUpdCnt; /* persits queue info after this nbr of updates - 0 -> persist only on shutdown */ + bool bSyncQueueFiles;/* if working with files, sync them after each write? */ int iHighWtrMrk; /* high water mark for disk-assisted memory queues */ int iLowWtrMrk; /* low water mark for disk-assisted memory queues */ int iDiscardMrk; /* if the queue is above this mark, low-severity messages are discarded */ int iFullDlyMrk; /* if the queue is above this mark, FULL_DELAYable message are put on hold */ int iLightDlyMrk; /* if the queue is above this mark, LIGHT_DELAYable message are put on hold */ int iDiscardSeverity;/* messages of this severity above are discarded on too-full queue */ - int bNeedDelQIF; /* does the QIF file need to be deleted when queue becomes empty? */ + bool bNeedDelQIF; /* does the QIF file need to be deleted when queue becomes empty? */ int toQShutdown; /* timeout for regular queue shutdown in ms */ int toActShutdown; /* timeout for long-running action shutdown in ms */ int toWrkShutdown; /* timeout for idle workers in ms, -1 means indefinite (0 is immediate) */ @@ -178,6 +178,7 @@ typedef struct queue_s { /* prototypes */ rsRetVal qqueueDestruct(qqueue_t **ppThis); +rsRetVal qqueueMultiEnqObj(qqueue_t *pThis, multi_submit_t *pMultiSub); rsRetVal qqueueEnqObj(qqueue_t *pThis, flowControl_t flwCtlType, void *pUsr); rsRetVal qqueueStart(qqueue_t *pThis); rsRetVal qqueueSetMaxFileSize(qqueue_t *pThis, size_t iMaxFileSize); @@ -186,6 +187,7 @@ rsRetVal qqueueConstruct(qqueue_t **ppThis, queueType_t qType, int iWorkerThread int iMaxQueueSize, rsRetVal (*pConsumer)(void*,void*)); PROTOTYPEObjClassInit(qqueue); PROTOTYPEpropSetMeth(qqueue, iPersistUpdCnt, int); +PROTOTYPEpropSetMeth(qqueue, bSyncQueueFiles, int); PROTOTYPEpropSetMeth(qqueue, iDeqtWinFromHr, int); PROTOTYPEpropSetMeth(qqueue, iDeqtWinToHr, int); PROTOTYPEpropSetMeth(qqueue, toQShutdown, long); diff --git a/runtime/rsyslog.c b/runtime/rsyslog.c index 8df100a1..443d0f41 100644 --- a/runtime/rsyslog.c +++ b/runtime/rsyslog.c @@ -77,6 +77,9 @@ #include "conf.h" #include "glbl.h" #include "errmsg.h" +#include "prop.h" +#include "rule.h" +#include "ruleset.h" /* forward definitions */ static rsRetVal dfltErrLogger(int, uchar *errMsg); @@ -144,20 +147,18 @@ rsrtInit(char **ppErrObj, obj_if_t *pObjIF) * class immediately after it is initialized. And, of course, we load those classes * first that we use ourselfs... -- rgerhards, 2008-03-07 */ + if(ppErrObj != NULL) *ppErrObj = "prop"; + CHKiRet(propClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "glbl"; CHKiRet(glblClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "datetime"; CHKiRet(datetimeClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "msg"; CHKiRet(msgClassInit(NULL)); - if(ppErrObj != NULL) *ppErrObj = "str,"; - CHKiRet(strmClassInit(NULL)); - if(ppErrObj != NULL) *ppErrObj = "wti"; - CHKiRet(wtiClassInit(NULL)); - if(ppErrObj != NULL) *ppErrObj = "wtp"; - CHKiRet(wtpClassInit(NULL)); - if(ppErrObj != NULL) *ppErrObj = "queue"; - CHKiRet(qqueueClassInit(NULL)); + if(ppErrObj != NULL) *ppErrObj = "ctok_token"; + CHKiRet(ctok_tokenClassInit(NULL)); + if(ppErrObj != NULL) *ppErrObj = "ctok"; + CHKiRet(ctokClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "vmstk"; CHKiRet(vmstkClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "sysvar"; @@ -168,12 +169,18 @@ rsrtInit(char **ppErrObj, obj_if_t *pObjIF) CHKiRet(vmopClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "vmprg"; CHKiRet(vmprgClassInit(NULL)); - if(ppErrObj != NULL) *ppErrObj = "ctok_token"; - CHKiRet(ctok_tokenClassInit(NULL)); - if(ppErrObj != NULL) *ppErrObj = "ctok"; - CHKiRet(ctokClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "expr"; CHKiRet(exprClassInit(NULL)); + if(ppErrObj != NULL) *ppErrObj = "rule"; + CHKiRet(ruleClassInit(NULL)); + if(ppErrObj != NULL) *ppErrObj = "ruleset"; + CHKiRet(rulesetClassInit(NULL)); + if(ppErrObj != NULL) *ppErrObj = "wti"; + CHKiRet(wtiClassInit(NULL)); + if(ppErrObj != NULL) *ppErrObj = "wtp"; + CHKiRet(wtpClassInit(NULL)); + if(ppErrObj != NULL) *ppErrObj = "queue"; + CHKiRet(qqueueClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "conf"; CHKiRet(confClassInit(NULL)); @@ -206,6 +213,8 @@ rsrtExit(void) /* do actual de-init only if we are the last runtime user */ confClassExit(); glblClassExit(); + rulesetClassExit(); + ruleClassExit(); objClassExit(); /* *THIS* *MUST/SHOULD?* always be the first class initilizer being called (except debug)! */ } diff --git a/runtime/rsyslog.h b/runtime/rsyslog.h index 835fff27..8979893a 100644 --- a/runtime/rsyslog.h +++ b/runtime/rsyslog.h @@ -29,21 +29,23 @@ /* ############################################################# * * # Config Settings # * * ############################################################# */ -#define RS_STRINGBUF_ALLOC_INCREMENT 128 +#define RS_STRINGBUF_ALLOC_INCREMENT 128 +/* MAXSIZE are absolute maxima, while BUFSIZE are just values after which + * processing is more time-intense. The BUFSIZE params currently add their + * value to the fixed size of the message object. + */ +#define CONF_TAG_MAXSIZE 512 /* a value that is deemed far too large for any valid TAG */ +#define CONF_HOSTNAME_MAXSIZE 512 /* a value that is deemed far too large for any valid HOSTNAME */ +#define CONF_RAWMSG_BUFSIZE 101 +#define CONF_TAG_BUFSIZE 32 +#define CONF_HOSTNAME_BUFSIZE 32 +#define CONF_PROP_BUFSIZE 16 /* should be close to sizeof(ptr) or lighly above it */ + /* ############################################################# * * # End Config Settings # * * ############################################################# */ -#ifndef NOLARGEFILE -# undef _LARGEFILE_SOURCE -# undef _LARGEFILE64_SOURCE -# undef _FILE_OFFSET_BITS -# define _LARGEFILE_SOURCE -# define _LARGEFILE64_SOURCE -# define _FILE_OFFSET_BITS 64 -#endif - /* portability: not all platforms have these defines, so we * define them here if they are missing. -- rgerhards, 2008-03-04 */ @@ -62,7 +64,9 @@ typedef unsigned char uchar;/* get rid of the unhandy "unsigned char" */ typedef struct thrdInfo thrdInfo_t; typedef struct obj_s obj_t; -typedef struct filed selector_t;/* TODO: this so far resides in syslogd.c, think about modularization */ +typedef struct ruleset_s ruleset_t; +typedef struct rule_s rule_t; +//typedef struct filed selector_t;/* TODO: this so far resides in syslogd.c, think about modularization */ typedef struct NetAddr netAddr_t; typedef struct netstrms_s netstrms_t; typedef struct netstrm_s netstrm_t; @@ -77,6 +81,7 @@ typedef struct nsdsel_gtls_s nsdsel_gtls_t; typedef obj_t nsd_t; typedef obj_t nsdsel_t; typedef struct msg msg_t; +typedef struct prop_s prop_t; typedef struct interface_s interface_t; typedef struct objInfo_s objInfo_t; typedef enum rsRetVal_ rsRetVal; /**< friendly type for global return value */ @@ -97,6 +102,8 @@ typedef struct strmLstnPortList_s strmLstnPortList_t; // TODO: rename? typedef long long int64; typedef long long unsigned uint64; typedef int64 number_t; /* type to use for numbers - TODO: maybe an autoconf option? */ +typedef char intTiny; /* 0..127! */ +typedef uchar uintTiny; /* 0..255! */ #ifdef __hpux typedef unsigned int u_int32_t; /* TODO: is this correct? */ @@ -114,6 +121,72 @@ typedef enum { eFLOWCTL_FULL_DELAY = 2 /**< delay possible for extended period of time */ } flowControl_t; +/* filter operations */ +typedef enum { + FIOP_NOP = 0, /* do not use - No Operation */ + FIOP_CONTAINS = 1, /* contains string? */ + FIOP_ISEQUAL = 2, /* is (exactly) equal? */ + FIOP_STARTSWITH = 3, /* starts with a string? */ + FIOP_REGEX = 4, /* matches a (BRE) regular expression? */ + FIOP_EREREGEX = 5 /* matches a ERE regular expression? */ +} fiop_t; + + +/* multi-submit support. + * This is done via a simple data structure, which holds the number of elements + * as well as an array of to-be-submitted messages. + * rgerhards, 2009-06-16 + */ +typedef struct multi_submit_s multi_submit_t; +struct multi_submit_s { + short maxElem; /* maximum number of Elements */ + short nElem; /* current number of Elements, points to the next one FREE */ + msg_t **ppMsgs; +}; + + +#ifndef _PATH_CONSOLE +#define _PATH_CONSOLE "/dev/console" +#endif + +/* properties are now encoded as (tiny) integers. I do not use an enum as I would like + * to keep the memory footprint small (and thus cache hits high). + * rgerhards, 2009-06-26 + */ +typedef uintTiny propid_t; +#define PROP_INVALID 0 +#define PROP_MSG 1 +#define PROP_TIMESTAMP 2 +#define PROP_HOSTNAME 3 +#define PROP_SYSLOGTAG 4 +#define PROP_RAWMSG 5 +#define PROP_INPUTNAME 6 +#define PROP_FROMHOST 7 +#define PROP_FROMHOST_IP 8 +#define PROP_PRI 9 +#define PROP_PRI_TEXT 10 +#define PROP_IUT 11 +#define PROP_SYSLOGFACILITY 12 +#define PROP_SYSLOGFACILITY_TEXT 13 +#define PROP_SYSLOGSEVERITY 14 +#define PROP_SYSLOGSEVERITY_TEXT 15 +#define PROP_TIMEGENERATED 16 +#define PROP_PROGRAMNAME 17 +#define PROP_PROTOCOL_VERSION 18 +#define PROP_STRUCTURED_DATA 19 +#define PROP_APP_NAME 20 +#define PROP_PROCID 21 +#define PROP_MSGID 22 +#define PROP_SYS_NOW 150 +#define PROP_SYS_YEAR 151 +#define PROP_SYS_MONTH 152 +#define PROP_SYS_DAY 153 +#define PROP_SYS_HOUR 154 +#define PROP_SYS_HHOUR 155 +#define PROP_SYS_QHOUR 156 +#define PROP_SYS_MINUTE 157 +#define PROP_SYS_MYHOSTNAME 158 + /* The error codes below are orginally "borrowed" from * liblogging. As such, we reserve values up to -2999 @@ -279,8 +352,14 @@ enum rsRetVal_ /** return value. All methods return this if not specified oth RS_RET_PREVIOUS_COMMITTED = -2122, /**< output plugin status: previous record was committed (an OK state!) */ RS_RET_ACTION_FAILED = -2123, /**< action failed and is now suspended (consider this permanent for the time being) */ RS_RET_NONFATAL_CONFIG_ERR = -2124, /**< non-fatal error during config processing */ + RS_RET_NON_SIZELIMITCMD = -2125, /**< size limit for file defined, but no size limit command given */ + RS_RET_SIZELIMITCMD_DIDNT_RESOLVE = -2126, /**< size limit command did not resolve situation */ + RS_RET_STREAM_DISABLED = -2127, /**< a file has been disabled (e.g. by size limit restriction) */ RS_RET_FILENAME_INVALID = -2140, /**< filename invalid, not found, no access, ... */ + RS_RET_ZLIB_ERR = -2141, /**< error during zlib call */ + RS_RET_VAR_NOT_FOUND = -2142, /**< variable not found */ RS_RET_EMPTY_MSG = -2143, /**< provided (raw) MSG is empty */ + RS_RET_PEER_CLOSED_CONN = -2144, /**< remote peer closed connection (information, no error) */ /* RainerScript error messages (range 1000.. 1999) */ RS_RET_SYSVAR_NOT_FOUND = 1001, /**< system variable could not be found (maybe misspelled) */ @@ -369,6 +448,10 @@ typedef enum rsObjectID rsObjID; # define O_CLOEXEC 0 #endif +/* some constants */ +#define MUTEX_ALREADY_LOCKED 0 +#define LOCK_MUTEX 1 + /* The following prototype is convenient, even though it may not be the 100% correct place.. -- rgerhards 2008-01-07 */ void dbgprintf(char *, ...) __attribute__((format(printf, 1, 2))); diff --git a/runtime/rule.c b/runtime/rule.c new file mode 100644 index 00000000..4c2c9edb --- /dev/null +++ b/runtime/rule.c @@ -0,0 +1,450 @@ +/* rule.c - rsyslog's rule object + * + * See file comment in rule.c for the overall structure of rule processing. + * + * Module begun 2009-06-10 by Rainer Gerhards + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * The rsyslog runtime library is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * The rsyslog runtime library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with the rsyslog runtime library. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. + */ + +#include "config.h" +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <ctype.h> + +#include "rsyslog.h" +#include "obj.h" +#include "action.h" +#include "rule.h" +#include "errmsg.h" +#include "vm.h" +#include "var.h" +#include "srUtils.h" +#include "unicode-helper.h" +#include "dirty.h" /* for getFIOPName */ + +/* static data */ +DEFobjStaticHelpers +DEFobjCurrIf(errmsg) +DEFobjCurrIf(expr) +DEFobjCurrIf(var) +DEFobjCurrIf(vm) + +/* iterate over all actions, this is often needed, for example when HUP processing + * must be done or a shutdown is pending. + */ +static rsRetVal +iterateAllActions(rule_t *pThis, rsRetVal (*pFunc)(void*, void*), void* pParam) +{ + return llExecFunc(&pThis->llActList, pFunc, pParam); +} + + + +/* helper to processMsg(), used to call the configured actions. It is + * executed from within llExecFunc() of the action list. + * rgerhards, 2007-08-02 + */ +typedef struct processMsgDoActions_s { + int bPrevWasSuspended; /* was the previous action suspended? */ + msg_t *pMsg; +} processMsgDoActions_t; +DEFFUNC_llExecFunc(processMsgDoActions) +{ + DEFiRet; + rsRetVal iRetMod; /* return value of module - we do not always pass that back */ + action_t *pAction = (action_t*) pData; + processMsgDoActions_t *pDoActData = (processMsgDoActions_t*) pParam; + + assert(pAction != NULL); + + if((pAction->bExecWhenPrevSusp == 1) && (pDoActData->bPrevWasSuspended == 0)) { + dbgprintf("not calling action because the previous one is not suspended\n"); + ABORT_FINALIZE(RS_RET_OK); + } + + iRetMod = actionCallAction(pAction, pDoActData->pMsg); + if(iRetMod == RS_RET_DISCARDMSG) { + ABORT_FINALIZE(RS_RET_DISCARDMSG); + } else if(iRetMod == RS_RET_SUSPENDED) { + /* indicate suspension for next module to be called */ + pDoActData->bPrevWasSuspended = 1; + } else { + pDoActData->bPrevWasSuspended = 0; + } + +finalize_it: + RETiRet; +} + + +/* This functions looks at the given message and checks if it matches the + * provided filter condition. + */ +static rsRetVal +shouldProcessThisMessage(rule_t *pRule, msg_t *pMsg, int *bProcessMsg) +{ + DEFiRet; + unsigned short pbMustBeFreed; + uchar *pszPropVal; + int bRet = 0; + size_t propLen; + vm_t *pVM = NULL; + var_t *pResult = NULL; + + ISOBJ_TYPE_assert(pRule, rule); + assert(pMsg != NULL); + + /* we first have a look at the global, BSD-style block filters (for tag + * and host). Only if they match, we evaluate the actual filter. + * rgerhards, 2005-10-18 + */ + if(pRule->eHostnameCmpMode == HN_NO_COMP) { + /* EMPTY BY INTENSION - we check this value first, because + * it is the one most often used, so this saves us time! + */ + } else if(pRule->eHostnameCmpMode == HN_COMP_MATCH) { + if(rsCStrSzStrCmp(pRule->pCSHostnameComp, (uchar*) getHOSTNAME(pMsg), getHOSTNAMELen(pMsg))) { + /* not equal, so we are already done... */ + dbgprintf("hostname filter '+%s' does not match '%s'\n", + rsCStrGetSzStrNoNULL(pRule->pCSHostnameComp), getHOSTNAME(pMsg)); + FINALIZE; + } + } else { /* must be -hostname */ + if(!rsCStrSzStrCmp(pRule->pCSHostnameComp, (uchar*) getHOSTNAME(pMsg), getHOSTNAMELen(pMsg))) { + /* not equal, so we are already done... */ + dbgprintf("hostname filter '-%s' does not match '%s'\n", + rsCStrGetSzStrNoNULL(pRule->pCSHostnameComp), getHOSTNAME(pMsg)); + FINALIZE; + } + } + + if(pRule->pCSProgNameComp != NULL) { + int bInv = 0, bEqv = 0, offset = 0; + if(*(rsCStrGetSzStrNoNULL(pRule->pCSProgNameComp)) == '-') { + if(*(rsCStrGetSzStrNoNULL(pRule->pCSProgNameComp) + 1) == '-') + offset = 1; + else { + bInv = 1; + offset = 1; + } + } + if(!rsCStrOffsetSzStrCmp(pRule->pCSProgNameComp, offset, + (uchar*) getProgramName(pMsg, LOCK_MUTEX), getProgramNameLen(pMsg, LOCK_MUTEX))) + bEqv = 1; + + if((!bEqv && !bInv) || (bEqv && bInv)) { + /* not equal or inverted selection, so we are already done... */ + DBGPRINTF("programname filter '%s' does not match '%s'\n", + rsCStrGetSzStrNoNULL(pRule->pCSProgNameComp), getProgramName(pMsg, LOCK_MUTEX)); + FINALIZE; + } + } + + /* done with the BSD-style block filters */ + + if(pRule->f_filter_type == FILTER_PRI) { + /* skip messages that are incorrect priority */ +dbgprintf("testing filter, f_pmask %d\n", pRule->f_filterData.f_pmask[pMsg->iFacility]); + if ( (pRule->f_filterData.f_pmask[pMsg->iFacility] == TABLE_NOPRI) || \ + ((pRule->f_filterData.f_pmask[pMsg->iFacility] & (1<<pMsg->iSeverity)) == 0) ) + bRet = 0; + else + bRet = 1; + } else if(pRule->f_filter_type == FILTER_EXPR) { + CHKiRet(vm.Construct(&pVM)); + CHKiRet(vm.ConstructFinalize(pVM)); + CHKiRet(vm.SetMsg(pVM, pMsg)); + CHKiRet(vm.ExecProg(pVM, pRule->f_filterData.f_expr->pVmprg)); + CHKiRet(vm.PopBoolFromStack(pVM, &pResult)); + dbgprintf("result of expression evaluation: %lld\n", pResult->val.num); + /* VM is destructed on function exit */ + bRet = (pResult->val.num) ? 1 : 0; + } else { + assert(pRule->f_filter_type == FILTER_PROP); /* assert() just in case... */ + pszPropVal = MsgGetProp(pMsg, NULL, pRule->f_filterData.prop.propID, &propLen, &pbMustBeFreed); + + /* Now do the compares (short list currently ;)) */ + switch(pRule->f_filterData.prop.operation ) { + case FIOP_CONTAINS: + if(rsCStrLocateInSzStr(pRule->f_filterData.prop.pCSCompValue, (uchar*) pszPropVal) != -1) + bRet = 1; + break; + case FIOP_ISEQUAL: + if(rsCStrSzStrCmp(pRule->f_filterData.prop.pCSCompValue, + pszPropVal, ustrlen(pszPropVal)) == 0) + bRet = 1; /* process message! */ + break; + case FIOP_STARTSWITH: + if(rsCStrSzStrStartsWithCStr(pRule->f_filterData.prop.pCSCompValue, + pszPropVal, ustrlen(pszPropVal)) == 0) + bRet = 1; /* process message! */ + break; + case FIOP_REGEX: + if(rsCStrSzStrMatchRegex(pRule->f_filterData.prop.pCSCompValue, + (unsigned char*) pszPropVal, 0, &pRule->f_filterData.prop.regex_cache) == RS_RET_OK) + bRet = 1; + break; + case FIOP_EREREGEX: + if(rsCStrSzStrMatchRegex(pRule->f_filterData.prop.pCSCompValue, + (unsigned char*) pszPropVal, 1, &pRule->f_filterData.prop.regex_cache) == RS_RET_OK) + bRet = 1; + break; + default: + /* here, it handles NOP (for performance reasons) */ + assert(pRule->f_filterData.prop.operation == FIOP_NOP); + bRet = 1; /* as good as any other default ;) */ + break; + } + + /* now check if the value must be negated */ + if(pRule->f_filterData.prop.isNegated) + bRet = (bRet == 1) ? 0 : 1; + + if(Debug) { + dbgprintf("Filter: check for property '%s' (value '%s') ", + propIDToName(pRule->f_filterData.prop.propID), pszPropVal); + if(pRule->f_filterData.prop.isNegated) + dbgprintf("NOT "); + dbgprintf("%s '%s': %s\n", + getFIOPName(pRule->f_filterData.prop.operation), + rsCStrGetSzStrNoNULL(pRule->f_filterData.prop.pCSCompValue), + bRet ? "TRUE" : "FALSE"); + } + + /* cleanup */ + if(pbMustBeFreed) + free(pszPropVal); + } + +finalize_it: + /* destruct in any case, not just on error, but it makes error handling much easier */ + if(pVM != NULL) + vm.Destruct(&pVM); + + if(pResult != NULL) + var.Destruct(&pResult); + + *bProcessMsg = bRet; + RETiRet; +} + + + +/* Process (consume) a received message. Calls the actions configured. + * rgerhards, 2005-10-13 + */ +static rsRetVal +processMsg(rule_t *pThis, msg_t *pMsg) +{ + int bProcessMsg; + processMsgDoActions_t DoActData; + DEFiRet; + + ISOBJ_TYPE_assert(pThis, rule); + assert(pMsg != NULL); + + /* first check the filters... */ + CHKiRet(shouldProcessThisMessage(pThis, pMsg, &bProcessMsg)); + if(bProcessMsg) { + DoActData.pMsg = pMsg; + DoActData.bPrevWasSuspended = 0; + CHKiRet(llExecFunc(&pThis->llActList, processMsgDoActions, (void*)&DoActData)); + } + +finalize_it: + RETiRet; +} + + +/* Standard-Constructor + */ +BEGINobjConstruct(rule) /* be sure to specify the object type also in END macro! */ +ENDobjConstruct(rule) + + +/* ConstructionFinalizer + * rgerhards, 2008-01-09 + */ +static rsRetVal +ruleConstructFinalize(rule_t *pThis) +{ + DEFiRet; + ISOBJ_TYPE_assert(pThis, rule); + + /* note: actionDestruct is from action.c API! */ + CHKiRet(llInit(&pThis->llActList, actionDestruct, NULL, NULL)); + +finalize_it: + RETiRet; +} + + +/* destructor for the rule object */ +BEGINobjDestruct(rule) /* be sure to specify the object type also in END and CODESTART macros! */ +CODESTARTobjDestruct(rule) + if(pThis->pCSHostnameComp != NULL) + rsCStrDestruct(&pThis->pCSHostnameComp); + if(pThis->pCSProgNameComp != NULL) + rsCStrDestruct(&pThis->pCSProgNameComp); + + if(pThis->f_filter_type == FILTER_PROP) { + if(pThis->f_filterData.prop.pCSCompValue != NULL) + rsCStrDestruct(&pThis->f_filterData.prop.pCSCompValue); + if(pThis->f_filterData.prop.regex_cache != NULL) + rsCStrRegexDestruct(&pThis->f_filterData.prop.regex_cache); + } else if(pThis->f_filter_type == FILTER_EXPR) { + if(pThis->f_filterData.f_expr != NULL) + expr.Destruct(&pThis->f_filterData.f_expr); + } + + llDestroy(&pThis->llActList); +ENDobjDestruct(rule) + + +/* set the associated ruleset */ +static rsRetVal +setAssRuleset(rule_t *pThis, ruleset_t *pRuleset) +{ + DEFiRet; + ISOBJ_TYPE_assert(pThis, rule); + ISOBJ_TYPE_assert(pRuleset, ruleset); + pThis->pRuleset = pRuleset; + RETiRet; +} + +/* get the associated ruleset (may be NULL if not set!) */ +static ruleset_t* +getAssRuleset(rule_t *pThis) +{ + ISOBJ_TYPE_assert(pThis, rule); + return pThis->pRuleset; +} + + +/* helper to DebugPrint, to print out all actions via + * the llExecFunc() facility. + */ +DEFFUNC_llExecFunc(dbgPrintInitInfoAction) +{ + DEFiRet; + iRet = actionDbgPrint((action_t*) pData); + dbgprintf("\n"); + RETiRet; +} + + +/* debugprint for the rule object */ +BEGINobjDebugPrint(rule) /* be sure to specify the object type also in END and CODESTART macros! */ + int i; +CODESTARTobjDebugPrint(rule) + dbgoprint((obj_t*) pThis, "rsyslog rule:\n"); + if(pThis->pCSProgNameComp != NULL) + dbgprintf("tag: '%s'\n", rsCStrGetSzStrNoNULL(pThis->pCSProgNameComp)); + if(pThis->eHostnameCmpMode != HN_NO_COMP) + dbgprintf("hostname: %s '%s'\n", + pThis->eHostnameCmpMode == HN_COMP_MATCH ? + "only" : "allbut", + rsCStrGetSzStrNoNULL(pThis->pCSHostnameComp)); + if(pThis->f_filter_type == FILTER_PRI) { + for (i = 0; i <= LOG_NFACILITIES; i++) + if (pThis->f_filterData.f_pmask[i] == TABLE_NOPRI) + dbgprintf(" X "); + else + dbgprintf("%2X ", pThis->f_filterData.f_pmask[i]); + } else if(pThis->f_filter_type == FILTER_EXPR) { + dbgprintf("EXPRESSION-BASED Filter: can currently not be displayed"); + } else { + dbgprintf("PROPERTY-BASED Filter:\n"); + dbgprintf("\tProperty.: '%s'\n", propIDToName(pThis->f_filterData.prop.propID)); + dbgprintf("\tOperation: "); + if(pThis->f_filterData.prop.isNegated) + dbgprintf("NOT "); + dbgprintf("'%s'\n", getFIOPName(pThis->f_filterData.prop.operation)); + dbgprintf("\tValue....: '%s'\n", + rsCStrGetSzStrNoNULL(pThis->f_filterData.prop.pCSCompValue)); + dbgprintf("\tAction...: "); + } + + dbgprintf("\nActions:\n"); + llExecFunc(&pThis->llActList, dbgPrintInitInfoAction, NULL); /* actions */ + + dbgprintf("\n"); +ENDobjDebugPrint(rule) + + +/* queryInterface function + * rgerhards, 2008-02-21 + */ +BEGINobjQueryInterface(rule) +CODESTARTobjQueryInterface(rule) + if(pIf->ifVersion != ruleCURR_IF_VERSION) { /* check for current version, increment on each change */ + ABORT_FINALIZE(RS_RET_INTERFACE_NOT_SUPPORTED); + } + + /* ok, we have the right interface, so let's fill it + * Please note that we may also do some backwards-compatibility + * work here (if we can support an older interface version - that, + * of course, also affects the "if" above). + */ + pIf->Construct = ruleConstruct; + pIf->ConstructFinalize = ruleConstructFinalize; + pIf->Destruct = ruleDestruct; + pIf->DebugPrint = ruleDebugPrint; + + pIf->IterateAllActions = iterateAllActions; + pIf->ProcessMsg = processMsg; + pIf->SetAssRuleset = setAssRuleset; + pIf->GetAssRuleset = getAssRuleset; +finalize_it: +ENDobjQueryInterface(rule) + + +/* Exit the rule class. + * rgerhards, 2009-04-06 + */ +BEGINObjClassExit(rule, OBJ_IS_CORE_MODULE) /* class, version */ + objRelease(errmsg, CORE_COMPONENT); + objRelease(expr, CORE_COMPONENT); + objRelease(var, CORE_COMPONENT); + objRelease(vm, CORE_COMPONENT); +ENDObjClassExit(rule) + + +/* Initialize the rule class. Must be called as the very first method + * before anything else is called inside this class. + * rgerhards, 2008-02-19 + */ +BEGINObjClassInit(rule, 1, OBJ_IS_CORE_MODULE) /* class, version */ + /* request objects we use */ + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(expr, CORE_COMPONENT)); + CHKiRet(objUse(var, CORE_COMPONENT)); + CHKiRet(objUse(vm, CORE_COMPONENT)); + + /* set our own handlers */ + OBJSetMethodHandler(objMethod_DEBUGPRINT, ruleDebugPrint); + OBJSetMethodHandler(objMethod_CONSTRUCTION_FINALIZER, ruleConstructFinalize); +ENDObjClassInit(rule) + +/* vi:set ai: + */ diff --git a/runtime/rule.h b/runtime/rule.h new file mode 100644 index 00000000..99ac44e7 --- /dev/null +++ b/runtime/rule.h @@ -0,0 +1,77 @@ +/* The rule object. + * + * This implements rules within rsyslog. + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * The rsyslog runtime library is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * The rsyslog runtime library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with the rsyslog runtime library. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. + */ +#ifndef INCLUDED_RULE_H +#define INCLUDED_RULE_H + +#include "linkedlist.h" +#include "regexp.h" +#include "expr.h" + +/* the rule object */ +struct rule_s { + BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ + /* filter properties */ + enum { + FILTER_PRI = 0, /* traditional PRI based filer */ + FILTER_PROP = 1, /* extended filter, property based */ + FILTER_EXPR = 2 /* extended filter, expression based */ + } f_filter_type; + EHostnameCmpMode eHostnameCmpMode; + cstr_t *pCSHostnameComp; /* hostname to check */ + cstr_t *pCSProgNameComp; /* tag to check or NULL, if not to be checked */ + union { + u_char f_pmask[LOG_NFACILITIES+1]; /* priority mask */ + struct { + fiop_t operation; + regex_t *regex_cache; /* cache for compiled REs, if such are used */ + cstr_t *pCSCompValue; /* value to "compare" against */ + bool isNegated; + propid_t propID; /* ID of the requested property */ + } prop; + expr_t *f_expr; /* expression object */ + } f_filterData; + + ruleset_t *pRuleset; /* associated ruleset */ + linkedList_t llActList; /* list of configured actions */ +}; + +/* interfaces */ +BEGINinterface(rule) /* name must also be changed in ENDinterface macro! */ + INTERFACEObjDebugPrint(rule); + rsRetVal (*Construct)(rule_t **ppThis); + rsRetVal (*ConstructFinalize)(rule_t __attribute__((unused)) *pThis); + rsRetVal (*Destruct)(rule_t **ppThis); + rsRetVal (*IterateAllActions)(rule_t *pThis, rsRetVal (*pFunc)(void*, void*), void *pParam); + rsRetVal (*ProcessMsg)(rule_t *pThis, msg_t *pMsg); + rsRetVal (*SetAssRuleset)(rule_t *pThis, ruleset_t*); + ruleset_t* (*GetAssRuleset)(rule_t *pThis); +ENDinterface(rule) +#define ruleCURR_IF_VERSION 1 /* increment whenever you change the interface structure! */ + + +/* prototypes */ +PROTOTYPEObj(rule); + +#endif /* #ifndef INCLUDED_RULE_H */ diff --git a/runtime/ruleset.c b/runtime/ruleset.c new file mode 100644 index 00000000..af61f24f --- /dev/null +++ b/runtime/ruleset.c @@ -0,0 +1,449 @@ +/* ruleset.c - rsyslog's ruleset object + * + * We have a two-way structure of linked lists: one global linked list + * (llAllRulesets) hold alls rule sets that we know. Included in each + * list is a list of rules (which contain a list of actions, but that's + * a different story). + * + * Usually, only a single rule set is executed. However, there exist some + * situations where all rules must be iterated over, for example on HUP. Thus, + * we also provide interfaces to do that. + * + * Module begun 2009-06-10 by Rainer Gerhards + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * The rsyslog runtime library is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * The rsyslog runtime library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with the rsyslog runtime library. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. + */ + +#include "config.h" +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <ctype.h> + +#include "rsyslog.h" +#include "obj.h" +#include "msg.h" +#include "ruleset.h" +#include "rule.h" +#include "errmsg.h" +#include "unicode-helper.h" + +static rsRetVal debugPrintAll(void); // TODO: remove! + +/* static data */ +DEFobjStaticHelpers +DEFobjCurrIf(errmsg) +DEFobjCurrIf(rule) + +linkedList_t llRulesets; /* this is NOT a pointer - no typo here ;) */ +ruleset_t *pCurrRuleset = NULL; /* currently "active" ruleset */ +ruleset_t *pDfltRuleset = NULL; /* currentl default ruleset, e.g. for binding to actions which have no other */ + +/* ---------- linked-list key handling functions ---------- */ + +/* destructor for linked list keys. + */ +static rsRetVal keyDestruct(void __attribute__((unused)) *pData) +{ + free(pData); + return RS_RET_OK; +} + + +/* ---------- END linked-list key handling functions ---------- */ + + +/* driver to iterate over all of this ruleset actions */ +typedef struct iterateAllActions_s { + rsRetVal (*pFunc)(void*, void*); + void *pParam; +} iterateAllActions_t; +DEFFUNC_llExecFunc(doIterateRulesetActions) +{ + DEFiRet; + rule_t* pRule = (rule_t*) pData; + iterateAllActions_t *pMyParam = (iterateAllActions_t*) pParam; + iRet = rule.IterateAllActions(pRule, pMyParam->pFunc, pMyParam->pParam); + RETiRet; +} +/* iterate over all actions of THIS rule set. + */ +static rsRetVal +iterateRulesetAllActions(ruleset_t *pThis, rsRetVal (*pFunc)(void*, void*), void* pParam) +{ + iterateAllActions_t params; + DEFiRet; + assert(pFunc != NULL); + + params.pFunc = pFunc; + params.pParam = pParam; + CHKiRet(llExecFunc(&(pThis->llRules), doIterateRulesetActions, ¶ms)); + +finalize_it: + RETiRet; +} + + +/* driver to iterate over all actions */ +DEFFUNC_llExecFunc(doIterateAllActions) +{ + DEFiRet; + ruleset_t* pThis = (ruleset_t*) pData; + iterateAllActions_t *pMyParam = (iterateAllActions_t*) pParam; + iRet = iterateRulesetAllActions(pThis, pMyParam->pFunc, pMyParam->pParam); + RETiRet; +} +/* iterate over ALL actions present in the WHOLE system. + * this is often needed, for example when HUP processing + * must be done or a shutdown is pending. + */ +static rsRetVal +iterateAllActions(rsRetVal (*pFunc)(void*, void*), void* pParam) +{ + iterateAllActions_t params; + DEFiRet; + assert(pFunc != NULL); + + params.pFunc = pFunc; + params.pParam = pParam; + CHKiRet(llExecFunc(&llRulesets, doIterateAllActions, ¶ms)); + +finalize_it: + RETiRet; +} + + + +/* helper to processMsg(), used to call the configured actions. It is + * executed from within llExecFunc() of the action list. + * rgerhards, 2007-08-02 + */ +DEFFUNC_llExecFunc(processMsgDoRules) +{ + ISOBJ_TYPE_assert(pData, rule); + return rule.ProcessMsg((rule_t*) pData, (msg_t*) pParam); +} + + +/* Process (consume) a received message. Calls the actions configured. + * rgerhards, 2005-10-13 + */ +static rsRetVal +processMsg(msg_t *pMsg) +{ + ruleset_t *pThis; + DEFiRet; + assert(pMsg != NULL); + + pThis = (pMsg->pRuleset == NULL) ? pDfltRuleset : pMsg->pRuleset; + ISOBJ_TYPE_assert(pThis, ruleset); + + CHKiRet(llExecFunc(&pThis->llRules, processMsgDoRules, pMsg)); + +finalize_it: + if(iRet == RS_RET_DISCARDMSG) + iRet = RS_RET_OK; + + RETiRet; +} + +/* Add a new rule to the end of the current rule set. We do a number + * of checks and ignore the rule if it does not pass them. + */ +static rsRetVal +addRule(ruleset_t *pThis, rule_t **ppRule) +{ + int iActionCnt; + DEFiRet; + + ISOBJ_TYPE_assert(pThis, ruleset); + ISOBJ_TYPE_assert(*ppRule, rule); + + CHKiRet(llGetNumElts(&(*ppRule)->llActList, &iActionCnt)); + if(iActionCnt == 0) { + errmsg.LogError(0, NO_ERRCODE, "warning: selector line without actions will be discarded"); + rule.Destruct(ppRule); + } else { + CHKiRet(llAppend(&pThis->llRules, NULL, *ppRule)); + dbgprintf("selector line successfully processed\n"); + } + +finalize_it: + RETiRet; +} + + +/* set name for ruleset */ +static rsRetVal setName(ruleset_t *pThis, uchar *pszName) +{ + DEFiRet; + free(pThis->pszName); + CHKmalloc(pThis->pszName = ustrdup(pszName)); + +finalize_it: + RETiRet; +} + + +/* get current ruleset + * We use a non-standard calling interface, as nothing can go wrong and it + * is really much more natural to return the pointer directly. + */ +static ruleset_t* +GetCurrent(void) +{ + return pCurrRuleset; +} + + +/* Find the ruleset with the given name and return a pointer to its object. + */ +static rsRetVal +GetRuleset(ruleset_t **ppRuleset, uchar *pszName) +{ + DEFiRet; + assert(ppRuleset != NULL); + assert(pszName != NULL); + + CHKiRet(llFind(&llRulesets, pszName, (void*) ppRuleset)); + +finalize_it: + RETiRet; +} + + +/* Set a new default rule set. If the default can not be found, no change happens. + */ +static rsRetVal +SetDefaultRuleset(uchar *pszName) +{ + ruleset_t *pRuleset; + DEFiRet; + assert(pszName != NULL); + + CHKiRet(GetRuleset(&pRuleset, pszName)); + pDfltRuleset = pRuleset; + dbgprintf("default rule set changed to %p: '%s'\n", pRuleset, pszName); + +finalize_it: + RETiRet; +} + + +/* Set a new current rule set. If the ruleset can not be found, no change happens. + */ +static rsRetVal +SetCurrRuleset(uchar *pszName) +{ + ruleset_t *pRuleset; + DEFiRet; + assert(pszName != NULL); + + CHKiRet(GetRuleset(&pRuleset, pszName)); + pCurrRuleset = pRuleset; + dbgprintf("current rule set changed to %p: '%s'\n", pRuleset, pszName); + +finalize_it: + RETiRet; +} + + +/* destructor we need to destruct rules inside our linked list contents. + */ +static rsRetVal +doRuleDestruct(void *pData) +{ + rule_t *pRule = (rule_t *) pData; + DEFiRet; + rule.Destruct(&pRule); + RETiRet; +} + + +/* Standard-Constructor + */ +BEGINobjConstruct(ruleset) /* be sure to specify the object type also in END macro! */ + CHKiRet(llInit(&pThis->llRules, doRuleDestruct, NULL, NULL)); +finalize_it: +ENDobjConstruct(ruleset) + + +/* ConstructionFinalizer + * This also adds the rule set to the list of all known rulesets. + */ +static rsRetVal +rulesetConstructFinalize(ruleset_t *pThis) +{ + uchar *keyName; + DEFiRet; + ISOBJ_TYPE_assert(pThis, ruleset); + + /* we must duplicate our name, as the key destructer would also + * free it, resulting in a double-free. It's also cleaner to have + * two separate copies. + */ + CHKmalloc(keyName = ustrdup(pThis->pszName)); + CHKiRet(llAppend(&llRulesets, keyName, pThis)); + + /* this now also is the new current ruleset */ + pCurrRuleset = pThis; + + /* and also the default, if so far none has been set */ + if(pDfltRuleset == NULL) + pDfltRuleset = pThis; + +finalize_it: + RETiRet; +} + + +/* destructor for the ruleset object */ +BEGINobjDestruct(ruleset) /* be sure to specify the object type also in END and CODESTART macros! */ +CODESTARTobjDestruct(ruleset) + dbgprintf("destructing ruleset %p, name %p\n", pThis, pThis->pszName); + llDestroy(&pThis->llRules); + free(pThis->pszName); +ENDobjDestruct(ruleset) + +/* this is a special destructor for the linkedList class. LinkedList does NOT + * provide a pointer to the pointer, but rather the raw pointer itself. So we + * must map this, otherwise the destructor will abort. + */ +static rsRetVal +rulesetDestructForLinkedList(void *pData) +{ + ruleset_t *pThis = (ruleset_t*) pData; + return rulesetDestruct(&pThis); +} + + +/* destruct ALL rule sets that reside in the system. This must + * be callable before unloading this module as the module may + * not be unloaded before unload of the actions is required. This is + * kind of a left-over from previous logic and may be optimized one + * everything runs stable again. -- rgerhards, 2009-06-10 + */ +static rsRetVal +destructAllActions(void) +{ + DEFiRet; + + CHKiRet(llDestroy(&llRulesets)); + CHKiRet(llInit(&llRulesets, rulesetDestructForLinkedList, keyDestruct, strcasecmp)); + pDfltRuleset = NULL; + +finalize_it: + RETiRet; +} + +/* helper for debugPrint(), initiates rule printing */ +DEFFUNC_llExecFunc(doDebugPrintRule) +{ + return rule.DebugPrint((rule_t*) pData); +} +/* debugprint for the ruleset object */ +BEGINobjDebugPrint(ruleset) /* be sure to specify the object type also in END and CODESTART macros! */ +CODESTARTobjDebugPrint(ruleset) + dbgoprint((obj_t*) pThis, "rsyslog ruleset %s:\n", pThis->pszName); + llExecFunc(&pThis->llRules, doDebugPrintRule, NULL); +ENDobjDebugPrint(ruleset) + + +/* helper for debugPrintAll(), prints a single ruleset */ +DEFFUNC_llExecFunc(doDebugPrintAll) +{ + return rulesetDebugPrint((ruleset_t*) pData); +} +/* debug print all rulesets + */ +static rsRetVal +debugPrintAll(void) +{ + DEFiRet; + dbgprintf("All Rulesets:\n"); + llExecFunc(&llRulesets, doDebugPrintAll, NULL); + dbgprintf("End of Rulesets.\n"); + RETiRet; +} + + +/* queryInterface function + * rgerhards, 2008-02-21 + */ +BEGINobjQueryInterface(ruleset) +CODESTARTobjQueryInterface(ruleset) + if(pIf->ifVersion != rulesetCURR_IF_VERSION) { /* check for current version, increment on each change */ + ABORT_FINALIZE(RS_RET_INTERFACE_NOT_SUPPORTED); + } + + /* ok, we have the right interface, so let's fill it + * Please note that we may also do some backwards-compatibility + * work here (if we can support an older interface version - that, + * of course, also affects the "if" above). + */ + pIf->Construct = rulesetConstruct; + pIf->ConstructFinalize = rulesetConstructFinalize; + pIf->Destruct = rulesetDestruct; + pIf->DebugPrint = rulesetDebugPrint; + + pIf->IterateAllActions = iterateAllActions; + pIf->DestructAllActions = destructAllActions; + pIf->AddRule = addRule; + pIf->ProcessMsg = processMsg; + pIf->SetName = setName; + pIf->DebugPrintAll = debugPrintAll; + pIf->GetCurrent = GetCurrent; + pIf->GetRuleset = GetRuleset; + pIf->SetDefaultRuleset = SetDefaultRuleset; + pIf->SetCurrRuleset = SetCurrRuleset; +finalize_it: +ENDobjQueryInterface(ruleset) + + +/* Exit the ruleset class. + * rgerhards, 2009-04-06 + */ +BEGINObjClassExit(ruleset, OBJ_IS_CORE_MODULE) /* class, version */ + llDestroy(&llRulesets); + objRelease(errmsg, CORE_COMPONENT); + objRelease(rule, CORE_COMPONENT); +ENDObjClassExit(ruleset) + + +/* Initialize the ruleset class. Must be called as the very first method + * before anything else is called inside this class. + * rgerhards, 2008-02-19 + */ +BEGINObjClassInit(ruleset, 1, OBJ_IS_CORE_MODULE) /* class, version */ + /* request objects we use */ + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(rule, CORE_COMPONENT)); + + /* set our own handlers */ + OBJSetMethodHandler(objMethod_DEBUGPRINT, rulesetDebugPrint); + OBJSetMethodHandler(objMethod_CONSTRUCTION_FINALIZER, rulesetConstructFinalize); + + /* prepare global data */ + CHKiRet(llInit(&llRulesets, rulesetDestructForLinkedList, keyDestruct, strcasecmp)); +ENDObjClassInit(ruleset) + +/* vi:set ai: + */ diff --git a/runtime/ruleset.h b/runtime/ruleset.h new file mode 100644 index 00000000..32571687 --- /dev/null +++ b/runtime/ruleset.h @@ -0,0 +1,60 @@ +/* The ruleset object. + * + * This implements rulesets within rsyslog. + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * The rsyslog runtime library is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * The rsyslog runtime library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with the rsyslog runtime library. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. + */ +#ifndef INCLUDED_RULESET_H +#define INCLUDED_RULESET_H + +#include "linkedlist.h" + +/* the ruleset object */ +struct ruleset_s { + BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ + linkedList_t llRules; /* this is NOT a pointer - no typo here ;) */ + uchar *pszName; /* name of our ruleset */ +}; + +/* interfaces */ +BEGINinterface(ruleset) /* name must also be changed in ENDinterface macro! */ + INTERFACEObjDebugPrint(ruleset); + rsRetVal (*DebugPrintAll)(void); + rsRetVal (*Construct)(ruleset_t **ppThis); + rsRetVal (*ConstructFinalize)(ruleset_t __attribute__((unused)) *pThis); + rsRetVal (*Destruct)(ruleset_t **ppThis); + rsRetVal (*IterateAllActions)(rsRetVal (*pFunc)(void*, void*), void* pParam); + rsRetVal (*DestructAllActions)(void); + rsRetVal (*AddRule)(ruleset_t *pThis, rule_t **ppRule); + rsRetVal (*SetName)(ruleset_t *pThis, uchar *pszName); + rsRetVal (*ProcessMsg)(msg_t *pMsg); + rsRetVal (*GetRuleset)(ruleset_t **ppThis, uchar*); + rsRetVal (*SetDefaultRuleset)(uchar*); + rsRetVal (*SetCurrRuleset)(uchar*); + ruleset_t* (*GetCurrent)(void); +ENDinterface(ruleset) +#define rulesetCURR_IF_VERSION 1 /* increment whenever you change the interface structure! */ + + +/* prototypes */ +PROTOTYPEObj(ruleset); + +#endif /* #ifndef INCLUDED_RULESET_H */ diff --git a/runtime/srUtils.h b/runtime/srUtils.h index bfce4cbb..16766312 100644 --- a/runtime/srUtils.h +++ b/runtime/srUtils.h @@ -92,6 +92,7 @@ void srSleep(int iSeconds, int iuSeconds); char *rs_strerror_r(int errnum, char *buf, size_t buflen); int decodeSyslogName(uchar *name, syslogName_t *codetab); int getSubString(uchar **ppSrc, char *pDst, size_t DstSize, char cSep); +rsRetVal getFileSize(uchar *pszName, off_t *pSize); /* mutex operations */ /* some macros to cancel-safe lock a mutex (it will automatically be released @@ -108,8 +109,6 @@ int getSubString(uchar **ppSrc, char *pDst, size_t DstSize, char cSep); #define mutex_cancelsafe_unlock(mut) pthread_cleanup_pop(1) /* some useful constants */ -#define MUTEX_ALREADY_LOCKED 0 -#define LOCK_MUTEX 1 #define DEFVARS_mutexProtection\ int iCancelStateSave; \ int bLockedOpIsLocked=0 @@ -124,4 +123,17 @@ int getSubString(uchar **ppSrc, char *pDst, size_t DstSize, char cSep); d_pthread_mutex_unlock(mut); \ pthread_setcancelstate(iCancelStateSave, NULL); \ } + +/* The unconditional versions of the macro always lock the mutex. They are preferred in + * complex scenarios, where the simple ones might get mixed up by multiple calls. + */ +#define DEFVARS_mutexProtection_uncond\ + int iCancelStateSave +#define BEGIN_MTX_PROTECTED_OPERATIONS_UNCOND(mut) \ + pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &iCancelStateSave); \ + d_pthread_mutex_lock(mut); +#define END_MTX_PROTECTED_OPERATIONS_UNCOND(mut) \ + d_pthread_mutex_unlock(mut); \ + pthread_setcancelstate(iCancelStateSave, NULL); + #endif diff --git a/runtime/srutils.c b/runtime/srutils.c index d01ca20d..1452c9b7 100644 --- a/runtime/srutils.c +++ b/runtime/srutils.c @@ -166,10 +166,22 @@ uchar *srUtilStrDup(uchar *pOld, size_t len) /* creates a path recursively - * Return 0 on success, -1 otherwise. On failure, errno - * hold the last OS error. - * Param "mode" holds the mode that all non-existing directories - * are to be created with. + * Return 0 on success, -1 otherwise. On failure, errno * hold the last OS error. + * Param "mode" holds the mode that all non-existing directories are to be + * created with. + * Note that we have a potential race inside that code, a race that even exists + * outside of the rsyslog process (if multiple instances run, or other programs + * generate directories): If the directory does not exist, a context switch happens, + * at that moment another process creates it, then our creation on the context + * switch back fails. This actually happened in practice, and depending on the + * configuration it is even likely to happen. We can not solve this situation + * with a mutex, as that works only within out process space. So the solution + * is that we take the optimistic approach, try the creation, and if it fails + * with "already exists" we go back and do one retry of the check/create + * sequence. That should then succeed. If the directory is still not found but + * the creation fails in the similar way, we return an error on that second + * try because otherwise we would potentially run into an endless loop. + * loop. -- rgerhards, 2010-03-25 */ int makeFileParentDirs(uchar *szFile, size_t lenFile, mode_t mode, uid_t uid, gid_t gid, int bFailOnChownFail) @@ -177,6 +189,8 @@ int makeFileParentDirs(uchar *szFile, size_t lenFile, mode_t mode, uchar *p; uchar *pszWork; size_t len; + int err; + int iTry = 0; int bErr = 0; assert(szFile != NULL); @@ -190,8 +204,9 @@ int makeFileParentDirs(uchar *szFile, size_t lenFile, mode_t mode, if(*p == '/') { /* temporarily terminate string, create dir and go on */ *p = '\0'; +again: if(access((char*)pszWork, F_OK)) { - if(mkdir((char*)pszWork, mode) == 0) { + if((err = mkdir((char*)pszWork, mode)) == 0) { if(uid != (uid_t) -1 || gid != (gid_t) -1) { /* we need to set owner/group */ if(chown((char*)pszWork, uid, gid) != 0) @@ -201,8 +216,13 @@ int makeFileParentDirs(uchar *szFile, size_t lenFile, mode_t mode, * to do so. */ } - } else + } else { + if(err == EEXIST && iTry == 0) { + iTry = 1; + goto again; + } bErr = 1; + } if(bErr) { int eSave = errno; free(pszWork); @@ -366,6 +386,7 @@ int getNumberDigits(long lNum) /* compute an absolute time timeout suitable for calls to pthread_cond_timedwait() + * iTimeout is in milliseconds * rgerhards, 2008-01-14 */ rsRetVal @@ -375,11 +396,12 @@ timeoutComp(struct timespec *pt, long iTimeout) assert(pt != NULL); /* compute timeout */ clock_gettime(CLOCK_REALTIME, pt); + pt->tv_sec += iTimeout / 1000; pt->tv_nsec += (iTimeout % 1000) * 1000000; /* think INTEGER arithmetic! */ if(pt->tv_nsec > 999999999) { /* overrun? */ pt->tv_nsec -= 1000000000; + ++pt->tv_sec; } - pt->tv_sec += iTimeout / 1000; ENDfunc return RS_RET_OK; /* so far, this is static... */ } @@ -553,6 +575,33 @@ int getSubString(uchar **ppSrc, char *pDst, size_t DstSize, char cSep) } +/* get the size of a file or return appropriate error code. If an error is returned, + * *pSize content is undefined. + * rgerhards, 2009-06-12 + */ +rsRetVal +getFileSize(uchar *pszName, off_t *pSize) +{ + int ret; + struct stat statBuf; + DEFiRet; + + ret = stat((char*) pszName, &statBuf); + if(ret == -1) { + switch(errno) { + case EACCES: ABORT_FINALIZE(RS_RET_NO_FILE_ACCESS); + case ENOTDIR: + case ENOENT: ABORT_FINALIZE(RS_RET_FILE_NOT_FOUND); + default: ABORT_FINALIZE(RS_RET_FILE_NO_STAT); + } + } + + *pSize = statBuf.st_size; + +finalize_it: + RETiRet; +} + /* vim:set ai: */ diff --git a/runtime/stream.c b/runtime/stream.c index 9997e685..869324ec 100644 --- a/runtime/stream.c +++ b/runtime/stream.c @@ -1,4 +1,3 @@ -//TODO: O_TRUC mode! /* The serial stream class. * * A serial stream provides serial data access. In theory, serial streams @@ -7,8 +6,17 @@ * "driver"). * * File begun on 2008-01-09 by RGerhards + * Large modifications in 2009-06 to support using it with omfile, including zip writer. + * Note that this file obtains the zlib wrapper object is needed, but it never frees it + * again. While this sounds like a leak (and one may argue it actually is), there is no + * harm associated with that. The reason is that strm is a core object, so it is terminated + * only when rsyslogd exists. As we could only release on termination (or else bear more + * overhead for keeping track of how many users we have), not releasing zlibw is OK, because + * it will be released when rsyslogd terminates. We may want to revisit this decision if + * it turns out to be problematic. Then, we need to quasi-refcount the number of accesses + * to the object. * - * Copyright 2008 Rainer Gerhards and Adiscon GmbH. + * Copyright 2008, 2009 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -39,24 +47,205 @@ #include <unistd.h> #include <sys/stat.h> /* required for HP UX */ #include <errno.h> +#include <pthread.h> #include "rsyslog.h" #include "stringbuf.h" #include "srUtils.h" #include "obj.h" #include "stream.h" +#include "unicode-helper.h" +#include "module-template.h" +#if HAVE_SYS_PRCTL_H +# include <sys/prctl.h> +#endif + +/* some platforms do not have large file support :( */ +#ifndef O_LARGEFILE +# define O_LARGEFILE 0 +#endif +#ifndef HAVE_LSEEK64 + typedef off_t off64_t; +# define lseek64(fd, offset, whence) lseek(fd, offset, whence) +#endif /* static data */ DEFobjStaticHelpers +DEFobjCurrIf(zlibw) + +/* forward definitions */ +static rsRetVal strmFlushInternal(strm_t *pThis); +static rsRetVal strmWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf); +static rsRetVal strmCloseFile(strm_t *pThis); +static void *asyncWriterThread(void *pPtr); +static rsRetVal doZipWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf); +static rsRetVal strmPhysWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf); + /* methods */ -/* first, we define type-specific handlers. The provide a generic functionality, +/* Try to resolve a size limit situation. This is used to support custom-file size handlers + * for omfile. It first runs the command, and then checks if we are still above the size + * treshold. Note that this works only with single file names, NOT with circular names. + * Note that pszCurrFName can NOT be taken from pThis, because the stream is closed when + * we are called (and that destroys pszCurrFName, as there is NO CURRENT file name!). So + * we need to receive the name as a parameter. + * initially wirtten 2005-06-21, moved to this class & updates 2009-06-01, both rgerhards + */ +static rsRetVal +resolveFileSizeLimit(strm_t *pThis, uchar *pszCurrFName) +{ + uchar *pParams; + uchar *pCmd; + uchar *p; + off_t actualFileSize; + rsRetVal localRet; + DEFiRet; + ISOBJ_TYPE_assert(pThis, strm); + assert(pszCurrFName != NULL); + + if(pThis->pszSizeLimitCmd == NULL) { + ABORT_FINALIZE(RS_RET_NON_SIZELIMITCMD); /* nothing we can do in this case... */ + } + + /* we first check if we have command line parameters. We assume this, + * when we have a space in the program name. If we find it, everything after + * the space is treated as a single argument. + */ + CHKmalloc(pCmd = ustrdup(pThis->pszSizeLimitCmd)); + + for(p = pCmd ; *p && *p != ' ' ; ++p) { + /* JUST SKIP */ + } + + if(*p == ' ') { + *p = '\0'; /* pretend string-end */ + pParams = p+1; + } else + pParams = NULL; + + /* the execProg() below is probably not great, but at least is is + * fairly secure now. Once we change the way file size limits are + * handled, we should also revisit how this command is run (and + * with which parameters). rgerhards, 2007-07-20 + */ + execProg(pCmd, 1, pParams); + + free(pCmd); + + localRet = getFileSize(pszCurrFName, &actualFileSize); + + if(localRet == RS_RET_OK && actualFileSize >= pThis->iSizeLimit) { + ABORT_FINALIZE(RS_RET_SIZELIMITCMD_DIDNT_RESOLVE); /* OK, it didn't work out... */ + } else if(localRet != RS_RET_FILE_NOT_FOUND) { + /* file not found is OK, the command may have moved away the file */ + ABORT_FINALIZE(localRet); + } + +finalize_it: + if(iRet != RS_RET_OK) { + if(iRet == RS_RET_SIZELIMITCMD_DIDNT_RESOLVE) { + DBGPRINTF("file size limit cmd for file '%s' did no resolve situation\n", pszCurrFName); + } else { + DBGPRINTF("file size limit cmd for file '%s' failed with code %d.\n", pszCurrFName, iRet); + } + pThis->bDisabled = 1; + } + + RETiRet; +} + + +/* Check if the file has grown beyond the configured omfile iSizeLimit + * and, if so, initiate processing. + */ +static rsRetVal +doSizeLimitProcessing(strm_t *pThis) +{ + uchar *pszCurrFName = NULL; + DEFiRet; + + ISOBJ_TYPE_assert(pThis, strm); + ASSERT(pThis->iSizeLimit != 0); + ASSERT(pThis->fd != -1); + + if(pThis->iCurrOffs >= pThis->iSizeLimit) { + /* strmCloseFile() destroys the current file name, so we + * need to preserve it. + */ + CHKmalloc(pszCurrFName = ustrdup(pThis->pszCurrFName)); + CHKiRet(strmCloseFile(pThis)); + CHKiRet(resolveFileSizeLimit(pThis, pszCurrFName)); + } + +finalize_it: + free(pszCurrFName); + RETiRet; +} + + +/* now, we define type-specific handlers. The provide a generic functionality, * but for this specific type of strm. The mapping to these handlers happens during * strm construction. Later on, handlers are called by pointers present in the * strm instance object. */ +/* do the physical open() call on a file. + */ +static rsRetVal +doPhysOpen(strm_t *pThis) +{ + int iFlags = 0; + DEFiRet; + ISOBJ_TYPE_assert(pThis, strm); + + /* compute which flags we need to provide to open */ + switch(pThis->tOperationsMode) { + case STREAMMODE_READ: + iFlags = O_CLOEXEC | O_NOCTTY | O_RDONLY; + break; + case STREAMMODE_WRITE: /* legacy mode used inside queue engine */ + iFlags = O_CLOEXEC | O_NOCTTY | O_WRONLY | O_CREAT; + break; + case STREAMMODE_WRITE_TRUNC: + iFlags = O_CLOEXEC | O_NOCTTY | O_WRONLY | O_CREAT | O_TRUNC; + break; + case STREAMMODE_WRITE_APPEND: + iFlags = O_CLOEXEC | O_NOCTTY | O_WRONLY | O_CREAT | O_APPEND; + break; + default:assert(0); + break; + } + if(pThis->sType == STREAMTYPE_NAMED_PIPE) { + DBGPRINTF("Note: stream '%s' is a named pipe, open with O_NONBLOCK\n", pThis->pszCurrFName); + iFlags |= O_NONBLOCK; + } + + pThis->fd = open((char*)pThis->pszCurrFName, iFlags | O_LARGEFILE, pThis->tOpenMode); + DBGPRINTF("file '%s' opened as #%d with mode %d\n", pThis->pszCurrFName, pThis->fd, pThis->tOpenMode); + if(pThis->fd == -1) { + char errStr[1024]; + int err = errno; + rs_strerror_r(err, errStr, sizeof(errStr)); + DBGOPRINT((obj_t*) pThis, "open error %d, file '%s': %s\n", errno, pThis->pszCurrFName, errStr); + if(err == ENOENT) + ABORT_FINALIZE(RS_RET_FILE_NOT_FOUND); + else + ABORT_FINALIZE(RS_RET_IO_ERROR); + } else { + if(!ustrcmp(pThis->pszCurrFName, UCHAR_CONSTANT(_PATH_CONSOLE)) || isatty(pThis->fd)) { + DBGPRINTF("file %d is a tty-type file\n", pThis->fd); + pThis->bIsTTY = 1; + } else { + pThis->bIsTTY = 0; + } + } + +finalize_it: + RETiRet; +} + + /* open a strm file * It is OK to call this function when the stream is already open. In that * case, it returns immediately with RS_RET_OK @@ -64,10 +253,8 @@ DEFobjStaticHelpers static rsRetVal strmOpenFile(strm_t *pThis) { DEFiRet; - int iFlags; ASSERT(pThis != NULL); - ASSERT(pThis->tOperationsMode == STREAMMODE_READ || pThis->tOperationsMode == STREAMMODE_WRITE); if(pThis->fd != -1) ABORT_FINALIZE(RS_RET_OK); @@ -88,56 +275,88 @@ static rsRetVal strmOpenFile(strm_t *pThis) } } - /* compute which flags we need to provide to open */ - if(pThis->tOperationsMode == STREAMMODE_READ) - iFlags = O_RDONLY; - else - iFlags = O_WRONLY | O_CREAT; - - iFlags |= pThis->iAddtlOpenFlags; - - pThis->fd = open((char*)pThis->pszCurrFName, iFlags | O_LARGEFILE, pThis->tOpenMode); - if(pThis->fd == -1) { - int ierrnoSave = errno; - char errmsg[1024]; - dbgoprint((obj_t*) pThis, "open error %d, file '%s': %s\n", errno, pThis->pszCurrFName, - rs_strerror_r(errno, errmsg, sizeof(errmsg))); - if(ierrnoSave == ENOENT) - ABORT_FINALIZE(RS_RET_FILE_NOT_FOUND); - else - ABORT_FINALIZE(RS_RET_IO_ERROR); - } + CHKiRet(doPhysOpen(pThis)); pThis->iCurrOffs = 0; + if(pThis->tOperationsMode == STREAMMODE_WRITE_APPEND) { + /* we need to obtain the current offset */ + off_t offset; + CHKiRet(getFileSize(pThis->pszCurrFName, &offset)); + pThis->iCurrOffs = offset; + } - dbgoprint((obj_t*) pThis, "opened file '%s' for %s (0x%x) as %d\n", pThis->pszCurrFName, - (pThis->tOperationsMode == STREAMMODE_READ) ? "READ" : "WRITE", iFlags, pThis->fd); + DBGOPRINT((obj_t*) pThis, "opened file '%s' for %s as %d\n", pThis->pszCurrFName, + (pThis->tOperationsMode == STREAMMODE_READ) ? "READ" : "WRITE", pThis->fd); finalize_it: RETiRet; } +/* wait for the output writer thread to be done. This must be called before actions + * that require data to be persisted. May be called in non-async mode and is a null + * operation than. Must be called with the mutex locked. + */ +static inline void +strmWaitAsyncWriterDone(strm_t *pThis) +{ + BEGINfunc + if(pThis->bAsyncWrite) { + /* awake writer thread and make it write out everything */ + while(pThis->iCnt > 0) { + pthread_cond_signal(&pThis->notEmpty); + d_pthread_cond_wait(&pThis->isEmpty, &pThis->mut); + } + } + ENDfunc +} + + /* close a strm file * Note that the bDeleteOnClose flag is honored. If it is set, the file will be * deleted after close. This is in support for the qRead thread. + * Note: it is valid to call this function when the physical file is closed. If so, + * strmCloseFile() will still check if there is any unwritten data inside buffers + * (this may be the case) and, if so, will open the file, write the data, and then + * close it again (this is done via strmFlushInternal and friends). */ static rsRetVal strmCloseFile(strm_t *pThis) { DEFiRet; ASSERT(pThis != NULL); - ASSERT(pThis->fd != -1); - dbgoprint((obj_t*) pThis, "file %d closing\n", pThis->fd); + DBGOPRINT((obj_t*) pThis, "file %d(%s) closing\n", pThis->fd, + (pThis->pszFName == NULL) ? "N/A" : (char*)pThis->pszFName); - if(pThis->tOperationsMode == STREAMMODE_WRITE) - strmFlush(pThis); + if(pThis->tOperationsMode != STREAMMODE_READ) { + strmFlushInternal(pThis); + if(pThis->bAsyncWrite) { + strmWaitAsyncWriterDone(pThis); + } + } - close(pThis->fd); // TODO: error check - pThis->fd = -1; + /* the file may already be closed (or never have opened), so guard + * against this. -- rgerhards, 2010-03-19 + */ + if(pThis->fd != -1) { + close(pThis->fd); + pThis->fd = -1; + } + + if(pThis->fdDir != -1) { + /* close associated directory handle, if it is open */ + close(pThis->fdDir); + pThis->fdDir = -1; + } if(pThis->bDeleteOnClose) { - unlink((char*) pThis->pszCurrFName); // TODO: check returncode + if(unlink((char*) pThis->pszCurrFName) == -1) { + char errStr[1024]; + int err = errno; + rs_strerror_r(err, errStr, sizeof(errStr)); + DBGPRINTF("error %d unlinking '%s' - ignored: %s\n", + errno, pThis->pszCurrFName, errStr); + } } pThis->iCurrOffs = 0; /* we are back at begin of file */ @@ -231,16 +450,13 @@ strmHandleEOF(strm_t *pThis) ISOBJ_TYPE_assert(pThis, strm); switch(pThis->sType) { case STREAMTYPE_FILE_SINGLE: + case STREAMTYPE_NAMED_PIPE: ABORT_FINALIZE(RS_RET_EOF); break; case STREAMTYPE_FILE_CIRCULAR: /* we have multiple files and need to switch to the next one */ /* TODO: think about emulating EOF in this case (not yet needed) */ -#if 0 - if(pThis->iMaxFiles == 0) /* TODO: why do we need this? ;) */ - ABORT_FINALIZE(RS_RET_EOF); -#endif - dbgoprint((obj_t*) pThis, "file %d EOF\n", pThis->fd); + DBGOPRINT((obj_t*) pThis, "file %d EOF\n", pThis->fd); CHKiRet(strmNextFile(pThis)); break; case STREAMTYPE_FILE_MONITOR: @@ -272,7 +488,7 @@ strmReadBuf(strm_t *pThis) */ CHKiRet(strmOpenFile(pThis)); iLenRead = read(pThis->fd, pThis->pIOBuf, pThis->sIOBufSize); - dbgoprint((obj_t*) pThis, "file %d read %ld bytes\n", pThis->fd, iLenRead); + DBGOPRINT((obj_t*) pThis, "file %d read %ld bytes\n", pThis->fd, iLenRead); if(iLenRead == 0) { CHKiRet(strmHandleEOF(pThis)); } else if(iLenRead < 0) @@ -297,14 +513,14 @@ finalize_it: * NOTE: needs to be enhanced to support sticking with a strm entry (if not * deleted). */ -rsRetVal strmReadChar(strm_t *pThis, uchar *pC) +static rsRetVal strmReadChar(strm_t *pThis, uchar *pC) { DEFiRet; ASSERT(pThis != NULL); ASSERT(pC != NULL); - /* DEV debug only: dbgoprint((obj_t*) pThis, "strmRead index %d, max %d\n", pThis->iBufPtr, pThis->iBufPtrMax); */ + /* DEV debug only: DBGOPRINT((obj_t*) pThis, "strmRead index %d, max %d\n", pThis->iBufPtr, pThis->iBufPtrMax); */ if(pThis->iUngetC != -1) { /* do we have an "unread" char that we need to provide? */ *pC = pThis->iUngetC; ++pThis->iCurrOffs; /* one more octet read */ @@ -331,7 +547,7 @@ finalize_it: * character buffering capability. * rgerhards, 2008-01-07 */ -rsRetVal strmUnreadChar(strm_t *pThis, uchar c) +static rsRetVal strmUnreadChar(strm_t *pThis, uchar c) { ASSERT(pThis != NULL); ASSERT(pThis->iUngetC == -1); @@ -353,7 +569,7 @@ rsRetVal strmUnreadChar(strm_t *pThis, uchar c) * are pthread_killed() upon termination. So if we use their native pointer, they * can cleanup (but only then). */ -rsRetVal +static rsRetVal strmReadLine(strm_t *pThis, cstr_t **ppCStr) { DEFiRet; @@ -362,19 +578,19 @@ strmReadLine(strm_t *pThis, cstr_t **ppCStr) ASSERT(pThis != NULL); ASSERT(ppCStr != NULL); - CHKiRet(rsCStrConstruct(ppCStr)); + CHKiRet(cstrConstruct(ppCStr)); /* now read the line */ CHKiRet(strmReadChar(pThis, &c)); while(c != '\n') { - CHKiRet(rsCStrAppendChar(*ppCStr, c)); + CHKiRet(cstrAppendChar(*ppCStr, c)); CHKiRet(strmReadChar(pThis, &c)); } CHKiRet(cstrFinalize(*ppCStr)); finalize_it: if(iRet != RS_RET_OK && *ppCStr != NULL) - rsCStrDestruct(ppCStr); + cstrDestruct(ppCStr); RETiRet; } @@ -385,26 +601,78 @@ finalize_it: BEGINobjConstruct(strm) /* be sure to specify the object type also in END macro! */ pThis->iCurrFNum = 1; pThis->fd = -1; + pThis->fdDir = -1; pThis->iUngetC = -1; pThis->sType = STREAMTYPE_FILE_SINGLE; pThis->sIOBufSize = glblGetIOBufSize(); - pThis->tOpenMode = 0600; /* TODO: make configurable */ + pThis->tOpenMode = 0600; ENDobjConstruct(strm) /* ConstructionFinalizer * rgerhards, 2008-01-09 */ -rsRetVal strmConstructFinalize(strm_t *pThis) +static rsRetVal strmConstructFinalize(strm_t *pThis) { + rsRetVal localRet; + int i; DEFiRet; ASSERT(pThis != NULL); - if(pThis->pIOBuf == NULL) { /* allocate our io buffer in case we have not yet */ - if((pThis->pIOBuf = (uchar*) malloc(sizeof(uchar) * pThis->sIOBufSize)) == NULL) - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - pThis->iBufPtrMax = 0; /* results in immediate read request */ + pThis->iBufPtrMax = 0; /* results in immediate read request */ + if(pThis->iZipLevel) { /* do we need a zip buf? */ + localRet = objUse(zlibw, LM_ZLIBW_FILENAME); + if(localRet != RS_RET_OK) { + pThis->iZipLevel = 0; + DBGPRINTF("stream was requested with zip mode, but zlibw module unavailable (%d) - using " + "without zip\n", localRet); + } else { + /* we use the same size as the original buf, as we would like + * to make sure we can write out everything with a SINGLE api call! + * We add another 128 bytes to take care of the gzip header and "all eventualities". + */ + CHKmalloc(pThis->pZipBuf = (Bytef*) malloc(sizeof(uchar) * (pThis->sIOBufSize + 128))); + } + } + + /* if we are set to sync, we must obtain a file handle to the directory for fsync() purposes */ + if(pThis->bSync && !pThis->bIsTTY) { + pThis->fdDir = open((char*)pThis->pszDir, O_RDONLY | O_CLOEXEC | O_NOCTTY); + if(pThis->fdDir == -1) { + char errStr[1024]; + int err = errno; + rs_strerror_r(err, errStr, sizeof(errStr)); + DBGPRINTF("error %d opening directory file for fsync() use - fsync for directory disabled: %s\n", + errno, errStr); + } + } + + DBGPRINTF("file stream %s params: flush interval %d, async write %d\n", + (pThis->pszFName == NULL) ? "N/A" : (char*)pThis->pszFName, + pThis->iFlushInterval, pThis->bAsyncWrite); + /* if we have a flush interval, we need to do async writes in any case */ + if(pThis->iFlushInterval != 0) { + pThis->bAsyncWrite = 1; + } + + /* if we work asynchronously, we need a couple of synchronization objects */ + if(pThis->bAsyncWrite) { + pthread_mutex_init(&pThis->mut, 0); + pthread_cond_init(&pThis->notFull, 0); + pthread_cond_init(&pThis->notEmpty, 0); + pthread_cond_init(&pThis->isEmpty, 0); + pThis->iCnt = pThis->iEnq = pThis->iDeq = 0; + for(i = 0 ; i < STREAM_ASYNC_NUMBUFS ; ++i) { + CHKmalloc(pThis->asyncBuf[i].pBuf = (uchar*) malloc(sizeof(uchar) * pThis->sIOBufSize)); + } + pThis->pIOBuf = pThis->asyncBuf[0].pBuf; + pThis->bStopWriter = 0; + if(pthread_create(&pThis->writerThreadID, NULL, asyncWriterThread, pThis) != 0) + DBGPRINTF("ERROR: stream %p cold not create writer thread\n", pThis); + } else { + /* we work synchronously, so we need to alloc a fixed pIOBuf */ + CHKmalloc(pThis->pIOBuf = (uchar*) malloc(sizeof(uchar) * pThis->sIOBufSize)); } finalize_it: @@ -412,24 +680,56 @@ finalize_it: } +/* stop the writer thread (we MUST be runnnig asynchronously when this method + * is called!). Note that the mutex must be locked! -- rgerhards, 2009-07-06 + */ +static inline void +stopWriter(strm_t *pThis) +{ + BEGINfunc + pThis->bStopWriter = 1; + pthread_cond_signal(&pThis->notEmpty); + d_pthread_mutex_unlock(&pThis->mut); + pthread_join(pThis->writerThreadID, NULL); + ENDfunc +} + + /* destructor for the strm object */ BEGINobjDestruct(strm) /* be sure to specify the object type also in END and CODESTART macros! */ + int i; CODESTARTobjDestruct(strm) - if(pThis->tOperationsMode == STREAMMODE_WRITE) - strmFlush(pThis); - - /* ... then free resources */ - if(pThis->fd != -1) - strmCloseFile(pThis); + if(pThis->bAsyncWrite) + /* Note: mutex will be unlocked in stopWriter! */ + d_pthread_mutex_lock(&pThis->mut); - if(pThis->pszDir != NULL) - free(pThis->pszDir); - if(pThis->pIOBuf != NULL) + /* strmClose() will handle read-only files as well as need to open + * files that have unwritten buffers. -- rgerhards, 2010-03-09 + */ + strmCloseFile(pThis); + + if(pThis->bAsyncWrite) { + stopWriter(pThis); + pthread_mutex_destroy(&pThis->mut); + pthread_cond_destroy(&pThis->notFull); + pthread_cond_destroy(&pThis->notEmpty); + pthread_cond_destroy(&pThis->isEmpty); + for(i = 0 ; i < STREAM_ASYNC_NUMBUFS ; ++i) { + free(pThis->asyncBuf[i].pBuf); + } + } else { free(pThis->pIOBuf); - if(pThis->pszCurrFName != NULL) - free(pThis->pszCurrFName); - if(pThis->pszFName != NULL) - free(pThis->pszFName); + } + + /* Finally, we can free the resources. + * IMPORTANT: we MUST free this only AFTER the ansyncWriter has been stopped, else + * we get random errors... + */ + free(pThis->pszDir); + free(pThis->pZipBuf); + free(pThis->pszCurrFName); + free(pThis->pszFName); + pThis->bStopWriter = 2; /* RG: use as flag for destruction */ ENDobjDestruct(strm) @@ -445,8 +745,11 @@ static rsRetVal strmCheckNextOutputFile(strm_t *pThis) if(pThis->fd == -1) FINALIZE; + /* wait for output to be empty, so that our counts are correct */ + strmWaitAsyncWriterDone(pThis); + if(pThis->iCurrOffs >= pThis->iMaxFileSize) { - dbgoprint((obj_t*) pThis, "max file size %ld reached for %d, now %ld - starting new file\n", + DBGOPRINT((obj_t*) pThis, "max file size %ld reached for %d, now %ld - starting new file\n", (long) pThis->iMaxFileSize, pThis->fd, (long) pThis->iCurrOffs); CHKiRet(strmNextFile(pThis)); } @@ -455,47 +758,387 @@ finalize_it: RETiRet; } + +/* try to recover a tty after a write error. This may have happend + * due to vhangup(), and, if so, we can simply re-open it. + */ +#ifdef linux +# define ERR_TTYHUP EIO +#else +# define ERR_TTYHUP EBADF +#endif +static rsRetVal +tryTTYRecover(strm_t *pThis, int err) +{ + DEFiRet; + ISOBJ_TYPE_assert(pThis, strm); + if(err == ERR_TTYHUP) { + close(pThis->fd); + CHKiRet(doPhysOpen(pThis)); + } + +finalize_it: + RETiRet; +} +#undef ER_TTYHUP + + +/* issue write() api calls until either the buffer is completely + * written or an error occured (it may happen that multiple writes + * are required, what is perfectly legal. On exit, *pLenBuf contains + * the number of bytes actually written. + * rgerhards, 2009-06-08 + */ +static rsRetVal +doWriteCall(strm_t *pThis, uchar *pBuf, size_t *pLenBuf) +{ + ssize_t lenBuf; + ssize_t iTotalWritten; + ssize_t iWritten; + char *pWriteBuf; + DEFiRet; + ISOBJ_TYPE_assert(pThis, strm); + + lenBuf = *pLenBuf; + pWriteBuf = (char*) pBuf; + iTotalWritten = 0; + do { + iWritten = write(pThis->fd, pWriteBuf, lenBuf); + if(iWritten < 0) { + char errStr[1024]; + int err = errno; + iWritten = 0; /* we have written NO bytes! */ + rs_strerror_r(err, errStr, sizeof(errStr)); + DBGPRINTF("log file (%d) write error %d: %s\n", pThis->fd, err, errStr); + if(err == EINTR) { + /*NO ERROR, just continue */; + } else { + if(pThis->bIsTTY) { + CHKiRet(tryTTYRecover(pThis, err)); + } else { + ABORT_FINALIZE(RS_RET_IO_ERROR); + /* Would it make sense to cover more error cases? So far, I + * do not see good reason to do so. + */ + } + } + } + /* advance buffer to next write position */ + iTotalWritten += iWritten; + lenBuf -= iWritten; + pWriteBuf += iWritten; + } while(lenBuf > 0); /* Warning: do..while()! */ + + DBGOPRINT((obj_t*) pThis, "file %d write wrote %d bytes\n", pThis->fd, (int) iWritten); + +finalize_it: + *pLenBuf = iTotalWritten; + RETiRet; +} + + + /* write memory buffer to a stream object. - * To support direct writes of large objects, this method may be called - * with a buffer pointing to some region other than the stream buffer itself. - * However, in that case the stream buffer must be empty (strmFlush() has to - * be called before), because we would otherwise mess up with the sequence - * inside the stream. -- rgerhards, 2008-01-10 */ -static rsRetVal strmWriteInternal(strm_t *pThis, uchar *pBuf, size_t lenBuf) +static inline rsRetVal +doWriteInternal(strm_t *pThis, uchar *pBuf, size_t lenBuf) +{ + DEFiRet; + + ASSERT(pThis != NULL); + + if(pThis->iZipLevel) { + CHKiRet(doZipWrite(pThis, pBuf, lenBuf)); + } else { + /* write without zipping */ + CHKiRet(strmPhysWrite(pThis, pBuf, lenBuf)); + } + +finalize_it: + RETiRet; +} + + +/* This function is called to "do" an async write call, what primarily means that + * the data is handed over to the writer thread (which will then do the actual write + * in parallel). Note that the stream mutex has already been locked by the + * strmWrite...() calls. Also note that we always have only a single producer, + * so we can simply serially assign the next free buffer to it and be sure that + * the very some producer comes back in sequence to submit the then-filled buffers. + * This also enables us to timout on partially written buffers. -- rgerhards, 2009-07-06 + */ +static inline rsRetVal +doAsyncWriteInternal(strm_t *pThis, size_t lenBuf) +{ + DEFiRet; + ISOBJ_TYPE_assert(pThis, strm); + + /* the -1 below is important, because we need one buffer for the main thread! */ + while(pThis->iCnt >= STREAM_ASYNC_NUMBUFS - 1) + d_pthread_cond_wait(&pThis->notFull, &pThis->mut); + + pThis->asyncBuf[pThis->iEnq % STREAM_ASYNC_NUMBUFS].lenBuf = lenBuf; + pThis->pIOBuf = pThis->asyncBuf[++pThis->iEnq % STREAM_ASYNC_NUMBUFS].pBuf; + + pThis->bDoTimedWait = 0; /* everything written, no need to timeout partial buffer writes */ + if(++pThis->iCnt == 1) + pthread_cond_signal(&pThis->notEmpty); + + RETiRet; +} + + +/* schedule writing to the stream. Depending on our concurrency settings, + * this either directly writes to the stream or schedules writing via + * the background thread. -- rgerhards, 2009-07-07 + */ +static rsRetVal +strmSchedWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) { DEFiRet; - int iWritten; ASSERT(pThis != NULL); - ASSERT(pBuf == pThis->pIOBuf || pThis->iBufPtr == 0); + + /* we need to reset the buffer pointer BEFORE calling the actual write + * function. Otherwise, in circular mode, the write function will + * potentially close the file, then close will flush and as the + * buffer pointer is nonzero, will re-call into this code here. In + * the end result, we than have a problem (and things are screwed + * up). So we reset the buffer pointer first, and all this can + * not happen. It is safe to do so, because that pointer is NOT + * used inside the write functions. -- rgerhads, 2010-03-10 + */ + pThis->iBufPtr = 0; /* we are at the begin of a new buffer */ + if(pThis->bAsyncWrite) { + CHKiRet(doAsyncWriteInternal(pThis, lenBuf)); + } else { + CHKiRet(doWriteInternal(pThis, pBuf, lenBuf)); + } + + +finalize_it: + RETiRet; +} + + + +/* This is the writer thread for asynchronous mode. + * -- rgerhards, 2009-07-06 + */ +static void* +asyncWriterThread(void *pPtr) +{ + int iDeq; + struct timespec t; + bool bTimedOut = 0; + strm_t *pThis = (strm_t*) pPtr; + ISOBJ_TYPE_assert(pThis, strm); + + BEGINfunc +# if HAVE_PRCTL && defined PR_SET_NAME + if(prctl(PR_SET_NAME, "rs:asyn strmwr", 0, 0, 0) != 0) { + DBGPRINTF("prctl failed, not setting thread name for '%s'\n", "stream writer"); + } +# endif + + while(1) { /* loop broken inside */ + d_pthread_mutex_lock(&pThis->mut); +dbgprintf("XXX: asyncWriterThread iterating %s\n", pThis->pszFName); + while(pThis->iCnt == 0) { + if(pThis->bStopWriter) { + pthread_cond_broadcast(&pThis->isEmpty); + d_pthread_mutex_unlock(&pThis->mut); + goto finalize_it; /* break main loop */ + } + if(bTimedOut && pThis->iBufPtr > 0) { + /* if we timed out, we need to flush pending data */ + strmFlushInternal(pThis); + bTimedOut = 0; + continue; /* now we should have data */ + } + bTimedOut = 0; + timeoutComp(&t, pThis->iFlushInterval * 2000); /* *1000 millisconds */ // TODO: check the 2000?!? + if(pThis->bDoTimedWait) { +dbgprintf("asyncWriter thread going to timeout sleep\n"); + if(pthread_cond_timedwait(&pThis->notEmpty, &pThis->mut, &t) != 0) { + int err = errno; + if(err == ETIMEDOUT) { + bTimedOut = 1; + } else { + bTimedOut = 1; + char errStr[1024]; + rs_strerror_r(err, errStr, sizeof(errStr)); + DBGPRINTF("stream async writer timeout with error (%d): %s - ignoring\n", + err, errStr); + } + } + } else { +dbgprintf("asyncWriter thread going to eternal sleep\n"); + d_pthread_cond_wait(&pThis->notEmpty, &pThis->mut); + } +dbgprintf("asyncWriter woke up\n"); + } + + bTimedOut = 0; /* we may have timed out, but there *is* work to do... */ + + iDeq = pThis->iDeq++ % STREAM_ASYNC_NUMBUFS; +dbgprintf("asyncWriter writes data\n"); + doWriteInternal(pThis, pThis->asyncBuf[iDeq].pBuf, pThis->asyncBuf[iDeq].lenBuf); + // TODO: error check????? 2009-07-06 + + --pThis->iCnt; + if(pThis->iCnt < STREAM_ASYNC_NUMBUFS) { + pthread_cond_signal(&pThis->notFull); + if(pThis->iCnt == 0) + pthread_cond_broadcast(&pThis->isEmpty); + } + d_pthread_mutex_unlock(&pThis->mut); + } + +finalize_it: + ENDfunc + return NULL; /* to keep pthreads happy */ +} + + +/* sync the file to disk, so that any unwritten data is persisted. This + * also syncs the directory and thus makes sure that the file survives + * fatal failure. Note that we do NOT return an error status if the + * sync fails. Doing so would probably cause more trouble than it + * is worth (read: data loss may occur where we otherwise might not + * have it). -- rgerhards, 2009-06-08 + */ +#undef SYNCCALL +#if HAVE_FDATASYNC +# define SYNCCALL(x) fdatasync(x) +#else +# define SYNCCALL(x) fsync(x) +#endif +static rsRetVal +syncFile(strm_t *pThis) +{ + int ret; + DEFiRet; + + if(pThis->bIsTTY) + FINALIZE; /* TTYs can not be synced */ + + DBGPRINTF("syncing file %d\n", pThis->fd); + ret = SYNCCALL(pThis->fd); + if(ret != 0) { + char errStr[1024]; + int err = errno; + rs_strerror_r(err, errStr, sizeof(errStr)); + DBGPRINTF("sync failed for file %d with error (%d): %s - ignoring\n", + pThis->fd, err, errStr); + } + + if(pThis->fdDir != -1) { + ret = fsync(pThis->fdDir); + } + +finalize_it: + RETiRet; +} +#undef SYNCCALL + +/* physically write to the output file. the provided data is ready for + * writing (e.g. zipped if we are requested to do that). + * Note that if the write() API fails, we do not reset any pointers, but return + * an error code. That means we may redo work in the next iteration. + * rgerhards, 2009-06-04 + */ +static rsRetVal +strmPhysWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) +{ + size_t iWritten; + DEFiRet; + ISOBJ_TYPE_assert(pThis, strm); if(pThis->fd == -1) CHKiRet(strmOpenFile(pThis)); - iWritten = write(pThis->fd, pBuf, lenBuf); - dbgoprint((obj_t*) pThis, "file %d write wrote %d bytes\n", pThis->fd, iWritten); - /* TODO: handle error case -- rgerhards, 2008-01-07 */ - - /* Now indicate buffer empty again. We do this in any case, because there - * is no way we could react more intelligently to an error during write. - * This MUST be done BEFORE strCheckNextOutputFile(), otherwise we have an - * endless loop. We reset the buffer pointer also in finalize_it - this is - * necessary if we run into problems. Not resetting it would again cause an - * endless loop. So it is better to loose some data (which also justifies - * duplicating that code, too...) -- rgerhards, 2008-01-10 - */ - pThis->iBufPtr = 0; + iWritten = lenBuf; + CHKiRet(doWriteCall(pThis, pBuf, &iWritten)); + pThis->iCurrOffs += iWritten; /* update user counter, if provided */ if(pThis->pUsrWCntr != NULL) *pThis->pUsrWCntr += iWritten; - if(pThis->sType == STREAMTYPE_FILE_CIRCULAR) + if(pThis->bSync) { + CHKiRet(syncFile(pThis)); + } + + if(pThis->sType == STREAMTYPE_FILE_CIRCULAR) { CHKiRet(strmCheckNextOutputFile(pThis)); + } else if(pThis->iSizeLimit != 0) { + CHKiRet(doSizeLimitProcessing(pThis)); + } + +finalize_it: + RETiRet; +} + + +/* write the output buffer in zip mode + * This means we compress it first and then do a physical write. + * Note that we always do a full deflateInit ... deflate ... deflateEnd + * sequence. While this is not optimal, we need to do it because we need + * to ensure that the file is readable even when we are aborted. Doing the + * full sequence brings us as far towards this goal as possible (and not + * doing it would be a total failure). It may be worth considering to + * add a config switch so that the user can decide the risk he is ready + * to take, but so far this is not yet implemented (not even requested ;)). + * rgerhards, 2009-06-04 + */ +static rsRetVal +doZipWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) +{ + z_stream zstrm; + int zRet; /* zlib return state */ + bool bzInitDone = FALSE; + DEFiRet; + assert(pThis != NULL); + assert(pBuf != NULL); + + /* allocate deflate state */ + zstrm.zalloc = Z_NULL; + zstrm.zfree = Z_NULL; + zstrm.opaque = Z_NULL; + zstrm.next_in = (Bytef*) pBuf; /* as of zlib doc, this must be set BEFORE DeflateInit2 */ + /* see note in file header for the params we use with deflateInit2() */ + zRet = zlibw.DeflateInit2(&zstrm, pThis->iZipLevel, Z_DEFLATED, 31, 9, Z_DEFAULT_STRATEGY); + if(zRet != Z_OK) { + DBGPRINTF("error %d returned from zlib/deflateInit2()\n", zRet); + ABORT_FINALIZE(RS_RET_ZLIB_ERR); + } + bzInitDone = TRUE; + + /* now doing the compression */ + zstrm.next_in = (Bytef*) pBuf; /* as of zlib doc, this must be set BEFORE DeflateInit2 */ + zstrm.avail_in = lenBuf; + /* run deflate() on buffer until everything has been compressed */ + do { + DBGPRINTF("in deflate() loop, avail_in %d, total_in %ld\n", zstrm.avail_in, zstrm.total_in); + zstrm.avail_out = pThis->sIOBufSize; + zstrm.next_out = pThis->pZipBuf; + zRet = zlibw.Deflate(&zstrm, Z_FINISH); /* no bad return value */ + DBGPRINTF("after deflate, ret %d, avail_out %d\n", zRet, zstrm.avail_out); + assert(zRet != Z_STREAM_ERROR); /* state not clobbered */ + if(zstrm.avail_out == pThis->sIOBufSize) + break; /* this is valid, indicates end of compression --> see zlib howto */ + CHKiRet(strmPhysWrite(pThis, (uchar*)pThis->pZipBuf, pThis->sIOBufSize - zstrm.avail_out)); + } while (zstrm.avail_out == 0); + assert(zstrm.avail_in == 0); /* all input will be used */ finalize_it: - pThis->iBufPtr = 0; /* see comment above */ + if(bzInitDone) { + zRet = zlibw.DeflateEnd(&zstrm); + if(zRet != Z_OK) { + DBGPRINTF("error %d returned from zlib/deflateEnd()\n", zRet); + } + } RETiRet; } @@ -505,21 +1148,49 @@ finalize_it: * and is automatically called when the output buffer is full. * rgerhards, 2008-01-10 */ -rsRetVal strmFlush(strm_t *pThis) +static rsRetVal +strmFlushInternal(strm_t *pThis) { DEFiRet; ASSERT(pThis != NULL); - dbgoprint((obj_t*) pThis, "file %d flush, buflen %ld\n", pThis->fd, (long) pThis->iBufPtr); + DBGOPRINT((obj_t*) pThis, "file %d(%s) flush, buflen %ld%s\n", pThis->fd, + (pThis->pszFName == NULL) ? "N/A" : (char*)pThis->pszFName, + (long) pThis->iBufPtr, (pThis->iBufPtr == 0) ? " (no need to flush)" : ""); - if(pThis->tOperationsMode == STREAMMODE_WRITE && pThis->iBufPtr > 0) { - iRet = strmWriteInternal(pThis, pThis->pIOBuf, pThis->iBufPtr); + if(pThis->tOperationsMode != STREAMMODE_READ && pThis->iBufPtr > 0) { + iRet = strmSchedWrite(pThis, pThis->pIOBuf, pThis->iBufPtr); } RETiRet; } +/* flush stream output buffer to persistent storage. This can be called at any time + * and is automatically called when the output buffer is full. This function is for + * use by EXTERNAL callers. Do NOT use it internally. It locks the async writer + * mutex if ther is need to do so. + * rgerhards, 2010-03-18 + */ +static rsRetVal +strmFlush(strm_t *pThis) +{ + DEFiRet; + + ASSERT(pThis != NULL); + + if(pThis->bAsyncWrite) + d_pthread_mutex_lock(&pThis->mut); + CHKiRet(strmFlushInternal(pThis)); + +finalize_it: + if(pThis->bAsyncWrite) + d_pthread_mutex_unlock(&pThis->mut); + + RETiRet; +} + + /* seek a stream to a specific location. Pending writes are flushed, read data * is invalidated. * rgerhards, 2008-01-12 @@ -533,10 +1204,10 @@ static rsRetVal strmSeek(strm_t *pThis, off64_t offs) if(pThis->fd == -1) strmOpenFile(pThis); else - strmFlush(pThis); - int64 i; - dbgoprint((obj_t*) pThis, "file %d seek, pos %ld\n", pThis->fd, (long) offs); - i = lseek64(pThis->fd, offs, SEEK_SET); + strmFlushInternal(pThis); + long long i; + DBGOPRINT((obj_t*) pThis, "file %d seek, pos %llu\n", pThis->fd, (long long unsigned) offs); + i = lseek64(pThis->fd, offs, SEEK_SET); // TODO: check error! pThis->iCurrOffs = offs; /* we are now at *this* offset */ pThis->iBufPtr = 0; /* buffer invalidated */ @@ -547,7 +1218,7 @@ static rsRetVal strmSeek(strm_t *pThis, off64_t offs) /* seek to current offset. This is primarily a helper to readjust the OS file * pointer after a strm object has been deserialized. */ -rsRetVal strmSeekCurrOffs(strm_t *pThis) +static rsRetVal strmSeekCurrOffs(strm_t *pThis) { DEFiRet; @@ -560,27 +1231,40 @@ rsRetVal strmSeekCurrOffs(strm_t *pThis) /* write a *single* character to a stream object -- rgerhards, 2008-01-10 */ -rsRetVal strmWriteChar(strm_t *pThis, uchar c) +static rsRetVal strmWriteChar(strm_t *pThis, uchar c) { DEFiRet; ASSERT(pThis != NULL); + if(pThis->bAsyncWrite) + d_pthread_mutex_lock(&pThis->mut); + + if(pThis->bDisabled) + ABORT_FINALIZE(RS_RET_STREAM_DISABLED); + /* if the buffer is full, we need to flush before we can write */ if(pThis->iBufPtr == pThis->sIOBufSize) { - CHKiRet(strmFlush(pThis)); + CHKiRet(strmFlushInternal(pThis)); } /* we now always have space for one character, so we simply copy it */ *(pThis->pIOBuf + pThis->iBufPtr) = c; pThis->iBufPtr++; finalize_it: + if(pThis->bAsyncWrite) + d_pthread_mutex_unlock(&pThis->mut); + RETiRet; } -/* write an integer value (actually a long) to a stream object */ -rsRetVal strmWriteLong(strm_t *pThis, long i) +/* write an integer value (actually a long) to a stream object + * Note that we do not need to lock the mutex here, because we call + * strmWrite(), which does the lock (aka: we must not lock it, else we + * would run into a recursive lock, resulting in a deadlock!) + */ +static rsRetVal strmWriteLong(strm_t *pThis, long i) { DEFiRet; uchar szBuf[32]; @@ -595,45 +1279,72 @@ finalize_it: } -/* write memory buffer to a stream object +/* write memory buffer to a stream object. + * process the data in chunks and copy it over to our buffer. The caller-provided data + * may theoritically be larger than our buffer. In that case, we do multiple copies. One + * may argue if it were more efficient to write out the caller-provided buffer in that case + * and earlier versions of rsyslog did this. However, this introduces a lot of complexity + * inside the buffered writer and potential performance bottlenecks when trying to solve + * it. Now keep in mind that we actually do (almost?) never have a case where the + * caller-provided buffer is larger than our one. So instead of optimizing a case + * which normally does not exist, we expect some degradation in its case but make us + * perform better in the regular cases. -- rgerhards, 2009-07-07 + * Note: the pThis->iBufPtr == pThis->sIOBufSize logic below looks a bit like an + * on-off error. In fact, it is not, because iBufPtr always points to the next + * *free* byte in the buffer. So if it is sIOBufSize - 1, there actually is one + * free byte left. This came up during a code walkthrough and was considered + * worth nothing. -- rgerhards, 2010-03-10 */ -rsRetVal strmWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) +static rsRetVal +strmWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) { DEFiRet; - size_t iPartial; + size_t iWrite; + size_t iOffset; ASSERT(pThis != NULL); ASSERT(pBuf != NULL); - /* check if the to-be-written data is larger than our buffer size */ - if(lenBuf >= pThis->sIOBufSize) { - /* it is - so we do a direct write, that is most efficient. - * TODO: is it really? think about disk block sizes! - */ - CHKiRet(strmFlush(pThis)); /* we need to flush first!!! */ - CHKiRet(strmWriteInternal(pThis, pBuf, lenBuf)); - } else { - /* data fits into a buffer - we just need to see if it - * fits into the current buffer... - */ - if(pThis->iBufPtr + lenBuf > pThis->sIOBufSize) { - /* nope, so we must split it */ - iPartial = pThis->sIOBufSize - pThis->iBufPtr; /* this fits in current buf */ - if(iPartial > 0) { /* the buffer was exactly full, can not write anything! */ - memcpy(pThis->pIOBuf + pThis->iBufPtr, pBuf, iPartial); - pThis->iBufPtr += iPartial; - } - CHKiRet(strmFlush(pThis)); /* get a new buffer for rest of data */ - memcpy(pThis->pIOBuf, pBuf + iPartial, lenBuf - iPartial); - pThis->iBufPtr = lenBuf - iPartial; - } else { - /* we have space, so we simply copy over the string */ - memcpy(pThis->pIOBuf + pThis->iBufPtr, pBuf, lenBuf); - pThis->iBufPtr += lenBuf; +//DBGPRINTF("strmWrite(%p, '%65.65s', %ld);, disabled %d, sizelim %ld, size %lld\n", pThis, pBuf,lenBuf, pThis->bDisabled, pThis->iSizeLimit, pThis->iCurrOffs); + if(pThis->bAsyncWrite) + d_pthread_mutex_lock(&pThis->mut); + + if(pThis->bDisabled) + ABORT_FINALIZE(RS_RET_STREAM_DISABLED); + + iOffset = 0; + do { + if(pThis->iBufPtr == pThis->sIOBufSize) { + CHKiRet(strmFlushInternal(pThis)); /* get a new buffer for rest of data */ } + iWrite = pThis->sIOBufSize - pThis->iBufPtr; /* this fits in current buf */ + if(iWrite > lenBuf) + iWrite = lenBuf; + memcpy(pThis->pIOBuf + pThis->iBufPtr, pBuf + iOffset, iWrite); + pThis->iBufPtr += iWrite; + iOffset += iWrite; + lenBuf -= iWrite; + } while(lenBuf > 0); + + /* now check if the buffer right at the end of the write is full and, if so, + * write it. This seems more natural than waiting (hours?) for the next message... + */ + if(pThis->iBufPtr == pThis->sIOBufSize) { + CHKiRet(strmFlushInternal(pThis)); /* get a new buffer for rest of data */ } finalize_it: + if(pThis->bAsyncWrite) { + if(pThis->bDoTimedWait == 0) { + /* we potentially have a partial buffer, so re-activate the + * writer thread that it can set and pick up timeouts. + */ + pThis->bDoTimedWait = 1; + pthread_cond_signal(&pThis->notEmpty); + } + d_pthread_mutex_unlock(&pThis->mut); + } + RETiRet; } @@ -646,34 +1357,27 @@ DEFpropSetMeth(strm, iFileNumDigits, int) DEFpropSetMeth(strm, tOperationsMode, int) DEFpropSetMeth(strm, tOpenMode, mode_t) DEFpropSetMeth(strm, sType, strmType_t) - -rsRetVal strmSetiMaxFiles(strm_t *pThis, int iNewVal) +DEFpropSetMeth(strm, iZipLevel, int) +DEFpropSetMeth(strm, bSync, int) +DEFpropSetMeth(strm, sIOBufSize, size_t) +DEFpropSetMeth(strm, iSizeLimit, off_t) +DEFpropSetMeth(strm, iFlushInterval, int) +DEFpropSetMeth(strm, pszSizeLimitCmd, uchar*) + +static rsRetVal strmSetiMaxFiles(strm_t *pThis, int iNewVal) { pThis->iMaxFiles = iNewVal; pThis->iFileNumDigits = getNumberDigits(iNewVal); return RS_RET_OK; } -rsRetVal strmSetiAddtlOpenFlags(strm_t *pThis, int iNewVal) -{ - DEFiRet; - - if(iNewVal & O_APPEND) - ABORT_FINALIZE(RS_RET_PARAM_ERROR); - - pThis->iAddtlOpenFlags = iNewVal; - -finalize_it: - RETiRet; -} - /* set the stream's file prefix * The passed-in string is duplicated. So if the caller does not need * it any longer, it must free it. * rgerhards, 2008-01-09 */ -rsRetVal +static rsRetVal strmSetFName(strm_t *pThis, uchar *pszName, size_t iLenName) { DEFiRet; @@ -687,7 +1391,7 @@ strmSetFName(strm_t *pThis, uchar *pszName, size_t iLenName) if(pThis->pszFName != NULL) free(pThis->pszFName); - if((pThis->pszFName = malloc(sizeof(uchar) * iLenName + 1)) == NULL) + if((pThis->pszFName = malloc(sizeof(uchar) * (iLenName + 1))) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); memcpy(pThis->pszFName, pszName, iLenName + 1); /* always think about the \0! */ @@ -703,7 +1407,7 @@ finalize_it: * it any longer, it must free it. * rgerhards, 2008-01-09 */ -rsRetVal +static rsRetVal strmSetDir(strm_t *pThis, uchar *pszDir, size_t iLenDir) { DEFiRet; @@ -714,8 +1418,7 @@ strmSetDir(strm_t *pThis, uchar *pszDir, size_t iLenDir) if(iLenDir < 1) ABORT_FINALIZE(RS_RET_FILE_PREFIX_MISSING); - if((pThis->pszDir = malloc(sizeof(uchar) * iLenDir + 1)) == NULL) - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + CHKmalloc(pThis->pszDir = malloc(sizeof(uchar) * iLenDir + 1)); memcpy(pThis->pszDir, pszDir, iLenDir + 1); /* always think about the \0! */ pThis->lenDir = iLenDir; @@ -747,7 +1450,7 @@ finalize_it: * * rgerhards, 2008-01-10 */ -rsRetVal strmRecordBegin(strm_t *pThis) +static rsRetVal strmRecordBegin(strm_t *pThis) { ASSERT(pThis != NULL); ASSERT(pThis->bInRecord == 0); @@ -755,7 +1458,7 @@ rsRetVal strmRecordBegin(strm_t *pThis) return RS_RET_OK; } -rsRetVal strmRecordEnd(strm_t *pThis) +static rsRetVal strmRecordEnd(strm_t *pThis) { DEFiRet; ASSERT(pThis != NULL); @@ -777,7 +1480,7 @@ rsRetVal strmRecordEnd(strm_t *pThis) * We do not serialize the dynamic properties. * rgerhards, 2008-01-10 */ -rsRetVal strmSerialize(strm_t *pThis, strm_t *pStrm) +static rsRetVal strmSerialize(strm_t *pThis, strm_t *pStrm) { DEFiRet; int i; @@ -786,7 +1489,7 @@ rsRetVal strmSerialize(strm_t *pThis, strm_t *pStrm) ISOBJ_TYPE_assert(pThis, strm); ISOBJ_TYPE_assert(pStrm, strm); - strmFlush(pThis); + strmFlushInternal(pThis); CHKiRet(obj.BeginSerialize(pStrm, (obj_t*) pThis)); objSerializeSCALAR(pStrm, iCurrFNum, INT); @@ -823,7 +1526,7 @@ finalize_it: * any new set overwrites the previous one. * rgerhards, 2008-02-27 */ -rsRetVal +static rsRetVal strmSetWCntr(strm_t *pThis, number_t *pWCnt) { DEFiRet; @@ -843,8 +1546,8 @@ strmSetWCntr(strm_t *pThis, number_t *pWCnt) /* This function can be used as a generic way to set properties. * rgerhards, 2008-01-11 */ -#define isProp(name) !rsCStrSzStrCmp(pProp->pcsName, (uchar*) name, sizeof(name) - 1) -rsRetVal strmSetProperty(strm_t *pThis, var_t *pProp) +#define isProp(name) !rsCStrSzStrCmp(pProp->pcsName, UCHAR_CONSTANT(name), sizeof(name) - 1) +static rsRetVal strmSetProperty(strm_t *pThis, var_t *pProp) { DEFiRet; @@ -883,7 +1586,7 @@ finalize_it: * reported on the second call may actually be lower than on the first call. This is due to * file circulation. A caller must deal with that. -- rgerhards, 2008-01-30 */ -rsRetVal +static rsRetVal strmGetCurrOffset(strm_t *pThis, int64 *pOffs) { DEFiRet; @@ -911,8 +1614,38 @@ CODESTARTobjQueryInterface(strm) * work here (if we can support an older interface version - that, * of course, also affects the "if" above). */ - /*xxxpIf->oID = OBJvm; SAMPLE */ - + pIf->Construct = strmConstruct; + pIf->ConstructFinalize = strmConstructFinalize; + pIf->Destruct = strmDestruct; + pIf->ReadChar = strmReadChar; + pIf->UnreadChar = strmUnreadChar; + pIf->ReadLine = strmReadLine; + pIf->SeekCurrOffs = strmSeekCurrOffs; + pIf->Write = strmWrite; + pIf->WriteChar = strmWriteChar; + pIf->WriteLong = strmWriteLong; + pIf->SetFName = strmSetFName; + pIf->SetDir = strmSetDir; + pIf->Flush = strmFlush; + pIf->RecordBegin = strmRecordBegin; + pIf->RecordEnd = strmRecordEnd; + pIf->Serialize = strmSerialize; + pIf->GetCurrOffset = strmGetCurrOffset; + pIf->SetWCntr = strmSetWCntr; + /* set methods */ + pIf->SetbDeleteOnClose = strmSetbDeleteOnClose; + pIf->SetiMaxFileSize = strmSetiMaxFileSize; + pIf->SetiMaxFiles = strmSetiMaxFiles; + pIf->SetiFileNumDigits = strmSetiFileNumDigits; + pIf->SettOperationsMode = strmSettOperationsMode; + pIf->SettOpenMode = strmSettOpenMode; + pIf->SetsType = strmSetsType; + pIf->SetiZipLevel = strmSetiZipLevel; + pIf->SetbSync = strmSetbSync; + pIf->SetsIOBufSize = strmSetsIOBufSize; + pIf->SetiSizeLimit = strmSetiSizeLimit; + pIf->SetiFlushInterval = strmSetiFlushInterval; + pIf->SetpszSizeLimitCmd = strmSetpszSizeLimitCmd; finalize_it: ENDobjQueryInterface(strm) @@ -929,7 +1662,5 @@ BEGINObjClassInit(strm, 1, OBJ_IS_CORE_MODULE) OBJSetMethodHandler(objMethod_CONSTRUCTION_FINALIZER, strmConstructFinalize); ENDObjClassInit(strm) - -/* - * vi:set ai: +/* vi:set ai: */ diff --git a/runtime/stream.h b/runtime/stream.h index 371358ab..369d5a0f 100644 --- a/runtime/stream.h +++ b/runtime/stream.h @@ -19,7 +19,29 @@ * can easily be persistet. The bottom line is that it makes much sense to * use this class whereever possible as its features may grow in the future. * - * Copyright 2008 Rainer Gerhards and Adiscon GmbH. + * An important note on writing gzip format via zlib (kept anonymous + * by request): + * + * -------------------------------------------------------------------------- + * We'd like to make sure the output file is in full gzip format + * (compatible with gzip -d/zcat etc). There is a flag in how the output + * is initialized within zlib to properly add the gzip wrappers to the + * output. (gzip is effectively a small metadata wrapper around raw + * zstream output.) + * + * I had written an old bit of code to do this - the documentation on + * deflatInit2() was pretty tricky to nail down on this specific feature: + * + * int deflateInit2 (z_streamp strm, int level, int method, int windowBits, + * int memLevel, int strategy); + * + * I believe "31" would be the value for the "windowBits" field that you'd + * want to try: + * + * deflateInit2(zstrmptr, 6, Z_DEFLATED, 31, 9, Z_DEFAULT_STRATEGY); + * -------------------------------------------------------------------------- + * + * Copyright 2008, 2009 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -47,20 +69,26 @@ #include "obj-types.h" #include "glbl.h" #include "stream.h" +#include "zlibw.h" +#include "apc.h" /* stream types */ typedef enum { STREAMTYPE_FILE_SINGLE = 0, /**< read a single file */ STREAMTYPE_FILE_CIRCULAR = 1, /**< circular files */ - STREAMTYPE_FILE_MONITOR = 2 /**< monitor a (third-party) file */ + STREAMTYPE_FILE_MONITOR = 2, /**< monitor a (third-party) file */ + STREAMTYPE_NAMED_PIPE = 3 /**< file is a named pipe (so far, tested for output only) */ } strmType_t; -typedef enum { +typedef enum { /* when extending, do NOT change existing modes! */ STREAMMMODE_INVALID = 0, STREAMMODE_READ = 1, - STREAMMODE_WRITE = 2 + STREAMMODE_WRITE = 2, + STREAMMODE_WRITE_TRUNC = 3, + STREAMMODE_WRITE_APPEND = 4 } strmMode_t; +#define STREAM_ASYNC_NUMBUFS 2 /* must be a power of 2 -- TODO: make configurable */ /* The strm_t data structure */ typedef struct strm_s { BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ @@ -71,61 +99,94 @@ typedef struct strm_s { int lenFName; strmMode_t tOperationsMode; mode_t tOpenMode; - int iAddtlOpenFlags; /* can be used to specifiy additional (compatible!) open flags */ int64 iMaxFileSize;/* maximum size a file may grow to */ int iMaxFiles; /* maximum number of files if a circular mode is in use */ int iFileNumDigits;/* min number of digits to use in file number (only in circular mode) */ - int bDeleteOnClose; /* set to 1 to auto-delete on close -- be careful with that setting! */ + bool bDeleteOnClose; /* set to 1 to auto-delete on close -- be careful with that setting! */ int64 iCurrOffs;/* current offset */ int64 *pUsrWCntr; /* NULL or a user-provided counter that receives the nbr of bytes written since the last CntrSet() */ /* dynamic properties, valid only during file open, not to be persistet */ - size_t sIOBufSize;/* size of IO buffer */ + bool bDisabled; /* should file no longer be written to? (currently set only if omfile file size limit fails) */ + bool bSync; /* sync this file after every write? */ + size_t sIOBufSize;/* size of IO buffer */ uchar *pszDir; /* Directory */ int lenDir; int fd; /* the file descriptor, -1 if closed */ + int fdDir; /* the directory's descriptor, in case bSync is requested (-1 if closed) */ uchar *pszCurrFName; /* name of current file (if open) */ - uchar *pIOBuf; /* io Buffer */ + uchar *pIOBuf; /* the iobuffer currently in use to gather data */ size_t iBufPtrMax; /* current max Ptr in Buffer (if partial read!) */ size_t iBufPtr; /* pointer into current buffer */ int iUngetC; /* char set via UngetChar() call or -1 if none set */ - int bInRecord; /* if 1, indicates that we are currently writing a not-yet complete record */ + bool bInRecord; /* if 1, indicates that we are currently writing a not-yet complete record */ + int iZipLevel; /* zip level (0..9). If 0, zip is completely disabled */ + Bytef *pZipBuf; + /* support for async flush procesing */ + bool bAsyncWrite; /* do asynchronous writes (always if a flush interval is given) */ + bool bStopWriter; /* shall writer thread terminate? */ + bool bDoTimedWait; /* instruct writer thread to do a times wait to support flush timeouts */ + int iFlushInterval; /* flush in which interval - 0, no flushing */ + apc_id_t apcID; /* id of current Apc request (used for cancelling) */ + pthread_mutex_t mut;/* mutex for flush in async mode */ + pthread_cond_t notFull; + pthread_cond_t notEmpty; + pthread_cond_t isEmpty; + unsigned short iEnq; /* this MUST be unsigned as we use module arithmetic (else invalid indexing happens!) */ + unsigned short iDeq; /* this MUST be unsigned as we use module arithmetic (else invalid indexing happens!) */ + short iCnt; /* current nbr of elements in buffer */ + struct { + uchar *pBuf; + size_t lenBuf; + } asyncBuf[STREAM_ASYNC_NUMBUFS]; + pthread_t writerThreadID; + int apcRequested; /* is an apc Requested? */ + /* support for omfile size-limiting commands, special counters, NOT persisted! */ + off_t iSizeLimit; /* file size limit, 0 = no limit */ + uchar *pszSizeLimitCmd; /* command to carry out when size limit is reached */ + bool bIsTTY; /* is this a tty file? */ } strm_t; + /* interfaces */ BEGINinterface(strm) /* name must also be changed in ENDinterface macro! */ + rsRetVal (*Construct)(strm_t **ppThis); + rsRetVal (*ConstructFinalize)(strm_t *pThis); + rsRetVal (*Destruct)(strm_t **ppThis); + rsRetVal (*SetMaxFileSize)(strm_t *pThis, int64 iMaxFileSize); + rsRetVal (*SetFileName)(strm_t *pThis, uchar *pszName, size_t iLenName); + rsRetVal (*ReadChar)(strm_t *pThis, uchar *pC); + rsRetVal (*UnreadChar)(strm_t *pThis, uchar c); + rsRetVal (*ReadLine)(strm_t *pThis, cstr_t **ppCStr); + rsRetVal (*SeekCurrOffs)(strm_t *pThis); + rsRetVal (*Write)(strm_t *pThis, uchar *pBuf, size_t lenBuf); + rsRetVal (*WriteChar)(strm_t *pThis, uchar c); + rsRetVal (*WriteLong)(strm_t *pThis, long i); + rsRetVal (*SetFName)(strm_t *pThis, uchar *pszPrefix, size_t iLenPrefix); + rsRetVal (*SetDir)(strm_t *pThis, uchar *pszDir, size_t iLenDir); + rsRetVal (*Flush)(strm_t *pThis); + rsRetVal (*RecordBegin)(strm_t *pThis); + rsRetVal (*RecordEnd)(strm_t *pThis); + rsRetVal (*Serialize)(strm_t *pThis, strm_t *pStrm); + rsRetVal (*GetCurrOffset)(strm_t *pThis, int64 *pOffs); + rsRetVal (*SetWCntr)(strm_t *pThis, number_t *pWCnt); + INTERFACEpropSetMeth(strm, bDeleteOnClose, int); + INTERFACEpropSetMeth(strm, iMaxFileSize, int); + INTERFACEpropSetMeth(strm, iMaxFiles, int); + INTERFACEpropSetMeth(strm, iFileNumDigits, int); + INTERFACEpropSetMeth(strm, tOperationsMode, int); + INTERFACEpropSetMeth(strm, tOpenMode, mode_t); + INTERFACEpropSetMeth(strm, sType, strmType_t); + INTERFACEpropSetMeth(strm, iZipLevel, int); + INTERFACEpropSetMeth(strm, bSync, int); + INTERFACEpropSetMeth(strm, sIOBufSize, size_t); + INTERFACEpropSetMeth(strm, iSizeLimit, off_t); + INTERFACEpropSetMeth(strm, iFlushInterval, int); + INTERFACEpropSetMeth(strm, pszSizeLimitCmd, uchar*); ENDinterface(strm) -#define strmCURR_IF_VERSION 1 /* increment whenever you change the interface structure! */ +#define strmCURR_IF_VERSION 2 /* increment whenever you change the interface structure! */ /* prototypes */ -rsRetVal strmConstruct(strm_t **ppThis); -rsRetVal strmConstructFinalize(strm_t __attribute__((unused)) *pThis); -rsRetVal strmDestruct(strm_t **ppThis); -rsRetVal strmSetMaxFileSize(strm_t *pThis, int64 iMaxFileSize); -rsRetVal strmSetFileName(strm_t *pThis, uchar *pszName, size_t iLenName); -rsRetVal strmReadChar(strm_t *pThis, uchar *pC); -rsRetVal strmUnreadChar(strm_t *pThis, uchar c); -rsRetVal strmReadLine(strm_t *pThis, cstr_t **ppCStr); -rsRetVal strmSeekCurrOffs(strm_t *pThis); -rsRetVal strmWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf); -rsRetVal strmWriteChar(strm_t *pThis, uchar c); -rsRetVal strmWriteLong(strm_t *pThis, long i); -rsRetVal strmSetFName(strm_t *pThis, uchar *pszPrefix, size_t iLenPrefix); -rsRetVal strmSetDir(strm_t *pThis, uchar *pszDir, size_t iLenDir); -rsRetVal strmFlush(strm_t *pThis); -rsRetVal strmRecordBegin(strm_t *pThis); -rsRetVal strmRecordEnd(strm_t *pThis); -rsRetVal strmSerialize(strm_t *pThis, strm_t *pStrm); -rsRetVal strmSetiAddtlOpenFlags(strm_t *pThis, int iNewVal); -rsRetVal strmGetCurrOffset(strm_t *pThis, int64 *pOffs); -rsRetVal strmSetWCntr(strm_t *pThis, number_t *pWCnt); PROTOTYPEObjClassInit(strm); -PROTOTYPEpropSetMeth(strm, bDeleteOnClose, int); -PROTOTYPEpropSetMeth(strm, iMaxFileSize, int); -PROTOTYPEpropSetMeth(strm, iMaxFiles, int); -PROTOTYPEpropSetMeth(strm, iFileNumDigits, int); -PROTOTYPEpropSetMeth(strm, tOperationsMode, int); -PROTOTYPEpropSetMeth(strm, tOpenMode, mode_t); -PROTOTYPEpropSetMeth(strm, sType, strmType_t); #endif /* #ifndef STREAM_H_INCLUDED */ diff --git a/runtime/stringbuf.c b/runtime/stringbuf.c index a2d9c599..93995b38 100644 --- a/runtime/stringbuf.c +++ b/runtime/stringbuf.c @@ -70,7 +70,6 @@ rsRetVal cstrConstruct(cstr_t **ppThis) pThis->pszBuf = NULL; pThis->iBufSize = 0; pThis->iStrLen = 0; - pThis->iAllocIncrement = RS_STRINGBUF_ALLOC_INCREMENT; *ppThis = pThis; finalize_it: @@ -153,24 +152,24 @@ void rsCStrDestruct(cstr_t **ppThis) * rgerhards, 2008-01-07 * changed to utilized realloc() -- rgerhards, 2009-06-16 */ -static rsRetVal +rsRetVal rsCStrExtendBuf(cstr_t *pThis, size_t iMinNeeded) { uchar *pNewBuf; - size_t iNewSize; + unsigned short iNewSize; DEFiRet; /* first compute the new size needed */ - if(iMinNeeded > pThis->iAllocIncrement) { - /* we allocate "n" iAllocIncrements. Usually, that should + if(iMinNeeded > RS_STRINGBUF_ALLOC_INCREMENT) { + /* we allocate "n" ALLOC_INCREMENTs. Usually, that should * leave some room after the absolutely needed one. It also * reduces memory fragmentation. Note that all of this are * integer operations (very important to understand what is * going on)! Parenthesis are for better readibility. */ - iNewSize = ((iMinNeeded / pThis->iAllocIncrement) + 1) * pThis->iAllocIncrement; + iNewSize = (iMinNeeded / RS_STRINGBUF_ALLOC_INCREMENT + 1) * RS_STRINGBUF_ALLOC_INCREMENT; } else { - iNewSize = pThis->iBufSize + pThis->iAllocIncrement; + iNewSize = pThis->iBufSize + RS_STRINGBUF_ALLOC_INCREMENT; } iNewSize += pThis->iBufSize; /* add current size */ @@ -224,7 +223,7 @@ rsRetVal rsCStrAppendStr(cstr_t *pThis, uchar* psz) /* append the contents of one cstr_t object to another * rgerhards, 2008-02-25 */ -rsRetVal rsCStrAppendCStr(cstr_t *pThis, cstr_t *pstrAppend) +rsRetVal cstrAppendCStr(cstr_t *pThis, cstr_t *pstrAppend) { return rsCStrAppendStrWithLen(pThis, pstrAppend->pBuf, pstrAppend->iStrLen); } @@ -245,58 +244,10 @@ finalize_it: } -rsRetVal rsCStrAppendChar(cstr_t *pThis, uchar c) -{ - DEFiRet; - - rsCHECKVALIDOBJECT(pThis, OIDrsCStr); - - if(pThis->iStrLen >= pThis->iBufSize) { - CHKiRet(rsCStrExtendBuf(pThis, 1)); /* need more memory! */ - } - - /* ok, when we reach this, we have sufficient memory */ - *(pThis->pBuf + pThis->iStrLen++) = c; - - /* check if we need to invalidate an sz representation! */ - if(pThis->pszBuf != NULL) { - free(pThis->pszBuf); - pThis->pszBuf = NULL; - } - -finalize_it: - RETiRet; -} - - -/* NEW VARIANT - * Append a character to the current string object. This may only be done until - * cstrFinalize() is called. - * rgerhards, 2009-06-16 - */ -rsRetVal cstrAppendChar(cstr_t *pThis, uchar c) -{ - DEFiRet; - - rsCHECKVALIDOBJECT(pThis, OIDrsCStr); - - if(pThis->iStrLen >= pThis->iBufSize) { - CHKiRet(rsCStrExtendBuf(pThis, 1)); /* need more memory! */ - } - - /* ok, when we reach this, we have sufficient memory */ - *(pThis->pBuf + pThis->iStrLen++) = c; - -finalize_it: - RETiRet; -} - - /* Sets the string object to the classigal sz-string provided. * Any previously stored vlaue is discarded. If a NULL pointer * the the new value (pszNew) is provided, an empty string is - * created (this is NOT an error!). Property iAllocIncrement is - * not modified by this function. + * created (this is NOT an error!). * rgerhards, 2005-10-18 */ rsRetVal rsCStrSetSzStr(cstr_t *pThis, uchar *pszNew) @@ -314,7 +265,6 @@ rsRetVal rsCStrSetSzStr(cstr_t *pThis, uchar *pszNew) pThis->iStrLen = strlen((char*)pszNew); pThis->iBufSize = pThis->iStrLen; pThis->pszBuf = NULL; - /* iAllocIncrement is NOT modified! */ /* now save the new value */ if((pThis->pBuf = (uchar*) malloc(sizeof(uchar) * pThis->iStrLen)) == NULL) { @@ -392,25 +342,6 @@ uchar* rsCStrGetSzStr(cstr_t *pThis) } -/* NEW VERSION for interface without separate psz buffer! */ -/* Returns the cstr data as a classical C sz string. We use that the - * Finalizer did properly terminate our string (but we may stil be NULL). - * So it is vital that the finalizer is called BEFORe this function here! - * The caller must not free or otherwise manipulate the returned string and must not - * destroy the CStr object as long as the ascii string is used. - * This function may return NULL, if the string is currently NULL. This - * is a feature, not a bug. If you need non-NULL in any case, use - * cstrGetSzStrNoNULL() instead. - * Note that due to the new single-buffer interface this function almost does nothing! - * rgerhards, 2006-09-16 - */ -uchar* cstrGetSzStr(cstr_t *pThis) -{ - rsCHECKVALIDOBJECT(pThis, OIDrsCStr); - return(pThis->pBuf); -} - - /* Converts the CStr object to a classical zero-terminated C string, * returns that string and destroys the CStr object. The returned string * MUST be freed by the caller. The function might return NULL if @@ -458,38 +389,6 @@ finalize_it: } -/* Finalize the string object. This must be called after all data is added to it - * but before that data is used. - * rgerhards, 2009-06-16 - */ -rsRetVal -cstrFinalize(cstr_t *pThis) -{ - DEFiRet; - rsCHECKVALIDOBJECT(pThis, OIDrsCStr); - - assert(pThis->bIsFinalized == 0); - - if(pThis->iStrLen > 0) { - /* terminate string only if one exists */ - CHKiRet(cstrAppendChar(pThis, '\0')); - --pThis->iStrLen; /* do NOT count the \0 byte */ - } - pThis->bIsFinalized = 1; - -finalize_it: - RETiRet; -} - - -void rsCStrSetAllocIncrement(cstr_t *pThis, int iNewIncrement) -{ - rsCHECKVALIDOBJECT(pThis, OIDrsCStr); - assert(iNewIncrement > 0); - pThis->iAllocIncrement = iNewIncrement; -} - - /* return the length of the current string * 2005-09-09 rgerhards * Please note: this is only a function in a debug build. @@ -497,7 +396,7 @@ void rsCStrSetAllocIncrement(cstr_t *pThis, int iNewIncrement) * This is due to performance reasons. */ #ifndef NDEBUG -int rsCStrLen(cstr_t *pThis) +int cstrLen(cstr_t *pThis) { rsCHECKVALIDOBJECT(pThis, OIDrsCStr); return(pThis->iStrLen); @@ -548,6 +447,27 @@ rsRetVal rsCStrTrimTrailingWhiteSpace(cstr_t *pThis) return RS_RET_OK; } +/* Trim trailing whitespace from a given string + */ +rsRetVal cstrTrimTrailingWhiteSpace(cstr_t *pThis) +{ + register int i; + register uchar *pC; + rsCHECKVALIDOBJECT(pThis, OIDrsCStr); + + i = pThis->iStrLen; + pC = pThis->pBuf + i - 1; + while(i > 0 && isspace((int)*pC)) { + --pC; + --i; + } + /* i now is the new string length! */ + pThis->iStrLen = i; + pThis->pBuf[pThis->iStrLen] = '0'; /* we always have this space */ + + return RS_RET_OK; +} + /* compare two string objects - works like strcmp(), but operates * on CStr objects. Please note that this version here is * faster in the majority of cases, simply because it can diff --git a/runtime/stringbuf.h b/runtime/stringbuf.h index d28aee26..c5130238 100644 --- a/runtime/stringbuf.h +++ b/runtime/stringbuf.h @@ -1,5 +1,5 @@ -/*! \file stringbuf.h - * \brief The counted string object +/* stringbuf.h + * The counted string object * * This is the byte-counted string class for rsyslog. It is a replacement * for classical \0 terminated string functions. We introduce it in @@ -11,8 +11,7 @@ * \date 2005-09-07 * Initial version begun. * - * All functions in this "class" start with rsCStr (rsyslog Counted String). - * Copyright 2005 + * Copyright 2005-2009 * Rainer Gerhards and Adiscon GmbH. All Rights Reserved. * * This file is part of the rsyslog runtime library. @@ -36,6 +35,8 @@ #ifndef _STRINGBUF_H_INCLUDED__ #define _STRINGBUF_H_INCLUDED__ 1 +#include <assert.h> + /** * The dynamic string buffer object. */ @@ -48,8 +49,6 @@ typedef struct cstr_s uchar *pszBuf; /**< pointer to the sz version of the string (after it has been created )*/ size_t iBufSize; /**< current maximum size of the string buffer */ size_t iStrLen; /**< length of the string in characters. */ - size_t iAllocIncrement; /**< the amount of bytes the string should be expanded if it needs to */ - bool bIsFinalized; /**< is this object finished and ready for use? (a debug aid, may be removed later TODO 2009-06-16) */ } cstr_t; @@ -67,14 +66,86 @@ rsRetVal rsCStrConstructFromCStr(cstr_t **ppThis, cstr_t *pFrom); void rsCStrDestruct(cstr_t **ppThis); #define cstrDestruct(x) rsCStrDestruct((x)) -/** - * Append a character to an existing string. If necessary, the - * method expands the string buffer. - * - * \param c Character to append to string. + +/* Append a character to the current string object. This may only be done until + * cstrFinalize() is called. + * rgerhards, 2009-06-16 */ -rsRetVal rsCStrAppendChar(cstr_t *pThis, uchar c); -rsRetVal cstrAppendChar(cstr_t *pThis, uchar c); +rsRetVal rsCStrExtendBuf(cstr_t *pThis, size_t iMinNeeded); /* our helper, NOT a public interface! */ +static inline rsRetVal cstrAppendChar(cstr_t *pThis, uchar c) +{ + rsRetVal iRet = RS_RET_OK; + + if(pThis->iStrLen >= pThis->iBufSize) { + CHKiRet(rsCStrExtendBuf(pThis, 1)); /* need more memory! */ + } + + /* ok, when we reach this, we have sufficient memory */ + *(pThis->pBuf + pThis->iStrLen++) = c; + +finalize_it: + return iRet; +} + + +/* some inline functions for things that are really frequently called... */ + +/* Finalize the string object. This must be called after all data is added to it + * but before that data is used. + * rgerhards, 2009-06-16 + */ +static inline rsRetVal +cstrFinalize(cstr_t *pThis) +{ + rsRetVal iRet = RS_RET_OK; + + if(pThis->iStrLen > 0) { + /* terminate string only if one exists */ + CHKiRet(cstrAppendChar(pThis, '\0')); + --pThis->iStrLen; /* do NOT count the \0 byte */ + } + +finalize_it: + return iRet; +} + + +/* Returns the cstr data as a classical C sz string. We use that the + * Finalizer did properly terminate our string (but we may stil be NULL). + * So it is vital that the finalizer is called BEFORe this function here! + * The caller must not free or otherwise manipulate the returned string and must not + * destroy the CStr object as long as the ascii string is used. + * This function may return NULL, if the string is currently NULL. This + * is a feature, not a bug. If you need non-NULL in any case, use + * cstrGetSzStrNoNULL() instead. + * Note that due to the new single-buffer interface this function almost does nothing! + * rgerhards, 2006-09-16 + */ +static inline uchar* cstrGetSzStr(cstr_t *pThis) +{ + rsCHECKVALIDOBJECT(pThis, OIDrsCStr); + return(pThis->pBuf); +} + + +/* Converts the CStr object to a classical sz string and returns that. + * Same restrictions as in cstrGetSzStr() applies (see there!). This + * function here guarantees that a valid string is returned, even if + * the CStr object currently holds a NULL pointer string buffer. If so, + * "" is returned. + * rgerhards 2005-10-19 + * WARNING: The returned pointer MUST NOT be freed, as it may be + * obtained from that constant memory pool (in case of NULL!) + */ +static inline uchar* cstrGetSzStrNoNULL(cstr_t *pThis) +{ + rsCHECKVALIDOBJECT(pThis, OIDrsCStr); + if(pThis->pBuf == NULL) + return (uchar*) ""; + else + return cstrGetSzStr(pThis); +} + /** * Truncate "n" number of characters from the end of the @@ -85,6 +156,7 @@ rsRetVal cstrAppendChar(cstr_t *pThis, uchar c); rsRetVal rsCStrTruncate(cstr_t *pThis, size_t nTrunc); rsRetVal rsCStrTrimTrailingWhiteSpace(cstr_t *pThis); +rsRetVal cstrTrimTrailingWhiteSpace(cstr_t *pThis); /** * Append a string to the buffer. For performance reasons, @@ -102,22 +174,6 @@ rsRetVal rsCStrAppendStr(cstr_t *pThis, uchar* psz); */ rsRetVal rsCStrAppendStrWithLen(cstr_t *pThis, uchar* psz, size_t iStrLen); -/** - * Set a new allocation incremet. This will influence - * the allocation the next time the string will be expanded. - * It can be set and changed at any time. If done immediately - * after custructing the StrB object, this will also be - * the inital allocation. - * - * \param iNewIncrement The new increment size - * - * \note It is possible to use a very low increment, e.g. 1 byte. - * This can generate a considerable overhead. We highly - * advise not to use an increment below 32 bytes, except - * if you are very well aware why you are doing it ;) - */ -void rsCStrSetAllocIncrement(cstr_t *pThis, int iNewIncrement); -#define rsCStrGetAllocIncrement(pThis) ((pThis)->iAllocIncrement) /** * Append an integer to the string. No special formatting is @@ -143,19 +199,22 @@ rsRetVal rsCStrSzStrMatchRegex(cstr_t *pCS1, uchar *psz, int iType, void *cache) void rsCStrRegexDestruct(void *rc); rsRetVal rsCStrConvertToNumber(cstr_t *pStr, number_t *pNumber); rsRetVal rsCStrConvertToBool(cstr_t *pStr, number_t *pBool); -rsRetVal rsCStrAppendCStr(cstr_t *pThis, cstr_t *pstrAppend); + +/* in migration */ +#define rsCStrAppendCStr(pThis, pstrAppend) cstrAppendCStr(pThis, pstrAppend) /* new calling interface */ rsRetVal cstrFinalize(cstr_t *pThis); rsRetVal cstrConvSzStrAndDestruct(cstr_t *pThis, uchar **ppSz, int bRetNULL); -uchar* cstrGetSzStr(cstr_t *pThis); +rsRetVal cstrAppendCStr(cstr_t *pThis, cstr_t *pstrAppend); /* now come inline-like functions */ #ifdef NDEBUG -# define rsCStrLen(x) ((int)((x)->iStrLen)) +# define cstrLen(x) ((int)((x)->iStrLen)) #else - int rsCStrLen(cstr_t *pThis); + int cstrLen(cstr_t *pThis); #endif +#define rsCStrLen(s) cstrLen((s)) #define rsCStrGetBufBeg(x) ((x)->pBuf) diff --git a/runtime/syslogd-types.h b/runtime/syslogd-types.h index be0dfdd8..4a26f993 100644 --- a/runtime/syslogd-types.h +++ b/runtime/syslogd-types.h @@ -76,25 +76,31 @@ enum _EHostnameCmpMode { }; typedef enum _EHostnameCmpMode EHostnameCmpMode; +/* time type numerical values for structure below */ +#define TIME_TYPE_UNINIT 0 +#define TIME_TYPE_RFC3164 1 +#define TIME_TYPE_RFC5424 2 /* rgerhards 2004-11-11: the following structure represents * a time as it is used in syslog. + * rgerhards, 2009-06-23: packed structure for better cache performance + * (but left ultimate decision about packing to compiler) */ struct syslogTime { - int timeType; /* 0 - unitinialized , 1 - RFC 3164, 2 - syslog-protocol */ - int year; - int month; - int day; - int hour; /* 24 hour clock */ - int minute; - int second; - int secfrac; /* fractional seconds (must be 32 bit!) */ - int secfracPrecision; + intTiny timeType; /* 0 - unitinialized , 1 - RFC 3164, 2 - syslog-protocol */ + intTiny month; + intTiny day; + intTiny hour; /* 24 hour clock */ + intTiny minute; + intTiny second; + intTiny secfracPrecision; + intTiny OffsetMinute; /* UTC offset in minutes */ + intTiny OffsetHour; /* UTC offset in hours + * full UTC offset minutes = OffsetHours*60 + OffsetMinute. Then use + * OffsetMode to know the direction. + */ char OffsetMode; /* UTC offset + or - */ - char OffsetHour; /* UTC offset in hours */ - int OffsetMinute; /* UTC offset in minutes */ - /* full UTC offset minutes = OffsetHours*60 + OffsetMinute. Then use - * OffsetMode to know the direction. - */ + short year; + int secfrac; /* fractional seconds (must be 32 bit!) */ }; typedef struct syslogTime syslogTime_t; diff --git a/runtime/sysvar.c b/runtime/sysvar.c index c102d1f5..4a6ace19 100644 --- a/runtime/sysvar.c +++ b/runtime/sysvar.c @@ -175,8 +175,6 @@ CODESTARTobjQueryInterface(sysvar) * work here (if we can support an older interface version - that, * of course, also affects the "if" above). */ - //xxxpIf->oID = "sysvar";//OBJsysvar; - pIf->Construct = sysvarConstruct; pIf->ConstructFinalize = sysvarConstructFinalize; pIf->Destruct = sysvarDestruct; diff --git a/runtime/unicode-helper.h b/runtime/unicode-helper.h index 36d76a78..7a776f68 100644 --- a/runtime/unicode-helper.h +++ b/runtime/unicode-helper.h @@ -4,6 +4,9 @@ * The following functions are wrappers which hopefully enable us to move * from 8-bit chars to unicode with relative ease when we finally attack this * + * Note: while we prefer inline functions, this leads to invalid references in + * core dumps. So in a debug build, we use macros where appropriate... + * * Begun 2009-05-21 RGerhards * * Copyright (C) 2009 by Rainer Gerhards and Adiscon GmbH @@ -31,6 +34,22 @@ #include <string.h> +#ifdef DEBUG +# define ustrncpy(psz1, psz2, len) strncpy((char*)(psz1), (char*)(psz2), (len)) +# define ustrdup(psz) (uchar*)strdup((char*)(psz)) +#else + static inline uchar* ustrncpy(uchar *psz1, uchar *psz2, size_t len) + { + return (uchar*) strncpy((char*) psz1, (char*) psz2, len); + } + + static inline uchar* ustrdup(uchar *psz) + { + return (uchar*) strdup((char*)psz); + } + +#endif /* #ifdef DEBUG */ + static inline int ustrcmp(uchar *psz1, uchar *psz2) { return strcmp((char*) psz1, (char*) psz2); @@ -41,13 +60,9 @@ static inline int ustrlen(uchar *psz) return strlen((char*) psz); } -static inline uchar* ustrdup(uchar *psz) -{ - return (uchar*) strdup((char*)psz); -} - #define UCHAR_CONSTANT(x) ((uchar*) (x)) +#define CHAR_CONVERT(x) ((char*) (x)) #endif /* multi-include protection */ /* vim:set ai: diff --git a/runtime/vm.c b/runtime/vm.c index 8cbf9e12..aaf3c879 100644 --- a/runtime/vm.c +++ b/runtime/vm.c @@ -82,11 +82,15 @@ rsfrAddFunction(uchar *szName, prsf_t rsf) /* unique name, so add to head of list */ CHKmalloc(pEntry = calloc(1, sizeof(rsf_entry_t))); CHKiRet(rsCStrConstructFromszStr(&pEntry->pName, szName)); + CHKiRet(cstrFinalize(pEntry->pName)); pEntry->rsf = rsf; pEntry->pNext = funcRegRoot; funcRegRoot = pEntry; finalize_it: + if(iRet != RS_RET_OK && iRet != RS_RET_DUP_FUNC_NAME) + free(pEntry); + RETiRet; } @@ -167,7 +171,7 @@ rsfrRemoveAll(void) while(pEntry != NULL) { pEntryDel = pEntry; pEntry = pEntry->pNext; - rsCStrDestruct(&pEntryDel->pName); + cstrDestruct(&pEntryDel->pName); free(pEntryDel); } funcRegRoot = NULL; @@ -405,6 +409,7 @@ CODESTARTop(STRADD) vmstk.PopString(pThis->pStk, &operand1); CHKiRet(rsCStrAppendCStr(operand1->val.pStr, operand2->val.pStr)); + CHKiRet(cstrFinalize(operand1->val.pStr)); /* we have a result, so let's push it */ vmstk.Push(pThis->pStk, operand1); @@ -554,12 +559,12 @@ rsf_tolower(vmstk_t *pStk, int numOperands) ABORT_FINALIZE(RS_RET_INVLD_NBR_ARGUMENTS); /* pop args and do operaton */ - CHKiRet(rsCStrConstruct(&pcstr)); + CHKiRet(cstrConstruct(&pcstr)); vmstk.PopString(pStk, &operand1); - pSrc = rsCStrGetSzStr(operand1->val.pStr); - iStrlen = strlen((char*)pSrc); + pSrc = cstrGetSzStr(operand1->val.pStr); + iStrlen = strlen((char*)pSrc); // TODO: use count from string! while(iStrlen--) { - CHKiRet(rsCStrAppendChar(pcstr, tolower(*pSrc++))); + CHKiRet(cstrAppendChar(pcstr, tolower(*pSrc++))); } /* Store result and cleanup */ diff --git a/runtime/vmop.c b/runtime/vmop.c index acacfc9e..ea627220 100644 --- a/runtime/vmop.c +++ b/runtime/vmop.c @@ -125,7 +125,7 @@ Obj2Str(vmop_t *pThis, cstr_t *pstrPrg) if(pThis->operand.pVar != NULL) CHKiRet(var.Obj2Str(pThis->operand.pVar, pstrPrg)); } - CHKiRet(rsCStrAppendChar(pstrPrg, '\n')); + CHKiRet(cstrAppendChar(pstrPrg, '\n')); finalize_it: RETiRet; diff --git a/runtime/wti.c b/runtime/wti.c index 544bffa7..abdf4add 100644 --- a/runtime/wti.c +++ b/runtime/wti.c @@ -41,7 +41,6 @@ #ifdef OS_SOLARIS # include <sched.h> -# define pthread_yield() sched_yield() #endif #include "rsyslog.h" @@ -51,6 +50,7 @@ #include "wti.h" #include "obj.h" #include "glbl.h" +#include "atomic.h" /* static data */ DEFobjStaticHelpers @@ -106,6 +106,7 @@ rsRetVal wtiSetState(wti_t *pThis, qWrkCmd_t tCmd, int bActiveOnly, int bLockMutex) { DEFiRet; + qWrkCmd_t tCurrCmd; DEFVARS_mutexProtection; ISOBJ_TYPE_assert(pThis, wti); @@ -113,13 +114,14 @@ wtiSetState(wti_t *pThis, qWrkCmd_t tCmd, int bActiveOnly, int bLockMutex) BEGIN_MTX_PROTECTED_OPERATIONS(&pThis->mut, bLockMutex); + tCurrCmd = pThis->tCurrCmd; /* all worker states must be followed sequentially, only termination can be set in any state */ - if( (bActiveOnly && (pThis->tCurrCmd < eWRKTHRD_RUN_CREATED)) - || (pThis->tCurrCmd > tCmd && !(tCmd == eWRKTHRD_TERMINATING || tCmd == eWRKTHRD_STOPPED))) { - dbgprintf("%s: command %d can not be accepted in current %d processing state - ignored\n", - wtiGetDbgHdr(pThis), tCmd, pThis->tCurrCmd); + if( (bActiveOnly && (tCurrCmd < eWRKTHRD_RUN_CREATED)) + || (tCurrCmd > tCmd && !(tCmd == eWRKTHRD_TERMINATING || tCmd == eWRKTHRD_STOPPED))) { + DBGPRINTF("%s: command %d can not be accepted in current %d processing state - ignored\n", + wtiGetDbgHdr(pThis), tCmd, tCurrCmd); } else { - dbgprintf("%s: receiving command %d\n", wtiGetDbgHdr(pThis), tCmd); + DBGPRINTF("%s: receiving command %d\n", wtiGetDbgHdr(pThis), tCmd); /* we could replace this with a simple if, but we leave the switch in in case we need * to add something at a later stage. -- rgerhards, 2008-09-30 */ @@ -143,7 +145,12 @@ wtiSetState(wti_t *pThis, qWrkCmd_t tCmd, int bActiveOnly, int bLockMutex) /* DO NOTHING */ break; } - pThis->tCurrCmd = tCmd; /* apply the new state */ + /* apply the new state */ + unsigned val = ATOMIC_CAS_VAL(pThis->tCurrCmd, tCurrCmd, tCmd); + if(val != tCurrCmd) { + DBGPRINTF("wtiSetState PROBLEM, tCurrCmd %d overwritten with %d, wanted to set %d\n", tCurrCmd, val, tCmd); + } + } END_MTX_PROTECTED_OPERATIONS(&pThis->mut); @@ -151,7 +158,7 @@ wtiSetState(wti_t *pThis, qWrkCmd_t tCmd, int bActiveOnly, int bLockMutex) } -/* Cancel the thread. If the thread is already cancelled or termination, +/* Cancel the thread. If the thread is already cancelled or terminated, * we do not again cancel it. But it is save and legal to call wtiCancelThrd() in * such situations. * rgerhards, 2008-02-26 @@ -165,11 +172,13 @@ wtiCancelThrd(wti_t *pThis) d_pthread_mutex_lock(&pThis->mut); + wtiProcessThrdChanges(pThis, MUTEX_ALREADY_LOCKED); /* process state change, so that we have current state vars */ + if(pThis->tCurrCmd >= eWRKTHRD_TERMINATING) { - dbgoprint((obj_t*) pThis, "canceling worker thread\n"); + dbgoprint((obj_t*) pThis, "canceling worker thread, curr stat %d\n", pThis->tCurrCmd); pthread_cancel(pThis->thrdID); wtiSetState(pThis, eWRKTHRD_TERMINATING, 0, MUTEX_ALREADY_LOCKED); - pThis->pWtp->bThrdStateChanged = 1; /* indicate change, so harverster will be called */ + ATOMIC_STORE_1_TO_INT(pThis->pWtp->bThrdStateChanged); /* indicate change, so harverster will be called */ } d_pthread_mutex_unlock(&pThis->mut); @@ -201,15 +210,13 @@ CODESTARTobjDestruct(wti) pthread_cond_destroy(&pThis->condExitDone); pthread_mutex_destroy(&pThis->mut); - if(pThis->pszDbgHdr != NULL) - free(pThis->pszDbgHdr); + free(pThis->pszDbgHdr); ENDobjDestruct(wti) /* Standard-Constructor for the wti object */ BEGINobjConstruct(wti) /* be sure to specify the object type also in END macro! */ - pThis->bOptimizeUniProc = glbl.GetOptimizeUniProc(); pthread_cond_init(&pThis->condExitDone, NULL); pthread_mutex_init(&pThis->mut, NULL); ENDobjConstruct(wti) @@ -320,7 +327,7 @@ wtiWorkerCancelCleanup(void *arg) d_pthread_mutex_lock(&pWtp->mut); wtiSetState(pThis, eWRKTHRD_TERMINATING, 0, MUTEX_ALREADY_LOCKED); /* TODO: sync access? I currently think it is NOT needed -- rgerhards, 2008-01-28 */ - pWtp->bThrdStateChanged = 1; /* indicate change, so harverster will be called */ + ATOMIC_STORE_1_TO_INT(pWtp->bThrdStateChanged); /* indicate change, so harverster will be called */ d_pthread_mutex_unlock(&pWtp->mut); pthread_setcancelstate(iCancelStateSave, NULL); @@ -329,27 +336,6 @@ wtiWorkerCancelCleanup(void *arg) /* generic worker thread framework - * - * Some special comments below, so that they do not clutter the main function code: - * - * On the use of pthread_testcancel(): - * Now make sure we can get canceled - it is not specified if pthread_setcancelstate() is - * a cancellation point in itself. As we run most of the time without cancel enabled, I fear - * we may never get cancelled if we do not create a cancellation point ourselfs. - * - * On the use of pthread_yield(): - * We yield to give the other threads a chance to obtain the mutex. If we do not - * do that, this thread may very well aquire the mutex again before another thread - * has even a chance to run. The reason is that mutex operations are free to be - * implemented in the quickest possible way (and they typically are!). That is, the - * mutex lock/unlock most probably just does an atomic memory swap and does not necessarily - * schedule other threads waiting on the same mutex. That can lead to the same thread - * aquiring the mutex ever and ever again while all others are starving for it. We - * have exactly seen this behaviour when we deliberately introduced a long-running - * test action which basically did a sleep. I understand that with real actions the - * likelihood of this starvation condition is very low - but it could still happen - * and would be very hard to debug. The yield() is a sure fix, its performance overhead - * should be well accepted given the above facts. -- rgerhards, 2008-01-10 */ #pragma GCC diagnostic ignored "-Wempty-body" rsRetVal @@ -377,11 +363,6 @@ wtiWorker(wti_t *pThis) while(1) { /* loop will be broken below - need to do mutex locks */ /* process any pending thread requests */ wtpProcessThrdChanges(pWtp); - pthread_testcancel(); /* see big comment in function header */ -# if !defined(__hpux) /* pthread_yield is missing there! */ - if(pThis->bOptimizeUniProc) - pthread_yield(); /* see big comment in function header */ -# endif /* if we have a rate-limiter set for this worker pool, let's call it. Please * keep in mind that the rate-limiter may hold us for an extended period @@ -433,7 +414,7 @@ wtiWorker(wti_t *pThis) pWtp->pfOnWorkerShutdown(pWtp->pUsr); wtiSetState(pThis, eWRKTHRD_TERMINATING, 0, MUTEX_ALREADY_LOCKED); - pWtp->bThrdStateChanged = 1; /* indicate change, so harverster will be called */ + ATOMIC_STORE_1_TO_INT(pWtp->bThrdStateChanged); /* indicate change, so harverster will be called */ d_pthread_mutex_unlock(&pThis->mut); pthread_setcancelstate(iCancelStateSave, NULL); diff --git a/runtime/wti.h b/runtime/wti.h index 6b60b833..72653b15 100644 --- a/runtime/wti.h +++ b/runtime/wti.h @@ -31,14 +31,13 @@ /* the worker thread instance class */ typedef struct wti_s { BEGINobjInstance; - int bOptimizeUniProc; /* cache for the equally-named global setting, pulled at time of queue creation */ pthread_t thrdID; /* thread ID */ qWrkCmd_t tCurrCmd; /* current command to be carried out by worker */ obj_t *pUsrp; /* pointer to an object meaningful for current user pointer (e.g. queue pUsr data elemt) */ wtp_t *pWtp; /* my worker thread pool (important if only the work thread instance is passed! */ pthread_cond_t condExitDone; /* signaled when the thread exit is done (once per thread existance) */ pthread_mutex_t mut; - int bShutdownRqtd; /* shutdown for this thread requested? 0 - no , 1 - yes */ + bool bShutdownRqtd; /* shutdown for this thread requested? 0 - no , 1 - yes */ uchar *pszDbgHdr; /* header string for debug messages */ } wti_t; diff --git a/runtime/wtp.c b/runtime/wtp.c index 04eb974f..0c66dd11 100644 --- a/runtime/wtp.c +++ b/runtime/wtp.c @@ -39,10 +39,13 @@ #include <fcntl.h> #include <unistd.h> #include <errno.h> +#include <atomic.h> +#if HAVE_SYS_PRCTL_H +# include <sys/prctl.h> +#endif #ifdef OS_SOLARIS # include <sched.h> -# define pthread_yield() sched_yield() #endif #include "rsyslog.h" @@ -51,6 +54,7 @@ #include "wtp.h" #include "wti.h" #include "obj.h" +#include "unicode-helper.h" #include "glbl.h" /* static data */ @@ -82,7 +86,6 @@ static rsRetVal NotImplementedDummy() { return RS_RET_OK; } /* Standard-Constructor for the wtp object */ BEGINobjConstruct(wtp) /* be sure to specify the object type also in END macro! */ - pThis->bOptimizeUniProc = glbl.GetOptimizeUniProc(); pthread_mutex_init(&pThis->mut, NULL); pthread_mutex_init(&pThis->mutThrdShutdwn, NULL); pthread_cond_init(&pThis->condThrdTrm, NULL); @@ -151,8 +154,7 @@ CODESTARTobjDestruct(wtp) pthread_mutex_destroy(&pThis->mut); pthread_mutex_destroy(&pThis->mutThrdShutdwn); - if(pThis->pszDbgHdr != NULL) - free(pThis->pszDbgHdr); + free(pThis->pszDbgHdr); ENDobjDestruct(wtp) @@ -179,9 +181,7 @@ wtpWakeupAllWrkr(wtp_t *pThis) DEFiRet; ISOBJ_TYPE_assert(pThis, wtp); - d_pthread_mutex_lock(pThis->pmutUsr); pthread_cond_broadcast(pThis->pcondBusy); - d_pthread_mutex_unlock(pThis->pmutUsr); RETiRet; } @@ -216,7 +216,7 @@ wtpProcessThrdChanges(wtp_t *pThis) */ do { /* reset the change marker */ - pThis->bThrdStateChanged = 0; + ATOMIC_STORE_0_TO_INT(pThis->bThrdStateChanged); /* go through all threads */ for(i = 0 ; i < pThis->iNumWorkerThreads ; ++i) { wtiProcessThrdChanges(pThis->pWrkr[i], LOCK_MUTEX); @@ -421,6 +421,8 @@ wtpWrkrExecCancelCleanup(void *arg) static void * wtpWorker(void *arg) /* the arg is actually a wti object, even though we are in wtp! */ { + uchar *pszDbgHdr; + uchar thrdName[32] = "rs:"; DEFiRet; DEFVARS_mutexProtection; wti_t *pWti = (wti_t*) arg; @@ -434,6 +436,15 @@ wtpWorker(void *arg) /* the arg is actually a wti object, even though we are in sigfillset(&sigSet); pthread_sigmask(SIG_BLOCK, &sigSet, NULL); +# if HAVE_PRCTL && defined PR_SET_NAME + /* set thread name - we ignore if the call fails, has no harsh consequences... */ + pszDbgHdr = wtpGetDbgHdr(pThis); + ustrncpy(thrdName+3, pszDbgHdr, 20); + if(prctl(PR_SET_NAME, thrdName, 0, 0, 0) != 0) { + DBGPRINTF("prctl failed, not setting thread name for '%s'\n", wtpGetDbgHdr(pThis)); + } +# endif + BEGIN_MTX_PROTECTED_OPERATIONS(&pThis->mut, LOCK_MUTEX); /* do some late initialization */ @@ -510,14 +521,6 @@ wtpStartWrkr(wtp_t *pThis, int bLockMutex) dbgprintf("%s: started with state %d, num workers now %d\n", wtpGetDbgHdr(pThis), iState, pThis->iCurNumWrkThrd); - /* we try to give the starting worker a little boost. It won't help much as we still - * hold the queue's mutex, but at least it has a chance to start on a single-CPU system. - */ -# if !defined(__hpux) /* pthread_yield is missing there! */ - if(pThis->bOptimizeUniProc) - pthread_yield(); -# endif - /* indicate we just started a worker and would like to see it running */ wtpSetInactivityGuard(pThis, 1, MUTEX_ALREADY_LOCKED); diff --git a/runtime/wtp.h b/runtime/wtp.h index b9cb07c5..1ce171cc 100644 --- a/runtime/wtp.h +++ b/runtime/wtp.h @@ -52,13 +52,12 @@ typedef enum { /* the worker thread pool (wtp) object */ typedef struct wtp_s { BEGINobjInstance; - int bOptimizeUniProc; /* cache for the equally-named global setting, pulled at time of queue creation */ wtpState_t wtpState; int iNumWorkerThreads;/* number of worker threads to use */ int iCurNumWrkThrd;/* current number of active worker threads */ struct wti_s **pWrkr;/* array with control structure for the worker thread(s) associated with this wtp */ int toWrkShutdown; /* timeout for idle workers in ms, -1 means indefinite (0 is immediate) */ - int bInactivityGuard;/* prevents inactivity due to race condition */ + bool bInactivityGuard;/* prevents inactivity due to race condition */ rsRetVal (*pConsumer)(void *); /* user-supplied consumer function for dewtpd messages */ /* synchronization variables */ pthread_mutex_t mutThrdShutdwn; /* mutex to guard thread shutdown processing */ diff --git a/runtime/zlibw.c b/runtime/zlibw.c new file mode 100644 index 00000000..2b386213 --- /dev/null +++ b/runtime/zlibw.c @@ -0,0 +1,125 @@ +/* The zlibwrap object. + * + * This is an rsyslog object wrapper around zlib. + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * The rsyslog runtime library is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * The rsyslog runtime library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with the rsyslog runtime library. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. + */ + +#include "config.h" +#include <string.h> +#include <assert.h> +#include <zlib.h> + +#include "rsyslog.h" +#include "module-template.h" +#include "obj.h" +#include "zlibw.h" + +MODULE_TYPE_LIB + +/* static data */ +DEFobjStaticHelpers + + +/* ------------------------------ methods ------------------------------ */ + +/* zlib make strong use of macros for its interface functions, so we can not simply + * pass function pointers to them. Instead, we create very small wrappers which call + * the relevant entry points. + */ + +static int myDeflateInit(z_streamp strm, int level) +{ + return deflateInit(strm, level); +} + +static int myDeflateInit2(z_streamp strm, int level, int method, int windowBits, int memLevel, int strategy) +{ + return deflateInit2(strm, level, method, windowBits, memLevel, strategy); +} + +static int myDeflateEnd(z_streamp strm) +{ + return deflateEnd(strm); +} + +static int myDeflate(z_streamp strm, int flush) +{ + return deflate(strm, flush); +} + + +/* queryInterface function + * rgerhards, 2008-03-05 + */ +BEGINobjQueryInterface(zlibw) +CODESTARTobjQueryInterface(zlibw) + if(pIf->ifVersion != zlibwCURR_IF_VERSION) { /* check for current version, increment on each change */ + ABORT_FINALIZE(RS_RET_INTERFACE_NOT_SUPPORTED); + } + + /* ok, we have the right interface, so let's fill it + * Please note that we may also do some backwards-compatibility + * work here (if we can support an older interface version - that, + * of course, also affects the "if" above). + */ + pIf->DeflateInit = myDeflateInit; + pIf->DeflateInit2 = myDeflateInit2; + pIf->Deflate = myDeflate; + pIf->DeflateEnd = myDeflateEnd; +finalize_it: +ENDobjQueryInterface(zlibw) + + +/* Initialize the zlibw class. Must be called as the very first method + * before anything else is called inside this class. + * rgerhards, 2008-02-19 + */ +BEGINAbstractObjClassInit(zlibw, 1, OBJ_IS_LOADABLE_MODULE) /* class, version */ + /* request objects we use */ + + /* set our own handlers */ +ENDObjClassInit(zlibw) + + +/* --------------- here now comes the plumbing that makes as a library module --------------- */ + + +BEGINmodExit +CODESTARTmodExit +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_LIB_QUERIES +ENDqueryEtryPt + + +BEGINmodInit() +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ + + CHKiRet(zlibwClassInit(pModInfo)); /* must be done after tcps_sess, as we use it */ + /* Initialize all classes that are in our module - this includes ourselfs */ +ENDmodInit +/* vi:set ai: + */ diff --git a/runtime/zlibw.h b/runtime/zlibw.h new file mode 100644 index 00000000..63d8f386 --- /dev/null +++ b/runtime/zlibw.h @@ -0,0 +1,46 @@ +/* The zlibw object. It encapsulates the zlib functionality. The primary + * purpose of this wrapper class is to enable rsyslogd core to be build without + * zlib libraries. + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * + * The rsyslog runtime library is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * The rsyslog runtime library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with the rsyslog runtime library. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. + */ +#ifndef INCLUDED_ZLIBW_H +#define INCLUDED_ZLIBW_H + +#include <zlib.h> + +/* interfaces */ +BEGINinterface(zlibw) /* name must also be changed in ENDinterface macro! */ + int (*DeflateInit)(z_streamp strm, int); + int (*DeflateInit2)(z_streamp strm, int level, int method, int windowBits, int memLevel, int strategy); + int (*Deflate)(z_streamp strm, int); + int (*DeflateEnd)(z_streamp strm); +ENDinterface(zlibw) +#define zlibwCURR_IF_VERSION 1 /* increment whenever you change the interface structure! */ + + +/* prototypes */ +PROTOTYPEObj(zlibw); + +/* the name of our library binary */ +#define LM_ZLIBW_FILENAME "lmzlibw" + +#endif /* #ifndef INCLUDED_ZLIBW_H */ @@ -297,6 +297,12 @@ Send(tcpclt_t *pThis, void *pData, char *msg, size_t len) CHKiRet(TCPSendBldFrame(pThis, &msg, &len, &bMsgMustBeFreed)); + if(pThis->iRebindInterval > 0 && ++pThis->iNumMsgs == pThis->iRebindInterval) { + /* we need to rebind, and use the retry logic for this*/ + CHKiRet(pThis->prepRetryFunc(pData)); /* try to recover */ + pThis->iNumMsgs = 0; + } + while(!bDone) { /* loop is broken when send succeeds or error occurs */ CHKiRet(pThis->initFunc(pData)); iRet = pThis->sendFunc(pData, msg, len); @@ -388,6 +394,13 @@ SetFraming(tcpclt_t *pThis, TCPFRAMINGMODE framing) pThis->tcp_framing = framing; RETiRet; } +static rsRetVal +SetRebindInterval(tcpclt_t *pThis, int iRebindInterval) +{ + DEFiRet; + pThis->iRebindInterval = iRebindInterval; + RETiRet; +} /* Standard-Constructor @@ -445,6 +458,7 @@ CODESTARTobjQueryInterface(tcpclt) pIf->SetSendFrame = SetSendFrame; pIf->SetSendPrepRetry = SetSendPrepRetry; pIf->SetFraming = SetFraming; + pIf->SetRebindInterval = SetRebindInterval; finalize_it: ENDobjQueryInterface(tcpclt) @@ -36,6 +36,8 @@ typedef struct tcpclt_s { short bResendLastOnRecon; /* should the last message be resent on a successful reconnect? */ size_t lenPrevMsg; /* session specific callbacks */ + int iRebindInterval; /* how often should the send socket be rebound? */ + int iNumMsgs; /* number of messages during current "rebind session" */ rsRetVal (*initFunc)(void*); rsRetVal (*sendFunc)(void*, char*, size_t); rsRetVal (*prepRetryFunc)(void*); @@ -55,8 +57,10 @@ BEGINinterface(tcpclt) /* name must also be changed in ENDinterface macro! */ rsRetVal (*SetSendFrame)(tcpclt_t*, rsRetVal (*)(void*, char*, size_t)); rsRetVal (*SetSendPrepRetry)(tcpclt_t*, rsRetVal (*)(void*)); rsRetVal (*SetFraming)(tcpclt_t*, TCPFRAMINGMODE framing); + /* v3, 2009-07-14*/ + rsRetVal (*SetRebindInterval)(tcpclt_t*, int iRebindInterval); ENDinterface(tcpclt) -#define tcpcltCURR_IF_VERSION 2 /* increment whenever you change the interface structure! */ +#define tcpcltCURR_IF_VERSION 3 /* increment whenever you change the interface structure! */ /* prototypes */ diff --git a/tcps_sess.c b/tcps_sess.c index cfee0523..ea9032b3 100644 --- a/tcps_sess.c +++ b/tcps_sess.c @@ -36,6 +36,7 @@ #include "rsyslog.h" #include "dirty.h" +#include "unicode-helper.h" #include "module-template.h" #include "net.h" #include "tcpsrv.h" @@ -45,6 +46,7 @@ #include "netstrm.h" #include "msg.h" #include "datetime.h" +#include "prop.h" /* static data */ @@ -52,10 +54,12 @@ DEFobjStaticHelpers DEFobjCurrIf(glbl) DEFobjCurrIf(errmsg) DEFobjCurrIf(netstrm) +DEFobjCurrIf(prop) DEFobjCurrIf(datetime) static int iMaxLine; /* maximum size of a single message */ + /* forward definitions */ static rsRetVal Close(tcps_sess_t *pThis); @@ -97,8 +101,10 @@ CODESTARTobjDestruct(tcps_sess) pThis->pSrv->pOnSessDestruct(&pThis->pUsr); } /* now destruct our own properties */ - free(pThis->fromHost); - free(pThis->fromHostIP); + if(pThis->fromHost != NULL) + CHKiRet(prop.Destruct(&pThis->fromHost)); + if(pThis->fromHostIP != NULL) + CHKiRet(prop.Destruct(&pThis->fromHostIP)); free(pThis->pMsg); ENDobjDestruct(tcps_sess) @@ -121,9 +127,13 @@ SetHost(tcps_sess_t *pThis, uchar *pszHost) ISOBJ_TYPE_assert(pThis, tcps_sess); - free(pThis->fromHost); - pThis->fromHost = pszHost; + if(pThis->fromHost == NULL) + CHKiRet(prop.Construct(&pThis->fromHost)); + + CHKiRet(prop.SetString(pThis->fromHost, pszHost, ustrlen(pszHost))); +finalize_it: + free(pszHost); /* we must free according to our (old) calling conventions */ RETiRet; } @@ -138,9 +148,13 @@ SetHostIP(tcps_sess_t *pThis, uchar *pszHostIP) ISOBJ_TYPE_assert(pThis, tcps_sess); - free(pThis->fromHostIP); - pThis->fromHostIP = pszHostIP; + if(pThis->fromHostIP == NULL) + CHKiRet(prop.Construct(&pThis->fromHostIP)); + CHKiRet(prop.SetString(pThis->fromHostIP, pszHostIP, ustrlen(pszHostIP))); + +finalize_it: + free(pszHostIP); RETiRet; } @@ -217,13 +231,18 @@ SetOnMsgReceive(tcps_sess_t *pThis, rsRetVal (*OnMsgReceive)(tcps_sess_t*, uchar * rgerhards, 2009-04-23 */ static rsRetVal -defaultDoSubmitMessage(tcps_sess_t *pThis, struct syslogTime *stTime, time_t ttGenTime) +defaultDoSubmitMessage(tcps_sess_t *pThis, struct syslogTime *stTime, time_t ttGenTime, multi_submit_t *pMultiSub) { msg_t *pMsg; DEFiRet; ISOBJ_TYPE_assert(pThis, tcps_sess); + if(pThis->iMsg == 0) { + DBGPRINTF("discarding zero-sized message\n"); + FINALIZE; + } + if(pThis->DoSubmitMessage != NULL) { pThis->DoSubmitMessage(pThis, pThis->pMsg, pThis->iMsg); FINALIZE; @@ -231,17 +250,23 @@ defaultDoSubmitMessage(tcps_sess_t *pThis, struct syslogTime *stTime, time_t ttG /* we now create our own message object and submit it to the queue */ CHKiRet(msgConstructWithTime(&pMsg, stTime, ttGenTime)); - /* first trim the buffer to what we have actually received */ - CHKmalloc(pMsg->pszRawMsg = malloc(sizeof(uchar) * pThis->iMsg)); - memcpy(pMsg->pszRawMsg, pThis->pMsg, pThis->iMsg); - pMsg->iLenRawMsg = pThis->iMsg; - MsgSetInputName(pMsg, pThis->pLstnInfo->pszInputName, pThis->pLstnInfo->lenInputName); + MsgSetRawMsg(pMsg, (char*)pThis->pMsg, pThis->iMsg); + MsgSetInputName(pMsg, pThis->pLstnInfo->pInputName); MsgSetFlowControlType(pMsg, eFLOWCTL_LIGHT_DELAY); pMsg->msgFlags = NEEDS_PARSING | PARSE_HOSTNAME; pMsg->bParseHOSTNAME = 1; MsgSetRcvFrom(pMsg, pThis->fromHost); CHKiRet(MsgSetRcvFromIP(pMsg, pThis->fromHostIP)); - CHKiRet(submitMsg(pMsg)); + MsgSetRuleset(pMsg, pThis->pLstnInfo->pRuleset); + + if(pMultiSub == NULL) { + CHKiRet(submitMsg(pMsg)); + } else { + pMultiSub->ppMsgs[pMultiSub->nElem++] = pMsg; + if(pMultiSub->nElem == pMultiSub->maxElem) + CHKiRet(multiSubmitMsg(pMultiSub)); + } + finalize_it: /* reset status variables */ @@ -295,7 +320,7 @@ PrepareClose(tcps_sess_t *pThis) */ dbgprintf("Extra data at end of stream in legacy syslog/tcp message - processing\n"); datetime.getCurrTime(&stTime, &ttGenTime); - defaultDoSubmitMessage(pThis, &stTime, ttGenTime); + defaultDoSubmitMessage(pThis, &stTime, ttGenTime, NULL); } finalize_it: @@ -314,10 +339,11 @@ Close(tcps_sess_t *pThis) ISOBJ_TYPE_assert(pThis, tcps_sess); netstrm.Destruct(&pThis->pStrm); - free(pThis->fromHost); - pThis->fromHost = NULL; /* not really needed, but... */ - free(pThis->fromHostIP); - pThis->fromHostIP = NULL; /* not really needed, but... */ + if(pThis->fromHost != NULL) { + prop.Destruct(&pThis->fromHost); + } + if(pThis->fromHostIP != NULL) + prop.Destruct(&pThis->fromHostIP); RETiRet; } @@ -330,7 +356,7 @@ Close(tcps_sess_t *pThis) * rgerhards, 2008-03-14 */ static rsRetVal -processDataRcvd(tcps_sess_t *pThis, char c, struct syslogTime *stTime, time_t ttGenTime) +processDataRcvd(tcps_sess_t *pThis, char c, struct syslogTime *stTime, time_t ttGenTime, multi_submit_t *pMultiSub) { DEFiRet; ISOBJ_TYPE_assert(pThis, tcps_sess); @@ -376,7 +402,7 @@ processDataRcvd(tcps_sess_t *pThis, char c, struct syslogTime *stTime, time_t tt if(pThis->iMsg >= iMaxLine) { /* emergency, we now need to flush, no matter if we are at end of message or not... */ dbgprintf("error: message received is larger than max msg size, we split it\n"); - defaultDoSubmitMessage(pThis, stTime, ttGenTime); + defaultDoSubmitMessage(pThis, stTime, ttGenTime, pMultiSub); /* we might think if it is better to ignore the rest of the * message than to treat it as a new one. Maybe this is a good * candidate for a configuration parameter... @@ -387,7 +413,7 @@ processDataRcvd(tcps_sess_t *pThis, char c, struct syslogTime *stTime, time_t tt if(( (c == '\n') || ((pThis->pSrv->addtlFrameDelim != TCPSRV_NO_ADDTL_DELIMITER) && (c == pThis->pSrv->addtlFrameDelim)) ) && pThis->eFraming == TCP_FRAMING_OCTET_STUFFING) { /* record delimiter? */ - defaultDoSubmitMessage(pThis, stTime, ttGenTime); + defaultDoSubmitMessage(pThis, stTime, ttGenTime, pMultiSub); pThis->inputState = eAtStrtFram; } else { /* IMPORTANT: here we copy the actual frame content to the message - for BOTH framing modes! @@ -404,7 +430,7 @@ processDataRcvd(tcps_sess_t *pThis, char c, struct syslogTime *stTime, time_t tt pThis->iOctetsRemain--; if(pThis->iOctetsRemain < 1) { /* we have end of frame! */ - defaultDoSubmitMessage(pThis, stTime, ttGenTime); + defaultDoSubmitMessage(pThis, stTime, ttGenTime, pMultiSub); pThis->inputState = eAtStrtFram; } } @@ -429,9 +455,12 @@ processDataRcvd(tcps_sess_t *pThis, char c, struct syslogTime *stTime, time_t tt * this *is* the *correct* reception step for all the data we received, because * we have just received a bunch of data! -- rgerhards, 2009-06-16 */ +#define NUM_MULTISUB 1024 static rsRetVal DataRcvd(tcps_sess_t *pThis, char *pData, size_t iLen) { + multi_submit_t multiSub; + msg_t *pMsgs[NUM_MULTISUB]; struct syslogTime stTime; time_t ttGenTime; char *pEnd; @@ -442,17 +471,26 @@ DataRcvd(tcps_sess_t *pThis, char *pData, size_t iLen) assert(iLen > 0); datetime.getCurrTime(&stTime, &ttGenTime); + multiSub.ppMsgs = pMsgs; + multiSub.maxElem = NUM_MULTISUB; + multiSub.nElem = 0; /* We now copy the message to the session buffer. */ pEnd = pData + iLen; /* this is one off, which is intensional */ while(pData < pEnd) { - CHKiRet(processDataRcvd(pThis, *pData++, &stTime, ttGenTime)); + CHKiRet(processDataRcvd(pThis, *pData++, &stTime, ttGenTime, &multiSub)); + } + + if(multiSub.nElem > 0) { + /* submit anything that was not yet submitted */ + CHKiRet(multiSubmitMsg(&multiSub)); } finalize_it: RETiRet; } +#undef NUM_MULTISUB /* queryInterface function @@ -499,6 +537,7 @@ CODESTARTObjClassExit(tcps_sess) objRelease(errmsg, CORE_COMPONENT); objRelease(netstrm, LM_NETSTRMS_FILENAME); objRelease(datetime, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); ENDObjClassExit(tcps_sess) @@ -511,6 +550,7 @@ BEGINObjClassInit(tcps_sess, 1, OBJ_IS_CORE_MODULE) /* class, version - CHANGE c CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(netstrm, LM_NETSTRMS_FILENAME)); CHKiRet(objUse(datetime, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); iMaxLine = glbl.GetMaxLine(); /* get maximum size we currently support */ @@ -521,7 +561,5 @@ BEGINObjClassInit(tcps_sess, 1, OBJ_IS_CORE_MODULE) /* class, version - CHANGE c OBJSetMethodHandler(objMethod_CONSTRUCTION_FINALIZER, tcps_sessConstructFinalize); ENDObjClassInit(tcps_sess) - - /* vim:set ai: */ diff --git a/tcps_sess.h b/tcps_sess.h index 5e59aaab..ec3a6af4 100644 --- a/tcps_sess.h +++ b/tcps_sess.h @@ -24,6 +24,7 @@ #define INCLUDED_TCPS_SESS_H #include "obj.h" +#include "prop.h" /* a forward-definition, we are somewhat cyclic */ struct tcpsrv_s; @@ -44,8 +45,8 @@ struct tcps_sess_s { int iOctetsRemain; /* Number of Octets remaining in message */ TCPFRAMINGMODE eFraming; uchar *pMsg; /* message (fragment) received */ - uchar *fromHost; - uchar *fromHostIP; + prop_t *fromHost; /* host name we received messages from */ + prop_t *fromHostIP; void *pUsr; /* a user-pointer */ rsRetVal (*DoSubmitMessage)(tcps_sess_t*, uchar*, int); /* submit message callback */ }; @@ -69,6 +69,7 @@ #include "netstrm.h" #include "nssel.h" #include "errmsg.h" +#include "ruleset.h" #include "unicode-helper.h" MODULE_TYPE_LIB @@ -81,12 +82,14 @@ MODULE_TYPE_LIB DEFobjStaticHelpers DEFobjCurrIf(conf) DEFobjCurrIf(glbl) +DEFobjCurrIf(ruleset) DEFobjCurrIf(tcps_sess) DEFobjCurrIf(errmsg) DEFobjCurrIf(net) DEFobjCurrIf(netstrms) DEFobjCurrIf(netstrm) DEFobjCurrIf(nssel) +DEFobjCurrIf(prop) /* add new listener port to listener port list @@ -104,8 +107,12 @@ addNewLstnPort(tcpsrv_t *pThis, uchar *pszPort) CHKmalloc(pEntry = malloc(sizeof(tcpLstnPortList_t))); pEntry->pszPort = pszPort; pEntry->pSrv = pThis; - CHKmalloc(pEntry->pszInputName = ustrdup(pThis->pszInputName)); - pEntry->lenInputName = ustrlen(pEntry->pszInputName); + pEntry->pRuleset = pThis->pRuleset; + + /* we need to create a property */ + CHKiRet(prop.Construct(&pEntry->pInputName)); + CHKiRet(prop.SetString(pEntry->pInputName, pThis->pszInputName, ustrlen(pThis->pszInputName))); + CHKiRet(prop.ConstructFinalize(pEntry->pInputName)); /* and add to list */ pEntry->pNext = pThis->pLstnPorts; @@ -247,14 +254,14 @@ static void deinit_tcp_listener(tcpsrv_t *pThis) pEntry = pThis->pLstnPorts; while(pEntry != NULL) { free(pEntry->pszPort); - free(pEntry->pszInputName); + prop.Destruct(&pEntry->pInputName); pDel = pEntry; pEntry = pEntry->pNext; free(pDel); } /* finally close our listen streams */ - for(i = 0 ; i < pThis->iLstnMax ; ++i) { + for(i = 0 ; i < pThis->iLstnCurr ; ++i) { netstrm.Destruct(pThis->ppLstn + i); } } @@ -273,12 +280,12 @@ addTcpLstn(void *pUsr, netstrm_t *pLstn) ISOBJ_TYPE_assert(pThis, tcpsrv); ISOBJ_TYPE_assert(pLstn, netstrm); - if(pThis->iLstnMax >= TCPLSTN_MAX_DEFAULT) + if(pThis->iLstnCurr >= pThis->iLstnMax) ABORT_FINALIZE(RS_RET_MAX_LSTN_REACHED); - pThis->ppLstn[pThis->iLstnMax] = pLstn; - pThis->ppLstnPort[pThis->iLstnMax] = pPortList; - ++pThis->iLstnMax; + pThis->ppLstn[pThis->iLstnCurr] = pLstn; + pThis->ppLstnPort[pThis->iLstnCurr] = pPortList; + ++pThis->iLstnCurr; finalize_it: RETiRet; @@ -320,15 +327,19 @@ finalize_it: static rsRetVal create_tcp_socket(tcpsrv_t *pThis) { - tcpLstnPortList_t *pEntry; DEFiRet; + rsRetVal localRet; + tcpLstnPortList_t *pEntry; ISOBJ_TYPE_assert(pThis, tcpsrv); /* init all configured ports */ pEntry = pThis->pLstnPorts; while(pEntry != NULL) { - CHKiRet(initTCPListener(pThis, pEntry)); + localRet = initTCPListener(pThis, pEntry); + if(localRet != RS_RET_OK) { + errmsg.LogError(0, localRet, "Could not create tcp listener, ignoring port %s.", pEntry->pszPort); + } pEntry = pEntry->pNext; } @@ -481,7 +492,7 @@ Run(tcpsrv_t *pThis) CHKiRet(nssel.ConstructFinalize(pSel)); /* Add the TCP listen sockets to the list of read descriptors. */ - for(i = 0 ; i < pThis->iLstnMax ; ++i) { + for(i = 0 ; i < pThis->iLstnCurr ; ++i) { CHKiRet(nssel.Add(pSel, pThis->ppLstn[i], NSDSEL_RD)); } @@ -497,7 +508,7 @@ Run(tcpsrv_t *pThis) /* wait for io to become ready */ CHKiRet(nssel.Wait(pSel, &nfds)); - for(i = 0 ; i < pThis->iLstnMax ; ++i) { + for(i = 0 ; i < pThis->iLstnCurr ; ++i) { CHKiRet(nssel.IsReady(pSel, pThis->ppLstn[i], NSDSEL_RD, &bIsReady, &nfds)); if(bIsReady) { dbgprintf("New connect on NSD %p.\n", pThis->ppLstn[i]); @@ -518,6 +529,14 @@ Run(tcpsrv_t *pThis) iRet = pThis->pRcvData(pThis->pSessions[iTCPSess], buf, sizeof(buf), &iRcvd); switch(iRet) { case RS_RET_CLOSED: + if(pThis->bEmitMsgOnClose) { + uchar *pszPeer; + int lenPeer; + errno = 0; + prop.GetString(pThis->pSessions[iTCPSess]->fromHostIP, &pszPeer, &lenPeer); + errmsg.LogError(0, RS_RET_PEER_CLOSED_CONN, "Netstream session %p closed by remote peer %s.\n", + pThis->pSessions[iTCPSess]->pStrm, pszPeer); + } pThis->pOnRegularClose(pThis->pSessions[iTCPSess]); tcps_sess.Destruct(&pThis->pSessions[iTCPSess]); break; @@ -568,7 +587,8 @@ finalize_it: /* this is a very special case - this time only we do not exit the /* Standard-Constructor */ BEGINobjConstruct(tcpsrv) /* be sure to specify the object type also in END macro! */ - pThis->iSessMax = TCPSESS_MAX_DEFAULT; /* TODO: useful default ;) */ + pThis->iSessMax = TCPSESS_MAX_DEFAULT; + pThis->iLstnMax = TCPLSTN_MAX_DEFAULT; pThis->addtlFrameDelim = TCPSRV_NO_ADDTL_DELIMITER; pThis->OnMsgReceive = NULL; ENDobjConstruct(tcpsrv) @@ -592,8 +612,8 @@ tcpsrvConstructFinalize(tcpsrv_t *pThis) CHKiRet(netstrms.ConstructFinalize(pThis->pNS)); /* set up listeners */ - CHKmalloc(pThis->ppLstn = calloc(TCPLSTN_MAX_DEFAULT, sizeof(netstrm_t*))); - CHKmalloc(pThis->ppLstnPort = calloc(TCPLSTN_MAX_DEFAULT, sizeof(tcpLstnPortList_t*))); + CHKmalloc(pThis->ppLstn = calloc(pThis->iLstnMax, sizeof(netstrm_t*))); + CHKmalloc(pThis->ppLstnPort = calloc(pThis->iLstnMax, sizeof(tcpLstnPortList_t*))); iRet = pThis->OpenLstnSocks(pThis); finalize_it: @@ -756,6 +776,26 @@ finalize_it: } +/* Set the ruleset (ptr) to use */ +static rsRetVal +SetRuleset(tcpsrv_t *pThis, ruleset_t *pRuleset) +{ + DEFiRet; + pThis->pRuleset = pRuleset; + RETiRet; +} + + +/* Set connection close notification */ +static rsRetVal +SetNotificationOnRemoteClose(tcpsrv_t *pThis, int bNewVal) +{ + DEFiRet; + pThis->bEmitMsgOnClose = bNewVal; + RETiRet; +} + + /* here follows a number of methods that shuffle authentication settings down * to the drivers. Drivers not supporting these settings may return an error * state. @@ -800,6 +840,20 @@ SetDrvrPermPeers(tcpsrv_t *pThis, permittedPeers_t *pPermPeers) * -------------------------------------------------------------------------- */ +/* set max number of listeners + * this must be called before ConstructFinalize, or it will have no effect! + * rgerhards, 2009-08-17 + */ +static rsRetVal +SetLstnMax(tcpsrv_t *pThis, int iMax) +{ + DEFiRet; + ISOBJ_TYPE_assert(pThis, tcpsrv); + pThis->iLstnMax = iMax; + RETiRet; +} + + /* set max number of sessions * this must be called before ConstructFinalize, or it will have no effect! * rgerhards, 2009-04-09 @@ -842,6 +896,7 @@ CODESTARTobjQueryInterface(tcpsrv) pIf->SetInputName = SetInputName; pIf->SetAddtlFrameDelim = SetAddtlFrameDelim; pIf->SetSessMax = SetSessMax; + pIf->SetLstnMax = SetLstnMax; pIf->SetDrvrMode = SetDrvrMode; pIf->SetDrvrAuthMode = SetDrvrAuthMode; pIf->SetDrvrPermPeers = SetDrvrPermPeers; @@ -856,6 +911,8 @@ CODESTARTobjQueryInterface(tcpsrv) pIf->SetCBOnRegularClose = SetCBOnRegularClose; pIf->SetCBOnErrClose = SetCBOnErrClose; pIf->SetOnMsgReceive = SetOnMsgReceive; + pIf->SetRuleset = SetRuleset; + pIf->SetNotificationOnRemoteClose = SetNotificationOnRemoteClose; finalize_it: ENDobjQueryInterface(tcpsrv) @@ -869,6 +926,8 @@ CODESTARTObjClassExit(tcpsrv) /* release objects we no longer need */ objRelease(tcps_sess, DONT_LOAD_LIB); objRelease(conf, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); + objRelease(ruleset, CORE_COMPONENT); objRelease(glbl, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); objRelease(netstrms, DONT_LOAD_LIB); @@ -892,6 +951,8 @@ BEGINObjClassInit(tcpsrv, 1, OBJ_IS_LOADABLE_MODULE) /* class, version - CHANGE CHKiRet(objUse(tcps_sess, DONT_LOAD_LIB)); CHKiRet(objUse(conf, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(ruleset, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); /* set our own handlers */ OBJSetMethodHandler(objMethod_DEBUGPRINT, tcpsrvDebugPrint); @@ -23,6 +23,7 @@ #define INCLUDED_TCPSRV_H #include "obj.h" +#include "prop.h" #include "tcps_sess.h" /* support for framing anomalies */ @@ -36,9 +37,9 @@ typedef enum ETCPsyslogFramingAnomaly { /* list of tcp listen ports */ struct tcpLstnPortList_s { uchar *pszPort; /**< the ports the listener shall listen on */ - uchar *pszInputName; /**< value to be used as input name */ - size_t lenInputName; /**< length of inputName */ + prop_t *pInputName; tcpsrv_t *pSrv; /**< pointer to higher-level server instance */ + ruleset_t *pRuleset; /**< associated ruleset */ tcpLstnPortList_t *pNext; /**< next port or NULL */ }; @@ -51,12 +52,16 @@ struct tcpsrv_s { int iDrvrMode; /**< mode of the stream driver to use */ uchar *pszDrvrAuthMode; /**< auth mode of the stream driver to use */ uchar *pszInputName; /**< value to be used as input name */ + ruleset_t *pRuleset; /**< ruleset to bind to */ permittedPeers_t *pPermPeers;/**< driver's permitted peers */ - int iLstnMax; /**< max nbr of listeners currently supported */ + bool bEmitMsgOnClose; /**< emit an informational message when the remote peer closes connection */ + int iLstnCurr; /**< max nbr of listeners currently supported */ netstrm_t **ppLstn; /**< our netstream listners */ tcpLstnPortList_t **ppLstnPort; /**< pointer to relevant listen port description */ + int iLstnMax; /**< max number of listners supported */ int iSessMax; /**< max number of sessions supported */ tcpLstnPortList_t *pLstnPorts; /**< head pointer for listen ports */ + int addtlFrameDelim; /**< additional frame delimiter for plain TCP syslog framing (e.g. to handle NetScreen) */ tcps_sess_t **pSessions;/**< array of all of our sessions */ void *pUsr; /**< a user-settable pointer (provides extensibility for "derived classes")*/ @@ -108,8 +113,12 @@ BEGINinterface(tcpsrv) /* name must also be changed in ENDinterface macro! */ rsRetVal (*SetSessMax)(tcpsrv_t *pThis, int iMaxSess); /* 2009-04-09 */ /* added v6 */ rsRetVal (*SetOnMsgReceive)(tcpsrv_t *pThis, rsRetVal (*OnMsgReceive)(tcps_sess_t*, uchar*, int)); /* 2009-05-24 */ + rsRetVal (*SetRuleset)(tcpsrv_t *pThis, ruleset_t*); /* 2009-06-12 */ + /* added v7 */ + rsRetVal (*SetLstnMax)(tcpsrv_t *pThis, int iMaxLstn); /* 2009-08-17 */ + rsRetVal (*SetNotificationOnRemoteClose)(tcpsrv_t *pThis, int bNewVal); /* 2009-10-01 */ ENDinterface(tcpsrv) -#define tcpsrvCURR_IF_VERSION 6 /* increment whenever you change the interface structure! */ +#define tcpsrvCURR_IF_VERSION 8 /* increment whenever you change the interface structure! */ /* change for v4: * - SetAddtlFrameDelim() added -- rgerhards, 2008-12-10 * - SetInputName() added -- rgerhards, 2008-12-10 @@ -49,62 +49,62 @@ static struct template *tplLastStatic = NULL; /* last static element of the temp -/* This functions converts a template into a string. It should - * actually be in template.c, but this requires larger re-structuring - * of the code (because all the property-access functions are static - * to this module). I have placed it next to the iov*() functions, as - * it is somewhat similiar in what it does. - * - * The function takes a pointer to a template and a pointer to a msg object. - * It the creates a string based on the template definition. A pointer - * to that string is returned to the caller. The caller MUST FREE that - * pointer when it is no longer needed. If the function fails, NULL - * is returned. - * If memory allocation fails in this function, we silently return - * NULL. The reason is that we can not do anything against it. And - * if we raise an alert, the memory situation might become even - * worse. So we prefer to let the caller deal with it. - * rgerhards, 2007-07-03 +/* helper to tplToString, extends buffer */ +#define ALLOC_INC 128 +static inline rsRetVal ExtendBuf(uchar **pBuf, size_t *pLenBuf, size_t iMinSize) +{ + uchar *pNewBuf; + size_t iNewSize; + DEFiRet; + + iNewSize = (iMinSize / ALLOC_INC + 1) * ALLOC_INC; + CHKmalloc(pNewBuf = (uchar*) realloc(*pBuf, iNewSize)); + *pBuf = pNewBuf; + *pLenBuf = iNewSize; + +finalize_it: + RETiRet; +} + + +/* This functions converts a template into a string. * - * rgerhards, 2007-09-05: I changed the interface to use the standard iRet - * "calling sequence". This greatly eases complexity when it comes to handling - * errors in called modules (plus, it is much nicer). + * The function takes a pointer to a template and a pointer to a msg object + * as well as a pointer to an output buffer and its size. Note that the output + * buffer pointer may be NULL, size 0, in which case a new one is allocated. + * The outpub buffer is grown as required. It is the caller's duty to free the + * buffer when it is done. Note that it is advisable to reuse memory, as this + * offers big performance improvements. + * rewritten 2009-06-19 rgerhards */ -rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar** ppSz) +rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar **ppBuf, size_t *pLenBuf) { DEFiRet; struct templateEntry *pTpe; - cstr_t *pCStr; + int iBuf; unsigned short bMustBeFreed; uchar *pVal; size_t iLenVal; assert(pTpl != NULL); assert(pMsg != NULL); - assert(ppSz != NULL); + assert(ppBuf != NULL); + assert(pLenBuf != NULL); /* loop through the template. We obtain one value * and copy it over to our dynamic string buffer. Then, we * free the obtained value (if requested). We continue this * loop until we got hold of all values. */ - CHKiRet(cstrConstruct(&pCStr)); - pTpe = pTpl->pEntryRoot; + iBuf = 0; while(pTpe != NULL) { if(pTpe->eEntryType == CONSTANT) { - CHKiRet_Hdlr(rsCStrAppendStrWithLen(pCStr, - (uchar *) pTpe->data.constant.pConstant, - pTpe->data.constant.iLenConstant) - ) { - dbgprintf("error %d during tplToString()\n", iRet); - /* it does not make sense to continue now */ - cstrDestruct(&pCStr); - FINALIZE; - } + pVal = (uchar*) pTpe->data.constant.pConstant; + iLenVal = pTpe->data.constant.iLenConstant; + bMustBeFreed = 0; } else if(pTpe->eEntryType == FIELD) { - pVal = (uchar*) MsgGetProp(pMsg, pTpe, NULL, &bMustBeFreed); - iLenVal = strlen((char*) pVal); + pVal = (uchar*) MsgGetProp(pMsg, pTpe, pTpe->data.field.propid, &iLenVal, &bMustBeFreed); /* we now need to check if we should use SQL option. In this case, * we must go over the generated string and escape '\'' characters. * rgerhards, 2005-09-22: the option values below look somewhat misplaced, @@ -115,30 +115,25 @@ rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar** ppSz) doSQLEscape(&pVal, &iLenVal, &bMustBeFreed, 1); else if(pTpl->optFormatForSQL == 2) doSQLEscape(&pVal, &iLenVal, &bMustBeFreed, 0); - /* value extracted, so lets copy */ - CHKiRet_Hdlr(rsCStrAppendStrWithLen(pCStr, (uchar*) pVal, iLenVal)) { - dbgprintf("error %d during tplToString()\n", iRet); - /* it does not make sense to continue now */ - cstrDestruct(&pCStr); - if(bMustBeFreed) - free(pVal); - FINALIZE; - } - if(bMustBeFreed) - free(pVal); } + /* got source, now copy over */ + if(iBuf + iLenVal + 1 >= *pLenBuf) /* we reserve one char for the final \0! */ + CHKiRet(ExtendBuf(ppBuf, pLenBuf, iBuf + iLenVal + 1)); + + if(iLenVal > 0) { /* may be zero depending on property */ + memcpy(*ppBuf + iBuf, pVal, iLenVal); + iBuf += iLenVal; + } + + if(bMustBeFreed) + free(pVal); + pTpe = pTpe->pNext; } - /* we are done with the template, now let's convert the result into a - * "real" (usable) string and discard the helper structures. - */ - CHKiRet(cstrFinalize(pCStr)); - CHKiRet(cstrConvSzStrAndDestruct(pCStr, &pVal, 0)); + (*ppBuf)[iBuf] = '\0'; /* space was reserved above (see copy) */ finalize_it: - *ppSz = (iRet == RS_RET_OK) ? pVal : NULL; - RETiRet; } @@ -158,6 +153,7 @@ rsRetVal tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr) struct templateEntry *pTpe; uchar **pArr; int iArr; + size_t propLen; unsigned short bMustBeFreed; uchar *pVal; @@ -178,7 +174,7 @@ rsRetVal tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr) if(pTpe->eEntryType == CONSTANT) { CHKmalloc(pArr[iArr] = (uchar*)strdup((char*) pTpe->data.constant.pConstant)); } else if(pTpe->eEntryType == FIELD) { - pVal = (uchar*) MsgGetProp(pMsg, pTpe, NULL, &bMustBeFreed); + pVal = (uchar*) MsgGetProp(pMsg, pTpe, pTpe->data.field.propid, &propLen, &bMustBeFreed); if(bMustBeFreed) { /* if it must be freed, it is our own private copy... */ pArr[iArr] = pVal; /* ... so we can use it! */ } else { @@ -378,7 +374,6 @@ static int do_Constant(unsigned char **pp, struct template *pTpl) if(cstrConstruct(&pStrB) != RS_RET_OK) return 1; - rsCStrSetAllocIncrement(pStrB, 32); /* process the message and expand escapes * (additional escapes can be added here if needed) */ @@ -431,10 +426,7 @@ static int do_Constant(unsigned char **pp, struct template *pTpl) } if((pTpe = tpeConstruct(pTpl)) == NULL) { - /* OK, we are out of luck. Let's invalidate the - * entry and that's it. - */ - pTpe->eEntryType = UNDEFINED; + rsCStrDestruct(&pStrB); return 1; } pTpe->eEntryType = CONSTANT; @@ -494,6 +486,8 @@ static void doOptions(unsigned char **pp, struct templateEntry *pTpe) pTpe->data.field.eDateFormat = tplFmtPgSQLDate; } else if(!strcmp((char*)Buf, "date-rfc3164")) { pTpe->data.field.eDateFormat = tplFmtRFC3164Date; + } else if(!strcmp((char*)Buf, "date-rfc3164-buggyday")) { + pTpe->data.field.eDateFormat = tplFmtRFC3164BuggyDate; } else if(!strcmp((char*)Buf, "date-rfc3339")) { pTpe->data.field.eDateFormat = tplFmtRFC3339Date; } else if(!strcmp((char*)Buf, "date-subseconds")) { @@ -567,10 +561,14 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) ++p; /* do NOT do this in tolower()! */ } - /* got the name*/ + /* got the name */ cstrFinalize(pStrB); - if(cstrConvSzStrAndDestruct(pStrB, &pTpe->data.field.pPropRepl, 0) != RS_RET_OK) + + if(propNameToID(pStrB, &pTpe->data.field.propid) != RS_RET_OK) { + cstrDestruct(&pStrB); return 1; + } + cstrDestruct(&pStrB); /* no longer needed, now use ID */ /* Check frompos, if it has an R, then topos should be a regex */ if(*p == ':') { @@ -1009,8 +1007,6 @@ void tplDeleteAll(void) regexp.regfree(&(pTpeDel->data.field.re)); } } - /*dbgprintf("(FIELD), value: '%s'", pTpeDel->data.field.pPropRepl);*/ - free(pTpeDel->data.field.pPropRepl); break; } /*dbgprintf("\n");*/ @@ -1066,8 +1062,6 @@ void tplDeleteNew(void) regexp.regfree(&(pTpeDel->data.field.re)); } } - /*dbgprintf("(FIELD), value: '%s'", pTpeDel->data.field.pPropRepl);*/ - free(pTpeDel->data.field.pPropRepl); break; } /*dbgprintf("\n");*/ @@ -1116,7 +1110,7 @@ void tplPrintList(void) pTpe->data.constant.pConstant); break; case FIELD: - dbgprintf("(FIELD), value: '%s' ", pTpe->data.field.pPropRepl); + dbgprintf("(FIELD), value: '%d' ", pTpe->data.field.propid); switch(pTpe->data.field.eDateFormat) { case tplFmtDefault: break; @@ -48,7 +48,7 @@ struct template { enum EntryTypes { UNDEFINED = 0, CONSTANT = 1, FIELD = 2 }; enum tplFormatTypes { tplFmtDefault = 0, tplFmtMySQLDate = 1, tplFmtRFC3164Date = 2, tplFmtRFC3339Date = 3, tplFmtPgSQLDate = 4, - tplFmtSecFrac = 5}; + tplFmtSecFrac = 5, tplFmtRFC3164BuggyDate = 6}; enum tplFormatCaseConvTypes { tplCaseConvNo = 0, tplCaseConvUpper = 1, tplCaseConvLower = 2 }; #include "msg.h" @@ -63,7 +63,7 @@ struct templateEntry { int iLenConstant; /* its length */ } constant; struct { - uchar *pPropRepl; /* pointer to property replacer string */ + propid_t propid; /* property to be used */ unsigned iFromPos; /* for partial strings only chars from this position ... */ unsigned iToPos; /* up to that one... */ #ifdef FEATURE_REGEXP @@ -127,7 +127,7 @@ void tplLastStaticInit(struct template *tpl); * rgerhards, 2007-08-06 */ rsRetVal tplToArray(struct template *pTpl, msg_t *pMsg, uchar*** ppArr); -rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar** ppSz); +rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar** ppSz, size_t *); rsRetVal doSQLEscape(uchar **pp, size_t *pLen, unsigned short *pbMustBeFreed, int escapeMode); rsRetVal templateInit(); diff --git a/tests/DiagTalker.java b/tests/DiagTalker.java index 04e12327..5a6f7dd5 100644 --- a/tests/DiagTalker.java +++ b/tests/DiagTalker.java @@ -30,7 +30,10 @@ public class DiagTalker { PrintWriter out = null; BufferedReader in = null; final String host = "127.0.0.1"; - final int port = 13500; + int port = 13500; + if(args.length > 1) { + port = Integer.parseInt(args[1]); + } try { diagSocket = new Socket(host, port); diff --git a/tests/Makefile.am b/tests/Makefile.am index 3f5a6c33..0045f00a 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -1,15 +1,47 @@ if ENABLE_TESTBENCH TESTRUNS = rt_init rscript -check_PROGRAMS = $(TESTRUNS) ourtail nettester tcpflood chkseq +check_PROGRAMS = $(TESTRUNS) ourtail nettester tcpflood chkseq randomgen TESTS = $(TESTRUNS) cfg.sh \ validation-run.sh \ imtcp-multiport.sh \ diskqueue.sh \ + diskqueue-fsync.sh \ manytcp.sh \ + sndrcv.sh \ + sndrcv_gzip.sh \ + asynwr_simple.sh \ + asynwr_timeout.sh \ + asynwr_small.sh \ + asynwr_tinybuf.sh \ + wr_large_async.sh \ + wr_large_sync.sh \ + asynwr_deadlock.sh \ + asynwr_deadlock2.sh \ + asynwr_deadlock4.sh \ + gzipwr_large.sh \ + gzipwr_large_dynfile.sh \ + dynfile_invld_async.sh \ + dynfile_invld_sync.sh \ + dynfile_invalid2.sh \ + complex1.sh \ + queue-persist.sh \ + pipeaction.sh + execonlyonce.sh \ queue-persist.sh if ENABLE_OMSTDOUT -TESTS += omod-if-array.sh parsertest.sh inputname.sh fieldtest.sh +TESTS += omod-if-array.sh \ + proprepltest.sh \ + parsertest.sh \ + timestamp.sh \ + inputname.sh \ + threadingmq.sh \ + threadingmqaq.sh \ + fieldtest.sh +endif + +if ENABLE_EXTENDED_TESTS +TESTS += random.sh endif check_JAVA = DiagTalker.java @@ -36,11 +68,38 @@ EXTRA_DIST= 1.rstest 2.rstest 3.rstest err1.rstest \ DevNull.cfgtest \ err1.rstest \ NoExistFile.cfgtest \ + timestamp.sh \ + testsuites/ts3164.conf \ + testsuites/mon1digit.ts3164 \ + testsuites/mon2digit.ts3164 \ + testsuites/Jan.ts3164 \ + testsuites/Feb.ts3164 \ + testsuites/Mar.ts3164 \ + testsuites/Apr.ts3164 \ + testsuites/May.ts3164 \ + testsuites/Jun.ts3164 \ + testsuites/Jul.ts3164 \ + testsuites/Aug.ts3164 \ + testsuites/Sep.ts3164 \ + testsuites/Oct.ts3164 \ + testsuites/Nov.ts3164 \ + testsuites/Dec.ts3164 \ + testsuites/ts3339.conf \ + testsuites/master.ts3339 \ + testsuites/tsmysql.conf \ + testsuites/master.tsmysql \ + testsuites/tspgsql.conf \ + testsuites/master.tspgsql \ + testsuites/subsecond.conf \ + testsuites/master.subsecond \ testsuites/parse1.conf \ testsuites/field1.conf \ testsuites/1.parse1 \ testsuites/2.parse1 \ testsuites/3.parse1 \ + testsuites/snare.parse1 \ + testsuites/oversizeTag-1.parse1 \ + testsuites/weird.parse1 \ testsuites/date1.parse1 \ testsuites/date2.parse1 \ testsuites/date3.parse1 \ @@ -51,6 +110,18 @@ EXTRA_DIST= 1.rstest 2.rstest 3.rstest err1.rstest \ testsuites/rfc5424-2.parse1 \ testsuites/rfc5424-3.parse1 \ testsuites/rfc5424-4.parse1 \ + testsuites/parse3.conf \ + testsuites/reallife.parse3 \ + testsuites/parse-nodate.conf \ + testsuites/samples.parse-nodate \ + testsuites/parse_invld_regex.conf \ + testsuites/samples.parse_invld_regex \ + testsuites/parse-3164-buggyday.conf \ + testsuites/samples.parse-3164-buggyday \ + testsuites/snare_ccoff_udp.conf \ + testsuites/samples.snare_ccoff_udp \ + testsuites/snare_ccoff_udp2.conf \ + testsuites/samples.snare_ccoff_udp2 \ testsuites/omod-if-array.conf \ testsuites/1.omod-if-array \ testsuites/1.field1 \ @@ -59,6 +130,8 @@ EXTRA_DIST= 1.rstest 2.rstest 3.rstest err1.rstest \ fieldtest.sh \ diskqueue.sh \ testsuites/diskqueue.conf \ + diskqueue-fsync.sh \ + testsuites/diskqueue-fsync.conf \ imtcp-multiport.sh \ testsuites/imtcp-multiport.conf \ manytcp.sh \ @@ -71,9 +144,63 @@ EXTRA_DIST= 1.rstest 2.rstest 3.rstest err1.rstest \ omod-if-array.sh \ diag.sh \ testsuites/diag-common.conf \ + testsuites/diag-common2.conf \ queue-persist.sh \ queue-persist-drvr.sh \ testsuites/queue-persist.conf \ + threadingmq.sh \ + testsuites/threadingmq.conf \ + threadingmqaq.sh \ + testsuites/threadingmqaq.conf \ + sndrcv_drvr.sh \ + sndrcv.sh \ + testsuites/sndrcv_sender.conf \ + testsuites/sndrcv_rcvr.conf \ + sndrcv_gzip.sh \ + testsuites/sndrcv_gzip_sender.conf \ + testsuites/sndrcv_gzip_rcvr.conf \ + pipeaction.sh \ + testsuites/pipeaction.conf \ + asynwr_simple.sh \ + testsuites/asynwr_simple.conf \ + asynwr_timeout.sh \ + testsuites/asynwr_timeout.conf \ + asynwr_small.sh \ + testsuites/asynwr_small.conf \ + asynwr_tinybuf.sh \ + testsuites/asynwr_tinybuf.conf \ + wr_large_async.sh \ + wr_large_sync.sh \ + wr_large.sh \ + testsuites/wr_large.conf \ + asynwr_deadlock.sh \ + testsuites/asynwr_deadlock.conf \ + asynwr_deadlock2.sh \ + testsuites/asynwr_deadlock2.conf \ + asynwr_deadlock4.sh \ + testsuites/asynwr_deadlock4.conf \ + gzipwr_large.sh \ + testsuites/gzipwr_large.conf \ + gzipwr_large_dynfile.sh \ + testsuites/gzipwr_large_dynfile.conf \ + complex1.sh \ + testsuites/complex1.conf \ + random.sh \ + testsuites/random.conf \ + dynfile_invld_async.sh \ + dynfile_invld_sync.sh \ + dynfile_cachemiss.sh \ + testsuites/dynfile_cachemiss.conf \ + dynfile_invalid2.sh \ + testsuites/dynfile_invalid2.conf \ + proprepltest.sh \ + testsuites/rfctag.conf \ + testsuites/master.rfctag \ + testsuites/nolimittag.conf \ + testsuites/master.nolimittag \ + execonlyonce.sh \ + testsuites/execonlyonce.conf \ + testsuites/execonlyonce.data \ DiagTalker.java \ cfg.sh @@ -83,6 +210,9 @@ chkseq_SOURCES = chkseq.c tcpflood_SOURCES = tcpflood.c tcpflood_LDADD = $(SOL_LIBS) +randomgen_SOURCES = randomgen.c +randomgen_LDADD = $(SOL_LIBS) + nettester_SOURCES = nettester.c getline.c nettester_LDADD = $(SOL_LIBS) diff --git a/tests/asynwr_deadlock.sh b/tests/asynwr_deadlock.sh new file mode 100755 index 00000000..dc08355e --- /dev/null +++ b/tests/asynwr_deadlock.sh @@ -0,0 +1,23 @@ +# This is test case from practice, with the version we introduced it, it +# caused a deadlock on shutdown. I have added it to the test suite to automatically +# detect such things in the future. +# +# added 2010-03-17 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo ================================================================================ +echo TEST: \[asynwr_deadlock.sh\]: a case known to have caused a deadlock in the past +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup asynwr_deadlock.conf +# just send one message +source $srcdir/diag.sh tcpflood -m1 +# sleep is important! need to make sure the instance is inactive +sleep 1 +# now try shutdown. The actual test is if the process does hang here! +echo "processing must continue soon" +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +source $srcdir/diag.sh seq-check 0 0 +source $srcdir/diag.sh exit diff --git a/tests/asynwr_deadlock2.sh b/tests/asynwr_deadlock2.sh new file mode 100755 index 00000000..1190f67a --- /dev/null +++ b/tests/asynwr_deadlock2.sh @@ -0,0 +1,69 @@ +# This is test case from practice, with the version we introduced it, it +# caused a deadlock during processing (when the a stream was purged from the +# dynafile cache). +# We added this as a standard test in the hopes that iw will help +# detect such things in the future. +# +# The problem that originally caused this test to fail was: +# We write to files asynchronously (with the async writer thread). There is +# no signaling done when the file stream is closed. That can lead to the writer +# process hanging in memory, that in turn leads to the main thread waiting on a +# condition that never occurs (because it would need to be signalled by the +# async writer). Even worse, in that case, the async writer was signalled invalid +# in such a way that when it received a wakeup, it thought it shall not terminate, +# but received a spurios wakeup due to timeout and no data to write. In that case +# it (correctly) concluded that it would not need to timeout until a new buffer write +# was done (in which case it would receive a wakeup). As such, it went into an eternal +# wait. However, the invalid signaling did not take into account that it did not +# signal the async writer to shut down. So the main thread went into a condition +# wait - and thus we had a deadlock. That situation occured only under very specific +# cirumstances. As far as the analysis goes, the following need to happen: +# 1. buffers on that file are being flushed +# 2. no new data arrives +# 3. the inactivity timeout has not yet expired +# 4. *then* (and only then) the stream is closed or destructed +# In that, 1 to 4 are prequisites for the deadlock which will happen in 4. However, +# for it to happen, we also need the right "timing". There is a race between the +# main thread and the async writer thread. The deadlock will only happen under +# the "right" circumstances, which basically means it will not happen always. +# In order to create this case as reliable as possible, I have used +# the "$OMFileFlushOnTXEnd on" directive +# inside my test case. It makes sure that #1 above happens. The test uses a dynafile +# cache size of 4, and the load generator generates data for 5 different dynafiles. +# So over time, we will hit a spot where 4 dynafiles are open and the 5th file name +# is generated. As such, one file needs to be discarded. Thanks to FlushOnTXEnd, we +# now likely have #2 in place and thanks to the load pattern generated, we most +# probably have #3 in place. During the dynafile cache displacement of the oldest +# entry, #4 is generated. At this point, we have the deadlock we are testing for. +# Note that this deadlock does not necessarily lead to a total lockup of rsyslogd. +# Parts of it continue to operate. But in our test setup, this means data is +# received and placed into the main queue. Once it's high water mark is hit, data +# is still being enqueued, but at a slow rate. So if one is patient enough, the load +# generator will be able to finish. However, rsyslogd will never process the data +# it received because it is locked in the deadlock caused by #4 above. +# Note that "$OMFileFlushOnTXEnd on" is not causing this behaviour. We just use it +# to (quite) reliably cause the failure condition. The failure described above +# (in version 4.6.1) was also present when the setting was set to "off", but its +# occurence was very much less probable - because the perquisites are then much +# harder to hit. without it, the test may need to run for several hours before +# we hit all failure conditions. +# +# added 2010-03-17 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo ================================================================================= +echo TEST: \[asynwr_deadlock2.sh\]: a case known to have caused a deadlock in the past +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup asynwr_deadlock2.conf +# send 20000 messages, each close to 2K (non-randomized!), so that we can fill +# the buffers and hopefully run into the "deadlock". +source $srcdir/diag.sh tcpflood -m20000 -d1800 -P129 -i1 -f5 +# the sleep below is needed to prevent too-early termination of the tcp listener +sleep 1 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +cat rsyslog.out.*.log > rsyslog.out.log +source $srcdir/diag.sh seq-check 1 20000 -E +source $srcdir/diag.sh exit diff --git a/tests/asynwr_deadlock4.sh b/tests/asynwr_deadlock4.sh new file mode 100755 index 00000000..a3452f5b --- /dev/null +++ b/tests/asynwr_deadlock4.sh @@ -0,0 +1,25 @@ +# This is test case from practice, with the version we introduced it, it +# caused a deadlock during processing. +# We added this as a standard test in the hopes that iw will help +# detect such things in the future. +# +# This is a test that is constructed similar to asynwr_deadlock2.sh, but +# can produce problems in a simpler way. +# +# added 2010-03-18 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo ================================================================================= +echo TEST: \[asynwr_deadlock4.sh\]: a case known to have caused a deadlock in the past +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup asynwr_deadlock4.conf +# send 20000 messages, each close to 2K (non-randomized!), so that we can fill +# the buffers and hopefully run into the "deadlock". +source $srcdir/diag.sh tcpflood -m20000 -d18 -P129 -i1 -f5 +# sleep is important! need to make sure the instance is inactive +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +source $srcdir/diag.sh seq-check 1 20000 -E +source $srcdir/diag.sh exit diff --git a/tests/asynwr_simple.sh b/tests/asynwr_simple.sh new file mode 100755 index 00000000..eb87443c --- /dev/null +++ b/tests/asynwr_simple.sh @@ -0,0 +1,18 @@ +# This is test driver for testing asynchronous file output. +# +# added 2010-03-09 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[asynwr_simple.sh\]: simple test for async file writing +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup asynwr_simple.conf +# send 35555 messages, make sure file size is not a multiple of +# 10K, the buffer size! +source $srcdir/diag.sh tcpflood -m35555 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +source $srcdir/diag.sh seq-check 0 35554 +source $srcdir/diag.sh exit diff --git a/tests/asynwr_small.sh b/tests/asynwr_small.sh new file mode 100755 index 00000000..97818f6e --- /dev/null +++ b/tests/asynwr_small.sh @@ -0,0 +1,26 @@ +# This tests async writing with only a small set of data. That +# shall result in data staying in buffers until shutdown, what +# then will trigger some somewhat complex logic in the stream +# writer (open, write, close all during the stream close +# opertion). It is vital that only few messages be sent. +# +# The main effort of this test is not (only) to see if we +# receive the data, but rather to see if we get into an abort +# condition. +# +# added 2010-03-09 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[asynwr_small.sh\]: test for async file writing for few messages +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup asynwr_small.conf +# send 4000 messages +source $srcdir/diag.sh tcpflood -m2 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +source $srcdir/diag.sh seq-check 0 1 +source $srcdir/diag.sh exit diff --git a/tests/asynwr_timeout.sh b/tests/asynwr_timeout.sh new file mode 100755 index 00000000..b0bc5c28 --- /dev/null +++ b/tests/asynwr_timeout.sh @@ -0,0 +1,21 @@ +# This test writes to the output buffers, let's the output +# write timeout (and write data) and then continue. The conf file +# has a 2 second timeout, so we wait 4 seconds to be on the save side. +# +# added 2010-03-09 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[asynwr_timeout.sh\]: test async file writing timeout writes +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup asynwr_timeout.conf +# send 35555 messages, make sure file size is not a multiple of +# 10K, the buffer size! +source $srcdir/diag.sh tcpflood -m 35555 +sleep 4 # wait for output writer to write and empty buffer +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +source $srcdir/diag.sh seq-check 0 35554 +source $srcdir/diag.sh exit diff --git a/tests/asynwr_tinybuf.sh b/tests/asynwr_tinybuf.sh new file mode 100755 index 00000000..8eae1e26 --- /dev/null +++ b/tests/asynwr_tinybuf.sh @@ -0,0 +1,19 @@ +# This tests async writing with a very small output buffer (1 byte!), +# so it stresses output buffer handling. This also means operations will +# be somewhat slow, so we send only a small amounts of data. +# +# added 2010-03-09 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[asynwr_tinybuf.sh\]: test async file writing with 1-byte buffer +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup asynwr_tinybuf.conf +# send 1000 messages, fairly enough to trigger problems +source $srcdir/diag.sh tcpflood -m1000 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh seq-check 0 999 +source $srcdir/diag.sh exit diff --git a/tests/chkseq.c b/tests/chkseq.c index 8c5fc61a..b22c8992 100644 --- a/tests/chkseq.c +++ b/tests/chkseq.c @@ -32,6 +32,7 @@ #include "config.h" #include <stdio.h> #include <stdlib.h> +#include <string.h> #include <getopt.h> int main(int argc, char *argv[]) @@ -40,13 +41,18 @@ int main(int argc, char *argv[]) int val; int i; int ret = 0; + int scanfOK; + int verbose = 0; + int bHaveExtraData = 0; int dupsPermitted = 0; int start = 0, end = 0; int opt; int nDups = 0; + int edLen; /* length of extra data */ + static char edBuf[500*1024]; /* buffer for extra data (pretty large to be on the save side...) */ char *file = NULL; - while((opt = getopt(argc, argv, "e:f:ds:")) != EOF) { + while((opt = getopt(argc, argv, "e:f:ds:vE")) != EOF) { switch((char)opt) { case 'f': file = optarg; @@ -60,8 +66,14 @@ int main(int argc, char *argv[]) case 's': start = atoi(optarg); break; - default:printf("Invalid call of chkseq\n"); - printf("Usage: chkseq file -sstart -eend -d\n"); + case 'v': + ++verbose; + break; + case 'E': + bHaveExtraData = 1; + break; + default:printf("Invalid call of chkseq, optchar='%c'\n", opt); + printf("Usage: chkseq file -sstart -eend -d -E\n"); exit(1); } } @@ -76,6 +88,10 @@ int main(int argc, char *argv[]) exit(1); } + if(verbose) { + printf("chkseq: start %d, end %d\n", start, end); + } + /* read file */ fp = fopen(file, "r"); if(fp == NULL) { @@ -85,7 +101,17 @@ int main(int argc, char *argv[]) } for(i = start ; i < end+1 ; ++i) { - if(fscanf(fp, "%d\n", &val) != 1) { + if(bHaveExtraData) { + scanfOK = fscanf(fp, "%d,%d,%s\n", &val, &edLen, edBuf) == 3 ? 1 : 0; + if(edLen != (int) strlen(edBuf)) { + printf("extra data length specified %d, but actually is %ld in record %d\n", + edLen, (long) strlen(edBuf), i); + exit(1); + } + } else { + scanfOK = fscanf(fp, "%d\n", &val) == 1 ? 1 : 0; + } + if(!scanfOK) { printf("scanf error in index i=%d\n", i); exit(1); } @@ -108,5 +134,10 @@ int main(int argc, char *argv[]) exit(1); } + if(!feof(fp)) { + printf("end of processing, but NOT end of file!\n"); + exit(1); + } + exit(ret); } diff --git a/tests/complex1.sh b/tests/complex1.sh new file mode 100755 index 00000000..7f3cd994 --- /dev/null +++ b/tests/complex1.sh @@ -0,0 +1,21 @@ +# This is a rather complex test that runs a number of features together. +# +# added 2010-03-16 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo ==================================================================================== +echo TEST: \[complex1.sh\]: complex test with gzip and multiple action queues +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup complex1.conf +# send 30,000 messages of 400 bytes plus header max, via three dest ports +source $srcdir/diag.sh tcpflood -m40000 -rd400 -P129 -f5 -n3 -c15 -i1 +sleep 2 # due to large messages, we need this time for the tcp receiver to settle... +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +ls rsyslog.out.*.log +zcat rsyslog.out.*.log > rsyslog.out.log +source $srcdir/diag.sh seq-check 1 40000 -E +source $srcdir/diag.sh exit diff --git a/tests/diag.sh b/tests/diag.sh index 1ceca75b..51ad5f6a 100755 --- a/tests/diag.sh +++ b/tests/diag.sh @@ -8,51 +8,67 @@ #valgrind="valgrind --log-fd=1" #valgrind="valgrind --tool=drd --log-fd=1" #valgrind="valgrind --tool=helgrind --log-fd=1" +#valgrind="valgrind --tool=exp-ptrcheck --log-fd=1" #set -o xtrace #export RSYSLOG_DEBUG="debug nostdout printmutexaction" #export RSYSLOG_DEBUGLOG="log" case $1 in 'init') $srcdir/killrsyslog.sh # kill rsyslogd if it runs for some reason cp $srcdir/testsuites/diag-common.conf diag-common.conf - rm -f rsyslogd.started work-*.conf + cp $srcdir/testsuites/diag-common2.conf diag-common2.conf + rm -f rsyslog.action.*.include + rm -f rsyslogd.started work-*.conf rsyslog.random.data + rm -f rsyslogd2.started work-*.conf rm -f work rsyslog.out.log rsyslog.out.log.save # common work files + rm -f rsyslog.out.*.log work-presort rm -rf test-spool + rm -f core.* vgcore.* mkdir test-spool ;; 'exit') rm -f rsyslogd.started work-*.conf diag-common.conf + rm -f rsyslogd2.started diag-common2.conf rsyslog.action.*.include rm -f work rsyslog.out.log rsyslog.out.log.save # common work files + rm -f rsyslog.out.*.log rsyslog.random.data work-presort rm -rf test-spool ;; 'startup') # start rsyslogd with default params. $2 is the config file name to use - # returns only after successful startup - $valgrind ../tools/rsyslogd -c4 -u2 -n -irsyslog.pid -M../runtime/.libs:../.libs -f$srcdir/testsuites/$2 & - $srcdir/diag.sh wait-startup + # returns only after successful startup, $3 is the instance (blank or 2!) + $valgrind ../tools/rsyslogd -c4 -u2 -n -irsyslog$3.pid -M../runtime/.libs:../.libs -f$srcdir/testsuites/$2 & + $srcdir/diag.sh wait-startup $3 ;; - 'wait-startup') # wait for rsyslogd startup - while test ! -f rsyslogd.started; do - true + 'wait-startup') # wait for rsyslogd startup ($2 is the instance) + while test ! -f rsyslogd$2.started; do + #true + sleep 0.1 # if this is not supported by all platforms, use above! done - echo "rsyslogd started with pid " `cat rsyslog.pid` + echo "rsyslogd$2 started with pid " `cat rsyslog$2.pid` ;; - 'wait-shutdown') # actually, we wait for rsyslog.pid to be deleted - while test -f rsyslog.pid; do - true + 'wait-shutdown') # actually, we wait for rsyslog.pid to be deleted. $2 is the + # instance + while test -f rsyslog$2.pid; do + #true + sleep 0.1 # if this is not supported by all platforms, use above! done ;; - 'wait-queueempty') # wait for main message queue to be empty - echo WaitMainQueueEmpty | java -classpath $abs_top_builddir DiagTalker + 'wait-queueempty') # wait for main message queue to be empty. $2 is the instance. + if [ "$2" == "2" ] + then + echo WaitMainQueueEmpty | java -classpath $abs_top_builddir DiagTalker + else + echo WaitMainQueueEmpty | java -classpath $abs_top_builddir DiagTalker 13501 + fi ;; - 'shutdown-when-empty') # shut rsyslogd down when main queue is empty - $srcdir/diag.sh wait-queueempty - kill `cat rsyslog.pid` + 'shutdown-when-empty') # shut rsyslogd down when main queue is empty. $2 is the instance. + $srcdir/diag.sh wait-queueempty $2 + kill `cat rsyslog$2.pid` # note: we do not wait for the actual termination! ;; - 'shutdown-immediate') # shut rsyslogd down without emptying the queue + 'shutdown-immediate') # shut rsyslogd down without emptying the queue. $2 is the instance. kill `cat rsyslog.pid` # note: we do not wait for the actual termination! ;; 'tcpflood') # do a tcpflood run and check if it worked params are passed to tcpflood - ./tcpflood $2 $3 $4 $5 $6 $7 $8 + ./tcpflood $2 $3 $4 $5 $6 $7 $8 $9 if [ "$?" -ne "0" ]; then echo "error during tcpflood! see rsyslog.out.log.save for what was written" cp rsyslog.out.log rsyslog.out.log.save @@ -60,10 +76,11 @@ case $1 in fi ;; 'injectmsg') # inject messages via our inject interface (imdiag) + echo injecting $3 messages echo injectmsg $2 $3 $4 $5 | java -classpath $abs_top_builddir DiagTalker # TODO: some return state checking? (does it really make sense here?) ;; - 'check-mainq-spool') # check if mainqueue spool files exist, if not abort (we just check .qi) + 'check-mainq-spool') # check if mainqueue spool files exist, if not abort (we just check .qi). echo There must exist some files now: ls -l test-spool if test ! -f test-spool/mainq.qi; then @@ -72,14 +89,47 @@ case $1 in exit 1 fi ;; - 'seq-check') # do the usual sequence check to see if everything was properly received + 'seq-check') # do the usual sequence check to see if everything was properly received. $2 is the instance. rm -f work + cp rsyslog.out.log work-presort sort < rsyslog.out.log > work - ./chkseq -fwork -e$2 $3 + # $4... are just to have the abilit to pass in more options... + ./chkseq -fwork -v -s$2 -e$3 $4 $5 $6 $7 if [ "$?" -ne "0" ]; then echo "sequence error detected" exit 1 fi ;; + 'seq-check2') # do the usual sequence check to see if everything was properly received. This is + # a duplicateof seq-check, but we could not change its calling conventions without + # breaking a lot of exitings test cases, so we preferred to duplicate the code here. + rm -f work2 + sort < rsyslog2.out.log > work2 + # $4... are just to have the abilit to pass in more options... + ./chkseq -fwork2 -v -s$2 -e$3 $4 $5 $6 $7 + if [ "$?" -ne "0" ]; then + echo "sequence error detected" + exit 1 + fi + ;; + 'gzip-seq-check') # do the usual sequence check, but for gzip files + rm -f work + ls -l rsyslog.out.log + gunzip < rsyslog.out.log | sort > work + ls -l work + # $4... are just to have the abilit to pass in more options... + ./chkseq -fwork -v -s$2 -e$3 $4 $5 $6 $7 + if [ "$?" -ne "0" ]; then + echo "sequence error detected" + exit 1 + fi + ;; + 'nettester') # perform nettester-based tests + # use -v for verbose output! + ./nettester -t$2 -i$3 + if [ "$?" -ne "0" ]; then + exit 1 + fi + ;; *) echo "invalid argument" $1 esac diff --git a/tests/diskqueue-fsync.sh b/tests/diskqueue-fsync.sh new file mode 100755 index 00000000..c7745930 --- /dev/null +++ b/tests/diskqueue-fsync.sh @@ -0,0 +1,16 @@ +# Test for disk-only queue mode (with fsync for queue files) +# This test checks if queue files can be correctly written +# and read back, but it does not test the transition from +# memory to disk mode for DA queues. +# added 2009-06-09 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +# uncomment for debugging support: +echo testing queue disk-only mode, fsync case +source $srcdir/diag.sh init +source $srcdir/diag.sh startup diskqueue-fsync.conf +# 1000 messages should be enough - the disk fsync test is very slow! +source $srcdir/diag.sh tcpflood -m1000 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 999 +source $srcdir/diag.sh exit diff --git a/tests/diskqueue.sh b/tests/diskqueue.sh index 2fe31db9..7a50d82e 100755 --- a/tests/diskqueue.sh +++ b/tests/diskqueue.sh @@ -5,11 +5,16 @@ # added 2009-04-17 by Rgerhards # This file is part of the rsyslog project, released under GPLv3 # uncomment for debugging support: -echo testing queue disk-only mode +echo diskqueue.sh: testing queue disk-only mode +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" source $srcdir/diag.sh init source $srcdir/diag.sh startup diskqueue.conf # 20000 messages should be enough - the disk test is slow enough ;) -source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 20000 +sleep 4 +source $srcdir/diag.sh tcpflood -m20000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 19999 source $srcdir/diag.sh exit diff --git a/tests/dynfile_cachemiss.sh b/tests/dynfile_cachemiss.sh new file mode 100755 index 00000000..6e2d9cca --- /dev/null +++ b/tests/dynfile_cachemiss.sh @@ -0,0 +1,34 @@ +# This test checks if omfile segfaults when a file open() in dynacache mode fails. +# The test is mimiced after a real-life scenario (which, of course, was much more +# complex). +# +# added 2010-03-09 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[dynfile_cachemiss.sh\]: test open fail for dynafiles with `cat rsyslog.action.1.include` +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup dynfile_cachemiss.conf +# we send handcrafted message. We have a dynafile cache of 4, and now send one message +# each to fill up the cache. +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.0.log:0" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.1.log:1" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.2.log:2" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.3.log:3" +# the next one has caused a segfault in practice +# note that /proc/rsyslog.error.file must not be creatable +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:/proc/rsyslog.error.file:boom" +# some more writes +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.0.log:4" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.1.log:5" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.2.log:6" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.3.log:7" +# done message generation +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +cat rsyslog.out.*.log > rsyslog.out.log +source $srcdir/diag.sh seq-check 0 7 +source $srcdir/diag.sh exit diff --git a/tests/dynfile_invalid2.sh b/tests/dynfile_invalid2.sh new file mode 100755 index 00000000..cb3ef51e --- /dev/null +++ b/tests/dynfile_invalid2.sh @@ -0,0 +1,34 @@ +# This test checks if omfile segfaults when a file open() in dynacache mode fails. +# The test is mimiced after a real-life scenario (which, of course, was much more +# complex). +# +# added 2010-03-22 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[dynfile_invalid2.sh\]: test open fail for dynafiles +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup dynfile_invalid2.conf +# we send handcrafted message. We have a dynafile cache of 4, and now send one message +# each to fill up the cache. +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.0.log:0" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.1.log:1" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.2.log:2" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.3.log:3" +# the next one has caused a segfault in practice +# note that /proc/rsyslog.error.file must not be creatable +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:/proc/rsyslog.error.file:boom" +# some more writes +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.0.log:4" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.1.log:5" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.2.log:6" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.3.log:7" +# done message generation +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +cat rsyslog.out.*.log > rsyslog.out.log +source $srcdir/diag.sh seq-check 0 7 +source $srcdir/diag.sh exit diff --git a/tests/dynfile_invld_async.sh b/tests/dynfile_invld_async.sh new file mode 100755 index 00000000..3c9b2045 --- /dev/null +++ b/tests/dynfile_invld_async.sh @@ -0,0 +1,2 @@ +echo "\$OMFileAsyncWriting on" > rsyslog.action.1.include +source $srcdir/dynfile_cachemiss.sh diff --git a/tests/dynfile_invld_sync.sh b/tests/dynfile_invld_sync.sh new file mode 100755 index 00000000..cc6b6451 --- /dev/null +++ b/tests/dynfile_invld_sync.sh @@ -0,0 +1,2 @@ +echo "\$OMFileAsyncWriting off" > rsyslog.action.1.include +source $srcdir/dynfile_cachemiss.sh diff --git a/tests/execonlyonce.sh b/tests/execonlyonce.sh new file mode 100755 index 00000000..b7f60849 --- /dev/null +++ b/tests/execonlyonce.sh @@ -0,0 +1,28 @@ +# Test for the $ActionExecOnlyOnceEveryInterval directive. +# We inject a couple of messages quickly during the interval, +# then wait until the interval expires, then quickly inject +# another set. After that, it is checked if exactly two messages +# have arrived. +# The once interval must be set to 3 seconds in the config file. +# added 2009-11-12 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo \[execonlyonce.sh\]: test for the $ActionExecOnlyOnceEveryInterval directive +source $srcdir/diag.sh init +source $srcdir/diag.sh startup execonlyonce.conf +source $srcdir/diag.sh tcpflood -m10 -i1 +# now wait until the interval definitely expires +sleep 4 # one more than the once inerval! +# and inject another couple of messages +source $srcdir/diag.sh tcpflood -m10 -i100 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages + +# now we need your custom logic to see if the result is equal to the +# expected result +cmp rsyslog.out.log testsuites/execonlyonce.data +if [ $? -eq 1 ] +then + echo "ERROR, output not as expected" + exit 1 +fi +source $srcdir/diag.sh exit diff --git a/tests/gzipwr_large.sh b/tests/gzipwr_large.sh new file mode 100755 index 00000000..ffce06f6 --- /dev/null +++ b/tests/gzipwr_large.sh @@ -0,0 +1,20 @@ +# This tests writing large data records in gzip mode. We use up to 10K +# record size. +# +# added 2010-03-10 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[gzipwr_large.sh\]: test for gzip file writing for large message sets +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup gzipwr_large.conf +# send 4000 messages of 10.000bytes plus header max, randomized +source $srcdir/diag.sh tcpflood -m4000 -r -d10000 -P129 +sleep 1 # due to large messages, we need this time for the tcp receiver to settle... +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +source $srcdir/diag.sh gzip-seq-check 0 3999 -E +source $srcdir/diag.sh exit diff --git a/tests/gzipwr_large_dynfile.sh b/tests/gzipwr_large_dynfile.sh new file mode 100755 index 00000000..73d44796 --- /dev/null +++ b/tests/gzipwr_large_dynfile.sh @@ -0,0 +1,36 @@ +# This tests writing large data records in gzip mode. We also write it to +# 5 different dynafiles, with a dynafile cache size set to 4. So this stresses +# both the input side, as well as zip writing, async writing and the dynafile +# cache logic. +# +# This test is a bit timing-dependent on the tcp reception side, so if it fails +# one may look into the timing first. The main issue is that the testbench +# currently has no good way to know if the tcp receiver is finished. This is NOT +# a problem in rsyslogd, but only of the testbench. +# +# Note that we do not yet have sufficient support for dynafiles in diag.sh, +# so we mangle some files here manually. +# +# added 2010-03-10 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo ==================================================================================== +echo TEST: \[gzipwr_large_dynfile.sh\]: test for gzip file writing for large message sets +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup gzipwr_large_dynfile.conf +# send 4000 messages of 10.000bytes plus header max, randomized +source $srcdir/diag.sh tcpflood -m4000 -r -d10000 -P129 -f5 +sleep 2 # due to large messages, we need this time for the tcp receiver to settle... +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +gunzip < rsyslog.out.0.log > rsyslog.out.log +gunzip < rsyslog.out.1.log >> rsyslog.out.log +gunzip < rsyslog.out.2.log >> rsyslog.out.log +gunzip < rsyslog.out.3.log >> rsyslog.out.log +gunzip < rsyslog.out.4.log >> rsyslog.out.log +#cat rsyslog.out.* > rsyslog.out.log +source $srcdir/diag.sh seq-check 0 3999 -E +source $srcdir/diag.sh exit diff --git a/tests/imtcp-multiport.sh b/tests/imtcp-multiport.sh index 702f8834..ad2b44f8 100755 --- a/tests/imtcp-multiport.sh +++ b/tests/imtcp-multiport.sh @@ -7,11 +7,13 @@ # # added 2009-05-22 by Rgerhards # This file is part of the rsyslog project, released under GPLv3 -echo testing imtcp multiple listeners +echo =============================================================================== +echo \[imtcp-multiport.sh\]: testing imtcp multiple listeners source $srcdir/diag.sh init source $srcdir/diag.sh startup imtcp-multiport.conf -source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 10000 +source $srcdir/diag.sh tcpflood -p13514 -m10000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 9999 source $srcdir/diag.sh exit # @@ -21,8 +23,9 @@ source $srcdir/diag.sh exit # source $srcdir/diag.sh init source $srcdir/diag.sh startup imtcp-multiport.conf -source $srcdir/diag.sh tcpflood 127.0.0.1 13515 1 10000 +source $srcdir/diag.sh tcpflood -p13515 -m10000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 9999 source $srcdir/diag.sh exit # @@ -32,7 +35,8 @@ source $srcdir/diag.sh exit # source $srcdir/diag.sh init source $srcdir/diag.sh startup imtcp-multiport.conf -source $srcdir/diag.sh tcpflood 127.0.0.1 13516 1 10000 +source $srcdir/diag.sh tcpflood -p13516 -m10000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 9999 source $srcdir/diag.sh exit diff --git a/tests/killrsyslog.sh b/tests/killrsyslog.sh index b1be757b..aac24909 100755 --- a/tests/killrsyslog.sh +++ b/tests/killrsyslog.sh @@ -2,6 +2,12 @@ if [ -e "rsyslog.pid" ] then echo rsyslog.pid exists, trying to shut down rsyslogd process `cat rsyslog.pid`. - kill `cat rsyslog.pid` + kill -9 `cat rsyslog.pid` + sleep 1 +fi +if [ -e "rsyslog2.pid" ] +then + echo rsyslog2.pid exists, trying to shut down rsyslogd process `cat rsyslog2.pid`. + kill -9 `cat rsyslog2.pid` sleep 1 fi diff --git a/tests/longrun.sh b/tests/longrun.sh new file mode 100755 index 00000000..407e9fa9 --- /dev/null +++ b/tests/longrun.sh @@ -0,0 +1,30 @@ +# This is a test-aid script to try running some tests through many iterations. +# It is not yet used in the automated testbench, but I keep this file so that +# I can use it whenever there is need to. As such, it currently does not have +# parameters but is expected to be edited as needed. -- rgerhards, 2010-03-10 +# +# use: ./longrun.sh testname.sh +# +# where testname.sh is the test to be run +# to change other params, you need to edit the settings here below: +MAXRUNS=10 +DISPLAYALIVE=100 +LOGFILE=runlog + +echo "logfile is $LOGFILE" +echo "executing test $1" + +date > $LOGFILE + +for (( i=0; $i < 10000; i++ )) + do + if [ $(( i % DISPLAYALIVE )) -eq 0 ]; then + echo "$i iterations done" + fi + $1 >> $LOGFILE + if [ "$?" -ne "0" ]; then + echo "Test failed in iteration $i, review $LOGFILE for details!" + exit 1 + fi + done +echo "No failure in $i iterations." diff --git a/tests/manytcp.sh b/tests/manytcp.sh index c55eb9c0..94a5c035 100755 --- a/tests/manytcp.sh +++ b/tests/manytcp.sh @@ -2,7 +2,10 @@ source $srcdir/diag.sh init source $srcdir/diag.sh startup manytcp.conf # the config file specifies exactly 1100 connections -source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1000 40000 +source $srcdir/diag.sh tcpflood -c1000 -m40000 +# the sleep below is needed to prevent too-early termination of the tcp listener +sleep 1 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # we need to wait until rsyslogd is finished! source $srcdir/diag.sh seq-check 0 39999 source $srcdir/diag.sh exit diff --git a/tests/nettester.c b/tests/nettester.c index 566f553b..07a1c052 100644 --- a/tests/nettester.c +++ b/tests/nettester.c @@ -38,6 +38,7 @@ #include <sys/socket.h> #include <sys/wait.h> #include <sys/stat.h> +#include <sys/time.h> #include <arpa/inet.h> #include <assert.h> #include <unistd.h> @@ -46,6 +47,7 @@ #include <signal.h> #include <netinet/in.h> #include <getopt.h> +#include <ctype.h> #define EXIT_FAILURE 1 #define INVALID_SOCKET -1 @@ -60,7 +62,11 @@ static char *testSuite = NULL; /* name of current test suite */ static int iPort = 12514; /* port which shall be used for sending data */ static char* pszCustomConf = NULL; /* custom config file, use -c conf to specify */ static int verbose = 0; /* verbose output? -v option */ +static int useDebugEnv = 0; /* activate debugging environment (for rsyslog debug log)? */ +/* these two are quick hacks... */ +int iFailed = 0; +int iTests = 0; /* provide user-friednly name of input mode */ @@ -79,14 +85,27 @@ static char *inputMode2Str(inputMode_t mode) void readLine(int fd, char *ln) { + char *orig = ln; char c; int lenRead; + + if(verbose) + fprintf(stderr, "begin readLine\n"); lenRead = read(fd, &c, 1); while(lenRead == 1 && c != '\n') { + if(c == '\0') { + *ln = c; + fprintf(stderr, "Warning: there was a '\\0'-Byte in the read response " + "right after this string: '%s'\n", orig); + c = '?'; + } *ln++ = c; - lenRead = read(fd, &c, 1); + lenRead = read(fd, &c, 1); } *ln = '\0'; + + if(verbose) + fprintf(stderr, "end readLine, val read '%s'\n", orig); } @@ -104,6 +123,7 @@ tcpSend(char *buf, int lenBuf) { static int sock = INVALID_SOCKET; struct sockaddr_in addr; + int retries; if(sock == INVALID_SOCKET) { /* first time, need to connect to target */ @@ -119,10 +139,20 @@ tcpSend(char *buf, int lenBuf) fprintf(stderr, "inet_aton() failed\n"); return(1); } - if(connect(sock, (struct sockaddr*)&addr, sizeof(addr)) != 0) { - fprintf(stderr, "connect() failed\n"); - return(1); - } + retries = 0; + while(1) { /* loop broken inside */ + if(connect(sock, (struct sockaddr*)&addr, sizeof(addr)) == 0) { + break; + } else { + if(retries++ == 50) { + ++iFailed; + fprintf(stderr, "connect() failed\n"); + return(1); + } else { + usleep(100000); /* 0.1 sec, these are us! */ + } + } + } } /* send test data */ @@ -189,11 +219,8 @@ int openPipe(char *configFile, pid_t *pid, int *pfd) "-M../runtime/.libs:../.libs", NULL }; char confFile[1024]; char *newenviron[] = { NULL }; - /* debug aide... - char *newenviron[] = { "RSYSLOG_DEBUG=debug nostdout", - "RSYSLOG_DEBUGLOG=tmp", NULL }; - */ - + char *newenvironDeb[] = { "RSYSLOG_DEBUG=debug nostdout", + "RSYSLOG_DEBUGLOG=log", NULL }; sprintf(confFile, "-f%s/testsuites/%s.conf", srcdir, (pszCustomConf == NULL) ? configFile : pszCustomConf); @@ -216,7 +243,7 @@ int openPipe(char *configFile, pid_t *pid, int *pfd) close(pipefd[1]); close(pipefd[0]); fclose(stdin); - execve("../tools/rsyslogd", newargv, newenviron); + execve("../tools/rsyslogd", newargv, (useDebugEnv) ? newenvironDeb : newenviron); } else { close(pipefd[1]); *pid = cpid; @@ -227,6 +254,62 @@ int openPipe(char *configFile, pid_t *pid, int *pfd) } +/* This function unescapes a string of testdata. That it, escape sequences + * are converted into their one-character equivalent. While doing so, it applies + * C-like semantics. This was made necessary for easy integration of control + * characters inside test cases. -- rgerhards, 2009-03-11 + * Currently supported: + * \\ single backslash + * \n, \t, \r as in C + * \nnn where nnn is a 1 to 3 character octal sequence + * Note that when a problem occurs, the end result is undefined. After all, this + * is for a testsuite generatort, it must not be 100% bullet proof (so do not + * copy this code into something that must be!). Also note that we do in-memory + * unescaping and assume that the string gets shorter but NEVER longer! + */ +void unescapeTestdata(char *testdata) +{ + char *pDst; + char *pSrc; + int i; + int c; + + pDst = pSrc = testdata; + while(*pSrc) { + if(*pSrc == '\\') { + switch(*++pSrc) { + case '\\': *pDst++ = *pSrc++; + break; + case 'n': *pDst++ = '\n'; + ++pSrc; + break; + case 'r': *pDst++ = '\r'; + ++pSrc; + break; + case 't': *pDst++ = '\t'; + ++pSrc; + break; + case '0': + case '1': + case '2': + case '3': c = *pSrc++ - '0'; + i = 1; /* we already processed one digit! */ + while(i < 3 && isdigit(*pSrc)) { + c = c * 8 + *pSrc++ - '0'; + ++i; + } + *pDst++ = c; + break; + default: break; + } + } else { + *pDst++ = *pSrc++; + } + } + *pDst = '\0'; +} + + /* Process a specific test case. File name is provided. * Needs to return 0 if all is OK, something else otherwise. */ @@ -247,41 +330,54 @@ processTestFile(int fd, char *pszFileName) /* skip comments at start of file */ - getline(&testdata, &lenLn, fp); while(!feof(fp)) { - if(*testdata == '#') - getline(&testdata, &lenLn, fp); - else - break; /* first non-comment */ - } - + getline(&testdata, &lenLn, fp); + while(!feof(fp)) { + if(*testdata == '#') + getline(&testdata, &lenLn, fp); + else + break; /* first non-comment */ + } - testdata[strlen(testdata)-1] = '\0'; /* remove \n */ - /* now we have the test data to send (we could use function pointers here...) */ - if(inputMode == inputUDP) { - if(udpSend(testdata, strlen(testdata)) != 0) - return(2); - } else { - if(tcpSend(testdata, strlen(testdata)) != 0) - return(2); - } + /* this is not perfect, but works ;) */ + if(feof(fp)) + break; - /* next line is expected output - * we do not care about EOF here, this will lead to a failure and thus - * draw enough attention. -- rgerhards, 2009-03-31 - */ - getline(&expected, &lenLn, fp); - expected[strlen(expected)-1] = '\0'; /* remove \n */ + ++iTests; /* increment test count, we now do one! */ + + testdata[strlen(testdata)-1] = '\0'; /* remove \n */ + /* now we have the test data to send (we could use function pointers here...) */ + unescapeTestdata(testdata); + if(inputMode == inputUDP) { + if(udpSend(testdata, strlen(testdata)) != 0) + return(2); + } else { + if(tcpSend(testdata, strlen(testdata)) != 0) + return(2); + } - /* pull response from server and then check if it meets our expectation */ - readLine(fd, buf); - if(strcmp(expected, buf)) { - printf("\nExpected Response:\n'%s'\nActual Response:\n'%s'\n", - expected, buf); - ret = 1; + /* next line is expected output + * we do not care about EOF here, this will lead to a failure and thus + * draw enough attention. -- rgerhards, 2009-03-31 + */ + getline(&expected, &lenLn, fp); + expected[strlen(expected)-1] = '\0'; /* remove \n */ + + /* pull response from server and then check if it meets our expectation */ + readLine(fd, buf); + if(strcmp(expected, buf)) { + ++iFailed; + printf("\nFile %s:\nExpected Response:\n'%s'\nActual Response:\n'%s'\n", + pszFileName, expected, buf); + ret = 1; + } + /* we need to free buffers, as we have potentially modified them! */ + free(testdata); + testdata = NULL; + free(expected); + expected = NULL; } - free(testdata); free(expected); fclose(fp); return(ret); @@ -297,8 +393,6 @@ processTestFile(int fd, char *pszFileName) int doTests(int fd, char *files) { - int iFailed = 0; - int iTests = 0; int ret; char *testFile; glob_t testFiles; @@ -313,7 +407,6 @@ doTests(int fd, char *files) if(stat((char*) testFile, &fileInfo) != 0) continue; /* continue with the next file if we can't stat() the file */ - ++iTests; /* all regular files are run through the test logic. Symlinks don't work. */ if(S_ISREG(fileInfo.st_mode)) { /* config file */ if(verbose) printf("processing test case '%s' ... ", testFile); @@ -321,8 +414,9 @@ doTests(int fd, char *files) if(ret == 0) { if(verbose) printf("successfully completed\n"); } else { - if(verbose) printf("failed!\n"); - ++iFailed; + if(!verbose) + printf("test '%s' ", testFile); + printf("failed!\n"); } } } @@ -365,11 +459,14 @@ int main(int argc, char *argv[]) char buf[4096]; char testcases[4096]; - while((opt = getopt(argc, argv, "c:i:p:t:v")) != EOF) { + while((opt = getopt(argc, argv, "dc:i:p:t:v")) != EOF) { switch((char)opt) { case 'c': pszCustomConf = optarg; break; + case 'd': + useDebugEnv = 1; + break; case 'i': if(!strcmp(optarg, "udp")) inputMode = inputUDP; @@ -390,7 +487,7 @@ int main(int argc, char *argv[]) verbose = 1; break; default:printf("Invalid call of nettester, invalid option '%c'.\n", opt); - printf("Usage: nettester -ttestsuite-name -iudp|tcp [-pport] [-ccustomConfFile] \n"); + printf("Usage: nettester -d -ttestsuite-name -iudp|tcp [-pport] [-ccustomConfFile] \n"); exit(1); } } diff --git a/tests/ourtail.c b/tests/ourtail.c index 6781b5fe..c31babb9 100644 --- a/tests/ourtail.c +++ b/tests/ourtail.c @@ -26,6 +26,7 @@ * * A copy of the GPL can be found in the file "COPYING" in this distribution. */ +#include "config.h" #include <stdio.h> int main(int __attribute__((unused)) argc, char __attribute__((unused)) *argv[]) @@ -40,4 +41,6 @@ int main(int __attribute__((unused)) argc, char __attribute__((unused)) *argv[]) for( ; c != EOF ; c = getchar()) putchar(c); + + return 0; } diff --git a/tests/parsertest.sh b/tests/parsertest.sh index afdb9469..fc68ab84 100755 --- a/tests/parsertest.sh +++ b/tests/parsertest.sh @@ -1,13 +1,33 @@ -echo test parsertest via udp -$srcdir/killrsyslog.sh # kill rsyslogd if it runs for some reason +echo TEST: \[parsertest.sh\]: various parser tests +source $srcdir/diag.sh init +source $srcdir/diag.sh nettester parse1 udp +source $srcdir/diag.sh nettester parse1 tcp +source $srcdir/diag.sh nettester parse3 udp +source $srcdir/diag.sh nettester parse3 tcp +source $srcdir/diag.sh nettester parse_invld_regex udp +source $srcdir/diag.sh nettester parse_invld_regex tcp +source $srcdir/diag.sh nettester parse-3164-buggyday udp +source $srcdir/diag.sh nettester parse-3164-buggyday tcp +source $srcdir/diag.sh nettester parse-nodate udp +source $srcdir/diag.sh nettester parse-nodate tcp +# the following samples can only be run over UDP as they are so +# malformed they break traditional syslog/tcp framing... +source $srcdir/diag.sh nettester snare_ccoff_udp udp +source $srcdir/diag.sh nettester snare_ccoff_udp2 udp -./nettester -tparse1 -iudp -if [ "$?" -ne "0" ]; then - exit 1 -fi +echo \[parsertest.sh]: redoing tests in IPv4-only mode +source $srcdir/diag.sh nettester parse1 udp -4 +source $srcdir/diag.sh nettester parse1 tcp -4 +source $srcdir/diag.sh nettester parse3 udp -4 +source $srcdir/diag.sh nettester parse3 tcp -4 +source $srcdir/diag.sh nettester parse_invld_regex udp -4 +source $srcdir/diag.sh nettester parse_invld_regex tcp -4 +source $srcdir/diag.sh nettester parse-3164-buggyday udp -4 +source $srcdir/diag.sh nettester parse-3164-buggyday tcp -4 +source $srcdir/diag.sh nettester parse-nodate udp -4 +source $srcdir/diag.sh nettester parse-nodate tcp -4 +# UDP-only tests +source $srcdir/diag.sh nettester snare_ccoff_udp udp -4 +source $srcdir/diag.sh nettester snare_ccoff_udp2 udp -4 -echo test parsertest via tcp -./nettester -tparse1 -itcp -if [ "$?" -ne "0" ]; then - exit 1 -fi +source $srcdir/diag.sh exit diff --git a/tests/pipeaction.sh b/tests/pipeaction.sh new file mode 100755 index 00000000..c2201011 --- /dev/null +++ b/tests/pipeaction.sh @@ -0,0 +1,33 @@ +# Test for the pipe output action. +# will create a fifo in the current directory, write to it and +# then do the usual sequence checks. +# added 2009-11-05 by RGerhards +echo =============================================================================== +echo \[pipeaction.sh\]: testing pipe output action + +# create the pipe and start a background process that copies data from +# it to the "regular" work file +source $srcdir/diag.sh init +rm -f rsyslog-testbench-fifo +mkfifo rsyslog-testbench-fifo +cp rsyslog-testbench-fifo rsyslog.out.log & +CPPROCESS=$! +echo background cp process id is $CPPROCESS + +# now do the usual run +source $srcdir/diag.sh startup pipeaction.conf +# 20000 messages should be enough +#source $srcdir/diag.sh tcpflood -m20000 +source $srcdir/diag.sh injectmsg 0 20000 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown + +# wait for the cp process to finish, do pipe-specific cleanup +echo waiting for background cp to terminate... +wait $CPPROCESS +rm -f rsyslog-testbench-fifo +echo background cp has terminated, continue test... + +# and continue the usual checks +source $srcdir/diag.sh seq-check 0 19999 +source $srcdir/diag.sh exit diff --git a/tests/proprepltest.sh b/tests/proprepltest.sh new file mode 100755 index 00000000..3c252e52 --- /dev/null +++ b/tests/proprepltest.sh @@ -0,0 +1,7 @@ +echo TEST: proprepltest.sh - various tests for the property replacer +source $srcdir/diag.sh init +source $srcdir/diag.sh nettester rfctag udp +source $srcdir/diag.sh nettester rfctag tcp +source $srcdir/diag.sh nettester nolimittag udp +source $srcdir/diag.sh nettester nolimittag tcp +source $srcdir/diag.sh init diff --git a/tests/random.sh b/tests/random.sh new file mode 100755 index 00000000..d1f392d3 --- /dev/null +++ b/tests/random.sh @@ -0,0 +1,20 @@ +# Test if rsyslog survives sending truely random data to it... +# +# added 2010-04-01 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[random.sh\]: testing random data +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup random.conf +# generate random data +./randomgen -f rsyslog.random.data -s 100000 +ls -l rsyslog.random.data +source $srcdir/diag.sh tcpflood -B -I rsyslog.random.data -c5 -C10 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +# we do not check anything yet, the point is if rsyslog survived ;) +# TODO: check for exit message, but we'll notice an abort anyhow, so not that important +source $srcdir/diag.sh exit diff --git a/tests/randomgen.c b/tests/randomgen.c new file mode 100644 index 00000000..9ba56954 --- /dev/null +++ b/tests/randomgen.c @@ -0,0 +1,130 @@ +/* generates random data for later use in test cases. Of course, + * we could generate random data during the testcase itself, but + * the core idea is that we record the random data so that we have + * a chance to reproduce a problem should it occur. IMHO this + * provides the best compromise, by a) having randomness but + * b) knowing what was used during the test. + * + * Params + * -f output file name (stdout if not given) + * -s size of test data, plain number is size in k, 1MB default + * -u uses /dev/urandom instead of libc random number generator + * (when available). Note that this is usually much slower. + * + * Part of the testbench for rsyslog. + * + * Copyright 2010 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +#include "config.h" +#include <stdio.h> +#include <stdlib.h> +#include <time.h> +#include <signal.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <arpa/inet.h> +#include <assert.h> +#include <unistd.h> +#include <string.h> +#include <netinet/in.h> + +#define EXIT_FAILURE 1 + +static char *fileName = NULL; /* name of output file */ +static int tryUseURandom = 0; /* try to use /dev/urandom? */ +static long long fileSize = 1024*1024; /* file size in K, 1MB default */ + + +/* generate the random file. This code really can be improved (e.g. read /dev/urandom + * when available) + */ +static inline void +genFile() +{ + long i; + FILE *fp; + FILE *rfp = NULL; + + if(fileName == NULL) { + fp = stdout; + } else { + if((fp = fopen(fileName, "w")) == NULL) { + perror(fileName); + } + } + + /* try to use /dev/urandom, if available */ + if(tryUseURandom) + rfp = fopen("/dev/urandom", "r"); + + if(rfp == NULL) { + /* fallback, use libc random number generator */ + for(i = 0 ; i < fileSize ; ++i) { + if(fputc((char) rand(), fp) == EOF) { + perror(fileName); + exit(1); + } + } + } else { + /* use /dev/urandom */ + printf("using /dev/urandom"); + for(i = 0 ; i < fileSize ; ++i) { + if(fputc(fgetc(rfp), fp) == EOF) { + perror(fileName); + exit(1); + } + } + } + + if(fileName != NULL) + fclose(fp); +} + + +/* Run the test. + * rgerhards, 2009-04-03 + */ +int main(int argc, char *argv[]) +{ + int ret = 0; + int opt; + + srand(time(NULL)); /* seed is good enough for our needs */ + + while((opt = getopt(argc, argv, "f:s:u")) != -1) { + switch (opt) { + case 'f': fileName = optarg; + break; + case 's': fileSize = atol(optarg) * 1024; + break; + case 'u': tryUseURandom = 1; + break; + default: printf("invalid option '%c' or value missing - terminating...\n", opt); + exit (1); + break; + } + } + + printf("generating random data file '%s' of %ldkb - may take a short while...\n", + fileName, (long) (fileSize / 1024)); + genFile(); + + exit(ret); +} diff --git a/tests/rt-init.c b/tests/rt-init.c index aaac7ed1..66a9ad32 100644 --- a/tests/rt-init.c +++ b/tests/rt-init.c @@ -21,10 +21,10 @@ * * A copy of the GPL can be found in the file "COPYING" in this distribution. */ -#include <stdio.h> - +#include "config.h" #include "rsyslog.h" #include "testbench.h" +#include <stdio.h> /* must be last, else we get a zlib compile error on some platforms */ MODULE_TYPE_TESTBENCH diff --git a/tests/runtime-dummy.c b/tests/runtime-dummy.c index 9cddd913..38e6bba1 100644 --- a/tests/runtime-dummy.c +++ b/tests/runtime-dummy.c @@ -25,7 +25,9 @@ * * A copy of the GPL can be found in the file "COPYING" in this distribution. */ +#include "config.h" #include <stdlib.h> +#include "rsyslog.h" int bReduceRepeatMsgs = 0; int repeatinterval = 30; @@ -37,5 +39,7 @@ void cflineClassic(void) {}; void selectorAddList(void) {}; void selectorConstruct(void) {}; void selectorDestruct(void) {}; +void getFIOPName(void) {}; +ruleset_t *pCurrRuleset; /* these are required by some dynamically loaded modules */ diff --git a/tests/sndrcv.sh b/tests/sndrcv.sh new file mode 100755 index 00000000..2fc3bd82 --- /dev/null +++ b/tests/sndrcv.sh @@ -0,0 +1,9 @@ +# This tests two rsyslog instances. Instance +# TWO sends data to instance ONE. A number of messages is injected into +# the instance 2 and we finally check if all those messages +# arrived at instance 1. +# added 2009-11-11 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo \[sndrcv.sh\]: testing sending and receiving via tcp +source $srcdir/sndrcv_drvr.sh sndrcv 50000 diff --git a/tests/sndrcv_drvr.sh b/tests/sndrcv_drvr.sh new file mode 100755 index 00000000..3d613069 --- /dev/null +++ b/tests/sndrcv_drvr.sh @@ -0,0 +1,50 @@ +# This is test driver for testing two rsyslog instances. It can be +# utilized by any test that just needs two instances with different +# config files, where messages are injected in instance TWO and +# (with whatever rsyslog mechanism) being relayed over to instance ONE, +# where they are written to the log file. After the run, the completeness +# of that log file is checked. +# The code is almost the same, but the config files differ (probably greatly) +# for different test cases. As such, this driver needs to be called with the +# config file name ($2). From that name, the sender and receiver config file +# names are automatically generated. +# So: $1 config file name, $2 number of messages +# +# A note on TLS testing: the current testsuite (in git!) already contains +# TLS test cases. However, getting these test cases correct is not simple. +# That's not a problem with the code itself, but rater a problem with +# synchronization in the test environment. So I have deciced to keep the +# TLS tests in, but not yet actually utilize them. This is most probably +# left as an excercise for future (devel) releases. -- rgerhards, 2009-11-11 +# +# added 2009-11-11 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +# uncomment for debugging support: +source $srcdir/diag.sh init +# start up the instances +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup $1_rcvr.conf +source $srcdir/diag.sh wait-startup +#export RSYSLOG_DEBUGLOG="log2" +#valgrind="valgrind" +source $srcdir/diag.sh startup $1_sender.conf 2 +source $srcdir/diag.sh wait-startup 2 +# may be needed by TLS (once we do it): sleep 30 + +# now inject the messages into instance 2. It will connect to instance 1, +# and that instance will record the data. +source $srcdir/diag.sh tcpflood -m$2 -i1 +sleep 2 # make sure all data is received in input buffers +# shut down sender when everything is sent, receiver continues to run concurrently +# may be needed by TLS (once we do it): sleep 60 +source $srcdir/diag.sh shutdown-when-empty 2 +source $srcdir/diag.sh wait-shutdown 2 +# now it is time to stop the receiver as well +source $srcdir/diag.sh shutdown-when-empty +source $srcdir/diag.sh wait-shutdown + +# may be needed by TLS (once we do it): sleep 60 +# do the final check +source $srcdir/diag.sh seq-check 1 $2 +source $srcdir/diag.sh exit diff --git a/tests/sndrcv_gzip.sh b/tests/sndrcv_gzip.sh new file mode 100755 index 00000000..4931f3d0 --- /dev/null +++ b/tests/sndrcv_gzip.sh @@ -0,0 +1,7 @@ +# This test is similar to tcpsndrcv, but it forwards messages in +# zlib-compressed format (our own syslog extension). +# rgerhards, 2009-11-11 +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo \[sndrcv_gzip.sh\]: testing sending and receiving via tcp in zlib mode +source $srcdir/sndrcv_drvr.sh sndrcv_gzip 50000 diff --git a/tests/tcpflood.c b/tests/tcpflood.c index 2ca796ca..d8c3f038 100644 --- a/tests/tcpflood.c +++ b/tests/tcpflood.c @@ -2,15 +2,36 @@ * messages over them. This is used for stress-testing. * * Params - * argv[1] target address - * argv[2] target port - * argv[3] number of connections - * argv[4] number of messages to send (connection is random) - * argv[5] initial message number (optional) + * -t target address (default 127.0.0.1) + * -p target port (default 13514) + * -n number of target ports (targets are in range -p..(-p+-n-1) + * -c number of connections (default 1) + * -m number of messages to send (connection is random) + * -i initial message number (optional) + * -P PRI to be used for generated messages (default is 167). + * Specify the plain number without leading zeros + * -d amount of extra data to add to message. If present, the + * number itself will be added as third field, and the data + * bytes as forth. Add -r to randomize the amount of extra + * data included in the range 1..(value of -d). + * -r randomize amount of extra data added (-d must be > 0) + * -f support for testing dynafiles. If given, include a dynafile ID + * in the range 0..(f-1) as the SECOND field, shifting all field values + * one field to the right. Zero (default) disables this functionality. + * -M the message to be sent. Disables all message format options, as + * only that exact same message is sent. + * -I read specified input file, do NOT generate own test data. The test + * completes when eof is reached. + * -B The specified file (-I) is binary. No data processing is done by + * tcpflood. If multiple connections are specified, data is read in + * chunks and spread across the connections without taking any record + * delemiters into account. + * -C when input from a file is read, this file is transmitted -C times + * (C like cycle, running out of meaningful option switches ;)) * * Part of the testbench for rsyslog. * - * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * Copyright 2009, 2010 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -47,12 +68,25 @@ /* Name of input file, must match $IncludeConfig in test suite .conf files */ #define NETTEST_INPUT_CONF_FILE "nettest.input.conf" /* name of input file, must match $IncludeConfig in .conf files */ -static char *targetIP; -static int targetPort; +#define MAX_EXTRADATA_LEN 100*1024 + +static char *targetIP = "127.0.0.1"; +static char *msgPRI = "167"; +static int targetPort = 13514; +static int numTargetPorts = 1; +static int dynFileIDs = 0; +static int extraDataLen = 0; /* amount of extra data to add to message */ +static int bRandomizeExtraData = 0; /* randomize amount of extra data added */ static int numMsgsToSend; /* number of messages to send */ -static int numConnections; /* number of connections to create */ +static int numConnections = 1; /* number of connections to create */ static int *sockArray; /* array of sockets to use */ static int msgNum = 0; /* initial message number to start with */ +static int bShowProgress = 1; /* show progress messages */ +static char *MsgToSend = NULL; /* if non-null, this is the actual message to send */ +static int bBinaryFile = 0; /* is -I file binary */ +static char *dataFile = NULL; /* name of data file, if NULL, generate own data */ +static int numFileIterations = 1;/* how often is file data to be sent? */ +FILE *dataFP = NULL; /* file pointer for data file, if used */ /* open a single tcp connection @@ -61,24 +95,42 @@ int openConn(int *fd) { int sock; struct sockaddr_in addr; + int port; + int retries = 0; + int rnd; if((sock=socket(AF_INET, SOCK_STREAM, 0))==-1) { perror("socket()"); return(1); } + /* randomize port if required */ + if(numTargetPorts > 1) { + rnd = rand(); /* easier if we need value for debug messages ;) */ + port = targetPort + (rnd % numTargetPorts); + } else { + port = targetPort; + } memset((char *) &addr, 0, sizeof(addr)); addr.sin_family = AF_INET; - addr.sin_port = htons(targetPort); + addr.sin_port = htons(port); if(inet_aton(targetIP, &addr.sin_addr)==0) { fprintf(stderr, "inet_aton() failed\n"); return(1); } - if(connect(sock, (struct sockaddr*)&addr, sizeof(addr)) != 0) { - perror("connect()"); - fprintf(stderr, "connect() failed\n"); - return(1); - } + while(1) { /* loop broken inside */ + if(connect(sock, (struct sockaddr*)&addr, sizeof(addr)) == 0) { + break; + } else { + if(retries++ == 50) { + perror("connect()"); + fprintf(stderr, "connect() failed\n"); + return(1); + } else { + usleep(100000); /* ms = 1000 us! */ + } + } + } *fd = sock; return 0; @@ -94,11 +146,13 @@ int openConnections(void) char msgBuf[128]; size_t lenMsg; - write(1, " open connections", sizeof(" open connections")-1); + if(bShowProgress) + write(1, " open connections", sizeof(" open connections")-1); sockArray = calloc(numConnections, sizeof(int)); for(i = 0 ; i < numConnections ; ++i) { if(i % 10 == 0) { - printf("\r%5.5d", i); + if(bShowProgress) + printf("\r%5.5d", i); //lenMsg = sprintf(msgBuf, "\r%5.5d", i); //write(1, msgBuf, lenMsg); } @@ -124,14 +178,17 @@ int openConnections(void) void closeConnections(void) { int i; - char msgBuf[128]; size_t lenMsg; + char msgBuf[128]; - write(1, " close connections", sizeof(" close connections")-1); + if(bShowProgress) + write(1, " close connections", sizeof(" close connections")-1); for(i = 0 ; i < numConnections ; ++i) { if(i % 10 == 0) { - lenMsg = sprintf(msgBuf, "\r%5.5d", i); - write(1, msgBuf, lenMsg); + if(bShowProgress) { + lenMsg = sprintf(msgBuf, "\r%5.5d", i); + write(1, msgBuf, lenMsg); + } } close(sockArray[i]); } @@ -141,6 +198,62 @@ void closeConnections(void) } +/* generate the message to be sent according to program command line parameters. + * this has been moved to its own function as we now have various different ways + * of constructing test messages. -- rgerhards, 2010-03-31 + */ +static inline void +genMsg(char *buf, size_t maxBuf, int *pLenBuf) +{ + int edLen; /* actual extra data length to use */ + char extraData[MAX_EXTRADATA_LEN + 1]; + char dynFileIDBuf[128] = ""; + static int numMsgsGen = 0; + int done; + + if(dataFP != NULL) { + /* get message from file */ + do { + done = 1; + *pLenBuf = fread(buf, 1, 1024, dataFP); + if(feof(dataFP)) { + if(--numFileIterations > 0) { + rewind(dataFP); + done = 0; /* need new iteration */ + } else { + *pLenBuf = 0; + goto finalize_it; + } + } + } while(!done); /* Attention: do..while()! */ + } else if(MsgToSend == NULL) { + if(dynFileIDs > 0) { + snprintf(dynFileIDBuf, maxBuf, "%d:", rand() % dynFileIDs); + } + if(extraDataLen == 0) { + *pLenBuf = snprintf(buf, maxBuf, "<%s>Mar 1 01:00:00 172.20.245.8 tag msgnum:%s%8.8d:\n", + msgPRI, dynFileIDBuf, msgNum); + } else { + if(bRandomizeExtraData) + edLen = ((long) rand() + extraDataLen) % extraDataLen + 1; + else + edLen = extraDataLen; + memset(extraData, 'X', edLen); + extraData[edLen] = '\0'; + *pLenBuf = snprintf(buf, maxBuf, "<%s>Mar 1 01:00:00 172.20.245.8 tag msgnum:%s%8.8d:%d:%s\n", + msgPRI, dynFileIDBuf, msgNum, edLen, extraData); + } + } else { + /* use fixed message format from command line */ + *pLenBuf = snprintf(buf, maxBuf, "%s\n", MsgToSend); + } + + if(numMsgsGen++ >= numMsgsToSend) + *pLenBuf = 0; /* indicate end of run */ + +finalize_it: ; +} + /* send messages to the tcp connections we keep open. We use * a very basic format that helps identify the message * (via msgnum:<number>: e.g. msgnum:00000001:). This format is suitable @@ -151,39 +264,53 @@ void closeConnections(void) */ int sendMessages(void) { - int i; + int i = 0; int socknum; int lenBuf; int lenSend; - char buf[2048]; - - srand(time(NULL)); /* seed is good enough for our needs */ - - printf("Sending %d messages.\n", numMsgsToSend); - printf("\r%5.5d messages sent", 0); - for(i = 0 ; i < numMsgsToSend ; ++i) { + char *statusText; + char buf[MAX_EXTRADATA_LEN + 1024]; + + if(dataFile == NULL) { + printf("Sending %d messages.\n", numMsgsToSend); + statusText = "messages"; + } else { + printf("Sending file '%s' %d times.\n", dataFile, numFileIterations); + statusText = "kb"; + } + if(bShowProgress) + printf("\r%8.8d %s sent", 0, statusText); + while(1) { /* broken inside loop! */ if(i < numConnections) socknum = i; else if(i >= numMsgsToSend - numConnections) socknum = i - (numMsgsToSend - numConnections); - else - socknum = rand() % numConnections; - lenBuf = sprintf(buf, "<167>Mar 1 01:00:00 172.20.245.8 tag msgnum:%8.8d:\n", msgNum); + else { + int rnd = rand(); + //socknum = rand() % numConnections; + socknum = rnd % numConnections; + } + genMsg(buf, sizeof(buf), &lenBuf); /* generate the message to send according to params */ + if(lenBuf == 0) + break; /* end of processing! */ lenSend = send(sockArray[socknum], buf, lenBuf, 0); if(lenSend != lenBuf) { printf("\r%5.5d\n", i); fflush(stdout); perror("send test data"); - printf("send() failed at socket %d, index %d, msgNum %d\n", socknum, i, msgNum); + printf("send() failed at socket %d, index %d, msgNum %d\n", + sockArray[socknum], i, msgNum); fflush(stderr); return(1); } if(i % 100 == 0) { - printf("\r%5.5d", i); + if(bShowProgress) + printf("\r%8.8d", i); } ++msgNum; + ++i; } - printf("\r%5.5d messages sent\n", i); + printf("\r%8.8d %s sent\n", i, statusText); return 0; } @@ -242,17 +369,18 @@ tcpSend(char *buf, int lenBuf) } -/* Run the test suite. This must be called with exactly one parameter, the - * name of the test suite. For details, see file header comment at the top - * of this file. +/* Run the test. * rgerhards, 2009-04-03 */ int main(int argc, char *argv[]) { int ret = 0; + int opt; struct sigaction sigAct; static char buf[1024]; + srand(time(NULL)); /* seed is good enough for our needs */ + /* on Solaris, we do not HAVE MSG_NOSIGNAL, so for this reason * we block SIGPIPE (not an issue for this program) */ @@ -262,19 +390,61 @@ int main(int argc, char *argv[]) sigaction(SIGPIPE, &sigAct, NULL); setvbuf(stdout, buf, _IONBF, 48); + + if(!isatty(1)) + bShowProgress = 0; + + while((opt = getopt(argc, argv, "f:t:p:c:C:m:i:I:P:d:n:M:rB")) != -1) { + switch (opt) { + case 't': targetIP = optarg; + break; + case 'p': targetPort = atoi(optarg); + break; + case 'n': numTargetPorts = atoi(optarg); + break; + case 'c': numConnections = atoi(optarg); + break; + case 'C': numFileIterations = atoi(optarg); + break; + case 'm': numMsgsToSend = atoi(optarg); + break; + case 'i': msgNum = atoi(optarg); + break; + case 'P': msgPRI = optarg; + break; + case 'd': extraDataLen = atoi(optarg); + if(extraDataLen > MAX_EXTRADATA_LEN) { + fprintf(stderr, "-d max is %d!\n", + MAX_EXTRADATA_LEN); + exit(1); + } + break; + case 'r': bRandomizeExtraData = 1; + break; + case 'f': dynFileIDs = atoi(optarg); + break; + case 'M': MsgToSend = optarg; + break; + case 'I': dataFile = optarg; + /* in this mode, we do not know the num messages to send, so + * we set a (high) number to keep the code happy. + */ + numMsgsToSend = 1000000; + break; + case 'B': bBinaryFile = 1; + break; + default: printf("invalid option '%c' or value missing - terminating...\n", opt); + exit (1); + break; + } + } - if(argc != 5 && argc != 6) { - printf("Invalid call of tcpflood\n"); - printf("Usage: tcpflood target-host target-port num-connections num-messages [initial msgnum]\n"); - exit(1); + if(dataFile != NULL) { + if((dataFP = fopen(dataFile, "r")) == NULL) { + perror(dataFile); + exit(1); + } } - - targetIP = argv[1]; - targetPort = atoi(argv[2]); - numConnections = atoi(argv[3]); - numMsgsToSend = atoi(argv[4]); - if(argc == 6) - msgNum = atoi(argv[5]); if(openConnections() != 0) { printf("error opening connections\n"); @@ -286,7 +456,6 @@ int main(int argc, char *argv[]) exit(1); } - //closeConnections(); printf("End of tcpflood Run\n"); exit(ret); diff --git a/tests/testsuites/1.retry.conf b/tests/testsuites/1.retry.conf new file mode 100644 index 00000000..c464b19c --- /dev/null +++ b/tests/testsuites/1.retry.conf @@ -0,0 +1,2 @@ +<167>Mar 6 16:57:54 172.20.245.8 %PIX-7-710005: UDP request discarded from SERVER1/2741 to test_app:255.255.255.255/61601 +167,Mar 6 16:57:54,172.20.245.8,%PIX-7-710005,%PIX-7-710005:, diff --git a/tests/testsuites/Apr.ts3164 b/tests/testsuites/Apr.ts3164 new file mode 100644 index 00000000..3134f224 --- /dev/null +++ b/tests/testsuites/Apr.ts3164 @@ -0,0 +1,3 @@ +<167>Apr 6 16:57:54 172.20.245.8 TAG: MSG +Apr 6 16:57:54 +#Only the first two lines are important, you may place anything behind them! diff --git a/tests/testsuites/Aug.ts3164 b/tests/testsuites/Aug.ts3164 new file mode 100644 index 00000000..d9a721eb --- /dev/null +++ b/tests/testsuites/Aug.ts3164 @@ -0,0 +1,3 @@ +<167>Aug 6 16:57:54 172.20.245.8 TAG: MSG +Aug 6 16:57:54 +#Only the first two lines are important, you may place anything behind them! diff --git a/tests/testsuites/Dec.ts3164 b/tests/testsuites/Dec.ts3164 new file mode 100644 index 00000000..080ba401 --- /dev/null +++ b/tests/testsuites/Dec.ts3164 @@ -0,0 +1,3 @@ +<167>Dec 6 16:57:54 172.20.245.8 TAG: MSG +Dec 6 16:57:54 +#Only the first two lines are important, you may place anything behind them! diff --git a/tests/testsuites/Feb.ts3164 b/tests/testsuites/Feb.ts3164 new file mode 100644 index 00000000..d1eaaa33 --- /dev/null +++ b/tests/testsuites/Feb.ts3164 @@ -0,0 +1,3 @@ +<167>Feb 6 16:57:54 172.20.245.8 TAG: MSG +Feb 6 16:57:54 +#Only the first two lines are important, you may place anything behind them! diff --git a/tests/testsuites/Jan.ts3164 b/tests/testsuites/Jan.ts3164 new file mode 100644 index 00000000..0cb1c8e2 --- /dev/null +++ b/tests/testsuites/Jan.ts3164 @@ -0,0 +1,3 @@ +<167>Jan 6 16:57:54 172.20.245.8 TAG: MSG +Jan 6 16:57:54 +#Only the first two lines are important, you may place anything behind them! diff --git a/tests/testsuites/Jul.ts3164 b/tests/testsuites/Jul.ts3164 new file mode 100644 index 00000000..562e1ec4 --- /dev/null +++ b/tests/testsuites/Jul.ts3164 @@ -0,0 +1,3 @@ +<167>Jul 6 16:57:54 172.20.245.8 TAG: MSG +Jul 6 16:57:54 +#Only the first two lines are important, you may place anything behind them! diff --git a/tests/testsuites/Jun.ts3164 b/tests/testsuites/Jun.ts3164 new file mode 100644 index 00000000..ede27e0e --- /dev/null +++ b/tests/testsuites/Jun.ts3164 @@ -0,0 +1,3 @@ +<167>Jun 6 16:57:54 172.20.245.8 TAG: MSG +Jun 6 16:57:54 +#Only the first two lines are important, you may place anything behind them! diff --git a/tests/testsuites/Mar.ts3164 b/tests/testsuites/Mar.ts3164 new file mode 100644 index 00000000..55dd5bc2 --- /dev/null +++ b/tests/testsuites/Mar.ts3164 @@ -0,0 +1,3 @@ +<167>Mar 6 16:57:54 172.20.245.8 TAG: MSG +Mar 6 16:57:54 +#Only the first two lines are important, you may place anything behind them! diff --git a/tests/testsuites/May.ts3164 b/tests/testsuites/May.ts3164 new file mode 100644 index 00000000..72a5a301 --- /dev/null +++ b/tests/testsuites/May.ts3164 @@ -0,0 +1,3 @@ +<167>May 6 16:57:54 172.20.245.8 TAG: MSG +May 6 16:57:54 +#Only the first two lines are important, you may place anything behind them! diff --git a/tests/testsuites/Nov.ts3164 b/tests/testsuites/Nov.ts3164 new file mode 100644 index 00000000..e8f00e01 --- /dev/null +++ b/tests/testsuites/Nov.ts3164 @@ -0,0 +1,3 @@ +<167>Nov 6 16:57:54 172.20.245.8 TAG: MSG +Nov 6 16:57:54 +#Only the first two lines are important, you may place anything behind them! diff --git a/tests/testsuites/Oct.ts3164 b/tests/testsuites/Oct.ts3164 new file mode 100644 index 00000000..01423fef --- /dev/null +++ b/tests/testsuites/Oct.ts3164 @@ -0,0 +1,3 @@ +<167>Oct 6 16:57:54 172.20.245.8 TAG: MSG +Oct 6 16:57:54 +#Only the first two lines are important, you may place anything behind them! diff --git a/tests/testsuites/Sep.ts3164 b/tests/testsuites/Sep.ts3164 new file mode 100644 index 00000000..6c9e48e0 --- /dev/null +++ b/tests/testsuites/Sep.ts3164 @@ -0,0 +1,3 @@ +<167>Sep 6 16:57:54 172.20.245.8 TAG: MSG +Sep 6 16:57:54 +#Only the first two lines are important, you may place anything behind them! diff --git a/tests/testsuites/asynwr_deadlock.conf b/tests/testsuites/asynwr_deadlock.conf new file mode 100644 index 00000000..dc4045b0 --- /dev/null +++ b/tests/testsuites/asynwr_deadlock.conf @@ -0,0 +1,14 @@ +# rgerhards, 2010-03-09 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%\n" + +$OMFileFlushOnTXEnd on +$OMFileFlushInterval 10 +$OMFileFlushIOBufferSize 10k +$OMFileAsyncWriting on +:msg, contains, "msgnum:" ./rsyslog.out.log;outfmt diff --git a/tests/testsuites/asynwr_deadlock2.conf b/tests/testsuites/asynwr_deadlock2.conf new file mode 100644 index 00000000..07811613 --- /dev/null +++ b/tests/testsuites/asynwr_deadlock2.conf @@ -0,0 +1,16 @@ +# rgerhards, 2010-03-17 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:3%,%msg:F,58:4%,%msg:F,58:5%\n" +$template dynfile,"rsyslog.out.%msg:F,58:2%.log" # use multiple dynafiles + +$OMFileFlushOnTXEnd on +$OMFileFlushInterval 10 +$OMFileIOBufferSize 10k +$OMFileAsyncWriting on +$DynaFileCacheSize 4 +local0.* ?dynfile;outfmt diff --git a/tests/testsuites/asynwr_deadlock4.conf b/tests/testsuites/asynwr_deadlock4.conf new file mode 100644 index 00000000..f4308ff1 --- /dev/null +++ b/tests/testsuites/asynwr_deadlock4.conf @@ -0,0 +1,16 @@ +# rgerhards, 2010-03-17 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:3%,%msg:F,58:4%,%msg:F,58:5%\n" +$template dynfile,"rsyslog.out.log" # use multiple dynafiles + +$OMFileFlushOnTXEnd on +$OMFileFlushInterval 10 +$OMFileIOBufferSize 10k +$OMFileAsyncWriting on +$DynaFileCacheSize 4 +local0.* ?dynfile;outfmt diff --git a/tests/testsuites/asynwr_simple.conf b/tests/testsuites/asynwr_simple.conf new file mode 100644 index 00000000..44b03f2b --- /dev/null +++ b/tests/testsuites/asynwr_simple.conf @@ -0,0 +1,15 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +$OMFileFlushOnTXEnd off +$OMFileFlushInterval 2 +$OMFileFlushIOBufferSize 10k +$OMFileAsyncWriting on +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/asynwr_small.conf b/tests/testsuites/asynwr_small.conf new file mode 100644 index 00000000..f04ce962 --- /dev/null +++ b/tests/testsuites/asynwr_small.conf @@ -0,0 +1,14 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +$OMFileFlushOnTXEnd off +$OMFileFlushInterval 2 +$OMFileAsyncWriting on +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/asynwr_timeout.conf b/tests/testsuites/asynwr_timeout.conf new file mode 100644 index 00000000..44b03f2b --- /dev/null +++ b/tests/testsuites/asynwr_timeout.conf @@ -0,0 +1,15 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +$OMFileFlushOnTXEnd off +$OMFileFlushInterval 2 +$OMFileFlushIOBufferSize 10k +$OMFileAsyncWriting on +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/asynwr_tinybuf.conf b/tests/testsuites/asynwr_tinybuf.conf new file mode 100644 index 00000000..01dec4d8 --- /dev/null +++ b/tests/testsuites/asynwr_tinybuf.conf @@ -0,0 +1,15 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +$OMFileFlushOnTXEnd off +$OMFileFlushInterval 2 +$OMFileIOBufferSize 1 +$OMFileAsyncWriting on +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/complex1.conf b/tests/testsuites/complex1.conf new file mode 100644 index 00000000..9e2441d4 --- /dev/null +++ b/tests/testsuites/complex1.conf @@ -0,0 +1,81 @@ +# complex test case with multiple actions in gzip mode +# rgerhards, 2009-05-22 +$MaxMessageSize 10k +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 + +$template outfmt,"%msg:F,58:3%,%msg:F,58:4%,%msg:F,58:5%\n" +$template dynfile,"rsyslog.out.%inputname%.%msg:F,58:2%.log" + +## RULESET with listener +$Ruleset R13514 +# queue params: +$ActionQueueTimeoutShutdown 60000 +$ActionQueueSize 5000 +$ActionQueueSaveOnShutdown on +$ActionQueueHighWaterMark 4900 +$ActionQueueLowWaterMark 3500 +$ActionQueueType FixedArray +$ActionQueueWorkerThreads 1 +# action params: +$OMFileFlushOnTXEnd off +$OMFileZipLevel 6 +#$OMFileIOBufferSize 256k +$DynaFileCacheSize 4 +$omfileFlushInterval 1 +*.* ?dynfile;outfmt +# listener +$InputTCPServerInputName 13514 +$InputTCPServerBindRuleset R13514 +$InputTCPServerRun 13514 + + +## RULESET with listener +$Ruleset R13515 +# queue params: +$ActionQueueTimeoutShutdown 60000 +$ActionQueueSize 5000 +$ActionQueueSaveOnShutdown on +$ActionQueueHighWaterMark 4900 +$ActionQueueLowWaterMark 3500 +$ActionQueueType FixedArray +$ActionQueueWorkerThreads 1 +# action params: +$OMFileFlushOnTXEnd off +$OMFileZipLevel 6 +$OMFileIOBufferSize 256k +$DynaFileCacheSize 4 +$omfileFlushInterval 1 +*.* ?dynfile;outfmt +# listener +$InputTCPServerInputName 13515 +$InputTCPServerBindRuleset R13515 +$InputTCPServerRun 13515 + + + +## RULESET with listener +$Ruleset R13516 +# queue params: +$ActionQueueTimeoutShutdown 60000 +$ActionQueueSize 5000 +$ActionQueueSaveOnShutdown on +$ActionQueueHighWaterMark 4900 +$ActionQueueLowWaterMark 3500 +$ActionQueueType FixedArray +$ActionQueueWorkerThreads 1 +# action params: +$OMFileFlushOnTXEnd off +$OMFileZipLevel 6 +$OMFileIOBufferSize 256k +$DynaFileCacheSize 4 +$omfileFlushInterval 1 +*.* ?dynfile;outfmt +# listener +$InputTCPServerInputName 13516 +$InputTCPServerBindRuleset R13516 +$InputTCPServerRun 13516 + + diff --git a/tests/testsuites/diag-common2.conf b/tests/testsuites/diag-common2.conf new file mode 100644 index 00000000..94f7e87f --- /dev/null +++ b/tests/testsuites/diag-common2.conf @@ -0,0 +1,16 @@ +# This is a config include file. It sets up rsyslog so that the +# diag system can successfully be used. Also, it generates a file +# "rsyslogd.started" after rsyslogd is initialized. This config file +# should be included in all tests that intend to use common code for +# controlling the daemon. +# NOTE: we assume that rsyslogd's current working directory is +# ./tests (or the distcheck equivalent), in particlular that this +# config file resides in the testsuites subdirectory. +# rgerhards, 2009-05-27 +$ModLoad ../plugins/imdiag/.libs/imdiag +$IMDiagServerRun 13501 + +$template startupfile,"rsyslogd2.started" # trick to use relative path names! +:syslogtag, contains, "rsyslogd" ?startupfile + +$ErrorMessagesToStderr off diff --git a/tests/testsuites/diskqueue-fsync.conf b/tests/testsuites/diskqueue-fsync.conf new file mode 100644 index 00000000..0a02c6ce --- /dev/null +++ b/tests/testsuites/diskqueue-fsync.conf @@ -0,0 +1,17 @@ +# Test for queue disk mode (see .sh file for details) +# rgerhards, 2009-04-17 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$InputTCPServerRun 13514 + +# set spool locations and switch queue to disk-only mode +$WorkDirectory test-spool +$MainMsgQueueSyncQueueFiles on +$MainMsgQueueTimeoutShutdown 10000 +$MainMsgQueueFilename mainq +$MainMsgQueueType disk + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/dynfile_cachemiss.conf b/tests/testsuites/dynfile_cachemiss.conf new file mode 100644 index 00000000..273ff176 --- /dev/null +++ b/tests/testsuites/dynfile_cachemiss.conf @@ -0,0 +1,14 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:3%\n" +$template dynfile,"%msg:F,58:2%.log" # complete name is in message +$OMFileFlushOnTXEnd on +$DynaFileCacheSize 4 +$IncludeConfig rsyslog.action.1.include +local0.* ?dynfile;outfmt diff --git a/tests/testsuites/dynfile_invalid2.conf b/tests/testsuites/dynfile_invalid2.conf new file mode 100644 index 00000000..6d94c40d --- /dev/null +++ b/tests/testsuites/dynfile_invalid2.conf @@ -0,0 +1,14 @@ +# simple async writing test +# rgerhards, 2010-03-22 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:3%\n" +$template dynfile,"%msg:F,58:2%.log" # complete name is in message +$OMFileFlushOnTXEnd off +$DynaFileCacheSize 4 +$omfileFlushInterval 1 +local0.* ?dynfile;outfmt diff --git a/tests/testsuites/execonlyonce.conf b/tests/testsuites/execonlyonce.conf new file mode 100644 index 00000000..085b970e --- /dev/null +++ b/tests/testsuites/execonlyonce.conf @@ -0,0 +1,12 @@ +# see the equally-named .sh file for details +# rgerhards, 2009-11-12 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +$ActionExecOnlyOnceEveryInterval 3 +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/execonlyonce.data b/tests/testsuites/execonlyonce.data new file mode 100644 index 00000000..3c54f3d4 --- /dev/null +++ b/tests/testsuites/execonlyonce.data @@ -0,0 +1,2 @@ +00000001 +00000100 diff --git a/tests/testsuites/gzipwr_large.conf b/tests/testsuites/gzipwr_large.conf new file mode 100644 index 00000000..54ad3bb3 --- /dev/null +++ b/tests/testsuites/gzipwr_large.conf @@ -0,0 +1,15 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$MaxMessageSize 10k +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%,%msg:F,58:3%,%msg:F,58:4%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +$OMFileFlushOnTXEnd off +$OMFileZipLevel 6 +$OMFileIOBufferSize 256k +local0.* ?dynfile;outfmt diff --git a/tests/testsuites/gzipwr_large_dynfile.conf b/tests/testsuites/gzipwr_large_dynfile.conf new file mode 100644 index 00000000..3a1b255a --- /dev/null +++ b/tests/testsuites/gzipwr_large_dynfile.conf @@ -0,0 +1,17 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$MaxMessageSize 10k +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:3%,%msg:F,58:4%,%msg:F,58:5%\n" +$template dynfile,"rsyslog.out.%msg:F,58:2%.log" # use multiple dynafiles +$OMFileFlushOnTXEnd off +$OMFileZipLevel 6 +$OMFileIOBufferSize 256k +$DynaFileCacheSize 4 +$omfileFlushInterval 1 +local0.* ?dynfile;outfmt diff --git a/tests/testsuites/master.nolimittag b/tests/testsuites/master.nolimittag new file mode 100644 index 00000000..502d9d5d --- /dev/null +++ b/tests/testsuites/master.nolimittag @@ -0,0 +1,11 @@ +<167>Mar 6 16:57:54 172.20.245.8 TAG: Rest of message... ++TAG:+ +# now one char, no colon +<167>Mar 6 16:57:54 172.20.245.8 0 Rest of message... ++0+ +# Now exactly with 32 characters +<167>Mar 6 16:57:54 172.20.245.8 01234567890123456789012345678901 Rest of message... ++01234567890123456789012345678901+ +# Now oversize, should be completely output with this config +<167>Mar 6 16:57:54 172.20.245.8 01234567890123456789012345678901-toolong Rest of message... ++01234567890123456789012345678901-toolong+ diff --git a/tests/testsuites/master.rfctag b/tests/testsuites/master.rfctag new file mode 100644 index 00000000..3f1e0c66 --- /dev/null +++ b/tests/testsuites/master.rfctag @@ -0,0 +1,11 @@ +<167>Mar 6 16:57:54 172.20.245.8 TAG: Rest of message... ++TAG:+ +# now one char, no colon +<167>Mar 6 16:57:54 172.20.245.8 0 Rest of message... ++0+ +# Now exactly with 32 characters +<167>Mar 6 16:57:54 172.20.245.8 01234567890123456789012345678901 Rest of message... ++01234567890123456789012345678901+ +# Now oversize, should be truncated with this config +<167>Mar 6 16:57:54 172.20.245.8 01234567890123456789012345678901-toolong Rest of message... ++01234567890123456789012345678901+ diff --git a/tests/testsuites/master.subsecond b/tests/testsuites/master.subsecond new file mode 100644 index 00000000..ee924877 --- /dev/null +++ b/tests/testsuites/master.subsecond @@ -0,0 +1,8 @@ +<34>1 2003-01-23T12:34:56.003Z mymachine.example.com su - ID47 - MSG +003 +# full precision +<34>1 2003-01-23T12:34:56.123456Z mymachine.example.com su - ID47 - MSG +123456 +# without +<34>1 2003-01-23T12:34:56Z mymachine.example.com su - ID47 - MSG +0 diff --git a/tests/testsuites/master.ts3339 b/tests/testsuites/master.ts3339 new file mode 100644 index 00000000..b4dd5f39 --- /dev/null +++ b/tests/testsuites/master.ts3339 @@ -0,0 +1,22 @@ +<34>1 2003-11-11T22:14:15.003Z mymachine.example.com su - ID47 - MSG +2003-11-11T22:14:15.003Z +# next test +<34>1 2003-01-11T22:14:15.003Z mymachine.example.com su - ID47 - MSG +2003-01-11T22:14:15.003Z +# next test +<34>1 2003-11-01T22:04:15.003Z mymachine.example.com su - ID47 - MSG +2003-11-01T22:04:15.003Z +# next test +<34>1 2003-11-11T02:14:15.003Z mymachine.example.com su - ID47 - MSG +2003-11-11T02:14:15.003Z +# next test +<34>1 2003-11-11T22:04:05.003Z mymachine.example.com su - ID47 - MSG +2003-11-11T22:04:05.003Z +# next test +<34>1 2003-11-11T22:04:05.003+02:00 mymachine.example.com su - ID47 - MSG +2003-11-11T22:04:05.003+02:00 +# next test +<34>1 2003-11-11T22:04:05.003+01:30 mymachine.example.com su - ID47 - MSG +2003-11-11T22:04:05.003+01:30 +<34>1 2003-11-11T22:04:05.123456+01:30 mymachine.example.com su - ID47 - MSG +2003-11-11T22:04:05.123456+01:30 diff --git a/tests/testsuites/master.tsmysql b/tests/testsuites/master.tsmysql new file mode 100644 index 00000000..dc6d85be --- /dev/null +++ b/tests/testsuites/master.tsmysql @@ -0,0 +1,2 @@ +<34>1 2003-01-23T12:34:56.003Z mymachine.example.com su - ID47 - MSG +20030123123456 diff --git a/tests/testsuites/master.tspgsql b/tests/testsuites/master.tspgsql new file mode 100644 index 00000000..d7ac19ff --- /dev/null +++ b/tests/testsuites/master.tspgsql @@ -0,0 +1,2 @@ +<34>1 2003-01-23T12:34:56.003Z mymachine.example.com su - ID47 - MSG +2003-01-23 12:34:56 diff --git a/tests/testsuites/mon1digit.ts3164 b/tests/testsuites/mon1digit.ts3164 new file mode 100644 index 00000000..0cb1c8e2 --- /dev/null +++ b/tests/testsuites/mon1digit.ts3164 @@ -0,0 +1,3 @@ +<167>Jan 6 16:57:54 172.20.245.8 TAG: MSG +Jan 6 16:57:54 +#Only the first two lines are important, you may place anything behind them! diff --git a/tests/testsuites/mon2digit.ts3164 b/tests/testsuites/mon2digit.ts3164 new file mode 100644 index 00000000..9606961c --- /dev/null +++ b/tests/testsuites/mon2digit.ts3164 @@ -0,0 +1,3 @@ +<167>Jan 16 16:57:54 172.20.245.8 TAG: MSG +Jan 16 16:57:54 +#Only the first two lines are important, you may place anything behind them! diff --git a/tests/testsuites/nolimittag.conf b/tests/testsuites/nolimittag.conf new file mode 100644 index 00000000..0b6ec387 --- /dev/null +++ b/tests/testsuites/nolimittag.conf @@ -0,0 +1,8 @@ +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format +$template fmt,"+%syslogtag%+\n" +*.* :omstdout:;fmt diff --git a/tests/testsuites/oversizeTag-1.parse1 b/tests/testsuites/oversizeTag-1.parse1 new file mode 100644 index 00000000..d45ba1f2 --- /dev/null +++ b/tests/testsuites/oversizeTag-1.parse1 @@ -0,0 +1,2 @@ +<38>Mar 27 19:06:53 source_server 0123456789012345678901234567890123456789: MSG part +38,auth,info,Mar 27 19:06:53,source_server,0123456789012345678901234567890123456789,0123456789012345678901234567890123456789:, MSG part diff --git a/tests/testsuites/parse-3164-buggyday.conf b/tests/testsuites/parse-3164-buggyday.conf new file mode 100644 index 00000000..937f423a --- /dev/null +++ b/tests/testsuites/parse-3164-buggyday.conf @@ -0,0 +1,8 @@ +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format that we can easily parse in expect +$template expect,"%PRI%,%syslogfacility-text%,%syslogseverity-text%,%timestamp:::date-rfc3164-buggyday%,%hostname%,%programname%,%syslogtag%,%msg%\n" +*.* :omstdout:;expect diff --git a/tests/testsuites/parse-nodate.conf b/tests/testsuites/parse-nodate.conf new file mode 100644 index 00000000..570638d9 --- /dev/null +++ b/tests/testsuites/parse-nodate.conf @@ -0,0 +1,14 @@ +# test is a test config that does not include the timestamp. This is necessary to +# test some illformed messages that do not contain a date. In that case, the system's +# current timestamp is used, and that of course is a bit hard to compare against +# a fixed template. So the solution in this case is to use a format that does +# not contain any timestamp. Maybe not optimal, but it works ;) +# rgerhards, 2010-03-19 +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format that we can easily parse +$template fmt,"%PRI%,%syslogfacility-text%,%syslogseverity-text%,%hostname%,%programname%,%syslogtag%,%msg%\n" +*.* :omstdout:;fmt diff --git a/tests/testsuites/parse1udp.conf b/tests/testsuites/parse1udp.conf new file mode 100644 index 00000000..0fb7d16d --- /dev/null +++ b/tests/testsuites/parse1udp.conf @@ -0,0 +1,9 @@ +$ModLoad ../plugins/omstdout/.libs/omstdout +$ModLoad ../plugins/imudp/.libs/imudp +$UDPServerRun 12514 + +$ErrorMessagesToStderr off + +# use a special format that we can easily parse in expect +$template expect,"%PRI%,%syslogfacility-text%,%syslogseverity-text%,%timestamp%,%hostname%,%programname%,%syslogtag%,%msg%\n" +*.* :omstdout:;expect diff --git a/tests/testsuites/parse3.conf b/tests/testsuites/parse3.conf new file mode 100644 index 00000000..8a3cb317 --- /dev/null +++ b/tests/testsuites/parse3.conf @@ -0,0 +1,10 @@ +# note: we need to strip off the TZ designator in the rfc3339 timestamp +# as this test otherwise fails in different timezones! +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format that we can easily parse in expect +$Template output,"%timereported:1:19:date-rfc3339,csv%, %hostname:::csv%, %programname:::csv%, %syslogtag:R,ERE,0,BLANK:[0-9]+--end:csv%, %syslogseverity:::csv%, %msg:::drop-last-lf,csv%\n" +*.* :omstdout:;output diff --git a/tests/testsuites/parse_invld_regex.conf b/tests/testsuites/parse_invld_regex.conf new file mode 100644 index 00000000..d18a2b3c --- /dev/null +++ b/tests/testsuites/parse_invld_regex.conf @@ -0,0 +1,10 @@ +# note: we need to strip off the TZ designator in the rfc3339 timestamp +# as this test otherwise fails in different timezones! +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format that we can easily parse in expect +$Template output,"%timereported:1:19:date-rfc3339,csv%, %hostname:::csv%, %programname:::csv%, %syslogtag:R,ERE,0,BLANK:[0-9+--end:csv%, %syslogseverity:::csv%, %msg:::drop-last-lf,csv%\n" +*.* :omstdout:;output diff --git a/tests/testsuites/pipeaction.conf b/tests/testsuites/pipeaction.conf new file mode 100644 index 00000000..f58b6d65 --- /dev/null +++ b/tests/testsuites/pipeaction.conf @@ -0,0 +1,16 @@ +# Test for pipe output action (see .sh file for details) +# rgerhards, 2009-11-05 +$IncludeConfig diag-common.conf + +$MainMsgQueueTimeoutShutdown 10000 + +# set spool locations and switch queue to disk-only mode +$WorkDirectory test-spool +$MainMsgQueueFilename mainq +$MainMsgQueueType disk + +$template outfmt,"%msg:F,58:2%\n" +# with pipes, we do not need to use absolute path names, so +# we can simply refer to our working pipe via the usual relative +# path name +:msg, contains, "msgnum:" |rsyslog-testbench-fifo;outfmt diff --git a/tests/testsuites/random.conf b/tests/testsuites/random.conf new file mode 100644 index 00000000..a7079df1 --- /dev/null +++ b/tests/testsuites/random.conf @@ -0,0 +1,13 @@ +# we write to /dev/null, as we have no chance to verify the output +# in any case. What we really check is that rsyslogd does not +# segfault or otherwise abort. +# rgerhards, 2010-04-01 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%rawmsg%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +*.* /dev/null diff --git a/tests/testsuites/reallife.parse3 b/tests/testsuites/reallife.parse3 new file mode 100644 index 00000000..dad3f56e --- /dev/null +++ b/tests/testsuites/reallife.parse3 @@ -0,0 +1,15 @@ +# New tests should be added to this file if there is no specific +# reason for not doing that. Initially, we could only handle one test +# case per file, but this restriction has been removed some time ago. +# So it is less troublesome (and easier to overlook) to have all related +# tests in a single file. +# This file contains a lot of real-life samples (of course mangled so +# that they can not be traced back to the original submitter). Note +# that IP addr 192.0.2.1 is specifically set aside for testing and +# documentation by IANA. +# rgerhards, 2009-10-19 +<175>Oct 16 2009 23:47:31 hostname tag This is a message +"2009-10-16T23:47:31", "hostname", "tag", "", "7", " This is a message" +# +<175>Oct 16 2009 23:47:31 hostname tag[1234] This is a message +"2009-10-16T23:47:31", "hostname", "tag", "1234", "7", " This is a message" diff --git a/tests/testsuites/rfctag.conf b/tests/testsuites/rfctag.conf new file mode 100644 index 00000000..8619e89e --- /dev/null +++ b/tests/testsuites/rfctag.conf @@ -0,0 +1,9 @@ +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format +# Note: the plus signs are necessary to detect truncated logs! +$template fmt,"+%syslogtag:1:32%+\n" +*.* :omstdout:;fmt diff --git a/tests/testsuites/samples.parse-3164-buggyday b/tests/testsuites/samples.parse-3164-buggyday new file mode 100644 index 00000000..e21df980 --- /dev/null +++ b/tests/testsuites/samples.parse-3164-buggyday @@ -0,0 +1,6 @@ +# in 3164-buggyday mode, we need to have a leading zero in front of the day +<38> Mar 7 19:06:53 example tag: testmessage (only date actually tested) +38,auth,info,Mar 07 19:06:53,example,tag,tag:, testmessage (only date actually tested) +# and now one with a complete date: +<38> Mar 17 19:06:53 example tag: testmessage (only date actually tested) +38,auth,info,Mar 17 19:06:53,example,tag,tag:, testmessage (only date actually tested) diff --git a/tests/testsuites/samples.parse-nodate b/tests/testsuites/samples.parse-nodate new file mode 100644 index 00000000..7f16181c --- /dev/null +++ b/tests/testsuites/samples.parse-nodate @@ -0,0 +1,6 @@ +<27>xapi: [error|xen3|15|Guest liveness monitor D:bca30ab3f1c1|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message) +27,daemon,err,localhost,xapi,xapi:, [error|xen3|15|Guest liveness monitor D:bca30ab3f1c1|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message) +# a message with just text (as permitted by rfc 3164) +# it is questionable if the current sample result is really correct as of 3164! +This is a message! +13,user,notice,This,is,is, a message! diff --git a/tests/testsuites/samples.parse_invld_regex b/tests/testsuites/samples.parse_invld_regex new file mode 100644 index 00000000..0d0e4ce3 --- /dev/null +++ b/tests/testsuites/samples.parse_invld_regex @@ -0,0 +1,16 @@ +# New tests should be added to this file if there is no specific +# reason for not doing that. Initially, we could only handle one test +# case per file, but this restriction has been removed some time ago. +# So it is less troublesome (and easier to overlook) to have all related +# tests in a single file. +# the actual message is not important. There is an error inside the conf +# file, and all messages will trigger the same problem. +# NOTE: it is correct that the "BAD REGULAR EXPRESSION" error message is +# *NOT* run through the rest of the propety replace, in specific through +# the CSV escaper. We do not do this because it could potentially lead +# to an obfuscated error message, and thus making problems hard to find. As +# this is a real error case, there is no problem in not obeying to the +# configured format. +# rgerhards, 2010-02-08 +<175>Feb 08 2008 23:47:31 hostname tag This is a message +"2008-02-08T23:47:31", "hostname", "tag", **NO MATCH** **BAD REGULAR EXPRESSION**, "7", " This is a message" diff --git a/tests/testsuites/samples.snare_ccoff_udp b/tests/testsuites/samples.snare_ccoff_udp new file mode 100644 index 00000000..010e44d5 --- /dev/null +++ b/tests/testsuites/samples.snare_ccoff_udp @@ -0,0 +1,14 @@ +# see comments in snare_ccoff_udp.conf +# note that some of these samples look pretty wild, but they are +# *real* cases (just mangled to anonymize them...) +# Sample 1 - note the absence of PRI! +windowsserver MSWinEventLog 1 Security 1167 Fri Mar 19 15:33:30 2010 540 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Successful Network Logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {79b6eb79-7bcc-8a2e-7dad-953c51dc00fd} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 10.11.11.3 Source Port: 3306 733\n +13,user,notice,localhost,windowsserver,windowsserver MSWinEventLog 1 Security 1167 Fri, Mar 19 15:33:30 2010 540 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Successful Network Logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {79b6eb79-7bcc-8a2e-7dad-953c51dc00fd} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 10.11.11.3 Source Port: 3306 733 +# Sample 2 +# the samples below need to be disabled for the "workaround patch" for the message +# parser to work. They need to be re-enabled once a final solution has been crafted +#windowsserver MSWinEventLog 1 Security 1166 Fri Mar 19 15:33:30 2010 576 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Special privileges assigned to new logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Privileges: SeSecurityPrivilege SeBackupPrivilege SeRestorePrivilege SeTakeOwnershipPrivilege SeDebugPrivilege SeSystemEnvironmentPrivilege SeLoadDriverPrivilege SeImpersonatePrivilege SeEnableDelegationPrivilege 732\n +#13,user,notice,localhost,windowsserver,windowsserver MSWinEventLog 1 Security 1166 Fri, Mar 19 15:33:30 2010 576 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Special privileges assigned to new logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Privileges: SeSecurityPrivilege SeBackupPrivilege SeRestorePrivilege SeTakeOwnershipPrivilege SeDebugPrivilege SeSystemEnvironmentPrivilege SeLoadDriverPrivilege SeImpersonatePrivilege SeEnableDelegationPrivilege 732 +# Sample 3 +#windowsserver MSWinEventLog 1 Security 1165 Fri Mar 19 15:33:30 2010 538 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff User Logoff: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF8830B) Logon Type: 3 731\n +#13,user,notice,localhost,windowsserver,windowsserver MSWinEventLog 1 Security 1165 Fri, Mar 19 15:33:30 2010 538 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff User Logoff: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF8830B) Logon Type: 3 731 diff --git a/tests/testsuites/samples.snare_ccoff_udp2 b/tests/testsuites/samples.snare_ccoff_udp2 new file mode 100644 index 00000000..337cd97c --- /dev/null +++ b/tests/testsuites/samples.snare_ccoff_udp2 @@ -0,0 +1,26 @@ +# see comments in snare_ccoff_udp.conf +# note that some of these samples look pretty wild, but they are +# *real* cases (just mangled to anonymize them...) +# +# NOTE +# The current responses are probably not correct (handling of messages without PRI). +# However, we keep them inside the test to be consistent. We should look at how +# PRI-less messages are handled and once we have fixed that, the test cases may need +# to be adapted. We do NOT try to preserve misbehaviour on such seriously malformed +# messages. +# +# this is a very simple test, though not snare-based +test +insert into windows (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('', 1, 'test',5, '20100321185328', '20100321185328', 1, '') +# and yet another one we have seen in practice +UX=Abcd-efg-hij-klmno; XXXXX=1111111111, Z123=192.12.231.245:11111, S1234=123456789, XXXXXX=111111111 +insert into windows (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values (' XXXXX=1111111111, Z123=192.12.231.245:11111, S1234=123456789, XXXXXX=111111111', 1, 'localhost',5, '20100321185328', '20100321185328', 1, 'UX=Abcd-efg-hij-klmno;') +# Sample 1 - note the absence of PRI! +windowsserver MSWinEventLog 1 Security 1167 Fri Mar 19 15:33:30 2010 540 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Successful Network Logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {79b6eb79-7bcc-8a2e-7dad-953c51dc00fd} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 10.11.11.3 Source Port: 3306 733\n +insert into windows (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values (' Mar 19 15:33:30 2010 540 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Successful Network Logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {79b6eb79-7bcc-8a2e-7dad-953c51dc00fd} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 10.11.11.3 Source Port: 3306 733', 1, 'localhost',5, '20100321185328', '20100321185328', 1, 'windowsserver MSWinEventLog 1 Security 1167 Fri') +# Sample 2 +windowsserver MSWinEventLog 1 Security 1166 Fri Mar 19 15:33:30 2010 576 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Special privileges assigned to new logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Privileges: SeSecurityPrivilege SeBackupPrivilege SeRestorePrivilege SeTakeOwnershipPrivilege SeDebugPrivilege SeSystemEnvironmentPrivilege SeLoadDriverPrivilege SeImpersonatePrivilege SeEnableDelegationPrivilege 732\n +insert into windows (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values (' Mar 19 15:33:30 2010 576 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Special privileges assigned to new logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Privileges: SeSecurityPrivilege SeBackupPrivilege SeRestorePrivilege SeTakeOwnershipPrivilege SeDebugPrivilege SeSystemEnvironmentPrivilege SeLoadDriverPrivilege SeImpersonatePrivilege SeEnableDelegationPrivilege 732', 1, 'localhost',5, '20100321185328', '20100321185328', 1, 'windowsserver MSWinEventLog 1 Security 1166 Fri') +# Sample 3 +windowsserver MSWinEventLog 1 Security 1165 Fri Mar 19 15:33:30 2010 538 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff User Logoff: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF8830B) Logon Type: 3 731\n +insert into windows (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values (' Mar 19 15:33:30 2010 538 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff User Logoff: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF8830B) Logon Type: 3 731', 1, 'localhost',5, '20100321185328', '20100321185328', 1, 'windowsserver MSWinEventLog 1 Security 1165 Fri') diff --git a/tests/testsuites/snare.parse1 b/tests/testsuites/snare.parse1 new file mode 100644 index 00000000..550b0703 --- /dev/null +++ b/tests/testsuites/snare.parse1 @@ -0,0 +1,83 @@ +# some parse test build around data in snare-format +<141>Mar 10 09:30:20 zuse.xysystems.local MSWinEventLog\0111\011Security\011563\011Wed Mar 10 09:30:15 2010\011538\011Security\011XYWS011$\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011User Logoff: User Name: XYWS011$ Domain: XYZSYSTEMS Logon ID: (0x0,0x5984789C) Logon Type: 3 \011552 +141,local1,notice,Mar 10 09:30:20,zuse.xysystems.local,MSWinEventLog#0111#011Security#011563#011Wed,MSWinEventLog#0111#011Security#011563#011Wed, Mar 10 09:30:15 2010#011538#011Security#011XYWS011$#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011User Logoff: User Name: XYWS011$ Domain: XYZSYSTEMS Logon ID: (0x0,0x5984789C) Logon Type: 3 #011552 +# +# NEXT MESSAGE +# +Mar 10 09:30:20 zuse.xysystems.local MSWinEventLog\0111\011Security\011564\011Wed Mar 10 09:30:19 2010\011540\011Security\011BACKUP1$\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: BACKUP1$ Domain: XYZSYSTEMS Logon ID: (0x0,0x59848DB4) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {f6f65903-1932-d229-4b75-64816121d569} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.31 Source Port: 0 \011553 +13,user,notice,Mar 10 09:30:20,zuse.xysystems.local,MSWinEventLog#0111#011Security#011564#011Wed,MSWinEventLog#0111#011Security#011564#011Wed, Mar 10 09:30:19 2010#011540#011Security#011BACKUP1$#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: BACKUP1$ Domain: XYZSYSTEMS Logon ID: (0x0,0x59848DB4) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {f6f65903-1932-d229-4b75-64816121d569} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.31 Source Port: 0 #011553 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011566\011Wed Mar 10 09:30:21 2010\011540\011Security\011aadminps\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: aadminps Domain: XYSYSTEMS Logon ID: (0x0,0x5984973C) Logon Type: 3 Logon Process: Authz Authentication Package: Kerberos Workstation Name: ZUSE Logon GUID: - Caller User Name: ZUSE$ Caller Domain: XYSYSTEMS Caller Logon ID: (0x0,0x3E7) Caller Process ID: 1004 Transited Services: - Source Network Address: - Source Port: - \011555 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011566#011Wed,MSWinEventLog#0111#011Security#011566#011Wed, Mar 10 09:30:21 2010#011540#011Security#011aadminps#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: aadminps Domain: XYSYSTEMS Logon ID: (0x0,0x5984973C) Logon Type: 3 Logon Process: Authz Authentication Package: Kerberos Workstation Name: ZUSE Logon GUID: - Caller User Name: ZUSE$ Caller Domain: XYSYSTEMS Caller Logon ID: (0x0,0x3E7) Caller Process ID: 1004 Transited Services: - Source Network Address: - Source Port: - #011555 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011567\011Wed Mar 10 09:30:21 2010\011538\011Security\011aadminps\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011User Logoff: User Name: aadminps Domain: XYSYSTEMS Logon ID: (0x0,0x5984973C) Logon Type: 3 \011556 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011567#011Wed,MSWinEventLog#0111#011Security#011567#011Wed, Mar 10 09:30:21 2010#011538#011Security#011aadminps#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011User Logoff: User Name: aadminps Domain: XYSYSTEMS Logon ID: (0x0,0x5984973C) Logon Type: 3 #011556 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011568\011Wed Mar 10 09:30:25 2010\011540\011Security\011ANONYMOUS LOGON\011Well Known Group\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: Domain: Logon ID: (0x0,0x5984AB6F) Logon Type: 3 Logon Process: NtLmSsp Authentication Package: NTLM Workstation Name: XYWS083 Logon GUID: - Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.3.91 Source Port: 0 \011557 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011568#011Wed,MSWinEventLog#0111#011Security#011568#011Wed, Mar 10 09:30:25 2010#011540#011Security#011ANONYMOUS LOGON#011Well Known Group#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: Domain: Logon ID: (0x0,0x5984AB6F) Logon Type: 3 Logon Process: NtLmSsp Authentication Package: NTLM Workstation Name: XYWS083 Logon GUID: - Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.3.91 Source Port: 0 #011557 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011569\011Wed Mar 10 09:30:25 2010\011540\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984ACA7) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2318 \011558 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011569#011Wed,MSWinEventLog#0111#011Security#011569#011Wed, Mar 10 09:30:25 2010#011540#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984ACA7) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2318 #011558 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011570\011Wed Mar 10 09:30:25 2010\011538\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984ACA7) Logon Type: 3 \011559 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011570#011Wed,MSWinEventLog#0111#011Security#011570#011Wed, Mar 10 09:30:25 2010#011538#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984ACA7) Logon Type: 3 #011559 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011571\011Wed Mar 10 09:30:25 2010\011540\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AD7C) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2319 \011560\ +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011571#011Wed,MSWinEventLog#0111#011Security#011571#011Wed, Mar 10 09:30:25 2010#011540#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AD7C) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2319 #011560 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011572\011Wed Mar 10 09:30:25 2010\011538\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AD7C) Logon Type: 3 \011561 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011572#011Wed,MSWinEventLog#0111#011Security#011572#011Wed, Mar 10 09:30:25 2010#011538#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AD7C) Logon Type: 3 #011561 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011573\011Wed Mar 10 09:30:25 2010\011680\011Security\011ettore.trezzani\011User\011Success Audit\011ZUSE\011Account Logon\011\011Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon account: ettore.trezzani Source Workstation: XYWS083 Error Code: 0x0 \011562 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011573#011Wed,MSWinEventLog#0111#011Security#011573#011Wed, Mar 10 09:30:25 2010#011680#011Security#011ettore.trezzani#011User#011Success Audit#011ZUSE#011Account Logon#011#011Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon account: ettore.trezzani Source Workstation: XYWS083 Error Code: 0x0 #011562 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011574\011Wed Mar 10 09:30:25 2010\011540\011Security\011ettore.trezzani\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: ettore.trezzani Domain: XYSYSTEMS Logon ID: (0x0,0x5984ADD5) Logon Type: 3 Logon Process: NtLmSsp Authentication Package: NTLM Workstation Name: XYWS083 Logon GUID: - Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.3.91 Source Port: 0 \011563 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011574#011Wed,MSWinEventLog#0111#011Security#011574#011Wed, Mar 10 09:30:25 2010#011540#011Security#011ettore.trezzani#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: ettore.trezzani Domain: XYSYSTEMS Logon ID: (0x0,0x5984ADD5) Logon Type: 3 Logon Process: NtLmSsp Authentication Package: NTLM Workstation Name: XYWS083 Logon GUID: - Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.3.91 Source Port: 0 #011563 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011575\011Wed Mar 10 09:30:25 2010\011540\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AE49) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2320 \011564 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011575#011Wed,MSWinEventLog#0111#011Security#011575#011Wed, Mar 10 09:30:25 2010#011540#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AE49) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2320 #011564 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011576\011Wed Mar 10 09:30:25 2010\011538\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AE49) Logon Type: 3 \011565 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011576#011Wed,MSWinEventLog#0111#011Security#011576#011Wed, Mar 10 09:30:25 2010#011538#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AE49) Logon Type: 3 #011565 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011577\011Wed Mar 10 09:30:25 2010\011540\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AF00) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2321 \011566 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011577#011Wed,MSWinEventLog#0111#011Security#011577#011Wed, Mar 10 09:30:25 2010#011540#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AF00) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2321 #011566 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011578\011Wed Mar 10 09:30:25 2010\011538\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AF00) Logon Type: 3 \011567 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011578#011Wed,MSWinEventLog#0111#011Security#011578#011Wed, Mar 10 09:30:25 2010#011538#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AF00) Logon Type: 3 #011567 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011579\011Wed Mar 10 09:30:25 2010\011538\011Security\011ANONYMOUS LOGON\011Well Known Group\011Success Audit\011ZUSE\011Logon/Logoff\011\011User Logoff: User Name: ANONYMOUS LOGON Domain: NT AUTHORITY Logon ID: (0x0,0x5984AB6F) Logon Type: 3 \011568 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011579#011Wed,MSWinEventLog#0111#011Security#011579#011Wed, Mar 10 09:30:25 2010#011538#011Security#011ANONYMOUS LOGON#011Well Known Group#011Success Audit#011ZUSE#011Logon/Logoff#011#011User Logoff: User Name: ANONYMOUS LOGON Domain: NT AUTHORITY Logon ID: (0x0,0x5984AB6F) Logon Type: 3 #011568 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:30 zuse.xysystems.local MSWinEventLog\0111\011Security\011580\011Wed Mar 10 09:30:29 2010\011540\011Security\011XYWSBADGE$\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: XYWSBADGE$ Domain: XYSYSTEMS Logon ID: (0x0,0x59852D73) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {4bc3c075-5a77-4648-5822-bfdf88b4c211} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.3.18 Source Port: 0 \011569 +141,local1,notice,Mar 10 09:30:30,zuse.xysystems.local,MSWinEventLog#0111#011Security#011580#011Wed,MSWinEventLog#0111#011Security#011580#011Wed, Mar 10 09:30:29 2010#011540#011Security#011XYWSBADGE$#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: XYWSBADGE$ Domain: XYSYSTEMS Logon ID: (0x0,0x59852D73) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {4bc3c075-5a77-4648-5822-bfdf88b4c211} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.3.18 Source Port: 0 #011569 diff --git a/tests/testsuites/snare_ccoff_udp.conf b/tests/testsuites/snare_ccoff_udp.conf new file mode 100644 index 00000000..6abbedf4 --- /dev/null +++ b/tests/testsuites/snare_ccoff_udp.conf @@ -0,0 +1,21 @@ +# This test some real-world snare cases. I don't like snare (no wonder +# as I have written EventReporter, the ultimate Windows-to-Syslog tool), +# but besides that snare generates severely malformed messages that +# really stress-test the rsyslog engine. They deserve to be beaten by someone ;) +# This test needs to be run over UDP only, as snare puts LF INTO some of the messages, +# which makes it impossible to try these out via traditional syslog/tcp +# added 2010-03-21 rgerhards +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# snare usses HT as field delimiter, so many users have turned off +# control character escaping to make any sense at all from these messages... +$EscapeControlCharactersOnReceive off + +# use a special format that we can easily check. We do NOT include a timestamp because +# the malformed snare messages usually do not contain one (and we can not check against +# the system time in our test cases). +$template fmt,"%PRI%,%syslogfacility-text%,%syslogseverity-text%,%hostname%,%programname%,%syslogtag%,%msg%\n" +*.* :omstdout:;fmt diff --git a/tests/testsuites/snare_ccoff_udp2.conf b/tests/testsuites/snare_ccoff_udp2.conf new file mode 100644 index 00000000..9115c14f --- /dev/null +++ b/tests/testsuites/snare_ccoff_udp2.conf @@ -0,0 +1,17 @@ +# Similar to snare_ccoff_udp_2, but with a different template. This template +# has triggered problems in the past, thus a test is granted. +# added 2010-03-21 rgerhards +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# snare usses HT as field delimiter, so many users have turned off +# control character escaping to make any sense at all from these messages... +$EscapeControlCharactersOnReceive off + +# we need to use a fixed timestamp, as otherwise we can not compare :( +# This could be improved in later versions of the testing tools, but requires +# modification to the rsyslog core... +$template fmt,"insert into windows (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg:::space-cc%', %syslogfacility%, '%HOSTNAME%',%syslogpriority%, '20100321185328', '20100321185328', %iut%, '%syslogtag:::space-cc%')\n",sql +*.* :omstdout:;fmt diff --git a/tests/testsuites/sndrcv_gzip_rcvr.conf b/tests/testsuites/sndrcv_gzip_rcvr.conf new file mode 100644 index 00000000..6f7ce34b --- /dev/null +++ b/tests/testsuites/sndrcv_gzip_rcvr.conf @@ -0,0 +1,11 @@ +# see equally-named shell file for details +# rgerhards, 2009-11-11 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +# then SENDER sends to this port (not tcpflood!) +$InputTCPServerRun 13515 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/sndrcv_gzip_sender.conf b/tests/testsuites/sndrcv_gzip_sender.conf new file mode 100644 index 00000000..c874c068 --- /dev/null +++ b/tests/testsuites/sndrcv_gzip_sender.conf @@ -0,0 +1,8 @@ +# see tcpsndrcv.sh for details +# rgerhards, 2009-11-11 +$IncludeConfig diag-common2.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$InputTCPServerRun 13514 + +*.* @@127.0.0.1:13515 diff --git a/tests/testsuites/sndrcv_rcvr.conf b/tests/testsuites/sndrcv_rcvr.conf new file mode 100644 index 00000000..6f7ce34b --- /dev/null +++ b/tests/testsuites/sndrcv_rcvr.conf @@ -0,0 +1,11 @@ +# see equally-named shell file for details +# rgerhards, 2009-11-11 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +# then SENDER sends to this port (not tcpflood!) +$InputTCPServerRun 13515 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/sndrcv_sender.conf b/tests/testsuites/sndrcv_sender.conf new file mode 100644 index 00000000..c874c068 --- /dev/null +++ b/tests/testsuites/sndrcv_sender.conf @@ -0,0 +1,8 @@ +# see tcpsndrcv.sh for details +# rgerhards, 2009-11-11 +$IncludeConfig diag-common2.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$InputTCPServerRun 13514 + +*.* @@127.0.0.1:13515 diff --git a/tests/testsuites/sndrcv_tls_anon_rcvr.conf b/tests/testsuites/sndrcv_tls_anon_rcvr.conf new file mode 100644 index 00000000..01143b22 --- /dev/null +++ b/tests/testsuites/sndrcv_tls_anon_rcvr.conf @@ -0,0 +1,22 @@ +# see equally-named shell file for details +# this is the config fil for the TLS server +# rgerhards, 2009-11-11 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp + +# certificates +$DefaultNetstreamDriverCAFile testsuites/x.509/ca.pem +$DefaultNetstreamDriverCertFile testsuites/x.509/client-cert.pem +$DefaultNetstreamDriverKeyFile testsuites/x.509/client-key.pem + +$DefaultNetstreamDriver gtls # use gtls netstream driver + +# then SENDER sends to this port (not tcpflood!) +$InputTCPServerStreamDriverMode 1 +$InputTCPServerStreamDriverAuthMode anon +$InputTCPServerRun 13515 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/sndrcv_tls_anon_sender.conf b/tests/testsuites/sndrcv_tls_anon_sender.conf new file mode 100644 index 00000000..4a944455 --- /dev/null +++ b/tests/testsuites/sndrcv_tls_anon_sender.conf @@ -0,0 +1,19 @@ +# see tcpsndrcv.sh for details +# this is the TLS client +# rgerhards, 2009-11-11 +$IncludeConfig diag-common2.conf + +# certificates +$DefaultNetstreamDriverCAFile testsuites/x.509/ca.pem +$DefaultNetstreamDriverCertFile testsuites/x.509/client-cert.pem +$DefaultNetstreamDriverKeyFile testsuites/x.509/client-key.pem + +# Note: no TLS for the listener, this is for tcpflood! +$ModLoad ../plugins/imtcp/.libs/imtcp +$InputTCPServerRun 13514 + +# set up the action +$DefaultNetstreamDriver gtls # use gtls netstream driver +$ActionSendStreamDriverMode 1 # require TLS for the connection +$ActionSendStreamDriverAuthMode anon +*.* @@127.0.0.1:13515 diff --git a/tests/testsuites/subsecond.conf b/tests/testsuites/subsecond.conf new file mode 100644 index 00000000..58c26cc7 --- /dev/null +++ b/tests/testsuites/subsecond.conf @@ -0,0 +1,8 @@ +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format +$template fmt,"%timestamp:::date-subseconds%\n" +*.* :omstdout:;fmt diff --git a/tests/testsuites/threadingmq.conf b/tests/testsuites/threadingmq.conf new file mode 100644 index 00000000..b98f9b5a --- /dev/null +++ b/tests/testsuites/threadingmq.conf @@ -0,0 +1,16 @@ +# Threading test, we run a tcp flood to via an +# engine instructed to use multiple threads +# rgerhards, 2009-06-26 +$IncludeConfig diag-common.conf + +$MainMsgQueueTimeoutShutdown 100000 + +$MainMsgQueueWorkerThreadMinimumMessages 10 +$MainMsgQueueWorkerThreads 5 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +# write quickly to the output file: +$OMFileFlushOnTXEnd off +$OMFileIOBufferSize 256k +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/threadingmqaq.conf b/tests/testsuites/threadingmqaq.conf new file mode 100644 index 00000000..f0d39057 --- /dev/null +++ b/tests/testsuites/threadingmqaq.conf @@ -0,0 +1,20 @@ +# Threading test, we run a tcp flood to via an +# engine instructed to use multiple threads +# rgerhards, 2009-06-26 +$IncludeConfig diag-common.conf + +$MainMsgQueueTimeoutShutdown 10000 + +$MainMsgQueueWorkerThreadMinimumMessages 10 +$MainMsgQueueWorkerThreads 5 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +# write quickly to the output file: +$OMFileFlushOnTXEnd off +$OMFileIOBufferSize 256k +# This time, also run the action queue detached +$ActionQueueWorkerThreadMinimumMessages 10 +$ActionQueueWorkerThreads 5 +$ActionQueueType LinkedList +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/ts3164.conf b/tests/testsuites/ts3164.conf new file mode 100644 index 00000000..7aa6a8ef --- /dev/null +++ b/tests/testsuites/ts3164.conf @@ -0,0 +1,8 @@ +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format +$template fmt,"%timestamp:::date-rfc3164%\n" +*.* :omstdout:;fmt diff --git a/tests/testsuites/ts3339.conf b/tests/testsuites/ts3339.conf new file mode 100644 index 00000000..df8f23ac --- /dev/null +++ b/tests/testsuites/ts3339.conf @@ -0,0 +1,8 @@ +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format +$template fmt,"%timestamp:::date-rfc3339%\n" +*.* :omstdout:;fmt diff --git a/tests/testsuites/tsmysql.conf b/tests/testsuites/tsmysql.conf new file mode 100644 index 00000000..f97d4b0a --- /dev/null +++ b/tests/testsuites/tsmysql.conf @@ -0,0 +1,8 @@ +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format +$template fmt,"%timestamp:::date-mysql%\n" +*.* :omstdout:;fmt diff --git a/tests/testsuites/tspgsql.conf b/tests/testsuites/tspgsql.conf new file mode 100644 index 00000000..eb18c091 --- /dev/null +++ b/tests/testsuites/tspgsql.conf @@ -0,0 +1,8 @@ +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format +$template fmt,"%timestamp:::date-pgsql%\n" +*.* :omstdout:;fmt diff --git a/tests/testsuites/upcase-date.parse1 b/tests/testsuites/upcase-date.parse1 new file mode 100644 index 00000000..2d21222a --- /dev/null +++ b/tests/testsuites/upcase-date.parse1 @@ -0,0 +1,4 @@ +<6>AUG 10 22:18:24 2009 netips-warden2-p [audit] user=[*SMS] src=192.168.11.11 iface=5 access=9 Update State Reset +6,kern,info,Aug 10 22:18:24,2009,,, netips-warden2-p [audit] user=[*SMS] src=192.168.11.11 iface=5 access=9 Update State Reset +#Example from RFC3164, section 5.4 +#Only the first two lines are important, you may place anything behind them! diff --git a/tests/testsuites/weird.parse1 b/tests/testsuites/weird.parse1 new file mode 100644 index 00000000..e8b90c74 --- /dev/null +++ b/tests/testsuites/weird.parse1 @@ -0,0 +1,37 @@ +# some really weird samples, some of them seen in practice, +# some other deliberately generated. The main point is that they +# should not cause an abort... +<14>Aug 30 23:00:05 X4711 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA +14,user,info,Aug 30 23:00:05,X4711,AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA,AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA, +# important: the following line has a SP at the end of the line! +<14>Aug 30 23:00:05 X4711 +14,user,info,Aug 30 23:00:05,X4711,,, +# and this one NOT +<14>Aug 30 23:00:05 X4711 +14,user,info,Aug 30 23:00:05,X4711,,, +# there is a SP at the end of the line +<14>Aug 30 23:00:05 +14,user,info,Aug 30 23:00:05,localhost,,, +# and here is no SP at the end of the line +<14>Aug 30 23:00:05 +14,user,info,Aug 30 23:00:05,localhost,,, +# unfortunately, I can not test missing dates with this test suite, because +# we would have the current date in the response, which we can not check against +# +# and now the same tests with RFC3339 data - this can make a difference +# as a different date parser is involved. +# +<14>2010-08-30T23:00:05Z X4711 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA +14,user,info,Aug 30 23:00:05,X4711,AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA,AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA, +# important: the following line has a SP at the end of the line! +<14>2010-08-30T23:00:05Z X4711 +14,user,info,Aug 30 23:00:05,X4711,,, +# and this one NOT +<14>2010-08-30T23:00:05Z X4711 +14,user,info,Aug 30 23:00:05,X4711,,, +# there is a SP at the end of the line +<14>2010-08-30T23:00:05Z +14,user,info,Aug 30 23:00:05,localhost,,, +# and here is no SP at the end of the line +<14>2010-08-30T23:00:05Z +14,user,info,Aug 30 23:00:05,localhost,,, diff --git a/tests/testsuites/wr_large.conf b/tests/testsuites/wr_large.conf new file mode 100644 index 00000000..b64f132b --- /dev/null +++ b/tests/testsuites/wr_large.conf @@ -0,0 +1,16 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$MaxMessageSize 10k +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%,%msg:F,58:3%,%msg:F,58:4%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +$OMFileFlushOnTXEnd off +$OMFileFlushInterval 2 +$OMFileIOBufferSize 256k +$IncludeConfig rsyslog.action.1.include +local0.* ?dynfile;outfmt diff --git a/tests/testsuites/x.509/ca-key.pem b/tests/testsuites/x.509/ca-key.pem new file mode 100644 index 00000000..1e1a0b26 --- /dev/null +++ b/tests/testsuites/x.509/ca-key.pem @@ -0,0 +1,15 @@ +-----BEGIN RSA PRIVATE KEY----- +MIICXQIBAAKBgQDZnIJGJH80j2DPBXdxgmFmBRMoBnpwQb8yhRJcJacaWigRAhp4 +wdo07rR+EpuBJHD/5ImIygUwCj/XWAs4JKm3LqK2ih1gUy/s6Tg2O5t3k11kdjEH +MKUxDOLs441dEwERPQtePEoy2POzViIyy959ZJorkdnwC4LBKdQVLEELlwIDAQAB +AoGAEQWvoRoAw1VF3tvQHJZ01Pyno3ViRX63HJYROhkN6b9MrAvsky6iyYo0nzoI +ZQE7P6EaaxNWdYwPs2IlOoaPqeos1sGVDaK/JFuja/DduoXBdCy9RFWRaugDX/1U +iMtjtu29euvegP0r2RIxaIl9dapF5alNH5MLMyBl7XTB+/kCQQDiwHnW8jS1paSc +/risF6Ie5rKuUfVDG8hqMEiKyczSHwUVYushwCclshjM6E1TBFZqMz/8PbFW51pK +OzFS2s6/AkEA9a4044RL3AWe37LIU4hbz2Y+auRvPh8x4i2cWLzdok8Rc1EHDGLN +eHBoOQ3Q2nQS94cOx6HxpRztzBgiwpTRKQJADX9BgV7nbkyO0N2EppG9j7NRvXiZ +bcYwlsmK99/tNjCsf8pkjpy+d8rzGPdW6vMeJbIpQ910OeUJhdOiKvllRwJBAIw3 +rP/dVd5xZseNpj/mp1+rnxwq3EK8UyAfoAgVYvlr3y3NpRQwn8yJezJ07CqB7QFR +F+JgTyZJaH7/l3cusGECQQCM3HmkADAKxX6RwKe8X0Kj/36rjXEMNoq0ZdXOB7Qz +f5N6og4Da9y/ZO+XMo6P3XR/TYIYrMD8nuoR33X69kb1 +-----END RSA PRIVATE KEY----- diff --git a/tests/testsuites/x.509/ca.pem b/tests/testsuites/x.509/ca.pem new file mode 100644 index 00000000..a733eb86 --- /dev/null +++ b/tests/testsuites/x.509/ca.pem @@ -0,0 +1,17 @@ +-----BEGIN CERTIFICATE----- +MIICyzCCAjagAwIBAgIESFo2XjALBgkqhkiG9w0BAQUwezELMAkGA1UEBhMCVVMx +EDAOBgNVBAoTB1NvbWVPcmcxDzANBgNVBAsTBlNvbWVPVTESMBAGA1UEBxMJU29t +ZXdoZXJlMQswCQYDVQQIEwJDQTEoMCYGA1UEAxMfc29tZU5hbWUgKG5vdCBuZWNl +c3NhcmlseSBETlMhKTAeFw0wODA2MTkxMDM1MTJaFw0xODA2MTcxMDM1MjVaMHsx +CzAJBgNVBAYTAlVTMRAwDgYDVQQKEwdTb21lT3JnMQ8wDQYDVQQLEwZTb21lT1Ux +EjAQBgNVBAcTCVNvbWV3aGVyZTELMAkGA1UECBMCQ0ExKDAmBgNVBAMTH3NvbWVO +YW1lIChub3QgbmVjZXNzYXJpbHkgRE5TISkwgZwwCwYJKoZIhvcNAQEBA4GMADCB +iAKBgNmcgkYkfzSPYM8Fd3GCYWYFEygGenBBvzKFElwlpxpaKBECGnjB2jTutH4S +m4EkcP/kiYjKBTAKP9dYCzgkqbcuoraKHWBTL+zpODY7m3eTXWR2MQcwpTEM4uzj +jV0TARE9C148SjLY87NWIjLL3n1kmiuR2fALgsEp1BUsQQuXAgMBAAGjYzBhMA8G +A1UdEwEB/wQFMAMBAf8wHgYDVR0RBBcwFYETc29tZW9uZUBleGFtcGxlLm5ldDAP +BgNVHQ8BAf8EBQMDBwQAMB0GA1UdDgQWBBT7/paNEKc65bcNe0NIhsj4cpl7iTAL +BgkqhkiG9w0BAQUDgYEAlv9ge8Koways837OLoZIam0s7wQCcwd9rWE05caps7BU +T4bfgab9U/e9mmrf3V/zXmtU6y8hhTXF5AcZv3/EmCVwsPRotgrJ+rHXTv5e2PO7 +/8C3K2Lhc89gF4qf4xZwlZU70RasKgCzZa5ivS2Y8pW6LUu6eqqgVw3pPJbW3TE= +-----END CERTIFICATE----- diff --git a/tests/testsuites/x.509/client-cert.pem b/tests/testsuites/x.509/client-cert.pem new file mode 100644 index 00000000..5bf39f81 --- /dev/null +++ b/tests/testsuites/x.509/client-cert.pem @@ -0,0 +1,16 @@ +-----BEGIN CERTIFICATE----- +MIICijCCAfWgAwIBAgIESFo7ITALBgkqhkiG9w0BAQUwezELMAkGA1UEBhMCVVMx +EDAOBgNVBAoTB1NvbWVPcmcxDzANBgNVBAsTBlNvbWVPVTESMBAGA1UEBxMJU29t +ZXdoZXJlMQswCQYDVQQIEwJDQTEoMCYGA1UEAxMfc29tZU5hbWUgKG5vdCBuZWNl +c3NhcmlseSBETlMhKTAeFw0wODA2MTkxMDU1MzJaFw0xMTAzMTYxMDU1MzlaMA0x +CzAJBgNVBAYTAlVTMIGcMAsGCSqGSIb3DQEBAQOBjAAwgYgCgYC+f6yCet2WJgmw +tgukOReI+avRHOfr2hLhIQkSzCOiNi0tNWMKmaQWw/D+y1FvLRq0wLDUyJK/36rB +67HKfscoNeClKTS8jhAs1mPjT57iyuoqK6VW/d2JoofklRCgDIZQrNfxHiOO+kN3 +ShLmkGqxkA3YyUty/JmF6PKWYIhQWQIDAQABo4GPMIGMMAwGA1UdEwEB/wQCMAAw +HQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMB0GA1UdEQQWMBSCEmNsaWVu +dC5leGFtcGxlLm5ldDAdBgNVHQ4EFgQUrDcwsuOF4RiHn0eboCplJSiUhfcwHwYD +VR0jBBgwFoAU+/6WjRCnOuW3DXtDSIbI+HKZe4kwCwYJKoZIhvcNAQEFA4GBAAAh +niy9ORW2AIb6lk/sa3iYczeYpGzxDM9bLZ1xSoIdoHM/v9gPG/WpAZ4ECHjx+Yk8 +4B/9gvaAmMi0FmcoIBQaEOe2P8tcIuzmum3N2F27F2+J4httiNDLJoseWVnXJUvS +dPyVOrKXdl5vVFpmViI5P+VzzMqbAQ6oNlMXIh6e +-----END CERTIFICATE----- diff --git a/tests/testsuites/x.509/client-key.pem b/tests/testsuites/x.509/client-key.pem new file mode 100644 index 00000000..05641213 --- /dev/null +++ b/tests/testsuites/x.509/client-key.pem @@ -0,0 +1,15 @@ +-----BEGIN RSA PRIVATE KEY----- +MIICWwIBAAKBgQC+f6yCet2WJgmwtgukOReI+avRHOfr2hLhIQkSzCOiNi0tNWMK +maQWw/D+y1FvLRq0wLDUyJK/36rB67HKfscoNeClKTS8jhAs1mPjT57iyuoqK6VW +/d2JoofklRCgDIZQrNfxHiOO+kN3ShLmkGqxkA3YyUty/JmF6PKWYIhQWQIDAQAB +AoGAVxrM+BqTIJlC/Ay5lP1QAB9di3ACserUkCFJY1F5h63rCU1sfIfVKl2s3+x6 +z3GZ0QV8tccCpv5wN1x8vqEqkbOvddM3rzpGkEC5PoyfCzuQBun1wnHK/JKjrfk5 +PvcaP60eTNjHZC7w78gOJJCzgzsEMrndtE+55diPmqGVtXMCQQDTZBy5WK8gZwMO +rRz1BKKyBeMYMfTJoJafGfxp0H8AUbTa0V2eb+el3kuzPCm3FQ6IgaHyGj2WqkAw +M0bfAfdXAkEA5rLna1t+2SCtgSd1DotndA4EsH4skBq9kFeD2/8T6Pf13zmBOq6O +4aNEOhgBE/R9/MI4XoU9MbOlkZvKvDuXzwJADdWSb6rXIza6o34+0+Yuw5nRB+dV +DtD8qoLn2wDzHtE6Fcv35YOLVHac26kHTd0J63MYZyDCgRa5Rq5EaBnX1wJAQYRF +XKPbXmZ9X9SI1dyZQMhKZKUwmqw9caSo+e1zBhKFbSOzo6q3QTVQxv7SL4ybyxCN +WaqVOmw+dR+9b7+s2QJAdNAw3r418rWKFKJJNTSqSqr1sYqiKvrQL6w2dpdpAeY4 +3VDCz/7/F9AEn3R7K3fZLQ7W6M62LSEjxxc1Y3LIpQ== +-----END RSA PRIVATE KEY----- diff --git a/tests/testsuites/x.509/machine-cert.pem b/tests/testsuites/x.509/machine-cert.pem new file mode 100644 index 00000000..fa2fd36e --- /dev/null +++ b/tests/testsuites/x.509/machine-cert.pem @@ -0,0 +1,18 @@ +-----BEGIN CERTIFICATE----- +MIIC7TCCAligAwIBAgIESFo4GTALBgkqhkiG9w0BAQUwezELMAkGA1UEBhMCVVMx +EDAOBgNVBAoTB1NvbWVPcmcxDzANBgNVBAsTBlNvbWVPVTESMBAGA1UEBxMJU29t +ZXdoZXJlMQswCQYDVQQIEwJDQTEoMCYGA1UEAxMfc29tZU5hbWUgKG5vdCBuZWNl +c3NhcmlseSBETlMhKTAeFw0wODA2MTkxMDQyNTRaFw0xMTAzMTYxMDQyNTdaMG8x +CzAJBgNVBAYTAlVTMRAwDgYDVQQKEwdTb21lT3JnMQ8wDQYDVQQLEwZTb21lT1Ux +EjAQBgNVBAcTCVNvbWV3aGVyZTELMAkGA1UECBMCQ0ExHDAaBgNVBAMTE21hY2hp +bmUuZXhhbXBsZS5uZXQwgZwwCwYJKoZIhvcNAQEBA4GMADCBiAKBgLJOW6lIHv8u +c6Ez7tiir64vI3aRuDmUACPybyWtyWqrLebzYtg+borWHj9y5di54NB5wpQhZQsQ +U2awNqanzUYeLGqbecbuxuLtsKlZ4knax+PwHOBTmIcN1SjbpII27Toe0VwHE5Vd +sygFFyorto6OeNLPrIcTFfwXQ2sVw325AgMBAAGjgZAwgY0wDAYDVR0TAQH/BAIw +ADAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwHgYDVR0RBBcwFYITbWFj +aGluZS5leGFtcGxlLm5ldDAdBgNVHQ4EFgQUDOHD29GdMfoDWwev4uDvItkLKKww +HwYDVR0jBBgwFoAU+/6WjRCnOuW3DXtDSIbI+HKZe4kwCwYJKoZIhvcNAQEFA4GB +AMt1iED7QzFL2Qk6VivoFY15S2XGF8rJTd3l00bwyLA5qLyLBGlB6z4qkYu7/7SW +5r7tet+1DezgHrj/1eU289m410wnQB8fGwcVLp6OX2PAlhNmVLcsipiN6rielAcP +aIg/VlBtoCFp/ymTLKgvh6DLKWhRUkFPqO2WtcQ3UUo+ +-----END CERTIFICATE----- diff --git a/tests/testsuites/x.509/machine-key.pem b/tests/testsuites/x.509/machine-key.pem new file mode 100644 index 00000000..808f00c9 --- /dev/null +++ b/tests/testsuites/x.509/machine-key.pem @@ -0,0 +1,15 @@ +-----BEGIN RSA PRIVATE KEY----- +MIICXAIBAAKBgQCyTlupSB7/LnOhM+7Yoq+uLyN2kbg5lAAj8m8lrclqqy3m82LY +Pm6K1h4/cuXYueDQecKUIWULEFNmsDamp81GHixqm3nG7sbi7bCpWeJJ2sfj8Bzg +U5iHDdUo26SCNu06HtFcBxOVXbMoBRcqK7aOjnjSz6yHExX8F0NrFcN9uQIDAQAB +AoGABHJs2c95Km8bpikX62I/VG5LiaD/wbvdtwfMWtm3PMhRKEHotLD169OERJvW +fK3CHCD1R+F/ViPNmLGLY2Oq/GqKjhKjg4sqAznw8TImBSgXCFho4sl38z+luP1o +TXFDgfV5HDDW1/F5kJlBIfXBLFdl4VO7E0ZnFt4FqSDRW2MCQQDRun/sBGM4i9hM +QdC+QwrdcgCScBpzbz4YXtI9TyGEqNahg8kXgIVUbzDdRmG68G2M98USzRs5DWB7 +YvYwmRoPAkEA2aUdUpFRb/n7XfsAiFLYOk96C82iCCQpJi0si34zlCAEbCRbQ6zw +gVDMCMSccnnWrVzqtxfN+rXycFTNyDFTtwJAPRwymfrNTnSxGcczo7y1NcE6GXFA +w9HuLfuzFtov0g/AOl/EAG0abHfZrSAM6gOUaDbp3YiWHhGfw1QamB6EUQJAClTb +MnsxeXZNZ2Wt3crI9uOk8IB/a5GD3osQbUK9Yg+vBg8nweuoswrJ1LS4lHqSJUKe +5bgckAUpEAoGhrVIuwJBAKIuqx/cSjF4Oa9xT6DzBRe7vAlKFq62lUV5SLfoSEgY +L5dvPBgAD0Styglny1s0Bu5FTlkxlFOMvUAD/O5hsQw= +-----END RSA PRIVATE KEY----- diff --git a/tests/testsuites/x.509/request.pem b/tests/testsuites/x.509/request.pem new file mode 100644 index 00000000..c612325c --- /dev/null +++ b/tests/testsuites/x.509/request.pem @@ -0,0 +1,10 @@ +-----BEGIN NEW CERTIFICATE REQUEST----- +MIIBWDCBxAIBADANMQswCQYDVQQGEwJVUzCBnDALBgkqhkiG9w0BAQEDgYwAMIGI +AoGAvn+sgnrdliYJsLYLpDkXiPmr0Rzn69oS4SEJEswjojYtLTVjCpmkFsPw/stR +by0atMCw1MiSv9+qweuxyn7HKDXgpSk0vI4QLNZj40+e4srqKiulVv3diaKH5JUQ +oAyGUKzX8R4jjvpDd0oS5pBqsZAN2MlLcvyZhejylmCIUFkCAwEAAaARMA8GCSqG +SIb3DQEJBzECEwAwCwYJKoZIhvcNAQEFA4GBAA6mBaHFuRvcJVNoU7wDFcDexjvC +QLpDpFRSbKcKdNEQLBRD8ZNVOY4WBXQE2pE84//QnygQPKPCHSqUVdPPBabi5y2E +A2XvgYyKsrFbsrpKrVkPz5oQB4V7FRytQaQoBi//BSOu3dMaimLcAhfNQZCrQeu8 +SYWdJi5OPvrYGvgT +-----END NEW CERTIFICATE REQUEST----- diff --git a/tests/threadingmq.sh b/tests/threadingmq.sh new file mode 100755 index 00000000..bdb5f35e --- /dev/null +++ b/tests/threadingmq.sh @@ -0,0 +1,16 @@ +# test many concurrent tcp connections +# we send 100,000 messages in the hopes that his puts at least a little bit +# of pressure on the threading subsystem. To really prove it, we would need to +# push messages for several minutes, but that takes too long during the +# automatted tests (hint: do this manually after suspect changes). Thankfully, +# in practice many threading bugs result in an abort rather quickly and these +# should be covered by this test here. +# rgerhards, 2009-06-26 +echo TEST: threadingmq.sh - main queue concurrency +source $srcdir/diag.sh init +source $srcdir/diag.sh startup threadingmq.conf +source $srcdir/diag.sh injectmsg 0 100000 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 99999 +source $srcdir/diag.sh exit diff --git a/tests/threadingmqaq.sh b/tests/threadingmqaq.sh new file mode 100755 index 00000000..b7764821 --- /dev/null +++ b/tests/threadingmqaq.sh @@ -0,0 +1,18 @@ +# test many concurrent tcp connections +# we send 100,000 messages in the hopes that his puts at least a little bit +# of pressure on the threading subsystem. To really prove it, we would need to +# push messages for several minutes, but that takes too long during the +# automatted tests (hint: do this manually after suspect changes). Thankfully, +# in practice many threading bugs result in an abort rather quickly and these +# should be covered by this test here. +# rgerhards, 2009-06-26 +echo TEST: threadingmqaq.sh - main/action queue concurrency +source $srcdir/diag.sh init +source $srcdir/diag.sh startup threadingmqaq.conf +#source $srcdir/diag.sh tcpflood -c2 -m100000 +#source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh injectmsg 0 100000 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 99999 +source $srcdir/diag.sh exit diff --git a/tests/timestamp.sh b/tests/timestamp.sh new file mode 100755 index 00000000..7699a4af --- /dev/null +++ b/tests/timestamp.sh @@ -0,0 +1,13 @@ +echo various timestamp tests +source $srcdir/diag.sh init +source $srcdir/diag.sh nettester ts3164 udp +source $srcdir/diag.sh nettester ts3164 tcp +source $srcdir/diag.sh nettester ts3339 udp +source $srcdir/diag.sh nettester ts3339 tcp +source $srcdir/diag.sh nettester tsmysql udp +source $srcdir/diag.sh nettester tsmysql tcp +source $srcdir/diag.sh nettester tspgsql udp +source $srcdir/diag.sh nettester tspgsql tcp +source $srcdir/diag.sh nettester subsecond udp +source $srcdir/diag.sh nettester subsecond tcp +source $srcdir/diag.sh init diff --git a/tests/validation-run.sh b/tests/validation-run.sh index 10981290..2e922283 100755 --- a/tests/validation-run.sh +++ b/tests/validation-run.sh @@ -22,17 +22,18 @@ # A copy of the GPL can be found in the file "COPYING" in this distribution. #set -x echo "testing a failed configuration verification run" -../tools/rsyslogd -u2 -c3 -N1 -f$srcdir/testsuites/invalid.conf +../tools/rsyslogd -dn -u2 -c4 -N1 -f$srcdir/testsuites/invalid.conf -M../runtime/.libs:../.libs if [ $? -ne 1 ]; then exit 1 fi echo testing a valid config verification run -../tools/rsyslogd -u2 -c3 -N1 -f$srcdir/testsuites/valid.conf +../tools/rsyslogd -u2 -c4 -N1 -f$srcdir/testsuites/valid.conf -M../runtime/.libs:../.libs if [ $? -ne 0 ]; then exit 1 fi echo testing empty config file -../tools/rsyslogd -u2 -c3 -N1 -f/dev/null +../tools/rsyslogd -u2 -c4 -N1 -f/dev/null -M../runtime/.libs:../.libs if [ $? -ne 1 ]; then exit 1 fi +echo SUCCESS: validation run tests diff --git a/tests/wr_large.sh b/tests/wr_large.sh new file mode 100755 index 00000000..84f12989 --- /dev/null +++ b/tests/wr_large.sh @@ -0,0 +1,16 @@ +# This tests async writing large data records. We use up to 10K +# record size. + +# added 2010-03-10 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +cat rsyslog.action.1.include +source $srcdir/diag.sh init +source $srcdir/diag.sh startup wr_large.conf +# send 4000 messages of 10.000bytes plus header max, randomized +source $srcdir/diag.sh tcpflood -m4000 -r -d10000 -P129 +sleep 1 # due to large messages, we need this time for the tcp receiver to settle... +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +source $srcdir/diag.sh seq-check 0 3999 -E +source $srcdir/diag.sh exit diff --git a/tests/wr_large_async.sh b/tests/wr_large_async.sh new file mode 100755 index 00000000..88a1acf8 --- /dev/null +++ b/tests/wr_large_async.sh @@ -0,0 +1,14 @@ +# This tests async writing large data records. We use up to 10K +# record size. + +# added 2010-03-10 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[wr_large_async.sh\]: test for file writing for large message sets +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +echo "\$OMFileAsyncWriting on" > rsyslog.action.1.include +source $srcdir/wr_large.sh diff --git a/tests/wr_large_sync.sh b/tests/wr_large_sync.sh new file mode 100755 index 00000000..a1c4fd77 --- /dev/null +++ b/tests/wr_large_sync.sh @@ -0,0 +1,14 @@ +# This tests async writing large data records. We use up to 10K +# record size. + +# added 2010-03-10 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[wr_large_sync.sh\]: test for file writing for large message sets +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +echo "\$OMFileAsyncWriting off" > rsyslog.action.1.include +source $srcdir/wr_large.sh diff --git a/tools/Makefile.am b/tools/Makefile.am index e523b854..1497d3be 100644 --- a/tools/Makefile.am +++ b/tools/Makefile.am @@ -13,6 +13,8 @@ rsyslogd_SOURCES = \ omfwd.h \ omfile.c \ omfile.h \ + ompipe.c \ + ompipe.h \ omdiscard.c \ omdiscard.h \ iminternal.c \ @@ -26,8 +28,10 @@ rsyslogd_LDADD = $(ZLIB_LIBS) $(PTHREADS_LIBS) $(RSRT_LIBS) $(SOL_LIBS) rsyslogd_LDFLAGS = -export-dynamic if ENABLE_DIAGTOOLS -sbin_PROGRAMS += rsyslog_diag_hostname msggen +sbin_PROGRAMS += rsyslog_diag_hostname msggen zpipe rsyslog_diag_hostname_SOURCES = gethostn.c +zpipe_SOURCES = zpipe.c +zpipe_LDADD = -lz msggen_SOURCES = msggen.c endif diff --git a/tools/msggen.c b/tools/msggen.c index 06244c18..29ade3a7 100644 --- a/tools/msggen.c +++ b/tools/msggen.c @@ -21,6 +21,7 @@ * A copy of the GPL can be found in the file "COPYING" in this distribution. */ +#include "config.h" #include <stdio.h> #include <syslog.h> diff --git a/tools/omfile.c b/tools/omfile.c index 3e845a73..24de052c 100644 --- a/tools/omfile.c +++ b/tools/omfile.c @@ -1,18 +1,22 @@ /* omfile.c * This is the implementation of the build-in file output module. * - * Handles: eTypeCONSOLE, eTypeTTY, eTypeFILE, eTypePIPE - * * NOTE: read comments in module-template.h to understand how this file * works! * * File begun on 2007-07-21 by RGerhards (extracted from syslogd.c) - * This file is under development and has not yet arrived at being fully - * self-contained and a real object. So far, it is mostly an excerpt - * of the "old" message code without any modifications. However, it - * helps to have things at the right place one we go to the meat of it. * - * Copyright 2007, 2008 Rainer Gerhards and Adiscon GmbH. + * A large re-write of this file was done in June, 2009. The focus was + * to introduce many more features (like zipped writing), clean up the code + * and make it more reliable. In short, that rewrite tries to provide a new + * solid basis for the next three to five years to come. During it, bugs + * may have been introduced ;) -- rgerhards, 2009-06-04 + * + * Note that as of 2010-02-28 this module does no longer handle + * pipes. These have been moved to ompipe, to reduced the entanglement + * between the two different functionalities. -- rgerhards + * + * Copyright 2007-2009 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -41,6 +45,7 @@ #include <assert.h> #include <errno.h> #include <ctype.h> +#include <libgen.h> #include <unistd.h> #include <sys/file.h> @@ -48,7 +53,7 @@ # include <fcntl.h> #endif -#include "syslogd.h" +#include "conf.h" #include "syslogd-types.h" #include "srUtils.h" #include "template.h" @@ -57,6 +62,9 @@ #include "cfsysline.h" #include "module-template.h" #include "errmsg.h" +#include "stream.h" +#include "unicode-helper.h" +#include "atomic.h" MODULE_TYPE_OUTPUT @@ -64,17 +72,23 @@ MODULE_TYPE_OUTPUT */ DEF_OMOD_STATIC_DATA DEFobjCurrIf(errmsg) +DEFobjCurrIf(strm) /* The following structure is a dynafile name cache entry. */ struct s_dynaFileCacheEntry { - uchar *pName; /* name currently open, if dynamic name */ - short fd; /* name associated with file name in cache */ + uchar *pName; /* name currently open, if dynamic name */ + strm_t *pStrm; /* our output stream */ time_t lastUsed; /* for LRU - last access */ }; typedef struct s_dynaFileCacheEntry dynaFileCacheEntry; +#define IOBUF_DFLT_SIZE 1024 /* default size for io buffers */ +#define FLUSH_INTRVL_DFLT 1 /* default buffer flush interval (in seconds) */ +#define USE_ASYNCWRITER_DFLT 0 /* default buffer use async writer */ +#define FLUSHONTX_DFLT 1 /* default for flush on TX end */ + /* globals for default values */ static int iDynaFileCacheSize = 10; /* max cache for dynamic files */ static int fCreateMode = 0644; /* mode to use when creating files */ @@ -84,21 +98,21 @@ static uid_t fileUID; /* UID to be used for newly created files */ static uid_t fileGID; /* GID to be used for newly created files */ static uid_t dirUID; /* UID to be used for newly created directories */ static uid_t dirGID; /* GID to be used for newly created directories */ -static int bCreateDirs; /* auto-create directories for dynaFiles: 0 - no, 1 - yes */ +static int bCreateDirs = 1;/* auto-create directories for dynaFiles: 0 - no, 1 - yes */ static int bEnableSync = 0;/* enable syncing of files (no dash in front of pathname in conf): 0 - no, 1 - yes */ -static uchar *pszTplName = NULL; /* name of the default template to use */ +static int iZipLevel = 0; /* zip compression mode (0..9 as usual) */ +static bool bFlushOnTXEnd = FLUSHONTX_DFLT;/* flush write buffers when transaction has ended? */ +static int64 iIOBufSize = IOBUF_DFLT_SIZE; /* size of an io buffer */ +static int iFlushInterval = FLUSH_INTRVL_DFLT; /* how often flush the output buffer on inactivity? */ +static int bUseAsyncWriter = USE_ASYNCWRITER_DFLT; /* should we enable asynchronous writing? */ +uchar *pszFileDfltTplName = NULL; /* name of the default template to use */ /* end globals for default values */ typedef struct _instanceData { uchar f_fname[MAXFNAME];/* file or template name (display only) */ - short fd; /* file descriptor for (current) file */ - enum { - eTypeFILE, - eTypeTTY, - eTypeCONSOLE, - eTypePIPE - } fileType; + strm_t *pStrm; /* our output stream */ + strmType_t strmType; /* stream type, used for named pipes */ char bDynamicName; /* 0 - static name, 1 - dynamic name (with properties) */ int fCreateMode; /* file creation mode for open() */ int fDirCreateMode; /* creation mode for mkdir() */ @@ -117,8 +131,13 @@ typedef struct _instanceData { * pointer points to the overall structure. */ dynaFileCacheEntry **dynCache; - off_t f_sizeLimit; /* file size limit, 0 = no limit */ - char *f_sizeLimitCmd; /* command to carry out when size limit is reached */ + off_t iSizeLimit; /* file size limit, 0 = no limit */ + uchar *pszSizeLimitCmd; /* command to carry out when size limit is reached */ + int iZipLevel; /* zip mode to use for this selector */ + int iIOBufSize; /* size of associated io buffer */ + int iFlushInterval; /* how fast flush buffer on inactivity? */ + bool bFlushOnTXEnd; /* flush write buffers when transaction has ended? */ + bool bUseAsyncWriter; /* use async stream writer? */ } instanceData; @@ -132,24 +151,23 @@ ENDisCompatibleWithFeature BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo if(pData->bDynamicName) { - dbgprintf("[dynamic]\n\ttemplate='%s'" - "\tfile cache size=%d\n" - "\tcreate directories: %s\n" - "\tfile owner %d, group %d\n" - "\tdirectory owner %d, group %d\n" - "\tfail if owner/group can not be set: %s\n", - pData->f_fname, - pData->iDynaFileCacheSize, - pData->bCreateDirs ? "yes" : "no", - pData->fileUID, pData->fileGID, - pData->dirUID, pData->dirGID, - pData->bFailOnChown ? "yes" : "no" - ); + dbgprintf("[dynamic]\n"); } else { /* regular file */ - dbgprintf("%s", pData->f_fname); - if (pData->fd == -1) - dbgprintf(" (unused)"); + dbgprintf("%s%s\n", pData->f_fname, + (pData->pStrm == NULL) ? " (unused)" : ""); } + + dbgprintf("\ttemplate='%s'\n", pData->f_fname); + dbgprintf("\tuse async writer=%d\n", pData->bUseAsyncWriter); + dbgprintf("\tflush on TX end=%d\n", pData->bFlushOnTXEnd); + dbgprintf("\tflush interval=%d\n", pData->iFlushInterval); + dbgprintf("\tfile cache size=%d\n", pData->iDynaFileCacheSize); + dbgprintf("\tcreate directories: %s\n", pData->bCreateDirs ? "yes" : "no"); + dbgprintf("\tfile owner %d, group %d\n", pData->fileUID, pData->fileGID); + dbgprintf("\tdirectory owner %d, group %d\n", pData->dirUID, pData->dirGID); + dbgprintf("\tdir create mode 0%3.3o, file create mode 0%3.3o\n", + pData->fDirCreateMode, pData->fCreateMode); + dbgprintf("\tfail if owner/group can not be set: %s\n", pData->bFailOnChown ? "yes" : "no"); ENDdbgPrintInstInfo @@ -168,13 +186,13 @@ rsRetVal setDynaFileCacheSize(void __attribute__((unused)) *pVal, int iNewVal) errmsg.LogError(0, RS_RET_VAL_OUT_OF_RANGE, "%s", errMsg); iRet = RS_RET_VAL_OUT_OF_RANGE; iNewVal = 1; - } else if(iNewVal > 10000) { + } else if(iNewVal > 1000) { snprintf((char*) errMsg, sizeof(errMsg)/sizeof(uchar), - "DynaFileCacheSize maximum is 10,000 (%d given), changed to 10,000.", iNewVal); + "DynaFileCacheSize maximum is 1,000 (%d given), changed to 1,000.", iNewVal); errno = 0; errmsg.LogError(0, RS_RET_VAL_OUT_OF_RANGE, "%s", errMsg); iRet = RS_RET_VAL_OUT_OF_RANGE; - iNewVal = 10000; + iNewVal = 1000; } iDynaFileCacheSize = iNewVal; @@ -199,14 +217,6 @@ static rsRetVal cflineParseOutchannel(instanceData *pData, uchar* p, omodStringR struct outchannel *pOch; char szBuf[128]; /* should be more than sufficient */ - /* this must always be a file, because we can not set a size limit - * on a pipe... - * rgerhards 2005-06-21: later, this will be a separate type, but let's - * emulate things for the time being. When everything runs, we can - * extend it... - */ - pData->fileType = eTypeFILE; - ++p; /* skip '$' */ i = 0; /* get outchannel name */ @@ -241,79 +251,21 @@ static rsRetVal cflineParseOutchannel(instanceData *pData, uchar* p, omodStringR } /* OK, we finally got a correct template. So let's use it... */ - strncpy((char*) pData->f_fname, (char*) pOch->pszFileTemplate, MAXFNAME); - pData->f_sizeLimit = pOch->uSizeLimit; + ustrncpy(pData->f_fname, pOch->pszFileTemplate, MAXFNAME); + pData->iSizeLimit = pOch->uSizeLimit; /* WARNING: It is dangerous "just" to pass the pointer. As we * never rebuild the output channel description, this is acceptable here. */ - pData->f_sizeLimitCmd = (char*) pOch->cmdOnSizeLimit; + pData->pszSizeLimitCmd = pOch->cmdOnSizeLimit; iRet = cflineParseTemplateName(&p, pOMSR, iEntry, iTplOpts, - (pszTplName == NULL) ? (uchar*)"RSYSLOG_FileFormat" : pszTplName); + (pszFileDfltTplName == NULL) ? (uchar*)"RSYSLOG_FileFormat" : pszFileDfltTplName); finalize_it: RETiRet; } -/* rgerhards 2005-06-21: Try to resolve a size limit - * situation. This first runs the command, and then - * checks if we are still above the treshold. - * returns 0 if ok, 1 otherwise - * TODO: consider moving the initial check in here, too - */ -int resolveFileSizeLimit(instanceData *pData) -{ - uchar *pParams; - uchar *pCmd; - uchar *p; - off_t actualFileSize; - ASSERT(pData != NULL); - - if(pData->f_sizeLimitCmd == NULL) - return 1; /* nothing we can do in this case... */ - - /* the execProg() below is probably not great, but at least is is - * fairly secure now. Once we change the way file size limits are - * handled, we should also revisit how this command is run (and - * with which parameters). rgerhards, 2007-07-20 - */ - /* we first check if we have command line parameters. We assume this, - * when we have a space in the program name. If we find it, everything after - * the space is treated as a single argument. - */ - if((pCmd = (uchar*)strdup((char*)pData->f_sizeLimitCmd)) == NULL) { - /* there is not much we can do - we make syslogd close the file in this case */ - return 1; - } - - for(p = pCmd ; *p && *p != ' ' ; ++p) { - /* JUST SKIP */ - } - - if(*p == ' ') { - *p = '\0'; /* pretend string-end */ - pParams = p+1; - } else - pParams = NULL; - - execProg(pCmd, 1, pParams); - - free(pCmd); - - pData->fd = open((char*) pData->f_fname, O_WRONLY|O_APPEND|O_CREAT|O_NOCTTY|O_CLOEXEC, - pData->fCreateMode); - - actualFileSize = lseek(pData->fd, 0, SEEK_END); - if(actualFileSize >= pData->f_sizeLimit) { - /* OK, it didn't work out... */ - return 1; - } - - return 0; -} - - /* This function deletes an entry from the dynamic file name * cache. A pointer to the cache must be passed in as well * as the index of the to-be-deleted entry. This index may @@ -321,34 +273,36 @@ int resolveFileSizeLimit(instanceData *pData) * function immediately returns. Parameter bFreeEntry is 1 * if the entry should be d_free()ed and 0 if not. */ -static void dynaFileDelCacheEntry(dynaFileCacheEntry **pCache, int iEntry, int bFreeEntry) +static rsRetVal +dynaFileDelCacheEntry(dynaFileCacheEntry **pCache, int iEntry, int bFreeEntry) { + DEFiRet; ASSERT(pCache != NULL); - BEGINfunc; - if(pCache[iEntry] == NULL) FINALIZE; DBGPRINTF("Removed entry %d for file '%s' from dynaCache.\n", iEntry, - pCache[iEntry]->pName == NULL ? "[OPEN FAILED]" : (char*)pCache[iEntry]->pName); - /* if the name is NULL, this is an improperly initilized entry which - * needs to be discarded. In this case, neither the file is to be closed - * not the name to be freed. - */ + pCache[iEntry]->pName == NULL ? UCHAR_CONSTANT("[OPEN FAILED]") : pCache[iEntry]->pName); + if(pCache[iEntry]->pName != NULL) { - close(pCache[iEntry]->fd); d_free(pCache[iEntry]->pName); pCache[iEntry]->pName = NULL; } + if(pCache[iEntry]->pStrm != NULL) { + strm.Destruct(&pCache[iEntry]->pStrm); + if(pCache[iEntry]->pStrm != NULL) /* safety check -- TODO: remove if no longer necessary */ + abort(); + } + if(bFreeEntry) { d_free(pCache[iEntry]); pCache[iEntry] = NULL; } finalize_it: - ENDfunc; + RETiRet; } @@ -356,7 +310,8 @@ finalize_it: * relevant files. Part of Shutdown and HUP processing. * rgerhards, 2008-10-23 */ -static inline void dynaFileFreeCacheEntries(instanceData *pData) +static inline void +dynaFileFreeCacheEntries(instanceData *pData) { register int i; ASSERT(pData != NULL); @@ -365,6 +320,7 @@ static inline void dynaFileFreeCacheEntries(instanceData *pData) for(i = 0 ; i < pData->iCurrCacheSize ; ++i) { dynaFileDelCacheEntry(pData->dynCache, i, 1); } + pData->iCurrElt = -1; /* invalidate current element */ ENDfunc; } @@ -392,25 +348,18 @@ static void dynaFileFreeCache(instanceData *pData) static rsRetVal prepareFile(instanceData *pData, uchar *newFileName) { + int fd; DEFiRet; - if(pData->fileType == eTypePIPE) { - pData->fd = open((char*) pData->f_fname, O_RDWR|O_NONBLOCK|O_CLOEXEC); - FINALIZE; /* we are done in this case */ - } - if(access((char*)newFileName, F_OK) == 0) { - /* file already exists */ - pData->fd = open((char*) newFileName, O_WRONLY|O_APPEND|O_CREAT|O_NOCTTY|O_CLOEXEC, - pData->fCreateMode); - } else { - pData->fd = -1; + if(access((char*)newFileName, F_OK) != 0) { /* file does not exist, create it (and eventually parent directories */ + fd = -1; if(pData->bCreateDirs) { /* We first need to create parent dirs if they are missing. * We do not report any errors here ourselfs but let the code * fall through to error handler below. */ - if(makeFileParentDirs(newFileName, strlen((char*)newFileName), + if(makeFileParentDirs(newFileName, ustrlen(newFileName), pData->fDirCreateMode, pData->dirUID, pData->dirGID, pData->bFailOnChown) != 0) { ABORT_FINALIZE(RS_RET_ERR); /* we give up */ @@ -419,17 +368,17 @@ prepareFile(instanceData *pData, uchar *newFileName) /* no matter if we needed to create directories or not, we now try to create * the file. -- rgerhards, 2008-12-18 (based on patch from William Tisater) */ - pData->fd = open((char*) newFileName, O_WRONLY|O_APPEND|O_CREAT|O_NOCTTY|O_CLOEXEC, + fd = open((char*) newFileName, O_WRONLY|O_APPEND|O_CREAT|O_NOCTTY|O_CLOEXEC, pData->fCreateMode); - if(pData->fd != -1) { + if(fd != -1) { /* check and set uid/gid */ if(pData->fileUID != (uid_t)-1 || pData->fileGID != (gid_t) -1) { /* we need to set owner/group */ - if(fchown(pData->fd, pData->fileUID, pData->fileGID) != 0) { + if(fchown(fd, pData->fileUID, pData->fileGID) != 0) { if(pData->bFailOnChown) { int eSave = errno; - close(pData->fd); - pData->fd = -1; + close(fd); + fd = -1; errno = eSave; } /* we will silently ignore the chown() failure @@ -437,19 +386,42 @@ prepareFile(instanceData *pData, uchar *newFileName) */ } } + close(fd); /* close again, as we need a stream further on */ } } -finalize_it: - /* this was "pData->fd != 0", which I think was a bug. I guess 0 was intended to mean - * non-open file descriptor. Anyhow, I leave this comment for the time being to that if - * problems surface, one at least knows what happened. -- rgerhards, 2009-03-19 - */ - if(pData->fd != -1 && isatty(pData->fd)) { - DBGPRINTF("file %d is a tty file\n", pData->fd); - pData->fileType = eTypeTTY; - untty(); - } + /* the copies below are clumpsy, but there is no way around given the + * anomalies in dirname() and basename() [they MODIFY the provided buffer...] + */ + uchar szNameBuf[MAXFNAME]; + uchar szDirName[MAXFNAME]; + uchar szBaseName[MAXFNAME]; + ustrncpy(szNameBuf, newFileName, MAXFNAME); + ustrncpy(szDirName, (uchar*)dirname((char*)szNameBuf), MAXFNAME); + ustrncpy(szNameBuf, newFileName, MAXFNAME); + ustrncpy(szBaseName, (uchar*)basename((char*)szNameBuf), MAXFNAME); + + CHKiRet(strm.Construct(&pData->pStrm)); + CHKiRet(strm.SetFName(pData->pStrm, szBaseName, ustrlen(szBaseName))); + CHKiRet(strm.SetDir(pData->pStrm, szDirName, ustrlen(szDirName))); + CHKiRet(strm.SetiZipLevel(pData->pStrm, pData->iZipLevel)); + CHKiRet(strm.SetsIOBufSize(pData->pStrm, (size_t) pData->iIOBufSize)); + CHKiRet(strm.SettOperationsMode(pData->pStrm, STREAMMODE_WRITE_APPEND)); + CHKiRet(strm.SettOpenMode(pData->pStrm, fCreateMode)); + CHKiRet(strm.SetbSync(pData->pStrm, pData->bSyncFile)); + CHKiRet(strm.SetsType(pData->pStrm, pData->strmType)); + CHKiRet(strm.SetiSizeLimit(pData->pStrm, pData->iSizeLimit)); + /* set the flush interval only if we actually use it - otherwise it will activate + * async processing, which is a real performance waste if we do not do buffered + * writes! -- rgerhards, 2009-07-06 + */ + if(pData->bUseAsyncWriter) + CHKiRet(strm.SetiFlushInterval(pData->pStrm, pData->iFlushInterval)); + if(pData->pszSizeLimitCmd != NULL) + CHKiRet(strm.SetpszSizeLimitCmd(pData->pStrm, ustrdup(pData->pszSizeLimitCmd))); + CHKiRet(strm.ConstructFinalize(pData->pStrm)); + +finalize_it: RETiRet; } @@ -462,15 +434,16 @@ finalize_it: * be written. * This is a helper to writeFile(). rgerhards, 2007-07-03 */ -static int prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) +static inline rsRetVal +prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) { time_t ttOldest; /* timestamp of oldest element */ int iOldest; int i; int iFirstFree; + rsRetVal localRet; dynaFileCacheEntry **pCache; - - BEGINfunc + DEFiRet; ASSERT(pData != NULL); ASSERT(newFileName != NULL); @@ -481,29 +454,31 @@ static int prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsg * I *hope* this will be a performance enhancement. */ if( (pData->iCurrElt != -1) - && !strcmp((char*) newFileName, (char*) pCache[pData->iCurrElt]->pName)) { + && !ustrcmp(newFileName, pCache[pData->iCurrElt]->pName)) { /* great, we are all set */ - pCache[pData->iCurrElt]->lastUsed = time(NULL); /* update timestamp for LRU */ - return 0; + pCache[pData->iCurrElt]->lastUsed = time(NULL); /* update timestamp for LRU */ // TODO: optimize time call! + // LRU needs only a strictly monotonically increasing counter, so such a one could do + FINALIZE; } /* ok, no luck. Now let's search the table if we find a matching spot. * While doing so, we also prepare for creation of a new one. */ + pData->iCurrElt = -1; /* invalid current element pointer */ iFirstFree = -1; /* not yet found */ iOldest = 0; /* we assume the first element to be the oldest - that will change as we loop */ ttOldest = time(NULL) + 1; /* there must always be an older one */ for(i = 0 ; i < pData->iCurrCacheSize ; ++i) { - if(pCache[i] == NULL) { + if(pCache[i] == NULL || pCache[i]->pName == NULL) { if(iFirstFree == -1) iFirstFree = i; } else { /* got an element, let's see if it matches */ - if(!strcmp((char*) newFileName, (char*) pCache[i]->pName)) { + if(!ustrcmp(newFileName, pCache[i]->pName)) { // RG: name == NULL? /* we found our element! */ - pData->fd = pCache[i]->fd; + pData->pStrm = pCache[i]->pStrm; pData->iCurrElt = i; pCache[i]->lastUsed = time(NULL); /* update timestamp for LRU */ - return 0; + FINALIZE; } /* did not find it - so lets keep track of the counters for LRU */ if(pCache[i]->lastUsed < ttOldest) { @@ -514,28 +489,46 @@ static int prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsg } /* we have not found an entry */ + + /* invalidate iCurrElt as we may error-exit out of this function when the currrent + * iCurrElt has been freed or otherwise become unusable. This is a precaution, and + * performance-wise it may be better to do that in each of the exits. However, that + * is error-prone, so I prefer to do it here. -- rgerhards, 2010-03-02 + */ + pData->iCurrElt = -1; + /* similarly, we need to set the current pStrm to NULL, because otherwise, if prepareFile() fails, + * we may end up using an old stream. This bug depends on how exactly prepareFile fails, + * but it* could be triggered in the common case of a failed open() system call. + * rgerhards, 2010-03-22 + */ + pData->pStrm = NULL; + if(iFirstFree == -1 && (pData->iCurrCacheSize < pData->iDynaFileCacheSize)) { /* there is space left, so set it to that index */ iFirstFree = pData->iCurrCacheSize++; } +// RG: this is the begin of a potential problem area + /* Note that the following code sequence does not work with the cache entry itself, + * but rather with pData->pStrm, the (sole) stream pointer in the non-dynafile case. + * The cache array is only updated after the open was successful. -- rgerhards, 2010-03-21 + */ if(iFirstFree == -1) { dynaFileDelCacheEntry(pCache, iOldest, 0); iFirstFree = iOldest; /* this one *is* now free ;) */ } else { /* we need to allocate memory for the cache structure */ - pCache[iFirstFree] = (dynaFileCacheEntry*) calloc(1, sizeof(dynaFileCacheEntry)); - if(pCache[iFirstFree] == NULL) { - DBGPRINTF("prepareDynfile(): could not alloc mem, discarding this request\n"); - return -1; - } + /* TODO: performance note: we could alloc all entries on startup, thus saving malloc + * overhead -- this may be something to consider in v5... + */ + CHKmalloc(pCache[iFirstFree] = (dynaFileCacheEntry*) calloc(1, sizeof(dynaFileCacheEntry))); } /* Ok, we finally can open the file */ - prepareFile(pData, newFileName); /* ignore exact error, we check fd below */ + localRet = prepareFile(pData, newFileName); /* ignore exact error, we check fd below */ - /* file is either open now or an error state set */ - if(pData->fd == -1) { + /* file is either open now or an error state set */ // RG: better check localRet? + if(pData->pStrm == NULL) { /* do not report anything if the message is an internally-generated * message. Otherwise, we could run into a never-ending loop. The bad * news is that we also lose errors on startup messages, but so it is. @@ -543,22 +536,45 @@ static int prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsg if(iMsgOpts & INTERNAL_MSG) { DBGPRINTF("Could not open dynaFile, discarding message\n"); } else { - errmsg.LogError(0, NO_ERRCODE, "Could not open dynamic file '%s' - discarding message", (char*)newFileName); + errmsg.LogError(0, NO_ERRCODE, "Could not open dynamic file '%s' - discarding message", newFileName); } - dynaFileDelCacheEntry(pCache, iFirstFree, 1); - pData->iCurrElt = -1; - return -1; + ABORT_FINALIZE(localRet); } - pCache[iFirstFree]->fd = pData->fd; - pCache[iFirstFree]->pName = (uchar*)strdup((char*)newFileName); /* TODO: check for NULL (very unlikely) */ + if((pCache[iFirstFree]->pName = ustrdup(newFileName)) == NULL) { + strm.Destruct(&pData->pStrm); /* need to free failed entry! */ + ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + } + pCache[iFirstFree]->pStrm = pData->pStrm; pCache[iFirstFree]->lastUsed = time(NULL); pData->iCurrElt = iFirstFree; DBGPRINTF("Added new entry %d for file cache, file '%s'.\n", iFirstFree, newFileName); - ENDfunc +finalize_it: + RETiRet; +} + + +/* do the actual write process. This function is to be called once we are ready for writing. + * It will do buffered writes and persist data only when the buffer is full. Note that we must + * be careful to detect when the file handle changed. + * rgerhards, 2009-06-03 + */ +static rsRetVal +doWrite(instanceData *pData, uchar *pszBuf, int lenBuf) +{ + DEFiRet; + ASSERT(pData != NULL); + ASSERT(pszBuf != NULL); + +dbgprintf("doWrite, pData->pStrm %p, lenBuf %d\n", pData->pStrm, lenBuf); + if(pData->pStrm != NULL){ + CHKiRet(strm.Write(pData->pStrm, pszBuf, lenBuf)); + FINALIZE; + } - return 0; +finalize_it: + RETiRet; } @@ -566,10 +582,9 @@ static int prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsg * will be called for all outputs using file semantics, * for example also for pipes. */ -static rsRetVal writeFile(uchar **ppString, unsigned iMsgOpts, instanceData *pData) +static rsRetVal +writeFile(uchar **ppString, unsigned iMsgOpts, instanceData *pData) { - off_t actualFileSize; - int iLenWritten; DEFiRet; ASSERT(pData != NULL); @@ -578,106 +593,30 @@ static rsRetVal writeFile(uchar **ppString, unsigned iMsgOpts, instanceData *pDa * check if it still is ok or a new file needs to be created */ if(pData->bDynamicName) { - if(prepareDynFile(pData, ppString[1], iMsgOpts) != 0) - ABORT_FINALIZE(RS_RET_SUSPENDED); /* whatever the failure was, we need to retry */ - } - - if(pData->fd == -1) { - rsRetVal iRetLocal; - iRetLocal = prepareFile(pData, pData->f_fname); - if((iRetLocal != RS_RET_OK) || (pData->fd == -1)) - ABORT_FINALIZE(RS_RET_SUSPENDED); /* whatever the failure was, we need to retry */ - } - - /* create the message based on format specified */ -again: - /* check if we have a file size limit and, if so, - * obey to it. - */ - if(pData->f_sizeLimit != 0) { - actualFileSize = lseek(pData->fd, 0, SEEK_END); - if(actualFileSize >= pData->f_sizeLimit) { - char errMsg[256]; - /* for now, we simply disable a file once it is - * beyond the maximum size. This is better than having - * us aborted by the OS... rgerhards 2005-06-21 - */ - (void) close(pData->fd); - /* try to resolve the situation */ - if(resolveFileSizeLimit(pData) != 0) { - /* didn't work out, so disable... */ - snprintf(errMsg, sizeof(errMsg), - "no longer writing to file %s; grown beyond configured file size of %lld bytes, actual size %lld - configured command did not resolve situation", - pData->f_fname, (long long) pData->f_sizeLimit, (long long) actualFileSize); - errno = 0; - errmsg.LogError(0, RS_RET_DISABLE_ACTION, "%s", errMsg); - ABORT_FINALIZE(RS_RET_DISABLE_ACTION); - } else { - snprintf(errMsg, sizeof(errMsg), - "file %s had grown beyond configured file size of %lld bytes, actual size was %lld - configured command resolved situation", - pData->f_fname, (long long) pData->f_sizeLimit, (long long) actualFileSize); - errno = 0; - errmsg.LogError(0, NO_ERRCODE, "%s", errMsg); - } + CHKiRet(prepareDynFile(pData, ppString[1], iMsgOpts)); + } else { /* "regular", non-dynafile */ + if(pData->pStrm == NULL) { + CHKiRet(prepareFile(pData, pData->f_fname)); } } - iLenWritten = write(pData->fd, ppString[0], strlen((char*)ppString[0])); -//dbgprintf("lenwritten: %d\n", iLenWritten); - if(iLenWritten < 0) { - int e = errno; - char errStr[1024]; - rs_strerror_r(errno, errStr, sizeof(errStr)); - DBGPRINTF("log file (%d) write error %d: %s\n", pData->fd, e, errStr); - - /* If a named pipe is full, we suspend this action for a while */ - if(pData->fileType == eTypePIPE && e == EAGAIN) - ABORT_FINALIZE(RS_RET_SUSPENDED); - - close(pData->fd); - pData->fd = -1; /* tell that fd is no longer open! */ - if(pData->bDynamicName && pData->iCurrElt != -1) { - /* in this case, we need to invalidate the name in the cache, too - * otherwise, an invalid fd may show up if we had a file name change. - * rgerhards, 2009-03-19 - */ - pData->dynCache[pData->iCurrElt]->fd = -1; - } - /* Check for EBADF on TTY's due to vhangup() - * Linux uses EIO instead (mrn 12 May 96) - */ - if((pData->fileType == eTypeTTY || pData->fileType == eTypeCONSOLE) -#ifdef linux - && e == EIO -#else - && e == EBADF -#endif - ) { - pData->fd = open((char*) pData->f_fname, O_WRONLY|O_APPEND|O_NOCTTY|O_CLOEXEC); - if (pData->fd < 0) { - iRet = RS_RET_SUSPENDED; - errmsg.LogError(0, NO_ERRCODE, "%s", pData->f_fname); - } else { - untty(); - goto again; - } - } else { - iRet = RS_RET_SUSPENDED; - errno = e; - errmsg.LogError(0, NO_ERRCODE, "%s", pData->f_fname); - } - } else if (pData->bSyncFile) { - fsync(pData->fd); - } + CHKiRet(doWrite(pData, ppString[0], strlen(CHAR_CONVERT(ppString[0])))); finalize_it: + if(iRet != RS_RET_OK) { + /* in v5, we shall return different states for message-caused failure (but only there!) */ + if(pData->strmType == STREAMTYPE_NAMED_PIPE) + iRet = RS_RET_DISABLE_ACTION; /* this is the traditional semantic -- rgerhards, 2010-01-15 */ + else + iRet = RS_RET_SUSPENDED; + } RETiRet; } BEGINcreateInstance CODESTARTcreateInstance - pData->fd = -1; + pData->pStrm = NULL; ENDcreateInstance @@ -685,8 +624,8 @@ BEGINfreeInstance CODESTARTfreeInstance if(pData->bDynamicName) { dynaFileFreeCache(pData); - } else if(pData->fd != -1) - close(pData->fd); + } else if(pData->pStrm != NULL) + strm.Destruct(&pData->pStrm); ENDfreeInstance @@ -696,39 +635,32 @@ ENDtryResume BEGINdoAction CODESTARTdoAction - DBGPRINTF(" (%s)\n", pData->f_fname); - iRet = writeFile(ppString, iMsgOpts, pData); + DBGPRINTF("file to log to: %s\n", pData->f_fname); + CHKiRet(writeFile(ppString, iMsgOpts, pData)); + if(pData->bFlushOnTXEnd) { + /* TODO v5: do this in endTransaction only! */ + CHKiRet(strm.Flush(pData->pStrm)); + } +finalize_it: ENDdoAction BEGINparseSelectorAct CODESTARTparseSelectorAct - /* yes, the if below is redundant, but I need it now. Will go away as - * the code further changes. -- rgerhards, 2007-07-25 - */ - if(*p == '$' || *p == '?' || *p == '|' || *p == '/' || *p == '-') { - if((iRet = createInstance(&pData)) != RS_RET_OK) { - ENDfunc - return iRet; /* this can not use RET_iRet! */ - } - } else { - /* this is not clean, but we need it for the time being - * TODO: remove when cleaning up modularization - */ - ENDfunc - return RS_RET_CONFLINE_UNPROCESSED; - } + if(!(*p == '$' || *p == '?' || *p == '/' || *p == '.' || *p == '-')) + ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); + + CHKiRet(createInstance(&pData)); if(*p == '-') { pData->bSyncFile = 0; p++; } else { - pData->bSyncFile = bEnableSync ? 1 : 0; + pData->bSyncFile = bEnableSync; } - pData->f_sizeLimit = 0; /* default value, use outchannels to configure! */ + pData->iSizeLimit = 0; /* default value, use outchannels to configure! */ - switch (*p) - { + switch(*p) { case '$': CODE_STD_STRING_REQUESTparseSelectorAct(1) /* rgerhards 2005-06-21: this is a special setting for output-channel @@ -738,13 +670,8 @@ CODESTARTparseSelectorAct * rgerhards, 2007-07-24: output-channels will go away. We keep them * for compatibility reasons, but seems to have been a bad idea. */ - if((iRet = cflineParseOutchannel(pData, p, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS)) == RS_RET_OK) { - pData->bDynamicName = 0; - pData->fCreateMode = fCreateMode; /* preserve current setting */ - pData->fDirCreateMode = fDirCreateMode; /* preserve current setting */ - pData->fd = open((char*) pData->f_fname, O_WRONLY|O_APPEND|O_CREAT|O_NOCTTY|O_CLOEXEC, - pData->fCreateMode); - } + CHKiRet(cflineParseOutchannel(pData, p, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS)); + pData->bDynamicName = 0; break; case '?': /* This is much like a regular file handle, but we need to obtain @@ -752,84 +679,71 @@ CODESTARTparseSelectorAct */ CODE_STD_STRING_REQUESTparseSelectorAct(2) ++p; /* eat '?' */ - if((iRet = cflineParseFileName(p, (uchar*) pData->f_fname, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, - (pszTplName == NULL) ? (uchar*)"RSYSLOG_FileFormat" : pszTplName)) - != RS_RET_OK) - break; + CHKiRet(cflineParseFileName(p, (uchar*) pData->f_fname, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, + (pszFileDfltTplName == NULL) ? (uchar*)"RSYSLOG_FileFormat" : pszFileDfltTplName)); /* "filename" is actually a template name, we need this as string 1. So let's add it * to the pOMSR. -- rgerhards, 2007-07-27 */ - if((iRet = OMSRsetEntry(*ppOMSR, 1, (uchar*)strdup((char*) pData->f_fname), OMSR_NO_RQD_TPL_OPTS)) != RS_RET_OK) - break; + CHKiRet(OMSRsetEntry(*ppOMSR, 1, ustrdup(pData->f_fname), OMSR_NO_RQD_TPL_OPTS)); pData->bDynamicName = 1; pData->iCurrElt = -1; /* no current element */ - pData->fCreateMode = fCreateMode; /* freeze current setting */ - pData->fDirCreateMode = fDirCreateMode; /* preserve current setting */ - pData->bCreateDirs = bCreateDirs; - pData->bFailOnChown = bFailOnChown; - pData->fileUID = fileUID; - pData->fileGID = fileGID; - pData->dirUID = dirUID; - pData->dirGID = dirGID; - pData->iDynaFileCacheSize = iDynaFileCacheSize; /* freeze current setting */ - /* we now allocate the cache table. We use calloc() intentionally, as we - * need all pointers to be initialized to NULL pointers. - */ - if((pData->dynCache = (dynaFileCacheEntry**) - calloc(iDynaFileCacheSize, sizeof(dynaFileCacheEntry*))) == NULL) { - iRet = RS_RET_OUT_OF_MEMORY; - DBGPRINTF("Could not allocate memory for dynaFileCache - selector disabled.\n"); - } + /* we now allocate the cache table */ + CHKmalloc(pData->dynCache = (dynaFileCacheEntry**) + calloc(iDynaFileCacheSize, sizeof(dynaFileCacheEntry*))); break; - case '|': + /* case '|': while pipe support has been removed, I leave the code in in case we + * need high-performance pipes at a later stage (unlikely). -- rgerhards, 2010-02-28 + */ case '/': + case '.': CODE_STD_STRING_REQUESTparseSelectorAct(1) - /* rgerhards, 2007-0726: first check if file or pipe */ + /* we now have *almost* the same semantics for files and pipes, but we still need + * to know we deal with a pipe, because we must do non-blocking opens in that case + * (to keep consistent with traditional semantics and prevent rsyslog from hanging). + */ if(*p == '|') { - pData->fileType = eTypePIPE; ++p; + pData->strmType = STREAMTYPE_NAMED_PIPE; } else { - pData->fileType = eTypeFILE; + pData->strmType = STREAMTYPE_FILE_SINGLE; } - /* rgerhards 2004-11-17: from now, we need to have different - * processing, because after the first comma, the template name - * to use is specified. So we need to scan for the first coma first - * and then look at the rest of the line. - */ - if((iRet = cflineParseFileName(p, (uchar*) pData->f_fname, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, - (pszTplName == NULL) ? (uchar*)"RSYSLOG_FileFormat" : pszTplName)) - != RS_RET_OK) - break; + CHKiRet(cflineParseFileName(p, (uchar*) pData->f_fname, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, + (pszFileDfltTplName == NULL) ? (uchar*)"RSYSLOG_FileFormat" : pszFileDfltTplName)); pData->bDynamicName = 0; - pData->fCreateMode = fCreateMode; /* preserve current setting */ - pData->fDirCreateMode = fDirCreateMode; - pData->bCreateDirs = bCreateDirs; - pData->bFailOnChown = bFailOnChown; - pData->fileUID = fileUID; - pData->fileGID = fileGID; - pData->dirUID = dirUID; - pData->dirGID = dirGID; - - /* at this stage, we ignore the return value of prepareFile, this is taken - * care of in later steps. -- rgerhards, 2009-03-19 + break; + default: + ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); + } + + /* freeze current paremeters for this action */ + pData->iDynaFileCacheSize = iDynaFileCacheSize; + pData->fCreateMode = fCreateMode; + pData->fDirCreateMode = fDirCreateMode; + pData->bCreateDirs = bCreateDirs; + pData->bFailOnChown = bFailOnChown; + pData->fileUID = fileUID; + pData->fileGID = fileGID; + pData->dirUID = dirUID; + pData->dirGID = dirGID; + pData->iZipLevel = iZipLevel; + pData->bFlushOnTXEnd = bFlushOnTXEnd; + pData->iIOBufSize = (int) iIOBufSize; + pData->iFlushInterval = iFlushInterval; + pData->bUseAsyncWriter = bUseAsyncWriter; + + if(pData->bDynamicName == 0) { + /* try open and emit error message if not possible. At this stage, we ignore the + * return value of prepareFile, this is taken care of in later steps. */ prepareFile(pData, pData->f_fname); - if(pData->fd < 0 ) { - pData->fd = -1; + if(pData->pStrm == NULL) { DBGPRINTF("Error opening log file: %s\n", pData->f_fname); errmsg.LogError(0, RS_RET_NO_FILE_ACCESS, "Could no open output file '%s'", pData->f_fname); - break; } - if(strcmp((char*) p, _PATH_CONSOLE) == 0) - pData->fileType = eTypeCONSOLE; - break; - default: - iRet = RS_RET_CONFLINE_UNPROCESSED; - break; } CODE_STD_FINALIZERparseSelectorAct ENDparseSelectorAct @@ -850,9 +764,14 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a fDirCreateMode = 0700; bCreateDirs = 1; bEnableSync = 0; - if(pszTplName != NULL) { - free(pszTplName); - pszTplName = NULL; + iZipLevel = 0; + bFlushOnTXEnd = FLUSHONTX_DFLT; + iIOBufSize = IOBUF_DFLT_SIZE; + iFlushInterval = FLUSH_INTRVL_DFLT; + bUseAsyncWriter = USE_ASYNCWRITER_DFLT; + if(pszFileDfltTplName != NULL) { + free(pszFileDfltTplName); + pszFileDfltTplName = NULL; } return RS_RET_OK; @@ -863,11 +782,10 @@ BEGINdoHUP CODESTARTdoHUP if(pData->bDynamicName) { dynaFileFreeCacheEntries(pData); - pData->iCurrElt = -1; /* invalidate current element */ } else { - if(pData->fd != -1) { - close(pData->fd); - pData->fd = -1; + if(pData->pStrm != NULL) { + strm.Destruct(&pData->pStrm); + pData->pStrm = NULL; } } ENDdoHUP @@ -875,8 +793,9 @@ ENDdoHUP BEGINmodExit CODESTARTmodExit - if(pszTplName != NULL) - free(pszTplName); + objRelease(errmsg, CORE_COMPONENT); + objRelease(strm, CORE_COMPONENT); + free(pszFileDfltTplName); ENDmodExit @@ -892,7 +811,13 @@ CODESTARTmodInit *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(strm, CORE_COMPONENT)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"dynafilecachesize", 0, eCmdHdlrInt, (void*) setDynaFileCacheSize, NULL, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"omfileziplevel", 0, eCmdHdlrInt, NULL, &iZipLevel, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"omfileflushinterval", 0, eCmdHdlrInt, NULL, &iFlushInterval, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"omfileasyncwriting", 0, eCmdHdlrBinary, NULL, &bUseAsyncWriter, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"omfileflushontxend", 0, eCmdHdlrBinary, NULL, &bFlushOnTXEnd, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"omfileiobuffersize", 0, eCmdHdlrSize, NULL, &iIOBufSize, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"dirowner", 0, eCmdHdlrUID, NULL, &dirUID, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"dirgroup", 0, eCmdHdlrGID, NULL, &dirGID, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"fileowner", 0, eCmdHdlrUID, NULL, &fileUID, STD_LOADABLE_MODULE_ID)); @@ -902,7 +827,7 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(omsdRegCFSLineHdlr((uchar *)"createdirs", 0, eCmdHdlrBinary, NULL, &bCreateDirs, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"failonchownfailure", 0, eCmdHdlrBinary, NULL, &bFailOnChown, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionfileenablesync", 0, eCmdHdlrBinary, NULL, &bEnableSync, STD_LOADABLE_MODULE_ID)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionfiledefaulttemplate", 0, eCmdHdlrGetWord, NULL, &pszTplName, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionfiledefaulttemplate", 0, eCmdHdlrGetWord, NULL, &pszFileDfltTplName, NULL)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID)); ENDmodInit /* vi:set ai: diff --git a/tools/omfile.h b/tools/omfile.h index 03e081f3..8dca6a88 100644 --- a/tools/omfile.h +++ b/tools/omfile.h @@ -3,7 +3,7 @@ * * File begun on 2007-07-21 by RGerhards (extracted from syslogd.c) * - * Copyright 2007 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2010 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -28,7 +28,11 @@ /* prototypes */ rsRetVal modInitFile(int iIFVersRequested __attribute__((unused)), int *ipIFVersProvided, rsRetVal (**pQueryEtryPt)(), rsRetVal (*pHostQueryEtryPt)(uchar*, rsRetVal (**)()), modInfo_t*); +/* the define below is dirty, but we need it for ompipe integration. There is no + * other way to have the functionality (well, one way would be to go through the + * globals, but that seems not yet justified. -- rgerhards, 2010-03-01 + */ +uchar *pszFileDfltTplName; #endif /* #ifndef OMFILE_H_INCLUDED */ -/* - * vi:set ai: +/* vi:set ai: */ diff --git a/tools/omfwd.c b/tools/omfwd.c index 97703c79..cbfc36a4 100644 --- a/tools/omfwd.c +++ b/tools/omfwd.c @@ -48,6 +48,7 @@ #endif #include <pthread.h> #include "syslogd.h" +#include "conf.h" #include "syslogd-types.h" #include "srUtils.h" #include "net.h" @@ -89,6 +90,7 @@ typedef struct _instanceData { char *port; int protocol; int iUDPRebindInterval; /* rebind interval */ + int iTCPRebindInterval; /* rebind interval */ int nXmit; /* number of transmissions since last (re-)bind */ # define FORW_UDP 0 # define FORW_TCP 1 @@ -103,6 +105,7 @@ static short iStrmDrvrMode = 0; /* mode for stream driver, driver-dependent (0 m static short bResendLastOnRecon = 0; /* should the last message be re-sent on a successful reconnect? */ static uchar *pszStrmDrvrAuthMode = NULL; /* authentication mode to use */ static int iUDPRebindInterval = 0; /* support for automatic re-binding (load balancers!). 0 - no rebind */ +static int iTCPRebindInterval = 0; /* support for automatic re-binding (load balancers!). 0 - no rebind */ static permittedPeers_t *pPermPeers = NULL; @@ -408,7 +411,7 @@ CODESTARTtryResume ENDtryResume BEGINdoAction - char *psz; /* temporary buffering */ + char *psz = NULL; /* temporary buffering */ register unsigned l; int iMaxLine; CODESTARTdoAction @@ -455,11 +458,14 @@ CODESTARTdoAction * rgerhards, 2006-11-30 */ dbgprintf("Compression failed, sending uncompressed message\n"); + free(out); } else if(destLen+1 < l) { /* only use compression if there is a gain in using it! */ dbgprintf("there is gain in compression, so we do it\n"); psz = (char*) out; l = destLen + 1; /* take care for the "z" at message start! */ + } else { + free(out); } ++destLen; } @@ -480,6 +486,12 @@ CODESTARTdoAction } } finalize_it: +# ifdef USE_NETZIP + if((psz != NULL) && (psz != (char*) ppString[0])) { + /* we need to free temporary buffer, alloced above - Naoya Nakazawa, 2010-01-11 */ + free(psz); + } +# endif ENDdoAction @@ -642,6 +654,7 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) /* copy over config data as needed */ pData->iUDPRebindInterval = iUDPRebindInterval; + pData->iTCPRebindInterval = iTCPRebindInterval; /* process template */ CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, @@ -656,6 +669,7 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) CHKiRet(tcpclt.SetSendFrame(pData->pTCPClt, TCPSendFrame)); CHKiRet(tcpclt.SetSendPrepRetry(pData->pTCPClt, TCPSendPrepRetry)); CHKiRet(tcpclt.SetFraming(pData->pTCPClt, tcp_framing)); + CHKiRet(tcpclt.SetRebindInterval(pData->pTCPClt, pData->iTCPRebindInterval)); pData->iStrmDrvrMode = iStrmDrvrMode; if(pszStrmDrvr != NULL) CHKmalloc(pData->pszStrmDrvr = (uchar*)strdup((char*)pszStrmDrvr)); @@ -727,6 +741,7 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a iStrmDrvrMode = 0; bResendLastOnRecon = 0; iUDPRebindInterval = 0; + iTCPRebindInterval = 0; return RS_RET_OK; } @@ -741,6 +756,7 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(net,LM_NET_FILENAME)); CHKiRet(regCfSysLineHdlr((uchar *)"actionforwarddefaulttemplate", 0, eCmdHdlrGetWord, NULL, &pszTplName, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionsendtcprebindinterval", 0, eCmdHdlrInt, NULL, &iTCPRebindInterval, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"actionsendudprebindinterval", 0, eCmdHdlrInt, NULL, &iUDPRebindInterval, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"actionsendstreamdriver", 0, eCmdHdlrGetWord, NULL, &pszStrmDrvr, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"actionsendstreamdrivermode", 0, eCmdHdlrInt, NULL, &iStrmDrvrMode, NULL)); diff --git a/tools/ompipe.c b/tools/ompipe.c new file mode 100644 index 00000000..541b6552 --- /dev/null +++ b/tools/ompipe.c @@ -0,0 +1,240 @@ +/* ompipe.c + * This is the implementation of the build-in pipe output module. + * Note that this module stems back to the "old" (4.4.2 and below) + * omfile. There were some issues with the new omfile code and pipes + * (namely in regard to xconsole), so we took out the pipe code and moved + * that to a separate module. That a) immediately solves the issue for a + * less common use case and probably makes it much easier to enhance + * file and pipe support (now independently) in the future (we always + * needed to think about pipes in omfile so far, what we now no longer + * need to, hopefully resulting in reduction of complexity). + * + * NOTE: read comments in module-template.h to understand how this pipe + * works! + * + * Copyright 2007-2010 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <stdarg.h> +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <errno.h> +#include <unistd.h> +#include <sys/file.h> + +#include "syslogd.h" +#include "syslogd-types.h" +#include "srUtils.h" +#include "template.h" +#include "ompipe.h" +#include "omfile.h" /* for dirty trick: access to $ActionFileDefaultTemplate value */ +#include "cfsysline.h" +#include "module-template.h" +#include "conf.h" +#include "errmsg.h" + +MODULE_TYPE_OUTPUT + +/* internal structures + */ +DEF_OMOD_STATIC_DATA +DEFobjCurrIf(errmsg) + + +/* globals for default values */ +/* end globals for default values */ + + +typedef struct _instanceData { + uchar f_fname[MAXFNAME];/* pipe or template name (display only) */ + short fd; /* pipe descriptor for (current) pipe */ +} instanceData; + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature + if(eFeat == sFEATURERepeatedMsgReduction) + iRet = RS_RET_OK; +ENDisCompatibleWithFeature + + +BEGINdbgPrintInstInfo +CODESTARTdbgPrintInstInfo + dbgprintf("pipe %s", pData->f_fname); + if (pData->fd == -1) + dbgprintf(" (unused)"); +ENDdbgPrintInstInfo + + +/* This is now shared code for all types of files. It simply prepares + * pipe access, which, among others, means the the pipe wil be opened + * and any directories in between will be created (based on config, of + * course). -- rgerhards, 2008-10-22 + * changed to iRet interface - 2009-03-19 + */ +static inline rsRetVal +preparePipe(instanceData *pData) +{ + DEFiRet; + pData->fd = open((char*) pData->f_fname, O_RDWR|O_NONBLOCK|O_CLOEXEC); + RETiRet; +} + + +/* rgerhards 2004-11-11: write to a pipe output. This + * will be called for all outputs using pipe semantics, + * for example also for pipes. + */ +static rsRetVal writePipe(uchar **ppString, instanceData *pData) +{ + int iLenWritten; + DEFiRet; + + ASSERT(pData != NULL); + + if(pData->fd == -1) { + rsRetVal iRetLocal; + iRetLocal = preparePipe(pData); + if((iRetLocal != RS_RET_OK) || (pData->fd == -1)) + ABORT_FINALIZE(RS_RET_SUSPENDED); /* whatever the failure was, we need to retry */ + } + + /* create the message based on format specified */ + iLenWritten = write(pData->fd, ppString[0], strlen((char*)ppString[0])); + if(iLenWritten < 0) { + int e = errno; + char errStr[1024]; + rs_strerror_r(errno, errStr, sizeof(errStr)); + DBGPRINTF("pipe (%d) write error %d: %s\n", pData->fd, e, errStr); + + /* If a named pipe is full, we suspend this action for a while */ + if(e == EAGAIN) + ABORT_FINALIZE(RS_RET_SUSPENDED); + + close(pData->fd); + pData->fd = -1; /* tell that fd is no longer open! */ + iRet = RS_RET_SUSPENDED; + errno = e; + errmsg.LogError(0, NO_ERRCODE, "%s", pData->f_fname); + } + +finalize_it: + RETiRet; +} + + +BEGINcreateInstance +CODESTARTcreateInstance + pData->fd = -1; +ENDcreateInstance + + +BEGINfreeInstance +CODESTARTfreeInstance + if(pData->fd != -1) + close(pData->fd); +ENDfreeInstance + + +BEGINtryResume +CODESTARTtryResume +ENDtryResume + +BEGINdoAction +CODESTARTdoAction + DBGPRINTF(" (%s)\n", pData->f_fname); + iRet = writePipe(ppString, pData); +ENDdoAction + + +BEGINparseSelectorAct +CODESTARTparseSelectorAct + /* yes, the if below is redundant, but I need it now. Will go away as + * the code further changes. -- rgerhards, 2007-07-25 + */ + if(*p == '|') { + if((iRet = createInstance(&pData)) != RS_RET_OK) { + ENDfunc + return iRet; /* this can not use RET_iRet! */ + } + } else { + /* this is not clean, but we need it for the time being + * TODO: remove when cleaning up modularization + */ + ENDfunc + return RS_RET_CONFLINE_UNPROCESSED; + } + + CODE_STD_STRING_REQUESTparseSelectorAct(1) + ++p; + /* rgerhards 2004-11-17: from now, we need to have different + * processing, because after the first comma, the template name + * to use is specified. So we need to scan for the first coma first + * and then look at the rest of the line. + */ + CHKiRet(cflineParseFileName(p, (uchar*) pData->f_fname, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, + (pszFileDfltTplName == NULL) ? (uchar*)"RSYSLOG_FileFormat" : pszFileDfltTplName)); + + /* at this stage, we ignore the return value of preparePipe, this is taken + * care of in later steps. -- rgerhards, 2009-03-19 + */ + preparePipe(pData); + + if(pData->fd < 0 ) { + pData->fd = -1; + DBGPRINTF("Error opening log pipe: %s\n", pData->f_fname); + errmsg.LogError(0, RS_RET_NO_FILE_ACCESS, "Could no open output pipe '%s'", pData->f_fname); + } +CODE_STD_FINALIZERparseSelectorAct +ENDparseSelectorAct + + +BEGINdoHUP +CODESTARTdoHUP + if(pData->fd != -1) { + close(pData->fd); + pData->fd = -1; + } +ENDdoHUP + + +BEGINmodExit +CODESTARTmodExit +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_doHUP +ENDqueryEtryPt + + +BEGINmodInit(Pipe) +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + CHKiRet(objUse(errmsg, CORE_COMPONENT)); +ENDmodInit +/* vi:set ai: + */ diff --git a/tools/ompipe.h b/tools/ompipe.h new file mode 100644 index 00000000..d17346c5 --- /dev/null +++ b/tools/ompipe.h @@ -0,0 +1,31 @@ +/* ompipe.h + * These are the definitions for the build-in pipe output module. + * + * Copyright 2007-2010 Rainer Gerhards and Adiscon GmbH. + * + * This pipe is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the pipe "COPYING" in this distribution. + */ +#ifndef OMPIPE_H_INCLUDED +#define OMPIPE_H_INCLUDED 1 + +/* prototypes */ +rsRetVal modInitPipe(int iIFVersRequested __attribute__((unused)), int *ipIFVersProvided, rsRetVal (**pQueryEtryPt)(), rsRetVal (*pHostQueryEtryPt)(uchar*, rsRetVal (**)()), modInfo_t*); + +#endif /* #ifndef OMPIPE_H_INCLUDED */ +/* vi:set ai: + */ diff --git a/tools/omshell.c b/tools/omshell.c index 7b815869..f8a68527 100644 --- a/tools/omshell.c +++ b/tools/omshell.c @@ -38,7 +38,7 @@ #include <stdlib.h> #include <string.h> #include <assert.h> -#include "syslogd.h" +#include "conf.h" #include "syslogd-types.h" #include "srUtils.h" #include "omshell.h" diff --git a/tools/omusrmsg.c b/tools/omusrmsg.c index 830bbc87..e61751dc 100644 --- a/tools/omusrmsg.c +++ b/tools/omusrmsg.c @@ -50,7 +50,15 @@ #include <assert.h> #include <signal.h> #include <sys/param.h> -#include <utmp.h> +#ifdef HAVE_UTMP_H +# include <utmp.h> +# define STRUCTUTMP struct utmp +# define UTNAME ut_name +#else +# include <utmpx.h> +# define STRUCTUTMP struct utmpx +# define UTNAME ut_user +#endif #include <unistd.h> #include <sys/uio.h> #include <sys/stat.h> @@ -66,7 +74,7 @@ #include "srUtils.h" #include "stringbuf.h" #include "syslogd-types.h" -#include "syslogd.h" +#include "conf.h" #include "omusrmsg.h" #include "module-template.h" #include "errmsg.h" @@ -124,19 +132,25 @@ ENDdbgPrintInstInfo * need! rgerhards 2005-03-18 */ #ifdef OS_BSD +/* Since version 900007, FreeBSD has a POSIX compliant <utmpx.h> */ +#if defined(__FreeBSD__) && (__FreeBSD_version >= 900007) +# define setutent(void) setutxent(void) +# define getutent(void) getutxent(void) +# define endutent(void) endutxent(void) +#else static FILE *BSD_uf = NULL; void setutent(void) { assert(BSD_uf == NULL); if ((BSD_uf = fopen(_PATH_UTMP, "r")) == NULL) { - errmsg.LogError(NO_ERRCODE, "%s", _PATH_UTMP); + errmsg.LogError(0, NO_ERRCODE, "%s", _PATH_UTMP); return; } } -struct utmp* getutent(void) +STRUCTUTMP* getutent(void) { - static struct utmp st_utmp; + static STRUCTUTMP st_utmp; if(fread((char *)&st_utmp, sizeof(st_utmp), 1, BSD_uf) != 1) return NULL; @@ -149,6 +163,7 @@ void endutent(void) fclose(BSD_uf); BSD_uf = NULL; } +#endif /* if defined(__FreeBSD__) */ #endif /* #ifdef OS_BSD */ @@ -173,8 +188,8 @@ static rsRetVal wallmsg(uchar* pMsg, instanceData *pData) int errnoSave; int ttyf; int wrRet; - struct utmp ut; - struct utmp *uptr; + STRUCTUTMP ut; + STRUCTUTMP *uptr; struct stat statb; DEFiRet; @@ -187,13 +202,13 @@ static rsRetVal wallmsg(uchar* pMsg, instanceData *pData) while((uptr = getutent())) { memcpy(&ut, uptr, sizeof(ut)); /* is this slot used? */ - if(ut.ut_name[0] == '\0') + if(ut.UTNAME[0] == '\0') continue; #ifndef OS_BSD if(ut.ut_type != USER_PROCESS) continue; #endif - if(!(strncmp (ut.ut_name,"LOGIN", 6))) /* paranoia */ + if(!(strncmp (ut.UTNAME,"LOGIN", 6))) /* paranoia */ continue; /* should we send the message to this user? */ @@ -203,7 +218,7 @@ static rsRetVal wallmsg(uchar* pMsg, instanceData *pData) i = MAXUNAMES; break; } - if(strncmp(pData->uname[i], ut.ut_name, UNAMESZ) == 0) + if(strncmp(pData->uname[i], ut.UTNAME, UNAMESZ) == 0) break; } if(i == MAXUNAMES) /* user not found? */ diff --git a/tools/regexp.c b/tools/regexp.c index c8e4c681..e8bba4f4 100644 --- a/tools/regexp.c +++ b/tools/regexp.c @@ -26,6 +26,7 @@ * * A copy of the GPL can be found in the file "COPYING" in this distribution. */ +#include "config.h" #include <stdio.h> #include <stdlib.h> #include <string.h> diff --git a/tools/rsyslog.conf.5 b/tools/rsyslog.conf.5 index 0a2422c6..e8a4ab92 100644 --- a/tools/rsyslog.conf.5 +++ b/tools/rsyslog.conf.5 @@ -80,7 +80,7 @@ used like this: .IP $ModLoad imudp .IP -$InputUDPServerRun 514 +$UDPServerRun 514 .TP .I imtcp Input plugin for plain TCP syslog. Replaces the deprecated -t @@ -164,8 +164,8 @@ a pattern of facilities and priorities belonging to the specified action. The selector field itself again consists of two parts, a facility and a priority, separated by a period ('.'). Both parts are case insensitive and can also be specified as decimal numbers, but don't do that, you have been warned. -Both facilities and priorities are described in rsyslog(3). The names mentioned -below correspond to the similar LOG_-values in /usr/include/rsyslog.h. +Both facilities and priorities are described in syslog(3). The names mentioned +below correspond to the similar LOG_-values in /usr/include/syslog.h. The facility is one of the following keywords: auth, authpriv, cron, daemon, kern, lpr, mail, mark, news, security (same as auth), syslog, user, uucp and diff --git a/tools/syslogd.c b/tools/syslogd.c index 5f6b480f..a03dcf0e 100644 --- a/tools/syslogd.c +++ b/tools/syslogd.c @@ -127,17 +127,21 @@ #include "omusrmsg.h" #include "omfwd.h" #include "omfile.h" +#include "ompipe.h" #include "omdiscard.h" #include "threads.h" #include "queue.h" #include "stream.h" #include "conf.h" -#include "vm.h" #include "errmsg.h" #include "datetime.h" #include "parser.h" -#include "sysvar.h" #include "unicode-helper.h" +#include "ruleset.h" +#include "rule.h" +#include "net.h" +#include "vm.h" +#include "prop.h" /* definitions for objects we access */ DEFobjCurrIf(obj) @@ -145,29 +149,17 @@ DEFobjCurrIf(glbl) DEFobjCurrIf(datetime) DEFobjCurrIf(conf) DEFobjCurrIf(expr) -DEFobjCurrIf(vm) -DEFobjCurrIf(var) DEFobjCurrIf(module) DEFobjCurrIf(errmsg) +DEFobjCurrIf(rule) +DEFobjCurrIf(ruleset) +DEFobjCurrIf(prop) DEFobjCurrIf(net) /* TODO: make go away! */ /* forward definitions */ static rsRetVal GlobalClassExit(void); -/* We define our own set of syslog defintions so that we - * do not need to rely on (possibly different) implementations. - * 2007-07-19 rgerhards - */ -/* missing definitions for solaris - * 2006-02-16 Rger - */ -#ifdef __sun -# define LOG_AUTHPRIV LOG_AUTH -#endif -#define INTERNAL_NOPRI 0x10 /* the "no priority" priority */ -#define LOG_FTP (11<<3) /* ftp daemon */ - #ifndef UTMP_FILE #ifdef UTMP_FILENAME @@ -186,7 +178,11 @@ static rsRetVal GlobalClassExit(void); #endif #ifndef _PATH_MODDIR -#define _PATH_MODDIR "/lib/rsyslog/" +# if defined(__FreeBSD__) +# define _PATH_MODDIR "/usr/local/lib/rsyslog/" +# else +# define _PATH_MODDIR "/lib/rsyslog/" +# endif #endif #if defined(SYSLOGD_PIDNAME) @@ -215,14 +211,12 @@ static rsRetVal GlobalClassExit(void); # endif #endif -#ifndef _PATH_DEV -# define _PATH_DEV "/dev/" -#endif - #ifndef _PATH_TTY -#define _PATH_TTY "/dev/tty" +# define _PATH_TTY "/dev/tty" #endif +static prop_t *pInternalInputName = NULL; /* there is only one global inputName for all internally-generated messages */ +static prop_t *pLocalHostIP = NULL; /* there is only one global IP for all internally-generated messages */ static uchar *ConfFile = (uchar*) _PATH_LOGCONF; /* read-only after startup */ static char *PidFile = _PATH_LOGPID; /* read-only after startup */ @@ -247,8 +241,6 @@ int repeatinterval[2] = { 30, 60 }; /* # of secs before flush */ #define LIST_DELIMITER ':' /* delimiter between two hosts */ -struct filed *Files = NULL; /* read-only after init() (but beware of sigusr1!) */ - static pid_t ppid; /* This is a quick and dirty hack used for spliting main/startup thread */ typedef struct legacyOptsLL_s { @@ -298,7 +290,8 @@ static queueType_t MainMsgQueType = QUEUETYPE_FIXED_ARRAY; /* type of the main m static uchar *pszMainMsgQFName = NULL; /* prefix for the main message queue file */ static int64 iMainMsgQueMaxFileSize = 1024*1024; static int iMainMsgQPersistUpdCnt = 0; /* persist queue info every n updates */ -static int iMainMsgQtoQShutdown = 0; /* queue shutdown */ +static int bMainMsgQSyncQeueFiles = 0; /* sync queue files on every write? */ +static int iMainMsgQtoQShutdown = 1500; /* queue shutdown (ms) */ static int iMainMsgQtoActShutdown = 1000; /* action shutdown (in phase 2) */ static int iMainMsgQtoEnq = 2000; /* timeout for queue enque */ static int iMainMsgQtoWrkShutdown = 60000; /* timeout for worker thread shutdown */ @@ -313,7 +306,8 @@ static int iMainMsgQueueDeqtWinToHr = 25; /* hour begin of time frame when que /* support for simple textual representation of FIOP names * rgerhards, 2005-09-27 */ -static char* getFIOPName(unsigned iFIOP) +char* +getFIOPName(unsigned iFIOP) { char *pRet; switch(iFIOP) { @@ -360,7 +354,8 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a iMainMsgQueMaxFileSize = 1024 * 1024; iMainMsgQueueNumWorkers = 1; iMainMsgQPersistUpdCnt = 0; - iMainMsgQtoQShutdown = 0; + bMainMsgQSyncQeueFiles = 0; + iMainMsgQtoQShutdown = 1500; iMainMsgQtoActShutdown = 1000; iMainMsgQtoEnq = 2000; iMainMsgQtoWrkShutdown = 60000; @@ -395,7 +390,6 @@ static char **crunch_list(char *list); static void reapchild(); static void debug_switch(); static void sighup_handler(); -static void freeSelectors(void); static void processImInternal(void); @@ -422,7 +416,8 @@ diagGetMainMsgQSize(int *piSize) { DEFiRet; assert(piSize != NULL); - *piSize = pMsgQueue->iQueueSize; + *piSize = (pMsgQueue->pqDA != NULL) ? pMsgQueue->pqDA->iQueueSize : 0; + *piSize += pMsgQueue->iQueueSize; RETiRet; } @@ -430,67 +425,6 @@ diagGetMainMsgQSize(int *piSize) /* ------------------------------ end support functions for imdiag ------------------------------ */ -/* function to destruct a selector_t object - * rgerhards, 2007-08-01 - */ -rsRetVal -selectorDestruct(void *pVal) -{ - selector_t *pThis = (selector_t *) pVal; - - assert(pThis != NULL); - - if(pThis->pCSHostnameComp != NULL) - rsCStrDestruct(&pThis->pCSHostnameComp); - if(pThis->pCSProgNameComp != NULL) - rsCStrDestruct(&pThis->pCSProgNameComp); - - if(pThis->f_filter_type == FILTER_PROP) { - if(pThis->f_filterData.prop.pCSPropName != NULL) - rsCStrDestruct(&pThis->f_filterData.prop.pCSPropName); - if(pThis->f_filterData.prop.pCSCompValue != NULL) - rsCStrDestruct(&pThis->f_filterData.prop.pCSCompValue); - if(pThis->f_filterData.prop.regex_cache != NULL) - rsCStrRegexDestruct(&pThis->f_filterData.prop.regex_cache); - } else if(pThis->f_filter_type == FILTER_EXPR) { - if(pThis->f_filterData.f_expr != NULL) - expr.Destruct(&pThis->f_filterData.f_expr); - } - - llDestroy(&pThis->llActList); - free(pThis); - - return RS_RET_OK; -} - - -/* function to construct a selector_t object - * rgerhards, 2007-08-01 - */ -rsRetVal -selectorConstruct(selector_t **ppThis) -{ - DEFiRet; - selector_t *pThis; - - assert(ppThis != NULL); - - if((pThis = (selector_t*) calloc(1, sizeof(selector_t))) == NULL) { - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } - CHKiRet(llInit(&pThis->llActList, actionDestruct, NULL, NULL)); - -finalize_it: - if(iRet != RS_RET_OK) { - if(pThis != NULL) { - selectorDestruct(pThis); - } - } - *ppThis = pThis; - RETiRet; -} - - /* rgerhards, 2005-10-24: crunch_list is called only during option processing. So * it is never called once rsyslogd is running (not even when HUPed). This code * contains some exits, but they are considered safe because they only happen @@ -562,7 +496,7 @@ static char **crunch_list(char *list) void untty(void) #ifdef HAVE_SETSID { - if ( !Debug ) { + if(!Debug) { setsid(); } return; @@ -571,18 +505,18 @@ void untty(void) { int i; - if ( !Debug ) { + if(!Debug) { i = open(_PATH_TTY, O_RDWR|O_CLOEXEC); if (i >= 0) { # if !defined(__hpux) - (void) ioctl(i, (int) TIOCNOTTY, (char *)0); + (void) ioctl(i, (int) TIOCNOTTY, NULL); # else /* TODO: we need to implement something for HP UX! -- rgerhards, 2008-03-04 */ /* actually, HP UX should have setsid, so the code directly above should * trigger. So the actual question is why it doesn't do that... */ # endif - (void) close(i); + close(i); } } } @@ -625,12 +559,14 @@ void untty(void) * interface change: bParseHostname removed, now in flags */ static inline rsRetVal printline(uchar *hname, uchar *hnameIP, uchar *msg, int flags, flowControl_t flowCtlType, - uchar *pszInputName, struct syslogTime *stTime, time_t ttGenTime) + prop_t *pInputName, struct syslogTime *stTime, time_t ttGenTime) { DEFiRet; register uchar *p; int pri; msg_t *pMsg; + prop_t *propFromHost = NULL; + prop_t *propFromHostIP = NULL; /* Now it is time to create the message object (rgerhards) */ if(stTime == NULL) { @@ -638,10 +574,10 @@ static inline rsRetVal printline(uchar *hname, uchar *hnameIP, uchar *msg, int f } else { CHKiRet(msgConstructWithTime(&pMsg, stTime, ttGenTime)); } - if(pszInputName != NULL) - MsgSetInputName(pMsg, pszInputName, ustrlen(pszInputName)); + if(pInputName != NULL) + MsgSetInputName(pMsg, pInputName); MsgSetFlowControlType(pMsg, flowCtlType); - MsgSetRawMsg(pMsg, (char*)msg); + MsgSetRawMsgWOSize(pMsg, (char*)msg); /* test for special codes */ pri = DEFUPRI; @@ -668,10 +604,12 @@ static inline rsRetVal printline(uchar *hname, uchar *hnameIP, uchar *msg, int f * being the local host). rgerhards 2004-11-16 */ if((pMsg->msgFlags & PARSE_HOSTNAME) == 0) - MsgSetHOSTNAME(pMsg, hname); - MsgSetRcvFrom(pMsg, hname); + MsgSetHOSTNAME(pMsg, hname, ustrlen(hname)); + MsgSetRcvFromStr(pMsg, hname, ustrlen(hname), &propFromHost); + CHKiRet(MsgSetRcvFromIPStr(pMsg, hnameIP, ustrlen(hnameIP), &propFromHostIP)); MsgSetAfterPRIOffs(pMsg, p - msg); - CHKiRet(MsgSetRcvFromIP(pMsg, hnameIP)); + prop.Destruct(&propFromHost); + prop.Destruct(&propFromHostIP); logmsg(pMsg, flags); @@ -734,7 +672,7 @@ finalize_it: */ rsRetVal parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int flags, flowControl_t flowCtlType, - uchar *pszInputName, struct syslogTime *stTime, time_t ttGenTime) + prop_t *pInputName, struct syslogTime *stTime, time_t ttGenTime) { DEFiRet; register int iMsg; @@ -845,7 +783,7 @@ parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int fla */ if(iMsg == iMaxLine) { *(pMsg + iMsg) = '\0'; /* space *is* reserved for this! */ - printline(hname, hnameIP, tmpline, flags, flowCtlType, pszInputName, stTime, ttGenTime); + printline(hname, hnameIP, tmpline, flags, flowCtlType, pInputName, stTime, ttGenTime); } else { /* This case in theory never can happen. If it happens, we have * a logic error. I am checking for it, because if I would not, @@ -854,7 +792,7 @@ parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int fla * (I couldn't do any more smart things anyway...). * rgerhards, 2007-9-20 */ - DBGPRINTF("internal error: iMsg > max msg size in printchopped()\n"); + DBGPRINTF("internal error: iMsg > max msg size in parseAndSubmitMessage()\n"); } FINALIZE; /* in this case, we are done... nothing left we can do */ } @@ -897,7 +835,7 @@ parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int fla *(pMsg + iMsg) = '\0'; /* space *is* reserved for this! */ /* typically, we should end up here! */ - printline(hname, hnameIP, tmpline, flags, flowCtlType, pszInputName, stTime, ttGenTime); + printline(hname, hnameIP, tmpline, flags, flowCtlType, pInputName, stTime, ttGenTime); finalize_it: if(tmpline != NULL) @@ -940,20 +878,20 @@ logmsgInternal(int iErr, int pri, uchar *msg, int flags) DEFiRet; CHKiRet(msgConstruct(&pMsg)); - MsgSetInputName(pMsg, UCHAR_CONSTANT("rsyslogd"), sizeof("rsyslogd")-1); - MsgSetRawMsg(pMsg, (char*)msg); - MsgSetHOSTNAME(pMsg, glbl.GetLocalHostName()); - MsgSetRcvFrom(pMsg, glbl.GetLocalHostName()); - MsgSetRcvFromIP(pMsg, UCHAR_CONSTANT("127.0.0.1")); + MsgSetInputName(pMsg, pInternalInputName); + MsgSetRawMsgWOSize(pMsg, (char*)msg); + MsgSetHOSTNAME(pMsg, glbl.GetLocalHostName(), ustrlen(glbl.GetLocalHostName())); + MsgSetRcvFrom(pMsg, glbl.GetLocalHostNameProp()); + MsgSetRcvFromIP(pMsg, pLocalHostIP); /* check if we have an error code associated and, if so, * adjust the tag. -- rgerhards, 2008-06-27 */ if(iErr == NO_ERRCODE) { - MsgSetTAG(pMsg, "rsyslogd:"); + MsgSetTAG(pMsg, UCHAR_CONSTANT("rsyslogd:"), sizeof("rsyslogd:") - 1); } else { - snprintf((char*)pszTag, sizeof(pszTag), "rsyslogd%d:", iErr); + size_t len = snprintf((char*)pszTag, sizeof(pszTag), "rsyslogd%d:", iErr); pszTag[32] = '\0'; /* just to make sure... */ - MsgSetTAG(pMsg, (char*)pszTag); + MsgSetTAG(pMsg, pszTag, len); } pMsg->iFacility = LOG_FAC(pri); pMsg->iSeverity = LOG_PRI(pri); @@ -985,233 +923,6 @@ finalize_it: RETiRet; } -/* This functions looks at the given message and checks if it matches the - * provided filter condition. If so, it returns true, else it returns - * false. This is a helper to logmsg() and meant to drive the decision - * process if a message is to be processed or not. As I expect this - * decision code to grow more complex over time AND logmsg() is already - * a very lengthy function, I thought a separate function is more appropriate. - * 2005-09-19 rgerhards - * 2008-02-25 rgerhards: changed interface, now utilizes iRet, bProcessMsg - * returns is message should be procesed. - */ -static rsRetVal shouldProcessThisMessage(selector_t *f, msg_t *pMsg, int *bProcessMsg) -{ - DEFiRet; - unsigned short pbMustBeFreed; - char *pszPropVal; - int bRet = 0; - vm_t *pVM = NULL; - var_t *pResult = NULL; - - assert(f != NULL); - assert(pMsg != NULL); - - /* we first have a look at the global, BSD-style block filters (for tag - * and host). Only if they match, we evaluate the actual filter. - * rgerhards, 2005-10-18 - */ - if(f->eHostnameCmpMode == HN_NO_COMP) { - /* EMPTY BY INTENSION - we check this value first, because - * it is the one most often used, so this saves us time! - */ - } else if(f->eHostnameCmpMode == HN_COMP_MATCH) { - if(rsCStrSzStrCmp(f->pCSHostnameComp, (uchar*) getHOSTNAME(pMsg), getHOSTNAMELen(pMsg))) { - /* not equal, so we are already done... */ - DBGPRINTF("hostname filter '+%s' does not match '%s'\n", - rsCStrGetSzStrNoNULL(f->pCSHostnameComp), getHOSTNAME(pMsg)); - FINALIZE; - } - } else { /* must be -hostname */ - if(!rsCStrSzStrCmp(f->pCSHostnameComp, (uchar*) getHOSTNAME(pMsg), getHOSTNAMELen(pMsg))) { - /* not equal, so we are already done... */ - DBGPRINTF("hostname filter '-%s' does not match '%s'\n", - rsCStrGetSzStrNoNULL(f->pCSHostnameComp), getHOSTNAME(pMsg)); - FINALIZE; - } - } - - if(f->pCSProgNameComp != NULL) { - int bInv = 0, bEqv = 0, offset = 0; - if(*(rsCStrGetSzStrNoNULL(f->pCSProgNameComp)) == '-') { - if(*(rsCStrGetSzStrNoNULL(f->pCSProgNameComp) + 1) == '-') - offset = 1; - else { - bInv = 1; - offset = 1; - } - } - if(!rsCStrOffsetSzStrCmp(f->pCSProgNameComp, offset, (uchar*) getProgramName(pMsg), getProgramNameLen(pMsg))) - bEqv = 1; - - if((!bEqv && !bInv) || (bEqv && bInv)) { - /* not equal or inverted selection, so we are already done... */ - DBGPRINTF("programname filter '%s' does not match '%s'\n", - rsCStrGetSzStrNoNULL(f->pCSProgNameComp), getProgramName(pMsg)); - FINALIZE; - } - } - - /* done with the BSD-style block filters */ - - if(f->f_filter_type == FILTER_PRI) { - /* skip messages that are incorrect priority */ - if ( (f->f_filterData.f_pmask[pMsg->iFacility] == TABLE_NOPRI) || - ((f->f_filterData.f_pmask[pMsg->iFacility] & (1<<pMsg->iSeverity)) == 0) ) - bRet = 0; - else - bRet = 1; - } else if(f->f_filter_type == FILTER_EXPR) { - CHKiRet(vm.Construct(&pVM)); - CHKiRet(vm.ConstructFinalize(pVM)); - CHKiRet(vm.SetMsg(pVM, pMsg)); - CHKiRet(vm.ExecProg(pVM, f->f_filterData.f_expr->pVmprg)); - CHKiRet(vm.PopBoolFromStack(pVM, &pResult)); - DBGPRINTF("result of expression evaluation: %lld\n", pResult->val.num); - /* VM is destructed on function exit */ - bRet = (pResult->val.num) ? 1 : 0; - } else { - assert(f->f_filter_type == FILTER_PROP); /* assert() just in case... */ - pszPropVal = MsgGetProp(pMsg, NULL, f->f_filterData.prop.pCSPropName, &pbMustBeFreed); - - /* Now do the compares (short list currently ;)) */ - switch(f->f_filterData.prop.operation ) { - case FIOP_CONTAINS: - if(rsCStrLocateInSzStr(f->f_filterData.prop.pCSCompValue, (uchar*) pszPropVal) != -1) - bRet = 1; - break; - case FIOP_ISEQUAL: - if(rsCStrSzStrCmp(f->f_filterData.prop.pCSCompValue, - (uchar*) pszPropVal, strlen(pszPropVal)) == 0) - bRet = 1; /* process message! */ - break; - case FIOP_STARTSWITH: - if(rsCStrSzStrStartsWithCStr(f->f_filterData.prop.pCSCompValue, - (uchar*) pszPropVal, strlen(pszPropVal)) == 0) - bRet = 1; /* process message! */ - break; - case FIOP_REGEX: - if(rsCStrSzStrMatchRegex(f->f_filterData.prop.pCSCompValue, - (unsigned char*) pszPropVal, 0, &f->f_filterData.prop.regex_cache) == RS_RET_OK) - bRet = 1; - break; - case FIOP_EREREGEX: - if(rsCStrSzStrMatchRegex(f->f_filterData.prop.pCSCompValue, - (unsigned char*) pszPropVal, 1, &f->f_filterData.prop.regex_cache) == RS_RET_OK) - bRet = 1; - break; - default: - /* here, it handles NOP (for performance reasons) */ - assert(f->f_filterData.prop.operation == FIOP_NOP); - bRet = 1; /* as good as any other default ;) */ - break; - } - - /* now check if the value must be negated */ - if(f->f_filterData.prop.isNegated) - bRet = (bRet == 1) ? 0 : 1; - - if(Debug) { - dbgprintf("Filter: check for property '%s' (value '%s') ", - rsCStrGetSzStrNoNULL(f->f_filterData.prop.pCSPropName), - pszPropVal); - if(f->f_filterData.prop.isNegated) - dbgprintf("NOT "); - dbgprintf("%s '%s': %s\n", - getFIOPName(f->f_filterData.prop.operation), - rsCStrGetSzStrNoNULL(f->f_filterData.prop.pCSCompValue), - bRet ? "TRUE" : "FALSE"); - } - - /* cleanup */ - if(pbMustBeFreed) - free(pszPropVal); - } - -finalize_it: - /* destruct in any case, not just on error, but it makes error handling much easier */ - if(pVM != NULL) - vm.Destruct(&pVM); - - if(pResult != NULL) - var.Destruct(&pResult); - - *bProcessMsg = bRet; - RETiRet; -} - - -/* helper to processMsg(), used to call the configured actions. It is - * executed from within llExecFunc() of the action list. - * rgerhards, 2007-08-02 - */ -typedef struct processMsgDoActions_s { - int bPrevWasSuspended; /* was the previous action suspended? */ - msg_t *pMsg; -} processMsgDoActions_t; -DEFFUNC_llExecFunc(processMsgDoActions) -{ - DEFiRet; - rsRetVal iRetMod; /* return value of module - we do not always pass that back */ - action_t *pAction = (action_t*) pData; - processMsgDoActions_t *pDoActData = (processMsgDoActions_t*) pParam; - - assert(pAction != NULL); - - if((pAction->bExecWhenPrevSusp == 1) && (pDoActData->bPrevWasSuspended == 0)) { - DBGPRINTF("not calling action because the previous one is not suspended\n"); - ABORT_FINALIZE(RS_RET_OK); - } - - iRetMod = actionCallAction(pAction, pDoActData->pMsg); - if(iRetMod == RS_RET_DISCARDMSG) { - ABORT_FINALIZE(RS_RET_DISCARDMSG); - } else if(iRetMod == RS_RET_SUSPENDED) { - /* indicate suspension for next module to be called */ - pDoActData->bPrevWasSuspended = 1; - } else { - pDoActData->bPrevWasSuspended = 0; - } - -finalize_it: - RETiRet; -} - - -/* Process (consume) a received message. Calls the actions configured. - * rgerhards, 2005-10-13 - */ -static void -processMsg(msg_t *pMsg) -{ - selector_t *f; - int bContinue; - int bProcessMsg; - processMsgDoActions_t DoActData; - rsRetVal iRet; - - BEGINfunc - assert(pMsg != NULL); - - /* log the message to the particular outputs */ - - bContinue = 1; - for (f = Files; f != NULL && bContinue ; f = f->f_next) { - /* first check the filters... */ - iRet = shouldProcessThisMessage(f, pMsg, &bProcessMsg); - if(!bProcessMsg) { - continue; - } - - /* ok -- from here, we have action-specific code, nothing really selector-specific -- rger 2007-08-01 */ - DoActData.pMsg = pMsg; - DoActData.bPrevWasSuspended = 0; - if(llExecFunc(&f->llActList, processMsgDoActions, (void*)&DoActData) == RS_RET_DISCARDMSG) - bContinue = 0; - } - ENDfunc -} - /* The consumer of dequeued messages. This function is called by the * queue engine on dequeueing of a message. It runs on a SEPARATE @@ -1229,7 +940,7 @@ msgConsumer(void __attribute__((unused)) *notNeeded, void *pUsr) if((pMsg->msgFlags & NEEDS_PARSING) != 0) { parseMsg(pMsg); } - processMsg(pMsg); + ruleset.ProcessMsg(pMsg); msgDestruct(&pMsg); RETiRet; @@ -1330,7 +1041,7 @@ static int parseRFCStructuredData(uchar **pp2parse, uchar *pResult, int *pLenStr iRet = 1; /* this is not valid! */ bCont = 0; } - } else if(*p2parse == '\\' && *(p2parse+1) == ']') { + } else if(*p2parse == '\\' && *(p2parse+1) == ']') { /* this is escaped, need to copy both */ *pResult++ = *p2parse++; *pResult++ = *p2parse++; @@ -1362,7 +1073,7 @@ static int parseRFCStructuredData(uchar **pp2parse, uchar *pResult, int *pLenStr return 0; } -/* parse a RFC-formatted syslog message. This function returns +/* parse a RFC5424-formatted syslog message. This function returns * 0 if processing of the message shall continue and 1 if something * went wrong and this messe should be ignored. This function has been * implemented in the effort to support syslog-protocol. Please note that @@ -1426,12 +1137,7 @@ int parseRFCSyslogMsg(msg_t *pMsg, int flags) /* HOSTNAME */ if(bContParse) { parseRFCField(&p2parse, pBuf, &lenMsg); - MsgSetHOSTNAME(pMsg, pBuf); - } else { - /* we can not parse, so we get the system we - * received the data from. - */ - MsgSetHOSTNAME(pMsg, getRcvFrom(pMsg)); + MsgSetHOSTNAME(pMsg, pBuf, ustrlen(pBuf)); } /* APP-NAME */ @@ -1459,7 +1165,7 @@ int parseRFCSyslogMsg(msg_t *pMsg, int flags) } /* MSG */ - MsgSetMSG(pMsg, (lenMsg == 0) ? "" : (char*)p2parse); + MsgSetMSGoffs(pMsg, p2parse - pMsg->pszRawMsg); free(pBuf); ENDfunc @@ -1483,23 +1189,20 @@ int parseRFCSyslogMsg(msg_t *pMsg, int flags) int parseLegacySyslogMsg(msg_t *pMsg, int flags) { uchar *p2parse; - char *pBuf; - char *pWork; - cstr_t *pStrB; - int iCnt; int lenMsg; int bTAGCharDetected; + int i; /* general index for parsing */ + uchar bufParseTAG[CONF_TAG_MAXSIZE]; + uchar bufParseHOSTNAME[CONF_HOSTNAME_MAXSIZE]; BEGINfunc assert(pMsg != NULL); assert(pMsg->pszRawMsg != NULL); - lenMsg = pMsg->iLenRawMsg - (pMsg->offAfterPRI + 1); -RUNLOG_VAR("%d", pMsg->offAfterPRI); -RUNLOG_VAR("%d", lenMsg); + lenMsg = pMsg->iLenRawMsg - pMsg->offAfterPRI; /* note: offAfterPRI is already the number of PRI chars (do not add one!) */ p2parse = pMsg->pszRawMsg + pMsg->offAfterPRI; /* point to start of text, after PRI */ /* Check to see if msg contains a timestamp. We start by assuming - * that the message timestamp is the time of reciption (which we + * that the message timestamp is the time of reception (which we * generated ourselfs and then try to actually find one inside the * message. There we go from high-to low precison and are done * when we find a matching one. -- rgerhards, 2008-09-16 @@ -1514,8 +1217,7 @@ RUNLOG_VAR("%d", lenMsg); if(datetime.ParseTIMESTAMP3164(&(pMsg->tTIMESTAMP), &p2parse, &lenMsg) == RS_RET_OK) { /* indeed, we got it! */ /* we are done - parse pointer is moved by ParseTIMESTAMP3164 */; - } else { - /* parse pointer needs to be restored, as we moved it off-by-one + } else {/* parse pointer needs to be restored, as we moved it off-by-one * for this try. */ --p2parse; @@ -1546,55 +1248,33 @@ RUNLOG_VAR("%d", lenMsg); * If I find them, I set a simple flag but continue. After parsing, I check the flag. * If it was set, then we most probably do not have a hostname but a TAG. Thus, I change * the fields. I think this logic shall work with any type of syslog message. + * rgerhards, 2009-06-23: and I now have extended this logic to every character + * that is not a valid hostname. */ bTAGCharDetected = 0; - if(flags & PARSE_HOSTNAME) { - /* TODO: quick and dirty memory allocation */ - /* the memory allocated is far too much in most cases. But on the plus side, - * it is quite fast... - rgerhards, 2007-09-20 - */ - if((pBuf = malloc(sizeof(char) * (lenMsg + 1))) == NULL) - return 1; - pWork = pBuf; - /* this is the actual parsing loop */ - while(lenMsg > 0 && *p2parse != ' ' && *p2parse != ':') { - if(*p2parse == '[' || *p2parse == ']' || *p2parse == '/') - bTAGCharDetected = 1; - *pWork++ = *p2parse++; - --lenMsg; + if(lenMsg > 0 && flags & PARSE_HOSTNAME) { + i = 0; + while(i < lenMsg && (isalnum(p2parse[i]) || p2parse[i] == '.' || p2parse[i] == '.' + || p2parse[i] == '_' || p2parse[i] == '-') && i < (CONF_HOSTNAME_MAXSIZE - 1)) { + bufParseHOSTNAME[i] = p2parse[i]; + ++i; } - /* we need to handle ':' seperately, because it terminates the - * TAG - so we also need to terminate the parser here! - * rgerhards, 2007-09-10 *p2parse points to a valid address here in - * any case. We can reach this point only if we are at end of string, - * or we have a ':' or ' '. What the if below does is check if we are - * not at end of string and, if so, advance the parse pointer. If we - * are already at end of string, *p2parse is equal to '\0', neither if - * will be true and the parse pointer remain as is. This is perfectly - * well. - */ - if(lenMsg > 0) { - if(*p2parse == ':') { - bTAGCharDetected = 1; - /* We will move hostname to tag, so preserve ':' (otherwise we - * will needlessly change the message format) */ - *pWork++ = *p2parse++; - } else if(*p2parse == ' ') { - ++p2parse; - } - --lenMsg; + + if(i == lenMsg) { + /* we have a message that is empty immediately after the hostname, + * but the hostname thus is valid! -- rgerhards, 2010-02-22 + */ + p2parse += i; + lenMsg -= i; + bufParseHOSTNAME[i] = '\0'; + MsgSetHOSTNAME(pMsg, bufParseHOSTNAME, i); + } else if(i > 0 && p2parse[i] == ' ' && isalnum(p2parse[i-1])) { + /* we got a hostname! */ + p2parse += i + 1; /* "eat" it (including SP delimiter) */ + lenMsg -= i + 1; + bufParseHOSTNAME[i] = '\0'; + MsgSetHOSTNAME(pMsg, bufParseHOSTNAME, i); } - *pWork = '\0'; - MsgAssignHOSTNAME(pMsg, pBuf); - } - /* check if we seem to have a TAG */ - if(bTAGCharDetected) { - /* indeed, this smells like a TAG, so lets use it for this. We take - * the HOSTNAME from the sender system instead. - */ - DBGPRINTF("HOSTNAME contains invalid characters, assuming it to be a TAG.\n"); - moveHOSTNAMEtoTAG(pMsg); - MsgSetHOSTNAME(pMsg, getRcvFrom(pMsg)); } /* now parse TAG - that should be present in message from all sources. @@ -1610,70 +1290,39 @@ RUNLOG_VAR("%d", lenMsg); * in RFC3164...). We now receive the full size, but will modify the * outputs so that only 32 characters max are used by default. */ - /* The following code in general is quick & dirty - I need to get - * it going for a test, rgerhards 2004-11-16 */ - /* lol.. we tried to solve it, just to remind ourselfs that 32 octets - * is the max size ;) we need to shuffle the code again... Just for - * the records: the code is currently clean, but we could optimize it! */ - if(!bTAGCharDetected) { - uchar *pszTAG; - if(cstrConstruct(&pStrB) != RS_RET_OK) - return 1; - rsCStrSetAllocIncrement(pStrB, 33); - pWork = pBuf; - iCnt = 0; - while(lenMsg > 0 && *p2parse != ':' && *p2parse != ' ') { - cstrAppendChar(pStrB, *p2parse++); - ++iCnt; - --lenMsg; - } - if(lenMsg > 0 && *p2parse == ':') { - ++p2parse; - --lenMsg; - cstrAppendChar(pStrB, ':'); - } - cstrFinalize(pStrB); - cstrConvSzStrAndDestruct(pStrB, &pszTAG, 1); - if(pszTAG == NULL) - { /* rger, 2005-11-10: no TAG found - this implies that what - * we have considered to be the HOSTNAME is most probably the - * TAG. We consider it so probable, that we now adjust it - * that way. So we pick up the previously set hostname, assign - * it to tag and use the sender system (from IP stack) as - * the hostname. This situation is the standard case with - * stock BSD syslogd. - */ - DBGPRINTF("No TAG in message, assuming that HOSTNAME is missing.\n"); - moveHOSTNAMEtoTAG(pMsg); - MsgSetHOSTNAME(pMsg, getRcvFrom(pMsg)); - } else { /* we have a TAG, so we can happily set it ;) */ - MsgAssignTAG(pMsg, pszTAG); - } - } else { - /* we have no TAG, so we ... */ - /*DO NOTHING*/; + i = 0; + while(lenMsg > 0 && *p2parse != ':' && *p2parse != ' ' && i < CONF_TAG_MAXSIZE) { + bufParseTAG[i++] = *p2parse++; + --lenMsg; } - } else { - /* we enter this code area when the user has instructed rsyslog NOT + if(lenMsg > 0 && *p2parse == ':') { + ++p2parse; + --lenMsg; + bufParseTAG[i++] = ':'; + } + + /* no TAG can only be detected if the message immediatly ends, in which case an empty TAG + * is considered OK. So we do not need to check for empty TAG. -- rgerhards, 2009-06-23 + */ + bufParseTAG[i] = '\0'; /* terminate string */ + MsgSetTAG(pMsg, bufParseTAG, i); + } else {/* we enter this code area when the user has instructed rsyslog NOT * to parse HOSTNAME and TAG - rgerhards, 2006-03-13 */ - if(!(flags & INTERNAL_MSG)) - { + if(!(flags & INTERNAL_MSG)) { DBGPRINTF("HOSTNAME and TAG not parsed by user configuraton.\n"); - MsgSetHOSTNAME(pMsg, getRcvFrom(pMsg)); } } /* The rest is the actual MSG */ - MsgSetMSG(pMsg, (lenMsg == 0) ? "" : (char*)p2parse); + MsgSetMSGoffs(pMsg, p2parse - pMsg->pszRawMsg); ENDfunc return 0; /* all ok */ } -/* submit a fully created message to the main message queue. The message is - * fully processed and parsed, so no parsing at all happens. This is primarily +/* submit a message to the main message queue. This is primarily * a hook to prevent the need for callers to know about the main message queue * (which may change in the future as we will probably have multiple rule * sets and thus queues...). @@ -1693,6 +1342,28 @@ submitMsg(msg_t *pMsg) } +/* submit multiple messages at once, very similar to submitMsg, just + * for multi_submit_t. + * rgerhards, 2009-06-16 + */ +rsRetVal +multiSubmitMsg(multi_submit_t *pMultiSub) +{ + int i; + DEFiRet; + assert(pMultiSub != NULL); + + for(i = 0 ; i < pMultiSub->nElem ; ++i) { + MsgPrepareEnqueue(pMultiSub->ppMsgs[i]); + } + + iRet = qqueueMultiEnqObj(pMsgQueue, pMultiSub); + pMultiSub->nElem = 0; + + RETiRet; +} + + /* Log a message to the appropriate log files, users, etc. based on * the priority. * rgerhards 2004-11-08: actually, this also decodes all but the PRI part. @@ -1776,7 +1447,6 @@ reapchild() DEFFUNC_llExecFunc(flushRptdMsgsActions) { action_t *pAction = (action_t*) pData; - assert(pAction != NULL); BEGINfunc @@ -1799,32 +1469,38 @@ DEFFUNC_llExecFunc(flushRptdMsgsActions) } -/* This method flushes reapeat messages. +/* This method flushes repeat messages. */ static void doFlushRptdMsgs(void) { - register selector_t *f; - - /* see if we need to flush any "message repeated n times"... - * Note that this interferes with objects running on other threads. - * We are using appropriate locking inside the function to handle that. - */ - for (f = Files; f != NULL ; f = f->f_next) { - llExecFunc(&f->llActList, flushRptdMsgsActions, NULL); - } + ruleset.IterateAllActions(flushRptdMsgsActions, NULL); } static void debug_switch() { + time_t tTime; + struct tm tp; struct sigaction sigAct; + time(&tTime); + localtime_r(&tTime, &tp); if(debugging_on == 0) { debugging_on = 1; - DBGPRINTF("Switching debugging_on to true\n"); + dbgprintf("\n"); + dbgprintf("\n"); + dbgprintf("********************************************************************************\n"); + dbgprintf("Switching debugging_on to true at %2.2d:%2.2d:%2.2d\n", + tp.tm_hour, tp.tm_min, tp.tm_sec); + dbgprintf("********************************************************************************\n"); } else { - DBGPRINTF("Switching debugging_on to false\n"); + dbgprintf("********************************************************************************\n"); + dbgprintf("Switching debugging_on to false at %2.2d:%2.2d:%2.2d\n", + tp.tm_hour, tp.tm_min, tp.tm_sec); + dbgprintf("********************************************************************************\n"); + dbgprintf("\n"); + dbgprintf("\n"); debugging_on = 0; } @@ -2003,6 +1679,16 @@ freeAllDynMemForTermination(void) } +/* Finalize and destruct all actions. + */ +static inline void +destructAllActions(void) +{ + ruleset.DestructAllActions(); + bHaveMainQueue = 0; // flag that internal messages need to be temporarily stored +} + + /* die() is called when the program shall end. This typically only occurs * during sigterm or during the initialization. * As die() is intended to shutdown rsyslogd, it is @@ -2053,7 +1739,7 @@ die(int sig) * repeated msgs. */ DBGPRINTF("Terminating outputs...\n"); - freeSelectors(); + destructAllActions(); DBGPRINTF("all primary multi-thread sources have been terminated - now doing aux cleanup...\n"); /* rger 2005-02-22 @@ -2078,6 +1764,12 @@ die(int sig) legacyOptsFree(); + /* destruct our global properties */ + if(pInternalInputName != NULL) + prop.Destruct(&pInternalInputName); + if(pLocalHostIP != NULL) + prop.Destruct(&pLocalHostIP); + /* terminate the remaining classes */ GlobalClassExit(); @@ -2089,7 +1781,7 @@ die(int sig) * rgerhards, 2007-08-03 * I have added some code now, but all that mod init/de-init should be moved to * init, so that modules are unloaded and reloaded on HUP to. Eventually it should go - * into freeSelectors() - but that needs to be seen. -- rgerhards, 2007-08-09 + * into destructAllActions() - but that needs to be seen. -- rgerhards, 2007-08-09 */ module.UnloadAndDestructAll(eMOD_LINK_ALL); @@ -2118,7 +1810,7 @@ static void doexit() /* set the maximum message size */ -static rsRetVal setMaxMsgSize(void __attribute__((unused)) *pVal, int iNewVal) +static rsRetVal setMaxMsgSize(void __attribute__((unused)) *pVal, long iNewVal) { return glbl.SetMaxLine(iNewVal); } @@ -2217,56 +1909,6 @@ static void doDropPrivUid(int iUid) } -/* helper to freeSelectors(), used with llExecFunc() to flush - * pending output. -- rgerhards, 2007-08-02 - * We do not need to lock the action object here as the processing - * queue is already empty and no other threads are running when - * we call this function. -- rgerhards, 2007-12-12 - */ -DEFFUNC_llExecFunc(freeSelectorsActions) -{ - action_t *pAction = (action_t*) pData; - - assert(pAction != NULL); - - /* flush any pending output */ - if(pAction->f_prevcount) { - actionWriteToAction(pAction); - } - - return RS_RET_OK; /* never fails ;) */ -} - - -/* Close all open log files and free selector descriptor array. - */ -static void freeSelectors(void) -{ - selector_t *f; - selector_t *fPrev; - - if(Files != NULL) { - DBGPRINTF("Freeing log structures.\n"); - - for(f = Files ; f != NULL ; f = f->f_next) { - llExecFunc(&f->llActList, freeSelectorsActions, NULL); - } - - /* actions flushed and ready for destruction - so do that... */ - f = Files; - while (f != NULL) { - fPrev = f; - f = f->f_next; - selectorDestruct(fPrev); - } - - /* Reflect the deletion of the selectors linked list. */ - Files = NULL; - bHaveMainQueue = 0; - } -} - - /* helper to generateConfigDAG, to print out all actions via * the llExecFunc() facility. * rgerhards, 2007-08-02 @@ -2343,14 +1985,14 @@ DEFFUNC_llExecFunc(generateConfigDAGAction) static rsRetVal generateConfigDAG(uchar *pszDAGFile) { - selector_t *f; + //rule_t *f; FILE *fp; int iActUnit = 1; - int bHasFilter = 0; /* filter associated with this action unit? */ - int bHadFilter; - int i; + //int bHasFilter = 0; /* filter associated with this action unit? */ + //int bHadFilter; + //int i; struct dag_info dagInfo; - char *pszFilterName; + //char *pszFilterName; char szConnectingNode[64]; DEFiRet; @@ -2379,6 +2021,8 @@ generateConfigDAG(uchar *pszDAGFile) strcpy(szConnectingNode, "act0_0"); dagInfo.bDiscarded = 0; +/* TODO: re-enable! */ +#if 0 for(f = Files; f != NULL ; f = f->f_next) { /* BSD-Style filters are currently ignored */ bHadFilter = bHasFilter; @@ -2434,6 +2078,7 @@ generateConfigDAG(uchar *pszDAGFile) ++iActUnit; } +#endif fprintf(fp, "\t%s -> act%d_0\n", szConnectingNode, iActUnit); fprintf(fp, "\tact%d_0\t\t[label=discard shape=box]\n" @@ -2445,20 +2090,6 @@ finalize_it: } -/* helper to dbPrintInitInfo, to print out all actions via - * the llExecFunc() facility. - * rgerhards, 2007-08-02 - */ -DEFFUNC_llExecFunc(dbgPrintInitInfoAction) -{ - DEFiRet; - iRet = actionDbgPrint((action_t*) pData); - DBGPRINTF("\n"); - - RETiRet; -} - - /* print debug information as part of init(). This pretty much * outputs the whole config of rsyslogd. I've moved this code * out of init() to clean it somewhat up. @@ -2466,47 +2097,7 @@ DEFFUNC_llExecFunc(dbgPrintInitInfoAction) */ static void dbgPrintInitInfo(void) { - selector_t *f; - int iSelNbr = 1; - int i; - - DBGPRINTF("\nActive selectors:\n"); - for (f = Files; f != NULL ; f = f->f_next) { - DBGPRINTF("Selector %d:\n", iSelNbr++); - if(f->pCSProgNameComp != NULL) - DBGPRINTF("tag: '%s'\n", rsCStrGetSzStrNoNULL(f->pCSProgNameComp)); - if(f->eHostnameCmpMode != HN_NO_COMP) - DBGPRINTF("hostname: %s '%s'\n", - f->eHostnameCmpMode == HN_COMP_MATCH ? - "only" : "allbut", - rsCStrGetSzStrNoNULL(f->pCSHostnameComp)); - if(f->f_filter_type == FILTER_PRI) { - for (i = 0; i <= LOG_NFACILITIES; i++) - if (f->f_filterData.f_pmask[i] == TABLE_NOPRI) { - DBGPRINTF(" X "); - } else { - DBGPRINTF("%2X ", f->f_filterData.f_pmask[i]); - } - } else if(f->f_filter_type == FILTER_EXPR) { - DBGPRINTF("EXPRESSION-BASED Filter: can currently not be displayed"); - } else { - DBGPRINTF("PROPERTY-BASED Filter:\n"); - DBGPRINTF("\tProperty.: '%s'\n", - rsCStrGetSzStrNoNULL(f->f_filterData.prop.pCSPropName)); - DBGPRINTF("\tOperation: "); - if(f->f_filterData.prop.isNegated) - DBGPRINTF("NOT "); - DBGPRINTF("'%s'\n", getFIOPName(f->f_filterData.prop.operation)); - DBGPRINTF("\tValue....: '%s'\n", - rsCStrGetSzStrNoNULL(f->f_filterData.prop.pCSCompValue)); - DBGPRINTF("\tAction...: "); - } - - DBGPRINTF("\nActions:\n"); - llExecFunc(&f->llActList, dbgPrintInitInfoAction, NULL); /* actions */ - - DBGPRINTF("\n"); - } + ruleset.DebugPrintAll(); DBGPRINTF("\n"); if(bDebugPrintTemplateList) tplPrintList(); @@ -2549,6 +2140,30 @@ static void dbgPrintInitInfo(void) } +/* Actually run the input modules. This happens after privileges are dropped, + * if that is requested. + */ +static rsRetVal +runInputModules(void) +{ + modInfo_t *pMod; + + BEGINfunc + /* loop through all modules and activate them (brr...) */ + pMod = module.GetNxtType(NULL, eMOD_IN); + while(pMod != NULL) { + if(pMod->mod.im.bCanRun) { + /* activate here */ + thrdCreate(pMod->mod.im.runInput, pMod->mod.im.afterRun); + } + pMod = module.GetNxtType(pMod, eMOD_IN); + } + + ENDfunc + return RS_RET_OK; /* intentional: we do not care about module errors */ +} + + /* Start the input modules. This function will probably undergo big changes * while we implement the input module interface. For now, it does the most * important thing to get at least my poor initial input modules up and @@ -2564,10 +2179,9 @@ startInputModules(void) /* loop through all modules and activate them (brr...) */ pMod = module.GetNxtType(NULL, eMOD_IN); while(pMod != NULL) { - if((iRet = pMod->mod.im.willRun()) == RS_RET_OK) { - /* activate here */ - thrdCreate(pMod->mod.im.runInput, pMod->mod.im.afterRun); - } else { + iRet = pMod->mod.im.willRun(); + pMod->mod.im.bCanRun = (iRet == RS_RET_OK); + if(!pMod->mod.im.bCanRun) { DBGPRINTF("module %lx will not run, iRet %d\n", (unsigned long) pMod, iRet); } pMod = module.GetNxtType(pMod, eMOD_IN); @@ -2584,15 +2198,16 @@ startInputModules(void) * else happens. -- rgerhards, 2008-07-28 */ static rsRetVal -init(void) +init() { - DEFiRet; rsRetVal localRet; int iNbrActions; int bHadConfigErr = 0; + ruleset_t *pRuleset; char cbuf[BUFSIZ]; char bufStartUpMsg[512]; struct sigaction sigAct; + DEFiRet; thrdTerminateAll(); /* stop all running input threads - TODO: reconsider location! */ @@ -2613,7 +2228,7 @@ init(void) /* Close all open log files and free log descriptor array. This also frees * all output-modules instance data. */ - freeSelectors(); + destructAllActions(); /* Unload all non-static modules */ DBGPRINTF("Unloading non-static modules.\n"); @@ -2634,6 +2249,11 @@ init(void) conf.ReInitConf(); + /* construct the default ruleset */ + ruleset.Construct(&pRuleset); + ruleset.SetName(pRuleset, UCHAR_CONSTANT("RSYSLOG_DefaultRuleset")); + ruleset.ConstructFinalize(pRuleset); + /* open the configuration file */ localRet = conf.processConfFile(ConfFile); CHKiRet(conf.GetNbrActActions(&iNbrActions)); @@ -2655,23 +2275,23 @@ init(void) * We ignore any errors while doing this - we would be lost anyhow... */ errmsg.LogError(0, NO_ERRCODE, "EMERGENCY CONFIGURATION ACTIVATED - fix rsyslog config file!"); - selector_t *f = NULL; /* note: we previously used _POSIY_TTY_NAME_MAX+1, but this turned out to be * too low on linux... :-S -- rgerhards, 2008-07-28 */ char szTTYNameBuf[128]; - conf.cfline((uchar*)"*.ERR\t" _PATH_CONSOLE, &f); - conf.cfline((uchar*)"syslog.*\t" _PATH_CONSOLE, &f); - conf.cfline((uchar*)"*.PANIC\t*", &f); - conf.cfline((uchar*)"syslog.*\troot", &f); + rule_t *pRule = NULL; /* initialization to NULL is *vitally* important! */ + conf.cfline(UCHAR_CONSTANT("*.ERR\t" _PATH_CONSOLE), &pRule); + conf.cfline(UCHAR_CONSTANT("syslog.*\t" _PATH_CONSOLE), &pRule); + conf.cfline(UCHAR_CONSTANT("*.PANIC\t*"), &pRule); + conf.cfline(UCHAR_CONSTANT("syslog.*\troot"), &pRule); if(ttyname_r(0, szTTYNameBuf, sizeof(szTTYNameBuf)) == 0) { snprintf(cbuf,sizeof(cbuf), "*.*\t%s", szTTYNameBuf); - conf.cfline((uchar*)cbuf, &f); + conf.cfline((uchar*)cbuf, &pRule); } else { DBGPRINTF("error %d obtaining controlling terminal, not using that emergency rule\n", errno); } - selectorAddList(f); + ruleset.AddRule(ruleset.GetCurrent(), &pRule); } legacyOptsHook(); @@ -2734,7 +2354,7 @@ init(void) exit(1); } /* name our main queue object (it's not fatal if it fails...) */ - obj.SetName((obj_t*) pMsgQueue, (uchar*) "main queue"); + obj.SetName((obj_t*) pMsgQueue, (uchar*) "main Q"); /* ... set some properties ... */ # define setQPROP(func, directive, data) \ @@ -2750,6 +2370,7 @@ init(void) setQPROP(qqueueSetsizeOnDiskMax, "$MainMsgQueueMaxDiskSpace", iMainMsgQueMaxDiskSpace); setQPROPstr(qqueueSetFilePrefix, "$MainMsgQueueFileName", pszMainMsgQFName); setQPROP(qqueueSetiPersistUpdCnt, "$MainMsgQueueCheckpointInterval", iMainMsgQPersistUpdCnt); + setQPROP(qqueueSetbSyncQueueFiles, "$MainMsgQueueSyncQueueFiles", bMainMsgQSyncQeueFiles); setQPROP(qqueueSettoQShutdown, "$MainMsgQueueTimeoutShutdown", iMainMsgQtoQShutdown ); setQPROP(qqueueSettoActShutdown, "$MainMsgQueueTimeoutActionCompletion", iMainMsgQtoActShutdown); setQPROP(qqueueSettoWrkShutdown, "$MainMsgQueueWorkerTimeoutThreadShutdown", iMainMsgQtoWrkShutdown); @@ -2778,9 +2399,12 @@ init(void) DBGPRINTF("Main processing queue is initialized and running\n"); /* the output part and the queue is now ready to run. So it is a good time - * to start the inputs. Please note that the net code above should be + * to initialize the inputs. Please note that the net code above should be * shuffled to down here once we have everything in input modules. * rgerhards, 2007-12-14 + * NOTE: as of 2009-06-29, the input modules are initialized, but not yet run. + * Keep in mind. though, that the outputs already run if the queue was + * persisted to disk. -- rgerhards */ startInputModules(); @@ -2809,49 +2433,48 @@ finalize_it: } -/* add a completely-processed selector (after config line parsing) to - * the linked list of selectors. We now need to check - * if it has any actions associated and, if so, link it to the linked - * list. If it has nothing associated with it, we can simply discard - * it. - * We have one special case during initialization: then, the current - * selector is NULL, which means we do not need to care about it at - * all. -- rgerhards, 2007-08-01 +/* Switch the default ruleset (that, what servcies bind to if nothing specific + * is specified). + * rgerhards, 2009-06-12 */ -rsRetVal -selectorAddList(selector_t *f) +static rsRetVal +setDefaultRuleset(void __attribute__((unused)) *pVal, uchar *pszName) { DEFiRet; - int iActionCnt; - static selector_t *nextp = NULL; /* TODO: make this go away (see comment below) */ + CHKiRet(ruleset.SetDefaultRuleset(pszName)); - if(f != NULL) { - CHKiRet(llGetNumElts(&f->llActList, &iActionCnt)); - if(iActionCnt == 0) { - errmsg.LogError(0, NO_ERRCODE, "warning: selector line without actions will be discarded"); - selectorDestruct(f); - } else { - /* successfully created an entry */ - DBGPRINTF("selector line successfully processed\n"); - /* TODO: we should use the linked list class for the selector list, else we need to add globals - * ... well nextp could be added temporarily... - * Thanks to varmojfekoj for having the idea to just use "Files" to make this - * code work. I had actually forgotten to fix the code here before moving to 1.18.0. - * And, of course, I also did not migrate the selector_t structure to the linked list class. - * However, that should still be one of the very next things to happen. - * rgerhards, 2007-08-06 - */ - if(Files == NULL) { - Files = f; - } else { - nextp->f_next = f; - } - nextp = f; - } +finalize_it: + free(pszName); /* no longer needed */ + RETiRet; +} + + +/* Switch to either an already existing rule set or start a new one. The + * named rule set becomes the new "current" rule set (what means that new + * actions are added to it). + * rgerhards, 2009-06-12 + */ +static rsRetVal +setCurrRuleset(void __attribute__((unused)) *pVal, uchar *pszName) +{ + ruleset_t *pRuleset; + rsRetVal localRet; + DEFiRet; + + localRet = ruleset.SetCurrRuleset(pszName); + + if(localRet == RS_RET_NOT_FOUND) { + DBGPRINTF("begin new current rule set '%s'\n", pszName); + CHKiRet(ruleset.Construct(&pRuleset)); + CHKiRet(ruleset.SetName(pRuleset, pszName)); + CHKiRet(ruleset.ConstructFinalize(pRuleset)); + } else { + ABORT_FINALIZE(localRet); } finalize_it: + free(pszName); /* no longer needed */ RETiRet; } @@ -2943,7 +2566,6 @@ DEFFUNC_llExecFunc(doHUPActions) static inline void doHUP(void) { - selector_t *f; char buf[512]; snprintf(buf, sizeof(buf) / sizeof(char), @@ -2956,11 +2578,10 @@ doHUP(void) if(glbl.GetHUPisRestart()) { DBGPRINTF("Received SIGHUP, configured to be restart, reloading rsyslogd.\n"); init(); /* main queue is stopped as part of init() */ + runInputModules(); } else { DBGPRINTF("Received SIGHUP, configured to be a non-restart type of HUP - notifying actions.\n"); - for(f = Files; f != NULL ; f = f->f_next) { - llExecFunc(&f->llActList, doHUPActions, NULL); - } + ruleset.IterateAllActions(doHUPActions, NULL); } } @@ -2992,6 +2613,7 @@ mainloop(void) * but a once-a-day wakeup should be quite acceptable. -- rgerhards, 2008-06-09 */ tvSelectTimeout.tv_sec = (bReduceRepeatMsgs == 1) ? TIMERINTVL : 86400 /*1 day*/; + //tvSelectTimeout.tv_sec = TIMERINTVL; /* TODO: change this back to the above code when we have a better solution for apc */ tvSelectTimeout.tv_usec = 0; select(1, NULL, NULL, NULL, &tvSelectTimeout); if(bFinished) @@ -3026,49 +2648,11 @@ mainloop(void) bHadHUP = 0; continue; } + // TODO: remove execScheduled(); /* handle Apc calls (if any) */ } ENDfunc } -/* If user is not root, prints warnings or even exits - * TODO: check all dynafiles for write permission - * ... but it is probably better to wait here until we have - * a module interface - rgerhards, 2007-07-23 - */ -static void checkPermissions() -{ -#if 0 - /* TODO: this function must either be redone or removed - now with the input modules, - * there is no such simple check we can do. What we can check, however, is if there is - * any input module active and terminate, if not. -- rgerhards, 2007-12-26 - */ - /* we are not root */ - if (geteuid() != 0) - { - fputs("WARNING: Local messages will not be logged! If you want to log them, run rsyslog as root.\n",stderr); -#ifdef SYSLOG_INET - /* udp enabled and port number less than or equal to 1024 */ - if ( AcceptRemote && (atoi(LogPort) <= 1024) ) - fprintf(stderr, "WARNING: Will not listen on UDP port %s. Use port number higher than 1024 or run rsyslog as root!\n", LogPort); - - /* tcp enabled and port number less or equal to 1024 */ - if( bEnableTCP && (atoi(TCPLstnPort) <= 1024) ) - fprintf(stderr, "WARNING: Will not listen on TCP port %s. Use port number higher than 1024 or run rsyslog as root!\n", TCPLstnPort); - - /* Neither explicit high UDP port nor explicit high TCP port. - * It is useless to run anymore */ - if( !(AcceptRemote && (atoi(LogPort) > 1024)) && !( bEnableTCP && (atoi(TCPLstnPort) > 1024)) ) - { -#endif - fprintf(stderr, "ERROR: Nothing to log, no reason to run. Please run rsyslog as root.\n"); - exit(EXIT_FAILURE); -#ifdef SYSLOG_INET - } -#endif - } -#endif -} - /* load build-in modules * very first version begun on 2007-07-23 by rgerhards @@ -3077,23 +2661,26 @@ static rsRetVal loadBuildInModules(void) { DEFiRet; - if((iRet = module.doModInit(modInitFile, (uchar*) "builtin-file", NULL)) != RS_RET_OK) { + if((iRet = module.doModInit(modInitFile, UCHAR_CONSTANT("builtin-file"), NULL)) != RS_RET_OK) { + RETiRet; + } + if((iRet = module.doModInit(modInitPipe, UCHAR_CONSTANT("builtin-pipe"), NULL)) != RS_RET_OK) { RETiRet; } #ifdef SYSLOG_INET - if((iRet = module.doModInit(modInitFwd, (uchar*) "builtin-fwd", NULL)) != RS_RET_OK) { + if((iRet = module.doModInit(modInitFwd, UCHAR_CONSTANT("builtin-fwd"), NULL)) != RS_RET_OK) { RETiRet; } #endif - if((iRet = module.doModInit(modInitShell, (uchar*) "builtin-shell", NULL)) != RS_RET_OK) { + if((iRet = module.doModInit(modInitShell, UCHAR_CONSTANT("builtin-shell"), NULL)) != RS_RET_OK) { RETiRet; } - if((iRet = module.doModInit(modInitDiscard, (uchar*) "builtin-discard", NULL)) != RS_RET_OK) { + if((iRet = module.doModInit(modInitDiscard, UCHAR_CONSTANT("builtin-discard"), NULL)) != RS_RET_OK) { RETiRet; } /* dirty, but this must be for the time being: the usrmsg module must always be - * loaded as last module. This is because it processes any time of action selector. + * loaded as last module. This is because it processes any type of action selector. * If we load it before other modules, these others will never have a chance of * working with the config file. We may change that implementation so that a user name * must start with an alnum, that would definitely help (but would it break backwards @@ -3101,8 +2688,7 @@ static rsRetVal loadBuildInModules(void) * User names now must begin with: * [a-zA-Z0-9_.] */ - if((iRet = module.doModInit(modInitUsrMsg, (uchar*) "builtin-usrmsg", NULL)) != RS_RET_OK) - RETiRet; + CHKiRet(module.doModInit(modInitUsrMsg, (uchar*) "builtin-usrmsg", NULL)); /* ok, initialization of the command handler probably does not 100% belong right in * this space here. However, with the current design, this is actually quite a good @@ -3112,6 +2698,8 @@ static rsRetVal loadBuildInModules(void) * This, I think, is the right thing to do. -- rgerhards, 2007-07-31 */ CHKiRet(regCfSysLineHdlr((uchar *)"actionresumeretrycount", 0, eCmdHdlrInt, NULL, &glbliActionResumeRetryCount, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"defaultruleset", 0, eCmdHdlrGetWord, setDefaultRuleset, NULL, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"ruleset", 0, eCmdHdlrGetWord, setCurrRuleset, NULL, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"mainmsgqueuefilename", 0, eCmdHdlrGetWord, NULL, &pszMainMsgQFName, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"mainmsgqueuesize", 0, eCmdHdlrInt, NULL, &iMainMsgQueueSize, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"mainmsgqueuehighwatermark", 0, eCmdHdlrInt, NULL, &iMainMsgQHighWtrMark, NULL)); @@ -3119,6 +2707,7 @@ static rsRetVal loadBuildInModules(void) CHKiRet(regCfSysLineHdlr((uchar *)"mainmsgqueuediscardmark", 0, eCmdHdlrInt, NULL, &iMainMsgQDiscardMark, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"mainmsgqueuediscardseverity", 0, eCmdHdlrSeverity, NULL, &iMainMsgQDiscardSeverity, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"mainmsgqueuecheckpointinterval", 0, eCmdHdlrInt, NULL, &iMainMsgQPersistUpdCnt, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"mainmsgqueuesyncqueuefiles", 0, eCmdHdlrBinary, NULL, &bMainMsgQSyncQeueFiles, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"mainmsgqueuetype", 0, eCmdHdlrGetWord, setMainMsgQueType, NULL, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"mainmsgqueueworkerthreads", 0, eCmdHdlrInt, NULL, &iMainMsgQueueNumWorkers, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"mainmsgqueuetimeoutshutdown", 0, eCmdHdlrInt, NULL, &iMainMsgQtoQShutdown, NULL)); @@ -3181,7 +2770,7 @@ static void printVersion(void) #else printf("\tFEATURE_REGEXP:\t\t\t\tNo\n"); #endif -#ifndef NOLARGEFILE +#if defined(_LARGE_FILES) || (defined (_FILE_OFFSET_BITS) && _FILE_OFFSET_BITS >= 64) printf("\tFEATURE_LARGEFILE:\t\t\tYes\n"); #else printf("\tFEATURE_LARGEFILE:\t\t\tNo\n"); @@ -3255,6 +2844,7 @@ static rsRetVal mainThread() if(Debug && debugging_on) { DBGPRINTF("Debugging enabled, SIGUSR1 to turn off debugging.\n"); } + /* Send a signal to the parent so it can terminate. */ if(myPid != ppid) @@ -3262,7 +2852,7 @@ static rsRetVal mainThread() /* If instructed to do so, we now drop privileges. Note that this is not 100% secure, - * because inputs and outputs are already running at this time. However, we can implement + * because outputs are already running at this time. However, we can implement * dropping of privileges rather quickly and it will work in many cases. While it is not * the ultimate solution, the current one is still much better than not being able to * drop privileges at all. Doing it correctly, requires a change in architecture, which @@ -3278,11 +2868,12 @@ static rsRetVal mainThread() glbl.SetHUPisRestart(0); /* we can not do restart-type HUPs with dropped privs */ } + /* finally let the inputs run... */ + runInputModules(); + /* END OF INTIALIZATION * ... but keep in mind that we might do a restart and thus init() might - * be called again. If that happens, we must shut down the worker thread, - * do the init() and then restart things. - * rgerhards, 2005-10-24 + * be called again. -- rgerhards, 2005-10-24 */ DBGPRINTF("initialization completed, transitioning to regular run mode\n"); @@ -3328,16 +2919,18 @@ InitGlobalClasses(void) CHKiRet(objUse(errmsg, CORE_COMPONENT)); pErrObj = "module"; CHKiRet(objUse(module, CORE_COMPONENT)); - pErrObj = "var"; - CHKiRet(objUse(var, CORE_COMPONENT)); pErrObj = "datetime"; CHKiRet(objUse(datetime, CORE_COMPONENT)); - pErrObj = "vm"; - CHKiRet(objUse(vm, CORE_COMPONENT)); pErrObj = "expr"; CHKiRet(objUse(expr, CORE_COMPONENT)); + pErrObj = "rule"; + CHKiRet(objUse(rule, CORE_COMPONENT)); + pErrObj = "ruleset"; + CHKiRet(objUse(ruleset, CORE_COMPONENT)); pErrObj = "conf"; CHKiRet(objUse(conf, CORE_COMPONENT)); + pErrObj = "prop"; + CHKiRet(objUse(prop, CORE_COMPONENT)); /* intialize some dummy classes that are not part of the runtime */ pErrObj = "action"; @@ -3378,34 +2971,15 @@ GlobalClassExit(void) /* first, release everything we used ourself */ objRelease(net, LM_NET_FILENAME);/* TODO: the dependency on net shall go away! -- rgerhards, 2008-03-07 */ + objRelease(prop, CORE_COMPONENT); objRelease(conf, CORE_COMPONENT); + objRelease(ruleset, CORE_COMPONENT); + objRelease(rule, CORE_COMPONENT); objRelease(expr, CORE_COMPONENT); vmClassExit(); /* this is hack, currently core_modules do not get this automatically called */ - objRelease(vm, CORE_COMPONENT); - objRelease(var, CORE_COMPONENT); objRelease(datetime, CORE_COMPONENT); /* TODO: implement the rest of the deinit */ -#if 0 - CHKiRet(datetimeClassInit(NULL)); - CHKiRet(msgClassInit(NULL)); - CHKiRet(strmClassInit(NULL)); - CHKiRet(wtiClassInit(NULL)); - CHKiRet(wtpClassInit(NULL)); - CHKiRet(qqueueClassInit(NULL)); - CHKiRet(vmstkClassInit(NULL)); - CHKiRet(sysvarClassInit(NULL)); - CHKiRet(vmClassInit(NULL)); - CHKiRet(vmopClassInit(NULL)); - CHKiRet(vmprgClassInit(NULL)); - CHKiRet(ctok_tokenClassInit(NULL)); - CHKiRet(ctokClassInit(NULL)); - CHKiRet(exprClassInit(NULL)); - - /* dummy "classes" */ - CHKiRet(actionClassInit()); - CHKiRet(templateInit()); -#endif /* dummy "classes */ strExit(); @@ -3499,7 +3073,6 @@ doGlblProcessInit(void) int i; DEFiRet; - checkPermissions(); thrdInit(); if( !(Debug || NoFork) ) @@ -3537,8 +3110,6 @@ doGlblProcessInit(void) fputs(" Already running.\n", stderr); exit(1); /* "good" exit, done if syslogd is already running */ } - } else { - debugging_on = 1; } /* tuck my process id away */ @@ -3646,6 +3217,7 @@ int realMain(int argc, char **argv) iCompatibilityMode = atoi(optarg); break; case 'd': /* debug - must be handled now, so that debug is active during init! */ + debugging_on = 1; Debug = 1; break; case 'e': /* log every message (no repeat message supression) */ @@ -3705,6 +3277,15 @@ int realMain(int argc, char **argv) /* doing some core initializations */ + /* we need to create the inputName property (only once during our lifetime) */ + CHKiRet(prop.Construct(&pInternalInputName)); + CHKiRet(prop.SetString(pInternalInputName, UCHAR_CONSTANT("rsyslogd"), sizeof("rsyslgod") - 1)); + CHKiRet(prop.ConstructFinalize(pInternalInputName)); + + CHKiRet(prop.Construct(&pLocalHostIP)); + CHKiRet(prop.SetString(pLocalHostIP, UCHAR_CONSTANT("127.0.0.1"), sizeof("127.0.0.1") - 1)); + CHKiRet(prop.ConstructFinalize(pLocalHostIP)); + /* get our host and domain names - we need to do this early as we may emit * error log messages, which need the correct hostname. -- rgerhards, 2008-04-04 */ @@ -3753,6 +3334,7 @@ int realMain(int argc, char **argv) */ glbl.SetLocalHostName(LocalHostName); glbl.SetLocalDomain(LocalDomain); + glbl.GenerateLocalHostNameProperty(); /* must be redone after conf processing, FQDN setting may have changed */ /* initialize the objects */ if((iRet = modInitIminternal()) != RS_RET_OK) { @@ -3854,11 +3436,12 @@ int realMain(int argc, char **argv) } else { fprintf(stderr, "error -p is no longer supported, use module imuxsock instead"); } + break; case 'q': /* add hostname if DNS resolving has failed */ - *net.pACLAddHostnameOnFail = 1; + *(net.pACLAddHostnameOnFail) = 1; break; case 'Q': /* dont resolve hostnames in ACL to IPs */ - *net.pACLDontResolve = 1; + *(net.pACLDontResolve) = 1; break; case 'r': /* accept remote messages */ if(iCompatibilityMode < 3) { @@ -3949,6 +3532,9 @@ int realMain(int argc, char **argv) if(!iConfigVerify) CHKiRet(doGlblProcessInit()); + /* re-generate local host name property, as the config may have changed our FQDN settings */ + glbl.GenerateLocalHostNameProperty(); + CHKiRet(mainThread()); /* do any de-init's that need to be done AFTER this comment */ diff --git a/tools/syslogd.h b/tools/syslogd.h index 8b9bd131..3dfdbe2b 100644 --- a/tools/syslogd.h +++ b/tools/syslogd.h @@ -29,66 +29,6 @@ #include "linkedlist.h" #include "expr.h" - -#ifndef _PATH_CONSOLE -#define _PATH_CONSOLE "/dev/console" -#endif - - -/* This structure represents the files that will have log - * copies printed. - * RGerhards 2004-11-08: Each instance of the filed structure - * describes what I call an "output channel". This is important - * to mention as we now allow database connections to be - * present in the filed structure. If helps immensely, if we - * think of it as the abstraction of an output channel. - * rgerhards, 2005-10-26: The structure below provides ample - * opportunity for non-thread-safety. Each of the variable - * accesses must be carefully evaluated, many of them probably - * be guarded by mutexes. But beware of deadlocks... - * rgerhards, 2007-08-01: as you can see, the structure has shrunk pretty much. I will - * remove some of the comments some time. It's still the structure that controls much - * of the processing that goes on in syslogd, but it now has lots of helpers. - */ -struct filed { - struct filed *f_next; /* next in linked list */ - /* filter properties */ - enum { - FILTER_PRI = 0, /* traditional PRI based filer */ - FILTER_PROP = 1, /* extended filter, property based */ - FILTER_EXPR = 2 /* extended filter, expression based */ - } f_filter_type; - EHostnameCmpMode eHostnameCmpMode; - cstr_t *pCSHostnameComp; /* hostname to check */ - cstr_t *pCSProgNameComp; /* tag to check or NULL, if not to be checked */ - union { - u_char f_pmask[LOG_NFACILITIES+1]; /* priority mask */ - struct { - cstr_t *pCSPropName; - enum { - FIOP_NOP = 0, /* do not use - No Operation */ - FIOP_CONTAINS = 1, /* contains string? */ - FIOP_ISEQUAL = 2, /* is (exactly) equal? */ - FIOP_STARTSWITH = 3, /* starts with a string? */ - FIOP_REGEX = 4, /* matches a (BRE) regular expression? */ - FIOP_EREREGEX = 5 /* matches a ERE regular expression? */ - } operation; - regex_t *regex_cache; /* cache for compiled REs, if such are used */ - cstr_t *pCSCompValue; /* value to "compare" against */ - char isNegated; /* actually a boolean ;) */ - } prop; - expr_t *f_expr; /* expression object */ - } f_filterData; - - linkedList_t llActList; /* list of configured actions */ -}; - - -#include "net.h" /* TODO: remove when you remoe isAllowedSender from here! */ -void untty(void); -rsRetVal selectorConstruct(selector_t **ppThis); -rsRetVal selectorDestruct(void *pVal); -rsRetVal selectorAddList(selector_t *f); /* the following prototypes should go away once we have an input * module interface -- rgerhards, 2007-12-12 */ diff --git a/tools/zpipe.c b/tools/zpipe.c new file mode 100644 index 00000000..38069425 --- /dev/null +++ b/tools/zpipe.c @@ -0,0 +1,262 @@ +/* zpipe.c: example of proper use of zlib's inflate() and deflate() + Not copyrighted -- provided to the public domain + Version 1.5 11 December 2005 Mark Adler + Version 2.0 03 June 2009 Rainer Gerhards */ + +/* RSYSLOG NOTE: + * This file is primarily been used as a testing aid for rsyslog. We do NOT + * properly maintain it and it has been brought to our attention that it may + * have some security issues. However, we prefer not to remove the file as it + * may turn out to be useful for further testing. All users are advised NOT + * to base any development on this version here, but rather look for the + * original zpipe.c by the authors mentioned above. + * + * This file is beeing distributed as part of rsyslog, but is just an + * add-on. Most importantly, rsyslog's copyright does not apply but + * rather the (non-) copyright stated above. + */ + +/* Version history: + 1.0 30 Oct 2004 First version + 1.1 8 Nov 2004 Add void casting for unused return values + Use switch statement for inflate() return values + 1.2 9 Nov 2004 Add assertions to document zlib guarantees + 1.3 6 Apr 2005 Remove incorrect assertion in inf() + 1.4 11 Dec 2005 Add hack to avoid MSDOS end-of-line conversions + Avoid some compiler warnings for input and output buffers + 2.0 03 Jun 2009 Add hack to support multiple deflate records inside a single + file on inflate. This is needed in order to support reading + files created by rsyslog's zip output writer. + */ + +#include "config.h" +#include <stdio.h> +#include <string.h> +#include <assert.h> +#include "zlib.h" + +#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(__CYGWIN__) +# include <fcntl.h> +# include <io.h> +# define SET_BINARY_MODE(file) setmode(fileno(file), O_BINARY) +#else +# define SET_BINARY_MODE(file) +#endif + +#define CHUNK 16384 + +/* Compress from file source to file dest until EOF on source. + def() returns Z_OK on success, Z_MEM_ERROR if memory could not be + allocated for processing, Z_STREAM_ERROR if an invalid compression + level is supplied, Z_VERSION_ERROR if the version of zlib.h and the + version of the library linked do not match, or Z_ERRNO if there is + an error reading or writing the files. */ +int def(FILE *source, FILE *dest, int level) +{ + int ret, flush; + unsigned have; + z_stream strm; + unsigned char in[CHUNK]; + unsigned char out[CHUNK]; + + /* allocate deflate state */ + strm.zalloc = Z_NULL; + strm.zfree = Z_NULL; + strm.opaque = Z_NULL; + ret = deflateInit(&strm, level); + if (ret != Z_OK) + return ret; + + /* compress until end of file */ + do { + strm.avail_in = fread(in, 1, CHUNK, source); + if (ferror(source)) { + (void)deflateEnd(&strm); + return Z_ERRNO; + } + flush = feof(source) ? Z_FINISH : Z_NO_FLUSH; + strm.next_in = in; + + /* run deflate() on input until output buffer not full, finish + compression if all of source has been read in */ + do { + strm.avail_out = CHUNK; + strm.next_out = out; + ret = deflate(&strm, flush); /* no bad return value */ + assert(ret != Z_STREAM_ERROR); /* state not clobbered */ + have = CHUNK - strm.avail_out; + if (fwrite(out, 1, have, dest) != have || ferror(dest)) { + (void)deflateEnd(&strm); + return Z_ERRNO; + } + } while (strm.avail_out == 0); + assert(strm.avail_in == 0); /* all input will be used */ + + /* done when last data in file processed */ + } while (flush != Z_FINISH); + assert(ret == Z_STREAM_END); /* stream will be complete */ + + /* clean up and return */ + (void)deflateEnd(&strm); + return Z_OK; +} + + +/* initialize stream for deflating (we need this in case of + * multiple records. + * rgerhards, 2009-06-03 + */ +int doInflateInit(z_stream *strm) +{ + int ret; + + /* allocate inflate state */ + strm->zalloc = Z_NULL; + strm->zfree = Z_NULL; + strm->opaque = Z_NULL; + strm->avail_in = 0; + strm->next_in = Z_NULL; + ret = inflateInit(strm); + return ret; +} + + +/* Decompress from file source to file dest until stream ends or EOF. + inf() returns Z_OK on success, Z_MEM_ERROR if memory could not be + allocated for processing, Z_DATA_ERROR if the deflate data is + invalid or incomplete, Z_VERSION_ERROR if the version of zlib.h and + the version of the library linked do not match, or Z_ERRNO if there + is an error reading or writing the files. */ +int inf(FILE *source, FILE *dest) +{ + int ret; + unsigned have; + z_stream strm; + unsigned char in[CHUNK]; + int len; + unsigned char *next_in_save; + unsigned char out[CHUNK]; + + ret = doInflateInit(&strm); + if (ret != Z_OK) + return ret; + + /* decompress until deflate stream ends or end of file */ + do { + len = fread(in, 1, CHUNK, source); + if (ferror(source)) { + (void)inflateEnd(&strm); + return Z_ERRNO; + } + if (len == 0) { + break; + } + strm.avail_in = len; + strm.next_in = in; + + /* run inflate() on input until output buffer not full */ + strm.avail_out = CHUNK; + strm.next_out = out; + do { + /* fprintf(stderr, "---inner LOOP---, avail_in %d, avail_out %d Byte 0: %x, 1: %x\n", strm.avail_in, strm.avail_out, *strm.next_in, *(strm.next_in+1));*/ + do { + ret = inflate(&strm, Z_NO_FLUSH); + assert(ret != Z_STREAM_ERROR); /* state not clobbered */ + switch (ret) { + case Z_NEED_DICT: + ret = Z_DATA_ERROR; /* and fall through */ + case Z_DATA_ERROR: + case Z_MEM_ERROR: + (void)inflateEnd(&strm); + return ret; + } + have = CHUNK - strm.avail_out; + if (fwrite(out, 1, have, dest) != have || ferror(dest)) { + (void)inflateEnd(&strm); + return Z_ERRNO; + } + } while (strm.avail_out == 0); + /* handle the case that more than one deflate record is contained + * in a single file. -- rgerhards, 2009-06-03 + */ + if(ret == Z_STREAM_END) { + len -= strm.total_in; + if(len > 0) { + next_in_save = strm.next_in; + (void)inflateEnd(&strm); + ret = doInflateInit(&strm); + if (ret != Z_OK) + return ret; + strm.avail_in = len; + strm.next_in = next_in_save; + strm.avail_out = CHUNK; + strm.next_out = out; + ret = Z_OK; /* continue outer loop */ + } + } + } while (strm.avail_in > 0); + + /* done when inflate() says it's done */ + } while (ret != Z_STREAM_END); + + /* clean up and return */ + (void)inflateEnd(&strm); + return ret == Z_STREAM_END ? Z_OK : Z_DATA_ERROR; +} + +/* report a zlib or i/o error */ +void zerr(int ret) +{ + fputs("zpipe: ", stdout); + switch (ret) { + case Z_ERRNO: + if (ferror(stdin)) + fputs("error reading stdin\n", stdout); + if (ferror(stdout)) + fputs("error writing stdout\n", stdout); + break; + case Z_STREAM_ERROR: + fputs("invalid compression level\n", stdout); + break; + case Z_DATA_ERROR: + fputs("invalid or incomplete deflate data\n", stdout); + break; + case Z_MEM_ERROR: + fputs("out of memory\n", stdout); + break; + case Z_VERSION_ERROR: + fputs("zlib version mismatch!\n", stdout); + } +} + +/* compress or decompress from stdin to stdout */ +int main(int argc, char **argv) +{ + int ret; + + /* avoid end-of-line conversions */ + SET_BINARY_MODE(stdin); + SET_BINARY_MODE(stdout); + + /* do compression if no arguments */ + if (argc == 1) { + ret = def(stdin, stdout, Z_DEFAULT_COMPRESSION); + if (ret != Z_OK) + zerr(ret); + return ret; + } + + /* do decompression if -d specified */ + else if (argc == 2 && strcmp(argv[1], "-d") == 0) { + ret = inf(stdin, stdout); + if (ret != Z_OK) + zerr(ret); + return ret; + } + + /* otherwise, report usage */ + else { + fputs("zpipe usage: zpipe [-d] < source > dest\n", stdout); + return 1; + } +} |