diff options
| author | Rainer Gerhards <rgerhards@adiscon.com> | 2009-11-13 10:00:23 +0100 |
|---|---|---|
| committer | Rainer Gerhards <rgerhards@adiscon.com> | 2009-11-13 10:00:23 +0100 |
| commit | 2a3c25a5c68db66ec35c155f7ecbe65079aed0e0 (patch) | |
| tree | b899a3cd256d60f3415753aa925c66fc0504373e | |
| parent | adb880f17bee807df6058ab3772eed3d35bd8d37 (diff) | |
| parent | 8b246de2a587454f9260ff91192d27a2e168ea2d (diff) | |
| download | rsyslog-2a3c25a5c68db66ec35c155f7ecbe65079aed0e0.tar.gz rsyslog-2a3c25a5c68db66ec35c155f7ecbe65079aed0e0.tar.xz rsyslog-2a3c25a5c68db66ec35c155f7ecbe65079aed0e0.zip | |
Begin new beta branch
Merge branch 'master' into beta
Conflicts:
ChangeLog
tests/Makefile.am
214 files changed, 6278 insertions, 2656 deletions
@@ -1,4 +1,117 @@ --------------------------------------------------------------------------- +Version 5.3.5 [DEVEL] (rgerhards), 2009-11-?? +- some light performance enhancement by replacing time() call with much + faster (at least under linux) gettimeofday() calls. +- some improvement of omfile performance with dynafiles + saved costly time() calls by employing a logical clock, which is + sufficient for the use case +- bugfix: omudpspoof miscalculated source and destination ports + while this was probably not noticed for source ports, it resulted in + almost all destination ports being wrong, except for the default port + of 514, which by virtue of its binary representation was calculated + correct (and probably thus the bug not earlier detected). +- bugfixes imported from earlier releases + * bugfix: named pipes did no longer work (they always got an open error) + this was a regression from the omfile rewrite in 4.5.0 + * bugfix(testbench): sequence check was not always performed correctly, + that could result in tests reporting success when they actually failed +- improved testbench: added tests for UDP forwarding and omudpspoof +- doc bugfix: omudpspoof had wrong config command names ("om" missing) +- bugfix [imported from 4.4.3]: $ActionExecOnlyOnceEveryInterval did + not work. +- [inport v4] improved testbench, contains now tcp and gzip test cases +- [import v4] added a so-called "On Demand Debug" mode, in which debug + output can be generated only after the process has started, but not right + from the beginning. This is assumed to be useful for hard-to-find bugs. + Also improved the doc on the debug system. +- bugfix: segfault on startup when -q or -Q option was given + [imported from v3-stable] +--------------------------------------------------------------------------- +Version 5.3.4 [DEVEL] (rgerhards), 2009-11-04 +- added the ability to create custom message parsers +- added $RulesetParser config directive that permits to bind specific + parsers to specific rulesets +- added omruleset output module, which provides great flexibility in + action processing. THIS IS A VERY IMPORTANT ADDITION, see its doc + for why. +- added the capability to have ruleset-specific main message queues + This offers considerable additional flexibility AND superior performance + (in cases where multiple inputs now can avoid lock contention) +- bugfix: correct default for escape ('#') character restored + This was accidently changed to '\\', thanks to David Lang for reporting +- bugfix(testbench): testcase did not properly wait for rsyslogd shutdown + thus some unpredictable behavior and a false negative test result + could occur. +--------------------------------------------------------------------------- +Version 5.3.3 [DEVEL] (rgerhards), 2009-10-27 +- simplified and thus speeded up the queue engine, also fixed some + potential race conditions (in very unusual shutdown conditions) + along the way. The threading model has seriously changes, so there may + be some regressions. +- enhanced test environment (inlcuding testbench): support for enhancing + probability of memory addressing failure by using non-NULL default + value for malloced memory (optional, only if requested by configure + option). This helps to track down some otherwise undetected issues + within the testbench. +- bugfix: potential abort if inputname property was not set + primarily a problem of imdiag +- bugfix: message processing states were not set correctly in all cases + however, this had no negative effect, as the message processing state + was not evaluated when a batch was deleted, and that was the only case + where the state could be wrong. +--------------------------------------------------------------------------- +Version 5.3.2 [DEVEL] (rgerhards), 2009-10-21 +- enhanced omfile to support transactional interface. This will increase + performance in many cases. +- added multi-ruleset support to imudp +- re-enabled input thread termination handling that does avoid thread + cancellation where possible. This provides a more reliable mode of + rsyslogd termination (canceling threads my result in not properly + freed resouces and potential later hangs, even though we perform + proper cancel handling in our code). This is part of an effort to + reduce thread cancellation as much as possible in rsyslog. + NOTE: the code previously written code for this functionality had a + subtle race condition. The new code solves that. +- enhanced immark to support non-cancel input module termination +- improved imudp so that epoll can be used in more environments, + fixed potential compile time problem if EPOLL_CLOEXEC is not available. +- some cleanup/slight improvement: + * changed imuxsock to no longer use deprecated submitAndParseMsg() IF + * changed submitAndParseMsg() interface to be a wrapper around the new + way of message creation/submission. This enables older plugins to be + used together with the new interface. The removal also enables us to + drop a lot of duplicate code, reducing complexity and increasing + maintainability. +- bugfix: segfault when starting up with an invalid .qi file for a disk queue + Failed for both pure disk as well as DA queues. Now, we emit an error + message and disable disk queueing facility. +- bugfix: potential segfault on messages with empty MSG part. This was a + recently introduced regression. +- bugfix: debug string larger than 1K were improperly displayed. Max size + is now 32K, and if a string is even longer it is meaningfully truncated. +--------------------------------------------------------------------------- +Version 5.3.1 [DEVEL] (rgerhards), 2009-10-05 +- added $AbortOnUncleanConfig directive - permits to prevent startup when + there are problems with the configuration file. See it's doc for + details. +- included some important fixes from v4-stable: + * bugfix: invalid handling of zero-sized messages + * bugfix: zero-sized UDP messages are no longer processed + * bugfix: random data could be appended to message + * bugfix: reverse lookup reduction logic in imudp do DNS queries too often +- bugfixes imported from 4.5.4: + * bugfix: potential segfault in stream writer on destruction + * bugfix: potential race in object loader (obj.c) during use/release + * bugfixes: potential problems in out file zip writer +--------------------------------------------------------------------------- +Version 5.3.0 [DEVEL] (rgerhards), 2009-09-14 +- begun to add simple GUI programs to gain insight into running rsyslogd + instances and help setup and troubleshooting (active via the + --enable-gui ./configure switch) +- changed imudp to utilize epoll(), where available. This shall provide + slightly better performance (just slightly because we called select() + rather infrequently on a busy system) +--------------------------------------------------------------------------- Version 5.1.6 [v5-beta] (rgerhards), 2009-10-15 - feature imports from v4.5.6 - bugfix: potential race condition when queue worker threads were @@ -22,6 +135,14 @@ Version 5.1.5 [v5-beta] (rgerhards), 2009-09-11 this option permites to process mark messages under all circumstances, even if an action was recently called. This can be useful to use mark messages as a kind of heartbeat. +- added new config option $InputUnixListenSocketCreatePath + to permit the auto-creation of pathes to additional log sockets. This + turns out to be useful if they reside on temporary file systems and + rsyslogd starts up before the daemons that create these sockets + (rsyslogd always creates the socket itself if it does not exist). +- added $LogRSyslogStatusMessages configuration directive + permitting to turn off rsyslog start/stop/HUP messages. See Debian + ticket http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=463793 - bugfix: hostnames with dashes in them were incorrectly treated as malformed, thus causing them to be treated as TAG (this was a regression introduced from the "rfc3164 strict" change in 4.5.0). Testbench has been @@ -29,6 +150,7 @@ Version 5.1.5 [v5-beta] (rgerhards), 2009-09-11 - bugfix: strings improperly reused, resulting in some message properties be populated with strings from previous messages. This was caused by an improper predicate check. +- added new config directive $omfileForceChown [import from 4.7.0] --------------------------------------------------------------------------- Version 5.1.4 [DEVEL] (rgerhards), 2009-08-20 - legacy syslog parser changed so that it now accepts date stamps in @@ -128,7 +250,32 @@ increase. - increased ompgsql performance by adapting to new transactional output module interface --------------------------------------------------------------------------- -Version 4.5.6 [v4-beta] (rgerhards), 2009-09-?? +Version 4.7.0 [v4-devel] (rgerhards), 2009-09-?? +- added function getenv() to RainerScript +- added new config option $InputUnixListenSocketCreatePath + to permit the auto-creation of pathes to additional log sockets. This + turns out to be useful if they reside on temporary file systems and + rsyslogd starts up before the daemons that create these sockets + (rsyslogd always creates the socket itself if it does not exist). +- added $LogRSyslogStatusMessages configuration directive + permitting to turn off rsyslog start/stop/HUP messages. See Debian + ticket http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=463793 +- added new config directive $omfileForceChown to (try to) fix some broken + system configs. + See ticket for details: http://bugzilla.adiscon.com/show_bug.cgi?id=150 +- imported changes from 4.5.6 and below +--------------------------------------------------------------------------- +Version 4.5.7 [v4-beta] (rgerhards), 2009-11-?? +- added a so-called "On Demand Debug" mode, in which debug output can + be generated only after the process has started, but not right from + the beginning. This is assumed to be useful for hard-to-find bugs. + Also improved the doc on the debug system. +- bugfix [imported from 4.4.3]: $ActionExecOnlyOnceEveryInterval did + not work. +--------------------------------------------------------------------------- +Version 4.5.6 [v4-beta] (rgerhards), 2009-11-05 +- bugfix: named pipes did no longer work (they always got an open error) + this was a regression from the omfile rewrite in 4.5.0 - bugfix(minor): diag function returned wrong queue memeber count for the main queue if an active DA queue existed. This had no relevance to real deployments (assuming they are not running the debug/diagnostic @@ -139,6 +286,11 @@ Version 4.5.6 [v4-beta] (rgerhards), 2009-09-?? * bugfix: zero-sized UDP messages are no longer processed * bugfix: random data could be appended to message * bugfix: reverse lookup reduction logic in imudp do DNS queries too often +- bugfix(testbench): testcase did not properly wait for rsyslod shutdown + thus some unpredictable behavior and a false negative test result + could occur. [BACKPORTED from v5] +- bugfix(testbench): sequence check was not always performed correctly, + that could result in tests reporting success when they actually failed --------------------------------------------------------------------------- Version 4.5.5 [v4-beta] (rgerhards), 2009-10-21 - added $InputTCPServerNotifyOnConnectionClose config directive @@ -259,6 +411,8 @@ Version 4.5.0 [DEVEL] (rgerhards), 2009-07-02 removes false positives (but may cause some trouble with hostname parsing). For details, see this bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=126 +- omfile rewrite to natively support zip files (includes large extension + of the stream class) - added configuration commands (see doc for explanations) * $OMFileZipLevel * $OMFileIOBufferSize @@ -280,8 +434,14 @@ Version 4.5.0 [DEVEL] (rgerhards), 2009-07-02 an abort --------------------------------------------------------------------------- Version 4.4.3 [v4-stable] (rgerhards), 2009-10-?? +- bugfix: $ActionExecOnlyOnceEveryInterval did not work. + This was a regression from the time() optimizations done in v4. + Bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=143 + Thanks to Klaus Tachtler for reporting this bug. - bugfix: potential hang condition on queue shutdown [imported from v3-stable] +- bugfix: segfault on startup when -q or -Q option was given + [imported from v3-stable] --------------------------------------------------------------------------- Version 4.4.2 [v4-stable] (rgerhards), 2009-10-09 - bugfix: invalid handling of zero-sized messages, could lead to mis- @@ -538,6 +698,9 @@ Version 3.22.2 [v3-stable] (rgerhards), 2009-07-?? [if librelp != 1.0.0 is used] - bugfix: sending syslog messages with zip compression did not work - bugfix: potential hang condition on queue shutdown +- bugfix: segfault on startup when -q or -Q option was given + bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=157 + Thanks to Jonas Nogueira for reporting this bug. - clarified use of $ActionsSendStreamDriver[AuthMode/PermittedPeers] in doc set (require TLS drivers) --------------------------------------------------------------------------- diff --git a/Makefile.am b/Makefile.am index 7ab48455..5f9d35fe 100644 --- a/Makefile.am +++ b/Makefile.am @@ -95,6 +95,10 @@ if ENABLE_OMSTDOUT SUBDIRS += plugins/omstdout endif +if ENABLE_OMRULESET +SUBDIRS += plugins/omruleset +endif + if ENABLE_OMUDPSPOOF SUBDIRS += plugins/omudpspoof endif @@ -127,6 +131,10 @@ if ENABLE_ORACLE SUBDIRS += plugins/omoracle endif +if ENABLE_GUI +SUBDIRS += java +endif + # tests are added as last element, because tests may need different # modules that need to be generated first SUBDIRS += tests @@ -138,5 +146,5 @@ SUBDIRS += tests # temporarily be removed below. The intent behind forcing everthing to compile # in a make distcheck is so that we detect code that accidently was not updated # when some global update happened. -DISTCHECK_CONFIGURE_FLAGS=--enable-gssapi_krb5 --enable-imfile --enable-snmp --enable-pgsql --enable-libdbi --enable-mysql --enable-omtemplate --enable-imtemplate --enable-relp --enable-rsyslogd --enable-mail --enable-klog --enable-diagtools --enable-gnutls --enable-omstdout --enable-omprog --enable-imdiag --enable-shave +DISTCHECK_CONFIGURE_FLAGS=--enable-gssapi_krb5 --enable-imfile --enable-snmp --enable-pgsql --enable-libdbi --enable-mysql --enable-omtemplate --enable-imtemplate --enable-relp --enable-rsyslogd --enable-mail --enable-klog --enable-diagtools --enable-gnutls --enable-omstdout --enable-omruleset --enable-omprog --enable-imdiag --enable-shave --enable-memcheck ACLOCAL_AMFLAGS = -I m4 @@ -50,7 +50,7 @@ #define NO_TIME_PROVIDED 0 /* indicate we do not provide any cached time */ /* forward definitions */ -static rsRetVal processBatchMain(action_t *pAction, batch_t *pBatch); +static rsRetVal processBatchMain(action_t *pAction, batch_t *pBatch, int*); /* object static data (once for all instances) */ /* TODO: make this an object! DEFobjStaticHelpers -- rgerhards, 2008-03-05 */ @@ -59,6 +59,7 @@ DEFobjCurrIf(datetime) DEFobjCurrIf(module) DEFobjCurrIf(errmsg) +static int iActExecOnceInterval = 0; /* execute action once every nn seconds */ static int iActExecEveryNthOccur = 0; /* execute action every n-th occurence (0,1=always) */ static time_t iActExecEveryNthOccurTO = 0; /* timeout for n-occurence setting (in seconds, 0=never) */ static int glbliActionResumeInterval = 30; @@ -128,7 +129,7 @@ getActNow(action_t *pThis) { assert(pThis != NULL); if(pThis->tActNow == -1) { - pThis->tActNow = time(NULL); /* good time call - the only one done */ + pThis->tActNow = datetime.GetTime(NULL); /* good time call - the only one done */ if(pThis->tLastExec > pThis->tActNow) { /* if we are traveling back in time, reset tLastExec */ pThis->tLastExec = (time_t) 0; @@ -210,7 +211,7 @@ rsRetVal actionDestruct(action_t *pThis) if(pThis->ppMsgs[i] != NULL) { switch(pThis->eParamPassing) { case ACT_ARRAY_PASSING: -#if 0 /* later! */ +#if 0 /* later, as an optimization. So far, we do the cleanup after each message */ iArr = 0; while(((char **)pThis->ppMsgs[i])[iArr] != NULL) { d_free(((char **)pThis->ppMsgs[i])[iArr++]); @@ -223,6 +224,9 @@ rsRetVal actionDestruct(action_t *pThis) case ACT_STRING_PASSING: d_free(pThis->ppMsgs[i]); break; + case ACT_MSG_PASSING: + /* No cleanup needed in this case */ + break; default: assert(0); } @@ -250,7 +254,7 @@ rsRetVal actionConstruct(action_t **ppThis) CHKmalloc(pThis = (action_t*) calloc(1, sizeof(action_t))); pThis->iResumeInterval = glbliActionResumeInterval; pThis->iResumeRetryCount = glbliActionResumeRetryCount; - pThis->tLastOccur = time(NULL); /* done once per action on startup only */ + pThis->tLastOccur = datetime.GetTime(NULL); /* done once per action on startup only */ pthread_mutex_init(&pThis->mutActExec, NULL); SYNC_OBJ_TOOL_INIT(pThis); @@ -291,7 +295,7 @@ actionConstructFinalize(action_t *pThis) * spec. -- rgerhards, 2008-01-30 */ CHKiRet(qqueueConstruct(&pThis->pQueue, ActionQueType, 1, iActionQueueSize, - (rsRetVal (*)(void*, batch_t*))processBatchMain)); + (rsRetVal (*)(void*, batch_t*, int*))processBatchMain)); obj.SetName((obj_t*) pThis->pQueue, pszQName); /* ... set some properties ... */ @@ -466,7 +470,7 @@ static void actionDisable(action_t *pThis) static inline void actionSuspend(action_t *pThis, time_t ttNow) { if(ttNow == NO_TIME_PROVIDED) - time(&ttNow); + datetime.GetTime(&ttNow); pThis->ttResumeRtry = ttNow + pThis->iResumeInterval * (pThis->iNbrResRtry / 10 + 1); actionSetState(pThis, ACT_STATE_SUSP); DBGPRINTF("earliest retry=%d\n", (int) pThis->ttResumeRtry); @@ -534,7 +538,7 @@ static rsRetVal actionTryResume(action_t *pThis) * is always in the past. So we can not avoid doing a fresh time() call * here. -- rgerhards, 2009-03-18 */ - time(&ttNow); /* cache "now" */ + datetime.GetTime(&ttNow); /* cache "now" */ if(ttNow > pThis->ttResumeRtry) { actionSetState(pThis, ACT_STATE_RTRY); /* back to retries */ } @@ -542,7 +546,7 @@ static rsRetVal actionTryResume(action_t *pThis) if(pThis->eState == ACT_STATE_RTRY) { if(ttNow == NO_TIME_PROVIDED) /* use cached result if we have it */ - time(&ttNow); + datetime.GetTime(&ttNow); CHKiRet(actionDoRetry(pThis, ttNow)); } @@ -623,6 +627,12 @@ static rsRetVal prepareDoActionParams(action_t *pAction, msg_t *pMsg) case ACT_ARRAY_PASSING: CHKiRet(tplToArray(pAction->ppTpl[i], pMsg, (uchar***) &(pAction->ppMsgs[i]))); break; + case ACT_MSG_PASSING: + /* we abuse the uchar* ptr, it now actually is a void*, but we can not + * change that other than by chaning the interface, what we don't like... + */ + pAction->ppMsgs[i] = (uchar*) pMsg; + break; default:assert(0); /* software bug if this happens! */ } } @@ -654,6 +664,7 @@ static rsRetVal cleanupDoActionParams(action_t *pAction) d_free(pAction->ppMsgs[i]); pAction->ppMsgs[i] = NULL; break; + case ACT_MSG_PASSING: case ACT_STRING_PASSING: break; default: @@ -798,7 +809,7 @@ finalize_it: * rgerhards, 2009-05-12 */ static rsRetVal -tryDoAction(action_t *pAction, batch_t *pBatch, int *pnElem) +tryDoAction(action_t *pAction, batch_t *pBatch, int *pnElem, int *pbShutdownImmediate) { int i; int iElemProcessed; @@ -814,19 +825,20 @@ tryDoAction(action_t *pAction, batch_t *pBatch, int *pnElem) iElemProcessed = 0; iCommittedUpTo = i; while(iElemProcessed <= *pnElem && i < pBatch->nElem) { + if(*pbShutdownImmediate) + ABORT_FINALIZE(RS_RET_FORCE_TERM); pMsg = (msg_t*) pBatch->pElem[i].pUsrp; - DBGPRINTF("submitBatch: i:%d, batch size %d, to process %d, pMsg: %p, state %d\n", i, pBatch->nElem, *pnElem, pMsg, pBatch->pElem[i].state);//remove later! if(pBatch->pElem[i].state != BATCH_STATE_DISC) { localRet = actionProcessMessage(pAction, pMsg); DBGPRINTF("action call returned %d\n", localRet); if(localRet == RS_RET_OK) { /* mark messages as committed */ - while(iCommittedUpTo < i) { + while(iCommittedUpTo <= i) { pBatch->pElem[iCommittedUpTo++].state = BATCH_STATE_COMM; } } else if(localRet == RS_RET_PREVIOUS_COMMITTED) { /* mark messages as committed */ - while(iCommittedUpTo < i - 1) { + while(iCommittedUpTo < i) { pBatch->pElem[iCommittedUpTo++].state = BATCH_STATE_COMM; } pBatch->pElem[i].state = BATCH_STATE_SUB; @@ -860,7 +872,7 @@ finalize_it: * rgerhards, 2009-05-12 */ static rsRetVal -submitBatch(action_t *pAction, batch_t *pBatch, int nElem) +submitBatch(action_t *pAction, batch_t *pBatch, int nElem, int *pbShutdownImmediate) { int i; int bDone; @@ -871,7 +883,9 @@ submitBatch(action_t *pAction, batch_t *pBatch, int nElem) bDone = 0; do { - localRet = tryDoAction(pAction, pBatch, &nElem); + localRet = tryDoAction(pAction, pBatch, &nElem, pbShutdownImmediate); + if(localRet == RS_RET_FORCE_TERM) + FINALIZE; if( localRet == RS_RET_OK || localRet == RS_RET_PREVIOUS_COMMITTED || localRet == RS_RET_DEFER_COMMIT) { @@ -902,13 +916,17 @@ submitBatch(action_t *pAction, batch_t *pBatch, int nElem) bDone = 1; } else { /* retry with half as much. Depth is log_2 batchsize, so recursion is not too deep */ - submitBatch(pAction, pBatch, nElem / 2); - submitBatch(pAction, pBatch, nElem - (nElem / 2)); + submitBatch(pAction, pBatch, nElem / 2, pbShutdownImmediate); + submitBatch(pAction, pBatch, nElem - (nElem / 2), pbShutdownImmediate); bDone = 1; } } - } while(!bDone); /* do .. while()! */ + } while(!bDone && !*pbShutdownImmediate); /* do .. while()! */ + if(*pbShutdownImmediate) + ABORT_FINALIZE(RS_RET_FORCE_TERM); + +finalize_it: RETiRet; } @@ -917,26 +935,13 @@ submitBatch(action_t *pAction, batch_t *pBatch, int nElem) * rgerhards, 2009-05-12 */ static rsRetVal -processAction(action_t *pAction, batch_t *pBatch) +processAction(action_t *pAction, batch_t *pBatch, int *pbShutdownImmediate) { - int i; - msg_t *pMsg; - rsRetVal localRet; DEFiRet; assert(pBatch != NULL); - pBatch->iDoneUpTo = 0; - /* TODO: think about action batches, must be handled at upper layer! - * MULTIQUEUE - */ - localRet = submitBatch(pAction, pBatch, pBatch->nElem); - CHKiRet(localRet); - - /* this must be moved away - up into the dequeue part of the queue, I guess, but that's for another day */ - for(i = 0 ; i < pBatch->nElem ; i++) { - pMsg = (msg_t*) pBatch->pElem[i].pUsrp; - } + CHKiRet(submitBatch(pAction, pBatch, pBatch->nElem, pbShutdownImmediate)); iRet = finishBatch(pAction); finalize_it: @@ -950,7 +955,7 @@ finalize_it: * rgerhards, 2009-04-22 */ static rsRetVal -processBatchMain(action_t *pAction, batch_t *pBatch) +processBatchMain(action_t *pAction, batch_t *pBatch, int *pbShutdownImmediate) { DEFiRet; @@ -964,7 +969,7 @@ processBatchMain(action_t *pAction, batch_t *pBatch) d_pthread_mutex_lock(&pAction->mutActExec); pthread_cleanup_push(mutexCancelCleanup, &pAction->mutActExec); - iRet = processAction(pAction, pBatch); + iRet = processAction(pAction, pBatch, pbShutdownImmediate); pthread_cleanup_pop(1); /* unlock mutex */ @@ -1127,7 +1132,7 @@ actionWriteToAction(action_t *pAction) * a purely logical point of view. However, if safes us to check the system time in * (those common) cases where ExecOnceInterval is not used. -- rgerhards, 2008-09-16 */ - if(pAction->f_time != 0 && pAction->iSecsExecOnceInterval > 0 && + if(pAction->iSecsExecOnceInterval > 0 && pAction->iSecsExecOnceInterval + pAction->tLastExec > getActNow(pAction)) { /* in this case we need to discard the message - its not yet time to exec the action */ DBGPRINTF("action not yet ready again to be executed, onceInterval %d, tCurr %d, tNext %d\n", @@ -1138,6 +1143,7 @@ actionWriteToAction(action_t *pAction) } /* we use reception time, not dequeue time - this is considered more appropriate and also faster ;) -- rgerhards, 2008-09-17 */ + pAction->tLastExec = getActNow(pAction); /* re-init time flags */ pAction->f_time = pAction->f_pMsg->ttGenTime; /* When we reach this point, we have a valid, non-disabled action. @@ -1260,47 +1266,6 @@ actionCallAction(action_t *pAction, msg_t *pMsg) #pragma GCC diagnostic warning "-Wempty-body" -/* add our cfsysline handlers - * rgerhards, 2008-01-28 - */ -rsRetVal -actionAddCfSysLineHdrl(void) -{ - DEFiRet; - - CHKiRet(regCfSysLineHdlr((uchar *)"actionname", 0, eCmdHdlrGetWord, NULL, &pszActionName, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuefilename", 0, eCmdHdlrGetWord, NULL, &pszActionQFName, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuesize", 0, eCmdHdlrInt, NULL, &iActionQueueSize, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionwriteallmarkmessages", 0, eCmdHdlrBinary, NULL, &bActionWriteAllMarkMsgs, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuedequeuebatchsize", 0, eCmdHdlrInt, NULL, &iActionQueueDeqBatchSize, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuemaxdiskspace", 0, eCmdHdlrSize, NULL, &iActionQueMaxDiskSpace, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuehighwatermark", 0, eCmdHdlrInt, NULL, &iActionQHighWtrMark, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuelowwatermark", 0, eCmdHdlrInt, NULL, &iActionQLowWtrMark, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuediscardmark", 0, eCmdHdlrInt, NULL, &iActionQDiscardMark, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuediscardseverity", 0, eCmdHdlrInt, NULL, &iActionQDiscardSeverity, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuecheckpointinterval", 0, eCmdHdlrInt, NULL, &iActionQPersistUpdCnt, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuesyncqueuefiles", 0, eCmdHdlrBinary, NULL, &bActionQSyncQeueFiles, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuetype", 0, eCmdHdlrGetWord, setActionQueType, NULL, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueueworkerthreads", 0, eCmdHdlrInt, NULL, &iActionQueueNumWorkers, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuetimeoutshutdown", 0, eCmdHdlrInt, NULL, &iActionQtoQShutdown, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuetimeoutactioncompletion", 0, eCmdHdlrInt, NULL, &iActionQtoActShutdown, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuetimeoutenqueue", 0, eCmdHdlrInt, NULL, &iActionQtoEnq, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueueworkertimeoutthreadshutdown", 0, eCmdHdlrInt, NULL, &iActionQtoWrkShutdown, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueueworkerthreadminimummessages", 0, eCmdHdlrInt, NULL, &iActionQWrkMinMsgs, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuemaxfilesize", 0, eCmdHdlrSize, NULL, &iActionQueMaxFileSize, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuesaveonshutdown", 0, eCmdHdlrBinary, NULL, &bActionQSaveOnShutdown, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuedequeueslowdown", 0, eCmdHdlrInt, NULL, &iActionQueueDeqSlowdown, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuedequeuetimebegin", 0, eCmdHdlrInt, NULL, &iActionQueueDeqtWinFromHr, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuedequeuetimeend", 0, eCmdHdlrInt, NULL, &iActionQueueDeqtWinToHr, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionexeconlyeverynthtime", 0, eCmdHdlrInt, NULL, &iActExecEveryNthOccur, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionexeconlyeverynthtimetimeout", 0, eCmdHdlrInt, NULL, &iActExecEveryNthOccurTO, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"repeatedmsgcontainsoriginalmsg", 0, eCmdHdlrBinary, NULL, &bActionRepMsgHasMsg, NULL)); - -finalize_it: - RETiRet; -} - - /* add an Action to the current selector * The pOMSR is freed, as it is not needed after this function. * Note: this function pulls global data that specifies action config state. @@ -1352,9 +1317,8 @@ addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, omodStringReques for(i = 0 ; i < pAction->iNumTpls ; ++i) { CHKiRet(OMSRgetEntry(pOMSR, i, &pTplName, &iTplOpts)); - /* Ok, we got everything, so it now is time to look up the - * template (Hint: templates MUST be defined before they are - * used!) + /* Ok, we got everything, so it now is time to look up the template + * (Hint: templates MUST be defined before they are used!) */ if((pAction->ppTpl[i] = tplFind((char*)pTplName, strlen((char*)pTplName))) == NULL) { snprintf(errMsg, sizeof(errMsg) / sizeof(char), @@ -1376,6 +1340,8 @@ addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, omodStringReques /* set parameter-passing mode */ if(iTplOpts & OMSR_TPL_AS_ARRAY) { pAction->eParamPassing = ACT_ARRAY_PASSING; + } else if(iTplOpts & OMSR_TPL_AS_MSG) { + pAction->eParamPassing = ACT_MSG_PASSING; } else { pAction->eParamPassing = ACT_STRING_PASSING; } @@ -1395,7 +1361,7 @@ addAction(action_t **ppAction, modInfo_t *pMod, void *pModData, omodStringReques pAction->eState = ACT_STATE_RDY; /* action is enabled */ if(bSuspended) - actionSuspend(pAction, time(NULL)); /* "good" time call, only during init and unavoidable */ + actionSuspend(pAction, datetime.GetTime(NULL)); /* "good" time call, only during init and unavoidable */ CHKiRet(actionConstructFinalize(pAction)); @@ -1417,6 +1383,17 @@ finalize_it: } +/* Reset config variables to default values. + * rgerhards, 2009-11-12 + */ +static rsRetVal +resetConfigVariables(uchar __attribute__((unused)) *pp, void __attribute__((unused)) *pVal) +{ + iActExecOnceInterval = 0; + return RS_RET_OK; +} + + /* TODO: we are not yet a real object, the ClassInit here just looks like it is.. */ rsRetVal actionClassInit(void) @@ -1428,6 +1405,36 @@ rsRetVal actionClassInit(void) CHKiRet(objUse(module, CORE_COMPONENT)); CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionname", 0, eCmdHdlrGetWord, NULL, &pszActionName, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuefilename", 0, eCmdHdlrGetWord, NULL, &pszActionQFName, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuesize", 0, eCmdHdlrInt, NULL, &iActionQueueSize, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionwriteallmarkmessages", 0, eCmdHdlrBinary, NULL, &bActionWriteAllMarkMsgs, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuedequeuebatchsize", 0, eCmdHdlrInt, NULL, &iActionQueueDeqBatchSize, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuemaxdiskspace", 0, eCmdHdlrSize, NULL, &iActionQueMaxDiskSpace, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuehighwatermark", 0, eCmdHdlrInt, NULL, &iActionQHighWtrMark, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuelowwatermark", 0, eCmdHdlrInt, NULL, &iActionQLowWtrMark, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuediscardmark", 0, eCmdHdlrInt, NULL, &iActionQDiscardMark, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuediscardseverity", 0, eCmdHdlrInt, NULL, &iActionQDiscardSeverity, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuecheckpointinterval", 0, eCmdHdlrInt, NULL, &iActionQPersistUpdCnt, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuesyncqueuefiles", 0, eCmdHdlrBinary, NULL, &bActionQSyncQeueFiles, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuetype", 0, eCmdHdlrGetWord, setActionQueType, NULL, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueueworkerthreads", 0, eCmdHdlrInt, NULL, &iActionQueueNumWorkers, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuetimeoutshutdown", 0, eCmdHdlrInt, NULL, &iActionQtoQShutdown, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuetimeoutactioncompletion", 0, eCmdHdlrInt, NULL, &iActionQtoActShutdown, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuetimeoutenqueue", 0, eCmdHdlrInt, NULL, &iActionQtoEnq, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueueworkertimeoutthreadshutdown", 0, eCmdHdlrInt, NULL, &iActionQtoWrkShutdown, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueueworkerthreadminimummessages", 0, eCmdHdlrInt, NULL, &iActionQWrkMinMsgs, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuemaxfilesize", 0, eCmdHdlrSize, NULL, &iActionQueMaxFileSize, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuesaveonshutdown", 0, eCmdHdlrBinary, NULL, &bActionQSaveOnShutdown, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuedequeueslowdown", 0, eCmdHdlrInt, NULL, &iActionQueueDeqSlowdown, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuedequeuetimebegin", 0, eCmdHdlrInt, NULL, &iActionQueueDeqtWinFromHr, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionqueuedequeuetimeend", 0, eCmdHdlrInt, NULL, &iActionQueueDeqtWinToHr, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionexeconlyeverynthtime", 0, eCmdHdlrInt, NULL, &iActExecEveryNthOccur, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionexeconlyeverynthtimetimeout", 0, eCmdHdlrInt, NULL, &iActExecEveryNthOccurTO, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionexeconlyonceeveryinterval", 0, eCmdHdlrInt, NULL, &iActExecOnceInterval, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"repeatedmsgcontainsoriginalmsg", 0, eCmdHdlrBinary, NULL, &bActionRepMsgHasMsg, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, NULL)); + finalize_it: RETiRet; } @@ -71,7 +71,7 @@ struct action_s { short f_ReduceRepeated;/* reduce repeated lines 0 - no, 1 - yes */ int f_prevcount; /* repetition cnt of prevline */ int f_repeatcount; /* number of "repeated" msgs */ - enum { ACT_STRING_PASSING = 0, ACT_ARRAY_PASSING = 1 } + enum { ACT_STRING_PASSING = 0, ACT_ARRAY_PASSING = 1, ACT_MSG_PASSING } eParamPassing; /* mode of parameter passing to action */ int iNumTpls; /* number of array entries for template element below */ struct template **ppTpl;/* array of template to use - strings must be passed to doAction @@ -95,7 +95,6 @@ typedef struct action_s action_t; rsRetVal actionConstruct(action_t **ppThis); rsRetVal actionConstructFinalize(action_t *pThis); rsRetVal actionDestruct(action_t *pThis); -rsRetVal actionAddCfSysLineHdrl(void); rsRetVal actionDbgPrint(action_t *pThis); rsRetVal actionSetGlobalResumeInterval(int iNewVal); rsRetVal actionDoAction(action_t *pAction); @@ -19,7 +19,7 @@ fi (cd $srcdir && autoreconf --verbose --force --install) || exit 1 -conf_flags="--enable-shave --cache-file=config.cache" +conf_flags="--cache-file=config.cache" if test x$NOCONFIGURE = x; then echo Running $srcdir/configure $conf_flags "$@" ... diff --git a/configure.ac b/configure.ac index 5bbc5e55..d850dfda 100644 --- a/configure.ac +++ b/configure.ac @@ -2,8 +2,11 @@ # Process this file with autoconf to produce a configure script. AC_PREREQ(2.61) -AC_INIT([rsyslog],[5.1.6],[rsyslog@lists.adiscon.com]) +AC_INIT([rsyslog],[5.3.5],[rsyslog@lists.adiscon.com]) AM_INIT_AUTOMAKE + +m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])]) + AC_CONFIG_SRCDIR([ChangeLog]) AC_CONFIG_MACRO_DIR([m4]) AC_CONFIG_HEADERS([config.h]) @@ -12,7 +15,7 @@ AC_GNU_SOURCE # check for Java compiler AC_CHECK_PROG(HAVE_JAVAC, [javac], [yes]) -if test x"$HAVE_JAVAC" = x"yes"; then +if test x"$HAVE_JAVAC" = x""; then AC_MSG_WARN([no javac found, disabling features depending on it]) fi @@ -104,7 +107,7 @@ AC_TYPE_SIGNAL AC_FUNC_STAT AC_FUNC_STRERROR_R AC_FUNC_VPRINTF -AC_CHECK_FUNCS([flock basename alarm clock_gettime gethostbyname gethostname gettimeofday localtime_r memset mkdir regcomp select setid socket strcasecmp strchr strdup strerror strndup strnlen strrchr strstr strtol strtoul uname ttyname_r epoll_wait getline malloc_trim prctl fdatasync]) +AC_CHECK_FUNCS([flock basename alarm clock_gettime gethostbyname gethostname gettimeofday localtime_r memset mkdir regcomp select setid socket strcasecmp strchr strdup strerror strndup strnlen strrchr strstr strtol strtoul uname ttyname_r getline malloc_trim prctl epoll_create epoll_create1 fdatasync]) # Check for MAXHOSTNAMELEN AC_MSG_CHECKING(for MAXHOSTNAMELEN) @@ -391,6 +394,21 @@ if test "$enable_valgrind" = "yes"; then fi +# memcheck +AC_ARG_ENABLE(memcheck, + [AS_HELP_STRING([--enable-memcheck],[Enable extended memory check support @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_memcheck="yes" ;; + no) enable_memcheck="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-memcheck) ;; + esac], + [enable_memcheck="no"] +) +if test "$enable_memcheck" = "yes"; then + AC_DEFINE(MEMCHECK, 1, [Defined if memcheck support settings are to be enabled (e.g. prevents dlclose()).]) +fi + + # compile diagnostic tools (small helpers usually not needed) AC_ARG_ENABLE(diagtools, [AS_HELP_STRING([--enable-diagtools],[Enable diagnostic tools @<:@default=no@:>@])], @@ -773,6 +791,40 @@ AC_ARG_ENABLE(omstdout, ) AM_CONDITIONAL(ENABLE_OMSTDOUT, test x$enable_omstdout = xyes) + +# settings for omruleset +AC_ARG_ENABLE(omruleset, + [AS_HELP_STRING([--enable-omruleset],[Compiles ruleset forwarding module @<:@default=yes@:>@])], + [case "${enableval}" in + yes) enable_omruleset="yes" ;; + no) enable_omruleset="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-omruleset) ;; + esac], + [enable_omruleset=yes] +) +AM_CONDITIONAL(ENABLE_OMRULESET, test x$enable_omruleset = xyes) + + +# building the GUI (mostly for diagnostic reasons) +AC_ARG_ENABLE(gui, + [AS_HELP_STRING([--enable-gui],[Enable GUI programs @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_gui="yes" ;; + no) enable_gui="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-gui) ;; + esac], + [enable_gui=no] +) +if test "x$enable_gui" = "xyes"; then + if test x$HAVE_JAVAC = x; then + AC_MSG_ERROR([GUI components need Java, but Java development system is not installed on this system]) + fi +fi +AM_CONDITIONAL(ENABLE_GUI, test x$enable_gui = xyes) + + +AC_SUBST(RELP_CFLAGS) +AC_SUBST(RELP_LIBS) # This provides a vehicle to integrate custom modules, that are not # part of rsyslog, into the build process. It is named cust1, so that # additional such modules can easily be added. @@ -828,12 +880,7 @@ AM_CONDITIONAL(ENABLE_OMTEMPLATE, test x$enable_omtemplate = xyes) # end of copy template - be sure to search for omtemplate to find everything! -SHAVE_INIT - - AC_CONFIG_FILES([Makefile \ - shave \ - shave-libtool \ runtime/Makefile \ tools/Makefile \ doc/Makefile \ @@ -848,6 +895,7 @@ AC_CONFIG_FILES([Makefile \ plugins/omtemplate/Makefile \ plugins/omprog/Makefile \ plugins/omstdout/Makefile \ + plugins/omruleset/Makefile \ plugins/imfile/Makefile \ plugins/imrelp/Makefile \ plugins/imdiag/Makefile \ @@ -862,6 +910,7 @@ AC_CONFIG_FILES([Makefile \ plugins/omoracle/Makefile \ plugins/omudpspoof/Makefile \ plugins/cust1/Makefile \ + java/Makefile \ tests/Makefile]) AC_OUTPUT @@ -874,6 +923,7 @@ echo " Regular expressions support enabled: $enable_regexp" echo " Zlib compression support enabled: $enable_zlib" echo " rsyslog runtime will be built: $enable_rsyslogrt" echo " rsyslogd will be built: $enable_rsyslogd" +echo " GUI components will be built: $enable_gui" echo " custom module 1 will be built: $enable_cust1" echo echo "---{ input plugins }---" @@ -886,6 +936,7 @@ echo "---{ output plugins }---" echo " Mail support enabled: $enable_mail" echo " omprog module will be compiled: $enable_omprog" echo " omstdout module will be compiled: $enable_omstdout" +echo " omruleset module will be compiled: $enable_omruleset" echo " omudpspoof module will be compiled: $enable_omudpspoof" echo " output template module will be compiled: $enable_omtemplate" echo @@ -906,5 +957,6 @@ echo " Testbench enabled: $enable_testbench" echo " Debug mode enabled: $enable_debug" echo " Runtime Instrumentation enabled: $enable_rtinst" echo " Diagnostic tools enabled: $enable_diagtools" +echo " Enhanced memory checking enabled: $enable_memcheck" echo " Valgrind support settings enabled: $enable_valgrind" echo @@ -31,10 +31,8 @@ rsRetVal multiSubmitMsg(multi_submit_t *pMultiSub); rsRetVal submitMsg(msg_t *pMsg); rsRetVal logmsgInternal(int iErr, int pri, uchar *msg, int flags); rsRetVal parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int flags, flowControl_t flowCtlTypeu, prop_t *pInputName, struct syslogTime *stTime, time_t ttGenTime); -int parseRFCSyslogMsg(msg_t *pMsg, int flags); -int parseLegacySyslogMsg(msg_t *pMsg, int flags); rsRetVal diagGetMainMsgQSize(int *piSize); /* for imdiag */ -char* getFIOPName(unsigned iFIOP); +rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName); /* Intervals at which we flush out "message repeated" messages, * in seconds after previous message is logged. After each flush, @@ -42,31 +40,13 @@ char* getFIOPName(unsigned iFIOP); * TODO: move this to action object! Only action.c and syslogd.c use it. */ extern int bActExecWhenPrevSusp; -extern int iActExecOnceInterval; extern int MarkInterval; extern int repeatinterval[2]; extern int bReduceRepeatMsgs; +extern qqueue_t *pMsgQueue; /* the main message queue */ #define MAXREPEAT ((int)((sizeof(repeatinterval) / sizeof(repeatinterval[0])) - 1)) #define REPEATTIME(f) ((f)->f_time + repeatinterval[(f)->f_repeatcount]) #define BACKOFF(f) { if (++(f)->f_repeatcount > MAXREPEAT) \ (f)->f_repeatcount = MAXREPEAT; \ } -extern int bDropTrailingLF; -extern uchar cCCEscapeChar; -extern int bEscapeCCOnRcv; -#ifdef USE_NETZIP -/* config param: minimum message size to try compression. The smaller - * the message, the less likely is any compression gain. We check for - * gain before we submit the message. But to do so we still need to - * do the (costly) compress() call. The following setting sets a size - * for which no call to compress() is done at all. This may result in - * a few more bytes being transmited but better overall performance. - * Note: I have not yet checked the minimum UDP packet size. It might be - * that we do not save anything by compressing very small messages, because - * UDP might need to pad ;) - * rgerhards, 2006-11-30 - */ -#define MIN_SIZE_FOR_COMPRESS 60 -#endif - #endif /* #ifndef DIRTY_H_INCLUDED */ diff --git a/doc/Makefile.am b/doc/Makefile.am index a447cddf..4832e757 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -90,7 +90,9 @@ html_files = \ rsconf1_moddir.html \ rsconf1_repeatedmsgreduction.html \ rsconf1_resetconfigvariables.html \ + rsconf1_rulesetcreatemainqueue.html \ rsconf1_umask.html \ + rulesetparser.html \ v3compatibility.html \ v4compatibility.html \ v5compatibility.html \ @@ -130,6 +132,8 @@ grfx_files = \ dataflow.png \ queue_analogy_tv.png \ gssapi.png \ + rfc5424layers.png \ + src/rfc5424layers.dia \ rsyslog-vers.png EXTRA_DIST = $(html_files) $(grfx_files) diff --git a/doc/debug.html b/doc/debug.html index de77f04a..46759986 100644 --- a/doc/debug.html +++ b/doc/debug.html @@ -1,41 +1,145 @@ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head> -<meta http-equiv="Content-Language" content="en"><title>Debug Support</title></head> +<html> +<head> +<meta http-equiv="Content-Language" content="en"> +<title>Rsyslog Debug Support</title></head> <body> -<h1>Debug Support</h1> +<h1>Rsyslog Debug Support</h1> <p> -Rsyslog provides a number of debug aids. Some of them are activated by +Rsyslog provides a number of debug aides. Some of them are activated by adding the --enable-rtinst ./configure option ("rtinst" means runtime instrumentation). Turning debugging on obviously costs some performance (in some cases considerable). </p> <p>This is document is just being created and thus terse.</p> -<p style="font-weight: bold;">Signals supported</p> -<p>SIGUSR1 - turns debug messages on and off (expect this signal -to go away over time)</p> -<p>SIGUSR2 - outputs debug information (including active threads +<h2>Signals supported</h2> +<p><b>SIGUSR1</b> - turns debug messages on and off. Note that for this +signal to work, rsyslogd must be running with debugging enabled, either +via the -d command line switch or the environment options specified below. +It is <b>not</b> required that rsyslog was compiled with debugging enabled +(but depending on the settings this may lead to better debug info). +<p><b>SIGUSR2</b> - outputs debug information (including active threads and a call stack) for the state when SIGUSR2 was received. This is a -one-time output. Can be sent as often as the user likes.</p> -<p style="font-weight: bold;">Environment Variables</p> -<p>There are two environment variables that set several debug settings. The "RSYSLOG_DEBUGLOG" (sample: RSYSLOG_DEBUGLOG="/path/to/debuglog/") +one-time output. Can be sent as often as the user likes. +<p><b>Note:</b> this signal <b>may go away</b> in later releases and may +be replaced by something else.</p> +<h2>Environment Variables</h2> +<p>There are two environment variables that set several debug settings: +<ul> +<li>The "RSYSLOG_DEBUGLOG" (sample: RSYSLOG_DEBUGLOG="/path/to/debuglog/") writes (allmost) all debug message to the specified log file in addition to stdout. Some system messages (e.g. segfault or abort message) are not written to the -file as we can not capture them. Runtime debug support is controlled by -"RSYSLOG_DEBUG". It contains an option string with the following -options possible (all are case insensitive):</p><ul><li><span style="font-weight: bold;">LogFuncFlow</span> - print out the logical flow of functions (entering and exiting them)</li><li><span style="font-weight: bold;">FileTrace</span> - specifies which files to trace LogFuncFlow. If <span style="font-weight: bold;">not</span> +file as we can not capture them. +<li>Runtime debug support is controlled by "RSYSLOG_DEBUG". +<p>The "RSYSLOG_DEBUG" environment variable contains an option string with the following +options possible (all are case insensitive):</p> +<ul> +<li><b>LogFuncFlow</b> - print out the logical flow of functions (entering and exiting them)</li> +<li><b>FileTrace</b> - specifies which files to trace LogFuncFlow. If <b>not</b> set (the default), a LogFuncFlow trace is provided for all files. Set to limit it to the files specified. FileTrace may be specified multiple times, one file each (e.g. export RSYSLOG_DEBUG="LogFuncFlow -FileTrace=vm.c FileTrace=expr.c"</li><li><span style="font-weight: bold;">PrintFuncDB</span> - print the content of the debug function database whenever debug information is printed (e.g. abort case)!</li><li><span style="font-weight: bold;">PrintAllDebugInfoOnExit</span> - print all debug information immediately before rsyslogd exits (<span style="font-weight: bold; font-style: italic;">currently not implemented!</span>)</li><li><span style="font-weight: bold;">PrintMutexAction</span> - print mutex action as it happens. Useful for finding deadlocks and such.</li><li><span style="font-weight: bold;">NoLogTimeStamp</span> - do not prefix log lines with a timestamp (default is to do that).</li><li><span style="font-weight: bold;">NoStdOut</span> - do not emit debug messages to stdout. If RSYSLOG_DEBUGLOG is not set, this means no messages will be displayed at all.</li><li><span style="font-weight: bold;">help</span> - display a very short list of commands - hopefully a life saver if you can't access the documentation...</li></ul> +FileTrace=vm.c FileTrace=expr.c"</li> +<li><b>PrintFuncDB</b> - print the content of the debug function database whenever debug information is printed (e.g. abort case)!</li> +<li><b>PrintAllDebugInfoOnExit</b> - print all debug information immediately before rsyslogd exits (<span style="font-weight: bold; font-style: italic;">currently not implemented!</span>)</li> +<li><b>PrintMutexAction</b> - print mutex action as it happens. Useful for finding deadlocks and such.</li> +<li><b>NoLogTimeStamp</b> - do not prefix log lines with a timestamp (default is to do that).</li> +<li><b>NoStdOut</b> - do not emit debug messages to stdout. If RSYSLOG_DEBUGLOG is not set, this means no messages will be displayed at all.</li> +<li><b>Debug</b> - if present, turns on the debug system and enables debug output +<li><b>DebugOnDemand</b> - if present, turns on the debug system but does not enable +debug output itself. You need to send SIGUSR1 to turn it on when desired. +<li><b>help</b> - display a very short list of commands - hopefully a life saver if you can't access the documentation...</li> +</ul> +</ul> +<h3>Why Environment Variables?</h3> +<p>You may ask why we use environment variables for debug-system parameters and not +the usual rsyslog.conf configuration commands. After all, environment variables force one +to change distro-specific configuration files, whereas regular configuration directives +would fit nicely into the one central rsyslog.conf. +<p>The problem here is that many settings of the debug system must be initialized +before the full rsyslog engine starts up. At that point, there is no such thing like +rsyslog.conf or the objects needed to process it present in an running instance. +And even if we would enable to change settings some time later, that would mean that +we can not correctly monitor (and debug) the initial startup phase of rsyslogd. What +makes matters worse is that during this startup phase (and never again later!) some +of the base debug structure needs to be created, at least if the build is +configured for that (many of these things only happen in --enable-rtinst mode). So +if we do not initialize the debug system <b>before</b> actually startig up the +rsyslog core, we get a number of data structures wrong. +<p>For these reasons, we utilize environment variables to initialize and configure +the debugging system. We understand this may be somewhat painful, but now you know +there are at least some good reasons for doing so. +<h2>Getting debug information from a running Instance</h2> +<p>It is possible to obtain debugging information from a running instance, but this requires +some setup. We assume that the instance runs in the background, so debug output to +stdout is not desired. As such, all debug information needs to go into a log file. +<p>To create this setup, you need to <ul> +<li>point the RSYSLOG_DEBUGLOG environment variable to a file that is accessible +during the while runtime (we strongly suggest a file in the local file system!) +<li>set RSYSLOG_DEBUG at least to "DebugOnDeman NoStdOut" +<li>make sure these environment variables are set in the correct (distro-specifc) +startup script if you do not run rsyslogd interactively </ul> +<p>These settings enable the capability to react to SIGUSR1. The signal will toggle +debug status when received. So send it one to turn debug loggin on, and send it again +to turn debug logging off again. The third time it will be turned on again ... and so on. +<p>On a typical system, you can signal rsyslogd as follows: +<pre> +kill -USR1 `cat /var/run/rsyslogd.pid` +</pre> +Important: there are backticks around the "cat"-command. If you use the regular +quote it won't work. The debug log will show whether debug logging has been turned +on or off. There is no other indication of the status. +<p>Note: running with DebugOnDemand by itself does in practice not have any performance +toll. However, switching debug logging on has a severe performance toll. Also, debug +logging synchronizes much of the code, removing a lot of concurrency and thus +potential race conditions. As such, the very same running instance may behave +very differently with debug logging turned on vs. off. The on-demand debug log +functionality is considered to be very valuable to analyze hard-to-find bugs that +only manifest after a long runtime. Turning debug logging on a failing instance +may reveal the cause of the failure. However, depending on the failure, debug logging +may not even be successfully be turned on. Also note that with this rsyslog version we cannot +obtain any debug information on events that happened <i>before</i> debug logging was +turned on. +<p>If an instance hangs, it is possible to obtain some useful information about the current +threads and their calling stack by sending SIGUSR2. However, the usefulness of that +information is very much depending on rsyslog compile-time settings, must importantly +the --enable-rtinst configure flag. Note that activating this option causes additional overhead +and slows down rsyslgod considerable. So if you do that, you need to check if it is +capable to handle the workload. Also, threading behavior is modified by the +runtime instrumentation. +<p>Sending SIGUSR2 writes new process state information to the log file each time +it is sent. So it may be useful to do that from time to time. It probably is most +useful if the process seems to hang, in which case it may (may!) be able to output +some diagnostic information on the current processing state. In that case, turning +on the mutex debugging options (see above) is probably useful. +<h2>Interpreting the Logs</h2> +<p>Debug logs are primarily meant for rsyslog developers. But they may still provide valuable +information to users. Just be warned that logs sometimes contains informaton the looks like +an error, but actually is none. We put a lot of extra information into the logs, and there +are some cases where it is OK for an error to happen, we just wanted to record it inside +the log. The code handles many cases automatically. So, in short, the log may not make sense to +you, but it (hopefully) makes sense to a developer. Note that we developers often need +many lines of the log file, it is relatively rare that a problem can be diagnosed by +looking at just a couple of (hundered) log records. +<h2>Security Risks</h2> +<p>The debug log will reveal potentially sensible information, including user accounts and +passwords, to anyone able to read the log file. As such, it is recommended to properly +guard access to the log file. Also, an instance running with debug log enabled runs much +slower than one without. An attacker may use this to place carry out a denial-of-service +attack or try to hide some information from the log file. As such, it is suggested to +enable DebugOnDemand mode only for a reason. Note that when no debug mode is enabled, +SIGUSR1 and SIGUSR2 are completely ignored. +<p>When running in any of the debug modes (including on demand mode), an interactive +instance of rsyslogd can be aborted by pressing ctl-c. +<p> <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> -project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer -Gerhards</a> and +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> -</body></html>
\ No newline at end of file +</body> +</html> diff --git a/doc/design.tex b/doc/design.tex index 53d25313..a3ec8f45 100644 --- a/doc/design.tex +++ b/doc/design.tex @@ -22,7 +22,7 @@ rgerhards@adiscon.com} \maketitle \begin{abstract} -This paper describes rsyslog design and internals. It is created to facilitate a discussion about the implementation of "batched queue processing". As such, it does not describe the full design of rsyslog but rather those elements that are relevant to queues. However, the document may be expanded in the future. +This paper describes rsyslog design and internals. It is created to facilitate a discussion about the implementation of "batched queue processing". As such, it does not describe the full design of rsyslog but rather those elements that are relevant to queues. However, the document may be expanded in the future. This is work in progress and should be considered with care. It is NOT updated during all phases of development. \end{abstract} \tableofcontents @@ -811,9 +811,37 @@ b) we push the failed message back to the main queue, but with an indication that it failed in an action. This is harder to implement and most importantly harder to understand/configure, but more flexible +\section{Network Stream Subsystem} +The idea of network streams was introduced when we implemented RFC5425 (syslog over TLS) in 2008. The core idea is to encapsulate all stream-oriented network data transfer into a single transport layer and make the upper layers independent of actual transport being used. This is in line with the traditional layer approaches in communication systems. + +Under this system, the upper layer provides plugins to send and receive streams of syslog data. Framing is provided by the upper layer. The upper layer itself is integrated in input and output plugins, which then are used to provide application-level syslog message objects to and from the rsyslog core. To these upper layers, the netstream layer provides reliable and sequenced message delivery with much of the same semantics as a usual TCP stream. + +\begin{figure} +\begin{center} +\includegraphics[scale=0.4]{tls.jpeg} +\end{center} +\caption{Objects at the Network Stream Layer} +\label{fig_netstream_objects} +\end{figure} + +At the netstream layer, we have a small set of generic classes, which are used for setup of the drivers and driver parameters. This is a very thin layer, mostly a wrapper. Once an actual lower-level netstream driver has been loaded, all parameters are passed through to it. + +Please note that both in theory and practice netstream drivers may call back into different netstream drivers. For example, the GnuTLS RFC5425 driver loads and calls back into the plain tcp driver, simply because that driver provides part of the required functionality and there is no point in re-implementing it for GnuTLS. + +The netstream driver layer does not only provide read and write calls but supports i/o multiplexing. To do so, it offers an interface that follows select() semantics. That permits an upper-layer comonent to request being blocked unless some data arrives. Note that due to the subleties in TLS processing, the upper layer may be awoken while there is no upper-layer work to do. This will properly be indicated by the netstream subsystem, is not an error and must be accepted and poperly handled by the upper layer. + +Using the nestream layer, we do not need to modify the input and output plugins while at the same time we can add additional transport providers. One weak spot in this design is the current configuration process. With the current system, we need to provide one configuration statement per driver property and we need to hardcode this. So if a new driver would require new properties, we still would need to modify the upper layers. This is unfortunate, but the current config system does not provide for any better way to handle the situation. Once we are able to create a new config system, we will address this by providing the ability to pass a string of parameters onto the driver, which will then have the ability to parse its content. So once we do this, we need to modify the driver interface, but the end result would be a simlification. + +So far, only drivers for GnuTLS and plain tcp are provided. However, during the design of the layer we also looked at openssl and Mozilla Network Security Services as well as kept an eye on the needs of Kerberos. In theory, it should not be a major problem to write drivers for these systems (but it most probably still is a lot of work to do). + +A final note on Kerberos: in order to keep compatible with previous protocol handling and due to constraints in testing environment and knowledge, we still support Kerberos not via the netstream layer but via special extension into the input and output modules. That, too, is unfortunate, but given the current resources at hand, there is no alternative to handling in that way. We would be very interested in moving over Kerberos to a netstream driver and any volunteer would be very welcome. + \section{Future Development} This section covers topics that can not currently be developed, but where important thoughts came up in discussions. For obvious reasons, the section has brainstorming character. +\subsection{Lock-Free Queuing} +On a very busy system, lock contention can limit performance. We should investigate ways to apply lock-free algorithms inside rsyslog. It is believed that at least for some scenarios, lock-free algorigthms can be applied with great benefit. To do so, we should introduce new queue modes, which will use very different semantics from what is described so far for the queue engine. Most importantly, in lock-free mode we will have limits on the number of producers and we will most probably not be able to guarantee audit-grade processing. The later is not a problem, because there are ample use cases that do not require audit-gradeness. + \subsection{Audit-Grade High Performance Queue Storage Driver} An audit grade driver must ensure that no message is lost, but should also be able to handle large workloads. The sequential disk driver does not support the later. diff --git a/doc/expression.html b/doc/expression.html index 9e37cb7a..c401d9ab 100644 --- a/doc/expression.html +++ b/doc/expression.html @@ -1,17 +1,22 @@ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html><head> -<meta http-equiv="Content-Language" content="en"><title>Expressions</title></head> +<meta http-equiv="Content-Language" content="en"> +<title>Expressions in rsyslog</title></head> <body> -<a href="rsyslog_conf_filter.html">back</a> -<h1>Expressions</h1> +<a href="rsyslog_conf_filter.html">back to rsyslog filter conditions</a> +<h1>Expressions in rsyslog</h1> <p>Rsyslog supports expressions at a growing number of places. So -far, they are supported for filtering messages.</p><p>Expression support is provided by RainerScript. For now, please see the formal expression definition in <a href="rainerscript.html">RainerScript ABNF</a>. It is the "expr" node.</p><p>C-like comments (/* some comment */) are supported <span style="font-weight: bold;">inside</span> the expression, but not yet in the rest of the configuration file.</p><p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +far, they are supported for filtering messages.</p> +<p>Expression support is provided by RainerScript. Please see the +<a href="rainerscript.html">RainerScript documentation</a> for more details.</p> +<p>C-like comments (/* some comment */) are supported <b>inside</b> the expression, +but not yet in the rest of the configuration file.</p> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> -project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer -Gerhards</a> and +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body></html> diff --git a/doc/imtcp.html b/doc/imtcp.html index 0ccdecc7..434b3903 100644 --- a/doc/imtcp.html +++ b/doc/imtcp.html @@ -1,21 +1,23 @@ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head> -<meta http-equiv="Content-Language" content="en"><title>TCP Syslog Input Module</title></head> +<html> +<head> +<meta http-equiv="Content-Language" content="en"> +<title>TCP Syslog Input Module</title> +</head> + <body> -<a href="rsyslog_conf_modules.html">back</a> +<a href="rsyslog_conf_modules.html">back to rsyslog module overview</a> <h1>TCP Syslog Input Module</h1> <p><b>Module Name: imtcp</b></p> -<p><b>Author: </b>Rainer Gerhards -<rgerhards@adiscon.com></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Multi-Ruleset Support: </b>since 4.5.0 and 5.1.1 <p><b>Description</b>:</p> <p>Provides the ability to receive syslog messages via TCP. -Encryption can be provided by using <a href="rsyslog_stunnel.html">stunnel</a> -(an alternative is the use -the <a href="imgssapi.html">imgssapi</a> -modul).</p> -<p>Multiple receivers may be configured by -specifying +Encryption is natively provided by selecting the approprioate network stream driver and +can also be provided by using <a href="rsyslog_stunnel.html">stunnel</a> +(an alternative is the use the <a href="imgssapi.html">imgssapi</a> module).</p> +<p>Multiple receivers may be configured by specifying $InputTCPServerRun multiple times. This is available since version 4.3.1, earlier versions do NOT support it. </p> @@ -49,8 +51,7 @@ after loading imtcp, otherwise it may have no effect.</li> Starts a TCP server on selected port</li> <li>$InputTCPMaxListeners <number><br> Sets the maximum number of listeners (server ports) supported. Default is 20. This must be set before the first $InputTCPServerRun directive.</li> -<li>$InputTCPMaxSessions <number><br> -Sets the maximum number of sessions supported. Default is 200. This must be set before the first $InputTCPServerRun directive</li> +<li>$InputTCPMaxSessions <number><br> Sets the maximum number of sessions supported. Default is 200. This must be set before the first $InputTCPServerRun directive</li> <li>$InputTCPServerStreamDriverMode <number><br> Sets the driver mode for the currently selected <a href="netstream.html">network stream driver</a>. <number> is driver specifc.</li> <li>$InputTCPServerInputName <name><br> @@ -63,6 +64,8 @@ Sets the authentication mode for the currently selected <a href="netstream.html" Sets permitted peer IDs. Only these peers are able to connect to the listener. <id-string> semantics depend on the currently selected AuthMode and <a href="netstream.html">network stream driver</a>. PermittedPeers may not be set in anonymous modes.</li> +<li>$InputTCPServerBindRuleset <ruleset><br> +Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> </ul> <b>Caveats/Known Bugs:</b> <ul> @@ -70,20 +73,22 @@ AuthMode and <a href="netstream.html">network stream driver</a>. Permitted <li>can not be loaded together with <a href="imgssapi.html">imgssapi</a> (which includes the functionality of imtcp)</li> </ul> -<p><b>Sample:</b></p> -<p>This sets up a TCP server on port 514:<br> +<p><b>Example:</b></p> +<p>This sets up a TCP server on port 514 and permits it to accept up to 500 connections:<br> </p> -<textarea rows="15" cols="60">$ModLoad imtcp # -needs to be done just once +<textarea rows="15" cols="60">$ModLoad imtcp # needs to be done just once +$InputTCPMaxSessions 500 $InputTCPServerRun 514 </textarea> +<p>Note that the parameters (here: max sessions) need to be set <b>before</b> the listener +is activated. Otherwise, the parameters will not apply. +</p> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> -<p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> +<p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer -Gerhards</a> and +Copyright © 2008,2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> -</body></html> +</body> +</html> diff --git a/doc/imudp.html b/doc/imudp.html new file mode 100644 index 00000000..f0e86307 --- /dev/null +++ b/doc/imudp.html @@ -0,0 +1,58 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html> +<head> +<meta http-equiv="Content-Language" content="en"> +<title>TCP Syslog Input Module</title> +</head> + +<body> +<a href="rsyslog_conf_modules.html">back to rsyslog module overview</a> + +<h1>UDP Syslog Input Module</h1> +<p><b>Module Name: imudp</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Multi-Ruleset Support: </b>since 5.3.2 +<p><b>Description</b>:</p> +<p>Provides the ability to receive syslog messages via UDP. +<p>Multiple receivers may be configured by specifying +$UDPServerRun multiple times. +</p> +<p><b>Configuration Directives</b>:</p> +<ul> +<li>$UDPServerAddress <IP><br> +local IP address (or name) the UDP listens should bind to</li> +<li>$UDPServerRun <port><br> +former -r<port> option, default 514, start UDP server on this +port, "*" means all addresses</li> +<li>$UDPServerTimeRequery <nbr-of-times><br> +this is a performance +optimization. Getting the system time is very costly. With this setting, imudp can +be instructed to obtain the precise time only once every n-times. This logic is +only activated if messages come in at a very fast rate, so doing less frequent +time calls should usually be acceptable. The default value is two, because we have +seen that even without optimization the kernel often returns twice the identical time. +You can set this value as high as you like, but do so at your own risk. The higher +the value, the less precise the timestamp. +<li>$InputUDPServerBindRuleset <ruleset><br> +Binds the listener to a specific <a href="multi_ruleset.html">ruleset</a>.</li> +</ul> +<b>Caveats/Known Bugs:</b> +<ul> +<li>currently none known</li> +</ul> +<p><b>Sample:</b></p> +<p>This sets up an UPD server on port 514:<br> +</p> +<textarea rows="15" cols="60">$ModLoad imudp # needs to be done just once +$UDPServerRun 514 +</textarea> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer +Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/imuxsock.html b/doc/imuxsock.html index 472470a0..15c365a6 100644 --- a/doc/imuxsock.html +++ b/doc/imuxsock.html @@ -46,6 +46,18 @@ Ignore timestamps included in the messages, applies to messages received via the <li><b>$SystemLogSocketName</b> <name-of-socket> -- former -p option</li> <li><b>$SystemLogFlowControl</b> [on/<b>off</b>] - specifies if flow control should be applied to the system log socket.</li> +<li><b>$InputUnixListenSocketCreatePath</b> [on/<b>off</b>] - create directories in the socket path +if they do not already exist. They are created with 0755 permissions with the owner being the process under +which rsyslogd runs. The default is not to create directories. Keep in mind, though, that rsyslogd always +creates the socket itself if it does not exist (just not the directories by default). +<br>Note that this statement affects the +next $AddUnixListenSocket directive that follows in sequence in the configuration file. It never works +on the system log socket (where it is deemed unnecessary). Also note that it is automatically +being reset to "off" after the $AddUnixListenSocket directive, so if you would have it active +for two additional listen sockets, you need to specify it in front of each one. This option is primarily considered +useful for defining additional sockets that reside on non-permanent file systems. As rsyslogd probably starts +up before the daemons that create these sockets, it is a vehicle to enable rsyslogd to listen to those +sockets even though their directories do not yet exist. [available since 4.7.0 and 5.3.0]</li> <li><b>$AddUnixListenSocket</b> <name-of-socket> adds additional unix socket, default none -- former -a option</li> <li><b>$InputUnixListenSocketHostName</b> <hostname> permits to override the hostname that shall be used inside messages taken from the <b>next</b> $AddUnixListenSocket socket. Note that @@ -57,20 +69,32 @@ that the local hostname can be overridden in cases where that is desired.</li> <br> This documentation is sparse and incomplete. <p><b>Sample:</b></p> -<p>The following sample is the minimum setup required to accept syslog messages from applications running on the local system.<br> +<p>The following sample is the minimum setup required to accept syslog messages from applications running +on the local system.<br> </p> <textarea rows="2" cols="70">$ModLoad imuxsock # needs to be done just once $SystemLogSocketFlowControl on # enable flow control (use if needed) </textarea> <p>The following sample is a configuration where rsyslogd pulls logs from two jails, and assigns different hostnames to each of the jails: </p> -<textarea rows="6" cols="60">$ModLoad imuxsock # needs to be done just once +<textarea rows="6" cols="70">$ModLoad imuxsock # needs to be done just once $InputUnixListenSocketHostName jail1.example.net $AddUnixListenSocket /jail/1/dev/log $InputUnixListenSocketHostName jail2.example.net $AddUnixListenSocket /jail/2/dev/log </textarea> +<p>The following sample is a configuration where rsyslogd reads the openssh log +messages via a separate socket, but this socket is created on a temporary file +system. As rsyslogd starts up before the sshd, it needs to create the socket +directories, because it otherwise can not open the socket and thus not listen +to openssh messages. Note that it is vital not to place any other socket between +the $InputUnixListenSocketCreatePath and the $InputUnixListenSocketHostName.</p> +<textarea rows="6" cols="70">$ModLoad imuxsock # needs to be done just once + +$InputUnixListenSocketCreatePath on # turn on for *next* socket +$InputUnixListenSocketHostName /var/run/sshd/dev/log +</textarea> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the diff --git a/doc/manual.html b/doc/manual.html index 90af6940..57f3d4a6 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -19,7 +19,7 @@ rsyslog support</a> available directly from the source!</p> <p><b>Please visit the <a href="http://www.rsyslog.com/sponsors">rsyslog sponsor's page</a> to honor the project sponsors or become one yourself!</b> We are very grateful for any help towards the project goals.</p> -<p><b>This documentation is for version 5.1.6 (devel branch) of rsyslog.</b> +<p><b>This documentation is for version 5.3.5 (devel branch) of rsyslog.</b> Visit the <i><a href="http://www.rsyslog.com/doc-status.html">rsyslog status page</a></i></b> to obtain current version information and project status. </p><p><b>If you like rsyslog, you might @@ -43,9 +43,10 @@ if you do not read the doc, but doing so will definitely improve your experience <li> <a href="property_replacer.html">property replacer, an important core component</a></li> <li>a commented <a href="sample.conf.html">sample rsyslog.conf</a> </li> <li><a href="bugs.html">rsyslog bug list</a></li> -<li><a href="rsyslog_packages.html"> rsyslog packages</a></li> +<li><a href="messageparser.html">understanding rsyslog message parsers</a></li> <li><a href="generic_design.html">backgrounder on generic syslog application design</a></li> <li><a href="modules.html">description of rsyslog modules</a></li> +<li><a href="rsyslog_packages.html">rsyslog packages</a></li> </ul> <p><b>We have some in-depth papers on</b></p> <ul> diff --git a/doc/messageparser.html b/doc/messageparser.html new file mode 100644 index 00000000..370db59f --- /dev/null +++ b/doc/messageparser.html @@ -0,0 +1,222 @@ +<html> +<head> +<title>Message parsers in rsyslog</title> +</head> +<body> +<a href="manual.html">rsyslog documentation</a> + +<h1>Message parsers in rsyslog</h1> +<p><small><i>Written by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> +(2009-11-06)</i></small></p> +<h2>Intro</h2> +<p>Message parsers are a feature of rsyslog 5.3.4 and above. In this article, I describe what +message parsers are, what they can do and how they relate to the relevant standards. I will +also describe what you can not do with time. Finally, I give some advice on implementing your +own custom parser. + +<h2>What are message parsers?</h2> +<p>Well, the quick answer is that message parsers are the component of rsyslog that +parses the syslog message after it is being received. Prior to rsyslog 5.3.4, message parsers +where built in into the rsyslog core itself and could not be modified (other than by modifying +the rsyslog code). +<p>In 5.3.4, we changed that: message parsers are now loadable modules (just +like input and output modules). That means that new message parsers can be added without +modifying the rsyslog core, even without contributing something back to the +project. +<p>But that doesn't answer what a message parser really is. What does ist mean to "parse a +message" and, maybe more importantly, what is a message? To answer these questions correctly, +we need to dig down into the relevant standards. +<a href="http://tools.ietf.org/html/rfc5424">RFC5424</a> specifies a layered architecture +for the syslog protocol: +<p align="center"><img src="rfc5424layers.png" alt="RFC5424 syslog protocol layers"> +<p>For us important is the distinction between the syslog transport and the upper layers. +The transport layer specifies how a stream of messages is assembled at the sender side and +how this stream of messages is disassembled into the individual messages at the receiver +side. In networking terminology, this is called "framing". The core idea is that +each message is put into a so-called "frame", which then is transmitted over the communications +link. +<p>The framing used is depending on the protocol. For example, in UDP the "frame"-equivalent is +a packet that is being sent (this also means that no two messages can travel within a single +UDP packet). In "plain tcp syslog", the industry standard, LF is used as a frame delimiter +(which also means that no multi-line message can properly be transmitted, a "design" flaw +in plain tcp syslog). In <a href="http://tools.ietf.org/html/rfc5425">RFC5425</a> there is +a header in front of each frame that contains the size of the message. With this framing, +any message content can properly be transferred. +<p>And now comes the important part: <b>message parsers do NOT operate at the transport +layer</b>, they operate, as their name implies, on messages. So we can not use message +parsers to change the underlying framing. For example, if a sender splits (for whatever +reason) a single message into two and encapsulates these into two frames, there is no way +a message parser could undo that. +<p>A typical example may be a multi-line message: let's assume some originator has generated +a message for the format "A\nB" (where \n means LF). If that message is being transmitted +via plain tcp syslog, the frame delimiter is LF. So the sender will delimite the frame with +LF, but otherwise send the message unmodified onto the wire (because that is how things are +-unfortunately- done in plain tcp syslog...). So wire will see "A\nB\n". When this +arrives at the receiver, the transport layer will undo the framing. When it sees the LF +after A, it thinks it finds a valid frame delimiter (in fact, this is the correct view!). So +the receive will extract one complete message A and one complete message B, not knowing +that they once were both part of a large multi-line message. These two messages are then +passed to the upper layers, where the message parsers receive them and extract information. +However, the message parsers never know (or even have a chance to see) that A and B +belonged together. Even further, in rsyslog there is no guarnatee that A will be parsed +before B - concurrent operations may cause the reverse order (and do so very validly). +<p>The important lesson is: <b>message parsers can not be used to fix a broken framing</b>. +You need a full protocol implementation to do that, what is the domain of input and +output modules. +<p>I have now told you what you can not do with message parsers. But what they are good for? +Thankfully, broken framing is not the primary problem of the syslog world. A wealth of different +formats is. Unfortunately, many real-world implementations violate the relevant standards +in one way or another. That makes it often very hard to extract meaningful information from +a message or to process messages from different sources by the same rules. In my article +<a href="syslog_parsing.html">syslog parsing in rsyslog</a> I have elaborated on all +the real-world evil that you can usually see. So I won't repeat that here. But in short, the +real problem is not the framing, but how to make malformed messages well-looking. +<p><b>This is what message parsers permit you to do: take a (well-known) malformed message, parse +it according to its semantics and generate perfectly valid internal message representations +from it.</b> So as long as messages are consistenly in the same wrong format (and they usually +are!), a message parser can look at that format, parse it, and make the message processable just +like it were wellformed in the first place. Plus, one can abuse the interface to do some other +"intersting" tricks, but that would take us to far. +<p>While this functionality may not sound exciting, it actually solves a very big issue (that you +only really understand if you have managed a system with various different syslog sources). +Note that we were often able to process malformed messages in the past with the help of the +property replacer and regular expressions. While this is nice, it has a performance hit. A +message parser is a C code, compiled to native language, and thus typically much faster than +any regular expression based method (depending, of course, on the quality of the implementation...). + +<h2>How are message parsers used?</h2> +<p>In a simlified view, rsyslog +<ol> +<li>first receives messages (via the input module), +<li><i>then parses them (at the message level!)</i> and +<li>then processes them (operating on the internal message representation). +</ol> +Message parsers are utilized in the second step (written in italics). +Thus, they take the raw message (NOT frame!) received from the remote system and create +the internal structure out of it that the other parts of rsyslog need in order to perform +their processing. Parsing is vital, because an unparsed message can not be processed in the +third stage, the actual application-level processing (like forwarding or writing to files). +<h3>Parser Chains and how they Operate</h3> +Rsyslog chains parsers together to provide flexibility. +A <b>parser chain</b> +contains all parsers that can potentially be used to parse a message. +It is assumed that there is some +way a parser can detect if the message it is being presented is supported by it. If so, the parser +will tell the rsyslog engine and parse the message. The rsyslog engine now calls each parser +inside the chain (in sequence!) until the first parser is able to parse the message. After one +parser has been found, the message is considered parsed and no others parsers are called on that +message. +<p>Side-note: this method implies there are some "not-so-dirty" tricks available to modify +the message by a parser module that declares itself as "unable to parse" but still does +some message modification. This was not a primary design goal, but may be utilized, and the +interface probably extended, to support generic filter modules. These would need to go +to the root of the parser chain. As mentioned, the current system already supports this. +<p>The position inside the parser chain can be thought of as a priority: parser sitting +earlier in the chain take precedence over those sitting later in it. So more specific +parser should go ealier in the chain. A good example of how this works is the default parser +set provided by rsyslog: rsyslog.rfc5424 and rsyslog.rfc3164, each one parses according to the +rfc that has named it. RFC5424 was designed to be distinguishable from RFC3164 message by the +sequence "1 " immediately after the so-called PRI-part (don't worry about these words, it is +sufficient if you understand there is a well-defined sequence used to indentify RFC5424 +messages). In contrary, RFC3164 actually permits everything as a valid message. Thus the +RFC3164 parser will always parse a message, sometimes with quite unexpected outcome (there is +a lot of guesswork involved in that parser, which unfortunately is unavoidable due to +existing techology limits). So the default parser chain is to try the RFC5424 parser first +and after it the RFC3164 parser. If we have a 5424-formatted message, that parser will +identify and parse it and the rsyslog engine will stop processing. But if we receive a +legacy syslog message, the RFC5424 will detect that it can not parse it, return this status +to the engine which then calls the next parser inside the chain. That usually happens to be +the RFC3164 parser, which will always process the message. But there could also be any other +parser inside the chain, and then each one would be called unless one that is able to parse +can be found. +<p>If we reversed the parser order, RFC5424 messages would incorrectly parsed. Why? Because the +RFC3164 parser will always parse every message, so if it were asked first, it would parse +(and misinterpret) the 5424-formatted message, return it did so and the rsyslog engine would +never call the 5424 parser. So oder of sequence is very important. +<p>What happens if no parser in the chain could parse a message? Well, then we could not +obtain the in-memory representation that is needed to further process the message. In that +case, rsyslog has no other choice than to discard the message. If it does so, it will emit +a warning message, but only in the first 1,000 incidents. This limit is a safety measure +against message-loops, which otherwise could quickly result from a parser chain +misconfiguration. <b>If you do not tolerate loss of unparsable messages, you must ensure +that each message can be parsed.</b> You can easily achive this by always using the +"rsyslog-rfc3164" parser as the <i>last</i> parser inside parser chains. That may result +in invalid parsing, but you will have a chance to see the invalid message (in debug mode, +a warning message will be written to the debug log each time a message is dropped due to +inability to parse it). +<h3>Where are parser chains used?</h3> +<p>We now know what parser chains are and how they operate. The question is now how many +parser chains can be active and how it is decicded which parser chain is used on which message. +This is controlled via <a href="multi_ruleset.html">rsyslog's rulesets</a>. In short, multiple +rulesets can be defined and there always exist at least one ruleset (for specifcs, follow +the <a href="multi_ruleset.html">link</a>). A parser chain is bound to a specific ruleset. +This is done by virtue of defining parsers via the +<a href="rsconf1_rulesetparser.html">$RulesetParser</a> configuration directive (for specifics, +see there). If no such directive is specified, the default parser chain is used. As of this +writing, the default parser chain always consists of "rsyslog.rfc5424", "rsyslog.rfc3164", in +that order. As soon as a parser is configured, the default list is cleared and the new parser +is added to the end of the (initially empty) ruleset's parser chain. +<p>The important point to know is that parser chains are defined on a per-ruleset basis. +<h3>Can I use different parser chains for different devices?</h3> +<p>The correct answer is: generally yes, but it depends. First of all, remember that input +modules (and specific listeners) may be bound to specific rulesets. As parser chains "reside" +in rulesets, binding to a ruleset also binds to the parser chain that is bound to that ruleset. +As a number one prequisite, the input module must support binding to different rulesets. Not +all do, but their number is growing. For example, the important +<a href="imudp.html">imudp</a> and <a href="imtcp.html">imtcp</a> input modules support +that functionality. Those that do not (for example <a href="im3195">im3195</a>) can only +utilize the default ruleset and thus the parser chain defined in that ruleset. +<p>If you do not know if the input module in question supports ruleset binding, check +its documentation page. Those that support it have the requiered directives. +<p>Note that it is currently under evaluation if rsyslog will support binding parser chains +to specific inputs directly, without depending on the ruleset. There are some concerns that +this may not be necessary but adds considerable complexity to the configuration. So this may +or may not be possible in the future. In any case, if we decide to add it, input modules +need to support it, so this functionality would require some time to implement. +<p>The coockbook recipe for using different parsers for different devices is given +as an actual in-depth example in the <a href="rscon1_rulesetsparser.html">$RulesetParser</a> +configuration directive doc page. In short, it is acomplished by defining specific rulesets +for the required parser chains, definining different listener ports for each of the devices +with different format and binding these listeners to the correct ruleset (and thus parser +chains). Using that approach, a variety of different message formats can be supported +via a single rsyslog instance. + +<h2>Which message parsers are available</h2> +<p>As of this writing, there exist only two message parsers, one for RFC5424 format and one for +legacy syslog (loosely described in +<a href="http://tools.ietf.org/html/rfc3164">RFC3164</a>). These parsers are built-in and +must not be explicitely loaded. However, message parsers can be added with relative ease +by anyone knowing to code in C. Then, they can be loaded via $ModLoad just like any +other loadable module. It is expected that the rsyslog project will be contributed additional +message parsers over time, so that at some point there hopefully is a rich choice of them +(I intend to add a browsable repository as soon as new parsers pop up). +<h3>How to write a message parser?</h3> +<p>As a prequisite, you need to know the exact format that the device is sending. Then, you need +moderate C coding skills, and a little bit of rsyslog internals. I guess the rsyslog specific part +should not be that hard, as almost all information can be gained from the existing parsers. They +are rather simple in structure and can be found under the "./tools" directory. They are named +pmrfc3164.c and pmrfc5424.c. You need to follow the usual loadable module guidelines. +It is my expectation that writing a parser should typically not take longer than a single +day, with maybe a day more to get aquainted with rsyslog. Of course, I am not sure if the number +is actually right. +<p>If you can not program or have no time to do it, Adiscon can also write a message parser +for you as +part of the <a href="http://www.rsyslog/professional-services">rsyslog professional services +offering</a>. +<h2>Conclusion</h2> +<p>Malformed syslog messages are a pain and unfortunately often seen in practice. Message parsers +provide a fast and efficient solution for this problem. Different parsers can be defined for +different devices, and they all convert message information into rsyslog's well-defined +internal format. Message parsers were first introduced in rsyslog 5.3.4 and also offer +some interesting ideas that may be explored in the future - up to full message normalization +capabilities. It is strongly recommended that anyone with a heterogenous environment take +a look at message parser capabilities. + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> +</body> +</html> diff --git a/doc/multi_ruleset.html b/doc/multi_ruleset.html index 8d8c614f..da65b4ba 100644 --- a/doc/multi_ruleset.html +++ b/doc/multi_ruleset.html @@ -23,7 +23,7 @@ write it to a file or forward it to a remote logging server. <p>A traditional configuration file is made up of one or more of these rules. When a new message arrives, its processing starts with the first rule (in order of appearance in rsyslog.conf) and continues for each rule until either all rules have been processed or -a so-called "e;discard" action happens, in which case processing stops and the +a so-called "discard" action happens, in which case processing stops and the message is thrown away (what also happens after the last rule has been processed). <p>The <b>multi-ruleset</b> support now permits to specify more than one such rule sequence. @@ -66,11 +66,18 @@ to seperate the messages by any other method. <pre>$InputTCPServerBindRuleset <name> </pre> -directive. Note that "name"e; must be the name of a ruleset that is already defined +directive. Note that "name" must be the name of a ruleset that is already defined at the time the bind directive is given. There are many ways to make sure this happens, but I personally think that it is best to define all rule sets at the top of rsyslog.conf and define the inputs at the bottom. This kind of reverses the traditional recommended ordering, but seems to be a really useful and straightforward way of doing things. +<h2>Why are rulesets important for different parser configurations?</h2> +<p>Custom message parsers, used to handle differnet (and potentially otherwise-invalid) +message formats, can be bound to rulesets. So multiple rulesets can be a very useful +way to handle devices sending messages in different malformed formats in a consistent +way. Unfortunately, this is not uncommon in the syslog world. An in-depth explanation +with configuration sample can be found at the +<a href="rsconf1_rulesetparser.html">$RulesetParser</a> configuration directive. <h2>Can I use a different Ruleset as the default?</h2> <p>This is possible by using the @@ -249,11 +256,12 @@ $InputTCPServerBindRuleset remote10516 $InputTCPServerRun 10516 </pre> -<p>Note that the "mail.*" rule inside the "remote10516"e; ruleset does +<p>Note that the "mail.*" rule inside the "remote10516" ruleset does not affect processing inside any other rule set, including the default rule set. <h2>Performance</h2> +<h3>Fewer Filters</h3> <p>No rule processing can be faster than not processing a rule at all. As such, it is useful for a high performance system to identify disjunct actions and try to split these off to different rule sets. In the example section, we had a case where three different tcp listeners @@ -263,6 +271,25 @@ is no need to check the reception service - instead messages are automatically p right rule set and can be processed by very simple rules (maybe even with "*.*"-filters, the fastest ones available). +<h3>Partitioning of Input Data</h3> +<p>Starting with rsyslog 5.3.4, rulesets permit higher concurrency. They offer the ability +to run on their own "main" queue. What that means is that a own queue is associated +with a specific rule set. That means that inputs bound to that ruleset do no longer need +to compete with each other when they enqueue a data element into the queue. Instead, enqueue +operations can be completed in parallel. +<p>An example: let us assume we have three TCP listeners. Without rulesets, each of them +needs to insert messages into the main message queue. So if each of them wants to +submit a newly arrived message into the queue at the same time, only one can do so +while the others need to wait. With multiple rulesets, its own queue can be created for each +ruleset. If now each listener is bound to its own ruleset, concurrent message submission is +possible. On a machine with a sufficiently large number of corse, this can result in +dramatic performance improvement. +<p>It is highly advised that high-performance systems define a dedicated ruleset, with a +dedicated queue for each of the inputs. +<p>By default, rulesets do <b>not</b> have their own queue. It must be activated via the +<a href="rsconf1_rulesetcreatemainqueue.html">$RulesetCreateMainQueue</a> directive. + +<h3>Future Enhancements</h3> <p>In the long term, multiple rule sets will probably lay the foundation for even better optimizations. So it is not a bad idea to get aquainted with them. diff --git a/doc/omoracle.html b/doc/omoracle.html index cfcf277f..2bb6aa5d 100644 --- a/doc/omoracle.html +++ b/doc/omoracle.html @@ -13,10 +13,11 @@ <p><b>Available since: </b>: 4.3.0 <p><b>Status: </b>: contributed module, not maitained by rsyslog core authors <p><b>Description</b>:</p> -<p>This module provides native support for logging to Oracle databases. It offers -superior performance over the more generic <a href="omlibdbi.html">omlibdbi</a> module. -It also includes a number of enhancements, most importantly prepared statements and -batching, what provides a big performance improvements. +<p>This module provides native support for logging to Oracle +databases. It offers superior performance over the more +generic <a href="omlibdbi.html">omlibdbi</a> module. It also includes +a number of enhancements, most importantly prepared statements and +batching, what provides a big performance improvement. </p> <p>Note that this module is maintained by its original author. If you need assistance with it, it is suggested to post questions to the @@ -63,7 +64,7 @@ it is suggested to post questions to the $OmoracleStatement \ insert into foo(hostname,message)values(:host,:message) - Also note that identifiers to placeholders are arbitrarry. You + Also note that identifiers to placeholders are arbitrary. You need to define the properties on the template in the correct order you want them passed to the statement! </pre> diff --git a/doc/omruleset.html b/doc/omruleset.html new file mode 100644 index 00000000..41d6ccfc --- /dev/null +++ b/doc/omruleset.html @@ -0,0 +1,140 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>ruleset output module (omruleset)</title> +</head> +<body> +<a href="rsyslog_conf_modules.html">rsyslog module reference</a> + +<h1>ruleset output/including module (omruleset)</h1> +<p><b>Module Name: omruleset</b></p> +<p><b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p> +<p><b>Available Since</b>: 5.3.4</p> +<p><b>Description</b>:</p> +<p>This is a very special "output" module. It permits to pass a message object +to another rule set. While this is a very simple action, it enables very +complex configurations, e.g. it supports high-speed "and" conditions, sending +data to the same file in a non-racy way, include-ruleset functionality as well as +some high-performance optimizations (in case the rule sets have the necessary +queue definitions). +<p>While it leads to a lot of power, this output module offers seamingly easy functionaltiy. +The complexity (and capablities) arise from how everthing can be combined. +<p>With this module, a message can be sent to processing to another ruleset. This is somewhat +similar to a "#include" in the C programming language. However, one needs to keep +on the mind that a ruleset can contain its own queue and that a queue can run in various modes. +<p>Note that if no queue is defined in the ruleset, the message is enqueued into the main message +queue. This most often is not optimal and means that message processing may be severely defered. +Also note that when the ruleset's target queue is full and no free space can be aquired +within the usual timeout, the message object may actually be lost. This is an extreme scenario, +but users building an audit-grade system need to know this restriction. For regular installations, +it should not really be relevant. +<p><b>At minimum, be sure you understand the +<a href="rsconf1_rulesetcreatemainqueue.html">$RulesetCreateMainQueue</a> +directive as well as the importance of statement order in rsyslog.conf before using omruleset!</b> +<p><b>Recommended Use:</b> +<ul> +<li>create rulesets specifically for omruleset +<li>create these rulesets with their own main queue +<li> decent queueing parameters (sizes, threads, etc) should be used +for the ruleset main queue. If in doubt, use the same parameters as for the +overall main queue. +<li>if you use multiple levels of ruleset nesting, double check for endless loops - the rsyslog +engine does not detect these +</ul> + +<p><b>Configuration Directives</b>:</p> +<ul> +<li><b>$ActionOmrulesetRulesetName</b> ruleset-to-submit-to<br> +This directive specifies the name of the ruleset that the message +provided to omruleset should be submitted to. This ruleset must already have +been defined. Note that the directive is automatically reset after each +:omruleset: action and there is no default. This is done to prevent accidential +loops in ruleset definition, what can happen very quickly. +The :omruleset: action will NOT be honored if no ruleset name has been defined. As usual, +the ruleset name must be specified in front of the action that it modifies. +</ul> +<p><b>Examples:</b></p> +<p>This example creates a ruleset for a write-to-file action. The idea here +is that the same file is written based on multiple filters, problems occur if the file is used +together with a buffer. That is because file buffers are action-specific, and so some partial +buffers would be written. With omruleset, we create a single action inside its own ruleset and +then pass all messages to it whenever we need to do so. Of course, such a simple situation could +also be solved by a more complex filter, but the method used here can also be utilized in +more complex scenarios (e.g. with multiple listeners). The example tries to keep it simple. +Note that we create a ruleset-specific main queue (for simplicity with the default main queue +parameters) in order to avoid re-queueing messages back into the main queue. +</p> +<textarea rows="30" cols="80">$ModLoad omruleset + +# define ruleset for commonly written file +$RuleSet commonAction +$RulesetCreateMainQueue on +*.* /path/to/file.log + +#switch back to default ruleset +$ruleset RSYSLOG_DefaultRuleset + +# begin first action +# note that we must first specify which ruleset to use for omruleset: +$ActionOmrulesetRulesetName CommonAction +mail.info :omruleset: +#end first action + +# begin second action +# note that we must first specify which ruleset to use for omruleset: +$ActionOmrulesetRulesetName CommonAction +:FROMHOST, isequal, "myhost.example.com" :omruleset: +#end second action + +# of course, we can have "regular" actions alongside :omrulset: actions +*.* /path/to/general-message-file.log +</textarea> +<p>The next example is used to creat a high-performance nested and filter condition. Here, +it is first checked if the message contains a string "error". If so, the message is forwarded +to another ruleset which then applies some filters. The advantage of this is that we can use +high-performance filters where we otherwise would need to use the (much slower) expression-based +filters. Also, this enables pipeline processing, in that second ruleset is executed in +parallel to the first one.</p> +<textarea rows="30" cols="80">$ModLoad omruleset + +# define "second" ruleset +$RuleSet nested +$RulesetCreateMainQueue on # again, we use our own queue +mail.* /path/to/mailerr.log +kernel.* /path/to/kernelerr.log +auth.* /path/to/autherr.log + +#switch back to default ruleset +$ruleset RSYSLOG_DefaultRuleset + +# begin first action - here we filter on "error" +# note that we must first specify which ruleset to use for omruleset: +$ActionOmrulesetRulesetName nested +:msg, contains, "error :omruleset: +#end first action + +# begin second action - as an example we can do anything else in +# this processing. Note that these actions are processed concurrently +# to the ruleset "nested" +:FROMHOST, isequal, "myhost.example.com" /path/to/host.log +#end second action + +# of course, we can have "regular" actions alongside :omrulset: actions +*.* /path/to/general-message-file.log +</textarea> +<p><b>Caveats/Known Bugs:</b> +<p>The current configuration file language is not really adequate for a complex construct +like omruleset. Unfortunately, more important work is currently preventing me from redoing the +config language. So use extreme care when nesting rulesets and be sure to test-run your +config before putting it into production, ensuring you have a suffciently large probe +of the traffic run over it. If problems arise, the +<a href="troubleshoot.html">rsyslog debug log</a> is your friend. +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/omstdout.html b/doc/omstdout.html new file mode 100644 index 00000000..0bd10cfb --- /dev/null +++ b/doc/omstdout.html @@ -0,0 +1,42 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html><head> +<meta http-equiv="Content-Language" content="en"> +<title>stdout output module (omstdout)</title> +</head> +<body> +<a href="rsyslog_conf_modules.html">rsyslog module reference</a> + +<h1>stdout output module (stdout)</h1> +<p><b>Module Name: omstdout</b></p> +<p><b>Author: </b>Rainer Gerhards +<rgerhards@adiscon.com></p> +<p><b>Available Since</b>: 4.1.6</p> +<p><b>Description</b>:</p> +<p>This module writes any messages that are passed to it to stdout. +It was developed for the rsyslog test suite. However, there may +(limited) other uses exists. Please not that we do not put too much +effort into the quality of this module as we do not expect it to +be used in real deployments. If you do, please drop us a note so +that we can enhance its priority! +<p><b>Configuration Directives</b>:</p> +<ul> +<li><b>$ActionOMStdoutArrayInterface</b> [on|<b>off</b><br> +This setting instructs omstdout to use the alternate +array based method of parameter passing. If used, the values +will be output with commas between the values but no other padding bytes. +This is a test aid for the alternate calling interface. +<li><b>$ActionOMStdoutEnsureLFEnding</b> [<b>on</b>|off<br> +Makes sure that each message is written with a terminating LF. This is needed for +the automatted tests. If the message contains a trailing LF, none is added. +</ul> +<b>Caveats/Known Bugs:</b> +<p>Currently none known. +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] +[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> +project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. +Released under the GNU GPL version 3 or higher.</font></p> +</body></html> diff --git a/doc/omudpspoof.html b/doc/omudpspoof.html index e5f963c7..64f83752 100644 --- a/doc/omudpspoof.html +++ b/doc/omudpspoof.html @@ -16,23 +16,23 @@ spoof the sender address. Also, it enables to circle through a number of source ports. <p><b>Configuration Directives</b>:</p> <ul> -<li><b>$ActionUDPSpoofSourceNameTemplate</b> <templatename><br> +<li><b>$ActionOMUDPSpoofSourceNameTemplate</b> <templatename><br> This MUST be specified. It is the name of the template that contains a numerical IP address that is to be used as the source system IP address. While it may often be a constant value, it can be generated as usual via the property replacer, as long as it is a valid IPv4 address. -<li><b>$ActionUDPSpoofTargetHost</b> <hostname><br> +<li><b>$ActionOMUDPSpoofTargetHost</b> <hostname><br> Host that the messages shall be sent to. <li><b>$ActionUDPSpoofTargetPort</b> <port><br> Remote port that the messages shall be sent to. -<li><b>$ActionUDPSpoofDefaultTemplate</b> <templatename><br> +<li><b>$ActionOMUDPSpoofDefaultTemplate</b> <templatename><br> This setting instructs omudpspoof to use a template different from the default template for all of its actions that do not have a template specified explicitely. -<li><b>$ActionUDPSpoofSourcePortStart</b> <number><br> +<li><b>$ActionOMUDPSpoofSourcePortStart</b> <number><br> Specifies the start value for circeling the source ports. Must be less than or equal to the end value. Default is 32000. -<li><b>$ActionUDPSpoofSourcePortEnd</b> <number><br> +<li><b>$ActionOMUDPSpoofSourcePortEnd</b> <number><br> Specifies the ending value for circeling the source ports. Must be less than or equal to the start value. Default is 42000. </ul> diff --git a/doc/professional_support.html b/doc/professional_support.html index 7724ede8..de3ac800 100644 --- a/doc/professional_support.html +++ b/doc/professional_support.html @@ -1,112 +1,27 @@ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<html><head> -<meta http-equiv="Content-Language" content="en"><title>Professional Support for Rsyslog</title> - +<html> +<head> +<meta http-equiv="Content-Language" content="en"> +<title>Professional Support for Rsyslog</title> </head> + <body> -<h1>Professional Services for Rsyslog</h1> +<h1><a href="http://www.rsyslog.com/Article412.phtml">Professional Services for Rsyslog</a></h1> <p>Professional services are being offered by <a href="http://www.adiscon.com">Adiscon</a>, the company -that sponsors rsyslog development. For details, please contact <a href="mailto:info%40adiscon.com">Adiscon Sales</a>. </p> +that sponsors rsyslog development. For details, please contact <a href="mailto:info%40adiscon.com">Adiscon Sales</a>.</p> -<h3>EMail Support Service</h3> -Price: 99.00 EURO <br> -Duration: 180 days -<br> -Support level: 8x5 -<p>Purchase rsyslog support directly from the source. This -contract provides priority email support. It is a great option if you -need to provide proof of software support in your organization. This -contract provides</p> -<ul> -<li>unlimited email support tickets during validity -</li> -<li><span style="font-weight: bold;">fixes for</span> -current and <span style="font-weight: bold;">past rsyslog -releases</span> -</li> -<li>advise on how to implement rsyslog in the best possible -way. -</li> -</ul> -<p>Under this contract, fixes for old rsyslog releases will be -provided / created, provided that it is possible to do that with the -code base in question. Phone support is not included.</p> -<h3>Custom-Written Config File</h3> -Price: 29.00 EURO -<br> -Duration: N/A -<br> -Support level: 8x5 -<p>Creating rsyslog config files is easy - but if you would like -to have that extra feature and have no time to do it, this service is -for you. Important: BEFORE you purchase this service, contact us and -inquire (via <a href="mailto:info%40adiscon.com">info@adiscon.com</a>) -whether or not your desired result can be achieved via rsyslog. Once -this is clear, order the service and we will ship a custom-made -configuration file within 5 working days (at latest, most often much -faster). For security reasons, we will not put passwords into the -configuration file, but will place easy to read comments in the places -where you need to put them in. The agreement is governed under German -law. You may also purchase this service if you would like to have your -own configuration file reviewed, e.g. for auditing purposes.</p> -<h3>Local Installation Support</h3> -<p>If you intend to install rsyslog on your system but would like -to do so with minimal effort and according to your specification, you -can ask us to perform the installation for you. You get a perfect -installation, exactly like you needed, but without a need to -touch the system. This is a perfect choice for the busy administrator! -<p>In order to perform this work, we just need ssh access to your -system and the proper permissions. -<p>We charge a low one-time fee for this service. For details, please -contact <a href="mailto:info@adiscon.com">info@adiscon.com</a>. -<h3>Local Installation Maintenance</h3> -<p>If you used our services to set up the system, why not keep it -running perfectly with maintenance support? Under this contract, we -assure you run a recent build that does not interfere with your -environment and we even carry out change requests you may have. So this -is a hassle-free, everything cared about solution. -<p>Again, all we need to have is ssh access and the proper permissions -to your machine. Of course, work will only be carried out when you -expect us to do so. You are always in control of what happens. This -is a perfect outsourcing solution for those who would like to run -a great logging system but can not afford the time to keep it -in perfect shape! -<p>We charge a low monthly fee for this service. For details, please -contact <a href="mailto:info@adiscon.com">info@adiscon.com</a>. -<h3>Custom Development</h3> -<p>Do you need an exotic feature that otherwise would not be implemented? -Do you need something really quick, quicker than it is available via -the regular development schedule? Then, you may consider funding -development for a specific functionality. We are always looking for -interesting projects. If you hire us to to do the job, you can be sure -to get the best possible and probably quickest solution, because we are -obviously at the heart of the source code. No need to get aquainted to -anything, no risk of misunderstanding program concepts. Benefit from -our vast syslog experience.</p> -<p>Please note that custom development is not limited to rsyslog. We offer -a number of logging solutions and can also work as part of your time -for specific requirements. The opportunities are endless, just ask. We -will work with you on your requirements and provide a quote on the -estimated cost. Just write to <a href="mailto:sales@adiscon.com">sales@adiscon.com</a> for details.</p><h3>Consulting Services</h3> -<p>Do you have demanding logging requirements? Why not talk to a -real logging professional? Instead of trying to find the solution -like a needle in the haystack, talk to the team that brought rsyslog, -phpLogCon, the Windows MonitorWare products and other logging -solutions. We sweat logging for over 15 years now and can help quickly. -Depending on your needs, consulting can be carried out via email, the -phone or on your premises (for larger or local projects). Everything is -possible, it just depends on your needs. Consulting services are -available in English and German. Just mail <a href="mailto:sales@adiscon.com">sales@adiscon.com</a> what you are interested in and we will work with you on a proposal that fits your needs. -</p><p></p><p>All agreements are -governed under German law. -</p> +<p>In an effort to keep the documentation set clean and enable Adiscon to enhance the +professional offerings, the actual +<a href="http://www.rsyslog.com/Article412.phtml">rsyslog service offering</a> +has been moved to a +<a href="http://www.rsyslog.com/Article412.phtml">separate page</a>. +Please follow that link for details. <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> -<p><font size="2">This documentation is part of the -<a href="http://www.rsyslog.com/">rsyslog</a> +<p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer -Gerhards</a> and +Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> -</body></html> +</body> +</html> diff --git a/doc/rainerscript.html b/doc/rainerscript.html index ef0e41cb..63a79040 100644 --- a/doc/rainerscript.html +++ b/doc/rainerscript.html @@ -51,13 +51,25 @@ of a and b should be tested as "a <> b". The "not" operator should be reserved to cases where it actually is needed to form a complex boolean expression. In those cases, parenthesis are highly recommended. +<h2>Functions</h2> +<p>RainerScript supports a currently quite limited set of functions: +<ul> +<li>getenv(str) - like the OS call, returns the value of the environment +variable, if it exists. Returns an empty string if it does not exist. +<li>strlen(str) - returns the length of the provided string +<li>tolower(str) - converts the provided string into lowercase +</ul> +<p>The following example can be used to build a dynamic filter based on some environment +variable: +<pre> +if $msg contains getenv('TRIGGERVAR') then /path/to/errfile +</pre> <p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer -Gerhards</a> and +Copyright © 2008, 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> -</body></html>
\ No newline at end of file +</body></html> diff --git a/doc/rfc5424layers.png b/doc/rfc5424layers.png Binary files differnew file mode 100644 index 00000000..70192cc0 --- /dev/null +++ b/doc/rfc5424layers.png diff --git a/doc/rsconf1_abortonuncleanconfig.html b/doc/rsconf1_abortonuncleanconfig.html new file mode 100644 index 00000000..77526c07 --- /dev/null +++ b/doc/rsconf1_abortonuncleanconfig.html @@ -0,0 +1,37 @@ +<html> +<head> +<title>rsyslog.conf file</title> +</head> +<body> +<a href="rsyslog_conf_global.html">rsyslog.conf configuration directive</a> + +<h2>$AboortOnUncleanConfig</h2> +<p><b>Type:</b> global configuration directive</p> +<p><b>Parameter Values:</b> boolean (on/off, yes/no)</p> +<p><b>Available since:</b> 5.3.1+</p> +<p><b>Default:</b> off</p> +<p><b>Description:</b></p> +<p>This directive permits to prevent rsyslog from running when the configuration file +is not clean. "Not Clean" means there are errors or some other annoyances that rsyslgod +reports on startup. This is a user-requested feature to have a strict startup mode. Note +that with the current code base it is not always possible to differentiate between an +real error and a warning-like condition. As such, the startup will also prevented if +warnings are present. I consider this a good thing in being "strict", but I admit +there also currently is no other way of doing it. +<p><b>Caveats:</b></p> +Note that the consequences of a failed rsyslogd startup can be much more serious than a +startup with only partial configuration. For example, log data may be lost or systems that +depend on the log server in question will not be able to send logs, what in the ultimate +result could result in a system hang on those systems. Also, the local system may hang when +the local log socket has become full and is not read. There exist many such scenarios. +As such, it is strongly recommended not to turn on this directive. + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 2 or higher.</font></p> +</body> +</html> diff --git a/doc/rsconf1_omfileforcechown.html b/doc/rsconf1_omfileforcechown.html new file mode 100644 index 00000000..7415a6f6 --- /dev/null +++ b/doc/rsconf1_omfileforcechown.html @@ -0,0 +1,64 @@ +<html> +<head> +<title>rsyslog.conf file</title> +</head> +<body> +<a href="rsyslog_conf_global.html">back</a> + +<h2>$omfileForceChown</h2> +<p><b>Type:</b> global configuration directive</p> +<p><b>Parameter Values:</b> boolean (on/off, yes/no)</p> +<p><b>Available since:</b> 4.7.0+, 5.3.0+</p> +<p><b>Default:</b> off</p> +<p><b>Description:</b></p> +<p>Forces rsyslogd to change the ownership for output files that already exist. Please note +that this tries to fix a potential problem that exists outside the scope of rsyslog. Actually, +it tries to fix invalid ownership/permission settings set by the original file creator. +<p>Rsyslog changes the ownership during initial execution with root privileges. When a privelege +drop is configured, privileges are dropped after the file owner ship is changed. Not that this currently +is a limitation in rsyslog's privilege drop code, which is on the TODO list to be removed. See Caveats +section below for the important implications. +<p><b>Caveats:</b></p> +<p>This directive tries to fix a problem that actually is outside the scope of rsyslog. As such, +there are a couple of restrictions and situations in which it will not work. <b>Users are strongly +encouraged to fix their system instead of turning this directive on</b> - it should only be used +as a last resort. +<p>At least in the following scenario, this directive will fail expectedly: +<p>It does not address +the situation that someone changes the ownership *after* rsyslogd has started. +Let's, for example, consider a log rotation script. +<ul> +<li>rsyslog is started +<li>ownership is changed +<li>privileges dropped +<li>log rotation (lr) script starts +<li>lr removes files +<li>lr creates new files with root:adm (or whatever else) +<li>lr HUPs rsyslogd +<li>rsyslogd closes files +<li>rsyslogd tries to open files +<li>rsyslogd tries to change ownership --> fail as we are non-root now +<li>file open fails +</ul> + +Please note that once the privilege drop code is refactored, this directive will +no longer work, because then privileges will be dropped before any action is performed, +and thus we will no longer be able to chown files that do not belong to the +user rsyslogd is configured to run under. + +<p>So <b>expect the directive to go away</b>. It will not +be removed in version 4, but may disappear at any time for any version greater than 4. + +<p><b>Sample:</b></p> +<p><code><b>$FileOwner loguser</b> +<br><b>$omfileForceChown on</b></code></p> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2007 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL +version 2 or higher.</font></p> +</body> +</html> diff --git a/doc/rsconf1_rulesetcreatemainqueue.html b/doc/rsconf1_rulesetcreatemainqueue.html new file mode 100644 index 00000000..5c1e0dec --- /dev/null +++ b/doc/rsconf1_rulesetcreatemainqueue.html @@ -0,0 +1,83 @@ +<html> +<head> +<title>RulesetCreateMainQueue - rsyslog.conf file</title> +</head> +<body> +<a href="rsyslog_conf_global.html">rsyslog.conf configuration directive</a> + +<h2>$RulesetCreateMainQueue</h2> +<p><b>Type:</b> ruleset-specific configuration directive</p> +<p><b>Parameter Values:</b> boolean (on/off, yes/no)</p> +<p><b>Available since:</b> 5.3.5+</p> +<p><b>Default:</b> off</p> +<p><b>Description:</b></p> +<p> +Rulesets may use their own "main" message queue for message submission. Specifying +this directive, <b>inside a ruleset definition</b>, turns this on. This is both a performance +enhancement and also permits different rulesets (and thus different inputs within the same +rsyslogd instance) to use different types of main message queues. +<p>The ruleset queue is created with the parameters that are specified for the main message +queue at the time the directive is given. If different queue configurations are desired, +different main message queue directives must be used in front of the $RulesetCreateMainQueue +directive. Note that this directive may only be given once per ruleset. If multiple statements +are specified, only the first is used and for the others error messages are emitted. +<p>Note that the final set of ruleset configuration directives specifies the parameters for +the default main message queue. +<p>To learn more about this feature, please be sure to read about +<a href="multi_ruleset.html">multi-ruleset support in rsyslog</a>. +<p><b>Caveats:</b></p> +The configuration statement "$RulesetCreateMainQueue off" has no effect at all. +The capability to specify this is an artifact of the current (ugly!) configuration +language. + +<p><b>Example:</b></p> +<p>This example sets up a tcp server with three listeners. Each of these +three listener is bound to a specific ruleset. As a performance optimization, +the rulesets all receive their own private queue. The result is that received messages +can be independently processed. With only a single main message queue, we would have +some lock contention between the messages. This does not happen here. Note that in this +example, we use different processing. Of course, all messages could also have been +processed in the same way ($IncludeConfig may be useful in that case!). +</p> +<textarea rows="30" cols="60">$ModLoad imtcp +# at first, this is a copy of the unmodified rsyslog.conf +#define rulesets first +$RuleSet remote10514 +$RulesetCreateMainQueue on # create ruleset-specific queue +*.* /var/log/remote10514 + +$RuleSet remote10515 +$RulesetCreateMainQueue on # create ruleset-specific queue +*.* /var/log/remote10515 + +$RuleSet remote10516 +$RulesetCreateMainQueue on # create ruleset-specific queue +mail.* /var/log/mail10516 +& ~ +# note that the discard-action will prevent this messag from +# being written to the remote10516 file - as usual... +*.* /var/log/remote10516 + +# and now define listners bound to the relevant ruleset +$InputTCPServerBindRuleset remote10514 +$InputTCPServerRun 10514 + +$InputTCPServerBindRuleset remote10515 +$InputTCPServerRun 10515 + +$InputTCPServerBindRuleset remote10516 +$InputTCPServerRun 10516 +</textarea> +<p>Note the positions of the directives. With the current config language, +position is very important. This is ugly, but unfortunately the way it currently +works. +</p> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 2 or higher.</font></p> +</body> +</html> diff --git a/doc/rsconf1_rulesetparser.html b/doc/rsconf1_rulesetparser.html new file mode 100644 index 00000000..84350431 --- /dev/null +++ b/doc/rsconf1_rulesetparser.html @@ -0,0 +1,119 @@ +<html> +<head> +<title>RulesetParser - rsyslog.conf file</title> +</head> +<body> +<a href="rsyslog_conf_global.html">rsyslog.conf configuration directive</a> + +<h2>$RulesetParser</h2> +<p><b>Type:</b> ruleset-specific configuration directive</p> +<p><b>Parameter Values:</b> string</p> +<p><b>Available since:</b> 5.3.4+</p> +<p><b>Default:</b> rsyslog.rfc5424;rsyslog.rfc5425</p> +<p><b>Description:</b></p> +<p> +This directive permits to specify which +<a href="messageparser.html">message parsers</a> should be used for the ruleset +in question. It no ruleset is explicitely specified, the default ruleset is used. Message +parsers are contained in (loadable) parser modules with the most common cases +(RFC3164 and RFC5424) being build-in into rsyslogd. +<p>When this directive is specified the first time for a ruleset, it will not only add the +parser to the ruleset's parser chain, it will also wipe out the default parser chain. +So if you need to have +them in addition to the custom parser, you need to specify those as well. +<p>Order of directives is important. Parsers are tried one after another, in the order +they are specified inside the config. As soon as a parser is able to parse the message, +it will do so and no other parsers will be executed. If no matching parser can be found, +the message will be discarded and a warning message be issued (but only for the first +1,000 instances of this problem, to prevent message generation loops). +<p>Note that the rfc3164 parser will <b>always</b> be able to parse a message - it may +just not be the format that you like. This has two important implications: 1) always place +that parser at the END of the parser list, or the other parsers after it will never +be tried and 2) if you would like to make sure no message is lost, placing the rfc3164 +parser at the end of the parser list ensures that. +<p>Multiple parser modules are very useful if you have various devices that emit +messages that are malformed in various ways. The route to take then is +<ul> +<li>make sure you find a custom parser for that device; if there is no one, you +may consider writing one yourself (it is not that hard) or getting one written +as part of +<a href="http://www.rsyslog.com/professional-servcies">Adiscon's professional services +for rsyslog</a>. +<li>load your custom parsers via $ModLoad +<li>create a ruleset for each malformed format; assign the custom parser to it +<li>create a specific listening port for all devices that emit the same +malformed format +<li>bind the listener to the ruleset with the required parser +</ul> +<p>Note that it may be cumbersome to add all rules to all rulesets. To avoid this, +you can either use $Include or <a href="omruleset.html">omruleset</a> +(what probably provides the best solution). +<p>More information about rulesets in general can be found in +<a href="multi_ruleset.html">multi-ruleset support in rsyslog</a>. +<p><b>Caveats:</b></p> +<p>currently none known</p> + +<p><b>Example:</b></p> +<p>This example assumes there are two devices emiting malformed messages via UDP. +We have two custom parsers for them, named "device1.parser" and +"device2.parser". In addition to that, we have a number of other +devices sending wellformed messages, also via UDP. +<p>The solution is to listen for data from the two devices on two special +ports (10514 and 10515 in this example), create a ruleset for each and +assign the custom parsers to them. The rest of the messages are received via +port 514 using the regular parsers. Processing shall be equal for all messages. +So we simply forward the malformed messages to the regular queue once they are parsed (keep +in mind that a message is never again parsed once any parser properly processed it). +</p> +<textarea rows="40" cols="80">$ModLoad imudp +$ModLoad pmdevice1 # load parser "device1.parser" for device 1 +$ModLoad pmdevice2 # load parser "device2.parser" for device 2 + +# define ruleset for the first device sending malformed data +$Ruleset maldev1 +$RulesetCreateMainQueue on # create ruleset-specific queue +$RulesetParser "device1.parser" # note: this deactivates the default parsers +# forward all messages to default ruleset: +$ActionOmrulesetRulesetName RSYSLOG_DefaultRuleset +*.* :omruleset: + +# define ruleset for the second device sending malformed data +$Ruleset maldev2 +$RulesetCreateMainQueue on # create ruleset-specific queue +$RulesetParser "device2.parser" # note: this deactivates the default parsers +# forward all messages to default ruleset: +$ActionOmrulesetRulesetName RSYSLOG_DefaultRuleset +*.* :omruleset: + +# switch back to default ruleset +$Ruleset RSYSLOG_DefaultRuleset +*.* /path/to/file +auth.info @authlogger.example.net +# whatever else you usually do... + + +# now define the inputs and bind them to the rulesets +# first the default listener (utilizing the default ruleset) +$UDPServerRun 514 + +# now the one with the parser for device type 1: +$InputUDPServerBindRuleset maldev1 +$UDPServerRun 10514 + +# and finally the one for device type 2: +$InputUDPServerBindRuleset maldev2 +$UDPServerRun 10515 +</textarea> +<p>Note the positions of the directives. With the current config language, +<b>sequence of statements is very important</b>. This is ugly, but unfortunately +the way it currently works. +</p> + +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual +index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> +<p><font size="2">This documentation is part of the +<a href="http://www.rsyslog.com/">rsyslog</a> project.<br> +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and +<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 2 or higher.</font></p> +</body> +</html> diff --git a/doc/rsyslog_conf_global.html b/doc/rsyslog_conf_global.html index 45eeabe6..beb90e02 100644 --- a/doc/rsyslog_conf_global.html +++ b/doc/rsyslog_conf_global.html @@ -17,6 +17,8 @@ appear as implementation progresses. many parameter settings modify queue parameters. If in doubt, use the default, it is usually well-chosen and applicable in most cases.</p> <ul> +<li><a href="rsconf1_abortonuncleanconfig.html">$AbortOnUncleanConfig</a> - abort startup if there is +any issue with the config file</li> <li><a href="rsconf1_actionexeconlywhenpreviousissuspended.html">$ActionExecOnlyWhenPreviousIsSuspended</a></li> <li>$ActionName <a_single_word> - used primarily for documentation, e.g. when generating a configuration graph. Available sice 4.3.1. @@ -158,7 +160,12 @@ Usually that should not be a big issue, as the restart-type HUP can easily be re something along the lines of "/etc/init.d/rsyslog restart". </li> <li><a href="rsconf1_includeconfig.html">$IncludeConfig</a></li><li>MainMsgQueueCheckpointInterval <number></li> -<li>$MainMsgQueueDequeueBatchSize <number> [default 32]</li> +<li><b>$LogRSyslogStatusMessages</b> [<b>on</b>/off] - If set to on (the default), +rsyslog emits message on startup and shutdown as well as when it is HUPed. +This information might be needed by some log analyzers. If set to off, no such +status messages are logged, what may be useful for other scenarios. +[available since 4.7.0 and 5.3.0] +<li><b>$MainMsgQueueDequeueBatchSize</b> <number> [default 32]</li> <li>$MainMsgQueueDequeueSlowdown <number> [number is timeout in <i> micro</i>seconds (1000000us is 1sec!), default 0 (no delay). Simple rate-limiting!]</li> @@ -231,6 +238,7 @@ error recovery thus can handle write errors without data loss. Note that this op severely reduces the effect of zip compression and should be switched to off for that use case. Note that the default -off- is primarily an aid to preserve the traditional syslogd behaviour.</li> +<li><a href="rsconf1_omfileforcechown.html">$omfileForceChown</a> - force ownership change for all files</li> <li><b>$RepeatedMsgContainsOriginalMsg</b> [on/<b>off</b>] - "last message repeated n times" messages, if generated, have a different format that contains the message that is being repeated. Note that only the first "n" characters are included, with n to be at least 80 characters, most @@ -241,10 +249,14 @@ large enough for the whole message. (Introduced with 4.1.5). Once set, it affect <li><a href="rsconf1_resetconfigvariables.html">$ResetConfigVariables</a></li> <li><b>$Ruleset</b> <i>name</i> - starts a new ruleset or switches back to one already defined. All following actions belong to that new rule set. -the <i>name</i> does not yet exist, it is created. To swith back to rsyslog's +the <i>name</i> does not yet exist, it is created. To switch back to rsyslog's default ruleset, specify "RSYSLOG_DefaultRuleset") as the name. All following actions belong to that new rule set. It is advised to also read our paper on <a href="multi_ruleset.html">using multiple rule sets in rsyslog</a>.</li> +<li><b><a href="rsconf1_rulesetcreatemainqueue.html">$RulesetCreateMainQueue</a></b> on - creates +a ruleset-specific main queue. +<li><b><a href="rsconf1_rulesetparser.html">$RulesetParser</a></b> - enables to set +a specific (list of) message parsers to be used with the ruleset. <li><b>$OptimizeForUniprocessor</b> [on/<b>off</b>] - turns on optimizatons which lead to better performance on uniprocessors. If you run on multicore-machiens, turning this off lessens CPU load. The default may change as uniprocessor systems become less common. [available since 4.1.0]</li> @@ -252,19 +264,6 @@ default may change as uniprocessor systems become less common. [available since to sysklogd), the domain part from a name that is within the same domain as the receiving system is stripped. If set to on, full names are always used.</li> <li>$WorkDirectory <name> (directory for spool and other work files)</li> -<li>$UDPServerAddress <IP> (imudp) -- local IP -address (or name) the UDP listens should bind to</li> -<li>$UDPServerRun <port> (imudp) -- former --r<port> option, default 514, start UDP server on this -port, "*" means all addresses</li> -<li>$UDPServerTimeRequery <nbr-of-times> (imudp) -- this is a performance -optimization. Getting the system time is very costly. With this setting, imudp can -be instructed to obtain the precise time only once every n-times. This logic is -only activated if messages come in at a very fast rate, so doing less frequent -time calls should usually be acceptable. The default value is two, because we have -seen that even without optimization the kernel often returns twice the identical time. -You can set this value as high as you like, but do so at your own risk. The higher -the value, the less precise the timestamp. <li><a href="droppriv.html">$PrivDropToGroup</a></li> <li><a href="droppriv.html">$PrivDropToGroupID</a></li> <li><a href="droppriv.html">$PrivDropToUser</a></li> diff --git a/doc/rsyslog_conf_modules.html b/doc/rsyslog_conf_modules.html index f9bdad4a..a4c8c423 100644 --- a/doc/rsyslog_conf_modules.html +++ b/doc/rsyslog_conf_modules.html @@ -11,6 +11,7 @@ what they do (list is currently not complete)</p> <li><a href="omsnmp.html">omsnmp</a> - SNMP trap output module</li> <li><a href="omstdout.html">omtdout</a> - stdout output module (mainly a test tool)</li> <li><a href="omrelp.html">omrelp</a> - RELP output module</li> +<li><a href="omruleset.html">omruleset</a> - forward message to another ruleset</li> <li>omgssapi - output module for GSS-enabled syslog</li> <li><a href="ommysql.html">ommysql</a> - output module for MySQL</li> <li>ompgsql - output module for PostgreSQL</li> @@ -25,7 +26,7 @@ permits rsyslog to alert folks by mail if something important happens</li> - input module for text files</li> <li><a href="imrelp.html">imrelp</a> - RELP input module</li> -<li>imudp - udp syslog message input</li> +<li><a href="imudp.html">imudp</a> - udp syslog message input</li> <li><a href="imtcp.html">imtcp</a> - input plugin for plain tcp syslog</li> <li><a href="imgssapi.html">imgssapi</a> - diff --git a/doc/rsyslog_secure_tls.html b/doc/rsyslog_secure_tls.html index be2811f4..b15e5a4e 100644 --- a/doc/rsyslog_secure_tls.html +++ b/doc/rsyslog_secure_tls.html @@ -51,7 +51,7 @@ google_ad_height = 125; src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> </script> </span> -I private keys have become known to third parties, the system does not provide +If private keys have become known to third parties, the system does not provide any security at all. Also, our solution bases on X.509 certificates and a (very limited) chain of trust. We have one instance (the CA) that issues all machine certificates. The machine certificate indentifies a particular machine. hile in diff --git a/doc/src/classes.dia b/doc/src/classes.dia Binary files differindex 70e91566..8cfcbd0c 100644 --- a/doc/src/classes.dia +++ b/doc/src/classes.dia diff --git a/doc/src/rfc5424layers.dia b/doc/src/rfc5424layers.dia Binary files differnew file mode 100644 index 00000000..300b7796 --- /dev/null +++ b/doc/src/rfc5424layers.dia diff --git a/doc/status.html b/doc/status.html index 02cc0d70..ff056489 100644 --- a/doc/status.html +++ b/doc/status.html @@ -2,12 +2,12 @@ <html><head><title>rsyslog status page</title></head> <body> <h2>rsyslog status page</h2> -<p>This page reflects the status as of 2009-08-21.</p> +<p>This page reflects the status as of 2009-11-05.</p> <h2>Current Releases</h2> -<p><b>v5 development:</b> 5.1.4 [2009-08-20] - -<a href="http://www.rsyslog.com/Article392.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-170.phtml">download</a> +<p><b>v5 development:</b> 5.3.4 [2009-11-04] - +<a href="http://www.rsyslog.com/Article423.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-185.phtml">download</a> <br> <!-- not at the moment! <b>v4 development:</b> 4.5.1 [2009-07-15] - @@ -15,13 +15,24 @@ <a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-167.phtml">download</a></p> --> -<br><b>v4-beta:</b> 4.5.2 [2009-08-21] - -<a href="http://www.rsyslog.com/Article395.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-172.phtml">download</a></p> +<!-- not at the moment! +<br><b>v5-beta:</b> 5.1.6 [2009-10-15] - +<a href="http://www.rsyslog.com/Article413.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-180.phtml">download</a> +--> + +<br><b>v4-beta:</b> 4.5.6 [2009-11-05] - +<a href="http://www.rsyslog.com/Article425.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-186.phtml">download</a></p> + +<p><b>v5 stable:</b> 5.2.0 [2009-11-02] (recommended to use +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-183.phtml">5.3.3</a> instead) - +<a href="http://www.rsyslog.com/Article421.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-184.phtml">download</a> -<p><b>v4 stable:</b> 4.4.0 [2009-08-21] - -<a href="http://www.rsyslog.com/Article394.phtml">change log</a> - -<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-171.phtml">download</a> +<br><b>v4 stable:</b> 4.4.2 [2009-10-09] - +<a href="http://www.rsyslog.com/Article409.phtml">change log</a> - +<a href="http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-179.phtml">download</a> <br><b>v3 stable:</b> 3.22.1 [2009-07-02] - <a href="http://www.rsyslog.com/Article381.phtml">change log</a> - diff --git a/doc/syslog_parsing.html b/doc/syslog_parsing.html index 57da6657..1ccec6f1 100644 --- a/doc/syslog_parsing.html +++ b/doc/syslog_parsing.html @@ -176,6 +176,19 @@ $template, MalfromedMsgFormater,"%timegenerated% %fromhost% %rawmsg:::drop-last- <p>This will make your log much nicer, but not look perfect. Experiment a bit with the available properties and replacer extraction options to fine-tune it to your needs. +<h2>The Ultimate Solution...</h2> +<p>Is available with rsyslog 5.3.4 and above. Here, we can define so-called custom +parsers. These are plugin modules, written in C and adapted to a specific message format +need. The big plus of custom parsers is that they offer excellent performance and unlimited +possibilities - far better than any work-around could do. Custom parsers can be +<a href="rsconf1_rulesetparser.html">bound to specific rule sets</a> +(and thus listening) ports with relative ease. The only con is that they must be written. +However, if you are lucky, a parser for your device may already exist. If not, you can +opt to write it yourself, what is not too hard if you know some C. Alternatively, +Adiscon can program one for you as part of the +<a href="http://www.rsyslog.com/professional-services">rsyslog professional services offering</a>. +In any case, you should seriously consider custom parsers as an alternative if you can not +reconfigure your device to send decent message format. <h2>Wrap-Up</h2> <p>Syslog message format is not sufficiently standardized. There exists a weak "standard" format, which is used by a good number of implementations. However, there @@ -183,14 +196,15 @@ exist many others, including mainstream vendor implementations, which have a (sometimes horribly) different format. Rsyslog tries to deal with anomalies but can not guess right in all instances. If possible, the sender should be configured to submit well-formed messages. If that is not possible, you can work around these -issues with rsyslog's property replacer and template system. +issues with rsyslog's property replacer and template system. Or you can use a suitable +message parser or write one for your needs. <p>I hope this is a useful guide. You may also have a look at the <a href="troubleshoot.html">rsyslog troubleshooting guide</a> for further help and places where to ask questions. <p>[<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br> -Copyright © 2008 by <a href="http://www.gerhards.net/rainer">Rainer +Copyright © 2009 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL version 3 or higher.</font></p> </body></html> diff --git a/doc/tls_cert_server.html b/doc/tls_cert_server.html index 9c68db5d..9c024bc9 100644 --- a/doc/tls_cert_server.html +++ b/doc/tls_cert_server.html @@ -37,6 +37,15 @@ src="http://pagead2.googlesyndication.com/pagead/show_ads.js"> </script> </span> <p><center><img src="tls_cert_100.jpg"></center> +<p><i><font color="red"><b>Important:</b> Keep in mind that the order of configuration directives +is very important in rsyslog. As such, the samples given below do only work if the given +order is preserved.</font> Re-ordering the directives can break configurations and has broken them +in practice. If you intend to re-order them, please be sure that you fully understand how +the configuration language works and, most importantly, which statements form a block together. +Please also note that we understand the the current configuration file format is +ugly. However, there has been more important work in the way of enhancing it. If you would like +to contribute some time to improve the config file language, please let us know. Any help +is appreciated (be it doc or coding work!).</i> <p>Steps to do: <ul> <li>make sure you have a functional CA (<a href="tls_cert_ca.html">Setting up the CA</a>) @@ -51,6 +51,7 @@ #include "obj.h" #include "errmsg.h" #include "gss-misc.h" +#include "debug.h" MODULE_TYPE_LIB @@ -177,7 +178,7 @@ static int recv_token(int s, gss_buffer_t tok) | lenbuf[3]); tok->length = ntohl(len); - tok->value = (char *) malloc(tok->length ? tok->length : 1); + tok->value = (char *) MALLOC(tok->length ? tok->length : 1); if (tok->length && tok->value == NULL) { errmsg.LogError(0, NO_ERRCODE, "Out of memory allocating token data\n"); return -1; diff --git a/java/Makefile.am b/java/Makefile.am new file mode 100644 index 00000000..67f5eb43 --- /dev/null +++ b/java/Makefile.am @@ -0,0 +1,35 @@ +# very rough support for compiling the java components of rsyslog +# Some usage notes: you need to use the Sun JDK compiler (jdk-devel) +# with this. At least it didn't work for me with the eclipse compiler. +# There is no real installation support. If you intend to run a program, +# change to the ./java subdirectory and issue +# java -cp . <class> +# e.g.: java -cp . com.rsyslog.gui.diaggui.DiagGUI +# or any equivalent command. +# +# I am very glad to hear suggestions about how to improve this part +# of the build system. -- rgerhards, 2009-08-27 + +javadir = $(top_builddir)/java +JAVAROOT = $(javadir) +# I don't know why CLASSPATH_ENV works this way, but at least it works... +CLASSPATH_ENV = CLASSPATH=$(javadir):$$CLASSPATH + +JAVA_SOURCE_FILES = \ + com/rsyslog/lib/DiagSess.java \ + com/rsyslog/lib/SyslogMessage.java \ + com/rsyslog/lib/SyslogMsgConsumer.java \ + com/rsyslog/lib/SyslogTrafficGenerator.java \ + com/rsyslog/lib/SyslogSender.java \ + com/rsyslog/lib/UDPSyslogSender.java \ + com/rsyslog/diag/DiagTalker.java \ + com/rsyslog/gui/simpServ/simpServ.java \ + com/rsyslog/gui/simpServ/simpServConsumer.java \ + com/rsyslog/gui/msggen/MsgGen.java \ + com/rsyslog/gui/diaggui/Counters.java \ + com/rsyslog/gui/diaggui/DiagGUI.java + + +java_JAVA = $(JAVA_SOURCE_FILES) + +dist_java = $(JAVA_SOURCE_FILES) diff --git a/java/com/rsyslog/diag/DiagTalker.java b/java/com/rsyslog/diag/DiagTalker.java new file mode 100644 index 00000000..c4e77e95 --- /dev/null +++ b/java/com/rsyslog/diag/DiagTalker.java @@ -0,0 +1,70 @@ +/* A yet very simple tool to talk to imdiag. + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +package com.rsyslog.diag; +import java.io.*; +import java.net.*; + +public class DiagTalker { + public static void main(String[] args) throws IOException { + + Socket diagSocket = null; + PrintWriter out = null; + BufferedReader in = null; + final String host = "127.0.0.1"; + final int port = 13500; + + try { + diagSocket = new Socket(host, port); + diagSocket.setSoTimeout(0); /* wait for lenghty operations */ + out = new PrintWriter(diagSocket.getOutputStream(), true); + in = new BufferedReader(new InputStreamReader( + diagSocket.getInputStream())); + } catch (UnknownHostException e) { + System.err.println("can not resolve " + host + "!"); + System.exit(1); + } catch (IOException e) { + System.err.println("Couldn't get I/O for " + + "the connection to: " + host + "."); + System.exit(1); + } + + BufferedReader stdIn = new BufferedReader( + new InputStreamReader(System.in)); + String userInput; + + try { + while ((userInput = stdIn.readLine()) != null) { + out.println(userInput); + System.out.println("imdiag returns: " + in.readLine()); + } + } catch (SocketException e) { + System.err.println("We had a socket exception and consider this to be OK: " + + e.getMessage()); + } + + out.close(); + in.close(); + stdIn.close(); + diagSocket.close(); + } +} + diff --git a/java/com/rsyslog/gui/diaggui/Counters.java b/java/com/rsyslog/gui/diaggui/Counters.java new file mode 100644 index 00000000..363bff43 --- /dev/null +++ b/java/com/rsyslog/gui/diaggui/Counters.java @@ -0,0 +1,138 @@ +/* Display some basic rsyslogd counter variables. + * + * Please note that this program requires imdiag to be loaded inside rsyslogd. + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +package com.rsyslog.gui.diaggui; +import java.awt.*; +import java.awt.event.*; +import java.io.*; +import java.util.*; + +import com.rsyslog.lib.DiagSess; + +public class Counters extends Frame { + + private TextField MainQItems; + private TextField RefreshInterval; + private Checkbox AutoRefresh; + private DiagSess diagSess; + private Timer timer; + + private void createDiagSess() { + try { + diagSess = new DiagSess("127.0.0.1", 13500); // TODO: values from GUI + diagSess.connect(); + } + catch(IOException e) { + System.out.println("Exception trying to open diag session:\n" + e.toString()); + } + } + + private void createGUI() { + MainQItems = new TextField(); + MainQItems.setColumns(8); + Panel pCenter = new Panel(); + pCenter.setLayout(new FlowLayout()); + pCenter.add(new Label("MainQ Items:")); + pCenter.add(MainQItems); + + RefreshInterval = new TextField(); + RefreshInterval.setColumns(5); + RefreshInterval.setText("100"); + AutoRefresh = new Checkbox("Auto Refresh", false); + AutoRefresh.addItemListener(new ItemListener() { + public void itemStateChanged(ItemEvent e) { + setAutoRefresh(); + } + + }); + Button b = new Button("Refresh now"); + b.addActionListener(new ActionListener() { + public void actionPerformed(ActionEvent e) { + refreshCounters(); + } + }); + Panel pSouth = new Panel(); + pSouth.setLayout(new FlowLayout()); + pSouth.add(AutoRefresh); + pSouth.add(new Label("Interval (ms):")); + pSouth.add(RefreshInterval); + pSouth.add(b); + + pack(); + setTitle("rsyslogd Counters"); + setLayout(new BorderLayout()); + add(pCenter, BorderLayout.CENTER); + add(pSouth, BorderLayout.SOUTH); + setSize(400,500); + } + + public Counters() { + addWindowListener(new WindowAdapter() { + public void windowClosing(WindowEvent e){ + Counters.this.dispose(); + } + }); + createGUI(); + createDiagSess(); + setAutoRefresh(); + setVisible(true); + } + + + private void startTimer() { + timer = new Timer(); + timer.scheduleAtFixedRate(new TimerTask() { + public void run() { + refreshCounters(); + } + }, 0, 100); + } + + private void stopTimer() { + if(timer != null) { + timer.cancel(); + timer = null; + } + } + + /** set auto-refresh mode. It is either turned on or off, depending on the + * status of the relevant check box. */ + private void setAutoRefresh() { + if(AutoRefresh.getState() == true) { + startTimer(); + } else { + stopTimer(); + } + } + + /** refresh counter display from rsyslogd data. Does a network round-trip. */ + private void refreshCounters() { + try { + String res = diagSess.request("getmainmsgqueuesize"); + MainQItems.setText(res); + } + catch(IOException e) { + System.out.println("Exception during request:\n" + e.toString()); + } + } +} diff --git a/java/com/rsyslog/gui/diaggui/DiagGUI.java b/java/com/rsyslog/gui/diaggui/DiagGUI.java new file mode 100644 index 00000000..1a03299c --- /dev/null +++ b/java/com/rsyslog/gui/diaggui/DiagGUI.java @@ -0,0 +1,77 @@ +/* A yet very simple diagnostic GUI for rsyslog. + * + * Please note that this program requires imdiag to be loaded inside rsyslogd. + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +package com.rsyslog.gui.diaggui; +import java.awt.*; +import java.awt.event.*; + +public class DiagGUI extends Frame { + public Counters counterWin; + public static void main(String args[]) { + new DiagGUI(); + } + + /** show counter window. creates it if not already present */ + public void showCounters() { + if(counterWin == null) { + counterWin = new Counters(); + } else { + counterWin.setVisible(true); + counterWin.toFront(); + } + } + + /** initialize the GUI. */ + public DiagGUI(){ + MenuItem item; + MenuBar menuBar = new MenuBar(); + Menu fileMenu = new Menu("File"); + item = new MenuItem("Exit"); + item.addActionListener(new ActionListener() { + public void actionPerformed(ActionEvent e) { + System.exit(0); + } + }); + fileMenu.add(item); + menuBar.add(fileMenu); + + Menu viewMenu = new Menu("View"); + item = new MenuItem("Counters"); + item.addActionListener(new ActionListener() { + public void actionPerformed(ActionEvent e) { + showCounters(); + } + }); + viewMenu.add(item); + menuBar.add(viewMenu); + + addWindowListener(new WindowAdapter() { + public void windowClosing(WindowEvent e){ + System.exit(0); + } + }); + setMenuBar(menuBar); + setSize(300,400); + setVisible(true); + } +} diff --git a/java/com/rsyslog/gui/msggen/MsgGen.java b/java/com/rsyslog/gui/msggen/MsgGen.java new file mode 100644 index 00000000..c57027ff --- /dev/null +++ b/java/com/rsyslog/gui/msggen/MsgGen.java @@ -0,0 +1,140 @@ +/* A yet very simple syslog message generator + * + * The purpose of this program is to provide a facility that enables + * to generate complex traffic patterns for testing purposes. It still is + * in its infancy, but hopefully will evolve. + * + * Note that this has been created as a stand-alone application because it + * was considered useful to have it as a separate program. But it should still + * be possible to call its class from any other program, specifically the debug + * GUI. + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +package com.rsyslog.gui.msggen; +import com.rsyslog.lib.*; +import java.awt.*; +import java.awt.event.*; +import javax.swing.*; + +public class MsgGen extends Frame { + private TextField target; + private TextField message; + private TextField nummsgs; + private TextField numthrds; + + public static void main(String args[]) { + new MsgGen(); + } + + /** creates the menu bar INCLUDING all menu handlers */ + private void createMenu() { + MenuItem item; + MenuBar menuBar = new MenuBar(); + Menu fileMenu = new Menu("File"); + item = new MenuItem("Exit"); + item.addActionListener(new ActionListener() { + public void actionPerformed(ActionEvent e) { + System.exit(0); + } + }); + fileMenu.add(item); + menuBar.add(fileMenu); + + addWindowListener(new WindowAdapter() { + public void windowClosing(WindowEvent e){ + System.exit(0); + } + }); + setMenuBar(menuBar); + } + + /** creates the main GUI */ + private void createGUI() { + //target = new TextField("127.0.0.1", 32); + target = new TextField("172.19.3.3", 32); + message = new TextField(80); + //message.setText("<161>Test malformed"); + message.setText("<5>iaalog[171652]: AIB|dcu|2009/08/12 14:48:43|mfa challenge|NNNNNNN|XX.XX.XX.XX"); + nummsgs = new TextField("1000", 8); + numthrds = new TextField("10", 5); + Panel pCenter = new Panel(); + + Panel pnl = new Panel(); + pnl.setLayout(new FlowLayout()); + pnl.add(new Label("Target Host:")); + pnl.add(target); + pCenter.add(pnl); + + pnl = new Panel(); + pnl.setLayout(new FlowLayout()); + pnl.add(new Label("Number of Msgs:")); + pnl.add(nummsgs); + pCenter.add(pnl); + + pnl = new Panel(); + pnl.setLayout(new FlowLayout()); + pnl.add(new Label("Msg:")); + pnl.add(message); + pCenter.add(pnl); + + Panel pSouth = new Panel(); + pSouth.setLayout(new FlowLayout()); + + pnl = new Panel(); + pnl.setLayout(new FlowLayout()); + pnl.add(new Label("Number of Threads:")); + pnl.add(numthrds); + pSouth.add(pnl); + + Button b = new Button("Start Test"); + b.addActionListener(new ActionListener() { + public void actionPerformed(ActionEvent e) { + performTest(); + } + }); + pSouth.add(b); + + pack(); + setTitle("Syslog Message Generator"); + setLayout(new BorderLayout()); + add(pCenter, BorderLayout.CENTER); + add(pSouth, BorderLayout.SOUTH); + setSize(800,400); + } + + /** perform the test, a potentially complex operation */ + private void performTest() { + for(short i = 0 ; i < Integer.parseInt(numthrds.getText()) ; ++ i) { + SyslogTrafficGenerator gen = + new SyslogTrafficGenerator(target.getText(), message.getText(), + Long.parseLong(nummsgs.getText())); + gen.start(); + } + } + + + /** initialize the GUI. */ + public MsgGen(){ + createMenu(); + createGUI(); + setVisible(true); + } +} diff --git a/java/com/rsyslog/gui/simpServ/simpServ.java b/java/com/rsyslog/gui/simpServ/simpServ.java new file mode 100644 index 00000000..2a83dad0 --- /dev/null +++ b/java/com/rsyslog/gui/simpServ/simpServ.java @@ -0,0 +1,45 @@ +/** + * Implementation of a tcp-based syslog server. + * + * @author Rainer Gerhards + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see http://www.gnu.org/licenses/. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ + +package com.rsyslog.gui.simpServ; +import com.rsyslog.lib.*; +//import com.rsyslog.gui.*; + +public class simpServ { + + public static void main(String args[]) { + try { + simpServConsumer cons = new simpServConsumer(); + System.out.println("Starting server on port " + args[0] + "\n"); + SyslogServerTCP myServ = new + SyslogServerTCP(Integer.parseInt(args[0]), cons); + myServ.start(); + System.out.println("Press ctl-c to terminate\n"); + } + catch(Exception e) { + System.out.println("Error: " + e.toString()); + } + } +} diff --git a/java/com/rsyslog/gui/simpServ/simpServConsumer.java b/java/com/rsyslog/gui/simpServ/simpServConsumer.java new file mode 100644 index 00000000..588f2640 --- /dev/null +++ b/java/com/rsyslog/gui/simpServ/simpServConsumer.java @@ -0,0 +1,32 @@ +/** A syslog message consumer for the simple syslog server. + * + * @author Rainer Gerhards + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see http://www.gnu.org/licenses/. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +package com.rsyslog.gui.simpServ; +import com.rsyslog.lib.*; + +class simpServConsumer implements SyslogMsgConsumer { + public void consumeMsg(String ln) { + SyslogMessage msg = new SyslogMessage(ln); + System.out.println("Line received '" + msg.getRawMsgAfterPRI() + "'\n"); + } +} diff --git a/java/com/rsyslog/lib/DiagSess.java b/java/com/rsyslog/lib/DiagSess.java new file mode 100644 index 00000000..799b9a4a --- /dev/null +++ b/java/com/rsyslog/lib/DiagSess.java @@ -0,0 +1,78 @@ +/* The diagnostic session to an imdiag module (running inside rsyslogd). + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +package com.rsyslog.lib; +import java.io.*; +import java.net.*; + +public class DiagSess { + + private String host = new String("127.0.0.1"); + private int port = 13500; + int timeout = 0; + private PrintWriter out = null; + private BufferedReader in = null; + private Socket diagSocket = null; + + /** set connection timeout */ + public void setTimeout(int timeout_) { + timeout = timeout_; + } + + public DiagSess(String host_, int port_) { + host = host_; + port = port_; + } + + /** connect to remote server. Initializes everything for request-reply + * processing. + * + * @throws IOException + */ + public void connect() throws IOException { + diagSocket = new Socket(host, port); + diagSocket.setSoTimeout(timeout); + out = new PrintWriter(diagSocket.getOutputStream(), true); + in = new BufferedReader(new InputStreamReader( + diagSocket.getInputStream())); + + } + + /** end session with remote server. */ + public void disconnect() throws IOException { + out.close(); + in.close(); + diagSocket.close(); + } + + /** issue a request to imdiag and return its response. + * + * @param req request string + * @return response string (unparsed) + * @throws IOException + */ + public String request(String req) throws IOException { + out.println(req); + String resp = in.readLine(); + return resp; + } + +} diff --git a/java/com/rsyslog/lib/SyslogMessage.java b/java/com/rsyslog/lib/SyslogMessage.java new file mode 100644 index 00000000..b544a6db --- /dev/null +++ b/java/com/rsyslog/lib/SyslogMessage.java @@ -0,0 +1,75 @@ +/** + * Implementation of the syslog message object. + * + * This is a limited-capability implementation of a syslog message together + * with all its properties. It is limit to what is currently needed and may + * be extended as further need arises. + * + * @author Rainer Gerhards + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see http://www.gnu.org/licenses/. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +package com.rsyslog.lib; + +public class SyslogMessage { + + /** message as received from the wire */ + private String rawmsg; + /** the rawmsg without the PRI part */ + private String rawMsgAfterPRI; + /** PRI part */ + private int pri; + + /** a very simple syslog parser. So far, it only parses out the + * PRI part of the message. May be extended later. Rawmsg must have + * been set before the parser is called. It will populate "all" other + * fields. + */ + private void parse() { + int i; + if(rawmsg.charAt(0) == '<') { + pri = 0; + for(i = 1 ; Character.isDigit(rawmsg.charAt(i)) && i < 4 ; ++i) { + pri = pri * 10 + rawmsg.charAt(i) - '0'; + } + if(rawmsg.charAt(i) != '>') + /* not a real cure, but sufficient for the current + * mini-parser... */ + --i; + rawMsgAfterPRI = rawmsg.substring(i + 1); + } else { + pri = 116; + rawMsgAfterPRI = rawmsg; + } + } + + public SyslogMessage(String _rawmsg) { + rawmsg = _rawmsg; + parse(); + } + + public String getRawMsg() { + return rawmsg; + } + + public String getRawMsgAfterPRI() { + return rawMsgAfterPRI; + } +} diff --git a/java/com/rsyslog/lib/SyslogMsgConsumer.java b/java/com/rsyslog/lib/SyslogMsgConsumer.java new file mode 100644 index 00000000..42c9931a --- /dev/null +++ b/java/com/rsyslog/lib/SyslogMsgConsumer.java @@ -0,0 +1,29 @@ +/** + * Interface for SyslogConsumers. + * + * @author Rainer Gerhards + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see http://www.gnu.org/licenses/. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +package com.rsyslog.lib; + +public interface SyslogMsgConsumer { + public void consumeMsg(String msg); +} diff --git a/java/com/rsyslog/lib/SyslogSender.java b/java/com/rsyslog/lib/SyslogSender.java new file mode 100644 index 00000000..fc0e3fec --- /dev/null +++ b/java/com/rsyslog/lib/SyslogSender.java @@ -0,0 +1,96 @@ +/** + * This class specifies all methods common to syslog senders. It also implements + * some generic ways to send data. Actual syslog senders (e.g. UDP, TCP, ...) shall + * be derived from this class. + * + * This is a limited-capability implementation of a syslog message together + * with all its properties. It is limit to what is currently needed and may + * be extended as further need arises. + * + * @author Rainer Gerhards + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see http://www.gnu.org/licenses/. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +package com.rsyslog.lib; + +public abstract class SyslogSender { + + /** the rawmsg without the PRI part */ + private String target; + + /** the rawmsg without the PRI part */ + private boolean isConnected = false; + + /** Constructs Sender, sets target system. + * @param target the system to connect to. Syntax of target is depending + * on the underlying transport. + */ + public SyslogSender(String target) { + this.target = target; + } + + + /** send a message on the wire. + * This needs a complete formatted message, which will be extended by + * the transport framing, if necessary. + * + * @param MSG a validly formatted syslog message as of the RFC (all parts) + * @throws Exception (depending on transport) + */ + protected abstract void sendTransport(String MSG) throws Exception; + + /** send an alread-formatted message. + * Sends a preformatted syslog message payload to the target. Connects + * to the target if not already connected. + * + * @param MSG a validly formatted syslog message as of the RFC (all parts) + * @throws Exception (depending on transport) + */ + public void sendMSG(String MSG) throws Exception { + if(!isConnected) + connect(); + sendTransport(MSG); + } + + /** connect to the target. + * Note that this may be a null operation if there is no session-like entity + * in the underlying transport (as is for example in UDP). + */ + public void connect() throws Exception { + /* the default implementation does (almost) nothing */ + isConnected = true; + } + + /** disconnects from the target. + * Note that this may be a null operation if there is no session-like entity + * in the underlying transport (as is for example in UDP). + */ + public void disconnect() { + /* the default implementation does (almost) nothing */ + isConnected = false; + } + + /** return target of this Sender. + * @returns target as initially set + */ + public String getTarget() { + return target; + } +} diff --git a/java/com/rsyslog/lib/SyslogServerTCP.java b/java/com/rsyslog/lib/SyslogServerTCP.java new file mode 100644 index 00000000..d5376a32 --- /dev/null +++ b/java/com/rsyslog/lib/SyslogServerTCP.java @@ -0,0 +1,126 @@ +/** + * Implementation of a tcp-based syslog server. + * + * This is a limited-capability implementation of a syslog tcp server. + * + * @author Rainer Gerhards + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see http://www.gnu.org/licenses/. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +package com.rsyslog.lib; + +import com.rsyslog.lib.SyslogMsgConsumer; +import java.io.*; +import java.net.*; + + +/** a small test consumer */ +/* +class TestConsumer implements SyslogMsgConsumer { + public void consumeMsg(String ln) { + System.out.println("Line received '" + ln + "'\n"); + } +} +*/ + +public class SyslogServerTCP extends Thread { + private ServerSocket lstnSock; + private boolean contRun; /* continue processing requests? */ + public SyslogMsgConsumer consumer; + + /** Process a single connection */ + class Session extends Thread { + private Socket sock; + private SyslogServerTCP srvr; + + public Session(Socket so, SyslogServerTCP _srvr) { + sock = so; + srvr = _srvr; + } + + public void run() { + try { + BufferedReader data = new BufferedReader( + new InputStreamReader(sock.getInputStream())); + + String ln = data.readLine(); + while(ln != null) { + srvr.getConsumer().consumeMsg(ln); + ln = data.readLine(); + } + System.out.println("End of Session.\n"); + sock.close(); + } + catch(Exception e) { + /* we ignore any errors we may have... */ + System.out.println("Session exception " + e.toString()); + } + } + } + + /** a small test driver */ +/* + public static void main(String args[]) { + try { + SyslogMsgConsumer cons = new TestConsumer(); + System.out.println("Starting server on port " + args[0] + "\n"); + SyslogServerTCP myServ = new + SyslogServerTCP(Integer.parseInt(args[0]), cons); + myServ.start(); + System.out.println("Press ctl-c to terminate\n"); + } + catch(Exception e) { + System.out.println("Fehler! " + e.toString()); + } + } +*/ + + public SyslogServerTCP(int port, SyslogMsgConsumer cons) throws java.io.IOException { + if(lstnSock != null) + terminate(); + lstnSock = new ServerSocket(port); + consumer = cons; + contRun = true; + } + + public void terminate() { + contRun = false; + } + + public SyslogMsgConsumer getConsumer() { + return consumer; + } + + public void run() { + try { + while(contRun) { + Socket sock = lstnSock.accept(); + System.out.println("New connection request! " + sock.toString()); + Thread sess = new Session(sock, this); + sock = null; + sess.start(); + } + } + catch(Exception e) { + System.out.println("Error during server run " + e.toString()); + } + + } +} diff --git a/java/com/rsyslog/lib/SyslogTrafficGenerator.java b/java/com/rsyslog/lib/SyslogTrafficGenerator.java new file mode 100644 index 00000000..79a99495 --- /dev/null +++ b/java/com/rsyslog/lib/SyslogTrafficGenerator.java @@ -0,0 +1,81 @@ +/** + * This class is a syslog traffic generator. It is primarily intended to be used + * together with testing tools, but may have some use cases outside that domain. + * + * @author Rainer Gerhards + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see http://www.gnu.org/licenses/. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +package com.rsyslog.lib; + +public class SyslogTrafficGenerator extends Thread { + + /** the target host to receive traffic */ + private String target; + + /** the message (template) to be sent */ + private String message; + + /** number of messages to be sent */ + private long nummsgs; + + /** Constructs Sender, sets target system. + * @param target the system to connect to. Syntax of target is depending + * on the underlying transport. + */ + public SyslogTrafficGenerator(String target, String message, long nummsgs) { + this.target = target; + this.message = message; + this.nummsgs = nummsgs; + } + + /** Generates the traffic. Stops when either called to terminate + * or the max number of messages have been sent. Note that all + * necessary properties must have been set up before starting the + * generator thread! + */ + private void performTest() throws Exception { + int doDisp = 0; + UDPSyslogSender sender = new UDPSyslogSender(target); + for(long i = 0 ; i < nummsgs ; ++i) { + sender.sendMSG(message + " " + Long.toString(i) + " " + this.toString() + "\0"); + if((doDisp++ % 1000) == 0) + System.out.println(this.toString() + " send message " + Long.toString(i)); + sleep(1); + } + } + + +/** Wrapper around the real traffic generator, catches exceptions. + */ + public void run() { + System.out.println("traffic generator " + this.toString() + " thread started"); + try { + performTest(); + } + catch(Exception e) { + /* at some time, we may find a more intelligent way to + * handle this! ;) + */ + System.out.println(e.toString()); + } + System.out.println("traffic generator " + this.toString() + " thread finished"); + } +} diff --git a/java/com/rsyslog/lib/UDPSyslogSender.java b/java/com/rsyslog/lib/UDPSyslogSender.java new file mode 100644 index 00000000..1a2c4726 --- /dev/null +++ b/java/com/rsyslog/lib/UDPSyslogSender.java @@ -0,0 +1,75 @@ +/** + * A UDP transport implementation of a syslog sender. + * + * Note that there is an anomaly in this version of the code: we query the remote system + * address only once during the connection setup and resue it. If we potentially run for + * an extended period of time, the remote address may change, what we do not reflect. For + * the current use case, this is acceptable, but if this code is put into more wide-spread + * use outside of debugging, a periodic requery should be added. + * + * @author Rainer Gerhards + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see http://www.gnu.org/licenses/. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +package com.rsyslog.lib; +import java.net.*; + +public class UDPSyslogSender extends SyslogSender { + + private final int port = 514; // TODO: take from target! + private InetAddress targetAddr; + + /** the socket to communicate over with the remote system. */ + private DatagramSocket sock; + + /** Constructs Sender, sets target system. + * @param target the system to connect to. Syntax of target is depending + * on the underlying transport. + */ + public UDPSyslogSender(String target) throws Exception { + super(target); + } + + /** send a message on the wire. + * This needs a complete formatted message, which will be extended by + * the transport framing, if necessary. + * + * @param MSG a validly formatted syslog message as of the RFC (all parts) + * @throws Exception (depending on transport) + */ + protected void sendTransport(String MSG) throws Exception { + byte msg[] = MSG.getBytes(); + DatagramPacket pkt = new DatagramPacket(msg, msg.length, targetAddr, port); + sock.send(pkt); + } + + + /** connect to the target. + * For UDP, this means we create the socket. + */ + public void connect() throws Exception { + super.connect(); + sock = new DatagramSocket(); + + // TODO: we should extract the actual hostname & port! + targetAddr = InetAddress.getByName(getTarget()); + } + +} diff --git a/m4/shave.m4 b/m4/shave.m4 deleted file mode 100644 index e647e579..00000000 --- a/m4/shave.m4 +++ /dev/null @@ -1,99 +0,0 @@ -dnl Make automake/libtool output more friendly to humans -dnl -dnl Copyright (c) 2009, Damien Lespiau <damien.lespiau@gmail.com> -dnl -dnl Permission is hereby granted, free of charge, to any person -dnl obtaining a copy of this software and associated documentation -dnl files (the "Software"), to deal in the Software without -dnl restriction, including without limitation the rights to use, -dnl copy, modify, merge, publish, distribute, sublicense, and/or sell -dnl copies of the Software, and to permit persons to whom the -dnl Software is furnished to do so, subject to the following -dnl conditions: -dnl -dnl The above copyright notice and this permission notice shall be -dnl included in all copies or substantial portions of the Software. -dnl -dnl THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -dnl EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -dnl OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -dnl NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -dnl HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -dnl WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -dnl FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -dnl OTHER DEALINGS IN THE SOFTWARE. -dnl -dnl SHAVE_INIT([shavedir],[default_mode]) -dnl -dnl shavedir: the directory where the shave scripts are, it defaults to -dnl $(top_builddir) -dnl default_mode: (enable|disable) default shave mode. This parameter -dnl controls shave's behaviour when no option has been -dnl given to configure. It defaults to disable. -dnl -dnl * SHAVE_INIT should be called late in your configure.(ac|in) file (just -dnl before AC_CONFIG_FILE/AC_OUTPUT is perfect. This macro rewrites CC and -dnl LIBTOOL, you don't want the configure tests to have these variables -dnl re-defined. -dnl * This macro requires GNU make's -s option. - -AC_DEFUN([_SHAVE_ARG_ENABLE], -[ - AC_ARG_ENABLE([shave], - AS_HELP_STRING( - [--enable-shave], - [use shave to make the build pretty [[default=$1]]]),, - [enable_shave=$1] - ) -]) - -AC_DEFUN([SHAVE_INIT], -[ - dnl you can tweak the default value of enable_shave - m4_if([$2], [enable], [_SHAVE_ARG_ENABLE(yes)], [_SHAVE_ARG_ENABLE(no)]) - - if test x"$enable_shave" = xyes; then - dnl where can we find the shave scripts? - m4_if([$1],, - [shavedir="$ac_pwd"], - [shavedir="$ac_pwd/$1"]) - AC_SUBST(shavedir) - - dnl make is now quiet - AC_SUBST([MAKEFLAGS], [-s]) - AC_SUBST([AM_MAKEFLAGS], ['`test -z $V && echo -s`']) - - dnl we need sed - AC_CHECK_PROG(SED,sed,sed,false) - - dnl substitute libtool - SHAVE_SAVED_LIBTOOL=$LIBTOOL - LIBTOOL="${SHELL} ${shavedir}/shave-libtool '${SHAVE_SAVED_LIBTOOL}'" - AC_SUBST(LIBTOOL) - - dnl substitute cc/cxx - SHAVE_SAVED_CC=$CC - SHAVE_SAVED_CXX=$CXX - SHAVE_SAVED_FC=$FC - SHAVE_SAVED_F77=$F77 - SHAVE_SAVED_OBJC=$OBJC - CC="${SHELL} ${shavedir}/shave cc ${SHAVE_SAVED_CC}" - CXX="${SHELL} ${shavedir}/shave cxx ${SHAVE_SAVED_CXX}" - FC="${SHELL} ${shavedir}/shave fc ${SHAVE_SAVED_FC}" - F77="${SHELL} ${shavedir}/shave f77 ${SHAVE_SAVED_F77}" - OBJC="${SHELL} ${shavedir}/shave objc ${SHAVE_SAVED_OBJC}" - AC_SUBST(CC) - AC_SUBST(CXX) - AC_SUBST(FC) - AC_SUBST(F77) - AC_SUBST(OBJC) - - V=@ - else - V=1 - fi - Q='$(V:1=)' - AC_SUBST(V) - AC_SUBST(Q) -]) - diff --git a/outchannel.c b/outchannel.c index 74c18218..b9b4a6b1 100644 --- a/outchannel.c +++ b/outchannel.c @@ -38,6 +38,7 @@ #include "stringbuf.h" #include "outchannel.h" #include "dirty.h" +#include "debug.h" static struct outchannel *ochRoot = NULL; /* the root of the outchannel list */ static struct outchannel *ochLast = NULL; /* points to the last element of the outchannel list */ @@ -212,7 +213,7 @@ struct outchannel *ochAddLine(char* pName, uchar** ppRestOfConfLine) return NULL; pOch->iLenName = strlen(pName); - pOch->pszName = (char*) malloc(sizeof(char) * (pOch->iLenName + 1)); + pOch->pszName = (char*) MALLOC(sizeof(char) * (pOch->iLenName + 1)); if(pOch->pszName == NULL) { dbgprintf("ochAddLine could not alloc memory for outchannel name!"); pOch->iLenName = 0; @@ -37,6 +37,7 @@ #include "rsyslog.h" #include "net.h" /* struct NetAddr */ #include "parse.h" +#include "debug.h" /* ################################################################# * * private members * @@ -429,7 +430,7 @@ rsRetVal parsAddrWithBits(rsParsObj *pThis, struct NetAddr **pIP, int *pBits) switch(getaddrinfo ((char*)pszIP+1, NULL, &hints, &res)) { case 0: - (*pIP)->addr.NetAddr = malloc (res->ai_addrlen); + (*pIP)->addr.NetAddr = MALLOC (res->ai_addrlen); memcpy ((*pIP)->addr.NetAddr, res->ai_addr, res->ai_addrlen); freeaddrinfo (res); break; @@ -468,7 +469,7 @@ rsRetVal parsAddrWithBits(rsParsObj *pThis, struct NetAddr **pIP, int *pBits) switch(getaddrinfo ((char*)pszIP, NULL, &hints, &res)) { case 0: - (*pIP)->addr.NetAddr = malloc (res->ai_addrlen); + (*pIP)->addr.NetAddr = MALLOC (res->ai_addrlen); memcpy ((*pIP)->addr.NetAddr, res->ai_addr, res->ai_addrlen); freeaddrinfo (res); break; diff --git a/plugins/imdiag/imdiag.c b/plugins/imdiag/imdiag.c index 9602f50d..2f7e5fee 100644 --- a/plugins/imdiag/imdiag.c +++ b/plugins/imdiag/imdiag.c @@ -213,7 +213,6 @@ doInjectMsg(int iNum) MsgSetInputName(pMsg, pInputName); MsgSetFlowControlType(pMsg, eFLOWCTL_NO_DELAY); pMsg->msgFlags = NEEDS_PARSING | PARSE_HOSTNAME; - pMsg->bParseHOSTNAME = 1; MsgSetRcvFrom(pMsg, pRcvDummy); CHKiRet(MsgSetRcvFromIP(pMsg, pRcvIPDummy)); CHKiRet(submitMsg(pMsg)); @@ -264,6 +263,9 @@ waitMainQEmpty(tcps_sess_t *pSess) CHKiRet(diagGetMainMsgQSize(&iMsgQueueSize)); while(iMsgQueueSize > 0) { + /* DEV DEBUG ONLY if(iPrint++ % 500) + printf("imdiag: main msg queue size: %d\n", iMsgQueueSize); + */ if(iPrint++ % 500 == 0) dbgprintf("imdiag sleeping, wait mainq drain, curr size %d\n", iMsgQueueSize); srSleep(0,2); /* wait a little bit */ @@ -294,7 +296,7 @@ OnMsgReceived(tcps_sess_t *pSess, uchar *pRcv, int iLenMsg) * WITHOUT a termination \0 char. So we need to convert it to one * before proceeding. */ - CHKmalloc(pszMsg = malloc(sizeof(uchar) * (iLenMsg + 1))); + CHKmalloc(pszMsg = MALLOC(sizeof(uchar) * (iLenMsg + 1))); memcpy(pszMsg, pRcv, iLenMsg); pszMsg[iLenMsg] = '\0'; diff --git a/plugins/imfile/imfile.c b/plugins/imfile/imfile.c index 7c588f90..8a10e26f 100644 --- a/plugins/imfile/imfile.c +++ b/plugins/imfile/imfile.c @@ -5,7 +5,7 @@ * * Work originally begun on 2008-02-01 by Rainer Gerhards * - * Copyright 2008 Rainer Gerhards and Adiscon GmbH. + * Copyright 2008,2009 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -107,7 +107,6 @@ static rsRetVal enqLine(fileInfo_t *pInfo, cstr_t *cstrLine) MsgSetTAG(pMsg, pInfo->pszTag, pInfo->lenTag); pMsg->iFacility = LOG_FAC(pInfo->iFacility); pMsg->iSeverity = LOG_PRI(pInfo->iSeverity); - pMsg->bParseHOSTNAME = 0; CHKiRet(submitMsg(pMsg)); finalize_it: RETiRet; @@ -214,7 +213,7 @@ static rsRetVal pollFile(fileInfo_t *pThis, int *pbHadFileData) } finalize_it: - /*EMPTY - just to keep the compiler happy, do NOT remove*/; + ; /*EMPTY STATEMENT - needed to keep compiler happy - see below! */ /* Note: the problem above is that pthread:cleanup_pop() is a macro which * evaluates to something like "} while(0);". So the code would become * "finalize_it: }", that is a label without a statement. The C standard does @@ -244,27 +243,12 @@ finalize_it: * IMPORTANT: the calling interface of this function can NOT be modified. It actually is * called by pthreads. The provided argument is currently not being used. */ -/* ------------------------------------------------------------------------------------------ * - * DO NOT TOUCH the following code - it will soon be part of the module generation macros! */ static void inputModuleCleanup(void __attribute__((unused)) *arg) { BEGINfunc -/* END no-touch zone * - * ------------------------------------------------------------------------------------------ */ - - - - /* so far not needed */ - - - -/* ------------------------------------------------------------------------------------------ * - * DO NOT TOUCH the following code - it will soon be part of the module generation macros! */ ENDfunc } -/* END no-touch zone * - * ------------------------------------------------------------------------------------------ */ /* This function is called by the framework to gather the input. The module stays @@ -292,28 +276,22 @@ BEGINrunInput int i; int bHadFileData; /* were there at least one file with data during this run? */ CODESTARTrunInput - /* ------------------------------------------------------------------------------------------ * - * DO NOT TOUCH the following code - it will soon be part of the module generation macros! */ pthread_cleanup_push(inputModuleCleanup, NULL); - while(1) { /* endless loop - do NOT break; out of it! */ - /* END no-touch zone * - * ------------------------------------------------------------------------------------------ */ + while(1) { - do { - bHadFileData = 0; - for(i = 0 ; i < iFilPtr ; ++i) { - pollFile(&files[i], &bHadFileData); - } - } while(iFilPtr > 1 && bHadFileData == 1); /* waring: do...while()! */ + do { + bHadFileData = 0; + for(i = 0 ; i < iFilPtr ; ++i) { + pollFile(&files[i], &bHadFileData); + } + } while(iFilPtr > 1 && bHadFileData == 1); /* warning: do...while()! */ - /* Note: the additional 10ns wait is vitally important. It guards rsyslog against totally - * hogging the CPU if the users selects a polling interval of 0 seconds. It doesn't hurt any - * other valid scenario. So do not remove. -- rgerhards, 2008-02-14 - */ - srSleep(iPollInterval, 10); + /* Note: the additional 10ns wait is vitally important. It guards rsyslog against totally + * hogging the CPU if the users selects a polling interval of 0 seconds. It doesn't hurt any + * other valid scenario. So do not remove. -- rgerhards, 2008-02-14 + */ + srSleep(iPollInterval, 10); - /* ------------------------------------------------------------------------------------------ * - * DO NOT TOUCH the following code - it will soon be part of the module generation macros! */ } /*NOTREACHED*/ diff --git a/plugins/imgssapi/imgssapi.c b/plugins/imgssapi/imgssapi.c index f9c1f86b..14830ce2 100644 --- a/plugins/imgssapi/imgssapi.c +++ b/plugins/imgssapi/imgssapi.c @@ -56,6 +56,7 @@ #include "errmsg.h" #include "netstrm.h" #include "glbl.h" +#include "debug.h" MODULE_TYPE_INPUT @@ -407,7 +408,7 @@ OnSessAcceptGSS(tcpsrv_t *pThis, tcps_sess_t *pSess) */ char *buf; int ret = 0; - CHKmalloc(buf = (char*) malloc(sizeof(char) * (glbl.GetMaxLine() + 1))); + CHKmalloc(buf = (char*) MALLOC(sizeof(char) * (glbl.GetMaxLine() + 1))); dbgprintf("GSS-API Trying to accept TCP session %p\n", pSess); diff --git a/plugins/imklog/bsd.c b/plugins/imklog/bsd.c index 6d7b6c98..b7899353 100644 --- a/plugins/imklog/bsd.c +++ b/plugins/imklog/bsd.c @@ -75,6 +75,7 @@ #include "rsyslog.h" #include "imklog.h" +#include "debug.h" /* globals */ static int fklog = -1; /* /dev/klog */ @@ -132,7 +133,7 @@ readklog(void) if((size_t) iMaxLine < sizeof(bufRcv) - 1) { pRcv = bufRcv; } else { - if((pRcv = (uchar*) malloc(sizeof(uchar) * (iMaxLine + 1))) == NULL) + if((pRcv = (uchar*) MALLOC(sizeof(uchar) * (iMaxLine + 1))) == NULL) iMaxLine = sizeof(bufRcv) - 1; /* better this than noting */ } diff --git a/plugins/imklog/imklog.c b/plugins/imklog/imklog.c index 7994c5eb..c59ce04f 100644 --- a/plugins/imklog/imklog.c +++ b/plugins/imklog/imklog.c @@ -111,7 +111,6 @@ enqMsg(uchar *msg, uchar* pszTag, int iFacility, int iSeverity) MsgSetTAG(pMsg, pszTag, ustrlen(pszTag)); pMsg->iFacility = LOG_FAC(iFacility); pMsg->iSeverity = LOG_PRI(iSeverity); - pMsg->bParseHOSTNAME = 0; CHKiRet(submitMsg(pMsg)); finalize_it: diff --git a/plugins/imklog/ksym.c b/plugins/imklog/ksym.c index f636a7bb..058b2cfa 100644 --- a/plugins/imklog/ksym.c +++ b/plugins/imklog/ksym.c @@ -122,6 +122,7 @@ #include "imklog.h" #include "ksyms.h" #include "module.h" +#include "debug.h" int num_syms = 0; @@ -523,7 +524,7 @@ static int AddSymbol(unsigned long address, char *symbol) return(0); /* Then the space for the symbol. */ - sym_array[num_syms].name = (char *) malloc(strlen(symbol)*sizeof(char) + 1); + sym_array[num_syms].name = (char *) MALLOC(strlen(symbol)*sizeof(char) + 1); if ( sym_array[num_syms].name == NULL ) return(0); diff --git a/plugins/imklog/ksym_mod.c b/plugins/imklog/ksym_mod.c index be5fdee9..82978892 100644 --- a/plugins/imklog/ksym_mod.c +++ b/plugins/imklog/ksym_mod.c @@ -106,6 +106,7 @@ #include "rsyslog.h" #include "imklog.h" #include "ksyms.h" +#include "debug.h" #define KSYMS "/proc/kallsyms" @@ -289,7 +290,7 @@ struct Module *AddModule(module) struct Module *mp; if ( num_modules == 0 ) { - sym_array_modules = (struct Module *)malloc(sizeof(struct Module)); + sym_array_modules = (struct Module *)MALLOC(sizeof(struct Module)); if ( sym_array_modules == NULL ) { diff --git a/plugins/immark/immark.c b/plugins/immark/immark.c index 8504f872..5d48369e 100644 --- a/plugins/immark/immark.c +++ b/plugins/immark/immark.c @@ -42,6 +42,8 @@ #include "module-template.h" #include "errmsg.h" #include "msg.h" +#include "srUtils.h" +#include "glbl.h" MODULE_TYPE_INPUT @@ -50,8 +52,16 @@ MODULE_TYPE_INPUT /* Module static data */ DEF_IMOD_STATIC_DATA +DEFobjCurrIf(glbl) static int iMarkMessagePeriod = DEFAULT_MARK_PERIOD; +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature + if(eFeat == sFEATURENonCancelInputTermination) + iRet = RS_RET_OK; +ENDisCompatibleWithFeature + + /* This function is called to gather input. It must terminate only * a) on failure (iRet set accordingly) * b) on termination of the input module (as part of the unload process) @@ -71,16 +81,13 @@ CODESTARTrunInput * right into the sleep below. */ while(1) { - /* we do not need to handle the RS_RET_TERMINATE_NOW case any - * special because we just need to terminate. This may be different - * if a cleanup is needed. But for now, we can just use CHKiRet(). - * rgerhards, 2007-12-17 - */ - CHKiRet(thrdSleep(pThrd, iMarkMessagePeriod, 0)); /* seconds, micro seconds */ + srSleep(iMarkMessagePeriod, 0); /* seconds, micro seconds */ + + if(glbl.GetGlobalInputTermState() == 1) + break; /* terminate input! */ + logmsgInternal(NO_ERRCODE, LOG_INFO, (uchar*)"-- MARK --", MARK); } -finalize_it: - return iRet; ENDrunInput @@ -106,6 +113,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_IMOD_QUERIES +CODEqueryEtryPt_IsCompatibleWithFeature_IF_OMOD_QUERIES ENDqueryEtryPt static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __attribute__((unused)) *pVal) @@ -119,9 +127,9 @@ BEGINmodInit() CODESTARTmodInit *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr + CHKiRet(objUse(glbl, CORE_COMPONENT)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"markmessageperiod", 0, eCmdHdlrInt, NULL, &iMarkMessagePeriod, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID)); ENDmodInit -/* - * vi:set ai: +/* vi:set ai: */ diff --git a/plugins/imtcp/imtcp.c b/plugins/imtcp/imtcp.c index c56593f2..176b5b18 100644 --- a/plugins/imtcp/imtcp.c +++ b/plugins/imtcp/imtcp.c @@ -171,7 +171,7 @@ static rsRetVal setRuleset(void __attribute__((unused)) *pVal, uchar *pszName) localRet = ruleset.GetRuleset(&pRuleset, pszName); if(localRet == RS_RET_NOT_FOUND) { - errmsg.LogError(0, NO_ERRCODE, "error: ruleset '%s' not found - ignored", pszName); + errmsg.LogError(0, RS_RET_RULESET_NOT_FOUND, "error: ruleset '%s' not found - ignored", pszName); } CHKiRet(localRet); pBindRuleset = pRuleset; diff --git a/plugins/imtemplate/imtemplate.c b/plugins/imtemplate/imtemplate.c index 366408a0..e5e43025 100644 --- a/plugins/imtemplate/imtemplate.c +++ b/plugins/imtemplate/imtemplate.c @@ -77,6 +77,7 @@ #include "cfsysline.h" /* access to config file objects */ #include "module-template.h" /* generic module interface code - very important, read it! */ #include "srUtils.h" /* some utility functions */ +#include "debug.h" /* some debug helper functions */ MODULE_TYPE_INPUT /* must be present for input modules, do not remove */ @@ -137,7 +138,7 @@ imtemplateMyFunc(int iMyParam) * ABORT_FINALIZE(retcode) * just like FINALIZE, except that iRet is set to the provided error * code before control is transferred, e.g. - * if((ptr = malloc(20)) == NULL) + * if((ptr = MALLOC(20)) == NULL) * ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); * * In order for all this to work, you need to define finalize_it, e.g. @@ -231,49 +232,25 @@ CODESTARTrunInput * logs an error message as syslogd, just as printf, e.g. * errmsg.LogError(NO_ERRCODE, "Error %d occured during %s", 1, "test"); * - * There are several ways how a message can be enqueued. This part of the - * interface is currently underspecified. Have a look at the function definitions - * in syslogd.c (sorry, folks...). - * - * If you received a full syslog message that must be decoded by a message - * parser, parseAndSubmitMessage() is the way to go. It's not just a funny name - * but also a quite some legacy. Consequently, its interface is, ummm, not - * well designed. - * parseAndSubmitMessage((char*)fromHost, (char*) pRcvBuf, lenRcvd, bParseHost); - * fromHost - * is the host that we received the message from (a string) - * pRcvBuf - * is the received (to-be-decoded) message - * lenRcvd - * is the length of the received message. Please note that pRcvBuf is - * NOT a standard C-string. Most importantly it is NOT expected to be - * \0-terminated. Thus the lenght is vitally imporant (if it is wrong, - * rsyslogd will probably segfault). - * bParseHost - * is a boolean (0-no, 1-yes). It tells the parser whether or not - * a hostname should be parsed from the message. This is important - * for sources that are known not to provide a hostname. - * Use define MSG_PARSE_HOSTNAME and MSG_DONT_PARSE_HOSTNAME - * - * Another, more elaborate, way is to create the message object ourselves and - * pass it to the rule engine. That way is more appropriate if the message + * To submit the message to the queue engine, we must create the message + * object and fill it with data. If it contains a syslog message that must + * be parsed, we can add a flag that requests parsing. Otherwise, we must + * fill the properties ourselves. That is appropriate if the message * does not need to be parsed, for example when reading text (log) files. In that way, * we can set the message properties as of our liking. This is how it works: * msg_t *pMsg; CHKiRet(msgConstruct(&pMsg)); - MsgSetUxTradMsg(pMsg, msg); MsgSetRawMsg(pMsg, msg); MsgSetHOSTNAME(pMsg, LocalHostName); MsgSetTAG(pMsg, "rsyslogd:"); pMsg->iFacility = LOG_FAC(pri); pMsg->iSeverity = LOG_PRI(pri); - pMsg->bParseHOSTNAME = 0; flags |= INTERNAL_MSG; logmsg(pMsg, flags); / * some time, CHKiRet() will work here, too [today NOT!] * / * - * Note that UxTradMsg is a wild construct. For the time being, set it to - * the raw message text. I am hard thinking at dropping that beast at all... + * NOTE: for up-to-date usage samples, see the other provided input modules. + * A good starting point is probably imuxsock. * * This example probably does not set all message properties (but the ones * that are of practical importance). If you need all, check msg.h. Use @@ -314,7 +291,7 @@ CODESTARTwillRun if(udpLstnSocks == NULL) ABORT_FINALIZE(RS_RET_NO_RUN); - if((pRcvBuf = malloc(glbl.GetMaxLine * sizeof(char))) == NULL) { + if((pRcvBuf = MALLOC(glbl.GetMaxLine * sizeof(char))) == NULL) { ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } * diff --git a/plugins/imudp/imudp.c b/plugins/imudp/imudp.c index a393cf96..307b684f 100644 --- a/plugins/imudp/imudp.c +++ b/plugins/imudp/imudp.c @@ -32,6 +32,9 @@ #include <errno.h> #include <unistd.h> #include <netdb.h> +#if HAVE_SYS_EPOLL_H +# include <sys/epoll.h> +#endif #include "rsyslog.h" #include "dirty.h" #include "net.h" @@ -44,6 +47,7 @@ #include "parser.h" #include "datetime.h" #include "prop.h" +#include "ruleset.h" #include "unicode-helper.h" MODULE_TYPE_INPUT @@ -57,6 +61,7 @@ DEFobjCurrIf(glbl) DEFobjCurrIf(net) DEFobjCurrIf(datetime) DEFobjCurrIf(prop) +DEFobjCurrIf(ruleset) static int iMaxLine; /* maximum UDP message size supported */ static time_t ttLastDiscard = 0; /* timestamp when a message from a non-permitted sender was last discarded @@ -65,13 +70,14 @@ static time_t ttLastDiscard = 0; /* timestamp when a message from a non-permitte */ static int *udpLstnSocks = NULL; /* Internet datagram sockets, first element is nbr of elements * read-only after init(), but beware of restart! */ +static ruleset_t **udpRulesets = NULL; /* ruleset to be used with sockets in question (entry 0 is empty) */ static uchar *pszBindAddr = NULL; /* IP to bind socket to */ static uchar *pRcvBuf = NULL; /* receive buffer (for a single packet). We use a global and alloc * it so that we can check available memory in willRun() and request * termination if we can not get it. -- rgerhards, 2007-12-27 */ static prop_t *pInputName = NULL; /* our inputName currently is always "imudp", and this will hold it */ -// TODO: static ruleset_t *pBindRuleset = NULL; /* ruleset to bind listener to (use system default if unspecified) */ +static ruleset_t *pBindRuleset = NULL; /* ruleset to bind listener to (use system default if unspecified) */ #define TIME_REQUERY_DFLT 2 static int iTimeRequery = TIME_REQUERY_DFLT;/* how often is time to be queried inside tight recv loop? 0=always */ @@ -90,6 +96,7 @@ static rsRetVal addListner(void __attribute__((unused)) *pVal, uchar *pNewVal) int *newSocks; int *tmpSocks; int iSrc, iDst; + ruleset_t **tmpRulesets; /* check which address to bind to. We could do this more compact, but have not * done so in order to make the code more readable. -- rgerhards, 2007-12-27 @@ -110,26 +117,40 @@ static rsRetVal addListner(void __attribute__((unused)) *pVal, uchar *pNewVal) if(udpLstnSocks == NULL) { /* esay, we can just replace it */ udpLstnSocks = newSocks; +RUNLOG_VAR("%d", newSocks[0]); + CHKmalloc(udpRulesets = (ruleset_t**) MALLOC(sizeof(ruleset_t*) * (newSocks[0] + 1))); + for(iDst = 1 ; iDst <= newSocks[0] ; ++iDst) + udpRulesets[iDst] = pBindRuleset; } else { /* we need to add them */ - if((tmpSocks = malloc(sizeof(int) * (1 + newSocks[0] + udpLstnSocks[0]))) == NULL) { - dbgprintf("out of memory trying to allocate udp listen socket array\n"); + tmpSocks = (int*) MALLOC(sizeof(int) * (1 + newSocks[0] + udpLstnSocks[0])); + tmpRulesets = (ruleset_t**) MALLOC(sizeof(ruleset_t*) * (1 + newSocks[0] + udpLstnSocks[0])); + if(tmpSocks == NULL || tmpRulesets == NULL) { + DBGPRINTF("out of memory trying to allocate udp listen socket array\n"); /* in this case, we discard the new sockets but continue with what we * already have */ free(newSocks); + free(tmpSocks); + free(tmpRulesets); ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } else { /* ready to copy */ iDst = 1; - for(iSrc = 1 ; iSrc <= udpLstnSocks[0] ; ++iSrc) - tmpSocks[iDst++] = udpLstnSocks[iSrc]; - for(iSrc = 1 ; iSrc <= newSocks[0] ; ++iSrc) - tmpSocks[iDst++] = newSocks[iSrc]; + for(iSrc = 1 ; iSrc <= udpLstnSocks[0] ; ++iSrc, ++iDst) { + tmpSocks[iDst] = udpLstnSocks[iSrc]; + tmpRulesets[iDst] = udpRulesets[iSrc]; + } + for(iSrc = 1 ; iSrc <= newSocks[0] ; ++iSrc, ++iDst) { + tmpSocks[iDst] = newSocks[iSrc]; + tmpRulesets[iDst] = pBindRuleset; + } tmpSocks[0] = udpLstnSocks[0] + newSocks[0]; free(newSocks); free(udpLstnSocks); udpLstnSocks = tmpSocks; + free(udpRulesets); + udpRulesets = tmpRulesets; } } } @@ -141,7 +162,6 @@ finalize_it: } -#if 0 /* TODO: implement when tehre is time, requires restructure of socket array! */ /* accept a new ruleset to bind. Checks if it exists and complains, if not */ static rsRetVal setRuleset(void __attribute__((unused)) *pVal, uchar *pszName) @@ -162,7 +182,6 @@ finalize_it: free(pszName); /* no longer needed */ RETiRet; } -#endif /* This function is a helper to runInput. I have extracted it @@ -180,8 +199,8 @@ finalize_it: * on scheduling order. -- rgerhards, 2008-10-02 */ static inline rsRetVal -processSocket(int fd, struct sockaddr_storage *frominetPrev, int *pbIsPermitted, - uchar *fromHost, uchar *fromHostFQDN, uchar *fromHostIP) +processSocket(thrdInfo_t *pThrd, int fd, struct sockaddr_storage *frominetPrev, int *pbIsPermitted, + uchar *fromHost, uchar *fromHostFQDN, uchar *fromHostIP, ruleset_t *pRuleset) { DEFiRet; int iNbrTimeUsed; @@ -195,8 +214,11 @@ processSocket(int fd, struct sockaddr_storage *frominetPrev, int *pbIsPermitted, prop_t *propFromHostIP = NULL; char errStr[1024]; + assert(pThrd != NULL); iNbrTimeUsed = 0; while(1) { /* loop is terminated if we have a bad receive, done below in the body */ + if(pThrd->bShallStop == TRUE) + ABORT_FINALIZE(RS_RET_FORCE_TERM); socklen = sizeof(struct sockaddr_storage); lenRcvBuf = recvfrom(fd, (char*) pRcvBuf, iMaxLine, 0, (struct sockaddr *)&frominet, &socklen); if(lenRcvBuf < 0) { @@ -205,7 +227,7 @@ processSocket(int fd, struct sockaddr_storage *frominetPrev, int *pbIsPermitted, DBGPRINTF("INET socket error: %d = %s.\n", errno, errStr); errmsg.LogError(errno, NO_ERRCODE, "recvfrom inet"); } - ABORT_FINALIZE(RS_RET_ERR); + ABORT_FINALIZE(RS_RET_ERR); // this most often is NOT an error, state is not checked by caller! } if(lenRcvBuf == 0) @@ -230,7 +252,7 @@ processSocket(int fd, struct sockaddr_storage *frominetPrev, int *pbIsPermitted, if(glbl.GetOption_DisallowWarning) { time_t tt; - time(&tt); + datetime.GetTime(&tt); if(tt > ttLastDiscard + 60) { ttLastDiscard = tt; errmsg.LogError(0, NO_ERRCODE, @@ -251,12 +273,11 @@ processSocket(int fd, struct sockaddr_storage *frominetPrev, int *pbIsPermitted, CHKiRet(msgConstructWithTime(&pMsg, &stTime, ttGenTime)); MsgSetRawMsg(pMsg, (char*)pRcvBuf, lenRcvBuf); MsgSetInputName(pMsg, pInputName); + MsgSetRuleset(pMsg, pRuleset); MsgSetFlowControlType(pMsg, eFLOWCTL_NO_DELAY); pMsg->msgFlags = NEEDS_PARSING | PARSE_HOSTNAME; - pMsg->bParseHOSTNAME = 1; MsgSetRcvFromStr(pMsg, fromHost, ustrlen(fromHost), &propFromHost); CHKiRet(MsgSetRcvFromIPStr(pMsg, fromHostIP, ustrlen(fromHostIP), &propFromHostIP)); -dbgprintf("XXX: submitting msg to queue\n"); CHKiRet(submitMsg(pMsg)); } } @@ -271,20 +292,88 @@ finalize_it: } -/* This function is called to gather input. - * Note that udpLstnSocks must be non-NULL because otherwise we would not have - * indicated that we want to run (or we have a programming error ;)). -- rgerhards, 2008-10-02 - * rgerhards, 2008-10-07: I have implemented a very simple, yet in most cases probably - * highly efficient "name caching". Before querying a name, I now check if the name to be - * queried is the same as the one queried in the last message processed. If that is the - * case, we can simple re-use the previous value. This algorithm works quite well with - * few sender, especially if they emit messages in bursts. The more sender and the - * more intermixed messages arrive, the less this algorithm works, but the overhead - * is so minimal (a simple memory compare and move) that this does not hurt. Even - * with a real name lookup cache, this optimization here is useful as it is quicker - * than even a cache lookup). +/* This function implements the main reception loop. Depending on the environment, + * we either use the traditional (but slower) select() or the Linux-specific epoll() + * interface. ./configure settings control which one is used. + * rgerhards, 2009-09-09 */ -BEGINrunInput +#if defined(HAVE_EPOLL_CREATE1) || defined(HAVE_EPOLL_CREATE) +#define NUM_EPOLL_EVENTS 10 +rsRetVal rcvMainLoop(thrdInfo_t *pThrd) +{ + DEFiRet; + int nfds; + int efd; + int i; + struct sockaddr_storage frominetPrev; + int bIsPermitted; + uchar fromHost[NI_MAXHOST]; + uchar fromHostIP[NI_MAXHOST]; + uchar fromHostFQDN[NI_MAXHOST]; + struct epoll_event *udpEPollEvt = NULL; + struct epoll_event currEvt[NUM_EPOLL_EVENTS]; + char errStr[1024]; + + /* start "name caching" algo by making sure the previous system indicator + * is invalidated. + */ + bIsPermitted = 0; + memset(&frominetPrev, 0, sizeof(frominetPrev)); + + CHKmalloc(udpEPollEvt = calloc(udpLstnSocks[0], sizeof(struct epoll_event))); + +# if defined(EPOLL_CLOEXEC) && defined(HAVE_EPOLL_CREATE1) + DBGPRINTF("imudp uses epoll_create1()\n"); + efd = epoll_create1(EPOLL_CLOEXEC); +# else + DBGPRINTF("imudp uses epoll_create()\n"); + efd = epoll_create(NUM_EPOLL_EVENTS); +# endif + if(efd < 0) { + DBGPRINTF("epoll_create1() could not create fd\n"); + ABORT_FINALIZE(RS_RET_IO_ERROR); + } + + /* fill the epoll set - we need to do this only once, as the set + * can not change dyamically. + */ + for (i = 0; i < *udpLstnSocks; i++) { + if (udpLstnSocks[i+1] != -1) { + udpEPollEvt[i].events = EPOLLIN | EPOLLET; + udpEPollEvt[i].data.u64 = i+1; + if(epoll_ctl(efd, EPOLL_CTL_ADD, udpLstnSocks[i+1], &(udpEPollEvt[i])) < 0) { + rs_strerror_r(errno, errStr, sizeof(errStr)); + errmsg.LogError(errno, NO_ERRCODE, "epoll_ctrl failed on fd %d with %s\n", + udpLstnSocks[i+1], errStr); + } + } + } + + while(1) { + /* wait for io to become ready */ + nfds = epoll_wait(efd, currEvt, NUM_EPOLL_EVENTS, -1); + DBGPRINTF("imudp: epoll_wait() returned with %d fds\n", nfds); + + if(pThrd->bShallStop == TRUE) + break; /* terminate input! */ + + for(i = 0 ; i < nfds ; ++i) { + processSocket(pThrd, udpLstnSocks[currEvt[i].data.u64], &frominetPrev, &bIsPermitted, + fromHost, fromHostFQDN, fromHostIP, udpRulesets[currEvt[i].data.u64]); + } + } + +finalize_it: + if(udpEPollEvt != NULL) + free(udpEPollEvt); + + RETiRet; +} +#else /* #if HAVE_EPOLL_CREATE1 */ +/* this is the code for the select() interface */ +rsRetVal rcvMainLoop(thrdInfo_t *pThrd) +{ + DEFiRet; int maxfds; int nfds; int i; @@ -294,16 +383,14 @@ BEGINrunInput uchar fromHost[NI_MAXHOST]; uchar fromHostIP[NI_MAXHOST]; uchar fromHostFQDN[NI_MAXHOST]; -CODESTARTrunInput + /* start "name caching" algo by making sure the previous system indicator * is invalidated. */ bIsPermitted = 0; memset(&frominetPrev, 0, sizeof(frominetPrev)); - /* this is an endless loop - it is terminated when the thread is - * signalled to do so. This, however, is handled by the framework, - * right into the sleep below. - */ + DBGPRINTF("imudp uses select()\n"); + while(1) { /* Add the Unix Domain Sockets to the list of read * descriptors. @@ -338,15 +425,29 @@ CODESTARTrunInput for(i = 0; nfds && i < *udpLstnSocks; i++) { if(FD_ISSET(udpLstnSocks[i+1], &readfds)) { - processSocket(udpLstnSocks[i+1], &frominetPrev, &bIsPermitted, - fromHost, fromHostFQDN, fromHostIP); + processSocket(pThrd, udpLstnSocks[i+1], &frominetPrev, &bIsPermitted, + fromHost, fromHostFQDN, fromHostIP, udpRulesets[i+1]); --nfds; /* indicate we have processed one descriptor */ } } /* end of a run, back to loop for next recv() */ } - return iRet; + RETiRet; +} +#endif /* #if HAVE_EPOLL_CREATE1 */ + +/* This function is called to gather input. + * Note that udpLstnSocks must be non-NULL because otherwise we would not have + * indicated that we want to run (or we have a programming error ;)). -- rgerhards, 2008-10-02 + */ +BEGINrunInput +CODESTARTrunInput + /* this is an endless loop - it is terminated when the thread is + * signalled to do so. This, however, is handled by the framework, + * right into the sleep below. + */ + iRet = rcvMainLoop(pThrd); ENDrunInput @@ -366,9 +467,7 @@ CODESTARTwillRun iMaxLine = glbl.GetMaxLine(); - if((pRcvBuf = malloc((iMaxLine + 1) * sizeof(char))) == NULL) { - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } + CHKmalloc(pRcvBuf = MALLOC((iMaxLine + 1) * sizeof(char))); finalize_it: ENDwillRun @@ -380,6 +479,8 @@ CODESTARTafterRun if(udpLstnSocks != NULL) { net.closeUDPListenSockets(udpLstnSocks); udpLstnSocks = NULL; + free(udpRulesets); + udpRulesets = NULL; } if(pRcvBuf != NULL) { free(pRcvBuf); @@ -397,6 +498,7 @@ CODESTARTmodExit objRelease(glbl, CORE_COMPONENT); objRelease(datetime, CORE_COMPONENT); objRelease(prop, CORE_COMPONENT); + objRelease(ruleset, CORE_COMPONENT); objRelease(net, LM_NET_FILENAME); ENDmodExit @@ -420,10 +522,6 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a free(pszBindAddr); pszBindAddr = NULL; } - if(udpLstnSocks != NULL) { - net.closeUDPListenSockets(udpLstnSocks); - udpLstnSocks = NULL; - } iTimeRequery = TIME_REQUERY_DFLT;/* the default is to query only every second time */ return RS_RET_OK; } @@ -437,13 +535,12 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(glbl, CORE_COMPONENT)); CHKiRet(objUse(datetime, CORE_COMPONENT)); CHKiRet(objUse(prop, CORE_COMPONENT)); + CHKiRet(objUse(ruleset, CORE_COMPONENT)); CHKiRet(objUse(net, LM_NET_FILENAME)); /* register config file handlers */ - /* TODO: add - but this requires more changes, no time right now... - CHKiRet(omsdRegCFSLineHdlr((uchar *)"udpserverbindruleset", 0, eCmdHdlrGetWord, + CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputudpserverbindruleset", 0, eCmdHdlrGetWord, setRuleset, NULL, STD_LOADABLE_MODULE_ID)); - */ CHKiRet(omsdRegCFSLineHdlr((uchar *)"udpserverrun", 0, eCmdHdlrGetWord, addListner, NULL, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"udpserveraddress", 0, eCmdHdlrGetWord, diff --git a/plugins/imuxsock/imuxsock.c b/plugins/imuxsock/imuxsock.c index c099be56..cf6df56c 100644 --- a/plugins/imuxsock/imuxsock.c +++ b/plugins/imuxsock/imuxsock.c @@ -6,7 +6,7 @@ * * File begun on 2007-12-20 by RGerhards (extracted from syslogd.c) * - * Copyright 2007 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2009 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -45,6 +45,7 @@ #include "glbl.h" #include "msg.h" #include "prop.h" +#include "debug.h" MODULE_TYPE_INPUT @@ -70,6 +71,7 @@ DEFobjCurrIf(errmsg) DEFobjCurrIf(glbl) DEFobjCurrIf(prop) +static prop_t *pLocalHostIP = NULL; /* there is only one global IP for all internally-generated messages */ static prop_t *pInputName = NULL; /* our inputName currently is always "imudp", and this will hold it */ static int startIndexUxLocalSockets; /* process funix from that index on (used to * suppress local logging. rgerhards 2005-08-01 @@ -77,8 +79,9 @@ static int startIndexUxLocalSockets; /* process funix from that index on (used t */ static int funixParseHost[MAXFUNIX] = { 0, }; /* should parser parse host name? read-only after startup */ static int funixFlags[MAXFUNIX] = { IGNDATE, }; /* should parser parse host name? read-only after startup */ +static int funixCreateSockPath[MAXFUNIX] = { 0, }; /* auto-creation of socket directory? */ static uchar *funixn[MAXFUNIX] = { (uchar*) _PATH_LOG }; /* read-only after startup */ -static uchar *funixHName[MAXFUNIX] = { NULL, }; /* host-name override - if set, use this instead of actual name */ +static prop_t *funixHName[MAXFUNIX] = { NULL, }; /* host-name override - if set, use this instead of actual name */ static int funixFlowCtl[MAXFUNIX] = { eFLOWCTL_NO_DELAY, }; /* flow control settings for this socket */ static int funix[MAXFUNIX] = { -1, }; /* read-only after startup */ static int nfunix = 1; /* number of Unix sockets open / read-only after startup */ @@ -89,6 +92,8 @@ static uchar *pLogSockName = NULL; static uchar *pLogHostName = NULL; /* host name to use with this socket */ static int bUseFlowCtl = 0; /* use flow control or not (if yes, only LIGHT is used! */ static int bIgnoreTimestamp = 1; /* ignore timestamps present in the incoming message? */ +#define DFLT_bCreateSockPath 0 +static int bCreateSockPath = DFLT_bCreateSockPath; /* auto-create socket path? */ /* set the timestamp ignore / not ignore option for the system @@ -119,29 +124,41 @@ static rsRetVal setSystemLogFlowControl(void __attribute__((unused)) *pVal, int * rgerhards, 2007-12-20 * added capability to specify hostname for socket -- rgerhards, 2008-08-01 */ -static rsRetVal addLstnSocketName(void __attribute__((unused)) *pVal, uchar *pNewVal) +static rsRetVal +addLstnSocketName(void __attribute__((unused)) *pVal, uchar *pNewVal) { + DEFiRet; + if(nfunix < MAXFUNIX) { if(*pNewVal == ':') { funixParseHost[nfunix] = 1; - } - else { + } else { funixParseHost[nfunix] = 0; } - funixHName[nfunix] = pLogHostName; - pLogHostName = NULL; /* re-init for next, not freed because funixHName[] now owns it */ + CHKiRet(prop.Construct(&(funixHName[nfunix]))); + if(pLogHostName == NULL) { + CHKiRet(prop.SetString(funixHName[nfunix], glbl.GetLocalHostName(), ustrlen(glbl.GetLocalHostName()))); + } else { + CHKiRet(prop.SetString(funixHName[nfunix], pLogHostName, ustrlen(pLogHostName))); + /* reset hostname for next socket */ + free(pLogHostName); + pLogHostName = NULL; + } + CHKiRet(prop.ConstructFinalize(funixHName[nfunix])); funixFlowCtl[nfunix] = bUseFlowCtl ? eFLOWCTL_LIGHT_DELAY : eFLOWCTL_NO_DELAY; funixFlags[nfunix] = bIgnoreTimestamp ? IGNDATE : NOFLAG; + funixCreateSockPath[nfunix] = bCreateSockPath; funixn[nfunix++] = pNewVal; - } - else { + } else { errmsg.LogError(0, NO_ERRCODE, "Out of unix socket name descriptors, ignoring %s\n", pNewVal); } - return RS_RET_OK; +finalize_it: + RETiRet; } + /* free the funixn[] socket names - needed as cleanup on several places * note that nfunix is NOT reset! funixn[0] is never freed, as it comes from * the constant memory pool - and if not, it is freeed via some other pointer. @@ -156,8 +173,7 @@ static rsRetVal discardFunixn(void) funixn[i] = NULL; } if(funixHName[i] != NULL) { - free(funixHName[i]); - funixHName[i] = NULL; + prop.Destruct(&(funixHName[i])); } } @@ -165,7 +181,7 @@ static rsRetVal discardFunixn(void) } -static int create_unix_socket(const char *path) +static int create_unix_socket(const char *path, int bCreatePath) { struct sockaddr_un sunx; int fd; @@ -177,6 +193,9 @@ static int create_unix_socket(const char *path) memset(&sunx, 0, sizeof(sunx)); sunx.sun_family = AF_UNIX; + if(bCreatePath) { + makeFileParentDirs((uchar*)path, strlen(path), 0755, -1, -1, 0); + } (void) strncpy(sunx.sun_path, path, sizeof(sunx.sun_path)); fd = socket(AF_UNIX, SOCK_DGRAM, 0); if (fd < 0 || bind(fd, (struct sockaddr *) &sunx, SUN_LEN(&sunx)) < 0 || @@ -190,6 +209,35 @@ static int create_unix_socket(const char *path) } +/* submit received message to the queue engine + */ +static inline rsRetVal +SubmitMsg(uchar *pRcv, int lenRcv, int iSock) +{ + msg_t *pMsg; + DEFiRet; + + /* we now create our own message object and submit it to the queue */ + CHKiRet(msgConstruct(&pMsg)); + MsgSetRawMsg(pMsg, (char*)pRcv, lenRcv); + MsgSetInputName(pMsg, pInputName); + MsgSetFlowControlType(pMsg, funixFlowCtl[iSock]); + + if(funixParseHost[iSock]) { + pMsg->msgFlags = funixFlags[iSock] | NEEDS_PARSING | PARSE_HOSTNAME; + } else { + pMsg->msgFlags = funixFlags[iSock] | NEEDS_PARSING; + } + + MsgSetRcvFrom(pMsg, funixHName[iSock]); + CHKiRet(MsgSetRcvFromIP(pMsg, pLocalHostIP)); + CHKiRet(submitMsg(pMsg)); + +finalize_it: + RETiRet; +} + + /* This function receives data from a socket indicated to be ready * to receive and submits the message received for processing. * rgerhards, 2007-12-20 @@ -218,16 +266,13 @@ static rsRetVal readSocket(int fd, int iSock) if((size_t) iMaxLine < sizeof(bufRcv) - 1) { pRcv = bufRcv; } else { - CHKmalloc(pRcv = (uchar*) malloc(sizeof(uchar) * (iMaxLine + 1))); + CHKmalloc(pRcv = (uchar*) MALLOC(sizeof(uchar) * (iMaxLine + 1))); } iRcvd = recv(fd, pRcv, iMaxLine, 0); dbgprintf("Message from UNIX socket: #%d\n", fd); if (iRcvd > 0) { - parseAndSubmitMessage(funixHName[iSock] == NULL ? glbl.GetLocalHostName() : funixHName[iSock], - (uchar*)"127.0.0.1", pRcv, - iRcvd, funixParseHost[iSock] ? (funixFlags[iSock] | PARSE_HOSTNAME) : funixFlags[iSock], - funixFlowCtl[iSock], pInputName, NULL, 0); + CHKiRet(SubmitMsg(pRcv, iRcvd, iSock)); } else if (iRcvd < 0 && errno != EINTR) { char errStr[1024]; rs_strerror_r(errno, errStr, sizeof(errStr)); @@ -286,6 +331,8 @@ CODESTARTrunInput break; /* terminate input! */ for (i = 0; i < nfunix && nfds > 0; i++) { + if(glbl.GetGlobalInputTermState() == 1) + ABORT_FINALIZE(RS_RET_FORCE_TERM); /* terminate input! */ if ((fd = funix[i]) != -1 && FD_ISSET(fd, &readfds)) { readSocket(fd, i); --nfds; /* indicate we have processed one */ @@ -293,6 +340,7 @@ CODESTARTrunInput } } +finalize_it: RETiRet; ENDrunInput @@ -308,7 +356,7 @@ CODESTARTwillRun /* initialize and return if will run or not */ for (i = startIndexUxLocalSockets ; i < nfunix ; i++) { - if ((funix[i] = create_unix_socket((char*) funixn[i])) != -1) + if ((funix[i] = create_unix_socket((char*) funixn[i], funixCreateSockPath[i])) != -1) dbgprintf("Opened UNIX socket '%s' (fd %d).\n", funixn[i], funix[i]); } @@ -383,6 +431,7 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a nfunix = 1; bIgnoreTimestamp = 1; bUseFlowCtl = 0; + bCreateSockPath = DFLT_bCreateSockPath; return RS_RET_OK; } @@ -405,6 +454,15 @@ CODEmodInit_QueryRegCFSLineHdlr funix[i] = -1; } + CHKiRet(prop.Construct(&pLocalHostIP)); + CHKiRet(prop.SetString(pLocalHostIP, UCHAR_CONSTANT("127.0.0.1"), sizeof("127.0.0.1") - 1)); + CHKiRet(prop.ConstructFinalize(pLocalHostIP)); + + /* now init listen socket zero, the local log socket */ + CHKiRet(prop.Construct(&(funixHName[0]))); + CHKiRet(prop.SetString(funixHName[0], glbl.GetLocalHostName(), ustrlen(glbl.GetLocalHostName()))); + CHKiRet(prop.ConstructFinalize(funixHName[0])); + /* register config file handlers */ CHKiRet(omsdRegCFSLineHdlr((uchar *)"omitlocallogging", 0, eCmdHdlrBinary, NULL, &bOmitLocalLogging, STD_LOADABLE_MODULE_ID)); @@ -416,6 +474,8 @@ CODEmodInit_QueryRegCFSLineHdlr NULL, &pLogHostName, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputunixlistensocketflowcontrol", 0, eCmdHdlrBinary, NULL, &bUseFlowCtl, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputunixlistensocketcreatepath", 0, eCmdHdlrBinary, + NULL, &bCreateSockPath, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"addunixlistensocket", 0, eCmdHdlrGetWord, addLstnSocketName, NULL, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, diff --git a/plugins/omgssapi/omgssapi.c b/plugins/omgssapi/omgssapi.c index 7b5a46e1..49d3f07e 100644 --- a/plugins/omgssapi/omgssapi.c +++ b/plugins/omgssapi/omgssapi.c @@ -193,7 +193,7 @@ static rsRetVal TCPSendGSSInit(void *pvData) base = (gss_base_service_name == NULL) ? "host" : gss_base_service_name; out_tok.length = strlen(pData->f_hname) + strlen(base) + 2; - CHKmalloc(out_tok.value = malloc(out_tok.length)); + CHKmalloc(out_tok.value = MALLOC(out_tok.length)); strcpy(out_tok.value, base); strcat(out_tok.value, "@"); strcat(out_tok.value, pData->f_hname); @@ -409,13 +409,13 @@ CODESTARTdoAction * hard-coded but this may be changed to a config parameter. * rgerhards, 2006-11-30 */ - if(pData->compressionLevel && (l > MIN_SIZE_FOR_COMPRESS)) { + if(pData->compressionLevel && (l > CONF_MIN_SIZE_FOR_COMPRESS)) { Bytef *out; uLongf destLen = sizeof(out) / sizeof(Bytef); uLong srcLen = l; int ret; /* TODO: optimize malloc sequence? -- rgerhards, 2008-09-02 */ - CHKmalloc(out = (Bytef*) malloc(iMaxLine + iMaxLine/100 + 12)); + CHKmalloc(out = (Bytef*) MALLOC(iMaxLine + iMaxLine/100 + 12)); out[0] = 'z'; out[1] = '\0'; ret = compress2((Bytef*) out+1, &destLen, (Bytef*) psz, @@ -554,7 +554,7 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) tmp = ++p; for(i=0 ; *p && isdigit((int) *p) ; ++p, ++i) /* SKIP AND COUNT */; - pData->port = malloc(i + 1); + pData->port = MALLOC(i + 1); if(pData->port == NULL) { errmsg.LogError(0, NO_ERRCODE, "Could not get memory to store syslog forwarding port, " "using default port, results may not be what you intend\n"); diff --git a/plugins/ommail/ommail.c b/plugins/ommail/ommail.c index 3a7669c9..324e1a77 100644 --- a/plugins/ommail/ommail.c +++ b/plugins/ommail/ommail.c @@ -50,6 +50,7 @@ #include "cfsysline.h" #include "module-template.h" #include "errmsg.h" +#include "datetime.h" #include "glbl.h" MODULE_TYPE_OUTPUT @@ -59,6 +60,7 @@ MODULE_TYPE_OUTPUT DEF_OMOD_STATIC_DATA DEFobjCurrIf(errmsg) DEFobjCurrIf(glbl) +DEFobjCurrIf(datetime) /* we add a little support for multiple recipients. We do this via a * singly-linked list, enqueued from the top. -- rgerhards, 2008-08-04 @@ -478,7 +480,7 @@ mkSMTPTimestamp(uchar *pszBuf, size_t lenBuf) static const char szDay[][4] = {"Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"}; static const char szMonth[][4] = {"Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"}; - time(&tCurr); + datetime.GetTime(&tCurr); gmtime_r(&tCurr, &tmCurr); snprintf((char*)pszBuf, lenBuf, "Date: %s, %2d %s %4d %2d:%02d:%02d UT\r\n", szDay[tmCurr.tm_wday], tmCurr.tm_mday, szMonth[tmCurr.tm_mon], 1900 + tmCurr.tm_year, tmCurr.tm_hour, tmCurr.tm_min, tmCurr.tm_sec); @@ -669,6 +671,7 @@ CODESTARTmodExit freeConfigVariables(); /* release what we no longer need */ + objRelease(datetime, CORE_COMPONENT); objRelease(glbl, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); ENDmodExit @@ -698,6 +701,7 @@ CODEmodInit_QueryRegCFSLineHdlr /* tell which objects we need */ CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(datetime, CORE_COMPONENT)); dbgprintf("ommail version %s initializing\n", VERSION); diff --git a/plugins/omoracle/omoracle.c b/plugins/omoracle/omoracle.c index 331b7dd4..48ee1fa4 100644 --- a/plugins/omoracle/omoracle.c +++ b/plugins/omoracle/omoracle.c @@ -47,9 +47,9 @@ $OmoracleStatement \ insert into foo(hostname,message)values(:host,:message) - Also note that identifiers to placeholders are arbitrarry. You - need to define the properties on the template in the correct order - you want them passed to the statement! + Also note that identifiers to placeholders are arbitrary. You need + to define the properties on the template in the correct order you + want them passed to the statement! This file is licensed under the terms of the GPL version 3 or, at your choice, any later version. Exceptionally (perhaps), you are @@ -87,7 +87,8 @@ MODULE_TYPE_OUTPUT DEF_OMOD_STATIC_DATA DEFobjCurrIf(errmsg) -/** */ +/** Structure defining a batch of items to be sent to the database in + * the same statement execution. */ struct oracle_batch { /* Batch size */ @@ -162,8 +163,10 @@ static int oci_errors(void* handle, ub4 htype, sword status) return OCI_SUCCESS; break; case OCI_SUCCESS_WITH_INFO: - errmsg.LogError(0, NO_ERRCODE, "OCI SUCCESS - With info\n"); - break; + OCIErrorGet(handle, 1, NULL, &errcode, buf, sizeof buf, htype); + errmsg.LogError(0, NO_ERRCODE, "OCI SUCCESS - With info: %s", + buf); + return OCI_SUCCESS_WITH_INFO; case OCI_NEED_DATA: errmsg.LogError(0, NO_ERRCODE, "OCI NEEDS MORE DATA\n"); break; @@ -180,6 +183,9 @@ static int oci_errors(void* handle, ub4 htype, sword status) break; case OCI_INVALID_HANDLE: errmsg.LogError(0, NO_ERRCODE, "OCI INVALID HANDLE\n"); + /* In this case we may have to trigger a call to + * tryResume(). */ + return RS_RET_SUSPENDED; break; case OCI_STILL_EXECUTING: errmsg.LogError(0, NO_ERRCODE, "Still executing...\n"); @@ -332,6 +338,48 @@ CODESTARTcreateInstance finalize_it: ENDcreateInstance +/* Analyses the errors during a batch statement execution, and logs + * all the corresponding ORA-MESSAGES, together with some useful + * information. */ +static void log_detailed_err(instanceData* pData) +{ + DEFiRet; + int errs, i, row, code, j; + OCIError *er = NULL, *er2 = NULL; + unsigned char buf[MAX_BUFSIZE]; + + OCIAttrGet(pData->statement, OCI_HTYPE_STMT, &errs, 0, + OCI_ATTR_NUM_DML_ERRORS, pData->error); + errmsg.LogError(0, NO_ERRCODE, "OCI: %d errors in execution of " + "statement: %s", errs, pData->txt_statement); + + CHECKENV(pData->environment, + OCIHandleAlloc(pData->environment, &er, OCI_HTYPE_ERROR, + 0, NULL)); + CHECKENV(pData->environment, + OCIHandleAlloc(pData->environment, &er2, OCI_HTYPE_ERROR, + 0, NULL)); + + for (i = 0; i < errs; i++) { + OCIParamGet(pData->error, OCI_HTYPE_ERROR, + er2, &er, i); + OCIAttrGet(er, OCI_HTYPE_ERROR, &row, 0, + OCI_ATTR_DML_ROW_OFFSET, er2); + errmsg.LogError(0, NO_ERRCODE, "OCI failure in row %d:", row); + for (j = 0; j < pData->batch.arguments; j++) + errmsg.LogError(0, NO_ERRCODE, "%s", + pData->batch.parameters[j][row]); + OCIErrorGet(er, 1, NULL, &code, buf, sizeof buf, + OCI_HTYPE_ERROR); + errmsg.LogError(0, NO_ERRCODE, "FAILURE DETAILS: %s", buf); + } + +finalize_it: + OCIHandleFree(er, OCI_HTYPE_ERROR); + OCIHandleFree(er2, OCI_HTYPE_ERROR); +} + + /* Inserts all stored statements into the database, releasing any * allocated memory. */ static int insert_to_db(instanceData* pData) @@ -346,6 +394,10 @@ static int insert_to_db(instanceData* pData) OCI_BATCH_ERRORS)); finalize_it: + if (iRet == OCI_SUCCESS_WITH_INFO) { + log_detailed_err(pData); + iRet = RS_RET_OK; + } pData->batch.n = 0; OCITransCommit(pData->service, pData->error, 0); dbgprintf ("omoracle insertion to DB %s\n", iRet == RS_RET_OK ? diff --git a/plugins/omrelp/omrelp.c b/plugins/omrelp/omrelp.c index d5ef8b4f..349e45aa 100644 --- a/plugins/omrelp/omrelp.c +++ b/plugins/omrelp/omrelp.c @@ -43,6 +43,7 @@ #include "module-template.h" #include "glbl.h" #include "errmsg.h" +#include "debug.h" MODULE_TYPE_OUTPUT @@ -260,7 +261,7 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) tmp = ++p; for(i=0 ; *p && isdigit((int) *p) ; ++p, ++i) /* SKIP AND COUNT */; - pData->port = malloc(i + 1); + pData->port = MALLOC(i + 1); if(pData->port == NULL) { errmsg.LogError(0, NO_ERRCODE, "Could not get memory to store relp port, " "using default port, results may not be what you intend\n"); diff --git a/plugins/omruleset/Makefile.am b/plugins/omruleset/Makefile.am new file mode 100644 index 00000000..fdd91a6e --- /dev/null +++ b/plugins/omruleset/Makefile.am @@ -0,0 +1,8 @@ +pkglib_LTLIBRARIES = omruleset.la + +omruleset_la_SOURCES = omruleset.c +omruleset_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) +omruleset_la_LDFLAGS = -module -avoid-version +omruleset_la_LIBADD = + +EXTRA_DIST = diff --git a/plugins/omruleset/omruleset.c b/plugins/omruleset/omruleset.c new file mode 100644 index 00000000..eebe1708 --- /dev/null +++ b/plugins/omruleset/omruleset.c @@ -0,0 +1,220 @@ +/* omruleset.c + * This is a very special output module. It permits to pass a message object + * to another rule set. While this is a very simple action, it enables very + * complex configurations, e.g. it supports high-speed "and" conditions, sending + * data to the same file in a non-racy way, include functionality as well as + * some high-performance optimizations (in case the rule sets have the necessary + * queue definitions). So while this code is small, it is pretty important. + * + * NOTE: read comments in module-template.h for details on the calling interface! + * + * File begun on 2009-11-02 by RGerhards + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <stdarg.h> +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <signal.h> +#include <errno.h> +#include <unistd.h> +#include "conf.h" +#include "syslogd-types.h" +//#include "srUtils.h" +#include "template.h" +#include "module-template.h" +#include "errmsg.h" +#include "ruleset.h" +#include "cfsysline.h" +#include "dirty.h" + +MODULE_TYPE_OUTPUT + +/* static data */ +DEFobjCurrIf(ruleset); +DEFobjCurrIf(errmsg); + +/* internal structures + */ +DEF_OMOD_STATIC_DATA + +/* config variables */ +ruleset_t *pRuleset = NULL; /* ruleset to enqueue message to (NULL = Default, not recommended) */ + + +typedef struct _instanceData { + ruleset_t *pRuleset; /* ruleset to enqueue to */ +} instanceData; + + +BEGINcreateInstance +CODESTARTcreateInstance +ENDcreateInstance + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature +ENDisCompatibleWithFeature + + +BEGINfreeInstance +CODESTARTfreeInstance +ENDfreeInstance + + +BEGINdbgPrintInstInfo +CODESTARTdbgPrintInstInfo +ENDdbgPrintInstInfo + + +BEGINtryResume +CODESTARTtryResume +ENDtryResume + +/* Note that we change the flow control type to "no delay", because at this point in + * rsyslog procesing we can not really slow down the producer any longer, as we already + * work off a queue. So a delay would just block out execution for longer than needed. + */ +BEGINdoAction + msg_t *pMsg; +CODESTARTdoAction + pMsg = (msg_t*) ppString[0]; + DBGPRINTF(":omruleset: forwarding message %p to ruleset %p\n", pMsg, pData->pRuleset); + MsgSetFlowControlType(pMsg, eFLOWCTL_NO_DELAY); + MsgSetRuleset(pMsg, pData->pRuleset); + submitMsg(MsgAddRef(pMsg)); +ENDdoAction + +/* set the ruleset name */ +static rsRetVal +setRuleset(void __attribute__((unused)) *pVal, uchar *pszName) +{ + rsRetVal localRet; + DEFiRet; + + localRet = ruleset.GetRuleset(&pRuleset, pszName); + if(localRet == RS_RET_NOT_FOUND) { + errmsg.LogError(0, RS_RET_RULESET_NOT_FOUND, "error: ruleset '%s' not found - ignored", pszName); + } + CHKiRet(localRet); + +finalize_it: + free(pszName); /* no longer needed */ + RETiRet; +} + + +BEGINparseSelectorAct + int iTplOpts; +CODESTARTparseSelectorAct +CODE_STD_STRING_REQUESTparseSelectorAct(1) + /* first check if this config line is actually for us */ + if(strncmp((char*) p, ":omruleset:", sizeof(":omruleset:") - 1)) { + ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); + } + + if(pRuleset == NULL) { + errmsg.LogError(0, RS_RET_NO_RULESET, "error: no ruleset was specified, use " + "$ActionOmrulesetRulesetName directive first!"); + ABORT_FINALIZE(RS_RET_NO_RULESET); + } + + /* ok, if we reach this point, we have something for us */ + p += sizeof(":omruleset:") - 1; /* eat indicator sequence (-1 because of '\0'!) */ + CHKiRet(createInstance(&pData)); + + /* check if a non-standard template is to be applied */ + if(*(p-1) == ';') + --p; + iTplOpts = OMSR_TPL_AS_MSG; + /* we call the message below because we need to call it via our interface definition. However, + * the format specified (if any) is always ignored. + */ + CHKiRet(cflineParseTemplateName(&p, *ppOMSR, 0, iTplOpts, (uchar*) "RSYSLOG_FileFormat")); + pData->pRuleset = pRuleset; + pRuleset = NULL; /* re-set, because there is a high risk of unwanted behavior if we leave it in! */ +CODE_STD_FINALIZERparseSelectorAct +ENDparseSelectorAct + + +BEGINmodExit +CODESTARTmodExit + objRelease(errmsg, CORE_COMPONENT); + objRelease(ruleset, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_OMOD_QUERIES +ENDqueryEtryPt + + + +/* Reset config variables for this module to default values. + */ +static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __attribute__((unused)) *pVal) +{ + DEFiRet; + pRuleset = NULL; + RETiRet; +} + + +BEGINmodInit() + rsRetVal localRet; + rsRetVal (*pomsrGetSupportedTplOpts)(unsigned long *pOpts); + unsigned long opts; + int bMsgPassingSupported; /* does core support template passing as an array? */ +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + /* check if the rsyslog core supports parameter passing code */ + bMsgPassingSupported = 0; + localRet = pHostQueryEtryPt((uchar*)"OMSRgetSupportedTplOpts", &pomsrGetSupportedTplOpts); + if(localRet == RS_RET_OK) { + /* found entry point, so let's see if core supports msg passing */ + CHKiRet((*pomsrGetSupportedTplOpts)(&opts)); + if(opts & OMSR_TPL_AS_MSG) + bMsgPassingSupported = 1; + } else if(localRet != RS_RET_ENTRY_POINT_NOT_FOUND) { + ABORT_FINALIZE(localRet); /* Something else went wrong, what is not acceptable */ + } + + if(!bMsgPassingSupported) { + DBGPRINTF("omruleset: msg-passing is not supported by rsyslog core, can not continue.\n"); + ABORT_FINALIZE(RS_RET_NO_MSG_PASSING); + } + + CHKiRet(objUse(ruleset, CORE_COMPONENT)); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + + CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionomrulesetrulesetname", 0, eCmdHdlrGetWord, + setRuleset, NULL, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, + resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID)); +ENDmodInit + +/* vi:set ai: + */ diff --git a/plugins/omsnmp/omsnmp.c b/plugins/omsnmp/omsnmp.c index 4db60e62..b973b09d 100644 --- a/plugins/omsnmp/omsnmp.c +++ b/plugins/omsnmp/omsnmp.c @@ -222,7 +222,7 @@ static rsRetVal omsnmp_sendsnmp(instanceData *pData, uchar *psz) ABORT_FINALIZE(RS_RET_DISABLE_ACTION); } - pdu->enterprise = (oid *) malloc(enterpriseoidlen * sizeof(oid)); + pdu->enterprise = (oid *) MALLOC(enterpriseoidlen * sizeof(oid)); memcpy(pdu->enterprise, enterpriseoid, enterpriseoidlen * sizeof(oid)); pdu->enterprise_length = enterpriseoidlen; diff --git a/plugins/omtesting/omtesting.c b/plugins/omtesting/omtesting.c index 8f6cdbe5..9442f691 100644 --- a/plugins/omtesting/omtesting.c +++ b/plugins/omtesting/omtesting.c @@ -53,6 +53,7 @@ #include "dirty.h" #include "syslogd-types.h" #include "module-template.h" +#include "conf.h" #include "cfsysline.h" MODULE_TYPE_OUTPUT diff --git a/plugins/omudpspoof/omudpspoof.c b/plugins/omudpspoof/omudpspoof.c index 8eb63c73..50bc6c9a 100644 --- a/plugins/omudpspoof/omudpspoof.c +++ b/plugins/omudpspoof/omudpspoof.c @@ -72,6 +72,7 @@ #include "errmsg.h" #include "dirty.h" #include "unicode-helper.h" +#include "debug.h" #include <libnet.h> @@ -180,7 +181,8 @@ ENDdbgPrintInstInfo /* Send a message via UDP * rgehards, 2007-12-20 */ -static rsRetVal UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) +static inline rsRetVal +UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, size_t len) { struct addrinfo *r; int lsent = 0; @@ -207,9 +209,10 @@ static rsRetVal UDPSend(instanceData *pData, uchar *pszSourcename, char *msg, si for (r = pData->f_addr; r; r = r->ai_next) { tempaddr = (struct sockaddr_in *)r->ai_addr; libnet_clear_packet(libnet_handle); + /* note: libnet does need ports in host order NOT in network byte order! -- rgerhards, 2009-11-12 */ udp = libnet_build_udp( - pData->sourcePort, /* source port */ - tempaddr->sin_port, /* destination port */ + ntohs(pData->sourcePort),/* source port */ + ntohs(tempaddr->sin_port),/* destination port */ LIBNET_UDP_H + len, /* packet length */ 0, /* checksum */ (u_char*)msg, /* payload */ @@ -339,13 +342,13 @@ CODESTARTdoAction * hard-coded but this may be changed to a config parameter. * rgerhards, 2006-11-30 */ - if(pData->compressionLevel && (l > MIN_SIZE_FOR_COMPRESS)) { + if(pData->compressionLevel && (l > CONF_MIN_SIZE_FOR_COMPRESS)) { Bytef *out; uLongf destLen = iMaxLine + iMaxLine/100 +12; /* recommended value from zlib doc */ uLong srcLen = l; int ret; /* TODO: optimize malloc sequence? -- rgerhards, 2008-09-02 */ - CHKmalloc(out = (Bytef*) malloc(destLen)); + CHKmalloc(out = (Bytef*) MALLOC(destLen)); out[0] = 'z'; out[1] = '\0'; ret = compress2((Bytef*) out+1, &destLen, (Bytef*) psz, diff --git a/runtime/apc.c b/runtime/apc.c index c2f61266..3c6b7ec4 100644 --- a/runtime/apc.c +++ b/runtime/apc.c @@ -40,9 +40,11 @@ #include "obj.h" #include "apc.h" #include "srUtils.h" +#include "datetime.h" /* static data */ DEFobjStaticHelpers +DEFobjCurrIf(datetime) /* following is a used to implement a monotonically increasing id for the apcs. That * ID can be used to cancel an apc request. Note that the ID is generated with modulo @@ -200,7 +202,7 @@ unlistCurrent(apc_list_t **ppList) DEFiRet; assert(ppList != NULL); - time(&tCurr); + datetime.GetTime(&tCurr); if(apcListRoot == NULL || apcListRoot->pApc->ttExec > tCurr) { *ppList = NULL; @@ -375,7 +377,7 @@ ENDobjQueryInterface(apc) * rgerhards, 2009-04-06 */ BEGINObjClassExit(apc, OBJ_IS_CORE_MODULE) /* class, version */ - //objRelease(apcstk, CORE_COMPONENT); + objRelease(datetime, CORE_COMPONENT); pthread_mutex_destroy(&listMutex); ENDObjClassExit(apc) @@ -386,7 +388,7 @@ ENDObjClassExit(apc) */ BEGINObjClassInit(apc, 1, OBJ_IS_CORE_MODULE) /* class, version */ /* request objects we use */ - //CHKiRet(objUse(apcstk, CORE_COMPONENT)); + CHKiRet(objUse(datetime, CORE_COMPONENT)); /* set our own handlers */ OBJSetMethodHandler(objMethod_DEBUGPRINT, apcDebugPrint); diff --git a/runtime/batch.h b/runtime/batch.h index 031718a7..2b3aa83e 100644 --- a/runtime/batch.h +++ b/runtime/batch.h @@ -34,7 +34,7 @@ typedef enum { BATCH_STATE_RDY = 0, /* object ready for processing */ BATCH_STATE_BAD = 1, /* unrecoverable failure while processing, do NOT resubmit to same action */ - BATCH_STATE_SUB = 2, /* message submitted for processing, outcome yet unkonwn */ + BATCH_STATE_SUB = 2, /* message submitted for processing, outcome yet unknown */ BATCH_STATE_COMM = 3, /* message successfully commited */ BATCH_STATE_DISC = 4, /* discarded - processed OK, but do not submit to any other action */ } batch_state_t; diff --git a/runtime/conf.c b/runtime/conf.c index 2e37edf2..d6e6ccf6 100644 --- a/runtime/conf.c +++ b/runtime/conf.c @@ -95,17 +95,16 @@ DEFobjCurrIf(ruleset) static int iNbrActions = 0; /* number of currently defined actions */ -/* The following global variables are used for building +/* The following module-global variables are used for building * tag and host selector lines during startup and config reload. * This is stored as a global variable pool because of its ease. It is * also fairly compatible with multi-threading as the stratup code must - * be run in a single thread anyways. So there can be no race conditions. These - * variables are no longer used once the configuration has been loaded (except, - * of course, during a reload). rgerhards 2005-10-18 + * be run in a single thread anyways. So there can be no race conditions. + * rgerhards 2005-10-18 */ -EHostnameCmpMode eDfltHostnameCmpMode; -cstr_t *pDfltHostnameCmp; -cstr_t *pDfltProgNameCmp; +static EHostnameCmpMode eDfltHostnameCmpMode = HN_NO_COMP; +static cstr_t *pDfltHostnameCmp = NULL; +static cstr_t *pDfltProgNameCmp = NULL; /* process a directory and include all of its files into @@ -1038,7 +1037,7 @@ static rsRetVal cflineDoFilter(uchar **pp, rule_t *f) DEFiRet; ASSERT(pp != NULL); - ASSERT(f != NULL); + ISOBJ_TYPE_assert(f, rule); /* check which filter we need to pull... */ switch(**pp) { @@ -1060,6 +1059,7 @@ static rsRetVal cflineDoFilter(uchar **pp, rule_t *f) * and, if so, we copy them over. rgerhards, 2005-10-18 */ if(pDfltProgNameCmp != NULL) { +RUNLOG_STR("dflt ProgNameCmp != NULL, setting opCSProgNameComp"); CHKiRet(rsCStrConstructFromCStr(&(f->pCSProgNameComp), pDfltProgNameCmp)); } @@ -1248,6 +1248,15 @@ ENDobjQueryInterface(conf) */ BEGINObjClassExit(conf, OBJ_IS_CORE_MODULE) /* CHANGE class also in END MACRO! */ CODESTARTObjClassExit(conf) + /* free no-longer needed module-global variables */ + if(pDfltHostnameCmp != NULL) { + rsCStrDestruct(&pDfltHostnameCmp); + } + + if(pDfltProgNameCmp != NULL) { + rsCStrDestruct(&pDfltProgNameCmp); + } + /* release objects we no longer need */ objRelease(expr, CORE_COMPONENT); objRelease(ctok, CORE_COMPONENT); diff --git a/runtime/conf.h b/runtime/conf.h index 6db1623e..d85d1f82 100644 --- a/runtime/conf.h +++ b/runtime/conf.h @@ -49,10 +49,6 @@ ENDinterface(conf) PROTOTYPEObj(conf); -/* TODO: remove them below (means move the config init code) -- rgerhards, 2008-02-19 */ -extern EHostnameCmpMode eDfltHostnameCmpMode; -extern cstr_t *pDfltHostnameCmp; -extern cstr_t *pDfltProgNameCmp; /* TODO: the following 2 need to go in conf obj interface... */ rsRetVal cflineParseTemplateName(uchar** pp, omodStringRequest_t *pOMSR, int iEntry, int iTplOpts, uchar *dfltTplName); rsRetVal cflineParseFileName(uchar* p, uchar *pFileName, omodStringRequest_t *pOMSR, int iEntry, int iTplOpts, uchar *pszTpl); diff --git a/runtime/datetime.c b/runtime/datetime.c index 6160bd7c..4ab4318d 100644 --- a/runtime/datetime.c +++ b/runtime/datetime.c @@ -127,6 +127,24 @@ static void getCurrTime(struct syslogTime *t, time_t *ttSeconds) } +/* A fast alternative to getCurrTime() and time() that only obtains + * a timestamp like time() does. I was told that gettimeofday(), at + * least under Linux, is much faster than time() and I could confirm + * this testing. So I created that function as a replacement. + * rgerhards, 2009-11-12 + */ +static time_t +getTime(time_t *ttSeconds) +{ + struct timeval tp; + + if(gettimeofday(&tp, NULL) == -1) + return -1; + + if(ttSeconds != NULL) + *ttSeconds = tp.tv_sec; + return tp.tv_sec; +} /******************************************************************* @@ -831,6 +849,7 @@ CODESTARTobjQueryInterface(datetime) * of course, also affects the "if" above). */ pIf->getCurrTime = getCurrTime; + pIf->GetTime = getTime; pIf->ParseTIMESTAMP3339 = ParseTIMESTAMP3339; pIf->ParseTIMESTAMP3164 = ParseTIMESTAMP3164; pIf->formatTimestampToMySQL = formatTimestampToMySQL; diff --git a/runtime/datetime.h b/runtime/datetime.h index 8140eb71..1de3db95 100644 --- a/runtime/datetime.h +++ b/runtime/datetime.h @@ -41,8 +41,10 @@ BEGINinterface(datetime) /* name must also be changed in ENDinterface macro! */ int (*formatTimestamp3339)(struct syslogTime *ts, char* pBuf); int (*formatTimestamp3164)(struct syslogTime *ts, char* pBuf); int (*formatTimestampSecFrac)(struct syslogTime *ts, char* pBuf); + /* v3, 2009-11-12 */ + time_t (*GetTime)(time_t *ttSeconds); ENDinterface(datetime) -#define datetimeCURR_IF_VERSION 2 /* increment whenever you change the interface structure! */ +#define datetimeCURR_IF_VERSION 3 /* increment whenever you change the interface structure! */ /* interface changes: * 1 - initial version * 2 - not compatible to 1 - bugfix required ParseTIMESTAMP3164 to accept char ** as diff --git a/runtime/debug.c b/runtime/debug.c index 959d56a3..545ac876 100644 --- a/runtime/debug.c +++ b/runtime/debug.c @@ -51,6 +51,7 @@ #include "rsyslog.h" #include "debug.h" #include "atomic.h" +#include "cfsysline.h" #include "obj.h" @@ -960,7 +961,7 @@ void dbgprintf(char *fmt, ...) { va_list ap; - char pszWriteBuf[1024]; + char pszWriteBuf[32*1024]; size_t lenWriteBuf; if(!(Debug && debugging_on)) @@ -969,6 +970,16 @@ dbgprintf(char *fmt, ...) va_start(ap, fmt); lenWriteBuf = vsnprintf(pszWriteBuf, sizeof(pszWriteBuf), fmt, ap); va_end(ap); + + if(lenWriteBuf >= sizeof(pszWriteBuf)) { + /* if we need to truncate, do it in a somewhat useful way... */ + pszWriteBuf[sizeof(pszWriteBuf) - 5] = '!'; + pszWriteBuf[sizeof(pszWriteBuf) - 4] = '.'; + pszWriteBuf[sizeof(pszWriteBuf) - 3] = '.'; + pszWriteBuf[sizeof(pszWriteBuf) - 2] = '.'; + pszWriteBuf[sizeof(pszWriteBuf) - 1] = '\n'; + lenWriteBuf = sizeof(pszWriteBuf); + } dbgprint(NULL, pszWriteBuf, lenWriteBuf); } @@ -1244,6 +1255,20 @@ dbgPrintNameIsInList(const uchar *pName, dbgPrintName_t *pRoot) } +/* this is a special version of malloc that fills the alloced memory with + * HIGHVALUE, as this helps to identify bugs. -- rgerhards, 2009-10-22 + */ +void * +dbgmalloc(size_t size) +{ + void *pRet; + pRet = malloc(size); + if(pRet != NULL) + memset(pRet, 0xff, size); + return pRet; +} + + /* read in the runtime options * rgerhards, 2008-02-28 */ @@ -1276,6 +1301,7 @@ dbgGetRuntimeOptions(void) "NoLogTimestamp\n" "Nostdoout\n" "filetrace=file (may be provided multiple times)\n" + "DebugOnDemand - enables debugging on USR1, but does not turn on output\n" "\nSee debug.html in your doc set or http://www.rsyslog.com for details\n"); exit(1); } else if(!strcasecmp((char*)optname, "debug")) { @@ -1284,6 +1310,13 @@ dbgGetRuntimeOptions(void) */ Debug = 1; debugging_on = 1; + } else if(!strcasecmp((char*)optname, "debugondemand")) { + /* Enables debugging, but turns off debug output */ + Debug = 1; + debugging_on = 1; + dbgprintf("Note: debug on demand turned on via configuraton file, " + "use USR1 signal to activate.\n"); + debugging_on = 0; } else if(!strcasecmp((char*)optname, "logfuncflow")) { bLogFuncFlow = 1; } else if(!strcasecmp((char*)optname, "logallocfree")) { diff --git a/runtime/debug.h b/runtime/debug.h index dcbfb930..8d9c1ceb 100644 --- a/runtime/debug.h +++ b/runtime/debug.h @@ -3,7 +3,7 @@ * Definitions for the debug and run-time analysis support module. * Contains a lot of macros. * - * Copyright 2008 Rainer Gerhards and Adiscon GmbH. + * Copyright 2008, 2009 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -99,6 +99,7 @@ void dbgExitFunc(dbgFuncDB_t *pFuncDB, int iStackPtrRestore, int iRet); void dbgSetExecLocation(int iStackPtr, int line); void dbgSetThrdName(uchar *pszName); void dbgPrintAllDebugInfo(void); +void *dbgmalloc(size_t size); /* macros */ #define DBGPRINTF(...) if(Debug) { dbgprintf(__VA_ARGS__); } @@ -126,6 +127,12 @@ void dbgPrintAllDebugInfo(void); # define RUNLOG_STR(str) #endif +#ifdef MEMCHECK +# define MALLOC(x) dbgmalloc(x) +#else +# define MALLOC(x) malloc(x) +#endif + /* mutex operations */ #define MUTOP_LOCKWAIT 1 #define MUTOP_LOCK 2 diff --git a/runtime/glbl.c b/runtime/glbl.c index f27b8e73..71c2ed0d 100644 --- a/runtime/glbl.c +++ b/runtime/glbl.c @@ -56,6 +56,7 @@ DEFobjCurrIf(prop) */ static uchar *pszWorkDir = NULL; static int bOptimizeUniProc = 1; /* enable uniprocessor optimizations */ +static int bParseHOSTNAMEandTAG = 1; /* parser modification (based on startup params!) */ static int bPreserveFQDN = 0; /* should FQDNs always be preserved? */ static int iMaxLine = 2048; /* maximum length of a syslog message */ static int iDefPFFamily = PF_UNSPEC; /* protocol family (IPv4, IPv6 or both) */ @@ -94,6 +95,7 @@ static dataType Get##nameFunc(void) \ return(nameVar); \ } +SIMP_PROP(ParseHOSTNAMEandTAG, bParseHOSTNAMEandTAG, int) SIMP_PROP(OptimizeUniProc, bOptimizeUniProc, int) SIMP_PROP(PreserveFQDN, bPreserveFQDN, int) SIMP_PROP(MaxLine, iMaxLine, int) @@ -183,6 +185,7 @@ finalize_it: RETiRet; } + /* return our local hostname as a string property */ static prop_t* @@ -266,6 +269,7 @@ CODESTARTobjQueryInterface(glbl) pIf->Set##name = Set##name; SIMP_PROP(MaxLine); SIMP_PROP(OptimizeUniProc); + SIMP_PROP(ParseHOSTNAMEandTAG); SIMP_PROP(PreserveFQDN); SIMP_PROP(DefPFFamily); SIMP_PROP(DropMalPTRMsgs); diff --git a/runtime/glbl.h b/runtime/glbl.h index 0d0c8210..7506f16b 100644 --- a/runtime/glbl.h +++ b/runtime/glbl.h @@ -64,9 +64,11 @@ BEGINinterface(glbl) /* name must also be changed in ENDinterface macro! */ /* added v4, 2009-07-20 */ int (*GetGlobalInputTermState)(void); void (*SetGlobalInputTermination)(void); + /* added v5, 2009-11-03 */ + SIMP_PROP(ParseHOSTNAMEandTAG, int) #undef SIMP_PROP ENDinterface(glbl) -#define glblCURR_IF_VERSION 4 /* increment whenever you change the interface structure! */ +#define glblCURR_IF_VERSION 5 /* increment whenever you change the interface structure! */ /* version 2 had PreserveFQDN added - rgerhards, 2008-12-08 */ /* the remaining prototypes */ diff --git a/runtime/module-template.h b/runtime/module-template.h index d49da2c9..18aad650 100644 --- a/runtime/module-template.h +++ b/runtime/module-template.h @@ -46,6 +46,9 @@ DEFobjCurrIf(obj) #define DEF_LMOD_STATIC_DATA \ DEF_MOD_STATIC_DATA +#define DEF_PMOD_STATIC_DATA \ + DEFobjCurrIf(obj) \ + DEF_MOD_STATIC_DATA /* Macro to define the module type. Each module can only have a single type. If @@ -65,6 +68,7 @@ static rsRetVal modGetType(eModType_t *modType) \ #define MODULE_TYPE_INPUT MODULE_TYPE(eMOD_IN) #define MODULE_TYPE_OUTPUT MODULE_TYPE(eMOD_OUT) +#define MODULE_TYPE_PARSER MODULE_TYPE(eMOD_PARSER) #define MODULE_TYPE_LIB \ DEF_LMOD_STATIC_DATA \ MODULE_TYPE(eMOD_LIB) @@ -400,6 +404,18 @@ static rsRetVal queryEtryPt(uchar *name, rsRetVal (**pEtryPoint)())\ #define CODEqueryEtryPt_STD_LIB_QUERIES \ CODEqueryEtryPt_STD_MOD_QUERIES +/* the following definition is the standard block for queryEtryPt for PARSER + * modules. This can be used if no specific handling (e.g. to cover version + * differences) is needed. + */ +#define CODEqueryEtryPt_STD_PMOD_QUERIES \ + CODEqueryEtryPt_STD_MOD_QUERIES \ + else if(!strcmp((char*) name, "parse")) {\ + *pEtryPoint = parse;\ + } else if(!strcmp((char*) name, "GetParserName")) {\ + *pEtryPoint = GetParserName;\ + } + /* modInit() * This has an extra parameter, which is the specific name of the modInit * function. That is needed for built-in modules, which must have unique @@ -590,5 +606,31 @@ static rsRetVal doHUP(instanceData __attribute__((unused)) *pData)\ } +/* parse() - main entry point of parser modules + */ +#define BEGINparse \ +static rsRetVal parse(msg_t *pMsg)\ +{\ + DEFiRet; + +#define CODESTARTparse \ + assert(pMsg != NULL); + +#define ENDparse \ + RETiRet;\ +} + + +/* function to specify the parser name. This is done via a single command which + * receives a ANSI string as parameter. + */ +#define PARSER_NAME(x) \ +static rsRetVal GetParserName(uchar **ppSz)\ +{\ + *ppSz = UCHAR_CONSTANT(x);\ + return RS_RET_OK;\ +} + + /* vim:set ai: */ diff --git a/runtime/modules.c b/runtime/modules.c index bdb15e7f..fd3468d8 100644 --- a/runtime/modules.c +++ b/runtime/modules.c @@ -11,7 +11,7 @@ * * File begun on 2007-07-22 by RGerhards * - * Copyright 2007 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007, 2009 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -57,10 +57,12 @@ #include "cfsysline.h" #include "modules.h" #include "errmsg.h" +#include "parser.h" /* static data */ DEFobjStaticHelpers DEFobjCurrIf(errmsg) +DEFobjCurrIf(parser) /* we must ensure that only one thread at one time tries to load or unload * modules, otherwise we may see race conditions. This first came up with @@ -94,7 +96,6 @@ static rsRetVal dummyEndTransaction() } static rsRetVal dummyIsCompatibleWithFeature() { -dbgprintf("XXX: dummy isCompatibleWithFeature called!\n"); return RS_RET_INCOMPATIBLE; } @@ -403,10 +404,13 @@ finalize_it: static rsRetVal doModInit(rsRetVal (*modInit)(int, int*, rsRetVal(**)(), rsRetVal(*)(), modInfo_t*), uchar *name, void *pModHdlr) { - DEFiRet; rsRetVal localRet; modInfo_t *pNew = NULL; + uchar *pParserName; + parser_t *pParser; /* used for parser modules */ + rsRetVal (*GetParserName)(uchar**); rsRetVal (*modGetType)(eModType_t *pType); + DEFiRet; assert(modInit != NULL); @@ -475,6 +479,33 @@ doModInit(rsRetVal (*modInit)(int, int*, rsRetVal(**)(), rsRetVal(*)(), modInfo_ break; case eMOD_LIB: break; + case eMOD_PARSER: + /* first, we need to obtain the parser object. We could not do that during + * init as that would have caused class bootstrap issues which are not + * absolutely necessary. Note that we can call objUse() multiple times, it + * handles that. + */ + CHKiRet(objUse(parser, CORE_COMPONENT)); + /* here, we create a new parser object */ + CHKiRet((*pNew->modQueryEtryPt)((uchar*)"parse", &pNew->mod.pm.parse)); + CHKiRet((*pNew->modQueryEtryPt)((uchar*)"GetParserName", &GetParserName)); + CHKiRet(GetParserName(&pParserName)); + CHKiRet(parser.Construct(&pParser)); + + /* check some features */ + localRet = pNew->isCompatibleWithFeature(sFEATUREAutomaticSanitazion); + if(localRet == RS_RET_OK){ + CHKiRet(parser.SetDoSanitazion(pParser, TRUE)); + } + localRet = pNew->isCompatibleWithFeature(sFEATUREAutomaticPRIParsing); + if(localRet == RS_RET_OK){ + CHKiRet(parser.SetDoPRIParsing(pParser, TRUE)); + } + + CHKiRet(parser.SetName(pParser, pParserName)); + CHKiRet(parser.SetModPtr(pParser, pNew)); + CHKiRet(parser.ConstructFinalize(pParser)); + break; } pNew->pszName = (uchar*) strdup((char*)name); /* we do not care if strdup() fails, we can accept that */ @@ -521,6 +552,9 @@ static void modPrintList(void) case eMOD_LIB: dbgprintf("library"); break; + case eMOD_PARSER: + dbgprintf("parser"); + break; } dbgprintf(" module.\n"); dbgprintf("Entry points:\n"); @@ -867,6 +901,7 @@ BEGINObjClassExit(module, OBJ_IS_LOADABLE_MODULE) /* CHANGE class also in END MA CODESTARTObjClassExit(module) /* release objects we no longer need */ objRelease(errmsg, CORE_COMPONENT); + objRelease(parser, CORE_COMPONENT); /* We have a problem in our reference counting, which leads to this function * being called too early. This usually is no problem, but if we destroy * the mutex object, we get into trouble. So rather than finding the root cause, diff --git a/runtime/modules.h b/runtime/modules.h index 71e3199c..62f86ded 100644 --- a/runtime/modules.h +++ b/runtime/modules.h @@ -12,7 +12,7 @@ * * File begun on 2007-07-22 by RGerhards * - * Copyright 2007, 2008 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2009 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -51,9 +51,10 @@ #define CURR_MOD_IF_VERSION 5 typedef enum eModType_ { - eMOD_IN, /* input module */ - eMOD_OUT, /* output module */ - eMOD_LIB /* library module - this module provides one or many interfaces */ + eMOD_IN = 0, /* input module */ + eMOD_OUT = 1, /* output module */ + eMOD_LIB = 2, /* library module */ + eMOD_PARSER = 3 /* parser module */ } eModType_t; @@ -73,7 +74,7 @@ typedef enum eModLinkType_ { eMOD_LINK_ALL /* special: all linkage types, e.g. for unload */ } eModLinkType_t; -typedef struct modInfo_s { +struct modInfo_s { struct modInfo_s *pPrev; /* support for creating a double linked module list */ struct modInfo_s *pNext; /* support for creating a linked module list */ int iIFVers; /* Interface version of module */ @@ -117,7 +118,10 @@ typedef struct modInfo_s { rsRetVal (*parseSelectorAct)(uchar**, void**,omodStringRequest_t**); } om; struct { /* data for library modules */ - } fm; + } lm; + struct { /* data for parser modules */ + rsRetVal (*parse)(msg_t*); + } pm; } mod; void *pModHdlr; /* handler to the dynamic library holding the module */ # ifdef DEBUG @@ -126,7 +130,7 @@ typedef struct modInfo_s { */ modUsr_t *pModUsrRoot; # endif -} modInfo_t; +}; /* interfaces */ BEGINinterface(module) /* name must also be changed in ENDinterface macro! */ @@ -152,6 +156,5 @@ extern uchar *pModDir; /* read-only after startup */ #endif /* #ifndef MODULES_H_INCLUDED */ -/* - * vi:set ai: +/* vi:set ai: */ diff --git a/runtime/msg.c b/runtime/msg.c index d28ee350..623c5b4a 100644 --- a/runtime/msg.c +++ b/runtime/msg.c @@ -288,7 +288,7 @@ static inline void getInputName(msg_t *pM, uchar **ppsz, int *plen) { BEGINfunc - if(pM == NULL) { + if(pM == NULL || pM->pInputName == NULL) { *ppsz = UCHAR_CONSTANT(""); *plen = 0; } else { @@ -626,13 +626,12 @@ static inline rsRetVal msgBaseConstruct(msg_t **ppThis) msg_t *pM; assert(ppThis != NULL); - CHKmalloc(pM = malloc(sizeof(msg_t))); + CHKmalloc(pM = MALLOC(sizeof(msg_t))); objConstructSetObjInfo(pM); /* intialize object helper entities */ /* initialize members in ORDER they appear in structure (think "cache line"!) */ pM->flowCtlType = 0; pM->bDoLock = 0; - pM->bParseHOSTNAME = 0; pM->iRefCount = 1; pM->iSeverity = -1; pM->iFacility = -1; @@ -861,7 +860,6 @@ msg_t* MsgDup(msg_t* pOld) pNew->iRefCount = 1; pNew->iSeverity = pOld->iSeverity; pNew->iFacility = pOld->iFacility; - pNew->bParseHOSTNAME = pOld->bParseHOSTNAME; pNew->msgFlags = pOld->msgFlags; pNew->iProtocolVersion = pOld->iProtocolVersion; pNew->ttGenTime = pOld->ttGenTime; @@ -935,7 +933,7 @@ msg_t* MsgDup(msg_t* pOld) * We do not serialize the cache properties. We re-create them when needed. * This saves us a lot of memory. Performance is no concern, as serializing * is a so slow operation that recration of the caches does not count. Also, - * we do not serialize bParseHOSTNAME, as this is only a helper variable + * we do not serialize --currently none--, as this is only a helper variable * during msg construction - and never again used later. * rgerhards, 2008-01-03 */ @@ -1229,7 +1227,7 @@ static inline char *getTimeReported(msg_t *pM, enum tplFormatTypes eFmt) case tplFmtMySQLDate: MsgLock(pM); if(pM->pszTIMESTAMP_MySQL == NULL) { - if((pM->pszTIMESTAMP_MySQL = malloc(15)) == NULL) { + if((pM->pszTIMESTAMP_MySQL = MALLOC(15)) == NULL) { MsgUnlock(pM); return ""; } @@ -1240,7 +1238,7 @@ static inline char *getTimeReported(msg_t *pM, enum tplFormatTypes eFmt) case tplFmtPgSQLDate: MsgLock(pM); if(pM->pszTIMESTAMP_PgSQL == NULL) { - if((pM->pszTIMESTAMP_PgSQL = malloc(21)) == NULL) { + if((pM->pszTIMESTAMP_PgSQL = MALLOC(21)) == NULL) { MsgUnlock(pM); return ""; } @@ -1281,7 +1279,7 @@ static inline char *getTimeGenerated(msg_t *pM, enum tplFormatTypes eFmt) case tplFmtDefault: MsgLock(pM); if(pM->pszRcvdAt3164 == NULL) { - if((pM->pszRcvdAt3164 = malloc(16)) == NULL) { + if((pM->pszRcvdAt3164 = MALLOC(16)) == NULL) { MsgUnlock(pM); return ""; } @@ -1292,7 +1290,7 @@ static inline char *getTimeGenerated(msg_t *pM, enum tplFormatTypes eFmt) case tplFmtMySQLDate: MsgLock(pM); if(pM->pszRcvdAt_MySQL == NULL) { - if((pM->pszRcvdAt_MySQL = malloc(15)) == NULL) { + if((pM->pszRcvdAt_MySQL = MALLOC(15)) == NULL) { MsgUnlock(pM); return ""; } @@ -1303,7 +1301,7 @@ static inline char *getTimeGenerated(msg_t *pM, enum tplFormatTypes eFmt) case tplFmtPgSQLDate: MsgLock(pM); if(pM->pszRcvdAt_PgSQL == NULL) { - if((pM->pszRcvdAt_PgSQL = malloc(21)) == NULL) { + if((pM->pszRcvdAt_PgSQL = MALLOC(21)) == NULL) { MsgUnlock(pM); return ""; } @@ -1314,7 +1312,7 @@ static inline char *getTimeGenerated(msg_t *pM, enum tplFormatTypes eFmt) case tplFmtRFC3164Date: MsgLock(pM); if(pM->pszRcvdAt3164 == NULL) { - if((pM->pszRcvdAt3164 = malloc(16)) == NULL) { + if((pM->pszRcvdAt3164 = MALLOC(16)) == NULL) { MsgUnlock(pM); return ""; } @@ -1325,7 +1323,7 @@ static inline char *getTimeGenerated(msg_t *pM, enum tplFormatTypes eFmt) case tplFmtRFC3339Date: MsgLock(pM); if(pM->pszRcvdAt3339 == NULL) { - if((pM->pszRcvdAt3339 = malloc(33)) == NULL) { + if((pM->pszRcvdAt3339 = MALLOC(33)) == NULL) { MsgUnlock(pM); return ""; } @@ -1576,7 +1574,7 @@ void MsgSetTAG(msg_t *pMsg, uchar* pszBuf, size_t lenBuf) /* small enough: use fixed buffer (faster!) */ pBuf = pMsg->TAG.szBuf; } else { - if((pBuf = (uchar*) malloc(pMsg->iLenTAG + 1)) == NULL) { + if((pBuf = (uchar*) MALLOC(pMsg->iLenTAG + 1)) == NULL) { /* truncate message, better than completely loosing it... */ pBuf = pMsg->TAG.szBuf; pMsg->iLenTAG = CONF_TAG_BUFSIZE - 1; @@ -1941,7 +1939,7 @@ void MsgSetHOSTNAME(msg_t *pThis, uchar* pszHOSTNAME, int lenHOSTNAME) if(pThis->iLenHOSTNAME < CONF_HOSTNAME_BUFSIZE) { /* small enough: use fixed buffer (faster!) */ pThis->pszHOSTNAME = pThis->szHOSTNAME; - } else if((pThis->pszHOSTNAME = (uchar*) malloc(pThis->iLenHOSTNAME + 1)) == NULL) { + } else if((pThis->pszHOSTNAME = (uchar*) MALLOC(pThis->iLenHOSTNAME + 1)) == NULL) { /* truncate message, better than completely loosing it... */ pThis->pszHOSTNAME = pThis->szHOSTNAME; pThis->iLenHOSTNAME = CONF_HOSTNAME_BUFSIZE - 1; @@ -1993,7 +1991,7 @@ rsRetVal MsgReplaceMSG(msg_t *pThis, uchar* pszMSG, int lenMSG) lenNew = pThis->iLenRawMsg + lenMSG - pThis->iLenMSG; if(lenMSG > pThis->iLenMSG && lenNew >= CONF_RAWMSG_BUFSIZE) { /* we have lost our "bet" and need to alloc a new buffer ;) */ - CHKmalloc(bufNew = malloc(lenNew + 1)); + CHKmalloc(bufNew = MALLOC(lenNew + 1)); memcpy(bufNew, pThis->pszRawMsg, pThis->offMSG); if(pThis->pszRawMsg != pThis->szRawMsg) free(pThis->pszRawMsg); @@ -2024,7 +2022,7 @@ void MsgSetRawMsg(msg_t *pThis, char* pszRawMsg, size_t lenMsg) if(pThis->iLenRawMsg < CONF_RAWMSG_BUFSIZE) { /* small enough: use fixed buffer (faster!) */ pThis->pszRawMsg = pThis->szRawMsg; - } else if((pThis->pszRawMsg = (uchar*) malloc(pThis->iLenRawMsg + 1)) == NULL) { + } else if((pThis->pszRawMsg = (uchar*) MALLOC(pThis->iLenRawMsg + 1)) == NULL) { /* truncate message, better than completely loosing it... */ pThis->pszRawMsg = pThis->szRawMsg; pThis->iLenRawMsg = CONF_RAWMSG_BUFSIZE - 1; @@ -2080,7 +2078,7 @@ static uchar *getNOW(eNOWType eNow) uchar *pBuf; struct syslogTime t; - if((pBuf = (uchar*) malloc(sizeof(uchar) * tmpBUFSIZE)) == NULL) { + if((pBuf = (uchar*) MALLOC(sizeof(uchar) * tmpBUFSIZE)) == NULL) { return NULL; } @@ -2214,7 +2212,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, pRes = (uchar*)getPRI(pMsg); break; case PROP_PRI_TEXT: - pBuf = malloc(20 * sizeof(uchar)); + pBuf = MALLOC(20 * sizeof(uchar)); if(pBuf == NULL) { *pbMustBeFreed = 0; return UCHAR_CONSTANT("**OUT OF MEMORY**"); @@ -2370,7 +2368,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, /* we got our end pointer, now do the copy */ /* TODO: code copied from below, this is a candidate for a separate function */ iLen = pFldEnd - pFld + 1; /* the +1 is for an actual char, NOT \0! */ - pBufStart = pBuf = malloc((iLen + 1) * sizeof(char)); + pBufStart = pBuf = MALLOC((iLen + 1) * sizeof(char)); if(pBuf == NULL) { if(*pbMustBeFreed == 1) free(pRes); @@ -2416,7 +2414,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, ; /*DO NOTHING*/ } else { iLen = iTo - iFrom + 1; /* the +1 is for an actual char, NOT \0! */ - pBufStart = pBuf = malloc((iLen + 1) * sizeof(char)); + pBufStart = pBuf = MALLOC((iLen + 1) * sizeof(char)); if(pBuf == NULL) { if(*pbMustBeFreed == 1) free(pRes); @@ -2521,7 +2519,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, iLenBuf = pmatch[pTpe->data.field.iSubMatchToUse].rm_eo - pmatch[pTpe->data.field.iSubMatchToUse].rm_so; - pB = malloc((iLenBuf + 1) * sizeof(uchar)); + pB = MALLOC((iLenBuf + 1) * sizeof(uchar)); if (pB == NULL) { if (*pbMustBeFreed == 1) @@ -2578,7 +2576,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, uchar *pBStart; uchar *pB; uchar *pSrc; - pBStart = pB = malloc((bufLen + 1) * sizeof(char)); + pBStart = pB = MALLOC((bufLen + 1) * sizeof(char)); if(pB == NULL) { if(*pbMustBeFreed == 1) free(pRes); @@ -2625,7 +2623,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, } if(bDropped) { - pDst = pDstStart = malloc(iLenBuf + 1); + pDst = pDstStart = MALLOC(iLenBuf + 1); if(pDst == NULL) { if(*pbMustBeFreed == 1) free(pRes); @@ -2661,7 +2659,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, } else { if(bufLen == -1) bufLen = ustrlen(pRes); - pDst = pDstStart = malloc(bufLen + 1); + pDst = pDstStart = MALLOC(bufLen + 1); if(pDst == NULL) { if(*pbMustBeFreed == 1) free(pRes); @@ -2701,7 +2699,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, int i; iLenBuf += iNumCC * 4; - pBStart = pB = malloc((iLenBuf + 1) * sizeof(uchar)); + pBStart = pB = MALLOC((iLenBuf + 1) * sizeof(uchar)); if(pB == NULL) { if(*pbMustBeFreed == 1) free(pRes); @@ -2747,7 +2745,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, } if(bDropped) { - pDst = pDstStart = malloc(iLenBuf + 1); + pDst = pDstStart = MALLOC(iLenBuf + 1); if(pDst == NULL) { if(*pbMustBeFreed == 1) free(pRes); @@ -2783,7 +2781,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, } else { if(bufLen == -1) bufLen = ustrlen(pRes); - pDst = pDstStart = malloc(bufLen + 1); + pDst = pDstStart = MALLOC(bufLen + 1); if(pDst == NULL) { if(*pbMustBeFreed == 1) free(pRes); @@ -2840,7 +2838,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, /* check if we need to obtain a private copy */ if(*pbMustBeFreed == 0) { /* ok, original copy, need a private one */ - pB = malloc((iLn + 1) * sizeof(uchar)); + pB = MALLOC((iLn + 1) * sizeof(uchar)); if(pB == NULL) { *pbMustBeFreed = 0; return UCHAR_CONSTANT("**OUT OF MEMORY**"); @@ -2869,7 +2867,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, bufLen = ustrlen(pRes); iBufLen = bufLen; /* the malloc may be optimized, we currently use the worst case... */ - pBStart = pDst = malloc((2 * iBufLen + 3) * sizeof(uchar)); + pBStart = pDst = MALLOC((2 * iBufLen + 3) * sizeof(uchar)); if(pDst == NULL) { if(*pbMustBeFreed == 1) free(pRes); diff --git a/runtime/msg.h b/runtime/msg.h index b006cbec..9101cef7 100644 --- a/runtime/msg.h +++ b/runtime/msg.h @@ -60,7 +60,6 @@ struct msg { once data has entered the queue, this property is no longer needed. */ pthread_mutex_t mut; bool bDoLock; /* use the mutex? */ - bool bParseHOSTNAME; /* should the hostname be parsed from the message? */ short iRefCount; /* reference counter (0 = unused) */ /* background: the hostname is not present on "regular" messages * received via UNIX domain sockets from the same machine. However, @@ -209,6 +208,16 @@ MsgSetRawMsgSize(msg_t *pMsg, size_t newLen) } +/* get the ruleset that is associated with the ruleset. + * May be NULL. -- rgerhards, 2009-10-27 + */ +static inline ruleset_t* +MsgGetRuleset(msg_t *pMsg) +{ + return pMsg->pRuleset; +} + + #endif /* #ifndef MSG_H_INCLUDED */ /* vim:set ai: */ diff --git a/runtime/net.c b/runtime/net.c index 5cafe522..dfae53e2 100644 --- a/runtime/net.c +++ b/runtime/net.c @@ -173,7 +173,7 @@ AddPermittedPeerWildcard(permittedPeers_t *pPeer, uchar* pszStr, size_t lenStr) /* alloc memory for the domain component. We may waste a byte or * two, but that's ok. */ - CHKmalloc(pNew->pszDomainPart = malloc(lenStr +1 )); + CHKmalloc(pNew->pszDomainPart = MALLOC(lenStr +1 )); } if(pszStr[0] == '*') { @@ -695,7 +695,7 @@ static rsRetVal AddAllowedSender(struct AllowedSenders **ppRoot, struct AllowedS case AF_INET: /* add IPv4 */ iSignificantBits = 32; allowIP.flags = 0; - if((allowIP.addr.NetAddr = malloc(res->ai_addrlen)) == NULL) { + if((allowIP.addr.NetAddr = MALLOC(res->ai_addrlen)) == NULL) { ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } memcpy(allowIP.addr.NetAddr, res->ai_addr, res->ai_addrlen); @@ -710,7 +710,7 @@ static rsRetVal AddAllowedSender(struct AllowedSenders **ppRoot, struct AllowedS iSignificantBits = 32; allowIP.flags = 0; - if((allowIP.addr.NetAddr = malloc(sizeof(struct sockaddr_in))) + if((allowIP.addr.NetAddr = MALLOC(sizeof(struct sockaddr_in))) == NULL) { ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } @@ -732,7 +732,7 @@ static rsRetVal AddAllowedSender(struct AllowedSenders **ppRoot, struct AllowedS iSignificantBits = 128; allowIP.flags = 0; - if((allowIP.addr.NetAddr = malloc(res->ai_addrlen)) == NULL) { + if((allowIP.addr.NetAddr = MALLOC(res->ai_addrlen)) == NULL) { ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } memcpy(allowIP.addr.NetAddr, res->ai_addr, res->ai_addrlen); @@ -1306,7 +1306,7 @@ getLocalHostname(uchar **ppName) do { if(buf == NULL) { buf_len = 128; /* Initial guess */ - CHKmalloc(buf = malloc(buf_len)); + CHKmalloc(buf = MALLOC(buf_len)); } else { buf_len += buf_len; CHKmalloc(buf = realloc (buf, buf_len)); @@ -1370,7 +1370,7 @@ int *create_udp_socket(uchar *hostname, uchar *pszPort, int bIsServer) /* Count max number of sockets we may open */ for (maxs = 0, r = res; r != NULL ; r = r->ai_next, maxs++) /* EMPTY */; - socks = malloc((maxs+1) * sizeof(int)); + socks = MALLOC((maxs+1) * sizeof(int)); if (socks == NULL) { errmsg.LogError(0, NO_ERRCODE, "couldn't allocate memory for UDP sockets, suspending UDP message reception"); freeaddrinfo(res); diff --git a/runtime/net.h b/runtime/net.h index ec364b1c..a50b6fcb 100644 --- a/runtime/net.h +++ b/runtime/net.h @@ -149,8 +149,8 @@ BEGINinterface(net) /* name must also be changed in ENDinterface macro! */ /* v5 interface additions */ int (*CmpHost)(struct sockaddr_storage *, struct sockaddr_storage*, size_t); /* data members - these should go away over time... TODO */ - int *pACLAddHostnameOnFail; /* add hostname to acl when DNS resolving has failed */ - int *pACLDontResolve; /* add hostname to acl instead of resolving it to IP(s) */ + int pACLAddHostnameOnFail; /* add hostname to acl when DNS resolving has failed */ + int pACLDontResolve; /* add hostname to acl instead of resolving it to IP(s) */ ENDinterface(net) #define netCURR_IF_VERSION 5 /* increment whenever you change the interface structure! */ diff --git a/runtime/nsd_gtls.c b/runtime/nsd_gtls.c index 79ceffb3..74c142f2 100644 --- a/runtime/nsd_gtls.c +++ b/runtime/nsd_gtls.c @@ -44,6 +44,7 @@ #include "stringbuf.h" #include "errmsg.h" #include "net.h" +#include "datetime.h" #include "nsd_ptcp.h" #include "nsdsel_gtls.h" #include "nsd_gtls.h" @@ -61,6 +62,7 @@ DEFobjStaticHelpers DEFobjCurrIf(errmsg) DEFobjCurrIf(glbl) DEFobjCurrIf(net) +DEFobjCurrIf(datetime) DEFobjCurrIf(nsd_ptcp) static int bGlblSrvrInitDone = 0; /**< 0 - server global init not yet done, 1 - already done */ @@ -129,7 +131,7 @@ readFile(uchar *pszFile, gnutls_datum_t *pBuf) ABORT_FINALIZE(RS_RET_FILE_TOO_LARGE); } - CHKmalloc(pBuf->data = malloc(stat_st.st_size)); + CHKmalloc(pBuf->data = MALLOC(stat_st.st_size)); pBuf->size = stat_st.st_size; if(read(fd, pBuf->data, stat_st.st_size) != stat_st.st_size) { errmsg.LogError(0, RS_RET_IO_ERROR, "error or incomplete read of file '%s'", pszFile); @@ -1016,7 +1018,7 @@ gtlsChkPeerCertValidity(nsd_gtls_t *pThis) } /* get current time for certificate validation */ - if(time(&ttNow) == -1) + if(datetime.GetTime(&ttNow) == -1) ABORT_FINALIZE(RS_RET_SYS_ERR); /* as it looks, we need to validate the expiration dates ourselves... @@ -1479,7 +1481,7 @@ Rcv(nsd_t *pNsd, uchar *pBuf, ssize_t *pLenBuf) if(pThis->pszRcvBuf == NULL) { /* we have no buffer, so we need to malloc one */ - CHKmalloc(pThis->pszRcvBuf = malloc(NSD_GTLS_MAX_RCVBUF)); + CHKmalloc(pThis->pszRcvBuf = MALLOC(NSD_GTLS_MAX_RCVBUF)); pThis->lenRcvBuf = -1; } @@ -1694,6 +1696,7 @@ CODESTARTObjClassExit(nsd_gtls) objRelease(nsd_ptcp, LM_NSD_PTCP_FILENAME); objRelease(net, LM_NET_FILENAME); objRelease(glbl, CORE_COMPONENT); + objRelease(datetime, CORE_COMPONENT); objRelease(errmsg, CORE_COMPONENT); ENDObjClassExit(nsd_gtls) @@ -1705,6 +1708,7 @@ ENDObjClassExit(nsd_gtls) BEGINObjClassInit(nsd_gtls, 1, OBJ_IS_LOADABLE_MODULE) /* class, version */ /* request objects we use */ CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(datetime, CORE_COMPONENT)); CHKiRet(objUse(glbl, CORE_COMPONENT)); CHKiRet(objUse(net, LM_NET_FILENAME)); CHKiRet(objUse(nsd_ptcp, LM_NSD_PTCP_FILENAME)); diff --git a/runtime/nsd_ptcp.c b/runtime/nsd_ptcp.c index 54ee0666..fe31ab40 100644 --- a/runtime/nsd_ptcp.c +++ b/runtime/nsd_ptcp.c @@ -296,12 +296,12 @@ FillRemHost(nsd_ptcp_t *pThis, struct sockaddr *pAddr) * memory consumption) */ len = strlen((char*)szIP) + 1; /* +1 for \0 byte */ - if((pThis->pRemHostIP = malloc(len)) == NULL) + if((pThis->pRemHostIP = MALLOC(len)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); memcpy(pThis->pRemHostIP, szIP, len); len = strlen((char*)szHname) + 1; /* +1 for \0 byte */ - if((pThis->pRemHostName = malloc(len)) == NULL) { + if((pThis->pRemHostName = MALLOC(len)) == NULL) { free(pThis->pRemHostIP); /* prevent leak */ pThis->pRemHostIP = NULL; ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); diff --git a/runtime/obj.c b/runtime/obj.c index aebea332..45dac776 100644 --- a/runtime/obj.c +++ b/runtime/obj.c @@ -48,7 +48,7 @@ * * File begun on 2008-01-04 by RGerhards * - * Copyright 2008 Rainer Gerhards and Adiscon GmbH. + * Copyright 2008, 2009 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -90,6 +90,7 @@ #include "cfsysline.h" #include "unicode-helper.h" #include "apc.h" +#include "datetime.h" /* static data */ DEFobjCurrIf(obj) /* we define our own interface, as this is expected by some macros! */ @@ -1129,7 +1130,7 @@ UseObj(char *srcFile, uchar *pObjName, uchar *pObjFile, interface_t *pIf) /* DEV debug only: dbgprintf("source file %s requests object '%s', ifIsLoaded %d\n", srcFile, pObjName, pIf->ifIsLoaded); */ - d_pthread_mutex_lock(&mutObjGlobalOp); + pthread_mutex_lock(&mutObjGlobalOp); if(pIf->ifIsLoaded == 1) { ABORT_FINALIZE(RS_RET_OK); /* we are already set */ @@ -1170,7 +1171,7 @@ UseObj(char *srcFile, uchar *pObjName, uchar *pObjFile, interface_t *pIf) pIf->ifIsLoaded = 1; /* we are happy */ finalize_it: - d_pthread_mutex_unlock(&mutObjGlobalOp); + pthread_mutex_unlock(&mutObjGlobalOp); if(pStr != NULL) rsCStrDestruct(&pStr); @@ -1193,7 +1194,7 @@ ReleaseObj(char *srcFile, uchar *pObjName, uchar *pObjFile, interface_t *pIf) /* dev debug only dbgprintf("source file %s releasing object '%s', ifIsLoaded %d\n", srcFile, pObjName, pIf->ifIsLoaded); */ - d_pthread_mutex_lock(&mutObjGlobalOp); + pthread_mutex_lock(&mutObjGlobalOp); if(pObjFile == NULL) FINALIZE; /* if it is not a lodable module, we do not need to do anything... */ @@ -1214,7 +1215,7 @@ ReleaseObj(char *srcFile, uchar *pObjName, uchar *pObjFile, interface_t *pIf) pIf->ifIsLoaded = 0; /* indicated "no longer valid" */ finalize_it: - d_pthread_mutex_unlock(&mutObjGlobalOp); + pthread_mutex_unlock(&mutObjGlobalOp); if(pStr != NULL) rsCStrDestruct(&pStr); @@ -1330,8 +1331,9 @@ objClassInit(modInfo_t *pModInfo) CHKiRet(objGetObjInterface(&obj)); /* get ourselves ;) */ /* init classes we use (limit to as few as possible!) */ - CHKiRet(apcClassInit(pModInfo)); CHKiRet(errmsgClassInit(pModInfo)); + CHKiRet(datetimeClassInit(pModInfo)); + CHKiRet(apcClassInit(pModInfo)); CHKiRet(cfsyslineInit()); CHKiRet(varClassInit(pModInfo)); CHKiRet(moduleClassInit(pModInfo)); diff --git a/runtime/objomsr.c b/runtime/objomsr.c index 8dddc4b8..1d442c61 100644 --- a/runtime/objomsr.c +++ b/runtime/objomsr.c @@ -67,31 +67,25 @@ rsRetVal OMSRconstruct(omodStringRequest_t **ppThis, int iNumEntries) assert(ppThis != NULL); assert(iNumEntries >= 0); - if((pThis = calloc(1, sizeof(omodStringRequest_t))) == NULL) { - iRet = RS_RET_OUT_OF_MEMORY; - goto abort_it; - } + CHKmalloc(pThis = calloc(1, sizeof(omodStringRequest_t))); /* got the structure, so fill it */ pThis->iNumEntries = iNumEntries; /* allocate string for template name array. The individual strings will be * allocated as the code progresses (we do not yet know the string sizes) */ - if((pThis->ppTplName = calloc(iNumEntries, sizeof(uchar*))) == NULL) { - OMSRdestruct(pThis); - pThis = NULL; - iRet = RS_RET_OUT_OF_MEMORY; - goto abort_it; - } + CHKmalloc(pThis->ppTplName = calloc(iNumEntries, sizeof(uchar*))); + /* allocate the template options array. */ - if((pThis->piTplOpts = calloc(iNumEntries, sizeof(int))) == NULL) { - OMSRdestruct(pThis); - pThis = NULL; - iRet = RS_RET_OUT_OF_MEMORY; - goto abort_it; - } + CHKmalloc(pThis->piTplOpts = calloc(iNumEntries, sizeof(int))); -abort_it: +finalize_it: + if(iRet != RS_RET_OK) { + if(pThis != NULL) { + OMSRdestruct(pThis); + pThis = NULL; + } + } *ppThis = pThis; RETiRet; } @@ -155,7 +149,7 @@ OMSRgetSupportedTplOpts(unsigned long *pOpts) { DEFiRet; assert(pOpts != NULL); - *pOpts = OMSR_RQD_TPL_OPT_SQL | OMSR_TPL_AS_ARRAY; + *pOpts = OMSR_RQD_TPL_OPT_SQL | OMSR_TPL_AS_ARRAY | OMSR_TPL_AS_MSG; RETiRet; } diff --git a/runtime/objomsr.h b/runtime/objomsr.h index 75ad0fb8..e59b774f 100644 --- a/runtime/objomsr.h +++ b/runtime/objomsr.h @@ -27,8 +27,12 @@ /* define flags for required template options */ #define OMSR_NO_RQD_TPL_OPTS 0 #define OMSR_RQD_TPL_OPT_SQL 1 +/* only one of OMSR_TPL_AS_ARRAY or _AS_MSG must be specified, if both are given + * results are unpredictable. + */ #define OMSR_TPL_AS_ARRAY 2 /* introduced in 4.1.6, 2009-04-03 */ -/* next option is 4, 8, 16, ... */ +#define OMSR_TPL_AS_MSG 4 /* introduced in 5.3.4, 2009-11-02 */ +/* next option is 8, 16, 32, ... */ struct omodStringRequest_s { /* strings requested by output module for doAction() */ int iNumEntries; /* number of array entries for data elements below */ diff --git a/runtime/parser.c b/runtime/parser.c index 466066e7..38f72986 100644 --- a/runtime/parser.c +++ b/runtime/parser.c @@ -37,7 +37,13 @@ #include "dirty.h" #include "msg.h" #include "obj.h" +#include "datetime.h" #include "errmsg.h" +#include "parser.h" +#include "ruleset.h" +#include "unicode-helper.h" +#include "dirty.h" +#include "cfsysline.h" /* some defines */ #define DEFUPRI (LOG_USER|LOG_NOTICE) @@ -46,26 +52,165 @@ DEFobjStaticHelpers DEFobjCurrIf(glbl) DEFobjCurrIf(errmsg) +DEFobjCurrIf(datetime) +DEFobjCurrIf(ruleset) /* static data */ +/* config data */ +static uchar cCCEscapeChar = '#';/* character to be used to start an escape sequence for control chars */ +static int bEscapeCCOnRcv = 1; /* escape control characters on reception: 0 - no, 1 - yes */ +static int bDropTrailingLF = 1; /* drop trailing LF's on reception? */ + +/* This is the list of all parsers known to us. + * This is also used to unload all modules on shutdown. + */ +parserList_t *pParsLstRoot = NULL; + +/* this is the list of the default parsers, to be used if no others + * are specified. + */ +parserList_t *pDfltParsLst = NULL; + + +/* intialize (but NOT allocate) a parser list. Primarily meant as a hook + * which can be used to extend the list in the future. So far, just sets + * it to NULL. + */ +static rsRetVal +InitParserList(parserList_t **pListRoot) +{ + *pListRoot = NULL; + return RS_RET_OK; +} + + +/* destruct a parser list. The list elements are destroyed, but the parser objects + * themselves are not modified. (That is done at a late stage during rsyslogd + * shutdown and need not be considered here.) + */ +static rsRetVal +DestructParserList(parserList_t **ppListRoot) +{ + parserList_t *pParsLst; + parserList_t *pParsLstDel; + + pParsLst = *ppListRoot; + while(pParsLst != NULL) { + pParsLstDel = pParsLst; + pParsLst = pParsLst->pNext; + free(pParsLstDel); + } + *ppListRoot = NULL; + return RS_RET_OK; +} + -/* this is a dummy class init +/* Add a parser to the list. We use a VERY simple and ineffcient algorithm, + * but it is employed only for a few milliseconds during config processing. So + * I prefer to keep it very simple and with simple data structures. Unfortunately, + * we need to preserve the order, but I don't like to add a tail pointer as that + * would require a container object. So I do the extra work to skip to the tail + * when adding elements... + * rgerhards, 2009-11-03 */ -rsRetVal parserClassInit(void) +static rsRetVal +AddParserToList(parserList_t **ppListRoot, parser_t *pParser) { + parserList_t *pThis; + parserList_t *pTail; DEFiRet; - /* request objects we use */ - CHKiRet(objGetObjInterface(&obj)); /* this provides the root pointer for all other queries */ - CHKiRet(objUse(glbl, CORE_COMPONENT)); - CHKiRet(objUse(errmsg, CORE_COMPONENT)); -// TODO: free components! see action.c + CHKmalloc(pThis = MALLOC(sizeof(parserList_t))); + pThis->pParser = pParser; + pThis->pNext = NULL; + + if(*ppListRoot == NULL) { + pThis->pNext = *ppListRoot; + *ppListRoot = pThis; + } else { + /* find tail first */ + for(pTail = *ppListRoot ; pTail->pNext != NULL ; pTail = pTail->pNext) + /* just search, do nothing else */; + /* add at tail */ + pTail->pNext = pThis; + } + +finalize_it: + RETiRet; +} + + +/* find a parser based on the provided name */ +static rsRetVal +FindParser(parser_t **ppParser, uchar *pName) +{ + parserList_t *pThis; + DEFiRet; + + for(pThis = pParsLstRoot ; pThis != NULL ; pThis = pThis->pNext) { + if(ustrcmp(pThis->pParser->pName, pName) == 0) { + *ppParser = pThis->pParser; + FINALIZE; /* found it, iRet still eq. OK! */ + } + } + + iRet = RS_RET_PARSER_NOT_FOUND; + finalize_it: RETiRet; } +/* --- END helper functions for parser list handling --- */ + +/* Add a an already existing parser to the default list. As usual, order + * of calls is important (most importantly, that means the legacy parser, + * which can process everything, MUST be added last!). + * rgerhards, 2009-11-04 + */ +static rsRetVal +AddDfltParser(uchar *pName) +{ + parser_t *pParser; + DEFiRet; + + CHKiRet(FindParser(&pParser, pName)); + CHKiRet(AddParserToList(&pDfltParsLst, pParser)); + dbgprintf("Parser '%s' added to default parser set.\n", pName); + +finalize_it: + RETiRet; +} + + + +BEGINobjConstruct(parser) /* be sure to specify the object type also in END macro! */ +ENDobjConstruct(parser) + +/* ConstructionFinalizer. The most important chore is to add the parser object + * to our global list of available parsers. + * rgerhards, 2009-11-03 + */ +rsRetVal parserConstructFinalize(parser_t *pThis) +{ + DEFiRet; + + ISOBJ_TYPE_assert(pThis, parser); + CHKiRet(AddParserToList(&pParsLstRoot, pThis)); + DBGPRINTF("Parser '%s' added to list of available parsers.\n", pThis->pName); + +finalize_it: + RETiRet; +} + +BEGINobjDestruct(parser) /* be sure to specify the object type also in END and CODESTART macros! */ +CODESTARTobjDestruct(parser) + dbgprintf("destructing parser '%s'\n", pThis->pName); + free(pThis->pName); +ENDobjDestruct(parser) + + /* uncompress a received message if it is compressed. * pMsg->pszRawMsg buffer is updated. * rgerhards, 2008-10-09 @@ -96,7 +241,7 @@ static inline rsRetVal uncompressMessage(msg_t *pMsg) */ int ret; iLenDefBuf = glbl.GetMaxLine(); - CHKmalloc(deflateBuf = malloc(sizeof(uchar) * (iLenDefBuf + 1))); + CHKmalloc(deflateBuf = MALLOC(sizeof(uchar) * (iLenDefBuf + 1))); ret = uncompress((uchar *) deflateBuf, &iLenDefBuf, (uchar *) pszMsg+1, lenMsg-1); DBGPRINTF("Compressed message uncompressed with status %d, length: new %ld, old %d.\n", ret, (long) iLenDefBuf, (int) (lenMsg-1)); @@ -153,7 +298,7 @@ finalize_it: * rgerhards, 2007-09-14 */ static inline rsRetVal -sanitizeMessage(msg_t *pMsg) +SanitizeMsg(msg_t *pMsg) { DEFiRet; uchar *pszMsg; @@ -169,10 +314,6 @@ sanitizeMessage(msg_t *pMsg) assert(pMsg != NULL); assert(pMsg->iLenRawMsg > 0); -# ifdef USE_NETZIP - CHKiRet(uncompressMessage(pMsg)); -# endif - pszMsg = pMsg->pszRawMsg; lenMsg = pMsg->iLenRawMsg; @@ -223,7 +364,7 @@ sanitizeMessage(msg_t *pMsg) if(maxDest < sizeof(szSanBuf)) pDst = szSanBuf; else - CHKmalloc(pDst = malloc(sizeof(uchar) * (iMaxLine + 1))); + CHKmalloc(pDst = MALLOC(sizeof(uchar) * (iMaxLine + 1))); iSrc = iDst = 0; while(iSrc < lenMsg && iDst < maxDest - 3) { /* leave some space if last char must be escaped */ if(iscntrl((int) pszMsg[iSrc])) { @@ -231,14 +372,14 @@ sanitizeMessage(msg_t *pMsg) * can not handle it! -- rgerhards, 2009-08-26 */ if(pszMsg[iSrc] == '\0' || bEscapeCCOnRcv) { - /* we are configured to escape control characters. Please note - * that this most probably break non-western character sets like - * Japanese, Korean or Chinese. rgerhards, 2007-07-17 - */ - pDst[iDst++] = cCCEscapeChar; - pDst[iDst++] = '0' + ((pszMsg[iSrc] & 0300) >> 6); - pDst[iDst++] = '0' + ((pszMsg[iSrc] & 0070) >> 3); - pDst[iDst++] = '0' + ((pszMsg[iSrc] & 0007)); + /* we are configured to escape control characters. Please note + * that this most probably break non-western character sets like + * Japanese, Korean or Chinese. rgerhards, 2007-07-17 + */ + pDst[iDst++] = cCCEscapeChar; + pDst[iDst++] = '0' + ((pszMsg[iSrc] & 0300) >> 6); + pDst[iDst++] = '0' + ((pszMsg[iSrc] & 0070) >> 3); + pDst[iDst++] = '0' + ((pszMsg[iSrc] & 0007)); } } else { pDst[iDst++] = pszMsg[iSrc]; @@ -255,27 +396,18 @@ finalize_it: RETiRet; } - -/* Parse a received message. The object's rawmsg property is taken and - * parsed according to the relevant standards. This can later be - * extended to support configured parsers. - * rgerhards, 2008-10-09 +/* A standard parser to parse out the PRI. This is made available in + * this module as it is expected that allmost all parsers will need + * that functionality and so they do not need to implement it themsleves. */ -rsRetVal parseMsg(msg_t *pMsg) +static inline rsRetVal +ParsePRI(msg_t *pMsg) { - DEFiRet; - uchar *msg; int pri; + uchar *msg; int lenMsg; int iPriText; - - if(pMsg->iLenRawMsg == 0) - ABORT_FINALIZE(RS_RET_EMPTY_MSG); - - CHKiRet(sanitizeMessage(pMsg)); - - /* we needed to sanitize first, because we otherwise do not have a C-string we can print... */ - DBGPRINTF("msg parser: flags %x, from '%s', msg '%s'\n", pMsg->msgFlags, getRcvFrom(pMsg), pMsg->pszRawMsg); + DEFiRet; /* pull PRI */ lenMsg = pMsg->iLenRawMsg; @@ -299,31 +431,247 @@ rsRetVal parseMsg(msg_t *pMsg) pMsg->iFacility = LOG_FAC(pri); pMsg->iSeverity = LOG_PRI(pri); MsgSetAfterPRIOffs(pMsg, msg - pMsg->pszRawMsg); + RETiRet; +} + + +/* Parse a received message. The object's rawmsg property is taken and + * parsed according to the relevant standards. This can later be + * extended to support configured parsers. + * rgerhards, 2008-10-09 + */ +static rsRetVal +ParseMsg(msg_t *pMsg) +{ + rsRetVal localRet; + parserList_t *pParserList; + parser_t *pParser; + bool bIsSanitized; + bool bPRIisParsed; + static int iErrMsgRateLimiter = 0; + DEFiRet; - /* rger 2005-11-24 (happy thanksgiving!): we now need to check if we have - * a traditional syslog message or one formatted according to syslog-protocol. - * We need to apply different parsers depending on that. We use the - * -protocol VERSION field for the detection. + if(pMsg->iLenRawMsg == 0) + ABORT_FINALIZE(RS_RET_EMPTY_MSG); + +# ifdef USE_NETZIP + CHKiRet(uncompressMessage(pMsg)); +# endif + + /* we take the risk to print a non-sanitized string, because this is the best we can get + * (and that functionality is too important for debugging to drop it...). */ - if(msg[0] == '1' && msg[1] == ' ') { - dbgprintf("Message has syslog-protocol format.\n"); - setProtocolVersion(pMsg, 1); - if(parseRFCSyslogMsg(pMsg, pMsg->msgFlags) == 1) { - msgDestruct(&pMsg); - ABORT_FINALIZE(RS_RET_ERR); // TODO: we need to handle these cases! + DBGPRINTF("msg parser: flags %x, from '%s', msg '%.50s'\n", pMsg->msgFlags, + getRcvFrom(pMsg), pMsg->pszRawMsg); + + /* we now need to go through our list of parsers and see which one is capable of + * parsing the message. Note that the first parser that requires message sanitization + * will cause it to happen. After that, access to the unsanitized message is no + * loger possible. + */ + pParserList = ruleset.GetParserList(pMsg); + if(pParserList == NULL) { + pParserList = pDfltParsLst; + } + DBGPRINTF("parse using parser list %p%s.\n", pParserList, + (pParserList == pDfltParsLst) ? " (the default list)" : ""); + + bIsSanitized = FALSE; + bPRIisParsed = FALSE; + while(pParserList != NULL) { + pParser = pParserList->pParser; + if(pParser->bDoSanitazion && bIsSanitized == FALSE) { + CHKiRet(SanitizeMsg(pMsg)); + if(pParser->bDoPRIParsing && bPRIisParsed == FALSE) { + CHKiRet(ParsePRI(pMsg)); + bPRIisParsed = TRUE; + } + bIsSanitized = TRUE; } - } else { /* we have legacy syslog */ - dbgprintf("Message has legacy syslog format.\n"); - setProtocolVersion(pMsg, 0); - if(parseLegacySyslogMsg(pMsg, pMsg->msgFlags) == 1) { - msgDestruct(&pMsg); - ABORT_FINALIZE(RS_RET_ERR); // TODO: we need to handle these cases! + localRet = pParser->pModule->mod.pm.parse(pMsg); + dbgprintf("Parser '%s' returned %d\n", pParser->pName, localRet); + if(localRet != RS_RET_COULD_NOT_PARSE) + break; + pParserList = pParserList->pNext; + } + + /* We need to log a warning message and drop the message if we did not find a parser. + * Note that we log at most the first 1000 message, as this may very well be a problem + * that causes a message generation loop. We do not synchronize that counter, it doesn't + * matter if we log a handful messages more than we should... + */ + if(localRet != RS_RET_OK) { + if(++iErrMsgRateLimiter > 1000) { + errmsg.LogError(0, localRet, "Error: one message could not be processed by " + "any parser, message is being discarded (start of raw msg: '%.50s')", + pMsg->pszRawMsg); } + DBGPRINTF("No parser could process the message (state %d), we need to discard it.\n", localRet); + ABORT_FINALIZE(localRet); } - /* finalize message object */ + /* "finalize" message object */ pMsg->msgFlags &= ~NEEDS_PARSING; /* this message is now parsed */ finalize_it: RETiRet; } + +/* set the parser name - string is copied over, call can continue to use it, + * but must free it if desired. + */ +static rsRetVal +SetName(parser_t *pThis, uchar *name) +{ + DEFiRet; + + ISOBJ_TYPE_assert(pThis, parser); + assert(name != NULL); + + if(pThis->pName != NULL) { + free(pThis->pName); + pThis->pName = NULL; + } + + CHKmalloc(pThis->pName = ustrdup(name)); + +finalize_it: + RETiRet; +} + + +/* set a pointer to "our" module. Note that no module + * pointer must already be set. + */ +static rsRetVal +SetModPtr(parser_t *pThis, modInfo_t *pMod) +{ + ISOBJ_TYPE_assert(pThis, parser); + assert(pMod != NULL); + assert(pThis->pModule == NULL); + pThis->pModule = pMod; + return RS_RET_OK; +} + + +/* Specify if we should do standard message sanitazion before we pass the data + * down to the parser. + */ +static rsRetVal +SetDoSanitazion(parser_t *pThis, int bDoIt) +{ + ISOBJ_TYPE_assert(pThis, parser); + pThis->bDoSanitazion = bDoIt; + return RS_RET_OK; +} + + +/* Specify if we should do standard PRI parsing before we pass the data + * down to the parser module. + */ +static rsRetVal +SetDoPRIParsing(parser_t *pThis, int bDoIt) +{ + ISOBJ_TYPE_assert(pThis, parser); + pThis->bDoPRIParsing = bDoIt; + return RS_RET_OK; +} + + +/* queryInterface function-- rgerhards, 2009-11-03 + */ +BEGINobjQueryInterface(parser) +CODESTARTobjQueryInterface(parser) + if(pIf->ifVersion != parserCURR_IF_VERSION) { /* check for current version, increment on each change */ + ABORT_FINALIZE(RS_RET_INTERFACE_NOT_SUPPORTED); + } + + /* ok, we have the right interface, so let's fill it + * Please note that we may also do some backwards-compatibility + * work here (if we can support an older interface version - that, + * of course, also affects the "if" above). + */ + pIf->Construct = parserConstruct; + pIf->ConstructFinalize = parserConstructFinalize; + pIf->Destruct = parserDestruct; + pIf->SetName = SetName; + pIf->SetModPtr = SetModPtr; + pIf->SetDoSanitazion = SetDoSanitazion; + pIf->SetDoPRIParsing = SetDoPRIParsing; + pIf->ParseMsg = ParseMsg; + pIf->SanitizeMsg = SanitizeMsg; + pIf->InitParserList = InitParserList; + pIf->DestructParserList = DestructParserList; + pIf->AddParserToList = AddParserToList; + pIf->AddDfltParser = AddDfltParser; + pIf->FindParser = FindParser; +finalize_it: +ENDobjQueryInterface(parser) + + + +/* Reset config variables to default values. + * rgerhards, 2007-07-17 + */ +static rsRetVal +resetConfigVariables(uchar __attribute__((unused)) *pp, void __attribute__((unused)) *pVal) +{ + cCCEscapeChar = '#'; + bEscapeCCOnRcv = 1; /* default is to escape control characters */ + bDropTrailingLF = 1; /* default is to drop trailing LF's on reception */ + + return RS_RET_OK; +} + +/* This destroys the master parserlist and all of its parser entries. MUST only be + * done when the module is shut down. Parser modules are NOT unloaded, rsyslog + * does that at a later stage for all dynamically loaded modules. + */ +static void +destroyMasterParserList(void) +{ + parserList_t *pParsLst; + parserList_t *pParsLstDel; + + pParsLst = pParsLstRoot; + while(pParsLst != NULL) { + parserDestruct(&pParsLst->pParser); + pParsLstDel = pParsLst; + pParsLst = pParsLst->pNext; + free(pParsLstDel); + } +} + +/* Exit our class. + * rgerhards, 2009-11-04 + */ +BEGINObjClassExit(parser, OBJ_IS_CORE_MODULE) /* class, version */ + DestructParserList(&pDfltParsLst); + destroyMasterParserList(); + objRelease(glbl, CORE_COMPONENT); + objRelease(errmsg, CORE_COMPONENT); + objRelease(datetime, CORE_COMPONENT); + objRelease(ruleset, CORE_COMPONENT); +ENDObjClassExit(parser) + + +/* Initialize the parser class. Must be called as the very first method + * before anything else is called inside this class. + * rgerhards, 2009-11-02 + */ +BEGINObjClassInit(parser, 1, OBJ_IS_CORE_MODULE) /* class, version */ + /* request objects we use */ + CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(datetime, CORE_COMPONENT)); + CHKiRet(objUse(ruleset, CORE_COMPONENT)); + + CHKiRet(regCfSysLineHdlr((uchar *)"controlcharacterescapeprefix", 0, eCmdHdlrGetChar, NULL, &cCCEscapeChar, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"droptrailinglfonreception", 0, eCmdHdlrBinary, NULL, &bDropTrailingLF, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"escapecontrolcharactersonreceive", 0, eCmdHdlrBinary, NULL, &bEscapeCCOnRcv, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, NULL)); + + InitParserList(&pParsLstRoot); + InitParserList(&pDfltParsLst); +ENDObjClassInit(parser) + diff --git a/runtime/parser.h b/runtime/parser.h index cec9c083..c4f63021 100644 --- a/runtime/parser.h +++ b/runtime/parser.h @@ -1,8 +1,6 @@ /* header for parser.c - * This is not yet an object, but contains all those code necessary to - * parse syslog messages. * - * Copyright 2008 Rainer Gerhards and Adiscon GmbH. + * Copyright 2008,2009 Rainer Gerhards and Adiscon GmbH. * * This file is part of the rsyslog runtime library. * @@ -21,10 +19,53 @@ * * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. */ -#ifndef INCLUDED_PARSE_H -#define INCLUDED_PARSE_H +#ifndef INCLUDED_PARSER_H +#define INCLUDED_PARSER_H -extern rsRetVal parserClassInit(void); -extern rsRetVal parseMsg(msg_t*); -#endif /* #ifndef INCLUDED_PARSE_H */ +/* we create a small helper object, a list of parsers, that we can use to + * build a chain of them whereever this is needed (initially thought to be + * used in ruleset.c as well as ourselvs). + */ +struct parserList_s { + parser_t *pParser; + parserList_t *pNext; +}; + + +/* the parser object, a dummy because we have only static methods */ +struct parser_s { + BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ + uchar *pName; /* name of this parser */ + modInfo_t *pModule; /* pointer to parser's module */ + bool bDoSanitazion; /* do standard message sanitazion before calling parser? */ + bool bDoPRIParsing; /* do standard PRI parsing before calling parser? */ +}; + +/* interfaces */ +BEGINinterface(parser) /* name must also be changed in ENDinterface macro! */ + INTERFACEObjDebugPrint(var); + rsRetVal (*Construct)(parser_t **ppThis); + rsRetVal (*ConstructFinalize)(parser_t *pThis); + rsRetVal (*Destruct)(parser_t **ppThis); + rsRetVal (*SetName)(parser_t *pThis, uchar *name); + rsRetVal (*SetModPtr)(parser_t *pThis, modInfo_t *pMod); + rsRetVal (*SetDoSanitazion)(parser_t *pThis, int); + rsRetVal (*SetDoPRIParsing)(parser_t *pThis, int); + rsRetVal (*FindParser)(parser_t **ppThis, uchar*name); + rsRetVal (*InitParserList)(parserList_t **pListRoot); + rsRetVal (*DestructParserList)(parserList_t **pListRoot); + rsRetVal (*AddParserToList)(parserList_t **pListRoot, parser_t *pParser); + /* static functions */ + rsRetVal (*ParseMsg)(msg_t *pMsg); + rsRetVal (*SanitizeMsg)(msg_t *pMsg); + rsRetVal (*AddDfltParser)(uchar *); +ENDinterface(parser) +#define parserCURR_IF_VERSION 1 /* increment whenever you change the interface above! */ + + +/* prototypes */ +PROTOTYPEObj(parser); + + +#endif /* #ifndef INCLUDED_PARSER_H */ diff --git a/runtime/prop.c b/runtime/prop.c index d188b2ed..94d1bd49 100644 --- a/runtime/prop.c +++ b/runtime/prop.c @@ -83,7 +83,7 @@ static rsRetVal SetString(prop_t *pThis, uchar *psz, int len) if(len < CONF_PROP_BUFSIZE) { memcpy(pThis->szVal.sz, psz, len + 1); } else { - CHKmalloc(pThis->szVal.psz = malloc(len + 1)); + CHKmalloc(pThis->szVal.psz = MALLOC(len + 1)); memcpy(pThis->szVal.psz, psz, len + 1); } diff --git a/runtime/queue.c b/runtime/queue.c index 101052a1..b29ec7ac 100644 --- a/runtime/queue.c +++ b/runtime/queue.c @@ -55,6 +55,9 @@ #include "wti.h" #include "msg.h" #include "atomic.h" +#include "errmsg.h" +#include "datetime.h" +#include "unicode-helper.h" #include "msg.h" /* TODO: remove once we remove MsgAddRef() call */ #ifdef OS_SOLARIS @@ -65,14 +68,15 @@ DEFobjStaticHelpers DEFobjCurrIf(glbl) DEFobjCurrIf(strm) +DEFobjCurrIf(errmsg) +DEFobjCurrIf(datetime) /* forward-definitions */ +static inline rsRetVal doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr); static rsRetVal qqueueChkPersist(qqueue_t *pThis, int nUpdates); -static rsRetVal SetEnqOnly(qqueue_t *pThis, int bEnqOnly, int bLockMutex); static rsRetVal RateLimiter(qqueue_t *pThis); static int qqueueChkStopWrkrDA(qqueue_t *pThis); static rsRetVal GetDeqBatchSize(qqueue_t *pThis, int *pVal); -static int qqueueIsIdleDA(qqueue_t *pThis); static rsRetVal ConsumerDA(qqueue_t *pThis, wti_t *pWti); static rsRetVal batchProcessed(qqueue_t *pThis, wti_t *pWti); @@ -143,7 +147,7 @@ static inline rsRetVal tdlAdd(qqueue_t *pQueue, qDeqID deqID, int nElemDeq) ISOBJ_TYPE_assert(pQueue, qqueue); assert(pQueue->toDeleteLst != NULL); - CHKmalloc(pNew = malloc(sizeof(toDeleteLst_t))); + CHKmalloc(pNew = MALLOC(sizeof(toDeleteLst_t))); pNew->deqID = deqID; pNew->nElemDeq = nElemDeq; @@ -228,7 +232,8 @@ static inline void queueDrain(qqueue_t *pThis) * this point in time. The mutex must be locked when * ths function is called. -- rgerhards, 2008-01-25 */ -static inline rsRetVal qqueueAdviseMaxWorkers(qqueue_t *pThis) +static inline rsRetVal +qqueueAdviseMaxWorkers(qqueue_t *pThis) { DEFiRet; int iMaxWorkers; @@ -236,48 +241,18 @@ static inline rsRetVal qqueueAdviseMaxWorkers(qqueue_t *pThis) ISOBJ_TYPE_assert(pThis, qqueue); if(!pThis->bEnqOnly) { - if(pThis->bRunsDA) { - /* if we have not yet reached the high water mark, there is no need to start a - * worker. -- rgerhards, 2008-01-26 - */ - if(getLogicalQueueSize(pThis) >= pThis->iHighWtrMrk || pThis->bQueueStarted == 0) { - wtpAdviseMaxWorkers(pThis->pWtpDA, 1); /* disk queues have always one worker */ - } - } - /* regular workers always run */ - if(pThis->qType == QUEUETYPE_DISK || pThis->iMinMsgsPerWrkr == 0) { - iMaxWorkers = 1; + if(pThis->bIsDA && getLogicalQueueSize(pThis) >= pThis->iHighWtrMrk) { + wtpAdviseMaxWorkers(pThis->pWtpDA, 1); /* disk queues have always one worker */ } else { - iMaxWorkers = getLogicalQueueSize(pThis) / pThis->iMinMsgsPerWrkr + 1; + if(getLogicalQueueSize(pThis) == 0) { + iMaxWorkers = 0; + } else if(pThis->qType == QUEUETYPE_DISK || pThis->iMinMsgsPerWrkr == 0) { + iMaxWorkers = 1; + } else { + iMaxWorkers = getLogicalQueueSize(pThis) / pThis->iMinMsgsPerWrkr + 1; + } + wtpAdviseMaxWorkers(pThis->pWtpReg, iMaxWorkers); } - wtpAdviseMaxWorkers(pThis->pWtpReg, iMaxWorkers); /* disk queues have always one worker */ - } - - RETiRet; -} - - -/* Destruct DA queue. This is the last part of DA-to-normal-mode - * transistion. This is called asynchronously and some time quite a - * while after the actual transistion. The key point is that we need to - * do it at some later time, because we need to destruct the DA queue. That, - * however, can not be done in a thread that has been signalled - * This is to be called when we revert back to our own queue. - * This function must be called with the queue mutex locked (the wti - * class ensures this). - * rgerhards, 2008-01-15 - */ -static rsRetVal -TurnOffDAMode(qqueue_t *pThis) -{ - DEFiRet; - - ISOBJ_TYPE_assert(pThis, qqueue); - ASSERT(pThis->bRunsDA); - if(getLogicalQueueSize(pThis->pqDA) == 0) { - pThis->bRunsDA = 0; /* tell the world we are back in non-DA mode */ - DBGOPRINT((obj_t*) pThis, "disk-assistance has been turned off, disk queue was empty (iRet %d)\n", - iRet); } RETiRet; @@ -308,15 +283,7 @@ qqueueChkIsDA(qqueue_t *pThis) } -/* Start disk-assisted queue mode. All internal settings are changed. This is supposed - * to be called from the DA worker, which must have been started before. The most important - * chore of this function is to create the DA queue object. If that function fails, - * the DA worker should return with an appropriate state, which in turn should lead to - * a re-set to non-DA mode in the Enq process. The queue mutex must be locked when this - * function is called, else a number of races will happen. - * Please note that this function may be called *while* we in DA mode. This is due to the - * fact that the DA worker calls it and the DA worker may be suspended (and restarted) due - * to inactivity timeouts. +/* Start disk-assisted queue mode. * rgerhards, 2008-01-15 */ static rsRetVal @@ -348,33 +315,22 @@ StartDA(qqueue_t *pThis) CHKiRet(qqueueSetbSyncQueueFiles(pThis->pqDA, pThis->bSyncQueueFiles)); CHKiRet(qqueueSettoActShutdown(pThis->pqDA, pThis->toActShutdown)); CHKiRet(qqueueSettoEnq(pThis->pqDA, pThis->toEnq)); - CHKiRet(SetEnqOnly(pThis->pqDA, pThis->bDAEnqOnly, MUTEX_ALREADY_LOCKED)); CHKiRet(qqueueSetiDeqtWinFromHr(pThis->pqDA, pThis->iDeqtWinFromHr)); CHKiRet(qqueueSetiDeqtWinToHr(pThis->pqDA, pThis->iDeqtWinToHr)); + CHKiRet(qqueueSettoQShutdown(pThis->pqDA, pThis->toQShutdown)); CHKiRet(qqueueSetiHighWtrMrk(pThis->pqDA, 0)); CHKiRet(qqueueSetiDiscardMrk(pThis->pqDA, 0)); - // experimental: XXX - CHKiRet(qqueueSettoWrkShutdown(pThis->pqDA, 0)); - - if(pThis->toQShutdown == 0) { - CHKiRet(qqueueSettoQShutdown(pThis->pqDA, 0)); /* if the user really wants... */ - } else { - /* we use the shortest possible shutdown (0 is endless!) because when we run on disk AND - * have an obviously large backlog, we can't finish it in any case. So there is no point - * in holding shutdown longer than necessary. -- rgerhards, 2008-01-15 - */ - CHKiRet(qqueueSettoQShutdown(pThis->pqDA, 1)); - } - iRet = qqueueStart(pThis->pqDA); /* file not found is expected, that means it is no previous QIF available */ - if(iRet != RS_RET_OK && iRet != RS_RET_FILE_NOT_FOUND) + if(iRet != RS_RET_OK && iRet != RS_RET_FILE_NOT_FOUND) { + errno = 0; /* else an errno is shown in errmsg! */ + errmsg.LogError(errno, iRet, "error starting up disk queue, using pure in-memory mode"); + pThis->bIsDA = 0; /* disable memory mode */ FINALIZE; /* something is wrong */ + } - //pthread_cond_broadcast(&pThis->condDAReady); /* signal we are now initialized and ready to go ;) */ - - DBGOPRINT((obj_t*) pThis, "is now running in disk assisted mode, disk queue 0x%lx\n", + DBGOPRINT((obj_t*) pThis, "DA queue initialized, disk queue 0x%lx\n", qqueueGetID(pThis->pqDA)); finalize_it: @@ -397,7 +353,7 @@ finalize_it: * rgerhards, 2008-01-16 */ static rsRetVal -InitDA(qqueue_t *pThis, int bEnqOnly, int bLockMutex) +InitDA(qqueue_t *pThis, int bLockMutex) { DEFiRet; DEFVARS_mutexProtection; @@ -412,91 +368,32 @@ InitDA(qqueue_t *pThis, int bEnqOnly, int bLockMutex) * rgerhards, 2008-01-24 * NOTE: this is the DA worker *pool*, not the DA queue! */ - if(pThis->pWtpDA == NULL) { - lenBuf = snprintf((char*)pszBuf, sizeof(pszBuf), "%s:DAwpool", obj.GetName((obj_t*) pThis)); - CHKiRet(wtpConstruct (&pThis->pWtpDA)); - CHKiRet(wtpSetDbgHdr (pThis->pWtpDA, pszBuf, lenBuf)); - CHKiRet(wtpSetpfChkStopWrkr (pThis->pWtpDA, (rsRetVal (*)(void *pUsr, int)) qqueueChkStopWrkrDA)); - CHKiRet(wtpSetpfGetDeqBatchSize (pThis->pWtpDA, (rsRetVal (*)(void *pUsr, int*)) GetDeqBatchSize)); - CHKiRet(wtpSetpfIsIdle (pThis->pWtpDA, (rsRetVal (*)(void *pUsr, wtp_t*)) qqueueIsIdleDA)); - CHKiRet(wtpSetpfDoWork (pThis->pWtpDA, (rsRetVal (*)(void *pUsr, void *pWti)) ConsumerDA)); - CHKiRet(wtpSetpfObjProcessed (pThis->pWtpDA, (rsRetVal (*)(void *pUsr, wti_t *pWti)) batchProcessed)); - CHKiRet(wtpSetpfOnWorkerShutdown(pThis->pWtpDA, (rsRetVal (*)(void *pUsr)) TurnOffDAMode)); - CHKiRet(wtpSetpmutUsr (pThis->pWtpDA, pThis->mut)); - CHKiRet(wtpSetpcondBusy (pThis->pWtpDA, &pThis->notEmpty)); - CHKiRet(wtpSetiNumWorkerThreads (pThis->pWtpDA, 1)); - CHKiRet(wtpSettoWrkShutdown (pThis->pWtpDA, pThis->toWrkShutdown)); - CHKiRet(wtpSetpUsr (pThis->pWtpDA, pThis)); - CHKiRet(wtpConstructFinalize (pThis->pWtpDA)); - } + lenBuf = snprintf((char*)pszBuf, sizeof(pszBuf), "%s:DAwpool", obj.GetName((obj_t*) pThis)); + CHKiRet(wtpConstruct (&pThis->pWtpDA)); + CHKiRet(wtpSetDbgHdr (pThis->pWtpDA, pszBuf, lenBuf)); + CHKiRet(wtpSetpfChkStopWrkr (pThis->pWtpDA, (rsRetVal (*)(void *pUsr, int)) qqueueChkStopWrkrDA)); + CHKiRet(wtpSetpfGetDeqBatchSize (pThis->pWtpDA, (rsRetVal (*)(void *pUsr, int*)) GetDeqBatchSize)); + CHKiRet(wtpSetpfDoWork (pThis->pWtpDA, (rsRetVal (*)(void *pUsr, void *pWti)) ConsumerDA)); + CHKiRet(wtpSetpfObjProcessed (pThis->pWtpDA, (rsRetVal (*)(void *pUsr, wti_t *pWti)) batchProcessed)); + CHKiRet(wtpSetpmutUsr (pThis->pWtpDA, pThis->mut)); + CHKiRet(wtpSetpcondBusy (pThis->pWtpDA, &pThis->notEmpty)); + CHKiRet(wtpSetiNumWorkerThreads (pThis->pWtpDA, 1)); + CHKiRet(wtpSettoWrkShutdown (pThis->pWtpDA, pThis->toWrkShutdown)); + CHKiRet(wtpSetpUsr (pThis->pWtpDA, pThis)); + CHKiRet(wtpConstructFinalize (pThis->pWtpDA)); /* if we reach this point, we have a "good" DA worker pool */ - /* indicate we now run in DA mode - this is reset by the DA worker if it fails */ - pThis->bDAEnqOnly = bEnqOnly; - /* now construct the actual queue (if it does not already exist) */ if(pThis->pqDA == NULL) { CHKiRet(StartDA(pThis)); } - pThis->bRunsDA = 1; - - /* now we must now adivse the wtp that we need one worker. If none is yet active, - * that will also start one up. If we forgot that step, everything would be stalled - * until the next enqueue request. - */ - wtpAdviseMaxWorkers(pThis->pWtpDA, 1); /* DA queues always have just one worker max */ - finalize_it: END_MTX_PROTECTED_OPERATIONS(pThis->mut); RETiRet; } -/* check if we need to start disk assisted mode and send some signals to - * keep it running if we are already in it. It also checks if DA mode is - * partially initialized, in which case it waits for initialization to - * complete. - * rgerhards, 2008-01-14 - */ -static rsRetVal -ChkStrtDA(qqueue_t *pThis) -{ - DEFiRet; - - ISOBJ_TYPE_assert(pThis, qqueue); - - /* if we do not hit the high water mark, we have nothing to do */ - if(getPhysicalQueueSize(pThis) != pThis->iHighWtrMrk) - ABORT_FINALIZE(RS_RET_OK); - - if(pThis->bRunsDA) { - /* then we need to signal that we are at the high water mark again. If that happens - * on our way down the queue, that doesn't matter, because then nobody is waiting - * on the condition variable. - * (Remember that a DA queue stops draining the queue once it has reached the low - * water mark and restarts it when the high water mark is reached again - this is - * what this code here is responsible for. Please note that all workers may have been - * terminated due to the inactivity timeout, thus we need to advise the pool that - * we need at least one). - */ - DBGOPRINT((obj_t*) pThis, "%d entries - passed high water mark in DA mode, send notify\n", - getPhysicalQueueSize(pThis)); - qqueueAdviseMaxWorkers(pThis); - } else { - /* this is the case when we are currently not running in DA mode. So it is time - * to turn it back on. - */ - DBGOPRINT((obj_t*) pThis, "%d entries - passed high water mark for disk-assisted mode, initiating...\n", - getPhysicalQueueSize(pThis)); - InitDA(pThis, QUEUE_MODE_ENQDEQ, MUTEX_ALREADY_LOCKED); /* initiate DA mode */ - } - -finalize_it: - RETiRet; -} - - /* --------------- end code for disk-assisted queue modes -------------------- */ @@ -516,7 +413,7 @@ static rsRetVal qConstructFixedArray(qqueue_t *pThis) if(pThis->iMaxQueueSize == 0) ABORT_FINALIZE(RS_RET_QSIZE_ZERO); - if((pThis->tVars.farray.pBuf = malloc(sizeof(void *) * pThis->iMaxQueueSize)) == NULL) { + if((pThis->tVars.farray.pBuf = MALLOC(sizeof(void *) * pThis->iMaxQueueSize)) == NULL) { ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } @@ -587,26 +484,6 @@ static rsRetVal qDelFixedArray(qqueue_t *pThis) } -/* reset the logical dequeue pointer to the physical dequeue position. - * This is only needed after we cancelled workers (during queue shutdown). - */ -static rsRetVal -qUnDeqAllFixedArray(qqueue_t *pThis) -{ - DEFiRet; - - ISOBJ_TYPE_assert(pThis, qqueue); - - DBGOPRINT((obj_t*) pThis, "resetting FixedArray deq index to %ld (was %ld), logical dequeue count %d\n", - pThis->tVars.farray.head, pThis->tVars.farray.deqhead, pThis->nLogDeq); - - pThis->tVars.farray.deqhead = pThis->tVars.farray.head; - pThis->nLogDeq = 0; - - RETiRet; -} - - /* -------------------- linked list -------------------- */ @@ -644,7 +521,7 @@ static rsRetVal qAddLinkedList(qqueue_t *pThis, void* pUsr) qLinkedList_t *pEntry; DEFiRet; - CHKmalloc((pEntry = (qLinkedList_t*) malloc(sizeof(qLinkedList_t)))); + CHKmalloc((pEntry = (qLinkedList_t*) MALLOC(sizeof(qLinkedList_t)))); pEntry->pNext = NULL; pEntry->pUsr = pUsr; @@ -698,26 +575,6 @@ static rsRetVal qDelLinkedList(qqueue_t *pThis) } -/* reset the logical dequeue pointer to the physical dequeue position. - * This is only needed after we cancelled workers (during queue shutdown). - */ -static rsRetVal -qUnDeqAllLinkedList(qqueue_t *pThis) -{ - DEFiRet; - - ASSERT(pThis != NULL); - - DBGOPRINT((obj_t*) pThis, "resetting LinkedList deq ptr to %p (was %p), logical dequeue count %d\n", - pThis->tVars.linklist.pDelRoot, pThis->tVars.linklist.pDeqRoot, pThis->nLogDeq); - - pThis->tVars.linklist.pDeqRoot = pThis->tVars.linklist.pDelRoot; - pThis->nLogDeq = 0; - - RETiRet; -} - - /* -------------------- disk -------------------- */ @@ -733,44 +590,6 @@ finalize_it: } -/* This method checks if we have a QIF file for the current queue (no matter of - * queue mode). Returns RS_RET_OK if we have a QIF file or an error status otherwise. - * rgerhards, 2008-01-15 - */ -static rsRetVal -qqueueHaveQIF(qqueue_t *pThis) -{ - DEFiRet; - uchar pszQIFNam[MAXFNAME]; - size_t lenQIFNam; - struct stat stat_buf; - - ISOBJ_TYPE_assert(pThis, qqueue); - - if(pThis->pszFilePrefix == NULL) - ABORT_FINALIZE(RS_RET_NO_FILEPREFIX); - - /* Construct file name */ - lenQIFNam = snprintf((char*)pszQIFNam, sizeof(pszQIFNam) / sizeof(uchar), "%s/%s.qi", - (char*) glbl.GetWorkDir(), (char*)pThis->pszFilePrefix); - - /* check if the file exists */ - if(stat((char*) pszQIFNam, &stat_buf) == -1) { - if(errno == ENOENT) { - DBGOPRINT((obj_t*) pThis, "no .qi file found\n"); - ABORT_FINALIZE(RS_RET_FILE_NOT_FOUND); - } else { - DBGOPRINT((obj_t*) pThis, "error %d trying to access .qi file\n", errno); - ABORT_FINALIZE(RS_RET_IO_ERROR); - } - } - /* If we reach this point, we have a .qi file */ - -finalize_it: - RETiRet; -} - - /* The method loads the persistent queue information. * rgerhards, 2008-01-11 */ @@ -920,9 +739,12 @@ static rsRetVal qDestructDisk(qqueue_t *pThis) ASSERT(pThis != NULL); - strm.Destruct(&pThis->tVars.disk.pWrite); - strm.Destruct(&pThis->tVars.disk.pReadDeq); - strm.Destruct(&pThis->tVars.disk.pReadDel); + if(pThis->tVars.disk.pWrite != NULL) + strm.Destruct(&pThis->tVars.disk.pWrite); + if(pThis->tVars.disk.pReadDeq != NULL) + strm.Destruct(&pThis->tVars.disk.pReadDeq); + if(pThis->tVars.disk.pReadDel != NULL) + strm.Destruct(&pThis->tVars.disk.pReadDel); RETiRet; } @@ -999,16 +821,6 @@ finalize_it: } -/* This is a dummy function for disks - we do not need to reset anything - * because everything is already persisted... - */ -static rsRetVal -qUnDeqAllDisk(__attribute__((unused)) qqueue_t *pThis) -{ - return RS_RET_OK; -} - - /* -------------------- direct (no queueing) -------------------- */ static rsRetVal qConstructDirect(qqueue_t __attribute__((unused)) *pThis) { @@ -1041,7 +853,7 @@ static rsRetVal qAddDirect(qqueue_t *pThis, void* pUsr) batchObj.pUsrp = (obj_t*) pUsr; singleBatch.nElem = 1; /* there always is only one in direct mode */ singleBatch.pElem = &batchObj; - iRet = pThis->pConsumer(pThis->pUsr, &singleBatch); + iRet = pThis->pConsumer(pThis->pUsr, &singleBatch, &pThis->bShutdownImmediate); objDestruct(pUsr); RETiRet; @@ -1053,12 +865,6 @@ static rsRetVal qDelDirect(qqueue_t __attribute__((unused)) *pThis) return RS_RET_OK; } -static rsRetVal -qUnDeqAllDirect(__attribute__((unused)) qqueue_t *pThis) -{ - return RS_RET_OK; -} - /* --------------- end type-specific handlers -------------------- */ @@ -1112,15 +918,17 @@ qqueueDeq(qqueue_t *pThis, void **ppUsr) } -/* Try to terminate queue worker threads within the regular shutdown interval. - * Both the regular and DA queue (if it exists) is waited for, but on the same timeout. - * After this function returns, the workers must either be finished or some force - * to finish them must be applied. - * This function also instructs the DA worker pool (if it exists) to terminate. This is done - * in preparation of final queue shutdown. - * rgerhards, 2009-05-27 +/* Try to shut down regular and DA queue workers, within the queue timeout + * period. That means processing continues as usual. This is the expected + * usual case, where during shutdown those messages remaining are being + * processed. At this point, it is acceptable that the queue can not be + * fully depleted, that case is handled in the next step. During this phase, + * we first shut down the main queue DA worker to prevent new data to arrive + * at the DA queue, and then we ask the regular workers of both the Regular + * and DA queue to try complete processing. + * rgerhards, 2009-10-14 */ -static rsRetVal +static inline rsRetVal tryShutdownWorkersWithinQueueTimeout(qqueue_t *pThis) { struct timespec tTimeout; @@ -1130,30 +938,26 @@ tryShutdownWorkersWithinQueueTimeout(qqueue_t *pThis) ISOBJ_TYPE_assert(pThis, qqueue); ASSERT(pThis->pqParent == NULL); /* detect invalid calling sequence */ - d_pthread_mutex_lock(pThis->mut); /* some workers may be running in parallel! */ - if(getPhysicalQueueSize(pThis) > 0) { - if(pThis->bRunsDA) { - /* We may have waited on the low water mark. As it may have changed, we - * see if we reactivate the worker. - */ - wtpAdviseMaxWorkers(pThis->pWtpDA, 1); - } + if(pThis->bIsDA) { + /* We need to lock the mutex, as otherwise we may have a race that prevents + * us from awaking the DA worker. */ + d_pthread_mutex_lock(pThis->mut); + + /* tell regular queue DA worker to stop shuffling messages to DA queue... */ + pThis->pqDA->bEnqOnly = 1; + wtpSetState(pThis->pWtpDA, wtpState_SHUTDOWN_IMMEDIATE); + wtpAdviseMaxWorkers(pThis->pWtpDA, 1); + DBGOPRINT((obj_t*) pThis, "awoke DA worker, told it to shut down.\n"); + + /* also tell the DA queue worker to shut down, so that it already knows... */ + wtpSetState(pThis->pqDA->pWtpReg, wtpState_SHUTDOWN); + wtpAdviseMaxWorkers(pThis->pqDA->pWtpReg, 1); /* awake its lone worker */ + DBGOPRINT((obj_t*) pThis, "awoke DA queue regular worker, told it to shut down when done.\n"); + + d_pthread_mutex_unlock(pThis->mut); } - d_pthread_mutex_unlock(pThis->mut); - /* Now wait for the queue's workers to shut down. Note that we run into the code even if we just found - * out there are no active workers - that doesn't matter: the wtp knows about that and so will - * return immediately. - * We do not yet care about the DA worker - that will be handled down later in the process. - * Note that we must not request shutdown right now - that may introduce a race: if the regular queue - * still runs DA assisted and the DA worker gets scheduled first, it will terminate itself (if the DA - * queue happens to be empty at that instant). Then the regular worker enqueues messages, what will lead - * to a restart of the worker. Of course, everything will continue to run, but in a bit sub-optimal way - * (from a performance point of view). So we don't do anything right now. The DA queue will continue to - * process messages and shutdown itself in any case if there is nothing to do. So we don't loose anything - * by not requesting shutdown now. - * rgerhards, 2008-01-25 - */ + /* first calculate absolute timeout - we need the absolute value here, because we need to coordinate * shutdown of both the regular and DA queue on *the same* timeout. */ @@ -1180,16 +984,6 @@ tryShutdownWorkersWithinQueueTimeout(qqueue_t *pThis) } else { DBGOPRINT((obj_t*) pThis, "DA queue worker shut down.\n"); } - /* we also instruct the DA worker pool to shutdown ASAP. If we need it for persisting - * the queue, it is restarted at a later stage. We don't care here if a timeout happens. - */ - DBGOPRINT((obj_t*) pThis, "trying shutdown of main queue DA worker pool\n"); - iRetLocal = wtpShutdownAll(pThis->pWtpDA, wtpState_SHUTDOWN_IMMEDIATE, &tTimeout); - if(iRetLocal == RS_RET_TIMED_OUT) { - DBGOPRINT((obj_t*) pThis, "shutdown timed out on main queue DA worker pool (this is OK)\n"); - } else { - DBGOPRINT((obj_t*) pThis, "main queue DA worker pool shut down on first try.\n"); - } } RETiRet; @@ -1197,11 +991,12 @@ tryShutdownWorkersWithinQueueTimeout(qqueue_t *pThis) /* Try to shut down regular and DA queue workers, within the action timeout - * period. Note that the main queue DA worker is still unaffected (and may shuffle - * data to the disk queue while we terminate the other workers). Not finishing - * processing all messages is now OK (but they may be preserved later, depending - * on bSaveOnShutdown setting). - * rgerhards, 2009-05-27 + * period. This aborts processing, but at the end of the current action, in + * a well-defined manner. During this phase, we terminate all three worker + * pools, including the regular queue DA worker if it not yet has terminated. + * Not finishing processing all messages is OK (and expected) at this stage + * (they may be preserved later, depending * on bSaveOnShutdown setting). + * rgerhards, 2009-10-14 */ static rsRetVal tryShutdownWorkersWithinActionTimeout(qqueue_t *pThis) @@ -1210,20 +1005,20 @@ tryShutdownWorkersWithinActionTimeout(qqueue_t *pThis) rsRetVal iRetLocal; DEFiRet; +RUNLOG_STR("trying to shutdown workers within Action Timeout"); ISOBJ_TYPE_assert(pThis, qqueue); ASSERT(pThis->pqParent == NULL); /* detect invalid calling sequence */ /* instruct workers to finish ASAP, even if still work exists */ - /* note that we modify bEnqOnly directly, because going through the method would - * startup some workers again. So this is OK here. -- rgerhards, 2009-05-28 - */ pThis->bEnqOnly = 1; - /* need to set this so that the DA queue begins shutdown in parallel! */ - if(pThis->pqDA != NULL) { + pThis->bShutdownImmediate = 1; + /* now DA queue */ + if(pThis->bIsDA) { pThis->pqDA->bEnqOnly = 1; - wtpSetState(pThis->pqDA->pWtpReg, wtpState_SHUTDOWN_IMMEDIATE); + pThis->pqDA->bShutdownImmediate = 1; } +// TODO: make sure we have at minimum a 10ms timeout - workers deserve a chance... /* now give the queue workers a last chance to gracefully shut down (based on action timeout setting) */ timeoutComp(&tTimeout, pThis->toActShutdown); DBGOPRINT((obj_t*) pThis, "trying immediate shutdown of regular workers (if any)\n"); @@ -1247,17 +1042,14 @@ tryShutdownWorkersWithinActionTimeout(qqueue_t *pThis) DBGOPRINT((obj_t*) pThis, "unexpected iRet state %d after trying immediate shutdown of the DA " "queue in disk save mode. Continuing, but results are unpredictable\n", iRetLocal); } - /* and now we need to check the DA worker itself (the one that shuffles data to the disk). This - * is necessary because we may be in a situation where the DA queue regular worker and the - * main queue worker stopped rather quickly. In this case, there is almost no time (and - * probably no thread switch!) between the point where we instructed the main queue DA - * worker to shutdown and this code location. In consequence, it may not even have - * noticed that it should should down, less acutally done this. So we provide it with a - * fixed 100ms timeout to try complete its work, what usually should be sufficient. - * rgerhards, 2009-10-06 + + /* and now we need to terminate the DA worker itself. We always grant it a 100ms timeout, + * which should be sufficient and usually not be required (it is expected to have finished + * long before while we were processing the queue timeout in shutdown phase 1). + * rgerhards, 2009-10-14 */ timeoutComp(&tTimeout, 100); - DBGOPRINT((obj_t*) pThis, "last try for regular shutdown of main queue DA worker pool\n"); + DBGOPRINT((obj_t*) pThis, "trying regular shutdown of main queue DA worker pool\n"); iRetLocal = wtpShutdownAll(pThis->pWtpDA, wtpState_SHUTDOWN_IMMEDIATE, &tTimeout); if(iRetLocal == RS_RET_TIMED_OUT) { DBGOPRINT((obj_t*) pThis, "shutdown timed out on main queue DA worker pool " @@ -1272,7 +1064,7 @@ tryShutdownWorkersWithinActionTimeout(qqueue_t *pThis) /* This function cancels all remaining regular workers for both the main and the DA - * queue. The main queue's DA worker pool continues to run (if it exists and is active). + * queue. * rgerhards, 2009-05-29 */ static rsRetVal @@ -1306,7 +1098,7 @@ cancelWorkers(qqueue_t *pThis) * big trouble when resetting the logical dequeue pointer. This operation can only be * done when *no* worker is running. So time for a shutdown... -- rgerhards, 2009-05-28 */ - DBGOPRINT((obj_t*) pThis, "checking to see if we need to cancel the main queue's DA worker pool\n"); + DBGOPRINT((obj_t*) pThis, "checking to see if main queue DA worker pool needs to be cancelled\n"); iRetLocal = wtpCancelAll(pThis->pWtpDA); /* returns immediately if all threads already have terminated */ } @@ -1341,13 +1133,6 @@ ShutdownWorkers(qqueue_t *pThis) DBGOPRINT((obj_t*) pThis, "initiating worker thread shutdown sequence\n"); - /* we reduce the low water mark in any case. This is not absolutely necessary, but - * it is useful because we enable DA mode at several spots below and so we do not need - * to think about the low water mark each time. - */ - pThis->iHighWtrMrk = 1; /* if we do not do this, the DA queue will not stop! */ - pThis->iLowWtrMrk = 0; - CHKiRet(tryShutdownWorkersWithinQueueTimeout(pThis)); if(getPhysicalQueueSize(pThis) > 0) { @@ -1375,7 +1160,7 @@ finalize_it: * to modify some parameters before the queue is actually started. */ rsRetVal qqueueConstruct(qqueue_t **ppThis, queueType_t qType, int iWorkerThreads, - int iMaxQueueSize, rsRetVal (*pConsumer)(void*, batch_t*)) + int iMaxQueueSize, rsRetVal (*pConsumer)(void*, batch_t*,int*)) { DEFiRet; qqueue_t *pThis; @@ -1384,9 +1169,7 @@ rsRetVal qqueueConstruct(qqueue_t **ppThis, queueType_t qType, int iWorkerThread ASSERT(pConsumer != NULL); ASSERT(iWorkerThreads >= 0); - if((pThis = (qqueue_t *)calloc(1, sizeof(qqueue_t))) == NULL) { - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } + CHKmalloc(pThis = (qqueue_t *)calloc(1, sizeof(qqueue_t))); /* we have an object, so let's fill the properties */ objConstructSetObjInfo(pThis); @@ -1397,7 +1180,7 @@ rsRetVal qqueueConstruct(qqueue_t **ppThis, queueType_t qType, int iWorkerThread pThis->iFullDlyMrk = iMaxQueueSize - (iMaxQueueSize / 100) * 3; /* default 97% */ pThis->iLightDlyMrk = iMaxQueueSize - (iMaxQueueSize / 100) * 30; /* default 70% */ - pThis->lenSpoolDir = strlen((char*)pThis->pszSpoolDir); + pThis->lenSpoolDir = ustrlen(pThis->pszSpoolDir); pThis->iMaxFileSize = 1024 * 1024; /* default is 1 MiB */ pThis->iQueueSize = 0; pThis->nLogDeq = 0; @@ -1418,7 +1201,6 @@ rsRetVal qqueueConstruct(qqueue_t **ppThis, queueType_t qType, int iWorkerThread pThis->qAdd = qAddFixedArray; pThis->qDeq = qDeqFixedArray; pThis->qDel = qDelFixedArray; - pThis->qUnDeqAll = qUnDeqAllFixedArray; break; case QUEUETYPE_LINKEDLIST: pThis->qConstruct = qConstructLinkedList; @@ -1426,7 +1208,6 @@ rsRetVal qqueueConstruct(qqueue_t **ppThis, queueType_t qType, int iWorkerThread pThis->qAdd = qAddLinkedList; pThis->qDeq = (rsRetVal (*)(qqueue_t*,void**)) qDeqLinkedList; pThis->qDel = (rsRetVal (*)(qqueue_t*)) qDelLinkedList; - pThis->qUnDeqAll = qUnDeqAllLinkedList; break; case QUEUETYPE_DISK: pThis->qConstruct = qConstructDisk; @@ -1434,7 +1215,6 @@ rsRetVal qqueueConstruct(qqueue_t **ppThis, queueType_t qType, int iWorkerThread pThis->qAdd = qAddDisk; pThis->qDeq = qDeqDisk; pThis->qDel = qDelDisk; - pThis->qUnDeqAll = qUnDeqAllDisk; /* special handling */ pThis->iNumWorkerThreads = 1; /* we need exactly one worker */ break; @@ -1443,7 +1223,6 @@ rsRetVal qqueueConstruct(qqueue_t **ppThis, queueType_t qType, int iWorkerThread pThis->qDestruct = qDestructDirect; pThis->qAdd = qAddDirect; pThis->qDel = qDelDirect; - pThis->qUnDeqAll = qUnDeqAllDirect; break; } @@ -1456,7 +1235,7 @@ finalize_it: /* This function checks if the provided message shall be discarded and does so, if needed. * In DA mode, we do not discard any messages as we assume the disk subsystem is fast enough to * provide real-time creation of spool files. - * Note: cached copies of iQueueSize and bRunsDA are provided so that no mutex locks are required. + * Note: cached copies of iQueueSize is provided so that no mutex locks are required. * The caller must have obtained them while the mutex was locked. Of course, these values may no * longer be current, but that is OK for the discard check. At worst, the message is either processed * or discarded when it should not have been. As discarding is in itself somewhat racy and erratic, @@ -1466,7 +1245,7 @@ finalize_it: * the return state! * rgerhards, 2008-01-24 */ -static int qqueueChkDiscardMsg(qqueue_t *pThis, int iQueueSize, int bRunsDA, void *pUsr) +static int qqueueChkDiscardMsg(qqueue_t *pThis, int iQueueSize, void *pUsr) { DEFiRet; rsRetVal iRetLocal; @@ -1475,7 +1254,7 @@ static int qqueueChkDiscardMsg(qqueue_t *pThis, int iQueueSize, int bRunsDA, voi ISOBJ_TYPE_assert(pThis, qqueue); ISOBJ_assert(pUsr); - if(pThis->iDiscardMrk > 0 && iQueueSize >= pThis->iDiscardMrk && bRunsDA == 0) { + if(pThis->iDiscardMrk > 0 && iQueueSize >= pThis->iDiscardMrk) { iRetLocal = objGetSeverity(pUsr, &iSeverity); if(iRetLocal == RS_RET_OK && iSeverity >= pThis->iDiscardSeverity) { DBGOPRINT((obj_t*) pThis, "queue nearly full (%d entries), discarded severity %d message\n", @@ -1519,11 +1298,7 @@ dbgprintf("delete batch from store, new sizes: log %d, phys %d\n", getLogicalQue /* remove messages from the physical queue store that are fully processed. This is - * controlled via the to-delete list. We can only delete those elements, that are - * at the current physical tail of the queue. If the batch is from another position, - * we schedule it for deletion, but actual deletion will happen at a later call - * of this function here. We always delete as much as possible, which includes - * picking up things from the to-delete list. + * controlled via the to-delete list. */ static inline rsRetVal DeleteBatchFromQStore(qqueue_t *pThis, batch_t *pBatch) @@ -1537,7 +1312,7 @@ DeleteBatchFromQStore(qqueue_t *pThis, batch_t *pBatch) pTdl = tdlPeek(pThis); /* get current head element */ if(pTdl == NULL) { /* to-delete list empty */ - DoDeleteBatchFromQStore(pThis, pBatch->nElemDeq); + DoDeleteBatchFromQStore(pThis, pBatch->nElem); } else if(pBatch->deqID == pThis->deqIDDel) { deqIDDel = pThis->deqIDDel; pTdl = tdlPeek(pThis); @@ -1547,10 +1322,12 @@ DeleteBatchFromQStore(qqueue_t *pThis, batch_t *pBatch) ++deqIDDel; pTdl = tdlPeek(pThis); } + /* old entries deleted, now delete current ones... */ + DoDeleteBatchFromQStore(pThis, pBatch->nElem); } else { /* can not delete, insert into to-delete list */ dbgprintf("not at head of to-delete list, enqueue %d\n", (int) pBatch->deqID); - CHKiRet(tdlAdd(pThis, pBatch->deqID, pBatch->nElemDeq)); + CHKiRet(tdlAdd(pThis, pBatch->deqID, pBatch->nElem)); } finalize_it: @@ -1559,7 +1336,8 @@ finalize_it: /* Delete a batch of processed user objects from the queue, which includes - * destructing the objects themself. + * destructing the objects themself. Any entries not marked as finally + * processed are enqueued again. The new enqueue is necessary because we have a * rgerhards, 2009-05-13 */ static inline rsRetVal @@ -1567,6 +1345,8 @@ DeleteProcessedBatch(qqueue_t *pThis, batch_t *pBatch) { int i; void *pUsr; + int nEnqueued = 0; + rsRetVal localRet; DEFiRet; ISOBJ_TYPE_assert(pThis, qqueue); @@ -1574,9 +1354,23 @@ DeleteProcessedBatch(qqueue_t *pThis, batch_t *pBatch) for(i = 0 ; i < pBatch->nElem ; ++i) { pUsr = pBatch->pElem[i].pUsrp; + if( pBatch->pElem[i].state == BATCH_STATE_RDY + || pBatch->pElem[i].state == BATCH_STATE_SUB) { + localRet = doEnqSingleObj(pThis, eFLOWCTL_NO_DELAY, + (obj_t*)MsgAddRef((msg_t*) pUsr)); + ++nEnqueued; + if(localRet != RS_RET_OK) { + DBGPRINTF("error %d re-enqueuing unprocessed data element - discarded\n", localRet); + } + } objDestruct(pUsr); } + dbgprintf("we deleted %d objects and enqueued %d objects\n", i-nEnqueued, nEnqueued); + + if(nEnqueued > 0) + qqueueChkPersist(pThis, nEnqueued); + iRet = DeleteBatchFromQStore(pThis, pBatch); pBatch->nElem = pBatch->nElemDeq = 0; /* reset batch */ @@ -1586,7 +1380,11 @@ DeleteProcessedBatch(qqueue_t *pThis, batch_t *pBatch) /* dequeue as many user pointers as are available, until we hit the configured - * upper limit of pointers. + * upper limit of pointers. Note that this function also deletes all processed + * objects from the previous batch. However, it is perfectly valid that the + * previous batch contained NO objects at all. For example, this happens + * immediately after system startup or when a queue was exhausted and the queue + * worker needed to wait for new data. * This must only be called when the queue mutex is LOOKED, otherwise serious * malfunction will happen. */ @@ -1606,11 +1404,10 @@ DequeueConsumableElements(qqueue_t *pThis, wti_t *pWti, int *piRemainingQueueSiz nDequeued = nDiscarded = 0; while((iQueueSize = getLogicalQueueSize(pThis)) > 0 && nDequeued < pThis->iDeqBatchSize) { -dbgprintf("DequeueConsumableElements, index %d\n", nDequeued); CHKiRet(qqueueDeq(pThis, &pUsr)); /* check if we should discard this element */ - localRet = qqueueChkDiscardMsg(pThis, pThis->iQueueSize, pThis->bRunsDA, pUsr); + localRet = qqueueChkDiscardMsg(pThis, pThis->iQueueSize, pUsr); if(localRet == RS_RET_QUEUE_FULL) { ++nDiscarded; continue; @@ -1731,7 +1528,7 @@ RateLimiter(qqueue_t *pThis) iDelay = 0; if(pThis->iDeqtWinToHr != 25) { /* 25 means disabled */ /* time calls are expensive, so only do them when needed */ - time(&tCurr); + datetime.GetTime(&tCurr); localtime_r(&tCurr, &m); iHrCurr = m.tm_hour; @@ -1775,7 +1572,8 @@ RateLimiter(qqueue_t *pThis) } -/* This dequeues the next batch. +/* This dequeues the next batch. Note that this function must not be + * cancelled, else it will leave back an inconsistent state. * rgerhards, 2009-05-20 */ static inline rsRetVal @@ -1786,7 +1584,6 @@ DequeueForConsumer(qqueue_t *pThis, wti_t *pWti) ISOBJ_TYPE_assert(pThis, qqueue); ISOBJ_TYPE_assert(pWti, wti); -dbgprintf("YYY: deqeueu for consumer"); CHKiRet(DequeueConsumable(pThis, pWti)); if(pWti->batch.nElem == 0) @@ -1809,7 +1606,6 @@ batchProcessed(qqueue_t *pThis, wti_t *pWti) ISOBJ_TYPE_assert(pThis, qqueue); ISOBJ_TYPE_assert(pWti, wti); -dbgprintf("XXX: batchProcessed deletes %d records\n", pWti->batch.nElemDeq); DeleteProcessedBatch(pThis, &pWti->batch); qqueueChkPersist(pThis, pWti->batch.nElemDeq); @@ -1825,6 +1621,7 @@ dbgprintf("XXX: batchProcessed deletes %d records\n", pWti->batch.nElemDeq); static rsRetVal ConsumerReg(qqueue_t *pThis, wti_t *pWti) { + int iCancelStateSave; DEFiRet; ISOBJ_TYPE_assert(pThis, qqueue); @@ -1835,7 +1632,10 @@ ConsumerReg(qqueue_t *pThis, wti_t *pWti) /* we now have a non-idle batch of work, so we can release the queue mutex and process it */ d_pthread_mutex_unlock(pThis->mut); - CHKiRet(pThis->pConsumer(pThis->pUsr, &pWti->batch)); + /* at this spot, we may be cancelled */ + pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, &iCancelStateSave); + + CHKiRet(pThis->pConsumer(pThis->pUsr, &pWti->batch, &pThis->bShutdownImmediate)); /* we now need to check if we should deliberately delay processing a bit * and, if so, do that. -- rgerhards, 2008-01-30 @@ -1847,11 +1647,15 @@ ConsumerReg(qqueue_t *pThis, wti_t *pWti) srSleep(pThis->iDeqSlowdown / 1000000, pThis->iDeqSlowdown % 1000000); } + /* but now cancellation is no longer permitted */ + pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &iCancelStateSave); + /* now we are done, but need to re-aquire the mutex */ d_pthread_mutex_lock(pThis->mut); finalize_it: -dbgprintf("XXX: regular consumer finished, iret=%d, szlog %d sz phys %d\n", iRet, getLogicalQueueSize(pThis), getPhysicalQueueSize(pThis)); + dbgprintf("regular consumer finished, iret=%d, szlog %d sz phys %d\n", iRet, + getLogicalQueueSize(pThis), getPhysicalQueueSize(pThis)); RETiRet; } @@ -1869,6 +1673,7 @@ static rsRetVal ConsumerDA(qqueue_t *pThis, wti_t *pWti) { int i; + int iCancelStateSave; DEFiRet; ISOBJ_TYPE_assert(pThis, qqueue); @@ -1879,15 +1684,24 @@ ConsumerDA(qqueue_t *pThis, wti_t *pWti) /* we now have a non-idle batch of work, so we can release the queue mutex and process it */ d_pthread_mutex_unlock(pThis->mut); + /* at this spot, we may be cancelled */ + pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, &iCancelStateSave); + /* iterate over returned results and enqueue them in DA queue */ - for(i = 0 ; i < pWti->batch.nElem ; i++) { + for(i = 0 ; i < pWti->batch.nElem && !pThis->bShutdownImmediate ; i++) { /* TODO: we must add a generic "addRef" mechanism, because the disk queue enqueue destructs * the message. So far, we simply assume we always have msg_t, what currently is always the case. * rgerhards, 2009-05-28 */ - CHKiRet(qqueueEnqObj(pThis->pqDA, eFLOWCTL_NO_DELAY, (obj_t*)MsgAddRef((msg_t*)(pWti->batch.pElem[i].pUsrp)))); +dbgprintf("DA consumer pushes msg '%s'\n", ((msg_t*)(pWti->batch.pElem[i].pUsrp))->pszRawMsg); + CHKiRet(qqueueEnqObj(pThis->pqDA, eFLOWCTL_NO_DELAY, + (obj_t*)MsgAddRef((msg_t*)(pWti->batch.pElem[i].pUsrp)))); + pWti->batch.pElem[i].state = BATCH_STATE_COMM; /* commited to other queue! */ } + /* but now cancellation is no longer permitted */ + pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &iCancelStateSave); + /* now we are done, but need to re-aquire the mutex */ d_pthread_mutex_lock(pThis->mut); @@ -1899,12 +1713,6 @@ finalize_it: /* must only be called when the queue mutex is locked, else results * are not stable! - * If we are a child, we have done our duty when the queue is empty. In that case, - * we can terminate. - * Version for the DA worker thread. NOTE: the pThis->bRunsDA is different from - * the DA queue. - * If our queue is in destruction, we drain to the DA queue and so we shall not terminate - * until we have done so. */ static rsRetVal qqueueChkStopWrkrDA(qqueue_t *pThis) @@ -1913,25 +1721,6 @@ qqueueChkStopWrkrDA(qqueue_t *pThis) if(pThis->bEnqOnly) { iRet = RS_RET_TERMINATE_WHEN_IDLE; - } else { - if(pThis->bRunsDA) { - ASSERT(pThis->pqDA != NULL); - if( pThis->pqDA->bEnqOnly - && pThis->pqDA->sizeOnDiskMax > 0 - && pThis->pqDA->tVars.disk.sizeOnDisk > pThis->pqDA->sizeOnDiskMax) { - /* this queue can never grow, so we can give up... */ - iRet = RS_RET_TERMINATE_NOW; - } else if(getPhysicalQueueSize(pThis) < pThis->iHighWtrMrk && pThis->bQueueStarted == 1) { -dbgprintf("XXX: terminate_NOW DA worker: queue size %d, high water mark %d\n", getPhysicalQueueSize(pThis), pThis->iHighWtrMrk); - iRet = RS_RET_TERMINATE_NOW; -RUNLOG_STR("XXX: re-start reg worker"); -qqueueAdviseMaxWorkers(pThis); -RUNLOG_STR("XXX: done re-start reg worker"); - } - } else { - // experimental iRet = RS_RET_TERMINATE_NOW; - ; - } } RETiRet; @@ -1941,9 +1730,7 @@ RUNLOG_STR("XXX: done re-start reg worker"); /* must only be called when the queue mutex is locked, else results * are not stable! * If we are a child, we have done our duty when the queue is empty. In that case, - * we can terminate. - * Version for the regular worker thread. NOTE: the pThis->bRunsDA is different from - * the DA queue + * we can terminate. Version for the regular worker thread. */ static rsRetVal ChkStopWrkrReg(qqueue_t *pThis) @@ -1968,60 +1755,12 @@ GetDeqBatchSize(qqueue_t *pThis, int *pVal) DEFiRet; assert(pVal != NULL); *pVal = pThis->iDeqBatchSize; -if(pThis->pqParent != NULL) +if(pThis->pqParent != NULL) // TODO: check why we actually do this! *pVal = 16; RETiRet; } -/* must only be called when the queue mutex is locked, else results - * are not stable! DA worker version (pThis *is* the *main* queue, not DA!) - */ -static int -qqueueIsIdleDA(qqueue_t *pThis) -{ - return(getPhysicalQueueSize(pThis) <= pThis->iLowWtrMrk); -} -/* must only be called when the queue mutex is locked, else results - * are not stable! Regular worker version. - */ -static int -IsIdleReg(qqueue_t *pThis) -{ - return(getPhysicalQueueSize(pThis) == 0); -} - - -/* This function is called when a worker thread for the regular queue is shut down. - * If we are the primary queue, this is not really interesting to us. If, however, - * we are the DA (child) queue, that means the DA queue is empty. In that case, we - * need to signal the parent queue's DA worker, so that it can terminate DA mode. - * rgerhards, 2008-01-26 - * rgerhards, 2008-02-27: HOWEVER, in a shutdown condition, it may be that the parent's worker thread pool - * has already been terminated and destructed. This *is* a legal condition and happens - * from time to time in practice. So we need to signal only if there still is a - * parent DA worker queue. Please keep in mind that the the parent's DA worker - * pool is DIFFERENT from our (DA queue) regular worker pool. So when the parent's - * pWtpDA is destructed, there can still be some of our (DAq/wtp) threads be running. - * I am telling this, because I, too, always get confused by those... - */ -static rsRetVal -RegOnWrkrShutdown(qqueue_t *pThis) -{ - DEFiRet; - - ISOBJ_TYPE_assert(pThis, qqueue); - - if(pThis->pqParent != NULL) { - if(pThis->pqParent->pWtpDA != NULL) { /* see comment in function header from 2008-02-27 */ - wtpAdviseMaxWorkers(pThis->pqParent->pWtpDA, 1); /* reactivate DA worker (always 1) */ - } - } - - RETiRet; -} - - /* start up the queue - it must have been constructed and parameters defined * before. */ @@ -2029,8 +1768,6 @@ rsRetVal qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ { DEFiRet; - rsRetVal iRetLocal; - int bInitialized = 0; /* is queue already initialized? */ uchar pszBuf[64]; size_t lenBuf; @@ -2044,7 +1781,7 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ * influenced by properties which might have been set after queueConstruct () */ if(pThis->pqParent == NULL) { - pThis->mut = (pthread_mutex_t *) malloc (sizeof (pthread_mutex_t)); + pThis->mut = (pthread_mutex_t *) MALLOC (sizeof (pthread_mutex_t)); pthread_mutex_init(pThis->mut, NULL); } else { /* child queue, we need to use parent's mutex */ @@ -2072,8 +1809,7 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ if(pThis->qType == QUEUETYPE_DIRECT) FINALIZE; /* with direct queues, we are already finished... */ - /* create worker thread pools for regular operation. The DA pool is created on an as-needed - * basis, which potentially means never under most circumstances. + /* create worker thread pools for regular and DA operation. */ lenBuf = snprintf((char*)pszBuf, sizeof(pszBuf), "%s:Reg", obj.GetName((obj_t*) pThis)); CHKiRet(wtpConstruct (&pThis->pWtpReg)); @@ -2081,10 +1817,8 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ CHKiRet(wtpSetpfRateLimiter (pThis->pWtpReg, (rsRetVal (*)(void *pUsr)) RateLimiter)); CHKiRet(wtpSetpfChkStopWrkr (pThis->pWtpReg, (rsRetVal (*)(void *pUsr, int)) ChkStopWrkrReg)); CHKiRet(wtpSetpfGetDeqBatchSize (pThis->pWtpReg, (rsRetVal (*)(void *pUsr, int*)) GetDeqBatchSize)); - CHKiRet(wtpSetpfIsIdle (pThis->pWtpReg, (rsRetVal (*)(void *pUsr, wtp_t*)) IsIdleReg)); CHKiRet(wtpSetpfDoWork (pThis->pWtpReg, (rsRetVal (*)(void *pUsr, void *pWti)) ConsumerReg)); CHKiRet(wtpSetpfObjProcessed (pThis->pWtpReg, (rsRetVal (*)(void *pUsr, wti_t *pWti)) batchProcessed)); - CHKiRet(wtpSetpfOnWorkerShutdown(pThis->pWtpReg, (rsRetVal (*)(void *pUsr)) RegOnWrkrShutdown)); CHKiRet(wtpSetpmutUsr (pThis->pWtpReg, pThis->mut)); CHKiRet(wtpSetpcondBusy (pThis->pWtpReg, &pThis->notEmpty)); CHKiRet(wtpSetiNumWorkerThreads (pThis->pWtpReg, pThis->iNumWorkerThreads)); @@ -2092,27 +1826,11 @@ qqueueStart(qqueue_t *pThis) /* this is the ConstructionFinalizer */ CHKiRet(wtpSetpUsr (pThis->pWtpReg, pThis)); CHKiRet(wtpConstructFinalize (pThis->pWtpReg)); - /* initialize worker thread instances */ - if(pThis->bIsDA) { - /* If we are disk-assisted, we need to check if there is a QIF file - * which we need to load. -- rgerhards, 2008-01-15 - */ - iRetLocal = qqueueHaveQIF(pThis); - if(iRetLocal == RS_RET_OK) { - DBGOPRINT((obj_t*) pThis, "on-disk queue present, needs to be reloaded\n"); - InitDA(pThis, QUEUE_MODE_ENQDEQ, LOCK_MUTEX); /* initiate DA mode */ - bInitialized = 1; /* we are done */ - } else { - /* TODO: use logerror? -- rgerhards, 2008-01-16 */ - DBGOPRINT((obj_t*) pThis, "error %d trying to access on-disk queue files, starting without them. " - "Some data may be lost\n", iRetLocal); - } - } + /* set up DA system if we have a disk-assisted queue */ + if(pThis->bIsDA) + InitDA(pThis, LOCK_MUTEX); /* initiate DA mode */ - if(Debug && !bInitialized) { - DBGOPRINT((obj_t*) pThis, "queue starts up without (loading) any DA disk state (this is normal for the DA " - "queue itself!)\n"); - } + DBGOPRINT((obj_t*) pThis, "queue finished initialization\n"); /* if the queue already contains data, we need to start the correct number of worker threads. This can be * the case when a disk queue has been loaded. If we did not start it here, it would never start. @@ -2165,7 +1883,8 @@ static rsRetVal qqueuePersist(qqueue_t *pThis, int bIsCheckpoint) pThis->bNeedDelQIF = 0; } /* indicate spool file needs to be deleted */ - CHKiRet(strm.SetbDeleteOnClose(pThis->tVars.disk.pReadDel, 1)); + if(pThis->tVars.disk.pReadDel != NULL) /* may be NULL if we had a startup failure! */ + CHKiRet(strm.SetbDeleteOnClose(pThis->tVars.disk.pReadDel, 1)); FINALIZE; /* nothing left to do, so be happy */ } @@ -2255,10 +1974,16 @@ DoSaveOnShutdown(qqueue_t *pThis) ISOBJ_TYPE_assert(pThis, qqueue); - InitDA(pThis, QUEUE_MODE_ENQONLY, LOCK_MUTEX); /* switch to DA mode */ -dbgprintf("after InitDA, queue log %d, phys %d\n", getLogicalQueueSize(pThis), getPhysicalQueueSize(pThis)); - /* make sure we do not timeout before we are done */ - DBGOPRINT((obj_t*) pThis, "bSaveOnShutdown configured, infinite timeout set\n"); + /* we reduce the low water mark, otherwise the DA worker would terminate when + * it is reached. + */ + DBGOPRINT((obj_t*) pThis, "bSaveOnShutdown set, restarting DA worker...\n"); + pThis->bShutdownImmediate = 0; /* would termiante the DA worker! */ + pThis->iLowWtrMrk = 0; + wtpSetState(pThis->pWtpDA, wtpState_SHUTDOWN); /* shutdown worker (only) when done (was _IMMEDIATE!) */ + wtpAdviseMaxWorkers(pThis->pWtpDA, 1); /* restart DA worker */ + + DBGOPRINT((obj_t*) pThis, "waiting for DA worker to terminate...\n"); timeoutComp(&tTimeout, QUEUE_TIMEOUT_ETERNAL); /* and run the primary queue's DA worker to drain the queue */ iRetLocal = wtpShutdownAll(pThis->pWtpDA, wtpState_SHUTDOWN, &tTimeout); @@ -2276,8 +2001,6 @@ dbgprintf("after InitDA, queue log %d, phys %d\n", getLogicalQueueSize(pThis), g /* destructor for the queue object */ BEGINobjDestruct(qqueue) /* be sure to specify the object type also in END and CODESTART macros! */ CODESTARTobjDestruct(qqueue) - pThis->bQueueInDestruction = 1; /* indicate we are in destruction (modifies some behaviour) */ - /* shut down all workers * We do not need to shutdown workers when we are in enqueue-only mode or we are a * direct queue - because in both cases we have none... ;) @@ -2286,13 +2009,6 @@ CODESTARTobjDestruct(qqueue) if(pThis->qType != QUEUETYPE_DIRECT && !pThis->bEnqOnly && pThis->pqParent == NULL) ShutdownWorkers(pThis); - /* now all workers are terminated. Messages may exist. Also, some logically dequeued - * messages may never have been processed because their worker was terminated. So - * we need to reset the logical dequeue pointer, persist the queue if configured to do - * so and then destruct everything. -- rgerhards, 2009-05-26 - */ - CHKiRet(pThis->qUnDeqAll(pThis)); - if(pThis->bIsDA && getPhysicalQueueSize(pThis) > 0 && pThis->bSaveOnShutdown) { CHKiRet(DoSaveOnShutdown(pThis)); } @@ -2371,7 +2087,7 @@ qqueueSetFilePrefix(qqueue_t *pThis, uchar *pszPrefix, size_t iLenPrefix) if(pszPrefix == NULL) /* just unset the prefix! */ ABORT_FINALIZE(RS_RET_OK); - if((pThis->pszFilePrefix = malloc(sizeof(uchar) * iLenPrefix + 1)) == NULL) + if((pThis->pszFilePrefix = MALLOC(sizeof(uchar) * iLenPrefix + 1)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); memcpy(pThis->pszFilePrefix, pszPrefix, iLenPrefix + 1); pThis->lenFilePrefix = iLenPrefix; @@ -2413,12 +2129,8 @@ doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) /* first check if we need to discard this message (which will cause CHKiRet() to exit) */ - CHKiRet(qqueueChkDiscardMsg(pThis, pThis->iQueueSize, pThis->bRunsDA, pUsr)); + CHKiRet(qqueueChkDiscardMsg(pThis, pThis->iQueueSize, pUsr)); - /* then check if we need to add an assistance disk queue */ - if(pThis->bIsDA) - CHKiRet(ChkStrtDA(pThis)); - /* handle flow control * There are two different flow control mechanisms: basic and advanced flow control. * Basic flow control has always been implemented and protects the queue structures @@ -2462,6 +2174,7 @@ doEnqSingleObj(qqueue_t *pThis, flowControl_t flowCtlType, void *pUsr) && pThis->tVars.disk.sizeOnDisk > pThis->sizeOnDiskMax)) { DBGOPRINT((obj_t*) pThis, "enqueueMsg: queue FULL - waiting to drain.\n"); timeoutComp(&t, pThis->toEnq); +// TODO : handle enqOnly => discard! if(pthread_cond_timedwait(&pThis->notFull, pThis->mut, &t) != 0) { DBGOPRINT((obj_t*) pThis, "enqueueMsg: cond timeout, dropping message!\n"); objDestruct(pUsr); @@ -2500,7 +2213,6 @@ qqueueMultiEnqObj(qqueue_t *pThis, multi_submit_t *pMultiSub) } for(i = 0 ; i < pMultiSub->nElem ; ++i) { -dbgprintf("queueMultiEnq: %d\n", i); CHKiRet(doEnqSingleObj(pThis, pMultiSub->ppMsgs[i]->flowCtlType, (void*)pMultiSub->ppMsgs[i])); } @@ -2544,7 +2256,6 @@ finalize_it: if(pThis->qType != QUEUETYPE_DIRECT) { /* make sure at least one worker is running. */ qqueueAdviseMaxWorkers(pThis); -dbgprintf("YYY: call advise with mutex %p locked \n", pThis->mut); /* and release the mutex */ d_pthread_mutex_unlock(pThis->mut); pthread_setcancelstate(iCancelStateSave, NULL); @@ -2555,58 +2266,6 @@ dbgprintf("YYY: call advise with mutex %p locked \n", pThis->mut); } -/* set queue mode to enqueue only or not - * There is one subtle issue: this method may be called during queue - * construction or while it is running. In the former case, the queue - * mutex does not yet exist (it is NULL), while in the later case it - * must be locked. The function detects the state and operates as - * required. - * rgerhards, 2008-01-16 - */ -static rsRetVal -SetEnqOnly(qqueue_t *pThis, int bEnqOnly, int bLockMutex) -{ - DEFiRet; - DEFVARS_mutexProtection; - - ISOBJ_TYPE_assert(pThis, qqueue); - - /* for simplicity, we do one big mutex lock. This method is extremely seldom - * called, so that doesn't matter... -- rgerhards, 2008-01-16 - */ - if(pThis->mut != NULL) { - BEGIN_MTX_PROTECTED_OPERATIONS(pThis->mut, bLockMutex); - } - - if(bEnqOnly == pThis->bEnqOnly) - FINALIZE; /* no change, nothing to do */ - - if(pThis->bQueueStarted) { - /* we need to adjust queue operation only if we are not during initial param setup */ - if(bEnqOnly == 1) { - /* switch to enqueue-only mode */ - /* this means we need to terminate all workers - that's it... */ - DBGOPRINT((obj_t*) pThis, "switching to enqueue-only mode, terminating all worker threads\n"); - if(pThis->pWtpReg != NULL) - wtpWakeupAllWrkr(pThis->pWtpReg); - if(pThis->pWtpDA != NULL) - wtpWakeupAllWrkr(pThis->pWtpDA); - } else { - /* switch back to regular mode */ - ABORT_FINALIZE(RS_RET_NOT_IMPLEMENTED); /* we don't need this so far... */ - } - } - - pThis->bEnqOnly = bEnqOnly; - -finalize_it: - if(pThis->mut != NULL) { - END_MTX_PROTECTED_OPERATIONS(pThis->mut); - } - RETiRet; -} - - /* some simple object access methods */ DEFpropSetMeth(qqueue, bSyncQueueFiles, int) DEFpropSetMeth(qqueue, iPersistUpdCnt, int) @@ -2670,6 +2329,8 @@ BEGINObjClassInit(qqueue, 1, OBJ_IS_CORE_MODULE) /* request objects we use */ CHKiRet(objUse(glbl, CORE_COMPONENT)); CHKiRet(objUse(strm, CORE_COMPONENT)); + CHKiRet(objUse(datetime, CORE_COMPONENT)); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); /* now set our own handlers */ OBJSetMethodHandler(objMethod_SETPROPERTY, qqueueSetProperty); diff --git a/runtime/queue.h b/runtime/queue.h index 73c62b52..93573dae 100644 --- a/runtime/queue.h +++ b/runtime/queue.h @@ -55,14 +55,14 @@ typedef struct qLinkedList_S { /* the queue object */ -typedef struct queue_s { +struct queue_s { BEGINobjInstance; queueType_t qType; int nLogDeq; /* number of elements currently logically dequeued */ + int bShutdownImmediate; /* should all workers cease processing messages? */ bool bEnqOnly; /* does queue run in enqueue-only mode (1) or not (0)? */ bool bSaveOnShutdown;/* persists everthing on shutdown (if DA!)? 1-yes, 0-no */ bool bQueueStarted; /* has queueStart() been called on this queue? 1-yes, 0-no */ - bool bQueueInDestruction;/* 1 if queue is in destruction process, 0 otherwise */ int iQueueSize; /* Current number of elements in the queue */ int iMaxQueueSize; /* how large can the queue grow? */ int iNumWorkerThreads;/* number of worker threads to use */ @@ -101,10 +101,11 @@ typedef struct queue_s { * the user really wanted...). -- rgerhards, 2008-04-02 */ /* end dequeue time window */ - rsRetVal (*pConsumer)(void *,batch_t*); /* user-supplied consumer function for dequeued messages */ + rsRetVal (*pConsumer)(void *,batch_t*,int*); /* user-supplied consumer function for dequeued messages */ /* calling interface for pConsumer: arg1 is the global user pointer from this structure, arg2 is the * user pointer array that was dequeued (actual sample: for actions, arg1 is the pAction and arg2 - * is pointer to an array of message message pointers) + * is pointer to an array of message message pointers), arg3 is a pointer to an interger which is zero + * during normal operations and one if the consumer must urgently shut down. */ /* type-specific handlers (set during construction) */ rsRetVal (*qConstruct)(struct queue_s *pThis); @@ -112,7 +113,6 @@ typedef struct queue_s { rsRetVal (*qAdd)(struct queue_s *pThis, void *pUsr); rsRetVal (*qDeq)(struct queue_s *pThis, void **ppUsr); rsRetVal (*qDel)(struct queue_s *pThis); - rsRetVal (*qUnDeqAll)(struct queue_s *pThis); /* end type-specific handler */ /* synchronization variables */ pthread_mutex_t mutThrdMgmt; /* mutex for the queue's thread management */ @@ -138,7 +138,6 @@ typedef struct queue_s { qDeqID deqIDAdd; /* next dequeue ID to use during add to queue store */ qDeqID deqIDDel; /* queue store delete position */ int bIsDA; /* is this queue disk assisted? */ - int bRunsDA; /* is this queue actually *running* disk assisted? */ struct queue_s *pqDA; /* queue for disk-assisted modes */ struct queue_s *pqParent;/* pointer to the parent (if this is a child queue) */ int bDAEnqOnly; /* EnqOnly setting for DA queue */ @@ -161,14 +160,8 @@ typedef struct queue_s { strm_t *pReadDel; /* current file for deleting */ } disk; } tVars; -} qqueue_t; - -/* some symbolic constants for easier reference */ -#define QUEUE_MODE_ENQDEQ 0 -#define QUEUE_MODE_ENQONLY 1 +}; -#define QUEUE_IDX_DA_WORKER 0 /* index for the DA worker (fixed) */ -#define QUEUE_PTR_DA_WORKER(x) (&((pThis)->pWrkThrds[0])) /* the define below is an "eternal" timeout for the timeout settings which require a value. * It is one day, which is not really eternal, but comes close to it if we think about @@ -185,7 +178,7 @@ rsRetVal qqueueStart(qqueue_t *pThis); rsRetVal qqueueSetMaxFileSize(qqueue_t *pThis, size_t iMaxFileSize); rsRetVal qqueueSetFilePrefix(qqueue_t *pThis, uchar *pszPrefix, size_t iLenPrefix); rsRetVal qqueueConstruct(qqueue_t **ppThis, queueType_t qType, int iWorkerThreads, - int iMaxQueueSize, rsRetVal (*pConsumer)(void*,batch_t*)); + int iMaxQueueSize, rsRetVal (*pConsumer)(void*,batch_t*, int*)); PROTOTYPEObjClassInit(qqueue); PROTOTYPEpropSetMeth(qqueue, iPersistUpdCnt, int); PROTOTYPEpropSetMeth(qqueue, bSyncQueueFiles, int); diff --git a/runtime/rsyslog.c b/runtime/rsyslog.c index 443d0f41..a76ae25a 100644 --- a/runtime/rsyslog.c +++ b/runtime/rsyslog.c @@ -80,6 +80,7 @@ #include "prop.h" #include "rule.h" #include "ruleset.h" +#include "parser.h" /* forward definitions */ static rsRetVal dfltErrLogger(int, uchar *errMsg); @@ -151,8 +152,6 @@ rsrtInit(char **ppErrObj, obj_if_t *pObjIF) CHKiRet(propClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "glbl"; CHKiRet(glblClassInit(NULL)); - if(ppErrObj != NULL) *ppErrObj = "datetime"; - CHKiRet(datetimeClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "msg"; CHKiRet(msgClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "ctok_token"; @@ -183,6 +182,8 @@ rsrtInit(char **ppErrObj, obj_if_t *pObjIF) CHKiRet(qqueueClassInit(NULL)); if(ppErrObj != NULL) *ppErrObj = "conf"; CHKiRet(confClassInit(NULL)); + if(ppErrObj != NULL) *ppErrObj = "parser"; + CHKiRet(parserClassInit(NULL)); /* dummy "classes" */ if(ppErrObj != NULL) *ppErrObj = "str"; diff --git a/runtime/rsyslog.h b/runtime/rsyslog.h index 59e8458b..a5e8cf5d 100644 --- a/runtime/rsyslog.h +++ b/runtime/rsyslog.h @@ -40,6 +40,17 @@ #define CONF_TAG_BUFSIZE 32 #define CONF_HOSTNAME_BUFSIZE 32 #define CONF_PROP_BUFSIZE 16 /* should be close to sizeof(ptr) or lighly above it */ +#define CONF_MIN_SIZE_FOR_COMPRESS 60 /* config param: minimum message size to try compression. The smaller + * the message, the less likely is any compression gain. We check for + * gain before we submit the message. But to do so we still need to + * do the (costly) compress() call. The following setting sets a size + * for which no call to compress() is done at all. This may result in + * a few more bytes being transmited but better overall performance. + * Note: I have not yet checked the minimum UDP packet size. It might be + * that we do not save anything by compressing very small messages, because + * UDP might need to pad ;) + * rgerhards, 2006-11-30 + */ /* ############################################################# * @@ -107,6 +118,7 @@ typedef struct wti_s wti_t; typedef obj_t nsd_t; typedef obj_t nsdsel_t; typedef struct msg msg_t; +typedef struct queue_s qqueue_t; typedef struct prop_s prop_t; typedef struct interface_s interface_t; typedef struct objInfo_s objInfo_t; @@ -122,6 +134,9 @@ typedef struct vmstk_s vmstk_t; typedef struct batch_obj_s batch_obj_t; typedef struct batch_s batch_t; typedef struct wtp_s wtp_t; +typedef struct modInfo_s modInfo_t; +typedef struct parser_s parser_t; +typedef struct parserList_s parserList_t; typedef rsRetVal (*prsf_t)(struct vmstk_s*, int); /* pointer to a RainerScript function */ typedef uint64 qDeqID; /* queue Dequeue order ID. 32 bits is considered dangerously few */ @@ -386,6 +401,14 @@ enum rsRetVal_ /** return value. All methods return this if not specified oth RS_RET_NO_SRCNAME_TPL = -2150, /**< sourcename template was not specified where one was needed (omudpspoof spoof addr) */ RS_RET_HOST_NOT_SPECIFIED = -2151, /**< (target) host was not specified where it was needed */ RS_RET_ERR_LIBNET_INIT = -2152, /**< error initializing libnet */ + RS_RET_FORCE_TERM = -2153, /**< thread was forced to terminate by bShallShutdown, a state, not an error */ + RS_RET_RULES_QUEUE_EXISTS = -2154,/**< we were instructed to create a new ruleset queue, but one already exists */ + RS_RET_NO_CURR_RULESET = -2155,/**< no current ruleset exists (but one is required) */ + RS_RET_NO_MSG_PASSING = -2156,/**< output module interface parameter passing mode "MSG" is not available but required */ + RS_RET_RULESET_NOT_FOUND = -2157,/**< a required ruleset could not be found */ + RS_RET_NO_RULESET= -2158,/**< no ruleset name as specified where one was needed */ + RS_RET_PARSER_NOT_FOUND = -2159,/**< parser with the specified name was not found */ + RS_RET_COULD_NOT_PARSE = -2160,/**< (this) parser could not parse the message (no error, means try next one) */ /* RainerScript error messages (range 1000.. 1999) */ RS_RET_SYSVAR_NOT_FOUND = 1001, /**< system variable could not be found (maybe misspelled) */ diff --git a/runtime/rule.c b/runtime/rule.c index 182d616a..ae0da0d6 100644 --- a/runtime/rule.c +++ b/runtime/rule.c @@ -40,7 +40,6 @@ #include "var.h" #include "srUtils.h" #include "unicode-helper.h" -#include "dirty.h" /* for getFIOPName */ /* static data */ DEFobjStaticHelpers @@ -49,6 +48,35 @@ DEFobjCurrIf(expr) DEFobjCurrIf(var) DEFobjCurrIf(vm) + +/* support for simple textual representation of FIOP names + * rgerhards, 2005-09-27 + */ +static char* +getFIOPName(unsigned iFIOP) +{ + char *pRet; + switch(iFIOP) { + case FIOP_CONTAINS: + pRet = "contains"; + break; + case FIOP_ISEQUAL: + pRet = "isequal"; + break; + case FIOP_STARTSWITH: + pRet = "startswith"; + break; + case FIOP_REGEX: + pRet = "regex"; + break; + default: + pRet = "NOP"; + break; + } + return pRet; +} + + /* iterate over all actions, this is often needed, for example when HUP processing * must be done or a shutdown is pending. */ @@ -138,6 +166,7 @@ shouldProcessThisMessage(rule_t *pRule, msg_t *pMsg, int *bProcessMsg) } } +RUNLOG_VAR("%p", pRule->pCSProgNameComp); if(pRule->pCSProgNameComp != NULL) { int bInv = 0, bEqv = 0, offset = 0; if(*(rsCStrGetSzStrNoNULL(pRule->pCSProgNameComp)) == '-') { diff --git a/runtime/ruleset.c b/runtime/ruleset.c index 5ac9a8fd..1a77be2b 100644 --- a/runtime/ruleset.c +++ b/runtime/ruleset.c @@ -40,22 +40,24 @@ #include "rsyslog.h" #include "obj.h" +#include "cfsysline.h" #include "msg.h" #include "ruleset.h" #include "rule.h" #include "errmsg.h" +#include "parser.h" #include "unicode-helper.h" - -static rsRetVal debugPrintAll(void); // TODO: remove! +#include "dirty.h" /* for main ruleset queue creation */ /* static data */ DEFobjStaticHelpers DEFobjCurrIf(errmsg) DEFobjCurrIf(rule) +DEFobjCurrIf(parser) linkedList_t llRulesets; /* this is NOT a pointer - no typo here ;) */ ruleset_t *pCurrRuleset = NULL; /* currently "active" ruleset */ -ruleset_t *pDfltRuleset = NULL; /* currentl default ruleset, e.g. for binding to actions which have no other */ +ruleset_t *pDfltRuleset = NULL; /* current default ruleset, e.g. for binding to actions which have no other */ /* ---------- linked-list key handling functions ---------- */ @@ -141,6 +143,7 @@ DEFFUNC_llExecFunc(processMsgDoRules) rsRetVal iRet; ISOBJ_TYPE_assert(pData, rule); iRet = rule.ProcessMsg((rule_t*) pData, (msg_t*) pParam); +dbgprintf("ruleset: get iRet %d from rule.ProcessMsg()\n", iRet); return iRet; } @@ -161,13 +164,22 @@ processMsg(msg_t *pMsg) CHKiRet(llExecFunc(&pThis->llRules, processMsgDoRules, pMsg)); finalize_it: +dbgprintf("ruleset.ProcessMsg() returns %d\n", iRet); + RETiRet; +} - //if(iRet == RS_RET_DISCARDMSG) - //iRet = RS_RET_OK; - RETiRet; +/* return the ruleset-assigned parser list. NULL means use the default + * parser list. + * rgerhards, 2009-11-04 + */ +static parserList_t* +GetParserList(msg_t *pMsg) +{ + return (pMsg->pRuleset == NULL) ? pDfltRuleset->pParserLst : pMsg->pRuleset->pParserLst; } + /* Add a new rule to the end of the current rule set. We do a number * of checks and ignore the rule if it does not pass them. */ @@ -217,6 +229,19 @@ GetCurrent(void) } +/* get main queue associated with ruleset. If no ruleset-specifc main queue + * is set, the primary main message queue is returned. + * We use a non-standard calling interface, as nothing can go wrong and it + * is really much more natural to return the pointer directly. + */ +static qqueue_t* +GetRulesetQueue(ruleset_t *pThis) +{ + ISOBJ_TYPE_assert(pThis, ruleset); + return (pThis->pQueue == NULL) ? pMsgQueue : pThis->pQueue; +} + + /* Find the ruleset with the given name and return a pointer to its object. */ static rsRetVal @@ -322,6 +347,12 @@ finalize_it: BEGINobjDestruct(ruleset) /* be sure to specify the object type also in END and CODESTART macros! */ CODESTARTobjDestruct(ruleset) dbgprintf("destructing ruleset %p, name %p\n", pThis, pThis->pszName); + if(pThis->pQueue != NULL) { + qqueueDestruct(&pThis->pQueue); + } + if(pThis->pParserLst != NULL) { + parser.DestructParserList(&pThis->pParserLst); + } llDestroy(&pThis->llRules); free(pThis->pszName); ENDobjDestruct(ruleset) @@ -387,6 +418,81 @@ debugPrintAll(void) } +/* Create a ruleset-specific "main" queue for this ruleset. If one is already + * defined, an error message is emitted but nothing else is done. + * Note: we use the main message queue parameters for queue creation and access + * syslogd.c directly to obtain these. This is far from being perfect, but + * considered acceptable for the time being. + * rgerhards, 2009-10-27 + */ +static rsRetVal +rulesetCreateQueue(void __attribute__((unused)) *pVal, int *pNewVal) +{ + DEFiRet; + + if(pCurrRuleset == NULL) { + errmsg.LogError(0, RS_RET_NO_CURR_RULESET, "error: currently no specific ruleset specified, thus a " + "queue can not be added to it"); + ABORT_FINALIZE(RS_RET_NO_CURR_RULESET); + } + + if(pCurrRuleset->pQueue != NULL) { + errmsg.LogError(0, RS_RET_RULES_QUEUE_EXISTS, "error: ruleset already has a main queue, can not " + "add another one"); + ABORT_FINALIZE(RS_RET_RULES_QUEUE_EXISTS); + } + + if(pNewVal == 0) + FINALIZE; /* if it is turned off, we do not need to change anything ;) */ + + dbgprintf("adding a ruleset-specific \"main\" queue"); + CHKiRet(createMainQueue(&pCurrRuleset->pQueue, UCHAR_CONSTANT("ruleset"))); + +finalize_it: + RETiRet; +} + + +/* Add a ruleset specific parser to the ruleset. Note that adding the first + * parser automatically disables the default parsers. If they are needed as well, + * the must be added via explicit config directives. + * Note: this is the only spot in the code that requires the parser object. In order + * to solve some class init bootstrap sequence problems, we get the object handle here + * instead of during module initialization. Note that objUse() is capable of being + * called multiple times. + * rgerhards, 2009-11-04 + */ +static rsRetVal +rulesetAddParser(void __attribute__((unused)) *pVal, uchar *pName) +{ + parser_t *pParser; + DEFiRet; + + assert(pCurrRuleset != NULL); + + CHKiRet(objUse(parser, CORE_COMPONENT)); + iRet = parser.FindParser(&pParser, pName); + if(iRet == RS_RET_PARSER_NOT_FOUND) { + errmsg.LogError(0, RS_RET_PARSER_NOT_FOUND, "error: parser '%s' unknown at this time " + "(maybe defined too late in rsyslog.conf?)", pName); + ABORT_FINALIZE(RS_RET_NO_CURR_RULESET); + } else if(iRet != RS_RET_OK) { + errmsg.LogError(0, iRet, "error trying to find parser '%s'\n", pName); + FINALIZE; + } + + CHKiRet(parser.AddParserToList(&pCurrRuleset->pParserLst, pParser)); + + dbgprintf("added parser '%s' to ruleset '%s'\n", pName, pCurrRuleset->pszName); +RUNLOG_VAR("%p", pCurrRuleset->pParserLst); + +finalize_it: + d_free(pName); /* no longer needed */ + + RETiRet; +} + + /* queryInterface function * rgerhards, 2008-02-21 */ @@ -416,6 +522,8 @@ CODESTARTobjQueryInterface(ruleset) pIf->GetRuleset = GetRuleset; pIf->SetDefaultRuleset = SetDefaultRuleset; pIf->SetCurrRuleset = SetCurrRuleset; + pIf->GetRulesetQueue = GetRulesetQueue; + pIf->GetParserList = GetParserList; finalize_it: ENDobjQueryInterface(ruleset) @@ -427,6 +535,7 @@ BEGINObjClassExit(ruleset, OBJ_IS_CORE_MODULE) /* class, version */ llDestroy(&llRulesets); objRelease(errmsg, CORE_COMPONENT); objRelease(rule, CORE_COMPONENT); + objRelease(parser, CORE_COMPONENT); ENDObjClassExit(ruleset) @@ -445,6 +554,10 @@ BEGINObjClassInit(ruleset, 1, OBJ_IS_CORE_MODULE) /* class, version */ /* prepare global data */ CHKiRet(llInit(&llRulesets, rulesetDestructForLinkedList, keyDestruct, strcasecmp)); + + /* config file handlers */ + CHKiRet(regCfSysLineHdlr((uchar *)"rulesetparser", 0, eCmdHdlrGetWord, rulesetAddParser, NULL, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"rulesetcreatemainqueue", 0, eCmdHdlrBinary, rulesetCreateQueue, NULL, NULL)); ENDObjClassInit(ruleset) /* vi:set ai: diff --git a/runtime/ruleset.h b/runtime/ruleset.h index 32571687..222d773e 100644 --- a/runtime/ruleset.h +++ b/runtime/ruleset.h @@ -25,6 +25,7 @@ #ifndef INCLUDED_RULESET_H #define INCLUDED_RULESET_H +#include "queue.h" #include "linkedlist.h" /* the ruleset object */ @@ -32,6 +33,8 @@ struct ruleset_s { BEGINobjInstance; /* Data to implement generic object - MUST be the first data element! */ linkedList_t llRules; /* this is NOT a pointer - no typo here ;) */ uchar *pszName; /* name of our ruleset */ + qqueue_t *pQueue; /* "main" message queue, if the ruleset has its own (else NULL) */ + parserList_t *pParserLst;/* list of parsers to use for this ruleset */ }; /* interfaces */ @@ -50,8 +53,11 @@ BEGINinterface(ruleset) /* name must also be changed in ENDinterface macro! */ rsRetVal (*SetDefaultRuleset)(uchar*); rsRetVal (*SetCurrRuleset)(uchar*); ruleset_t* (*GetCurrent)(void); + qqueue_t* (*GetRulesetQueue)(ruleset_t*); + /* v3, 2009-11-04 */ + parserList_t* (*GetParserList)(msg_t *); ENDinterface(ruleset) -#define rulesetCURR_IF_VERSION 1 /* increment whenever you change the interface structure! */ +#define rulesetCURR_IF_VERSION 3 /* increment whenever you change the interface structure! */ /* prototypes */ diff --git a/runtime/srutils.c b/runtime/srutils.c index c403b312..7ddc3ba2 100644 --- a/runtime/srutils.c +++ b/runtime/srutils.c @@ -158,7 +158,7 @@ uchar *srUtilStrDup(uchar *pOld, size_t len) assert(pOld != NULL); - if((pNew = malloc(len + 1)) != NULL) + if((pNew = MALLOC(len + 1)) != NULL) memcpy(pNew, pOld, len + 1); return pNew; @@ -183,7 +183,7 @@ int makeFileParentDirs(uchar *szFile, size_t lenFile, mode_t mode, assert(lenFile > 0); len = lenFile + 1; /* add one for '\0'-byte */ - if((pszWork = malloc(sizeof(uchar) * len)) == NULL) + if((pszWork = MALLOC(sizeof(uchar) * len)) == NULL) return -1; memcpy(pszWork, szFile, len); for(p = pszWork+1 ; *p ; p++) @@ -326,7 +326,7 @@ rsRetVal genFileName(uchar **ppName, uchar *pDirName, size_t lenDirName, uchar * } lenName = lenDirName + 1 + lenFName + lenBuf + 1; /* last +1 for \0 char! */ - if((pName = malloc(sizeof(uchar) * lenName)) == NULL) + if((pName = MALLOC(sizeof(uchar) * lenName)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); /* got memory, now construct string */ diff --git a/runtime/stream.c b/runtime/stream.c index 58f16cce..3673db91 100644 --- a/runtime/stream.c +++ b/runtime/stream.c @@ -211,6 +211,7 @@ doPhysOpen(strm_t *pThis) } pThis->fd = open((char*)pThis->pszCurrFName, iFlags, pThis->tOpenMode); + DBGPRINTF("file '%s' opened as #%d with mode %d\n", pThis->pszCurrFName, pThis->fd, pThis->tOpenMode); if(pThis->fd == -1) { int ierrnoSave = errno; dbgoprint((obj_t*) pThis, "open error %d, file '%s'\n", errno, pThis->pszCurrFName); @@ -609,7 +610,7 @@ static rsRetVal strmConstructFinalize(strm_t *pThis) * to make sure we can write out everything with a SINGLE api call! * We add another 128 bytes to take care of the gzip header and "all eventualities". */ - CHKmalloc(pThis->pZipBuf = (Bytef*) malloc(sizeof(uchar) * pThis->sIOBufSize + 128)); + CHKmalloc(pThis->pZipBuf = (Bytef*) MALLOC(sizeof(uchar) * pThis->sIOBufSize + 128)); } } @@ -638,7 +639,7 @@ static rsRetVal strmConstructFinalize(strm_t *pThis) pthread_cond_init(&pThis->isEmpty, 0); pThis->iCnt = pThis->iEnq = pThis->iDeq = 0; for(i = 0 ; i < STREAM_ASYNC_NUMBUFS ; ++i) { - CHKmalloc(pThis->asyncBuf[i].pBuf = (uchar*) malloc(sizeof(uchar) * pThis->sIOBufSize)); + CHKmalloc(pThis->asyncBuf[i].pBuf = (uchar*) MALLOC(sizeof(uchar) * pThis->sIOBufSize)); } pThis->pIOBuf = pThis->asyncBuf[0].pBuf; pThis->bStopWriter = 0; @@ -646,7 +647,7 @@ static rsRetVal strmConstructFinalize(strm_t *pThis) DBGPRINTF("ERROR: stream %p cold not create writer thread\n", pThis); } else { /* we work synchronously, so we need to alloc a fixed pIOBuf */ - CHKmalloc(pThis->pIOBuf = (uchar*) malloc(sizeof(uchar) * pThis->sIOBufSize)); + CHKmalloc(pThis->pIOBuf = (uchar*) MALLOC(sizeof(uchar) * pThis->sIOBufSize)); } finalize_it: @@ -1322,7 +1323,7 @@ strmSetFName(strm_t *pThis, uchar *pszName, size_t iLenName) if(pThis->pszFName != NULL) free(pThis->pszFName); - if((pThis->pszFName = malloc(sizeof(uchar) * (iLenName + 1))) == NULL) + if((pThis->pszFName = MALLOC(sizeof(uchar) * (iLenName + 1))) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); memcpy(pThis->pszFName, pszName, iLenName + 1); /* always think about the \0! */ @@ -1349,7 +1350,7 @@ strmSetDir(strm_t *pThis, uchar *pszDir, size_t iLenDir) if(iLenDir < 1) ABORT_FINALIZE(RS_RET_FILE_PREFIX_MISSING); - if((pThis->pszDir = malloc(sizeof(uchar) * iLenDir + 1)) == NULL) + if((pThis->pszDir = MALLOC(sizeof(uchar) * iLenDir + 1)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); memcpy(pThis->pszDir, pszDir, iLenDir + 1); /* always think about the \0! */ diff --git a/runtime/stringbuf.c b/runtime/stringbuf.c index 93995b38..ccf115c1 100644 --- a/runtime/stringbuf.c +++ b/runtime/stringbuf.c @@ -90,7 +90,7 @@ rsRetVal rsCStrConstructFromszStr(cstr_t **ppThis, uchar *sz) CHKiRet(rsCStrConstruct(&pThis)); pThis->iBufSize = pThis->iStrLen = strlen((char *) sz); - if((pThis->pBuf = (uchar*) malloc(sizeof(uchar) * pThis->iStrLen)) == NULL) { + if((pThis->pBuf = (uchar*) MALLOC(sizeof(uchar) * pThis->iStrLen)) == NULL) { RSFREEOBJ(pThis); ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } @@ -119,7 +119,7 @@ rsRetVal rsCStrConstructFromCStr(cstr_t **ppThis, cstr_t *pFrom) CHKiRet(rsCStrConstruct(&pThis)); pThis->iBufSize = pThis->iStrLen = pFrom->iStrLen; - if((pThis->pBuf = (uchar*) malloc(sizeof(uchar) * pThis->iStrLen)) == NULL) { + if((pThis->pBuf = (uchar*) MALLOC(sizeof(uchar) * pThis->iStrLen)) == NULL) { RSFREEOBJ(pThis); ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); } @@ -267,7 +267,7 @@ rsRetVal rsCStrSetSzStr(cstr_t *pThis, uchar *pszNew) pThis->pszBuf = NULL; /* now save the new value */ - if((pThis->pBuf = (uchar*) malloc(sizeof(uchar) * pThis->iStrLen)) == NULL) { + if((pThis->pBuf = (uchar*) MALLOC(sizeof(uchar) * pThis->iStrLen)) == NULL) { RSFREEOBJ(pThis); return RS_RET_OUT_OF_MEMORY; } @@ -315,7 +315,7 @@ uchar* rsCStrGetSzStr(cstr_t *pThis) if(pThis->pBuf != NULL) if(pThis->pszBuf == NULL) { /* we do not yet have a usable sz version - so create it... */ - if((pThis->pszBuf = malloc((pThis->iStrLen + 1) * sizeof(uchar))) == NULL) { + if((pThis->pszBuf = MALLOC((pThis->iStrLen + 1) * sizeof(uchar))) == NULL) { /* TODO: think about what to do - so far, I have no bright * idea... rgerhards 2005-09-07 */ @@ -369,7 +369,7 @@ rsRetVal cstrConvSzStrAndDestruct(cstr_t *pThis, uchar **ppSz, int bRetNULL) if(pThis->pBuf == NULL) { if(bRetNULL == 0) { - CHKmalloc(pRetBuf = malloc(sizeof(uchar))); + CHKmalloc(pRetBuf = MALLOC(sizeof(uchar))); *pRetBuf = '\0'; } else { pRetBuf = NULL; diff --git a/runtime/strmsrv.c b/runtime/strmsrv.c index 3dc53a97..a122ca8a 100644 --- a/runtime/strmsrv.c +++ b/runtime/strmsrv.c @@ -165,7 +165,7 @@ addNewLstnPort(strmsrv_t *pThis, uchar *pszPort) ISOBJ_TYPE_assert(pThis, strmsrv); /* create entry */ - CHKmalloc(pEntry = malloc(sizeof(strmLstnPortList_t))); + CHKmalloc(pEntry = MALLOC(sizeof(strmLstnPortList_t))); pEntry->pszPort = pszPort; pEntry->pSrv = pThis; CHKmalloc(pEntry->pszInputName = ustrdup(pThis->pszInputName)); diff --git a/runtime/sync.c b/runtime/sync.c index a3053e28..15d5c80f 100644 --- a/runtime/sync.c +++ b/runtime/sync.c @@ -28,12 +28,13 @@ #include "rsyslog.h" #include "sync.h" +#include "debug.h" void SyncObjInit(pthread_mutex_t **mut) { - *mut = (pthread_mutex_t *) malloc (sizeof (pthread_mutex_t)); + *mut = (pthread_mutex_t *) MALLOC(sizeof (pthread_mutex_t)); pthread_mutex_init(*mut, NULL); } diff --git a/runtime/syslogd-types.h b/runtime/syslogd-types.h index 161ee06f..fcebd985 100644 --- a/runtime/syslogd-types.h +++ b/runtime/syslogd-types.h @@ -56,8 +56,10 @@ * applications I do not yet envision. -- rgerhards, 2007-07-24 */ typedef enum _syslogFeature { - sFEATURERepeatedMsgReduction = 1, - sFEATURENonCancelInputTermination = 2 + sFEATURERepeatedMsgReduction = 1, /* for output modules */ + sFEATURENonCancelInputTermination = 2, /* for input modules */ + sFEATUREAutomaticSanitazion = 3, /* for parser modules */ + sFEATUREAutomaticPRIParsing = 4 /* for parser modules */ } syslogFeature; /* we define our own facility and severities */ diff --git a/runtime/vm.c b/runtime/vm.c index d7cd52d5..a1d992c3 100644 --- a/runtime/vm.c +++ b/runtime/vm.c @@ -34,6 +34,7 @@ #include "vm.h" #include "sysvar.h" #include "stringbuf.h" +#include "unicode-helper.h" /* static data */ DEFobjStaticHelpers @@ -41,6 +42,8 @@ DEFobjCurrIf(vmstk) DEFobjCurrIf(var) DEFobjCurrIf(sysvar) +static pthread_mutex_t mutGetenv; /* we need to make this global because otherwise we can not guarantee proper init! */ + /* ------------------------------ function registry code and structures ------------------------------ */ /* we maintain a registry of known functions */ @@ -539,6 +542,42 @@ finalize_it: } +/* The getenv function. Note that we guard the OS call by a mutex, as that + * function is not guaranteed to be thread-safe. This implementation here is far from + * being optimal, at least we should cache the result. This is left TODO for + * a later revision. + * rgerhards, 2009-11-03 + */ +static rsRetVal +rsf_getenv(vmstk_t *pStk, int numOperands) +{ + DEFiRet; + var_t *operand1; + char *envResult; + cstr_t *pCstr; + + if(numOperands != 1) + ABORT_FINALIZE(RS_RET_INVLD_NBR_ARGUMENTS); + + /* pop args and do operaton (trivial case here...) */ + vmstk.PopString(pStk, &operand1); + d_pthread_mutex_lock(&mutGetenv); + envResult = getenv((char*) rsCStrGetSzStr(operand1->val.pStr)); + DBGPRINTF("rsf_getenv(): envvar '%s', return '%s'\n", rsCStrGetSzStr(operand1->val.pStr), + envResult == NULL ? "(NULL)" : envResult); + iRet = rsCStrConstructFromszStr(&pCstr, (envResult == NULL) ? UCHAR_CONSTANT("") : (uchar*)envResult); + d_pthread_mutex_unlock(&mutGetenv); + if(iRet != RS_RET_OK) + FINALIZE; /* need to do this after mutex is unlocked! */ + + /* Store result and cleanup */ + var.SetString(operand1, pCstr); + vmstk.Push(pStk, operand1); +finalize_it: + RETiRet; +} + + /* The "tolower" function, which converts its sole argument to lower case. * Quite honestly, currently this is primarily a test driver for me... * rgerhards, 2009-04-06 @@ -756,6 +795,8 @@ BEGINObjClassExit(vm, OBJ_IS_CORE_MODULE) /* class, version */ objRelease(sysvar, CORE_COMPONENT); objRelease(var, CORE_COMPONENT); objRelease(vmstk, CORE_COMPONENT); + + pthread_mutex_destroy(&mutGetenv); ENDObjClassExit(vm) @@ -776,6 +817,9 @@ BEGINObjClassInit(vm, 1, OBJ_IS_CORE_MODULE) /* class, version */ /* register built-in functions // TODO: move to its own module */ CHKiRet(rsfrAddFunction((uchar*)"strlen", rsf_strlen)); CHKiRet(rsfrAddFunction((uchar*)"tolower", rsf_tolower)); + CHKiRet(rsfrAddFunction((uchar*)"getenv", rsf_getenv)); + + pthread_mutex_init(&mutGetenv, NULL); ENDObjClassInit(vm) diff --git a/runtime/wti.c b/runtime/wti.c index 53b695b0..288670b6 100644 --- a/runtime/wti.c +++ b/runtime/wti.c @@ -119,7 +119,7 @@ wtiSetState(wti_t *pThis, bool bNewVal) * Note that when waiting for the thread to terminate, we do a busy wait, checking * progress every 10ms. It is very unlikely that we will ever cancel a thread * and, if so, it will only happen at the end of the rsyslog run. So doing this - * kind of not optimal wait is considered preferable over using condition variables. + * kind of non-optimal wait is considered preferable over using condition variables. * rgerhards, 2008-02-26 */ rsRetVal @@ -134,7 +134,6 @@ wtiCancelThrd(wti_t *pThis) pthread_cancel(pThis->thrdID); /* now wait until the thread terminates... */ while(wtiGetState(pThis)) { -//fprintf(stderr, "sleep loop for getState\n"); srSleep(0, 10000); } } @@ -184,7 +183,7 @@ finalize_it: /* cancellation cleanup handler for queueWorker () - * Updates admin structure and frees ressources. + * Most importantly, it must bring back the batch into a consistent state. * Keep in mind that cancellation is disabled if we run into * the cancel cleanup handler (and have been cancelled). * rgerhards, 2008-01-16 @@ -201,10 +200,9 @@ wtiWorkerCancelCleanup(void *arg) ISOBJ_TYPE_assert(pWtp, wtp); DBGPRINTF("%s: cancelation cleanup handler called.\n", wtiGetDbgHdr(pThis)); + pWtp->pfObjProcessed(pWtp->pUsr, pThis); + DBGPRINTF("%s: done cancelation cleanup handler.\n", wtiGetDbgHdr(pThis)); - /* call user supplied handler */ - pWtp->pfOnWorkerCancel(pThis->pWtp->pUsr, pThis->batch.pElem[0].pUsrp); - ENDfunc } @@ -222,11 +220,8 @@ doIdleProcessing(wti_t *pThis, wtp_t *pWtp, int *pbInactivityTOOccured) BEGINfunc DBGPRINTF("%s: worker IDLE, waiting for work.\n", wtiGetDbgHdr(pThis)); - pWtp->pfOnIdle(pWtp->pUsr, MUTEX_ALREADY_LOCKED); - if(pThis->bAlwaysRunning) { /* never shut down any started worker */ -dbgprintf("YYY/ZZZ: wti Idle wait cond busy, mutex %p\n", pWtp->pmutUsr); d_pthread_cond_wait(pWtp->pcondBusy, pWtp->pmutUsr); } else { timeoutComp(&t, pWtp->toWrkShutdown);/* get absolute timeout */ @@ -235,11 +230,16 @@ dbgprintf("YYY/ZZZ: wti Idle wait cond busy, mutex %p\n", pWtp->pmutUsr); *pbInactivityTOOccured = 1; /* indicate we had a timeout */ } } + dbgoprint((obj_t*) pThis, "worker awoke from idle processing\n"); ENDfunc } -/* generic worker thread framework +/* generic worker thread framework. Note that we prohibit cancellation + * during almost all times, because it can have very undesired side effects. + * However, we may need to cancel a thread if the consumer blocks for too + * long (during shutdown). So what we do is block cancellation, and every + * consumer must enable it during the periods where it is safe. */ #pragma GCC diagnostic ignored "-Wempty-body" rsRetVal @@ -258,8 +258,7 @@ wtiWorker(wti_t *pThis) dbgSetThrdName(pThis->pszDbgHdr); pthread_cleanup_push(wtiWorkerCancelCleanup, pThis); - - pWtp->pfOnWorkerStartup(pWtp->pUsr); + pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &iCancelStateSave); /* now we have our identity, on to real processing */ while(1) { /* loop will be broken below - need to do mutex locks */ @@ -267,10 +266,8 @@ wtiWorker(wti_t *pThis) pWtp->pfRateLimiter(pWtp->pUsr); } -dbgprintf("YYY/ZZZ: pre lock mutex\n"); d_pthread_mutex_lock(pWtp->pmutUsr); -dbgprintf("YYY/ZZZ: wti locks mutex %p\n", pWtp->pmutUsr); /* first check if we are in shutdown process (but evaluate a bit later) */ terminateRet = wtpChkStopWrkr(pWtp, MUTEX_ALREADY_LOCKED); if(terminateRet == RS_RET_TERMINATE_NOW) { @@ -288,7 +285,6 @@ dbgprintf("YYY/ZZZ: wti locks mutex %p\n", pWtp->pmutUsr); */ localRet = pWtp->pfDoWork(pWtp->pUsr, pThis); -dbgprintf("YYY/ZZZ: wti loop locked mutex %p again\n", pWtp->pmutUsr); if(localRet == RS_RET_IDLE) { if(terminateRet == RS_RET_TERMINATE_WHEN_IDLE || bInactivityTOOccured) { d_pthread_mutex_unlock(pWtp->pmutUsr); @@ -305,12 +301,8 @@ dbgprintf("YYY/ZZZ: wti loop locked mutex %p again\n", pWtp->pmutUsr); } /* indicate termination */ - d_pthread_mutex_lock(pWtp->pmutUsr); - pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &iCancelStateSave); pthread_cleanup_pop(0); /* remove cleanup handler */ - pWtp->pfOnWorkerShutdown(pWtp->pUsr); pthread_setcancelstate(iCancelStateSave, NULL); - d_pthread_mutex_unlock(pWtp->pmutUsr); RETiRet; } @@ -340,7 +332,7 @@ wtiSetDbgHdr(wti_t *pThis, uchar *pszMsg, size_t lenMsg) free(pThis->pszDbgHdr); } - if((pThis->pszDbgHdr = malloc(sizeof(uchar) * lenMsg + 1)) == NULL) + if((pThis->pszDbgHdr = MALLOC(sizeof(uchar) * lenMsg + 1)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); memcpy(pThis->pszDbgHdr, pszMsg, lenMsg + 1); /* always think about the \0! */ diff --git a/runtime/wtp.c b/runtime/wtp.c index 47b99fe8..060e6627 100644 --- a/runtime/wtp.c +++ b/runtime/wtp.c @@ -94,13 +94,8 @@ BEGINobjConstruct(wtp) /* be sure to specify the object type also in END macro! /* set all function pointers to "not implemented" dummy so that we can safely call them */ pThis->pfChkStopWrkr = NotImplementedDummy; pThis->pfGetDeqBatchSize = NotImplementedDummy; - pThis->pfIsIdle = NotImplementedDummy; pThis->pfDoWork = NotImplementedDummy; pThis->pfObjProcessed = NotImplementedDummy; - pThis->pfOnIdle = NotImplementedDummy; - pThis->pfOnWorkerCancel = NotImplementedDummy; - pThis->pfOnWorkerStartup = NotImplementedDummy; - pThis->pfOnWorkerShutdown = NotImplementedDummy; ENDobjConstruct(wtp) @@ -122,8 +117,7 @@ wtpConstructFinalize(wtp_t *pThis) /* alloc and construct workers - this can only be done in finalizer as we previously do * not know the max number of workers */ - if((pThis->pWrkr = malloc(sizeof(wti_t*) * pThis->iNumWorkerThreads)) == NULL) - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + CHKmalloc(pThis->pWrkr = MALLOC(sizeof(wti_t*) * pThis->iNumWorkerThreads)); for(i = 0 ; i < pThis->iNumWorkerThreads ; ++i) { CHKiRet(wtiConstruct(&pThis->pWrkr[i])); @@ -160,20 +154,6 @@ CODESTARTobjDestruct(wtp) ENDobjDestruct(wtp) -/* wake up all worker threads. - * rgerhards, 2008-01-16 - */ -rsRetVal -wtpWakeupAllWrkr(wtp_t *pThis) -{ - DEFiRet; - - ISOBJ_TYPE_assert(pThis, wtp); - pthread_cond_broadcast(pThis->pcondBusy); - RETiRet; -} - - /* Sent a specific state for the worker thread pool. -- rgerhards, 2008-01-21 * We do not need to do atomic instructions as set operations are only * called when terminating the pool, and then in strict sequence. So we @@ -239,8 +219,11 @@ wtpShutdownAll(wtp_t *pThis, wtpState_t tShutdownCmd, struct timespec *ptTimeout ISOBJ_TYPE_assert(pThis, wtp); + /* lock mutex to prevent races (may otherwise happen during idle processing and such...) */ + d_pthread_mutex_lock(pThis->pmutUsr); wtpSetState(pThis, tShutdownCmd); - wtpWakeupAllWrkr(pThis); + pthread_cond_broadcast(pThis->pcondBusy); /* wake up all workers */ + d_pthread_mutex_unlock(pThis->pmutUsr); /* wait for worker thread termination */ d_pthread_mutex_lock(&pThis->mutWtp); @@ -285,14 +268,14 @@ wtpCancelAll(wtp_t *pThis) } -/* cancellation cleanup handler for executing worker decrements the worker counter. - * This is also called when the the worker is normally shut down. - * rgerhards, 2009-07-20 +/* this function contains shared code for both regular worker shutdown as + * well as shutdown via cancellation. We can not simply use pthread_cleanup_pop(1) + * as this introduces a race in the debug system (RETiRet system). + * rgerhards, 2009-10-26 */ -static void -wtpWrkrExecCancelCleanup(void *arg) +static inline void +wtpWrkrExecCleanup(wti_t *pWti) { - wti_t *pWti = (wti_t*) arg; wtp_t *pThis; BEGINfunc @@ -307,11 +290,37 @@ wtpWrkrExecCancelCleanup(void *arg) DBGPRINTF("%s: Worker thread %lx, terminated, num workers now %d\n", wtpGetDbgHdr(pThis), (unsigned long) pWti, ATOMIC_FETCH_32BIT(pThis->iCurNumWrkThrd)); - pthread_cond_broadcast(&pThis->condThrdTrm); /* activate anyone waiting on thread shutdown */ ENDfunc } +/* cancellation cleanup handler for executing worker decrements the worker counter. + * rgerhards, 2009-07-20 + */ +static void +wtpWrkrExecCancelCleanup(void *arg) +{ + wti_t *pWti = (wti_t*) arg; + wtp_t *pThis; + + BEGINfunc + ISOBJ_TYPE_assert(pWti, wti); + pThis = pWti->pWtp; + ISOBJ_TYPE_assert(pThis, wtp); + DBGPRINTF("%s: Worker thread %lx requested to be cancelled.\n", + wtpGetDbgHdr(pThis), (unsigned long) pWti); + + wtpWrkrExecCleanup(pWti); + + ENDfunc + /* NOTE: we must call ENDfunc FIRST, because otherwise the schedule may activate the main + * thread after the broadcast, which could destroy the debug class, resulting in a potential + * segfault. So we need to do the broadcast as actually the last action in our processing + */ + pthread_cond_broadcast(&pThis->condThrdTrm); /* activate anyone waiting on thread shutdown */ +} + + /* wtp worker shell. This is started and calls into the actual * wti worker. * rgerhards, 2008-01-21 @@ -345,9 +354,15 @@ wtpWorker(void *arg) /* the arg is actually a wti object, even though we are in pthread_cleanup_push(wtpWrkrExecCancelCleanup, pWti); wtiWorker(pWti); - pthread_cleanup_pop(1); + pthread_cleanup_pop(0); + wtpWrkrExecCleanup(pWti); ENDfunc + /* NOTE: we must call ENDfunc FIRST, because otherwise the schedule may activate the main + * thread after the broadcast, which could destroy the debug class, resulting in a potential + * segfault. So we need to do the broadcast as actually the last action in our processing + */ + pthread_cond_broadcast(&pThis->condThrdTrm); /* activate anyone waiting on thread shutdown */ pthread_exit(0); } #pragma GCC diagnostic warning "-Wempty-body" @@ -411,7 +426,6 @@ wtpAdviseMaxWorkers(wtp_t *pThis, int nMaxWrkr) ISOBJ_TYPE_assert(pThis, wtp); -int nMaxWrkrTmp = nMaxWrkr; if(nMaxWrkr == 0) FINALIZE; @@ -419,16 +433,16 @@ int nMaxWrkrTmp = nMaxWrkr; nMaxWrkr = pThis->iNumWorkerThreads; nMissing = nMaxWrkr - ATOMIC_FETCH_32BIT(pThis->iCurNumWrkThrd); -dbgprintf("wtpAdviseMaxWorkers, nmax: %d, curr %d, missing %d\n", nMaxWrkrTmp, pThis->iNumWorkerThreads, nMissing); if(nMissing > 0) { - DBGPRINTF("%s: high activity - starting %d additional worker thread(s).\n", wtpGetDbgHdr(pThis), nMissing); + DBGPRINTF("%s: high activity - starting %d additional worker thread(s).\n", + wtpGetDbgHdr(pThis), nMissing); /* start the rqtd nbr of workers */ for(i = 0 ; i < nMissing ; ++i) { CHKiRet(wtpStartWrkr(pThis)); } } else { -dbgprintf("YYY: adivse signal cond busy"); +dbgprintf("YYY: wtpAdviseMaxWorkers, sufficient workers, just doing adivse signal cond busy\n"); pthread_cond_signal(pThis->pcondBusy); } @@ -448,13 +462,8 @@ DEFpropSetMethPTR(wtp, pcondBusy, pthread_cond_t) DEFpropSetMethFP(wtp, pfChkStopWrkr, rsRetVal(*pVal)(void*, int)) DEFpropSetMethFP(wtp, pfRateLimiter, rsRetVal(*pVal)(void*)) DEFpropSetMethFP(wtp, pfGetDeqBatchSize, rsRetVal(*pVal)(void*, int*)) -DEFpropSetMethFP(wtp, pfIsIdle, rsRetVal(*pVal)(void*, wtp_t*)) DEFpropSetMethFP(wtp, pfDoWork, rsRetVal(*pVal)(void*, void*)) DEFpropSetMethFP(wtp, pfObjProcessed, rsRetVal(*pVal)(void*, wti_t*)) -DEFpropSetMethFP(wtp, pfOnIdle, rsRetVal(*pVal)(void*, int)) -DEFpropSetMethFP(wtp, pfOnWorkerCancel, rsRetVal(*pVal)(void*, void*)) -DEFpropSetMethFP(wtp, pfOnWorkerStartup, rsRetVal(*pVal)(void*)) -DEFpropSetMethFP(wtp, pfOnWorkerShutdown, rsRetVal(*pVal)(void*)) /* set the debug header message @@ -478,7 +487,7 @@ wtpSetDbgHdr(wtp_t *pThis, uchar *pszMsg, size_t lenMsg) pThis->pszDbgHdr = NULL; } - if((pThis->pszDbgHdr = malloc(sizeof(uchar) * lenMsg + 1)) == NULL) + if((pThis->pszDbgHdr = MALLOC(sizeof(uchar) * lenMsg + 1)) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); memcpy(pThis->pszDbgHdr, pszMsg, lenMsg + 1); /* always think about the \0! */ diff --git a/runtime/wtp.h b/runtime/wtp.h index 0505b91c..05c02a8c 100644 --- a/runtime/wtp.h +++ b/runtime/wtp.h @@ -62,12 +62,7 @@ struct wtp_s { rsRetVal (*pfGetDeqBatchSize)(void *pUsr, int*); /* obtains max dequeue count from queue config */ rsRetVal (*pfObjProcessed)(void *pUsr, wti_t *pWti); /* indicate user object is processed */ rsRetVal (*pfRateLimiter)(void *pUsr); - rsRetVal (*pfIsIdle)(void *pUsr, wtp_t *pWtp); rsRetVal (*pfDoWork)(void *pUsr, void *pWti); - rsRetVal (*pfOnIdle)(void *pUsr, int); - rsRetVal (*pfOnWorkerCancel)(void *pUsr, void*pWti); - rsRetVal (*pfOnWorkerStartup)(void *pUsr); - rsRetVal (*pfOnWorkerShutdown)(void *pUsr); /* end user objects */ uchar *pszDbgHdr; /* header string for debug messages */ }; @@ -91,13 +86,8 @@ PROTOTYPEObjClassInit(wtp); PROTOTYPEpropSetMethFP(wtp, pfChkStopWrkr, rsRetVal(*pVal)(void*, int)); PROTOTYPEpropSetMethFP(wtp, pfRateLimiter, rsRetVal(*pVal)(void*)); PROTOTYPEpropSetMethFP(wtp, pfGetDeqBatchSize, rsRetVal(*pVal)(void*, int*)); -PROTOTYPEpropSetMethFP(wtp, pfIsIdle, rsRetVal(*pVal)(void*, wtp_t*)); PROTOTYPEpropSetMethFP(wtp, pfDoWork, rsRetVal(*pVal)(void*, void*)); PROTOTYPEpropSetMethFP(wtp, pfObjProcessed, rsRetVal(*pVal)(void*, wti_t*)); -PROTOTYPEpropSetMethFP(wtp, pfOnIdle, rsRetVal(*pVal)(void*, int)); -PROTOTYPEpropSetMethFP(wtp, pfOnWorkerCancel, rsRetVal(*pVal)(void*,void*)); -PROTOTYPEpropSetMethFP(wtp, pfOnWorkerStartup, rsRetVal(*pVal)(void*)); -PROTOTYPEpropSetMethFP(wtp, pfOnWorkerShutdown, rsRetVal(*pVal)(void*)); PROTOTYPEpropSetMeth(wtp, toWrkShutdown, long); PROTOTYPEpropSetMeth(wtp, wtpState, wtpState_t); PROTOTYPEpropSetMeth(wtp, iMaxWorkerThreads, int); diff --git a/shave-libtool.in b/shave-libtool.in deleted file mode 100644 index 54ebd690..00000000 --- a/shave-libtool.in +++ /dev/null @@ -1,109 +0,0 @@ -#!/bin/sh -# -# Copyright (c) 2009, Damien Lespiau <damien.lespiau@gmail.com> -# -# Permission is hereby granted, free of charge, to any person -# obtaining a copy of this software and associated documentation -# files (the "Software"), to deal in the Software without -# restriction, including without limitation the rights to use, -# copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the -# Software is furnished to do so, subject to the following -# conditions: -# -# The above copyright notice and this permission notice shall be -# included in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. - -# we need sed -SED=@SED@ -if test -z "$SED" ; then -SED=sed -fi - -lt_unmangle () -{ - last_result=`echo $1 | $SED -e 's#.libs/##' -e 's#[0-9a-zA-Z_\-\.]*_la-##'` -} - -# the real libtool to use -LIBTOOL="$1" -shift - -# if 1, don't print anything, the underlaying wrapper will do it -pass_though=0 - -# scan the arguments, keep the right ones for libtool, and discover the mode -preserved_args= - -# have we seen the --tag option of libtool in the command line ? -tag_seen=0 - -while test "$#" -gt 0; do - opt="$1" - shift - - case $opt in - --mode=*) - mode=`echo $opt | $SED -e 's/[-_a-zA-Z0-9]*=//'` - preserved_args="$preserved_args $opt" - ;; - -o) - lt_output="$1" - preserved_args="$preserved_args $opt" - ;; - --tag=*) - tag_seen=1 - preserved_args="$preserved_args $opt" - ;; - *) - preserved_args="$preserved_args $opt" - ;; - esac -done - -case "$mode" in -compile) - # shave will be called and print the actual CC/CXX/LINK line - preserved_args="$preserved_args --shave-mode=$mode" - pass_though=1 - ;; -link) - preserved_args="$preserved_args --shave-mode=$mode" - Q=" LINK " - ;; -*) - # let's u - # echo "*** libtool: Unimplemented mode: $mode, fill a bug report" - ;; -esac - -lt_unmangle "$lt_output" -output=$last_result - -# automake does not add a --tag switch to its libtool invocation when -# assembling a .s file and rely on libtool to infer the right action based -# on the compiler name. As shave is using CC to hook a wrapper, libtool gets -# confused. Let's detect these cases and add a --tag=CC option. -tag="" -if test $tag_seen -eq 0 -a x"$mode" = xcompile; then - tag="--tag=CC" -fi - -if test -z $V; then - if test $pass_though -eq 0; then - echo "$Q$output" - fi - $LIBTOOL --silent $tag $preserved_args -else - echo $LIBTOOL $tag $preserved_args - $LIBTOOL $tag $preserved_args -fi diff --git a/shave.in b/shave.in deleted file mode 100644 index afed42e1..00000000 --- a/shave.in +++ /dev/null @@ -1,102 +0,0 @@ -#!/bin/sh -# -# Copyright (c) 2009, Damien Lespiau <damien.lespiau@gmail.com> -# -# Permission is hereby granted, free of charge, to any person -# obtaining a copy of this software and associated documentation -# files (the "Software"), to deal in the Software without -# restriction, including without limitation the rights to use, -# copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the -# Software is furnished to do so, subject to the following -# conditions: -# -# The above copyright notice and this permission notice shall be -# included in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. - -# we need sed -SED=@SED@ -if test -z "$SED" ; then -SED=sed -fi - -lt_unmangle () -{ - last_result=`echo $1 | $SED -e 's#.libs/##' -e 's#[0-9a-zA-Z_\-\.]*_la-##'` -} - -# the tool to wrap (cc, cxx, ar, ranlib, ..) -tool="$1" -shift - -# the reel tool (to call) -REEL_TOOL="$1" -shift - -pass_through=0 -preserved_args= -while test "$#" -gt 0; do - opt="$1" - shift - - case $opt in - --shave-mode=*) - mode=`echo $opt | $SED -e 's/[-_a-zA-Z0-9]*=//'` - ;; - -o) - lt_output="$1" - preserved_args="$preserved_args $opt" - ;; - *) - preserved_args="$preserved_args $opt" - ;; - esac -done - -# mode=link is handled in the libtool wrapper -case "$mode,$tool" in -link,*) - pass_through=1 - ;; -*,cxx) - Q=" CXX " - ;; -*,cc) - Q=" CC " - ;; -*,fc) - Q=" FC " - ;; -*,f77) - Q=" F77 " - ;; -*,objc) - Q=" OBJC " - ;; -*,*) - # should not happen - Q=" CC " - ;; -esac - -lt_unmangle "$lt_output" -output=$last_result - -if test -z $V; then - if test $pass_through -eq 0; then - echo "$Q$output" - fi - $REEL_TOOL $preserved_args -else - echo $REEL_TOOL $preserved_args - $REEL_TOOL $preserved_args -fi @@ -169,7 +169,7 @@ TCPSendBldFrame(tcpclt_t *pThis, char **pmsg, size_t *plen, int *pbMustBeFreed) * I have added this comment so that the logic is not accidently * changed again. rgerhards, 2005-10-25 */ - if((buf = malloc((len + 2) * sizeof(char))) == NULL) { + if((buf = MALLOC((len + 2) * sizeof(char))) == NULL) { /* extreme mem shortage, try to solve * as good as we can. No point in calling * any alarms, they might as well run out @@ -217,7 +217,7 @@ TCPSendBldFrame(tcpclt_t *pThis, char **pmsg, size_t *plen, int *pbMustBeFreed) /* IETF20061218 iLenBuf = snprintf(szLenBuf, sizeof(szLenBuf)/sizeof(char), "%d ", len + iLenBuf);*/ - if((buf = malloc((len + iLenBuf) * sizeof(char))) == NULL) { + if((buf = MALLOC((len + iLenBuf) * sizeof(char))) == NULL) { /* we are out of memory. This is an extreme situation. We do not * call any alarm handlers because they most likely run out of mem, * too. We are brave enough to call debug output, though. Other than @@ -324,7 +324,7 @@ Send(tcpclt_t *pThis, void *pData, char *msg, size_t len) * happens is that we lose our message recovery buffer - anything else would * be worse, so don't try anything ;) -- rgerhards, 2008-03-12 */ - if((pThis->prevMsg = malloc(len)) != NULL) { + if((pThis->prevMsg = MALLOC(len)) != NULL) { memcpy(pThis->prevMsg, msg, len); pThis->lenPrevMsg = len; } diff --git a/tcps_sess.c b/tcps_sess.c index 09861ab9..a3cd2f30 100644 --- a/tcps_sess.c +++ b/tcps_sess.c @@ -47,6 +47,7 @@ #include "msg.h" #include "datetime.h" #include "prop.h" +#include "debug.h" /* static data */ @@ -72,7 +73,7 @@ BEGINobjConstruct(tcps_sess) /* be sure to specify the object type also in END m pThis->bAtStrtOfFram = 1; /* indicate frame header expected */ pThis->eFraming = TCP_FRAMING_OCTET_STUFFING; /* just make sure... */ /* now allocate the message reception buffer */ - CHKmalloc(pThis->pMsg = (uchar*) malloc(sizeof(uchar) * iMaxLine + 1)); + CHKmalloc(pThis->pMsg = (uchar*) MALLOC(sizeof(uchar) * iMaxLine + 1)); finalize_it: ENDobjConstruct(tcps_sess) @@ -251,12 +252,10 @@ defaultDoSubmitMessage(tcps_sess_t *pThis, struct syslogTime *stTime, time_t ttG MsgSetInputName(pMsg, pThis->pLstnInfo->pInputName); MsgSetFlowControlType(pMsg, eFLOWCTL_LIGHT_DELAY); pMsg->msgFlags = NEEDS_PARSING | PARSE_HOSTNAME; - pMsg->bParseHOSTNAME = 1; MsgSetRcvFrom(pMsg, pThis->fromHost); CHKiRet(MsgSetRcvFromIP(pMsg, pThis->fromHostIP)); MsgSetRuleset(pMsg, pThis->pLstnInfo->pRuleset); -dbgprintf("YYY: submitting msg to queue\n"); if(pMultiSub == NULL) { CHKiRet(submitMsg(pMsg)); } else { @@ -104,7 +104,7 @@ addNewLstnPort(tcpsrv_t *pThis, uchar *pszPort) ISOBJ_TYPE_assert(pThis, tcpsrv); /* create entry */ - CHKmalloc(pEntry = malloc(sizeof(tcpLstnPortList_t))); + CHKmalloc(pEntry = MALLOC(sizeof(tcpLstnPortList_t))); pEntry->pszPort = pszPort; pEntry->pSrv = pThis; pEntry->pRuleset = pThis->pRuleset; @@ -473,6 +473,7 @@ doReceive(tcpsrv_t *pThis, tcps_sess_t **ppSess) { char buf[128*1024]; /* reception buffer - may hold a partial or multiple messages */ ssize_t iRcvd; + rsRetVal localRet; DEFiRet; ISOBJ_TYPE_assert(pThis, tcpsrv); @@ -498,11 +499,11 @@ doReceive(tcpsrv_t *pThis, tcps_sess_t **ppSess) break; case RS_RET_OK: /* valid data received, process it! */ - if(tcps_sess.DataRcvd(*ppSess, buf, iRcvd) != RS_RET_OK) { + if((localRet = tcps_sess.DataRcvd(*ppSess, buf, iRcvd)) != RS_RET_OK) { /* in this case, something went awfully wrong. * We are instructed to terminate the session. */ - errmsg.LogError(0, NO_ERRCODE, "Tearing down TCP Session - see " + errmsg.LogError(0, localRet, "Tearing down TCP Session - see " "previous messages for reason(s)\n"); pThis->pOnErrClose(*ppSess); tcps_sess.Destruct(ppSess); @@ -565,6 +566,8 @@ Run(tcpsrv_t *pThis) break; /* terminate input! */ for(i = 0 ; i < pThis->iLstnCurr ; ++i) { + if(glbl.GetGlobalInputTermState() == 1) + ABORT_FINALIZE(RS_RET_FORCE_TERM); CHKiRet(nssel.IsReady(pSel, pThis->ppLstn[i], NSDSEL_RD, &bIsReady, &nfds)); if(bIsReady) { DBGPRINTF("New connect on NSD %p.\n", pThis->ppLstn[i]); @@ -576,6 +579,8 @@ Run(tcpsrv_t *pThis) /* now check the sessions */ iTCPSess = TCPSessGetNxtSess(pThis, -1); while(nfds && iTCPSess != -1) { + if(glbl.GetGlobalInputTermState() == 1) + ABORT_FINALIZE(RS_RET_FORCE_TERM); CHKiRet(nssel.IsReady(pSel, pThis->pSessions[iTCPSess]->pStrm, NSDSEL_RD, &bIsReady, &nfds)); if(bIsReady) { doReceive(pThis, &pThis->pSessions[iTCPSess]); @@ -118,7 +118,7 @@ rsRetVal tplToString(struct template *pTpl, msg_t *pMsg, uchar **ppBuf, size_t * doSQLEscape(&pVal, &iLenVal, &bMustBeFreed, 0); } /* got source, now copy over */ - if(iBuf + iLenVal + 1 >= *pLenBuf) /* we reserve one char for the final \0! */ + if(iBuf + iLenVal >= *pLenBuf) /* we reserve one char for the final \0! */ CHKiRet(ExtendBuf(ppBuf, pLenBuf, iBuf + iLenVal + 1)); if(iLenVal > 0) { /* may be zero depending on property */ @@ -743,7 +743,7 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) /* We get here ONLY if the regex end was found */ longitud = regex_end - p; /* Malloc for the regex string */ - regex_char = (unsigned char *) malloc(longitud + 1); + regex_char = (unsigned char *) MALLOC(longitud + 1); if(regex_char == NULL) { dbgprintf("Could not allocate memory for template parameter!\n"); pTpe->data.field.has_regex = 0; @@ -849,7 +849,7 @@ struct template *tplAddLine(char* pName, unsigned char** ppRestOfConfLine) return NULL; pTpl->iLenName = strlen(pName); - pTpl->pszName = (char*) malloc(sizeof(char) * (pTpl->iLenName + 1)); + pTpl->pszName = (char*) MALLOC(sizeof(char) * (pTpl->iLenName + 1)); if(pTpl->pszName == NULL) { dbgprintf("tplAddLine could not alloc memory for template name!"); pTpl->iLenName = 0; diff --git a/tests/DiagTalker.java b/tests/DiagTalker.java index 04e12327..5a6f7dd5 100644 --- a/tests/DiagTalker.java +++ b/tests/DiagTalker.java @@ -30,7 +30,10 @@ public class DiagTalker { PrintWriter out = null; BufferedReader in = null; final String host = "127.0.0.1"; - final int port = 13500; + int port = 13500; + if(args.length > 1) { + port = Integer.parseInt(args[1]); + } try { diagSocket = new Socket(host, port); diff --git a/tests/Makefile.am b/tests/Makefile.am index b175b414..62de7f28 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -10,9 +10,23 @@ TESTS = $(TESTRUNS) cfg.sh \ daqueue-persist.sh \ diskqueue.sh \ diskqueue-fsync.sh \ + rulesetmultiqueue.sh \ manytcp.sh \ + rsf_getenv.sh \ + sndrcv.sh \ + sndrcv_gzip.sh \ + sndrcv_udp.sh \ + sndrcv_udp_nonstdpt.sh \ + queue-persist.sh \ + pipeaction.sh \ + execonlyonce.sh \ queue-persist.sh +if ENABLE_OMUDPSPOOF +TESTS += sndrcv_omudpspoof.sh \ + sndrcv_omudpspoof_nonstdpt.sh +endif + if ENABLE_OMSTDOUT TESTS += omod-if-array.sh \ proprepltest.sh \ @@ -22,9 +36,15 @@ TESTS += omod-if-array.sh \ threadingmq.sh \ threadingmqaq.sh \ discard.sh \ + badqi.sh \ fieldtest.sh endif +if ENABLE_OMRULESET +TESTS += omruleset.sh \ + omruleset-queue.sh +endif + check_JAVA = DiagTalker.java endif # if ENABLE_TESTBENCH @@ -91,12 +111,17 @@ EXTRA_DIST= 1.rstest 2.rstest 3.rstest err1.rstest \ testsuites/rfc5424-3.parse1 \ testsuites/rfc5424-4.parse1 \ testsuites/malformed1.parse1 \ + testsuites/reallife.parse1 \ + testsuites/parse2.conf \ + testsuites/reallife.parse2 \ testsuites/omod-if-array.conf \ testsuites/1.omod-if-array \ testsuites/1.field1 \ killrsyslog.sh \ parsertest.sh \ fieldtest.sh \ + rsf_getenv.sh \ + testsuites/rsf_getenv.conf \ diskqueue.sh \ testsuites/diskqueue.conf \ arrayqueue.sh \ @@ -130,11 +155,44 @@ EXTRA_DIST= 1.rstest 2.rstest 3.rstest err1.rstest \ testsuites/threadingmq.conf \ threadingmqaq.sh \ testsuites/threadingmqaq.conf \ + sndrcv_drvr.sh \ + sndrcv.sh \ + testsuites/sndrcv_sender.conf \ + testsuites/sndrcv_rcvr.conf \ + sndrcv_udp.sh \ + testsuites/sndrcv_udp_sender.conf \ + testsuites/sndrcv_udp_rcvr.conf \ + sndrcv_udp_nonstdpt.sh \ + testsuites/sndrcv_udp_nonstdpt_sender.conf \ + testsuites/sndrcv_udp_nonstdpt_rcvr.conf \ + sndrcv_omudpspoof.sh \ + testsuites/sndrcv_omudpspoof_sender.conf \ + testsuites/sndrcv_omudpspoof_rcvr.conf \ + sndrcv_omudpspoof_nonstdpt.sh \ + testsuites/sndrcv_omudpspoof_nonstdpt_sender.conf \ + testsuites/sndrcv_omudpspoof_nonstdpt_rcvr.conf \ + sndrcv_gzip.sh \ + testsuites/sndrcv_gzip_sender.conf \ + testsuites/sndrcv_gzip_rcvr.conf \ + pipeaction.sh \ + testsuites/pipeaction.conf \ proprepltest.sh \ testsuites/rfctag.conf \ testsuites/master.rfctag \ testsuites/nolimittag.conf \ testsuites/master.nolimittag \ + rulesetmultiqueue.sh \ + testsuites/rulesetmultiqueue.conf \ + omruleset.sh \ + testsuites/omruleset.conf \ + omruleset-queue.sh \ + testsuites/omruleset-queue.conf \ + badqi.sh \ + testsuites/badqi.conf \ + bad_qi/dbq.qi \ + execonlyonce.sh \ + testsuites/execonlyonce.conf \ + testsuites/execonlyonce.data \ DiagTalker.java \ cfg.sh diff --git a/tests/arrayqueue.sh b/tests/arrayqueue.sh index 58fd24ae..71e1cc21 100755 --- a/tests/arrayqueue.sh +++ b/tests/arrayqueue.sh @@ -1,7 +1,8 @@ # Test for fixedArray queue mode # added 2009-05-20 by rgerhards # This file is part of the rsyslog project, released under GPLv3 -echo testing queue fixedArray queue mode +echo =============================================================================== +echo \[arrayqueue.sh\]: testing queue fixedArray queue mode source $srcdir/diag.sh init source $srcdir/diag.sh startup arrayqueue.conf diff --git a/tests/bad_qi/dbq.qi b/tests/bad_qi/dbq.qi new file mode 100644 index 00000000..5f56e41a --- /dev/null +++ b/tests/bad_qi/dbq.qi @@ -0,0 +1,29 @@ +!OPB:1:queue:1: ++iQueueSize:2:2:84: ++iUngottenObjs:2:1:1: ++tVars.disk.sizeOnDisk:2:5:57906: ++tVars.disk.bytesRead:2:4:1010: +>End +. +<Obj:1:strm:1: ++iCurrFNum:2:1:1: ++pszFName:1:3:dbq: ++iMaxFiles:2:8:10000000: ++bDeleteOnClose:2:1:0: ++sType:2:1:1: ++tOperationsMode:2:1:2: ++tOpenMode:2:3:384: ++iCurrOffs:2:5:57906: +>End +. +<Obj:1:strm:1: ++iCurrFNum:2:1:1: ++pszFName:1:3:dbq: ++iMaxFiles:2:8:10000000: ++bDeleteOnClose:2:1:1: ++sType:2:1:1: ++tOperationsMode:2:1:1: ++tOpenMode:2:3:384: ++iCurrOffs:2:4:1010: +>End +. diff --git a/tests/badqi.sh b/tests/badqi.sh new file mode 100755 index 00000000..81f8c1c2 --- /dev/null +++ b/tests/badqi.sh @@ -0,0 +1,16 @@ +# Test for a startup with a bad qi file. This tests simply tests +# if the rsyslog engine survives (we had segfaults in this situation +# in the past). +# added 2009-10-21 by RGerhards +# This file is part of the rsyslog project, released under GPLv3 +# uncomment for debugging support: +echo =============================================================================== +echo \[badqi.sh\]: test startup with invalid .qi file +source $srcdir/diag.sh init +source $srcdir/diag.sh startup badqi.conf +# we just inject a handful of messages so that we have something to wait for... +source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 20 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # wait for process to terminate +source $srcdir/diag.sh seq-check 0 19 +source $srcdir/diag.sh exit diff --git a/tests/cfg.sh b/tests/cfg.sh index cb838354..f850c4d1 100755 --- a/tests/cfg.sh +++ b/tests/cfg.sh @@ -31,6 +31,7 @@ # # A copy of the GPL can be found in the file "COPYING" in this distribution. #set -x +echo \[cfg.sh\]: rm -f tmp echo "local directory" # diff --git a/tests/chkseq.c b/tests/chkseq.c index 8c5fc61a..6334d787 100644 --- a/tests/chkseq.c +++ b/tests/chkseq.c @@ -40,13 +40,14 @@ int main(int argc, char *argv[]) int val; int i; int ret = 0; + int verbose = 0; int dupsPermitted = 0; int start = 0, end = 0; int opt; int nDups = 0; char *file = NULL; - while((opt = getopt(argc, argv, "e:f:ds:")) != EOF) { + while((opt = getopt(argc, argv, "e:f:ds:v")) != EOF) { switch((char)opt) { case 'f': file = optarg; @@ -60,6 +61,9 @@ int main(int argc, char *argv[]) case 's': start = atoi(optarg); break; + case 'v': + ++verbose; + break; default:printf("Invalid call of chkseq\n"); printf("Usage: chkseq file -sstart -eend -d\n"); exit(1); @@ -76,6 +80,10 @@ int main(int argc, char *argv[]) exit(1); } + if(verbose) { + printf("chkseq: start %d, end %d\n", start, end); + } + /* read file */ fp = fopen(file, "r"); if(fp == NULL) { diff --git a/tests/da-mainmsg-q.sh b/tests/da-mainmsg-q.sh index d502fca3..d9cc0d4d 100755 --- a/tests/da-mainmsg-q.sh +++ b/tests/da-mainmsg-q.sh @@ -7,6 +7,7 @@ # check everything recovers from DA mode correctly. # added 2009-04-22 by Rgerhards # This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== echo "[da-mainmsg-q.sh]: testing main message queue in DA mode (going to disk)" source $srcdir/diag.sh init source $srcdir/diag.sh startup da-mainmsg-q.conf @@ -27,5 +28,6 @@ source $srcdir/diag.sh injectmsg 2050 50 # clean up and check test result source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages -source $srcdir/diag.sh seq-check 2099 +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 2099 source $srcdir/diag.sh exit diff --git a/tests/daqueue-persist-drvr.sh b/tests/daqueue-persist-drvr.sh index d95991fc..7934eb2b 100755 --- a/tests/daqueue-persist-drvr.sh +++ b/tests/daqueue-persist-drvr.sh @@ -5,9 +5,12 @@ # added 2009-05-27 by Rgerhards # This file is part of the rsyslog project, released under GPLv3 # uncomment for debugging support: -echo testing memory daqueue persisting to disk, mode $1 +echo \[daqueue-persist-drvr.sh\]: testing memory daqueue persisting to disk, mode $1 source $srcdir/diag.sh init +#export RSYSLOG_DEBUG="debug nologfuncflow nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" + # prepare config echo \$MainMsgQueueType $1 > work-queuemode.conf echo "*.* :omtesting:sleep 0 1000" > work-delay.conf @@ -19,7 +22,9 @@ $srcdir/diag.sh shutdown-immediate $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh check-mainq-spool -#exit +echo "Enter phase 2, rsyslogd restart" + +exit # restart engine and have rest processed #remove delay @@ -27,5 +32,5 @@ echo "#" > work-delay.conf source $srcdir/diag.sh startup queue-persist.conf source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages $srcdir/diag.sh wait-shutdown -source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh seq-check 0 99999 source $srcdir/diag.sh exit diff --git a/tests/daqueue-persist.sh b/tests/daqueue-persist.sh index ff81c987..feb2a347 100755 --- a/tests/daqueue-persist.sh +++ b/tests/daqueue-persist.sh @@ -2,7 +2,7 @@ # to carry out multiple tests with different queue modes # added 2009-05-27 by Rgerhards # This file is part of the rsyslog project, released under GPLv3 -echo TEST: daqueue-persist.sh +echo \[daqueue-persist.sh\]: test data persisting at shutdown source $srcdir/daqueue-persist-drvr.sh LinkedList source $srcdir/daqueue-persist-drvr.sh FixedArray # the disk test should not fail, however, the config is extreme and using diff --git a/tests/diag.sh b/tests/diag.sh index d8ba43b8..98228b12 100755 --- a/tests/diag.sh +++ b/tests/diag.sh @@ -5,7 +5,7 @@ # not always able to convey back states to the upper-level test driver # begun 2009-05-27 by rgerhards # This file is part of the rsyslog project, released under GPLv3 -#valgrind="valgrind --log-fd=1" +#valgrind="valgrind --malloc-fill=ff --free-fill=fe --log-fd=1" #valgrind="valgrind --tool=drd --log-fd=1" #valgrind="valgrind --tool=helgrind --log-fd=1" #set -o xtrace @@ -14,29 +14,34 @@ case $1 in 'init') $srcdir/killrsyslog.sh # kill rsyslogd if it runs for some reason cp $srcdir/testsuites/diag-common.conf diag-common.conf + cp $srcdir/testsuites/diag-common2.conf diag-common2.conf rm -f rsyslogd.started work-*.conf + rm -f rsyslogd2.started work-*.conf rm -f work rsyslog.out.log rsyslog.out.log.save # common work files rm -rf test-spool rm -f core.* vgcore.* mkdir test-spool ;; 'exit') rm -f rsyslogd.started work-*.conf diag-common.conf + rm -f rsyslogd2.started diag-common2.conf rm -f work rsyslog.out.log rsyslog.out.log.save # common work files rm -rf test-spool + echo ------------------------------------------------------------------------------- ;; 'startup') # start rsyslogd with default params. $2 is the config file name to use - # returns only after successful startup - $valgrind ../tools/rsyslogd -c4 -u2 -n -irsyslog.pid -M../runtime/.libs:../.libs -f$srcdir/testsuites/$2 & - $srcdir/diag.sh wait-startup + # returns only after successful startup, $3 is the instance (blank or 2!) + $valgrind ../tools/rsyslogd -c4 -u2 -n -irsyslog$3.pid -M../runtime/.libs:../.libs -f$srcdir/testsuites/$2 & + $srcdir/diag.sh wait-startup $3 ;; - 'wait-startup') # wait for rsyslogd startup - while test ! -f rsyslogd.started; do + 'wait-startup') # wait for rsyslogd startup ($2 is the instance) + while test ! -f rsyslogd$2.started; do true done - echo "rsyslogd started with pid " `cat rsyslog.pid` + echo "rsyslogd$2 started with pid " `cat rsyslog$2.pid` ;; - 'wait-shutdown') # actually, we wait for rsyslog.pid to be deleted - while test -f rsyslog.pid; do + 'wait-shutdown') # actually, we wait for rsyslog.pid to be deleted. $2 is the + # instance + while test -f rsyslog$2.pid; do true done if [ -e core.* ] @@ -46,15 +51,20 @@ case $1 in exit 1 fi ;; - 'wait-queueempty') # wait for main message queue to be empty - echo WaitMainQueueEmpty | java -classpath $abs_top_builddir DiagTalker + 'wait-queueempty') # wait for main message queue to be empty. $2 is the instance. + if [ "$2" == "2" ] + then + echo WaitMainQueueEmpty | java -classpath $abs_top_builddir DiagTalker + else + echo WaitMainQueueEmpty | java -classpath $abs_top_builddir DiagTalker 13501 + fi ;; - 'shutdown-when-empty') # shut rsyslogd down when main queue is empty - $srcdir/diag.sh wait-queueempty - kill `cat rsyslog.pid` + 'shutdown-when-empty') # shut rsyslogd down when main queue is empty. $2 is the instance. + $srcdir/diag.sh wait-queueempty $2 + kill `cat rsyslog$2.pid` # note: we do not wait for the actual termination! ;; - 'shutdown-immediate') # shut rsyslogd down without emptying the queue + 'shutdown-immediate') # shut rsyslogd down without emptying the queue. $2 is the instance. kill `cat rsyslog.pid` # note: we do not wait for the actual termination! ;; @@ -67,10 +77,11 @@ case $1 in fi ;; 'injectmsg') # inject messages via our inject interface (imdiag) + echo injecting $3 messages echo injectmsg $2 $3 $4 $5 | java -classpath $abs_top_builddir DiagTalker # TODO: some return state checking? (does it really make sense here?) ;; - 'check-mainq-spool') # check if mainqueue spool files exist, if not abort (we just check .qi) + 'check-mainq-spool') # check if mainqueue spool files exist, if not abort (we just check .qi). echo There must exist some files now: ls -l test-spool if test ! -f test-spool/mainq.qi; then @@ -79,10 +90,24 @@ case $1 in exit 1 fi ;; - 'seq-check') # do the usual sequence check to see if everything was properly received + 'seq-check') # do the usual sequence check to see if everything was properly received. $2 is the instance. rm -f work sort < rsyslog.out.log > work - ./chkseq -fwork -e$2 $3 + # $4... are just to have the abilit to pass in more options... + # add -v to chkseq if you need more verbose output + ./chkseq -fwork -s$2 -e$3 $4 $5 $6 $7 + if [ "$?" -ne "0" ]; then + echo "sequence error detected" + exit 1 + fi + ;; + 'seq-check2') # do the usual sequence check to see if everything was properly received. This is + # a duplicateof seq-check, but we could not change its calling conventions without + # breaking a lot of exitings test cases, so we preferred to duplicate the code here. + rm -f work2 + sort < rsyslog2.out.log > work2 + # $4... are just to have the abilit to pass in more options... + ./chkseq -fwork2 -v -s$2 -e$3 $4 $5 $6 $7 if [ "$?" -ne "0" ]; then echo "sequence error detected" exit 1 @@ -90,7 +115,7 @@ case $1 in ;; 'nettester') # perform nettester-based tests # use -v for verbose output! - ./nettester -t$2 -i$3 + ./nettester -t$2 -i$3 $4 if [ "$?" -ne "0" ]; then exit 1 fi diff --git a/tests/discard.sh b/tests/discard.sh index 0fafc7d9..9824b361 100755 --- a/tests/discard.sh +++ b/tests/discard.sh @@ -4,7 +4,8 @@ # added 2009-07-30 by Rgerhards # This file is part of the rsyslog project, released under GPLv3 # uncomment for debugging support: -echo TEST discard.sh: testing discard functionality +echo =============================================================================== +echo \[discard.sh\]: testing discard functionality source $srcdir/diag.sh init source $srcdir/diag.sh startup discard.conf # 20000 messages should be enough - the disk test is slow enough ;) @@ -12,5 +13,5 @@ sleep 4 source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 10 1 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages source $srcdir/diag.sh wait-shutdown -source $srcdir/diag.sh seq-check 10 -s2 +source $srcdir/diag.sh seq-check 2 10 source $srcdir/diag.sh exit diff --git a/tests/diskqueue-fsync.sh b/tests/diskqueue-fsync.sh index 0282202d..c826d32c 100755 --- a/tests/diskqueue-fsync.sh +++ b/tests/diskqueue-fsync.sh @@ -5,7 +5,7 @@ # added 2009-06-09 by Rgerhards # This file is part of the rsyslog project, released under GPLv3 # uncomment for debugging support: -echo testing queue disk-only mode, fsync case +echo \[diskqueue-fsync.sh\]: testing queue disk-only mode, fsync case source $srcdir/diag.sh init source $srcdir/diag.sh startup diskqueue-fsync.conf # 1000 messages should be enough - the disk fsync test is very slow! diff --git a/tests/diskqueue.sh b/tests/diskqueue.sh index 8233d569..e91f0633 100755 --- a/tests/diskqueue.sh +++ b/tests/diskqueue.sh @@ -5,7 +5,7 @@ # added 2009-04-17 by Rgerhards # This file is part of the rsyslog project, released under GPLv3 # uncomment for debugging support: -echo diskqueue.sh: testing queue disk-only mode +echo \[diskqueue.sh\]: testing queue disk-only mode source $srcdir/diag.sh init source $srcdir/diag.sh startup diskqueue.conf # 20000 messages should be enough - the disk test is slow enough ;) diff --git a/tests/execonlyonce.sh b/tests/execonlyonce.sh new file mode 100755 index 00000000..6ddc89b5 --- /dev/null +++ b/tests/execonlyonce.sh @@ -0,0 +1,29 @@ +# Test for the $ActionExecOnlyOnceEveryInterval directive. +# We inject a couple of messages quickly during the interval, +# then wait until the interval expires, then quickly inject +# another set. After that, it is checked if exactly two messages +# have arrived. +# The once interval must be set to 3 seconds in the config file. +# added 2009-11-12 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo \[execonlyonce.sh\]: test for the $ActionExecOnlyOnceEveryInterval directive +source $srcdir/diag.sh init +source $srcdir/diag.sh startup execonlyonce.conf +source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 10 1 +# now wait until the interval definitely expires +sleep 4 # one more than the once inerval! +# and inject another couple of messages +source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 10 100 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown + +# now we need your custom logic to see if the result is equal to the +# expected result +cmp rsyslog.out.log testsuites/execonlyonce.data +if [ $? -eq 1 ] +then + echo "ERROR, output not as expected" + exit 1 +fi +source $srcdir/diag.sh exit diff --git a/tests/fieldtest.sh b/tests/fieldtest.sh index 482fa143..9875fda6 100755 --- a/tests/fieldtest.sh +++ b/tests/fieldtest.sh @@ -1,4 +1,4 @@ -echo test fieldtest via udp +echo \[fieldtest.sh\]: test fieldtest via udp $srcdir/killrsyslog.sh # kill rsyslogd if it runs for some reason ./nettester -tfield1 -iudp diff --git a/tests/imtcp-multiport.sh b/tests/imtcp-multiport.sh index 702f8834..47a33cb9 100755 --- a/tests/imtcp-multiport.sh +++ b/tests/imtcp-multiport.sh @@ -7,11 +7,13 @@ # # added 2009-05-22 by Rgerhards # This file is part of the rsyslog project, released under GPLv3 -echo testing imtcp multiple listeners +echo =============================================================================== +echo \[imtcp-multiport.sh\]: testing imtcp multiple listeners source $srcdir/diag.sh init source $srcdir/diag.sh startup imtcp-multiport.conf source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 10000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 9999 source $srcdir/diag.sh exit # @@ -23,6 +25,7 @@ source $srcdir/diag.sh init source $srcdir/diag.sh startup imtcp-multiport.conf source $srcdir/diag.sh tcpflood 127.0.0.1 13515 1 10000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 9999 source $srcdir/diag.sh exit # @@ -34,5 +37,6 @@ source $srcdir/diag.sh init source $srcdir/diag.sh startup imtcp-multiport.conf source $srcdir/diag.sh tcpflood 127.0.0.1 13516 1 10000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 9999 source $srcdir/diag.sh exit diff --git a/tests/inputname.sh b/tests/inputname.sh index e1a58517..71f11c1e 100755 --- a/tests/inputname.sh +++ b/tests/inputname.sh @@ -1,4 +1,4 @@ -echo testing $InputTCPServerInputName directive +echo \[inputname.sh\]: testing $InputTCPServerInputName directive $srcdir/killrsyslog.sh # kill rsyslogd if it runs for some reason echo port 12514 diff --git a/tests/killrsyslog.sh b/tests/killrsyslog.sh index c9b6e0ac..aac24909 100755 --- a/tests/killrsyslog.sh +++ b/tests/killrsyslog.sh @@ -5,3 +5,9 @@ then kill -9 `cat rsyslog.pid` sleep 1 fi +if [ -e "rsyslog2.pid" ] +then + echo rsyslog2.pid exists, trying to shut down rsyslogd process `cat rsyslog2.pid`. + kill -9 `cat rsyslog2.pid` + sleep 1 +fi diff --git a/tests/linkedlistqueue.sh b/tests/linkedlistqueue.sh index 72c2a403..e6d48a68 100755 --- a/tests/linkedlistqueue.sh +++ b/tests/linkedlistqueue.sh @@ -1,7 +1,7 @@ # Test for Linkedlist queue mode # added 2009-05-20 by rgerhards # This file is part of the rsyslog project, released under GPLv3 -echo testing queue Linkedlist queue mode +echo \[linkedlistqueue.sh\]: testing queue Linkedlist queue mode source $srcdir/diag.sh init source $srcdir/diag.sh startup linkedlistqueue.conf diff --git a/tests/manytcp.sh b/tests/manytcp.sh index c55eb9c0..06e2fac9 100755 --- a/tests/manytcp.sh +++ b/tests/manytcp.sh @@ -1,4 +1,5 @@ # test many concurrent tcp connections +echo \[manytcp.sh\]: test concurrent tcp connections source $srcdir/diag.sh init source $srcdir/diag.sh startup manytcp.conf # the config file specifies exactly 1100 connections diff --git a/tests/nettester.c b/tests/nettester.c index 2838b919..7f1a6b1f 100644 --- a/tests/nettester.c +++ b/tests/nettester.c @@ -62,6 +62,7 @@ static char *testSuite = NULL; /* name of current test suite */ static int iPort = 12514; /* port which shall be used for sending data */ static char* pszCustomConf = NULL; /* custom config file, use -c conf to specify */ static int verbose = 0; /* verbose output? -v option */ +static int IPv4Only = 0; /* use only IPv4 in rsyslogd call? */ /* these two are quick hacks... */ int iFailed = 0; @@ -221,7 +222,7 @@ int openPipe(char *configFile, pid_t *pid, int *pfd) int pipefd[2]; pid_t cpid; char *newargv[] = {"../tools/rsyslogd", "dummy", "-c4", "-u2", "-n", "-irsyslog.pid", - "-M../runtime/.libs:../.libs", NULL }; + "-M../runtime/.libs:../.libs", NULL, NULL}; char confFile[1024]; char *newenviron[] = { NULL }; /* debug aide... @@ -233,6 +234,9 @@ int openPipe(char *configFile, pid_t *pid, int *pfd) (pszCustomConf == NULL) ? configFile : pszCustomConf); newargv[1] = confFile; + if(IPv4Only) + newargv[(sizeof(newargv)/sizeof(char*)) - 2] = "-4"; + if (pipe(pipefd) == -1) { perror("pipe"); exit(EXIT_FAILURE); @@ -316,7 +320,7 @@ processTestFile(int fd, char *pszFileName) /* pull response from server and then check if it meets our expectation */ readLine(fd, buf); if(strlen(buf) == 0) { - printf("something went wrong - read a zero-length string from rsyslogd"); + printf("something went wrong - read a zero-length string from rsyslogd\n"); exit(1); } if(strcmp(expected, buf)) { @@ -326,10 +330,13 @@ processTestFile(int fd, char *pszFileName) ret = 1; } + /* clean up after the try */ + free(testdata); + testdata = NULL; + free(expected); + expected = NULL; } - free(testdata); - free(expected); fclose(fp); return(ret); } @@ -423,8 +430,11 @@ int main(int argc, char *argv[]) char buf[4096]; char testcases[4096]; - while((opt = getopt(argc, argv, "c:i:p:t:v")) != EOF) { + while((opt = getopt(argc, argv, "4c:i:p:t:v")) != EOF) { switch((char)opt) { + case '4': + IPv4Only = 1; + break; case 'c': pszCustomConf = optarg; break; diff --git a/tests/omod-if-array.sh b/tests/omod-if-array.sh index 2c2a8ef3..4e916f1e 100755 --- a/tests/omod-if-array.sh +++ b/tests/omod-if-array.sh @@ -1,4 +1,4 @@ -echo test omod-if-array via udp +echo \[omod-if-array.sh\]: test omod-if-array via udp $srcdir/killrsyslog.sh # kill rsyslogd if it runs for some reason ./nettester -tomod-if-array -iudp -p4711 diff --git a/tests/omruleset-queue.sh b/tests/omruleset-queue.sh new file mode 100755 index 00000000..cfb80c62 --- /dev/null +++ b/tests/omruleset-queue.sh @@ -0,0 +1,19 @@ +# test for omruleset. What we do is have the main queue forward +# all messages to a secondary ruleset via omruleset, which then does +# the actual file write. We check if all messages arrive at the file, +# what implies that omruleset works. No filters or special queue modes +# are used, but the ruleset uses its own queue. So we can also inject +# more messages without running into troubles. +# added 2009-11-02 by rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo \[omruleset-queue.sh\]: test for omruleset functionality with a ruleset queue +source $srcdir/diag.sh init +source $srcdir/diag.sh startup omruleset-queue.conf +source $srcdir/diag.sh injectmsg 0 20000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 19999 +source $srcdir/diag.sh exit diff --git a/tests/omruleset.sh b/tests/omruleset.sh new file mode 100755 index 00000000..dbc5cb31 --- /dev/null +++ b/tests/omruleset.sh @@ -0,0 +1,22 @@ +# Basic test for omruleset. What we do is have the main queue forward +# all messages to a secondary ruleset via omruleset, which then does +# the actual file write. We check if all messages arrive at the file, +# what implies that omruleset works. No filters or special queue modes +# are used, so the message is re-enqueued into the main message queue. +# We inject just 5,000 message because we may otherwise run into +# queue full conditions (as we use the same queue) and that +# would result in longer execution time. In any case, 5000 messages +# are well enough to test what we want to test. +# added 2009-11-02 by rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo \[omruleset.sh\]: basic test for omruleset functionality +source $srcdir/diag.sh init +source $srcdir/diag.sh startup omruleset.conf +source $srcdir/diag.sh injectmsg 0 5000 +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 4999 +source $srcdir/diag.sh exit diff --git a/tests/parsertest.sh b/tests/parsertest.sh index ef33256e..470d9375 100755 --- a/tests/parsertest.sh +++ b/tests/parsertest.sh @@ -1,5 +1,13 @@ -echo TEST: parsertest.sh - various parser tests +echo TEST: \[parsertest.sh\]: various parser tests source $srcdir/diag.sh init source $srcdir/diag.sh nettester parse1 udp source $srcdir/diag.sh nettester parse1 tcp +source $srcdir/diag.sh nettester parse2 udp +source $srcdir/diag.sh nettester parse2 tcp + +echo \[parsertest.sh]: redoing tests in IPv4-only mode +source $srcdir/diag.sh nettester parse1 udp -4 +source $srcdir/diag.sh nettester parse1 tcp -4 +source $srcdir/diag.sh nettester parse2 udp -4 +source $srcdir/diag.sh nettester parse2 tcp -4 source $srcdir/diag.sh init diff --git a/tests/pipeaction.sh b/tests/pipeaction.sh new file mode 100755 index 00000000..26a4c22a --- /dev/null +++ b/tests/pipeaction.sh @@ -0,0 +1,33 @@ +# Test for the pipe output action. +# will create a fifo in the current directory, write to it and +# then do the usual sequence checks. +# added 2009-11-05 by RGerhards +echo =============================================================================== +echo \[pipeaction.sh\]: testing pipe output action + +# create the pipe and start a background process that copies data from +# it to the "regular" work file +source $srcdir/diag.sh init +rm -f rsyslog-testbench-fifo +mkfifo rsyslog-testbench-fifo +cp rsyslog-testbench-fifo rsyslog.out.log & +CPPROCESS=$! +echo background cp process id is $CPPROCESS + +# now do the usual run +source $srcdir/diag.sh startup pipeaction.conf +# 20000 messages should be enough +#source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 20000 +source $srcdir/diag.sh injectmsg 0 20000 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown + +# wait for the cp process to finish, do pipe-specific cleanup +echo waiting for background cp to terminate... +wait $CPPROCESS +rm -f rsyslog-testbench-fifo +echo background cp has terminated, continue test... + +# and continue the usual checks +source $srcdir/diag.sh seq-check 0 19999 +source $srcdir/diag.sh exit diff --git a/tests/proprepltest.sh b/tests/proprepltest.sh index 3c252e52..2e59a31d 100755 --- a/tests/proprepltest.sh +++ b/tests/proprepltest.sh @@ -1,4 +1,4 @@ -echo TEST: proprepltest.sh - various tests for the property replacer +echo \[proprepltest.sh\]: various tests for the property replacer source $srcdir/diag.sh init source $srcdir/diag.sh nettester rfctag udp source $srcdir/diag.sh nettester rfctag tcp diff --git a/tests/queue-persist-drvr.sh b/tests/queue-persist-drvr.sh index ea5386a7..53fbcb8b 100755 --- a/tests/queue-persist-drvr.sh +++ b/tests/queue-persist-drvr.sh @@ -24,5 +24,6 @@ source $srcdir/diag.sh check-mainq-spool echo "#" > work-delay.conf source $srcdir/diag.sh startup queue-persist.conf source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +$srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 4999 source $srcdir/diag.sh exit diff --git a/tests/queue-persist.sh b/tests/queue-persist.sh index e05b3da3..ff1842bb 100755 --- a/tests/queue-persist.sh +++ b/tests/queue-persist.sh @@ -2,7 +2,7 @@ # to carry out multiple tests with different queue modes # added 2009-05-27 by Rgerhards # This file is part of the rsyslog project, released under GPLv3 -echo TEST: queue-persist.sh +echo \[queue-persist.sh\]: source $srcdir/queue-persist-drvr.sh LinkedList source $srcdir/queue-persist-drvr.sh FixedArray # the disk test should not fail, however, the config is extreme and using diff --git a/tests/rsf_getenv.sh b/tests/rsf_getenv.sh new file mode 100755 index 00000000..42de20fe --- /dev/null +++ b/tests/rsf_getenv.sh @@ -0,0 +1,17 @@ +# Test for the getenv() rainerscript function +# this is a quick test, but it gurantees that the code path is +# at least progressed (but we do not check for unset envvars!) +# added 2009-11-03 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +# uncomment for debugging support: +echo =============================================================================== +echo \[rsf_getenv.sh\]: testing RainerScript getenv\(\) function +export MSGNUM="msgnum:" +source $srcdir/diag.sh init +source $srcdir/diag.sh startup rsf_getenv.conf +source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 10000 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh seq-check 0 9999 +unset MSGNUM +source $srcdir/diag.sh exit diff --git a/tests/rulesetmultiqueue.sh b/tests/rulesetmultiqueue.sh new file mode 100755 index 00000000..aff55148 --- /dev/null +++ b/tests/rulesetmultiqueue.sh @@ -0,0 +1,33 @@ +# Test for disk-only queue mode +# This tests defines three rulesets, each one with its own queue. Then, it +# sends data to them and checks the outcome. Note that we do need to +# use some custom code as the test driver framework does not (yet?) +# support multi-output-file operations. +# added 2009-10-30 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo \[rulesetmultiqueu.sh\]: testing multiple queues via rulesets +source $srcdir/diag.sh init +rm -f rsyslog.out1.log rsyslog.out2.log rsyslog.out3.log +source $srcdir/diag.sh startup rulesetmultiqueue.conf +source $srcdir/diag.sh wait-startup +# now fill the three files (a bit sequentially, but they should +# still get their share of concurrency - to increase the chance +# we use three connections per set). +source $srcdir/diag.sh tcpflood 127.0.0.1 13514 3 20000 0 +source $srcdir/diag.sh tcpflood 127.0.0.1 13515 3 20000 20000 +source $srcdir/diag.sh tcpflood 127.0.0.1 13516 3 20000 40000 + +# in this version of the imdiag, we do not have the capability to poll +# all queues for emptyness. So we do a sleep in the hopes that this will +# sufficiently drain the queues. This is race, but the best we currently +# can do... - rgerhards, 2009-11-05 +sleep 2 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown +# now consolidate all logs into a single one so that we can use the +# regular check logic +cat rsyslog.out1.log rsyslog.out2.log rsyslog.out3.log > rsyslog.out.log +source $srcdir/diag.sh seq-check 0 59999 +rm -f rsyslog.out1.log rsyslog.out2.log rsyslog.out3.log +source $srcdir/diag.sh exit diff --git a/tests/runtime-dummy.c b/tests/runtime-dummy.c index 38e6bba1..5a9039bf 100644 --- a/tests/runtime-dummy.c +++ b/tests/runtime-dummy.c @@ -34,12 +34,13 @@ int repeatinterval = 30; int bActExecWhenPrevSusp = 0; int iActExecOnceInterval = 1; int MarkInterval = 30; +void *pMsgQueue = NULL; void cflineClassic(void) {}; void selectorAddList(void) {}; void selectorConstruct(void) {}; void selectorDestruct(void) {}; -void getFIOPName(void) {}; +rsRetVal createMainQueue(void) { return RS_RET_ERR; } ruleset_t *pCurrRuleset; /* these are required by some dynamically loaded modules */ diff --git a/tests/sndrcv.sh b/tests/sndrcv.sh new file mode 100755 index 00000000..2fc3bd82 --- /dev/null +++ b/tests/sndrcv.sh @@ -0,0 +1,9 @@ +# This tests two rsyslog instances. Instance +# TWO sends data to instance ONE. A number of messages is injected into +# the instance 2 and we finally check if all those messages +# arrived at instance 1. +# added 2009-11-11 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo \[sndrcv.sh\]: testing sending and receiving via tcp +source $srcdir/sndrcv_drvr.sh sndrcv 50000 diff --git a/tests/sndrcv_drvr.sh b/tests/sndrcv_drvr.sh new file mode 100755 index 00000000..9835a8eb --- /dev/null +++ b/tests/sndrcv_drvr.sh @@ -0,0 +1,49 @@ +# This is test driver for testing two rsyslog instances. It can be +# utilized by any test that just needs two instances with different +# config files, where messages are injected in instance TWO and +# (with whatever rsyslog mechanism) being relayed over to instance ONE, +# where they are written to the log file. After the run, the completeness +# of that log file is checked. +# The code is almost the same, but the config files differ (probably greatly) +# for different test cases. As such, this driver needs to be called with the +# config file name ($2). From that name, the sender and receiver config file +# names are automatically generated. +# So: $1 config file name, $2 number of messages +# +# A note on TLS testing: the current testsuite (in git!) already contains +# TLS test cases. However, getting these test cases correct is not simple. +# That's not a problem with the code itself, but rater a problem with +# synchronization in the test environment. So I have deciced to keep the +# TLS tests in, but not yet actually utilize them. This is most probably +# left as an excercise for future (devel) releases. -- rgerhards, 2009-11-11 +# +# added 2009-11-11 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +# uncomment for debugging support: +source $srcdir/diag.sh init +# start up the instances +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup $1_rcvr.conf +source $srcdir/diag.sh wait-startup +#export RSYSLOG_DEBUGLOG="log2" +#valgrind="valgrind" +source $srcdir/diag.sh startup $1_sender.conf 2 +source $srcdir/diag.sh wait-startup 2 +# may be needed by TLS (once we do it): sleep 30 + +# now inject the messages into instance 2. It will connect to instance 1, +# and that instance will record the data. +source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 $2 1 +# shut down sender when everything is sent, receiver continues to run concurrently +# may be needed by TLS (once we do it): sleep 60 +source $srcdir/diag.sh shutdown-when-empty 2 +source $srcdir/diag.sh wait-shutdown 2 +# now it is time to stop the receiver as well +source $srcdir/diag.sh shutdown-when-empty +source $srcdir/diag.sh wait-shutdown + +# may be needed by TLS (once we do it): sleep 60 +# do the final check +source $srcdir/diag.sh seq-check 1 $2 +source $srcdir/diag.sh exit diff --git a/tests/sndrcv_gzip.sh b/tests/sndrcv_gzip.sh new file mode 100755 index 00000000..4931f3d0 --- /dev/null +++ b/tests/sndrcv_gzip.sh @@ -0,0 +1,7 @@ +# This test is similar to tcpsndrcv, but it forwards messages in +# zlib-compressed format (our own syslog extension). +# rgerhards, 2009-11-11 +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo \[sndrcv_gzip.sh\]: testing sending and receiving via tcp in zlib mode +source $srcdir/sndrcv_drvr.sh sndrcv_gzip 50000 diff --git a/tests/sndrcv_omudpspoof.sh b/tests/sndrcv_omudpspoof.sh new file mode 100755 index 00000000..bb804d93 --- /dev/null +++ b/tests/sndrcv_omudpspoof.sh @@ -0,0 +1,10 @@ +# This runs sends and receives messages via UDP to the standard +# ports. Note that with UDP we can always have message loss. While this is +# less likely in a local environment, we strongly limit the amount of data +# we send in the hope to not lose any messages. However, failure of this +# test does not necessarily mean that the code is wrong (but it is very likely!) +# added 2009-11-11 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo \[sndrcv_omudpspoof.sh\]: testing sending and receiving via omudp +source $srcdir/sndrcv_drvr.sh sndrcv_omudpspoof 50 diff --git a/tests/sndrcv_omudpspoof_nonstdpt.sh b/tests/sndrcv_omudpspoof_nonstdpt.sh new file mode 100755 index 00000000..6aeb1a5f --- /dev/null +++ b/tests/sndrcv_omudpspoof_nonstdpt.sh @@ -0,0 +1,10 @@ +# This runs sends and receives messages via UDP to the standard +# ports. Note that with UDP we can always have message loss. While this is +# less likely in a local environment, we strongly limit the amount of data +# we send in the hope to not lose any messages. However, failure of this +# test does not necessarily mean that the code is wrong (but it is very likely!) +# added 2009-11-11 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo \[sndrcv_omudpspoof_nonstdpt.sh\]: testing sending and receiving via omudp +source $srcdir/sndrcv_drvr.sh sndrcv_omudpspoof_nonstdpt 50 diff --git a/tests/sndrcv_udp.sh b/tests/sndrcv_udp.sh new file mode 100755 index 00000000..274a414a --- /dev/null +++ b/tests/sndrcv_udp.sh @@ -0,0 +1,10 @@ +# This runs sends and receives messages via UDP to the standard +# ports. Note that with UDP we can always have message loss. While this is +# less likely in a local environment, we strongly limit the amount of data +# we send in the hope to not lose any messages. However, failure of this +# test does not necessarily mean that the code is wrong (but it is very likely!) +# added 2009-11-11 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo \[sndrcv_udp.sh\]: testing sending and receiving via udp +source $srcdir/sndrcv_drvr.sh sndrcv_udp 50 diff --git a/tests/sndrcv_udp_nonstdpt.sh b/tests/sndrcv_udp_nonstdpt.sh new file mode 100755 index 00000000..2ad2906f --- /dev/null +++ b/tests/sndrcv_udp_nonstdpt.sh @@ -0,0 +1,10 @@ +# This runs sends and receives messages via UDP to the non-standard port 2514 +# Note that with UDP we can always have message loss. While this is +# less likely in a local environment, we strongly limit the amount of data +# we send in the hope to not lose any messages. However, failure of this +# test does not necessarily mean that the code is wrong (but it is very likely!) +# added 2009-11-11 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo \[sndrcv_udp_nonstdpt.sh\]: testing sending and receiving via udp +source $srcdir/sndrcv_drvr.sh sndrcv_udp_nonstdpt 50 diff --git a/tests/testsuites/badqi.conf b/tests/testsuites/badqi.conf new file mode 100644 index 00000000..0ab059df --- /dev/null +++ b/tests/testsuites/badqi.conf @@ -0,0 +1,15 @@ +# Test for bad .qi file (see .sh file for details) +# rgerhards, 2009-10-21 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +# instruct to use bad .qi file +$WorkDirectory bad_qi +$ActionQueueType LinkedList +$ActionQueueFileName dbq +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/execonlyonce.conf b/tests/testsuites/execonlyonce.conf new file mode 100644 index 00000000..085b970e --- /dev/null +++ b/tests/testsuites/execonlyonce.conf @@ -0,0 +1,12 @@ +# see the equally-named .sh file for details +# rgerhards, 2009-11-12 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +$ActionExecOnlyOnceEveryInterval 3 +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/execonlyonce.data b/tests/testsuites/execonlyonce.data new file mode 100644 index 00000000..3c54f3d4 --- /dev/null +++ b/tests/testsuites/execonlyonce.data @@ -0,0 +1,2 @@ +00000001 +00000100 diff --git a/tests/testsuites/omruleset-queue.conf b/tests/testsuites/omruleset-queue.conf new file mode 100644 index 00000000..edc33c40 --- /dev/null +++ b/tests/testsuites/omruleset-queue.conf @@ -0,0 +1,20 @@ +# test for omruleset (see .sh file for details) +# rgerhards, 2009-11-02 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/omruleset/.libs/omruleset +$ModLoad ../plugins/imtcp/.libs/imtcp +$InputTCPServerRun 13514 + +$ruleset rsinclude +# create ruleset main queue with default parameters +$RulesetCreateMainQueue on +# make sure we do not terminate too early! +$MainMsgQueueTimeoutShutdown 10000 +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile;outfmt + +$ruleset RSYSLOG_DefaultRuleset +$ActionOmrulesetRulesetName rsinclude +*.* :omruleset: diff --git a/tests/testsuites/omruleset.conf b/tests/testsuites/omruleset.conf new file mode 100644 index 00000000..fa6b906e --- /dev/null +++ b/tests/testsuites/omruleset.conf @@ -0,0 +1,16 @@ +# Basic test for omruleset (see .sh file for details) +# rgerhards, 2009-11-02 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/omruleset/.libs/omruleset +$ModLoad ../plugins/imtcp/.libs/imtcp +$InputTCPServerRun 13514 + +$ruleset rsinclude +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile;outfmt + +$ruleset RSYSLOG_DefaultRuleset +$ActionOmrulesetRulesetName rsinclude +*.* :omruleset: diff --git a/tests/testsuites/parse2.conf b/tests/testsuites/parse2.conf new file mode 100644 index 00000000..04d910bc --- /dev/null +++ b/tests/testsuites/parse2.conf @@ -0,0 +1,8 @@ +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format that we can easily parse in expect +$template output,"%PRI%,%syslogfacility-text%,%syslogseverity-text%,%timestamp%,%programname%,%syslogtag%,%msg%\n" +*.* :omstdout:;output diff --git a/tests/testsuites/pipeaction.conf b/tests/testsuites/pipeaction.conf new file mode 100644 index 00000000..f58b6d65 --- /dev/null +++ b/tests/testsuites/pipeaction.conf @@ -0,0 +1,16 @@ +# Test for pipe output action (see .sh file for details) +# rgerhards, 2009-11-05 +$IncludeConfig diag-common.conf + +$MainMsgQueueTimeoutShutdown 10000 + +# set spool locations and switch queue to disk-only mode +$WorkDirectory test-spool +$MainMsgQueueFilename mainq +$MainMsgQueueType disk + +$template outfmt,"%msg:F,58:2%\n" +# with pipes, we do not need to use absolute path names, so +# we can simply refer to our working pipe via the usual relative +# path name +:msg, contains, "msgnum:" |rsyslog-testbench-fifo;outfmt diff --git a/tests/testsuites/reallife.parse1 b/tests/testsuites/reallife.parse1 new file mode 100644 index 00000000..a83d2dca --- /dev/null +++ b/tests/testsuites/reallife.parse1 @@ -0,0 +1,12 @@ +# New tests should be added to this file if there is no specific +# reason for not doing that. Initially, we could only handle one test +# case per file, but this restriction has been removed some time ago. +# So it is less troublesome (and easier to overlook) to have all related +# tests in a single file. +# This file contains a lot of real-life samples (of course mangled so +# that they can not be traced back to the original submitter). Note +# that IP addr 192.0.2.1 is specifically set aside for testing and +# documentation by IANA. +# rgerhards, 2009-10-19 +<29>Oct 16 20:47:24 example-p exam-pl[12345]: connect host= /192.0.2.1 +29,daemon,notice,Oct 16 20:47:24,example-p,exam-pl,exam-pl[12345]:, connect host= /192.0.2.1 diff --git a/tests/testsuites/reallife.parse2 b/tests/testsuites/reallife.parse2 new file mode 100644 index 00000000..c42f2526 --- /dev/null +++ b/tests/testsuites/reallife.parse2 @@ -0,0 +1,12 @@ +# New tests should be added to this file if there is no specific +# reason for not doing that. Initially, we could only handle one test +# case per file, but this restriction has been removed some time ago. +# So it is less troublesome (and easier to overlook) to have all related +# tests in a single file. +# This file contains a lot of real-life samples (of course mangled so +# that they can not be traced back to the original submitter). Note +# that IP addr 192.0.2.1 is specifically set aside for testing and +# documentation by IANA. +# rgerhards, 2009-10-19 +<175>Oct 16 23:47:31 #001 MSWinEventLog 0#011Security#01119023582#011Fri Oct 16 16:30:44 2009#011592#011Security#011rgabcde#011User#011Success Audit#011XSXSXSN01#011Detailed Tracking#011#0112572#01119013885 +175,local5,debug,Oct 16 23:47:31,#001,#001, MSWinEventLog 0#011Security#01119023582#011Fri Oct 16 16:30:44 2009#011592#011Security#011rgabcde#011User#011Success Audit#011XSXSXSN01#011Detailed Tracking#011#0112572#01119013885 diff --git a/tests/testsuites/rsf_getenv.conf b/tests/testsuites/rsf_getenv.conf new file mode 100644 index 00000000..2f2eb58c --- /dev/null +++ b/tests/testsuites/rsf_getenv.conf @@ -0,0 +1,17 @@ +# Test for RainerScript getenv() function (see .sh file for details) +# Note envvar MSGNUM must be set to "msgnum:" +# rgerhards, 2009-11-03 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +# set spool locations and switch queue to disk-only mode +$WorkDirectory test-spool +$MainMsgQueueFilename mainq +$MainMsgQueueType disk + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +if $msg contains getenv('MSGNUM') then ?dynfile;outfmt diff --git a/tests/testsuites/rulesetmultiqueue.conf b/tests/testsuites/rulesetmultiqueue.conf new file mode 100644 index 00000000..c8a82dd6 --- /dev/null +++ b/tests/testsuites/rulesetmultiqueue.conf @@ -0,0 +1,34 @@ +# Test for multiple ruleset queues (see .sh file for details) +# rgerhards, 2009-10-30 +$IncludeConfig diag-common.conf +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 + +# general definition +$template outfmt,"%msg:F,58:2%\n" + +# create the individual rulesets +$ruleset file1 +$RulesetCreateMainQueue on +$template dynfile1,"rsyslog.out1.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile1;outfmt + +$ruleset file2 +$RulesetCreateMainQueue on +$template dynfile2,"rsyslog.out2.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile2;outfmt + +$ruleset file3 +$RulesetCreateMainQueue on +$template dynfile3,"rsyslog.out3.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile3;outfmt + +# start listeners and bind them to rulesets +$InputTCPServerBindRuleset file1 +$InputTCPServerRun 13514 + +$InputTCPServerBindRuleset file2 +$InputTCPServerRun 13515 + +$InputTCPServerBindRuleset file3 +$InputTCPServerRun 13516 diff --git a/tests/testsuites/sndrcv_gzip_rcvr.conf b/tests/testsuites/sndrcv_gzip_rcvr.conf new file mode 100644 index 00000000..6f7ce34b --- /dev/null +++ b/tests/testsuites/sndrcv_gzip_rcvr.conf @@ -0,0 +1,11 @@ +# see equally-named shell file for details +# rgerhards, 2009-11-11 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +# then SENDER sends to this port (not tcpflood!) +$InputTCPServerRun 13515 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/sndrcv_gzip_sender.conf b/tests/testsuites/sndrcv_gzip_sender.conf new file mode 100644 index 00000000..c874c068 --- /dev/null +++ b/tests/testsuites/sndrcv_gzip_sender.conf @@ -0,0 +1,8 @@ +# see tcpsndrcv.sh for details +# rgerhards, 2009-11-11 +$IncludeConfig diag-common2.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$InputTCPServerRun 13514 + +*.* @@127.0.0.1:13515 diff --git a/tests/testsuites/sndrcv_omudpspoof_nonstdpt_rcvr.conf b/tests/testsuites/sndrcv_omudpspoof_nonstdpt_rcvr.conf new file mode 100644 index 00000000..65659f00 --- /dev/null +++ b/tests/testsuites/sndrcv_omudpspoof_nonstdpt_rcvr.conf @@ -0,0 +1,11 @@ +# see equally-named shell file for details +# rgerhards, 2009-11-12 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imudp/.libs/imudp +# then SENDER sends to this port (not tcpflood!) +$UDPServerRun 2514 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/sndrcv_omudpspoof_nonstdpt_sender.conf b/tests/testsuites/sndrcv_omudpspoof_nonstdpt_sender.conf new file mode 100644 index 00000000..29a30145 --- /dev/null +++ b/tests/testsuites/sndrcv_omudpspoof_nonstdpt_sender.conf @@ -0,0 +1,18 @@ +# see equally-named shell file for details +# rgerhards, 2009-11-11 +$IncludeConfig diag-common2.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +# this listener is for message generation by the test framework! +$InputTCPServerRun 13514 + +$ModLoad ../plugins/omudpspoof/.libs/omudpspoof +$template spoofaddr,"127.0.0.1" + +#begin action definition +$ActionOMUDPSpoofSourceNameTemplate spoofaddr +$ActionOMUDPSpoofTargetHost 127.0.0.1 +$ActionOMUDPSpoofTargetPort 2514 +$ActionOMUDPSpoofSourcePortStart 514 +$ActionOMUDPSpoofSourcePortEnd 514 +*.* :omudpspoof: diff --git a/tests/testsuites/sndrcv_omudpspoof_rcvr.conf b/tests/testsuites/sndrcv_omudpspoof_rcvr.conf new file mode 100644 index 00000000..e5401811 --- /dev/null +++ b/tests/testsuites/sndrcv_omudpspoof_rcvr.conf @@ -0,0 +1,11 @@ +# see equally-named shell file for details +# rgerhards, 2009-11-12 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imudp/.libs/imudp +# then SENDER sends to this port (not tcpflood!) +$UDPServerRun 514 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/sndrcv_omudpspoof_sender.conf b/tests/testsuites/sndrcv_omudpspoof_sender.conf new file mode 100644 index 00000000..c0d25935 --- /dev/null +++ b/tests/testsuites/sndrcv_omudpspoof_sender.conf @@ -0,0 +1,17 @@ +# see equally-named shell file for details +# rgerhards, 2009-11-11 +$IncludeConfig diag-common2.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +# this listener is for message generation by the test framework! +$InputTCPServerRun 13514 + +$ModLoad ../plugins/omudpspoof/.libs/omudpspoof +$template spoofaddr,"127.0.0.1" + +#begin action definition +$ActionOMUDPSpoofSourceNameTemplate spoofaddr +$ActionOMUDPSpoofTargetHost 127.0.0.1 +$ActionOMUDPSpoofSourcePortStart 514 +$ActionOMUDPSpoofSourcePortEnd 514 +*.* :omudpspoof: diff --git a/tests/testsuites/sndrcv_rcvr.conf b/tests/testsuites/sndrcv_rcvr.conf new file mode 100644 index 00000000..6f7ce34b --- /dev/null +++ b/tests/testsuites/sndrcv_rcvr.conf @@ -0,0 +1,11 @@ +# see equally-named shell file for details +# rgerhards, 2009-11-11 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +# then SENDER sends to this port (not tcpflood!) +$InputTCPServerRun 13515 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/sndrcv_sender.conf b/tests/testsuites/sndrcv_sender.conf new file mode 100644 index 00000000..f3d6ba53 --- /dev/null +++ b/tests/testsuites/sndrcv_sender.conf @@ -0,0 +1,9 @@ +# see tcpsndrcv.sh for details +# rgerhards, 2009-11-11 +$IncludeConfig diag-common2.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +# this listener is for message generation by the test framework! +$InputTCPServerRun 13514 + +*.* @@127.0.0.1:13515 diff --git a/tests/testsuites/sndrcv_tls_anon_rcvr.conf b/tests/testsuites/sndrcv_tls_anon_rcvr.conf new file mode 100644 index 00000000..01143b22 --- /dev/null +++ b/tests/testsuites/sndrcv_tls_anon_rcvr.conf @@ -0,0 +1,22 @@ +# see equally-named shell file for details +# this is the config fil for the TLS server +# rgerhards, 2009-11-11 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp + +# certificates +$DefaultNetstreamDriverCAFile testsuites/x.509/ca.pem +$DefaultNetstreamDriverCertFile testsuites/x.509/client-cert.pem +$DefaultNetstreamDriverKeyFile testsuites/x.509/client-key.pem + +$DefaultNetstreamDriver gtls # use gtls netstream driver + +# then SENDER sends to this port (not tcpflood!) +$InputTCPServerStreamDriverMode 1 +$InputTCPServerStreamDriverAuthMode anon +$InputTCPServerRun 13515 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/sndrcv_tls_anon_sender.conf b/tests/testsuites/sndrcv_tls_anon_sender.conf new file mode 100644 index 00000000..4a944455 --- /dev/null +++ b/tests/testsuites/sndrcv_tls_anon_sender.conf @@ -0,0 +1,19 @@ +# see tcpsndrcv.sh for details +# this is the TLS client +# rgerhards, 2009-11-11 +$IncludeConfig diag-common2.conf + +# certificates +$DefaultNetstreamDriverCAFile testsuites/x.509/ca.pem +$DefaultNetstreamDriverCertFile testsuites/x.509/client-cert.pem +$DefaultNetstreamDriverKeyFile testsuites/x.509/client-key.pem + +# Note: no TLS for the listener, this is for tcpflood! +$ModLoad ../plugins/imtcp/.libs/imtcp +$InputTCPServerRun 13514 + +# set up the action +$DefaultNetstreamDriver gtls # use gtls netstream driver +$ActionSendStreamDriverMode 1 # require TLS for the connection +$ActionSendStreamDriverAuthMode anon +*.* @@127.0.0.1:13515 diff --git a/tests/testsuites/sndrcv_udp_nonstdpt_rcvr.conf b/tests/testsuites/sndrcv_udp_nonstdpt_rcvr.conf new file mode 100644 index 00000000..65659f00 --- /dev/null +++ b/tests/testsuites/sndrcv_udp_nonstdpt_rcvr.conf @@ -0,0 +1,11 @@ +# see equally-named shell file for details +# rgerhards, 2009-11-12 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imudp/.libs/imudp +# then SENDER sends to this port (not tcpflood!) +$UDPServerRun 2514 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/sndrcv_udp_nonstdpt_sender.conf b/tests/testsuites/sndrcv_udp_nonstdpt_sender.conf new file mode 100644 index 00000000..2975f938 --- /dev/null +++ b/tests/testsuites/sndrcv_udp_nonstdpt_sender.conf @@ -0,0 +1,9 @@ +# see equally-named shell file for details +# rgerhards, 2009-11-11 +$IncludeConfig diag-common2.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +# this listener is for message generation by the test framework! +$InputTCPServerRun 13514 + +*.* @127.0.0.1:2514 diff --git a/tests/testsuites/sndrcv_udp_rcvr.conf b/tests/testsuites/sndrcv_udp_rcvr.conf new file mode 100644 index 00000000..e5401811 --- /dev/null +++ b/tests/testsuites/sndrcv_udp_rcvr.conf @@ -0,0 +1,11 @@ +# see equally-named shell file for details +# rgerhards, 2009-11-12 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imudp/.libs/imudp +# then SENDER sends to this port (not tcpflood!) +$UDPServerRun 514 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/sndrcv_udp_sender.conf b/tests/testsuites/sndrcv_udp_sender.conf new file mode 100644 index 00000000..28e913ef --- /dev/null +++ b/tests/testsuites/sndrcv_udp_sender.conf @@ -0,0 +1,9 @@ +# see equally-named shell file for details +# rgerhards, 2009-11-11 +$IncludeConfig diag-common2.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +# this listener is for message generation by the test framework! +$InputTCPServerRun 13514 + +*.* @127.0.0.1 diff --git a/tests/testsuites/threadingmq.conf b/tests/testsuites/threadingmq.conf index aa5197bb..b98f9b5a 100644 --- a/tests/testsuites/threadingmq.conf +++ b/tests/testsuites/threadingmq.conf @@ -3,11 +3,7 @@ # rgerhards, 2009-06-26 $IncludeConfig diag-common.conf -$ModLoad ../plugins/imtcp/.libs/imtcp -$MainMsgQueueTimeoutShutdown 10000 -$MaxOpenFiles 2000 -$InputTCPMaxSessions 1100 -$InputTCPServerRun 13514 +$MainMsgQueueTimeoutShutdown 100000 $MainMsgQueueWorkerThreadMinimumMessages 10 $MainMsgQueueWorkerThreads 5 diff --git a/tests/testsuites/threadingmqaq.conf b/tests/testsuites/threadingmqaq.conf index f1bb72df..f0d39057 100644 --- a/tests/testsuites/threadingmqaq.conf +++ b/tests/testsuites/threadingmqaq.conf @@ -3,11 +3,7 @@ # rgerhards, 2009-06-26 $IncludeConfig diag-common.conf -$ModLoad ../plugins/imtcp/.libs/imtcp $MainMsgQueueTimeoutShutdown 10000 -$MaxOpenFiles 2000 -$InputTCPMaxSessions 1100 -$InputTCPServerRun 13514 $MainMsgQueueWorkerThreadMinimumMessages 10 $MainMsgQueueWorkerThreads 5 diff --git a/tests/testsuites/x.509/ca-key.pem b/tests/testsuites/x.509/ca-key.pem new file mode 100644 index 00000000..1e1a0b26 --- /dev/null +++ b/tests/testsuites/x.509/ca-key.pem @@ -0,0 +1,15 @@ +-----BEGIN RSA PRIVATE KEY----- +MIICXQIBAAKBgQDZnIJGJH80j2DPBXdxgmFmBRMoBnpwQb8yhRJcJacaWigRAhp4 +wdo07rR+EpuBJHD/5ImIygUwCj/XWAs4JKm3LqK2ih1gUy/s6Tg2O5t3k11kdjEH +MKUxDOLs441dEwERPQtePEoy2POzViIyy959ZJorkdnwC4LBKdQVLEELlwIDAQAB +AoGAEQWvoRoAw1VF3tvQHJZ01Pyno3ViRX63HJYROhkN6b9MrAvsky6iyYo0nzoI +ZQE7P6EaaxNWdYwPs2IlOoaPqeos1sGVDaK/JFuja/DduoXBdCy9RFWRaugDX/1U +iMtjtu29euvegP0r2RIxaIl9dapF5alNH5MLMyBl7XTB+/kCQQDiwHnW8jS1paSc +/risF6Ie5rKuUfVDG8hqMEiKyczSHwUVYushwCclshjM6E1TBFZqMz/8PbFW51pK +OzFS2s6/AkEA9a4044RL3AWe37LIU4hbz2Y+auRvPh8x4i2cWLzdok8Rc1EHDGLN +eHBoOQ3Q2nQS94cOx6HxpRztzBgiwpTRKQJADX9BgV7nbkyO0N2EppG9j7NRvXiZ +bcYwlsmK99/tNjCsf8pkjpy+d8rzGPdW6vMeJbIpQ910OeUJhdOiKvllRwJBAIw3 +rP/dVd5xZseNpj/mp1+rnxwq3EK8UyAfoAgVYvlr3y3NpRQwn8yJezJ07CqB7QFR +F+JgTyZJaH7/l3cusGECQQCM3HmkADAKxX6RwKe8X0Kj/36rjXEMNoq0ZdXOB7Qz +f5N6og4Da9y/ZO+XMo6P3XR/TYIYrMD8nuoR33X69kb1 +-----END RSA PRIVATE KEY----- diff --git a/tests/testsuites/x.509/ca.pem b/tests/testsuites/x.509/ca.pem new file mode 100644 index 00000000..a733eb86 --- /dev/null +++ b/tests/testsuites/x.509/ca.pem @@ -0,0 +1,17 @@ +-----BEGIN CERTIFICATE----- +MIICyzCCAjagAwIBAgIESFo2XjALBgkqhkiG9w0BAQUwezELMAkGA1UEBhMCVVMx +EDAOBgNVBAoTB1NvbWVPcmcxDzANBgNVBAsTBlNvbWVPVTESMBAGA1UEBxMJU29t +ZXdoZXJlMQswCQYDVQQIEwJDQTEoMCYGA1UEAxMfc29tZU5hbWUgKG5vdCBuZWNl +c3NhcmlseSBETlMhKTAeFw0wODA2MTkxMDM1MTJaFw0xODA2MTcxMDM1MjVaMHsx +CzAJBgNVBAYTAlVTMRAwDgYDVQQKEwdTb21lT3JnMQ8wDQYDVQQLEwZTb21lT1Ux +EjAQBgNVBAcTCVNvbWV3aGVyZTELMAkGA1UECBMCQ0ExKDAmBgNVBAMTH3NvbWVO +YW1lIChub3QgbmVjZXNzYXJpbHkgRE5TISkwgZwwCwYJKoZIhvcNAQEBA4GMADCB +iAKBgNmcgkYkfzSPYM8Fd3GCYWYFEygGenBBvzKFElwlpxpaKBECGnjB2jTutH4S +m4EkcP/kiYjKBTAKP9dYCzgkqbcuoraKHWBTL+zpODY7m3eTXWR2MQcwpTEM4uzj +jV0TARE9C148SjLY87NWIjLL3n1kmiuR2fALgsEp1BUsQQuXAgMBAAGjYzBhMA8G +A1UdEwEB/wQFMAMBAf8wHgYDVR0RBBcwFYETc29tZW9uZUBleGFtcGxlLm5ldDAP +BgNVHQ8BAf8EBQMDBwQAMB0GA1UdDgQWBBT7/paNEKc65bcNe0NIhsj4cpl7iTAL +BgkqhkiG9w0BAQUDgYEAlv9ge8Koways837OLoZIam0s7wQCcwd9rWE05caps7BU +T4bfgab9U/e9mmrf3V/zXmtU6y8hhTXF5AcZv3/EmCVwsPRotgrJ+rHXTv5e2PO7 +/8C3K2Lhc89gF4qf4xZwlZU70RasKgCzZa5ivS2Y8pW6LUu6eqqgVw3pPJbW3TE= +-----END CERTIFICATE----- diff --git a/tests/testsuites/x.509/client-cert.pem b/tests/testsuites/x.509/client-cert.pem new file mode 100644 index 00000000..5bf39f81 --- /dev/null +++ b/tests/testsuites/x.509/client-cert.pem @@ -0,0 +1,16 @@ +-----BEGIN CERTIFICATE----- +MIICijCCAfWgAwIBAgIESFo7ITALBgkqhkiG9w0BAQUwezELMAkGA1UEBhMCVVMx +EDAOBgNVBAoTB1NvbWVPcmcxDzANBgNVBAsTBlNvbWVPVTESMBAGA1UEBxMJU29t +ZXdoZXJlMQswCQYDVQQIEwJDQTEoMCYGA1UEAxMfc29tZU5hbWUgKG5vdCBuZWNl +c3NhcmlseSBETlMhKTAeFw0wODA2MTkxMDU1MzJaFw0xMTAzMTYxMDU1MzlaMA0x +CzAJBgNVBAYTAlVTMIGcMAsGCSqGSIb3DQEBAQOBjAAwgYgCgYC+f6yCet2WJgmw +tgukOReI+avRHOfr2hLhIQkSzCOiNi0tNWMKmaQWw/D+y1FvLRq0wLDUyJK/36rB +67HKfscoNeClKTS8jhAs1mPjT57iyuoqK6VW/d2JoofklRCgDIZQrNfxHiOO+kN3 +ShLmkGqxkA3YyUty/JmF6PKWYIhQWQIDAQABo4GPMIGMMAwGA1UdEwEB/wQCMAAw +HQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMB0GA1UdEQQWMBSCEmNsaWVu +dC5leGFtcGxlLm5ldDAdBgNVHQ4EFgQUrDcwsuOF4RiHn0eboCplJSiUhfcwHwYD +VR0jBBgwFoAU+/6WjRCnOuW3DXtDSIbI+HKZe4kwCwYJKoZIhvcNAQEFA4GBAAAh +niy9ORW2AIb6lk/sa3iYczeYpGzxDM9bLZ1xSoIdoHM/v9gPG/WpAZ4ECHjx+Yk8 +4B/9gvaAmMi0FmcoIBQaEOe2P8tcIuzmum3N2F27F2+J4httiNDLJoseWVnXJUvS +dPyVOrKXdl5vVFpmViI5P+VzzMqbAQ6oNlMXIh6e +-----END CERTIFICATE----- diff --git a/tests/testsuites/x.509/client-key.pem b/tests/testsuites/x.509/client-key.pem new file mode 100644 index 00000000..05641213 --- /dev/null +++ b/tests/testsuites/x.509/client-key.pem @@ -0,0 +1,15 @@ +-----BEGIN RSA PRIVATE KEY----- +MIICWwIBAAKBgQC+f6yCet2WJgmwtgukOReI+avRHOfr2hLhIQkSzCOiNi0tNWMK +maQWw/D+y1FvLRq0wLDUyJK/36rB67HKfscoNeClKTS8jhAs1mPjT57iyuoqK6VW +/d2JoofklRCgDIZQrNfxHiOO+kN3ShLmkGqxkA3YyUty/JmF6PKWYIhQWQIDAQAB +AoGAVxrM+BqTIJlC/Ay5lP1QAB9di3ACserUkCFJY1F5h63rCU1sfIfVKl2s3+x6 +z3GZ0QV8tccCpv5wN1x8vqEqkbOvddM3rzpGkEC5PoyfCzuQBun1wnHK/JKjrfk5 +PvcaP60eTNjHZC7w78gOJJCzgzsEMrndtE+55diPmqGVtXMCQQDTZBy5WK8gZwMO +rRz1BKKyBeMYMfTJoJafGfxp0H8AUbTa0V2eb+el3kuzPCm3FQ6IgaHyGj2WqkAw +M0bfAfdXAkEA5rLna1t+2SCtgSd1DotndA4EsH4skBq9kFeD2/8T6Pf13zmBOq6O +4aNEOhgBE/R9/MI4XoU9MbOlkZvKvDuXzwJADdWSb6rXIza6o34+0+Yuw5nRB+dV +DtD8qoLn2wDzHtE6Fcv35YOLVHac26kHTd0J63MYZyDCgRa5Rq5EaBnX1wJAQYRF +XKPbXmZ9X9SI1dyZQMhKZKUwmqw9caSo+e1zBhKFbSOzo6q3QTVQxv7SL4ybyxCN +WaqVOmw+dR+9b7+s2QJAdNAw3r418rWKFKJJNTSqSqr1sYqiKvrQL6w2dpdpAeY4 +3VDCz/7/F9AEn3R7K3fZLQ7W6M62LSEjxxc1Y3LIpQ== +-----END RSA PRIVATE KEY----- diff --git a/tests/testsuites/x.509/machine-cert.pem b/tests/testsuites/x.509/machine-cert.pem new file mode 100644 index 00000000..fa2fd36e --- /dev/null +++ b/tests/testsuites/x.509/machine-cert.pem @@ -0,0 +1,18 @@ +-----BEGIN CERTIFICATE----- +MIIC7TCCAligAwIBAgIESFo4GTALBgkqhkiG9w0BAQUwezELMAkGA1UEBhMCVVMx +EDAOBgNVBAoTB1NvbWVPcmcxDzANBgNVBAsTBlNvbWVPVTESMBAGA1UEBxMJU29t +ZXdoZXJlMQswCQYDVQQIEwJDQTEoMCYGA1UEAxMfc29tZU5hbWUgKG5vdCBuZWNl +c3NhcmlseSBETlMhKTAeFw0wODA2MTkxMDQyNTRaFw0xMTAzMTYxMDQyNTdaMG8x +CzAJBgNVBAYTAlVTMRAwDgYDVQQKEwdTb21lT3JnMQ8wDQYDVQQLEwZTb21lT1Ux +EjAQBgNVBAcTCVNvbWV3aGVyZTELMAkGA1UECBMCQ0ExHDAaBgNVBAMTE21hY2hp +bmUuZXhhbXBsZS5uZXQwgZwwCwYJKoZIhvcNAQEBA4GMADCBiAKBgLJOW6lIHv8u +c6Ez7tiir64vI3aRuDmUACPybyWtyWqrLebzYtg+borWHj9y5di54NB5wpQhZQsQ +U2awNqanzUYeLGqbecbuxuLtsKlZ4knax+PwHOBTmIcN1SjbpII27Toe0VwHE5Vd +sygFFyorto6OeNLPrIcTFfwXQ2sVw325AgMBAAGjgZAwgY0wDAYDVR0TAQH/BAIw +ADAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwHgYDVR0RBBcwFYITbWFj +aGluZS5leGFtcGxlLm5ldDAdBgNVHQ4EFgQUDOHD29GdMfoDWwev4uDvItkLKKww +HwYDVR0jBBgwFoAU+/6WjRCnOuW3DXtDSIbI+HKZe4kwCwYJKoZIhvcNAQEFA4GB +AMt1iED7QzFL2Qk6VivoFY15S2XGF8rJTd3l00bwyLA5qLyLBGlB6z4qkYu7/7SW +5r7tet+1DezgHrj/1eU289m410wnQB8fGwcVLp6OX2PAlhNmVLcsipiN6rielAcP +aIg/VlBtoCFp/ymTLKgvh6DLKWhRUkFPqO2WtcQ3UUo+ +-----END CERTIFICATE----- diff --git a/tests/testsuites/x.509/machine-key.pem b/tests/testsuites/x.509/machine-key.pem new file mode 100644 index 00000000..808f00c9 --- /dev/null +++ b/tests/testsuites/x.509/machine-key.pem @@ -0,0 +1,15 @@ +-----BEGIN RSA PRIVATE KEY----- +MIICXAIBAAKBgQCyTlupSB7/LnOhM+7Yoq+uLyN2kbg5lAAj8m8lrclqqy3m82LY +Pm6K1h4/cuXYueDQecKUIWULEFNmsDamp81GHixqm3nG7sbi7bCpWeJJ2sfj8Bzg +U5iHDdUo26SCNu06HtFcBxOVXbMoBRcqK7aOjnjSz6yHExX8F0NrFcN9uQIDAQAB +AoGABHJs2c95Km8bpikX62I/VG5LiaD/wbvdtwfMWtm3PMhRKEHotLD169OERJvW +fK3CHCD1R+F/ViPNmLGLY2Oq/GqKjhKjg4sqAznw8TImBSgXCFho4sl38z+luP1o +TXFDgfV5HDDW1/F5kJlBIfXBLFdl4VO7E0ZnFt4FqSDRW2MCQQDRun/sBGM4i9hM +QdC+QwrdcgCScBpzbz4YXtI9TyGEqNahg8kXgIVUbzDdRmG68G2M98USzRs5DWB7 +YvYwmRoPAkEA2aUdUpFRb/n7XfsAiFLYOk96C82iCCQpJi0si34zlCAEbCRbQ6zw +gVDMCMSccnnWrVzqtxfN+rXycFTNyDFTtwJAPRwymfrNTnSxGcczo7y1NcE6GXFA +w9HuLfuzFtov0g/AOl/EAG0abHfZrSAM6gOUaDbp3YiWHhGfw1QamB6EUQJAClTb +MnsxeXZNZ2Wt3crI9uOk8IB/a5GD3osQbUK9Yg+vBg8nweuoswrJ1LS4lHqSJUKe +5bgckAUpEAoGhrVIuwJBAKIuqx/cSjF4Oa9xT6DzBRe7vAlKFq62lUV5SLfoSEgY +L5dvPBgAD0Styglny1s0Bu5FTlkxlFOMvUAD/O5hsQw= +-----END RSA PRIVATE KEY----- diff --git a/tests/testsuites/x.509/request.pem b/tests/testsuites/x.509/request.pem new file mode 100644 index 00000000..c612325c --- /dev/null +++ b/tests/testsuites/x.509/request.pem @@ -0,0 +1,10 @@ +-----BEGIN NEW CERTIFICATE REQUEST----- +MIIBWDCBxAIBADANMQswCQYDVQQGEwJVUzCBnDALBgkqhkiG9w0BAQEDgYwAMIGI +AoGAvn+sgnrdliYJsLYLpDkXiPmr0Rzn69oS4SEJEswjojYtLTVjCpmkFsPw/stR +by0atMCw1MiSv9+qweuxyn7HKDXgpSk0vI4QLNZj40+e4srqKiulVv3diaKH5JUQ +oAyGUKzX8R4jjvpDd0oS5pBqsZAN2MlLcvyZhejylmCIUFkCAwEAAaARMA8GCSqG +SIb3DQEJBzECEwAwCwYJKoZIhvcNAQEFA4GBAA6mBaHFuRvcJVNoU7wDFcDexjvC +QLpDpFRSbKcKdNEQLBRD8ZNVOY4WBXQE2pE84//QnygQPKPCHSqUVdPPBabi5y2E +A2XvgYyKsrFbsrpKrVkPz5oQB4V7FRytQaQoBi//BSOu3dMaimLcAhfNQZCrQeu8 +SYWdJi5OPvrYGvgT +-----END NEW CERTIFICATE REQUEST----- diff --git a/tests/threadingmq.sh b/tests/threadingmq.sh index 5c29ec60..ea5d7837 100755 --- a/tests/threadingmq.sh +++ b/tests/threadingmq.sh @@ -6,10 +6,11 @@ # in practice many threading bugs result in an abort rather quickly and these # should be covered by this test here. # rgerhards, 2009-06-26 -echo TEST: threadingmq.sh - main queue concurrency +echo \[threadingmq.sh\]: main queue concurrency source $srcdir/diag.sh init source $srcdir/diag.sh startup threadingmq.conf -source $srcdir/diag.sh tcpflood 127.0.0.1 13514 2 100000 +source $srcdir/diag.sh injectmsg 0 100000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 99999 source $srcdir/diag.sh exit diff --git a/tests/threadingmqaq.sh b/tests/threadingmqaq.sh index 009551fd..83f5fedc 100755 --- a/tests/threadingmqaq.sh +++ b/tests/threadingmqaq.sh @@ -6,10 +6,13 @@ # in practice many threading bugs result in an abort rather quickly and these # should be covered by this test here. # rgerhards, 2009-06-26 -echo TEST: threadingmqaq.sh - main/action queue concurrency +echo \[threadingmqaq.sh\]: main/action queue concurrency source $srcdir/diag.sh init source $srcdir/diag.sh startup threadingmqaq.conf -source $srcdir/diag.sh tcpflood 127.0.0.1 13514 2 100000 +#source $srcdir/diag.sh tcpflood 127.0.0.1 13514 2 100000 +#source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh injectmsg 0 100000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 99999 source $srcdir/diag.sh exit diff --git a/tests/timestamp.sh b/tests/timestamp.sh index 7699a4af..71416c33 100755 --- a/tests/timestamp.sh +++ b/tests/timestamp.sh @@ -1,4 +1,4 @@ -echo various timestamp tests +echo \[timestamp.sh\]: various timestamp tests source $srcdir/diag.sh init source $srcdir/diag.sh nettester ts3164 udp source $srcdir/diag.sh nettester ts3164 tcp diff --git a/tests/validation-run.sh b/tests/validation-run.sh index 2e922283..cc29482a 100755 --- a/tests/validation-run.sh +++ b/tests/validation-run.sh @@ -21,6 +21,7 @@ # # A copy of the GPL can be found in the file "COPYING" in this distribution. #set -x +echo \[validation-run.sh\]: testing configuraton validation echo "testing a failed configuration verification run" ../tools/rsyslogd -dn -u2 -c4 -N1 -f$srcdir/testsuites/invalid.conf -M../runtime/.libs:../.libs if [ $? -ne 1 ]; then @@ -5,7 +5,7 @@ * * File begun on 2007-12-14 by RGerhards * - * Copyright 2007 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007, 2009 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -29,6 +29,7 @@ #include <stdlib.h> #include <string.h> #include <signal.h> +#include <errno.h> #include <pthread.h> #include <assert.h> @@ -36,6 +37,7 @@ #include "dirty.h" #include "linkedlist.h" #include "threads.h" +#include "srUtils.h" /* linked list of currently-known threads */ static linkedList_t llThrds; @@ -44,7 +46,8 @@ static linkedList_t llThrds; /* Construct a new thread object */ -static rsRetVal thrdConstruct(thrdInfo_t **ppThis) +static rsRetVal +thrdConstruct(thrdInfo_t **ppThis) { DEFiRet; thrdInfo_t *pThis; @@ -52,13 +55,8 @@ static rsRetVal thrdConstruct(thrdInfo_t **ppThis) assert(ppThis != NULL); CHKmalloc(pThis = calloc(1, sizeof(thrdInfo_t))); - - /* OK, we got the element, now initialize members that should - * not be zero-filled. - */ - pThis->mutTermOK = (pthread_mutex_t *) malloc (sizeof (pthread_mutex_t)); - pthread_mutex_init (pThis->mutTermOK, NULL); - + pthread_mutex_init(&pThis->mutThrd, NULL); + pthread_cond_init(&pThis->condThrdTerm, NULL); *ppThis = pThis; finalize_it: @@ -78,13 +76,54 @@ static rsRetVal thrdDestruct(thrdInfo_t *pThis) if(pThis->bIsActive == 1) { thrdTerminate(pThis); } - free(pThis->mutTermOK); + pthread_mutex_destroy(&pThis->mutThrd); + pthread_cond_destroy(&pThis->condThrdTerm); free(pThis); RETiRet; } +/* terminate a thread via the non-cancel interface + * This is a separate function as it involves a bit more of code. + * rgerhads, 2009-10-15 + */ +static inline rsRetVal +thrdTerminateNonCancel(thrdInfo_t *pThis) +{ + struct timespec tTimeout; + int ret; + DEFiRet; + assert(pThis != NULL); + + DBGPRINTF("request term via SIGTTIN for input thread 0x%x\n", (unsigned) pThis->thrdID); + pThis->bShallStop = TRUE; + do { + d_pthread_mutex_lock(&pThis->mutThrd); + pthread_kill(pThis->thrdID, SIGTTIN); + timeoutComp(&tTimeout, 10); /* a fixed 10ms timeout, do after lock (may take long!) */ + ret = d_pthread_cond_timedwait(&pThis->condThrdTerm, &pThis->mutThrd, &tTimeout); + d_pthread_mutex_unlock(&pThis->mutThrd); + if(Debug) { + if(ret == ETIMEDOUT) { + dbgprintf("input thread term: had a timeout waiting on thread termination\n"); + } else if(ret == 0) { + dbgprintf("input thread term: thread returned normally and is terminated\n"); + } else { + char errStr[1024]; + int err = errno; + rs_strerror_r(err, errStr, sizeof(errStr)); + dbgprintf("input thread term: cond_wait returned with error %d: %s\n", + err, errStr); + } + } + } while(pThis->bIsActive); + DBGPRINTF("non-cancel input thread termination succeeded for thread 0x%x\n", (unsigned) pThis->thrdID); + + RETiRet; +} + + /* terminate a thread gracefully. */ rsRetVal thrdTerminate(thrdInfo_t *pThis) @@ -92,21 +131,14 @@ rsRetVal thrdTerminate(thrdInfo_t *pThis) DEFiRet; assert(pThis != NULL); -#if 0 // TODO: somehow does not really work yet! if(pThis->bNeedsCancel) { -#endif DBGPRINTF("request term via canceling for input thread 0x%x\n", (unsigned) pThis->thrdID); pthread_cancel(pThis->thrdID); -#if 0 // TODO: somehow does not really work yet! - if(pThis->bNeedsCancel) { + pThis->bIsActive = 0; } else { - - DBGPRINTF("request term via SIGTTIN for input thread 0x%x\n", (unsigned) pThis->thrdID); - pthread_kill(pThis->thrdID, SIGTTIN); + thrdTerminateNonCancel(pThis); } -#endif pthread_join(pThis->thrdID, NULL); /* wait for input thread to complete */ - pThis->bIsActive = 0; /* call cleanup function, if any */ if(pThis->pAfterRun != NULL) @@ -157,6 +189,16 @@ static void* thrdStarter(void *arg) iRet = pThis->pUsrThrdMain(pThis); dbgprintf("thrdStarter: usrThrdMain 0x%lx returned with iRet %d, exiting now.\n", (unsigned long) pThis->thrdID, iRet); + + /* signal master control that we exit (we do the mutex lock mostly to + * keep the thread debugger happer, it would not really be necessary with + * the logic we employ...) + */ + pThis->bIsActive = 0; + d_pthread_mutex_lock(&pThis->mutThrd); + pthread_cond_signal(&pThis->condThrdTerm); + d_pthread_mutex_unlock(&pThis->mutThrd); + ENDfunc pthread_exit(0); } @@ -203,37 +245,10 @@ rsRetVal thrdInit(void) rsRetVal thrdExit(void) { DEFiRet; - iRet = llDestroy(&llThrds); - - RETiRet; -} - - -/* thrdSleep() - a fairly portable way to put a thread to sleep. It - * will wake up when - * a) the wake-time is over - * b) the thread shall be terminated - * Returns RS_RET_OK if all went well, RS_RET_TERMINATE_NOW if the calling - * thread shall be terminated and any other state if an error happened. - * rgerhards, 2007-12-17 - */ -rsRetVal -thrdSleep(thrdInfo_t *pThis, int iSeconds, int iuSeconds) -{ - DEFiRet; - struct timeval tvSelectTimeout; - - assert(pThis != NULL); - tvSelectTimeout.tv_sec = iSeconds; - tvSelectTimeout.tv_usec = iuSeconds; /* micro seconds */ - select(1, NULL, NULL, NULL, &tvSelectTimeout); - if(pThis->bShallStop) - iRet = RS_RET_TERMINATE_NOW; RETiRet; } -/* - * vi:set ai: +/* vi:set ai: */ @@ -25,7 +25,8 @@ /* the thread object */ struct thrdInfo { - pthread_mutex_t *mutTermOK; /* Is it ok to terminate that thread now? */ + pthread_mutex_t mutThrd;/* mutex for handling long-running operations and shutdown */ + pthread_cond_t condThrdTerm;/* condition: thread terminates (used just for shutdown loop) */ int bIsActive; /* Is thread running? */ int bShallStop; /* set to 1 if the thread should be stopped ? */ rsRetVal (*pUsrThrdMain)(struct thrdInfo*); /* user thread main to be called in new thread */ @@ -40,7 +41,6 @@ rsRetVal thrdInit(void); rsRetVal thrdTerminate(thrdInfo_t *pThis); rsRetVal thrdTerminateAll(void); rsRetVal thrdCreate(rsRetVal (*thrdMain)(thrdInfo_t*), rsRetVal(*afterRun)(thrdInfo_t *), bool); -rsRetVal thrdSleep(thrdInfo_t *pThis, int iSeconds, int iuSeconds); /* macros (replace inline functions) */ diff --git a/tools/Makefile.am b/tools/Makefile.am index f0f9afab..32d6843b 100644 --- a/tools/Makefile.am +++ b/tools/Makefile.am @@ -15,6 +15,10 @@ rsyslogd_SOURCES = \ omfile.h \ omdiscard.c \ omdiscard.h \ + pmrfc5424.c \ + pmrfc5424.h \ + pmrfc3164.c \ + pmrfc3164.h \ iminternal.c \ iminternal.h \ pidfile.c \ diff --git a/tools/iminternal.c b/tools/iminternal.c index 0ceff3d8..bd1fa128 100644 --- a/tools/iminternal.c +++ b/tools/iminternal.c @@ -89,7 +89,7 @@ finalize_it: * The interface of this function is modelled after syslogd/logmsg(), * for which it is an "replacement". */ -rsRetVal iminternalAddMsg(int pri, msg_t *pMsg, int flags) +rsRetVal iminternalAddMsg(int pri, msg_t *pMsg) { DEFiRet; iminternal_t *pThis; @@ -100,7 +100,6 @@ rsRetVal iminternalAddMsg(int pri, msg_t *pMsg, int flags) pThis->pri = pri; pThis->pMsg = pMsg; - pThis->flags = flags; CHKiRet(llAppend(&llMsgs, NULL, (void*) pThis)); @@ -119,7 +118,7 @@ finalize_it: * from the list and return it to the caller. The caller is * responsible for freeing the message! */ -rsRetVal iminternalRemoveMsg(int *pPri, msg_t **ppMsg, int *pFlags) +rsRetVal iminternalRemoveMsg(int *pPri, msg_t **ppMsg) { DEFiRet; iminternal_t *pThis; @@ -127,11 +126,9 @@ rsRetVal iminternalRemoveMsg(int *pPri, msg_t **ppMsg, int *pFlags) assert(pPri != NULL); assert(ppMsg != NULL); - assert(pFlags != NULL); CHKiRet(llGetNextElt(&llMsgs, &llCookie, (void*)&pThis)); *pPri = pThis->pri; - *pFlags = pThis->flags; *ppMsg = pThis->pMsg; pThis->pMsg = NULL; /* we do no longer own it - important for destructor */ diff --git a/tools/iminternal.h b/tools/iminternal.h index 8dc0f171..f1062a15 100644 --- a/tools/iminternal.h +++ b/tools/iminternal.h @@ -35,15 +35,14 @@ struct iminternal_s { /* config file sysline parse entry */ int pri; msg_t *pMsg; /* the message (in all its glory) */ - int flags; }; typedef struct iminternal_s iminternal_t; /* prototypes */ rsRetVal modInitIminternal(void); rsRetVal modExitIminternal(void); -rsRetVal iminternalAddMsg(int pri, msg_t *pMsg, int flags); +rsRetVal iminternalAddMsg(int pri, msg_t *pMsg); rsRetVal iminternalHaveMsgReady(int* pbHaveOne); -rsRetVal iminternalRemoveMsg(int *pPri, msg_t **ppMsg, int *pFlags); +rsRetVal iminternalRemoveMsg(int *pPri, msg_t **ppMsg); #endif /* #ifndef IMINTERNAL_H_INCLUDED */ diff --git a/tools/omfile.c b/tools/omfile.c index 9d6b3d1d..f4945ba0 100644 --- a/tools/omfile.c +++ b/tools/omfile.c @@ -1,8 +1,6 @@ /* omfile.c * This is the implementation of the build-in file output module. * - * Handles: eTypeCONSOLE, eTypeTTY, eTypeFILE, eTypePIPE - * * NOTE: read comments in module-template.h to understand how this file * works! * @@ -76,12 +74,30 @@ DEF_OMOD_STATIC_DATA DEFobjCurrIf(errmsg) DEFobjCurrIf(strm) +/* for our current LRU mechanism, we need a monotonically increasing counters. We use + * it much like a "Lamport logical clock": we do not need the actual time, we just need + * to know the sequence in which files were accessed. So we use a simple counter to + * create that sequence. We use an unsigned 64 bit value which is extremely unlike to + * wrap within the lifetime of a process. If we process 1,000,000 file writes per + * second, the process could still exist over 500,000 years before a wrap to 0 happens. + * That should be sufficient (and even than, there would no really bad effect ;)). + * The variable below is the global counter/clock. + */ +static uint64 clockFileAccess = 0; +/* and the "tick" function */ +static inline uint64 +getClockFileAccess(void) +{ + return ATOMIC_INC_AND_FETCH(clockFileAccess); +} + + /* The following structure is a dynafile name cache entry. */ struct s_dynaFileCacheEntry { uchar *pName; /* name currently open, if dynamic name */ strm_t *pStrm; /* our output stream */ - time_t lastUsed; /* for LRU - last access */ // TODO: perforamcne change to counter (see other comment!) + uint64 clkTickAccessed;/* for LRU - based on clockFileAccess */ }; typedef struct s_dynaFileCacheEntry dynaFileCacheEntry; @@ -89,11 +105,13 @@ typedef struct s_dynaFileCacheEntry dynaFileCacheEntry; #define IOBUF_DFLT_SIZE 1024 /* default size for io buffers */ #define FLUSH_INTRVL_DFLT 1 /* default buffer flush interval (in seconds) */ +#define DFLT_bForceChown 0 /* globals for default values */ static int iDynaFileCacheSize = 10; /* max cache for dynamic files */ static int fCreateMode = 0644; /* mode to use when creating files */ static int fDirCreateMode = 0700; /* mode to use when creating files */ static int bFailOnChown; /* fail if chown fails? */ +static int bForceChown = DFLT_bForceChown; /* Force chown() on existing files? */ static uid_t fileUID; /* UID to be used for newly created files */ static uid_t fileGID; /* GID to be used for newly created files */ static uid_t dirUID; /* UID to be used for newly created directories */ @@ -116,6 +134,7 @@ typedef struct _instanceData { int fDirCreateMode; /* creation mode for mkdir() */ int bCreateDirs; /* auto-create directories? */ int bSyncFile; /* should the file by sync()'ed? 1- yes, 0- no */ + bool bForceChown; /* force chown() on existing files? */ uid_t fileUID; /* IDs for creation */ uid_t dirUID; gid_t fileGID; @@ -153,6 +172,7 @@ CODESTARTdbgPrintInstInfo "\tcreate directories: %s\n" "\tfile owner %d, group %d\n" "\tdirectory owner %d, group %d\n" + "\tforce chown() for all files: %s\n" "\tdir create mode 0%3.3o, file create mode 0%3.3o\n" "\tfail if owner/group can not be set: %s\n", pData->f_fname, @@ -160,6 +180,7 @@ CODESTARTdbgPrintInstInfo pData->bCreateDirs ? "yes" : "no", pData->fileUID, pData->fileGID, pData->dirUID, pData->dirGID, + pData->bForceChown ? "yes" : "no", pData->fDirCreateMode, pData->fCreateMode, pData->bFailOnChown ? "yes" : "no" ); @@ -349,7 +370,22 @@ prepareFile(instanceData *pData, uchar *newFileName) int fd; DEFiRet; - if(access((char*)newFileName, F_OK) != 0) { + if(access((char*)newFileName, F_OK) == 0) { + if(pData->bForceChown) { + /* Try to fix wrong ownership set by someone else. Note that this code + * will no longer work once we have made the $PrivDrop code fully secure. + * This change is based on an idea of Michael Terry, provided as part of + * the effort to make rsyslogd the Ubuntu default syslogd. + * rgerhards, 2009-09-11 + */ + if(chown((char*)newFileName, pData->fileUID, pData->fileGID) != 0) { + if(pData->bFailOnChown) { + int eSave = errno; + errno = eSave; + } + } + } + } else { /* file does not exist, create it (and eventually parent directories */ fd = -1; if(pData->bCreateDirs) { @@ -370,7 +406,7 @@ prepareFile(instanceData *pData, uchar *newFileName) pData->fCreateMode); if(fd != -1) { /* check and set uid/gid */ - if(pData->fileUID != (uid_t)-1 || pData->fileGID != (gid_t) -1) { + if(pData->bForceChown || pData->fileUID != (uid_t)-1 || pData->fileGID != (gid_t) -1) { /* we need to set owner/group */ if(fchown(fd, pData->fileUID, pData->fileGID) != 0) { if(pData->bFailOnChown) { @@ -435,7 +471,7 @@ finalize_it: static inline rsRetVal prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) { - time_t ttOldest; /* timestamp of oldest element */ + uint64 ctOldest; /* "timestamp" of oldest element */ int iOldest; int i; int iFirstFree; @@ -454,7 +490,7 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) if( (pData->iCurrElt != -1) && !ustrcmp(newFileName, pCache[pData->iCurrElt]->pName)) { /* great, we are all set */ - pCache[pData->iCurrElt]->lastUsed = time(NULL); /* update timestamp for LRU */ // TODO: optimize time call! + pCache[pData->iCurrElt]->clkTickAccessed = getClockFileAccess(); // LRU needs only a strictly monotonically increasing counter, so such a one could do FINALIZE; } @@ -465,7 +501,7 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) pData->iCurrElt = -1; /* invalid current element pointer */ iFirstFree = -1; /* not yet found */ iOldest = 0; /* we assume the first element to be the oldest - that will change as we loop */ - ttOldest = time(NULL) + 1; /* there must always be an older one */ + ctOldest = getClockFileAccess(); /* there must always be an older one */ for(i = 0 ; i < pData->iCurrCacheSize ; ++i) { if(pCache[i] == NULL) { if(iFirstFree == -1) @@ -475,12 +511,12 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) /* we found our element! */ pData->pStrm = pCache[i]->pStrm; pData->iCurrElt = i; - pCache[i]->lastUsed = time(NULL); /* update timestamp for LRU */ + pCache[i]->clkTickAccessed = getClockFileAccess(); /* update "timestamp" for LRU */ FINALIZE; } /* did not find it - so lets keep track of the counters for LRU */ - if(pCache[i]->lastUsed < ttOldest) { - ttOldest = pCache[i]->lastUsed; + if(pCache[i]->clkTickAccessed < ctOldest) { + ctOldest = pCache[i]->clkTickAccessed; iOldest = i; } } @@ -520,7 +556,7 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) CHKmalloc(pCache[iFirstFree]->pName = ustrdup(newFileName)); pCache[iFirstFree]->pStrm = pData->pStrm; - pCache[iFirstFree]->lastUsed = time(NULL); // monotonically increasing value! TODO: performance + pCache[iFirstFree]->clkTickAccessed = getClockFileAccess(); pData->iCurrElt = iFirstFree; DBGPRINTF("Added new entry %d for file cache, file '%s'.\n", iFirstFree, newFileName); @@ -604,12 +640,26 @@ BEGINtryResume CODESTARTtryResume ENDtryResume +BEGINbeginTransaction +CODESTARTbeginTransaction + /* we have nothing to do to begin a transaction */ +ENDbeginTransaction + + +BEGINendTransaction +CODESTARTendTransaction + if(pData->bFlushOnTXEnd) { + CHKiRet(strm.Flush(pData->pStrm)); + } +finalize_it: +ENDendTransaction + + BEGINdoAction CODESTARTdoAction DBGPRINTF("file to log to: %s\n", pData->f_fname); CHKiRet(writeFile(ppString, iMsgOpts, pData)); - if(pData->bFlushOnTXEnd) { - /* TODO v5: do this in endTransaction only! */ + if(!bCoreSupportsBatching && pData->bFlushOnTXEnd) { CHKiRet(strm.Flush(pData->pStrm)); } finalize_it: @@ -667,6 +717,11 @@ CODESTARTparseSelectorAct case '|': case '/': CODE_STD_STRING_REQUESTparseSelectorAct(1) + /* we now have the same semantics for files and pipes, but we need to skip over + * the pipe indicator traditionally seen in config files... + */ + if(*p == '|') + ++p; CHKiRet(cflineParseFileName(p, (uchar*) pData->f_fname, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, (pszTplName == NULL) ? (uchar*)"RSYSLOG_FileFormat" : pszTplName)); pData->bDynamicName = 0; @@ -681,6 +736,7 @@ CODESTARTparseSelectorAct pData->fDirCreateMode = fDirCreateMode; pData->bCreateDirs = bCreateDirs; pData->bFailOnChown = bFailOnChown; + pData->bForceChown = bForceChown; pData->fileUID = fileUID; pData->fileGID = fileGID; pData->dirUID = dirUID; @@ -715,6 +771,7 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a dirUID = -1; dirGID = -1; bFailOnChown = 1; + bForceChown = DFLT_bForceChown; iDynaFileCacheSize = 10; fCreateMode = 0644; fDirCreateMode = 0700; @@ -758,6 +815,7 @@ ENDmodExit BEGINqueryEtryPt CODESTARTqueryEtryPt CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_TXIF_OMOD_QUERIES /* we support the transactional interface! */ CODEqueryEtryPt_doHUP ENDqueryEtryPt @@ -768,6 +826,8 @@ CODESTARTmodInit CODEmodInit_QueryRegCFSLineHdlr CHKiRet(objUse(errmsg, CORE_COMPONENT)); CHKiRet(objUse(strm, CORE_COMPONENT)); + INITChkCoreFeature(bCoreSupportsBatching, CORE_FEATURE_BATCHING); + DBGPRINTF("omfile: %susing transactional output interface.\n", bCoreSupportsBatching ? "" : "not "); CHKiRet(omsdRegCFSLineHdlr((uchar *)"dynafilecachesize", 0, eCmdHdlrInt, (void*) setDynaFileCacheSize, NULL, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"omfileziplevel", 0, eCmdHdlrInt, NULL, &iZipLevel, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"omfileflushinterval", 0, eCmdHdlrInt, NULL, &iFlushInterval, STD_LOADABLE_MODULE_ID)); @@ -781,6 +841,7 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(omsdRegCFSLineHdlr((uchar *)"filecreatemode", 0, eCmdHdlrFileCreateMode, NULL, &fCreateMode, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"createdirs", 0, eCmdHdlrBinary, NULL, &bCreateDirs, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"failonchownfailure", 0, eCmdHdlrBinary, NULL, &bFailOnChown, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"omfileForceChown", 0, eCmdHdlrBinary, NULL, &bForceChown, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionfileenablesync", 0, eCmdHdlrBinary, NULL, &bEnableSync, STD_LOADABLE_MODULE_ID)); CHKiRet(regCfSysLineHdlr((uchar *)"actionfiledefaulttemplate", 0, eCmdHdlrGetWord, NULL, &pszTplName, NULL)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID)); diff --git a/tools/omfwd.c b/tools/omfwd.c index 38d9cfa6..76beb486 100644 --- a/tools/omfwd.c +++ b/tools/omfwd.c @@ -436,13 +436,13 @@ CODESTARTdoAction * hard-coded but this may be changed to a config parameter. * rgerhards, 2006-11-30 */ - if(pData->compressionLevel && (l > MIN_SIZE_FOR_COMPRESS)) { + if(pData->compressionLevel && (l > CONF_MIN_SIZE_FOR_COMPRESS)) { Bytef *out; uLongf destLen = iMaxLine + iMaxLine/100 +12; /* recommended value from zlib doc */ uLong srcLen = l; int ret; /* TODO: optimize malloc sequence? -- rgerhards, 2008-09-02 */ - CHKmalloc(out = (Bytef*) malloc(destLen)); + CHKmalloc(out = (Bytef*) MALLOC(destLen)); out[0] = 'z'; out[1] = '\0'; ret = compress2((Bytef*) out+1, &destLen, (Bytef*) psz, @@ -617,7 +617,7 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) tmp = ++p; for(i=0 ; *p && isdigit((int) *p) ; ++p, ++i) /* SKIP AND COUNT */; - pData->port = malloc(i + 1); + pData->port = MALLOC(i + 1); if(pData->port == NULL) { errmsg.LogError(0, NO_ERRCODE, "Could not get memory to store syslog forwarding port, " "using default port, results may not be what you intend\n"); diff --git a/tools/pmrfc3164.c b/tools/pmrfc3164.c new file mode 100644 index 00000000..5b684af5 --- /dev/null +++ b/tools/pmrfc3164.c @@ -0,0 +1,232 @@ +/* pmrfc3164.c + * This is a parser module for RFC3164(legacy syslog)-formatted messages. + * + * NOTE: read comments in module-template.h to understand how this file + * works! + * + * File begun on 2009-11-04 by RGerhards + * + * Copyright 2007, 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <errno.h> +#include <ctype.h> +#include "syslogd.h" +#include "conf.h" +#include "syslogd-types.h" +#include "template.h" +#include "msg.h" +#include "module-template.h" +#include "glbl.h" +#include "errmsg.h" +#include "parser.h" +#include "datetime.h" +#include "unicode-helper.h" + +MODULE_TYPE_PARSER +PARSER_NAME("rsyslog.rfc3164") + +/* internal structures + */ +DEF_PMOD_STATIC_DATA +DEFobjCurrIf(errmsg) +DEFobjCurrIf(glbl) +DEFobjCurrIf(parser) +DEFobjCurrIf(datetime) + + +/* static data */ +static int bParseHOSTNAMEandTAG; /* cache for the equally-named global param - performance enhancement */ + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature + if(eFeat == sFEATUREAutomaticSanitazion) + iRet = RS_RET_OK; + if(eFeat == sFEATUREAutomaticPRIParsing) + iRet = RS_RET_OK; +ENDisCompatibleWithFeature + + +/* parse a legay-formatted syslog message. + */ +BEGINparse + uchar *p2parse; + int lenMsg; + int bTAGCharDetected; + int i; /* general index for parsing */ + uchar bufParseTAG[CONF_TAG_MAXSIZE]; + uchar bufParseHOSTNAME[CONF_TAG_HOSTNAME]; +CODESTARTparse + dbgprintf("Message will now be parsed by the legacy syslog parser (one size fits all... ;)).\n"); + assert(pMsg != NULL); + assert(pMsg->pszRawMsg != NULL); + lenMsg = pMsg->iLenRawMsg - (pMsg->offAfterPRI + 1); + p2parse = pMsg->pszRawMsg + pMsg->offAfterPRI; /* point to start of text, after PRI */ + setProtocolVersion(pMsg, 0); + + /* Check to see if msg contains a timestamp. We start by assuming + * that the message timestamp is the time of reception (which we + * generated ourselfs and then try to actually find one inside the + * message. There we go from high-to low precison and are done + * when we find a matching one. -- rgerhards, 2008-09-16 + */ + if(datetime.ParseTIMESTAMP3339(&(pMsg->tTIMESTAMP), &p2parse, &lenMsg) == RS_RET_OK) { + /* we are done - parse pointer is moved by ParseTIMESTAMP3339 */; + } else if(datetime.ParseTIMESTAMP3164(&(pMsg->tTIMESTAMP), &p2parse, &lenMsg) == RS_RET_OK) { + /* we are done - parse pointer is moved by ParseTIMESTAMP3164 */; + } else if(*p2parse == ' ' && lenMsg > 1) { /* try to see if it is slighly malformed - HP procurve seems to do that sometimes */ + ++p2parse; /* move over space */ + --lenMsg; + if(datetime.ParseTIMESTAMP3164(&(pMsg->tTIMESTAMP), &p2parse, &lenMsg) == RS_RET_OK) { + /* indeed, we got it! */ + /* we are done - parse pointer is moved by ParseTIMESTAMP3164 */; + } else {/* parse pointer needs to be restored, as we moved it off-by-one + * for this try. + */ + --p2parse; + ++lenMsg; + } + } + + if(pMsg->msgFlags & IGNDATE) { + /* we need to ignore the msg data, so simply copy over reception date */ + memcpy(&pMsg->tTIMESTAMP, &pMsg->tRcvdAt, sizeof(struct syslogTime)); + } + + /* rgerhards, 2006-03-13: next, we parse the hostname and tag. But we + * do this only when the user has not forbidden this. I now introduce some + * code that allows a user to configure rsyslogd to treat the rest of the + * message as MSG part completely. In this case, the hostname will be the + * machine that we received the message from and the tag will be empty. This + * is meant to be an interim solution, but for now it is in the code. + */ + if(bParseHOSTNAMEandTAG && !(pMsg->msgFlags & INTERNAL_MSG)) { + /* parse HOSTNAME - but only if this is network-received! + * rger, 2005-11-14: we still have a problem with BSD messages. These messages + * do NOT include a host name. In most cases, this leads to the TAG to be treated + * as hostname and the first word of the message as the TAG. Clearly, this is not + * of advantage ;) I think I have now found a way to handle this situation: there + * are certain characters which are frequently used in TAG (e.g. ':'), which are + * *invalid* in host names. So while parsing the hostname, I check for these characters. + * If I find them, I set a simple flag but continue. After parsing, I check the flag. + * If it was set, then we most probably do not have a hostname but a TAG. Thus, I change + * the fields. I think this logic shall work with any type of syslog message. + * rgerhards, 2009-06-23: and I now have extended this logic to every character + * that is not a valid hostname. + */ + bTAGCharDetected = 0; + if(lenMsg > 0 && pMsg->msgFlags & PARSE_HOSTNAME) { + i = 0; + while(i < lenMsg && (isalnum(p2parse[i]) || p2parse[i] == '.' || p2parse[i] == '.' + || p2parse[i] == '_' || p2parse[i] == '-') && i < CONF_TAG_MAXSIZE) { + bufParseHOSTNAME[i] = p2parse[i]; + ++i; + } + + if(i > 0 && p2parse[i] == ' ' && isalnum(p2parse[i-1])) { + /* we got a hostname! */ + p2parse += i + 1; /* "eat" it (including SP delimiter) */ + lenMsg -= i + 1; + bufParseHOSTNAME[i] = '\0'; + MsgSetHOSTNAME(pMsg, bufParseHOSTNAME, i); + } + } + + /* now parse TAG - that should be present in message from all sources. + * This code is somewhat not compliant with RFC 3164. As of 3164, + * the TAG field is ended by any non-alphanumeric character. In + * practice, however, the TAG often contains dashes and other things, + * which would end the TAG. So it is not desirable. As such, we only + * accept colon and SP to be terminators. Even there is a slight difference: + * a colon is PART of the TAG, while a SP is NOT part of the tag + * (it is CONTENT). Starting 2008-04-04, we have removed the 32 character + * size limit (from RFC3164) on the tag. This had bad effects on existing + * envrionments, as sysklogd didn't obey it either (probably another bug + * in RFC3164...). We now receive the full size, but will modify the + * outputs so that only 32 characters max are used by default. + */ + i = 0; + while(lenMsg > 0 && *p2parse != ':' && *p2parse != ' ' && i < CONF_TAG_MAXSIZE) { + bufParseTAG[i++] = *p2parse++; + --lenMsg; + } + if(lenMsg > 0 && *p2parse == ':') { + ++p2parse; + --lenMsg; + bufParseTAG[i++] = ':'; + } + + /* no TAG can only be detected if the message immediatly ends, in which case an empty TAG + * is considered OK. So we do not need to check for empty TAG. -- rgerhards, 2009-06-23 + */ + bufParseTAG[i] = '\0'; /* terminate string */ + MsgSetTAG(pMsg, bufParseTAG, i); + } else {/* we enter this code area when the user has instructed rsyslog NOT + * to parse HOSTNAME and TAG - rgerhards, 2006-03-13 + */ + if(!(pMsg->msgFlags & INTERNAL_MSG)) { + DBGPRINTF("HOSTNAME and TAG not parsed by user configuraton.\n"); + } + } + + /* The rest is the actual MSG */ + MsgSetMSGoffs(pMsg, p2parse - pMsg->pszRawMsg); +ENDparse + + +BEGINmodExit +CODESTARTmodExit + /* release what we no longer need */ + objRelease(errmsg, CORE_COMPONENT); + objRelease(glbl, CORE_COMPONENT); + objRelease(parser, CORE_COMPONENT); + objRelease(datetime, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_PMOD_QUERIES +CODEqueryEtryPt_IsCompatibleWithFeature_IF_OMOD_QUERIES +ENDqueryEtryPt + + +BEGINmodInit(pmrfc3164) +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(parser, CORE_COMPONENT)); + CHKiRet(objUse(datetime, CORE_COMPONENT)); + + dbgprintf("rfc3164 parser init called\n"); + bParseHOSTNAMEandTAG = glbl.GetParseHOSTNAMEandTAG(); /* cache value, is set only during rsyslogd option processing */ + + +ENDmodInit + +/* vim:set ai: + */ diff --git a/tools/pmrfc3164.h b/tools/pmrfc3164.h new file mode 100644 index 00000000..24304094 --- /dev/null +++ b/tools/pmrfc3164.h @@ -0,0 +1,33 @@ +/* pmrfc3164.h + * These are the definitions for the RFC3164 parser module. + * + * File begun on 2009-11-04 by RGerhards + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +#ifndef PMRFC3164_H_INCLUDED +#define PMRFC3164_H_INCLUDED 1 + +/* prototypes */ +rsRetVal modInitpmrfc3164(int iIFVersRequested __attribute__((unused)), int *ipIFVersProvided, rsRetVal (**pQueryEtryPt)(), rsRetVal (*pHostQueryEtryPt)(uchar*, rsRetVal (**)()), modInfo_t*); + +#endif /* #ifndef PMRFC3164_H_INCLUDED */ +/* vi:set ai: + */ diff --git a/tools/pmrfc5424.c b/tools/pmrfc5424.c new file mode 100644 index 00000000..07994ade --- /dev/null +++ b/tools/pmrfc5424.c @@ -0,0 +1,329 @@ +/* pmrfc5424.c + * This is a parser module for RFC5424-formatted messages. + * + * NOTE: read comments in module-template.h to understand how this file + * works! + * + * File begun on 2009-11-03 by RGerhards + * + * Copyright 2007, 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <errno.h> +#include "syslogd.h" +#include "conf.h" +#include "syslogd-types.h" +#include "template.h" +#include "msg.h" +#include "module-template.h" +#include "glbl.h" +#include "errmsg.h" +#include "parser.h" +#include "datetime.h" +#include "unicode-helper.h" + +MODULE_TYPE_PARSER +PARSER_NAME("rsyslog.rfc5424") + +/* internal structures + */ +DEF_PMOD_STATIC_DATA +DEFobjCurrIf(errmsg) +DEFobjCurrIf(glbl) +DEFobjCurrIf(parser) +DEFobjCurrIf(datetime) + + +/* config data */ + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature + if(eFeat == sFEATUREAutomaticSanitazion) + iRet = RS_RET_OK; + if(eFeat == sFEATUREAutomaticPRIParsing) + iRet = RS_RET_OK; +ENDisCompatibleWithFeature + + +/* Helper to parseRFCSyslogMsg. This function parses a field up to + * (and including) the SP character after it. The field contents is + * returned in a caller-provided buffer. The parsepointer is advanced + * to after the terminating SP. The caller must ensure that the + * provided buffer is large enough to hold the to be extracted value. + * Returns 0 if everything is fine or 1 if either the field is not + * SP-terminated or any other error occurs. -- rger, 2005-11-24 + * The function now receives the size of the string and makes sure + * that it does not process more than that. The *pLenStr counter is + * updated on exit. -- rgerhards, 2009-09-23 + */ +static int parseRFCField(uchar **pp2parse, uchar *pResult, int *pLenStr) +{ + uchar *p2parse; + int iRet = 0; + + assert(pp2parse != NULL); + assert(*pp2parse != NULL); + assert(pResult != NULL); + + p2parse = *pp2parse; + + /* this is the actual parsing loop */ + while(*pLenStr > 0 && *p2parse != ' ') { + *pResult++ = *p2parse++; + --(*pLenStr); + } + + if(*pLenStr > 0 && *p2parse == ' ') { + ++p2parse; /* eat SP, but only if not at end of string */ + --(*pLenStr); + } else { + iRet = 1; /* there MUST be an SP! */ + } + *pResult = '\0'; + + /* set the new parse pointer */ + *pp2parse = p2parse; + return 0; +} + + +/* Helper to parseRFCSyslogMsg. This function parses the structured + * data field of a message. It does NOT parse inside structured data, + * just gets the field as whole. Parsing the single entities is left + * to other functions. The parsepointer is advanced + * to after the terminating SP. The caller must ensure that the + * provided buffer is large enough to hold the to be extracted value. + * Returns 0 if everything is fine or 1 if either the field is not + * SP-terminated or any other error occurs. -- rger, 2005-11-24 + * The function now receives the size of the string and makes sure + * that it does not process more than that. The *pLenStr counter is + * updated on exit. -- rgerhards, 2009-09-23 + */ +static int parseRFCStructuredData(uchar **pp2parse, uchar *pResult, int *pLenStr) +{ + uchar *p2parse; + int bCont = 1; + int iRet = 0; + int lenStr; + + assert(pp2parse != NULL); + assert(*pp2parse != NULL); + assert(pResult != NULL); + + p2parse = *pp2parse; + lenStr = *pLenStr; + + /* this is the actual parsing loop + * Remeber: structured data starts with [ and includes any characters + * until the first ] followed by a SP. There may be spaces inside + * structured data. There may also be \] inside the structured data, which + * do NOT terminate an element. + */ + if(lenStr == 0 || *p2parse != '[') + return 1; /* this is NOT structured data! */ + + if(*p2parse == '-') { /* empty structured data? */ + *pResult++ = '-'; + ++p2parse; + --lenStr; + } else { + while(bCont) { + if(lenStr < 2) { + /* we now need to check if we have only structured data */ + if(lenStr > 0 && *p2parse == ']') { + *pResult++ = *p2parse; + p2parse++; + lenStr--; + bCont = 0; + } else { + iRet = 1; /* this is not valid! */ + bCont = 0; + } + } else if(*p2parse == '\\' && *(p2parse+1) == ']') { + /* this is escaped, need to copy both */ + *pResult++ = *p2parse++; + *pResult++ = *p2parse++; + lenStr -= 2; + } else if(*p2parse == ']' && *(p2parse+1) == ' ') { + /* found end, just need to copy the ] and eat the SP */ + *pResult++ = *p2parse; + p2parse += 2; + lenStr -= 2; + bCont = 0; + } else { + *pResult++ = *p2parse++; + --lenStr; + } + } + } + + if(lenStr > 0 && *p2parse == ' ') { + ++p2parse; /* eat SP, but only if not at end of string */ + --lenStr; + } else { + iRet = 1; /* there MUST be an SP! */ + } + *pResult = '\0'; + + /* set the new parse pointer */ + *pp2parse = p2parse; + *pLenStr = lenStr; + return 0; +} + +/* parse a RFC5424-formatted syslog message. This function returns + * 0 if processing of the message shall continue and 1 if something + * went wrong and this messe should be ignored. This function has been + * implemented in the effort to support syslog-protocol. Please note that + * the name (parse *RFC*) stems from the hope that syslog-protocol will + * some time become an RFC. Do not confuse this with informational + * RFC 3164 (which is legacy syslog). + * + * currently supported format: + * + * <PRI>VERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP [SD-ID]s SP MSG + * + * <PRI> is already stripped when this function is entered. VERSION already + * has been confirmed to be "1", but has NOT been stripped from the message. + * + * rger, 2005-11-24 + */ +//static int parseRFCSyslogMsg(msg_t *pMsg, int flags) +BEGINparse + uchar *p2parse; + uchar *pBuf = NULL; + int lenMsg; + int bContParse = 1; +CODESTARTparse + assert(pMsg != NULL); + assert(pMsg->pszRawMsg != NULL); + p2parse = pMsg->pszRawMsg + pMsg->offAfterPRI; /* point to start of text, after PRI */ + lenMsg = pMsg->iLenRawMsg - pMsg->offAfterPRI; + + /* check if we are the right parser */ + if(lenMsg < 2 || p2parse[0] != '1' || p2parse[1] != ' ') { + ABORT_FINALIZE(RS_RET_COULD_NOT_PARSE); + } + DBGPRINTF("Message has RFC5424/syslog-protocol format.\n"); + setProtocolVersion(pMsg, 1); + p2parse += 2; + lenMsg -= 2; + + /* Now get us some memory we can use as a work buffer while parsing. + * We simply allocated a buffer sufficiently large to hold all of the + * message, so we can not run into any troubles. I think this is + * wiser than to use individual buffers. + */ + CHKmalloc(pBuf = MALLOC(sizeof(uchar) * (lenMsg + 1))); + + /* IMPORTANT NOTE: + * Validation is not actually done below nor are any errors handled. I have + * NOT included this for the current proof of concept. However, it is strongly + * advisable to add it when this code actually goes into production. + * rgerhards, 2005-11-24 + */ + + /* TIMESTAMP */ + if(datetime.ParseTIMESTAMP3339(&(pMsg->tTIMESTAMP), &p2parse, &lenMsg) == RS_RET_OK) { + if(pMsg->msgFlags & IGNDATE) { + /* we need to ignore the msg data, so simply copy over reception date */ + memcpy(&pMsg->tTIMESTAMP, &pMsg->tRcvdAt, sizeof(struct syslogTime)); + } + } else { + DBGPRINTF("no TIMESTAMP detected!\n"); + bContParse = 0; + } + + /* HOSTNAME */ + if(bContParse) { + parseRFCField(&p2parse, pBuf, &lenMsg); + MsgSetHOSTNAME(pMsg, pBuf, ustrlen(pBuf)); + } + + /* APP-NAME */ + if(bContParse) { + parseRFCField(&p2parse, pBuf, &lenMsg); + MsgSetAPPNAME(pMsg, (char*)pBuf); + } + + /* PROCID */ + if(bContParse) { + parseRFCField(&p2parse, pBuf, &lenMsg); + MsgSetPROCID(pMsg, (char*)pBuf); + } + + /* MSGID */ + if(bContParse) { + parseRFCField(&p2parse, pBuf, &lenMsg); + MsgSetMSGID(pMsg, (char*)pBuf); + } + + /* STRUCTURED-DATA */ + if(bContParse) { + parseRFCStructuredData(&p2parse, pBuf, &lenMsg); + MsgSetStructuredData(pMsg, (char*)pBuf); + } + + /* MSG */ + MsgSetMSGoffs(pMsg, p2parse - pMsg->pszRawMsg); + +finalize_it: + if(pBuf != NULL) + free(pBuf); +ENDparse + + +BEGINmodExit +CODESTARTmodExit + /* release what we no longer need */ + objRelease(errmsg, CORE_COMPONENT); + objRelease(glbl, CORE_COMPONENT); + objRelease(parser, CORE_COMPONENT); + objRelease(datetime, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_PMOD_QUERIES +CODEqueryEtryPt_IsCompatibleWithFeature_IF_OMOD_QUERIES +ENDqueryEtryPt + + +BEGINmodInit(pmrfc5424) +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(parser, CORE_COMPONENT)); + CHKiRet(objUse(datetime, CORE_COMPONENT)); + + dbgprintf("rfc5424 parser init called\n"); + dbgprintf("GetParserName addr %p\n", GetParserName); +ENDmodInit + +/* vim:set ai: + */ diff --git a/tools/pmrfc5424.h b/tools/pmrfc5424.h new file mode 100644 index 00000000..df2a1c81 --- /dev/null +++ b/tools/pmrfc5424.h @@ -0,0 +1,33 @@ +/* pmrfc5424.h + * These are the definitions for the RFCC5424 parser module. + * + * File begun on 2009-11-03 by RGerhards + * + * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +#ifndef PMRFC54254_H_INCLUDED +#define PMRFC54254_H_INCLUDED 1 + +/* prototypes */ +rsRetVal modInitpmrfc5424(int iIFVersRequested __attribute__((unused)), int *ipIFVersProvided, rsRetVal (**pQueryEtryPt)(), rsRetVal (*pHostQueryEtryPt)(uchar*, rsRetVal (**)()), modInfo_t*); + +#endif /* #ifndef PMRFC54254_H_INCLUDED */ +/* vi:set ai: + */ diff --git a/tools/syslogd.c b/tools/syslogd.c index 165c751b..820cf273 100644 --- a/tools/syslogd.c +++ b/tools/syslogd.c @@ -33,7 +33,7 @@ * to the database). * * rsyslog - An Enhanced syslogd Replacement. - * Copyright 2003-2008 Rainer Gerhards and Adiscon GmbH. + * Copyright 2003-2009 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -125,6 +125,8 @@ #include "omfwd.h" #include "omfile.h" #include "omdiscard.h" +#include "pmrfc5424.h" +#include "pmrfc3164.h" #include "threads.h" #include "wti.h" #include "queue.h" @@ -133,7 +135,6 @@ #include "errmsg.h" #include "datetime.h" #include "parser.h" -//#include "sysvar.h" #include "batch.h" #include "unicode-helper.h" #include "ruleset.h" @@ -145,7 +146,7 @@ /* definitions for objects we access */ DEFobjCurrIf(obj) DEFobjCurrIf(glbl) -DEFobjCurrIf(datetime) +DEFobjCurrIf(datetime) /* TODO: make go away! */ DEFobjCurrIf(conf) DEFobjCurrIf(expr) DEFobjCurrIf(module) @@ -153,6 +154,7 @@ DEFobjCurrIf(errmsg) DEFobjCurrIf(rule) DEFobjCurrIf(ruleset) DEFobjCurrIf(prop) +DEFobjCurrIf(parser) DEFobjCurrIf(net) /* TODO: make go away! */ @@ -160,18 +162,6 @@ DEFobjCurrIf(net) /* TODO: make go away! */ static rsRetVal GlobalClassExit(void); -#ifndef UTMP_FILE -#ifdef UTMP_FILENAME -#define UTMP_FILE UTMP_FILENAME -#else -#ifdef _PATH_UTMP -#define UTMP_FILE _PATH_UTMP -#else -#define UTMP_FILE "/etc/utmp" -#endif -#endif -#endif - #ifndef _PATH_LOGCONF #define _PATH_LOGCONF "/etc/rsyslog.conf" #endif @@ -219,8 +209,6 @@ static pid_t myPid; /* our pid for use in self-generated messages, e.g. on start /* mypid is read-only after the initial fork() */ static int bHadHUP = 0; /* did we have a HUP? */ -static int bParseHOSTNAMEandTAG = 1; /* global config var: should the hostname and tag be - * parsed inside message - rgerhards, 2006-03-13 */ static int bFinished = 0; /* used by termination signal handler, read-only except there * is either 0 or the number of the signal that requested the * termination. @@ -245,19 +233,18 @@ typedef struct legacyOptsLL_s { legacyOptsLL_t *pLegacyOptsLL = NULL; /* global variables for config file state */ -int bDropTrailingLF = 1; /* drop trailing LF's on reception? */ int iCompatibilityMode = 0; /* version we should be compatible with; 0 means sysklogd. It is the default, so if no -c<n> option is given, we make ourselvs as compatible to sysklogd as possible. */ +#define DFLT_bLogStatusMsgs 1 +static int bLogStatusMsgs = DFLT_bLogStatusMsgs; /* log rsyslog start/stop/HUP messages? */ static int bDebugPrintTemplateList = 1;/* output template list in debug mode? */ static int bDebugPrintCfSysLineHandlerList = 1;/* output cfsyslinehandler list in debug mode? */ static int bDebugPrintModuleList = 1;/* output module list in debug mode? */ -uchar cCCEscapeChar = '\\';/* character to be used to start an escape sequence for control chars */ -int bEscapeCCOnRcv = 1; /* escape control characters on reception: 0 - no, 1 - yes */ static int bErrMsgToStderr = 1; /* print error messages to stderr (in addition to everything else)? */ int bReduceRepeatMsgs; /* reduce repeated message - 0 - no, 1 - yes */ +int bAbortOnUncleanConfig = 0; /* abort run (rather than starting with partial config) if there was any issue in conf */ int bActExecWhenPrevSusp; /* execute action only when previous one was suspended? */ -int iActExecOnceInterval = 0; /* execute action once every nn seconds */ /* end global config file state variables */ int MarkInterval = 20 * 60; /* interval between marks in seconds - read-only after startup */ @@ -274,7 +261,7 @@ extern int errno; static uchar *pszConfDAGFile = NULL; /* name of config DAG file, non-NULL means generate one */ /* main message queue and its configuration parameters */ -static qqueue_t *pMsgQueue = NULL; /* the main message queue */ +qqueue_t *pMsgQueue = NULL; /* the main message queue */ static int iMainMsgQueueSize = 10000; /* size of the main message queue above */ static int iMainMsgQHighWtrMark = 8000; /* high water mark for disk-assisted queues */ static int iMainMsgQLowWtrMark = 2000; /* low water mark for disk-assisted queues */ @@ -299,47 +286,18 @@ static int iMainMsgQueueDeqtWinFromHr = 0; /* hour begin of time frame when qu static int iMainMsgQueueDeqtWinToHr = 25; /* hour begin of time frame when queue is to be dequeued */ -/* support for simple textual representation of FIOP names - * rgerhards, 2005-09-27 - */ -char* -getFIOPName(unsigned iFIOP) -{ - char *pRet; - switch(iFIOP) { - case FIOP_CONTAINS: - pRet = "contains"; - break; - case FIOP_ISEQUAL: - pRet = "isequal"; - break; - case FIOP_STARTSWITH: - pRet = "startswith"; - break; - case FIOP_REGEX: - pRet = "regex"; - break; - default: - pRet = "NOP"; - break; - } - return pRet; -} - - /* Reset config variables to default values. * rgerhards, 2007-07-17 */ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __attribute__((unused)) *pVal) { - cCCEscapeChar = '#'; + bLogStatusMsgs = DFLT_bLogStatusMsgs; bActExecWhenPrevSusp = 0; - iActExecOnceInterval = 0; bDebugPrintTemplateList = 1; bDebugPrintCfSysLineHandlerList = 1; bDebugPrintModuleList = 1; - bEscapeCCOnRcv = 1; /* default is to escape control characters */ bReduceRepeatMsgs = 0; + bAbortOnUncleanConfig = 0; free(pszMainMsgQFName); pszMainMsgQFName = NULL; iMainMsgQueueSize = 10000; @@ -387,7 +345,6 @@ static char **crunch_list(char *list); static void reapchild(); static void debug_switch(); static void sighup_handler(); -static void processImInternal(void); static int usage(void) @@ -395,7 +352,7 @@ static int usage(void) fprintf(stderr, "usage: rsyslogd [-c<version>] [-46AdnqQvwx] [-l<hostlist>] [-s<domainlist>]\n" " [-f<conffile>] [-i<pidfile>] [-N<level>] [-M<module load path>]\n" " [-u<number>]\n" - "To run rsyslogd in native mode, use \"rsyslogd -c3 <other options>\"\n\n" + "To run rsyslogd in native mode, use \"rsyslogd -c5 <other options>\"\n\n" "For further information see http://www.rsyslog.com/doc\n"); exit(1); /* "good" exit - done to terminate usage() */ } @@ -407,6 +364,8 @@ static int usage(void) */ /* return back the approximate current number of messages in the main message queue + * This number includes the messages that reside in an associated DA queue (if + * it exists) -- rgerhards, 2009-10-14 */ rsRetVal diagGetMainMsgQSize(int *piSize) @@ -451,7 +410,7 @@ static char **crunch_list(char *list) for (count=i=0; p[i]; i++) if (p[i] == LIST_DELIMITER) count++; - if ((result = (char **)malloc(sizeof(char *) * (count+2))) == NULL) { + if ((result = (char **)MALLOC(sizeof(char *) * (count+2))) == NULL) { printf ("Sorry, can't get enough memory, exiting.\n"); exit(0); /* safe exit, because only called during startup */ } @@ -463,7 +422,7 @@ static char **crunch_list(char *list) */ count = 0; while ((q=strchr(p, LIST_DELIMITER))) { - result[count] = (char *) malloc((q - p + 1) * sizeof(char)); + result[count] = (char *) MALLOC((q - p + 1) * sizeof(char)); if (result[count] == NULL) { printf ("Sorry, can't get enough memory, exiting.\n"); exit(0); /* safe exit, because only called during startup */ @@ -474,7 +433,7 @@ static char **crunch_list(char *list) count++; } if ((result[count] = \ - (char *)malloc(sizeof(char) * strlen(p) + 1)) == NULL) { + (char *)MALLOC(sizeof(char) * strlen(p) + 1)) == NULL) { printf ("Sorry, can't get enough memory, exiting.\n"); exit(0); /* safe exit, because only called during startup */ } @@ -520,52 +479,21 @@ void untty(void) #endif -/* Take a raw input line, decode the message, and print the message - * on the appropriate log files. - * rgerhards 2004-11-08: Please note - * that this function does only a partial decoding. At best, it splits - * the PRI part. No further decode happens. The rest is done in - * logmsg(). - * Added the iSource parameter so that we know if we have to parse - * HOSTNAME or not. rgerhards 2004-11-16. - * changed parameter iSource to bParseHost. For details, see comment in - * printchopped(). rgerhards 2005-10-06 - * rgerhards: 2008-03-06: added "flags" to allow an input module to specify - * flags, most importantly to request ignoring the messages' timestamp. - * - * rgerhards, 2008-03-19: - * I added an additional calling parameter to permit specifying the flow - * control capability of the source. - * - * rgerhards, 2008-05-16: - * I added an additional calling parameter (hnameIP) to enable specifying the IP - * of a remote host. - * - * rgerhards, 2008-09-11: - * Interface change: added new parameter "InputName", permits the input to provide - * a string that identifies it. May be NULL, but must be a valid char* pointer if - * non-NULL. - * - * rgerhards, 2008-10-06: - * Interface change: added new parameter "stTime", which enables the caller to provide - * a timestamp that is to be used as timegenerated instead of the current system time. - * This is meant to facilitate performance optimization. Some inputs support such modes. - * If stTime is NULL, the current system time is used. - * - * rgerhards, 2008-10-09: - * interface change: bParseHostname removed, now in flags +/* This takes a received message that must be decoded and submits it to + * the main message queue. This is a legacy function which is being provided + * to aid older input plugins that do not support message creation via + * the new interfaces themselves. It is not recommended to use this + * function for new plugins. -- rgerhards, 2009-10-12 */ -static inline rsRetVal printline(uchar *hname, uchar *hnameIP, uchar *msg, int flags, flowControl_t flowCtlType, +rsRetVal +parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int flags, flowControl_t flowCtlType, prop_t *pInputName, struct syslogTime *stTime, time_t ttGenTime) { - DEFiRet; - register uchar *p; - int pri; + prop_t *pProp = NULL; msg_t *pMsg; - prop_t *propFromHost = NULL; - prop_t *propFromHostIP = NULL; + DEFiRet; - /* Now it is time to create the message object (rgerhards) */ + /* we now create our own message object and submit it to the queue */ if(stTime == NULL) { CHKiRet(msgConstruct(&pMsg)); } else { @@ -573,278 +501,21 @@ static inline rsRetVal printline(uchar *hname, uchar *hnameIP, uchar *msg, int f } if(pInputName != NULL) MsgSetInputName(pMsg, pInputName); + MsgSetRawMsg(pMsg, (char*)msg, len); MsgSetFlowControlType(pMsg, flowCtlType); - MsgSetRawMsgWOSize(pMsg, (char*)msg); - - /* test for special codes */ - pri = DEFUPRI; - p = msg; - if (*p == '<') { - pri = 0; - while (isdigit((int) *++p)) - { - pri = 10 * pri + (*p - '0'); - } - if (*p == '>') - ++p; - } - if (pri &~ (LOG_FACMASK|LOG_PRIMASK)) - pri = DEFUPRI; - pMsg->iFacility = LOG_FAC(pri); - pMsg->iSeverity = LOG_PRI(pri); - - /* Now we look at the HOSTNAME. That is a bit complicated... - * If we have a locally received message, it does NOT - * contain any hostname information in the message itself. - * As such, the HOSTNAME is the same as the system that - * the message was received from (that, for obvious reasons, - * being the local host). rgerhards 2004-11-16 - */ - if((pMsg->msgFlags & PARSE_HOSTNAME) == 0) - MsgSetHOSTNAME(pMsg, hname, ustrlen(hname)); - MsgSetRcvFromStr(pMsg, hname, ustrlen(hname), &propFromHost); - CHKiRet(MsgSetRcvFromIPStr(pMsg, hnameIP, ustrlen(hnameIP), &propFromHostIP)); - MsgSetAfterPRIOffs(pMsg, p - msg); - prop.Destruct(&propFromHost); - prop.Destruct(&propFromHostIP); + pMsg->msgFlags = flags | NEEDS_PARSING; - logmsg(pMsg, flags); + MsgSetRcvFromStr(pMsg, hname, ustrlen(hname), &pProp); + CHKiRet(prop.Destruct(&pProp)); + CHKiRet(MsgSetRcvFromIPStr(pMsg, hnameIP, ustrlen(hnameIP), &pProp)); + CHKiRet(prop.Destruct(&pProp)); + CHKiRet(submitMsg(pMsg)); finalize_it: RETiRet; } -/* This takes a received message that must be decoded and submits it to - * the main message queue. The function calls the necessary parser. - * - * rgerhards, 2006-11-30: I have greatly changed this function. Formerly, - * it tried to reassemble multi-part messages, which is a legacy stock - * sysklogd concept. In essence, that was that messages not ending with - * \0 were glued together. As far as I can see, this is a sysklogd - * specific feature and, from looking at the code, seems to be used - * pretty seldom (if at all). I remove this now, not the least because it is totally - * incompatible with upcoming IETF syslog standards. If you experience - * strange behaviour with messages beeing split across multiple lines, - * this function here might be the place to look at. - * - * Some previous history worth noting: - * I added the "iSource" parameter. This is needed to distinguish between - * messages that have a hostname in them (received from the internet) and - * those that do not have (most prominently /dev/log). rgerhards 2004-11-16 - * And now I removed the "iSource" parameter and changed it to be "bParseHost", - * because all that it actually controls is whether the host is parsed or not. - * For rfc3195 support, we needed to modify the algo for host parsing, so we can - * no longer rely just on the source (rfc3195d forwarded messages arrive via - * unix domain sockets but contain the hostname). rgerhards, 2005-10-06 - * - * rgerhards, 2008-02-18: - * This function was previously called "printchopped"() and has been renamed - * as part of the effort to create a clean internal message submission interface. - * It also has been adopted to our usual calling interface, but currently does - * not provide any useful return states. But we now have the hook and things can - * improve in the future. <-- TODO! - * - * rgerhards, 2008-03-19: - * I added an additional calling parameter to permit specifying the flow - * control capability of the source. - * - * rgerhards, 2008-05-16: - * I added an additional calling parameter (hnameIP) to enable specifying the IP - * of a remote host. - * - * rgerhards, 2008-09-11: - * Interface change: added new parameter "InputName", permits the input to provide - * a string that identifies it. May be NULL, but must be a valid char* pointer if - * non-NULL. - * - * rgerhards, 2008-10-06: - * Interface change: added new parameter "stTime", which enables the caller to provide - * a timestamp that is to be used as timegenerated instead of the current system time. - * This is meant to facilitate performance optimization. Some inputs support such modes. - * If stTime is NULL, the current system time is used. - * - * rgerhards, 2008-10-09: - * interface change: bParseHostname removed, now in flags - */ -rsRetVal -parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int flags, flowControl_t flowCtlType, - prop_t *pInputName, struct syslogTime *stTime, time_t ttGenTime) -{ - DEFiRet; - register int iMsg; - uchar *pMsg; - uchar *pData; - uchar *pEnd; - int iMaxLine; - uchar *tmpline = NULL; -# ifdef USE_NETZIP - uchar *deflateBuf = NULL; - uLongf iLenDefBuf; -# endif - - assert(hname != NULL); - assert(hnameIP != NULL); - assert(msg != NULL); - assert(len >= 0); - - /* we first allocate work buffers large enough to hold the configured maximum - * size of a message. Over time, we should change this to a more optimal way, i.e. - * by calling the function with the actual length of the message to be parsed. - * rgerhards, 2008-09-02 - * - * TODO: optimize buffer handling */ - iMaxLine = glbl.GetMaxLine(); - CHKmalloc(tmpline = malloc(sizeof(uchar) * (iMaxLine + 1))); - - /* we first check if we have a NUL character at the very end of the - * message. This seems to be a frequent problem with a number of senders. - * So I have now decided to drop these NULs. However, if they are intentional, - * that may cause us some problems, e.g. with syslog-sign. On the other hand, - * current code always has problems with intentional NULs (as it needs to escape - * them to prevent problems with the C string libraries), so that does not - * really matter. Just to be on the save side, we'll log destruction of such - * NULs in the debug log. - * rgerhards, 2007-09-14 - */ - if(*(msg + len - 1) == '\0') { - DBGPRINTF("dropped NUL at very end of message\n"); - len--; - } - - /* then we check if we need to drop trailing LFs, which often make - * their way into syslog messages unintentionally. In order to remain - * compatible to recent IETF developments, we allow the user to - * turn on/off this handling. rgerhards, 2007-07-23 - */ - if(bDropTrailingLF && *(msg + len - 1) == '\n') { - DBGPRINTF("dropped LF at very end of message (DropTrailingLF is set)\n"); - len--; - } - - iMsg = 0; /* initialize receiving buffer index */ - pMsg = tmpline; /* set receiving buffer pointer */ - pData = msg; /* set source buffer pointer */ - pEnd = msg + len; /* this is one off, which is intensional */ - -# ifdef USE_NETZIP - /* we first need to check if we have a compressed record. If so, - * we must decompress it. - */ - if(len > 0 && *msg == 'z') { /* compressed data present? (do NOT change order if conditions!) */ - /* we have compressed data, so let's deflate it. We support a maximum - * message size of iMaxLine. If it is larger, an error message is logged - * and the message is dropped. We do NOT try to decompress larger messages - * as such might be used for denial of service. It might happen to later - * builds that such functionality be added as an optional, operator-configurable - * feature. - */ - int ret; - iLenDefBuf = iMaxLine; - CHKmalloc(deflateBuf = malloc(sizeof(uchar) * (iMaxLine + 1))); - ret = uncompress((uchar *) deflateBuf, &iLenDefBuf, (uchar *) msg+1, len-1); - DBGPRINTF("Compressed message uncompressed with status %d, length: new %ld, old %d.\n", - ret, (long) iLenDefBuf, len-1); - /* Now check if the uncompression worked. If not, there is not much we can do. In - * that case, we log an error message but ignore the message itself. Storing the - * compressed text is dangerous, as it contains control characters. So we do - * not do this. If someone would like to have a copy, this code here could be - * modified to do a hex-dump of the buffer in question. We do not include - * this functionality right now. - * rgerhards, 2006-12-07 - */ - if(ret != Z_OK) { - errmsg.LogError(0, NO_ERRCODE, "Uncompression of a message failed with return code %d " - "- enable debug logging if you need further information. " - "Message ignored.", ret); - FINALIZE; /* unconditional exit, nothing left to do... */ - } - pData = deflateBuf; - pEnd = deflateBuf + iLenDefBuf; - } -# else /* ifdef USE_NETZIP */ - /* in this case, we still need to check if the message is compressed. If so, we must - * tell the user we can not accept it. - */ - if(len > 0 && *msg == 'z') { - errmsg.LogError(0, NO_ERRCODE, "Received a compressed message, but rsyslogd does not have compression " - "support enabled. The message will be ignored."); - FINALIZE; - } -# endif /* ifdef USE_NETZIP */ - - while(pData < pEnd) { - if(iMsg >= iMaxLine) { - /* emergency, we now need to flush, no matter if - * we are at end of message or not... - */ - if(iMsg == iMaxLine) { - *(pMsg + iMsg) = '\0'; /* space *is* reserved for this! */ - printline(hname, hnameIP, tmpline, flags, flowCtlType, pInputName, stTime, ttGenTime); - } else { - /* This case in theory never can happen. If it happens, we have - * a logic error. I am checking for it, because if I would not, - * we would address memory invalidly with the code above. I - * do not care much about this case, just a debug log entry - * (I couldn't do any more smart things anyway...). - * rgerhards, 2007-9-20 - */ - DBGPRINTF("internal error: iMsg > max msg size in printchopped()\n"); - } - FINALIZE; /* in this case, we are done... nothing left we can do */ - } - if(*pData == '\0') { /* guard against \0 characters... */ - /* changed to the sequence (somewhat) proposed in - * draft-ietf-syslog-protocol-19. rgerhards, 2006-11-30 - */ - if(iMsg + 3 < iMaxLine) { /* do we have space? */ - *(pMsg + iMsg++) = cCCEscapeChar; - *(pMsg + iMsg++) = '0'; - *(pMsg + iMsg++) = '0'; - *(pMsg + iMsg++) = '0'; - } /* if we do not have space, we simply ignore the '\0'... */ - /* log an error? Very questionable... rgerhards, 2006-11-30 */ - /* decided: we do not log an error, it won't help... rger, 2007-06-21 */ - ++pData; - } else if(bEscapeCCOnRcv && iscntrl((int) *pData)) { - /* we are configured to escape control characters. Please note - * that this most probably break non-western character sets like - * Japanese, Korean or Chinese. rgerhards, 2007-07-17 - * Note: sysklogd logs octal values only for DEL and CCs above 127. - * For others, it logs ^n where n is the control char converted to an - * alphabet character. We like consistency and thus escape it to octal - * in all cases. If someone complains, we may change the mode. At least - * we known now what's going on. - * rgerhards, 2007-07-17 - */ - if(iMsg + 3 < iMaxLine) { /* do we have space? */ - *(pMsg + iMsg++) = cCCEscapeChar; - *(pMsg + iMsg++) = '0' + ((*pData & 0300) >> 6); - *(pMsg + iMsg++) = '0' + ((*pData & 0070) >> 3); - *(pMsg + iMsg++) = '0' + ((*pData & 0007)); - } /* again, if we do not have space, we ignore the char - see comment at '\0' */ - ++pData; - } else { - *(pMsg + iMsg++) = *pData++; - } - } - - *(pMsg + iMsg) = '\0'; /* space *is* reserved for this! */ - - /* typically, we should end up here! */ - printline(hname, hnameIP, tmpline, flags, flowCtlType, pInputName, stTime, ttGenTime); - -finalize_it: - if(tmpline != NULL) - free(tmpline); -# ifdef USE_NETZIP - if(deflateBuf != NULL) - free(deflateBuf); -# endif - RETiRet; -} - - /* this is a special function used to submit an error message. This * function is also passed to the runtime library as the generic error * message handler. -- rgerhards, 2008-04-17 @@ -859,13 +530,7 @@ submitErrMsg(int iErr, uchar *msg) /* rgerhards 2004-11-09: the following is a function that can be used - * to log a message orginating from the syslogd itself. In sysklogd code, - * this is done by simply calling logmsg(). However, logmsg() is changed in - * rsyslog so that it takes a msg "object". So it can no longer be called - * directly. This method here solves the need. It provides an interface that - * allows to construct a locally-generated message. Please note that this - * function here probably is only an interim solution and that we need to - * think on the best way to do this. + * to log a message orginating from the syslogd itself. */ rsRetVal logmsgInternal(int iErr, int pri, uchar *msg, int flags) @@ -880,6 +545,7 @@ logmsgInternal(int iErr, int pri, uchar *msg, int flags) MsgSetHOSTNAME(pMsg, glbl.GetLocalHostName(), ustrlen(glbl.GetLocalHostName())); MsgSetRcvFrom(pMsg, glbl.GetLocalHostNameProp()); MsgSetRcvFromIP(pMsg, pLocalHostIP); + MsgSetMSGoffs(pMsg, 0); /* check if we have an error code associated and, if so, * adjust the tag. -- rgerhards, 2008-06-27 */ @@ -892,8 +558,8 @@ logmsgInternal(int iErr, int pri, uchar *msg, int flags) } pMsg->iFacility = LOG_FAC(pri); pMsg->iSeverity = LOG_PRI(pri); - pMsg->bParseHOSTNAME = 0; flags |= INTERNAL_MSG; + pMsg->msgFlags = flags; /* we now check if we should print internal messages out to stderr. This was * suggested by HKS as a way to help people troubleshoot rsyslog configuration @@ -909,12 +575,12 @@ logmsgInternal(int iErr, int pri, uchar *msg, int flags) } if(bHaveMainQueue == 0) { /* not yet in queued mode */ - iminternalAddMsg(pri, pMsg, flags); + iminternalAddMsg(pri, pMsg); } else { /* we have the queue, so we can simply provide the * message to the queue engine. */ - logmsg(pMsg, flags); + submitMsg(pMsg); } finalize_it: RETiRet; @@ -928,495 +594,86 @@ finalize_it: * for the main queue. */ static rsRetVal -msgConsumer(void __attribute__((unused)) *notNeeded, batch_t *pBatch) +msgConsumer(void __attribute__((unused)) *notNeeded, batch_t *pBatch, int *pbShutdownImmediate) { int i; msg_t *pMsg; + rsRetVal localRet; DEFiRet; assert(pBatch != NULL); - for(i = 0 ; i < pBatch->nElem ; i++) { + for(i = 0 ; i < pBatch->nElem && !*pbShutdownImmediate ; i++) { pMsg = (msg_t*) pBatch->pElem[i].pUsrp; DBGPRINTF("msgConsumer processes msg %d/%d\n", i, pBatch->nElem); if((pMsg->msgFlags & NEEDS_PARSING) != 0) { - parseMsg(pMsg); + localRet = parser.ParseMsg(pMsg); + if(localRet == RS_RET_OK) + ruleset.ProcessMsg(pMsg); + } else { + ruleset.ProcessMsg(pMsg); } - ruleset.ProcessMsg(pMsg); + /* if we reach this point, the message is considered committed (by definition!) */ + pBatch->pElem[i].state = BATCH_STATE_COMM; } RETiRet; } -/* Helper to parseRFCSyslogMsg. This function parses a field up to - * (and including) the SP character after it. The field contents is - * returned in a caller-provided buffer. The parsepointer is advanced - * to after the terminating SP. The caller must ensure that the - * provided buffer is large enough to hold the to be extracted value. - * Returns 0 if everything is fine or 1 if either the field is not - * SP-terminated or any other error occurs. -- rger, 2005-11-24 - * The function now receives the size of the string and makes sure - * that it does not process more than that. The *pLenStr counter is - * updated on exit. -- rgerhards, 2009-09-23 - */ -static int parseRFCField(uchar **pp2parse, uchar *pResult, int *pLenStr) -{ - uchar *p2parse; - int iRet = 0; - - assert(pp2parse != NULL); - assert(*pp2parse != NULL); - assert(pResult != NULL); - - p2parse = *pp2parse; - - /* this is the actual parsing loop */ - while(*pLenStr > 0 && *p2parse != ' ') { - *pResult++ = *p2parse++; - --(*pLenStr); - } - - if(*pLenStr > 0 && *p2parse == ' ') { - ++p2parse; /* eat SP, but only if not at end of string */ - --(*pLenStr); - } else { - iRet = 1; /* there MUST be an SP! */ - } - *pResult = '\0'; - - /* set the new parse pointer */ - *pp2parse = p2parse; - return 0; -} - - -/* Helper to parseRFCSyslogMsg. This function parses the structured - * data field of a message. It does NOT parse inside structured data, - * just gets the field as whole. Parsing the single entities is left - * to other functions. The parsepointer is advanced - * to after the terminating SP. The caller must ensure that the - * provided buffer is large enough to hold the to be extracted value. - * Returns 0 if everything is fine or 1 if either the field is not - * SP-terminated or any other error occurs. -- rger, 2005-11-24 - * The function now receives the size of the string and makes sure - * that it does not process more than that. The *pLenStr counter is - * updated on exit. -- rgerhards, 2009-09-23 - */ -static int parseRFCStructuredData(uchar **pp2parse, uchar *pResult, int *pLenStr) -{ - uchar *p2parse; - int bCont = 1; - int iRet = 0; - int lenStr; - - assert(pp2parse != NULL); - assert(*pp2parse != NULL); - assert(pResult != NULL); - - p2parse = *pp2parse; - lenStr = *pLenStr; - - /* this is the actual parsing loop - * Remeber: structured data starts with [ and includes any characters - * until the first ] followed by a SP. There may be spaces inside - * structured data. There may also be \] inside the structured data, which - * do NOT terminate an element. - */ - if(lenStr == 0 || *p2parse != '[') - return 1; /* this is NOT structured data! */ - - if(*p2parse == '-') { /* empty structured data? */ - *pResult++ = '-'; - ++p2parse; - --lenStr; - } else { - while(bCont) { - if(lenStr < 2) { - /* we now need to check if we have only structured data */ - if(lenStr > 0 && *p2parse == ']') { - *pResult++ = *p2parse; - p2parse++; - lenStr--; - bCont = 0; - } else { - iRet = 1; /* this is not valid! */ - bCont = 0; - } - } else if(*p2parse == '\\' && *(p2parse+1) == ']') { - /* this is escaped, need to copy both */ - *pResult++ = *p2parse++; - *pResult++ = *p2parse++; - lenStr -= 2; - } else if(*p2parse == ']' && *(p2parse+1) == ' ') { - /* found end, just need to copy the ] and eat the SP */ - *pResult++ = *p2parse; - p2parse += 2; - lenStr -= 2; - bCont = 0; - } else { - *pResult++ = *p2parse++; - --lenStr; - } - } - } - - if(lenStr > 0 && *p2parse == ' ') { - ++p2parse; /* eat SP, but only if not at end of string */ - --lenStr; - } else { - iRet = 1; /* there MUST be an SP! */ - } - *pResult = '\0'; - - /* set the new parse pointer */ - *pp2parse = p2parse; - *pLenStr = lenStr; - return 0; -} - -/* parse a RFC5424-formatted syslog message. This function returns - * 0 if processing of the message shall continue and 1 if something - * went wrong and this messe should be ignored. This function has been - * implemented in the effort to support syslog-protocol. Please note that - * the name (parse *RFC*) stems from the hope that syslog-protocol will - * some time become an RFC. Do not confuse this with informational - * RFC 3164 (which is legacy syslog). - * - * currently supported format: - * - * <PRI>VERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP [SD-ID]s SP MSG - * - * <PRI> is already stripped when this function is entered. VERSION already - * has been confirmed to be "1", but has NOT been stripped from the message. - * - * rger, 2005-11-24 - */ -int parseRFCSyslogMsg(msg_t *pMsg, int flags) -{ - uchar *p2parse; - uchar *pBuf; - int lenMsg; - int bContParse = 1; - - BEGINfunc - assert(pMsg != NULL); - assert(pMsg->pszRawMsg != NULL); - p2parse = pMsg->pszRawMsg + pMsg->offAfterPRI; /* point to start of text, after PRI */ - lenMsg = pMsg->iLenRawMsg - pMsg->offAfterPRI; - - /* do a sanity check on the version and eat it (the caller checked this already) */ - assert(p2parse[0] == '1' && p2parse[1] == ' '); - p2parse += 2; - lenMsg -= 2; - - /* Now get us some memory we can use as a work buffer while parsing. - * We simply allocated a buffer sufficiently large to hold all of the - * message, so we can not run into any troubles. I think this is - * more wise then to use individual buffers. - */ - if((pBuf = malloc(sizeof(uchar) * (lenMsg + 1))) == NULL) - return 1; - - /* IMPORTANT NOTE: - * Validation is not actually done below nor are any errors handled. I have - * NOT included this for the current proof of concept. However, it is strongly - * advisable to add it when this code actually goes into production. - * rgerhards, 2005-11-24 - */ - - /* TIMESTAMP */ - if(datetime.ParseTIMESTAMP3339(&(pMsg->tTIMESTAMP), &p2parse, &lenMsg) == RS_RET_OK) { - if(flags & IGNDATE) { - /* we need to ignore the msg data, so simply copy over reception date */ - memcpy(&pMsg->tTIMESTAMP, &pMsg->tRcvdAt, sizeof(struct syslogTime)); - } - } else { - DBGPRINTF("no TIMESTAMP detected!\n"); - bContParse = 0; - } - - /* HOSTNAME */ - if(bContParse) { - parseRFCField(&p2parse, pBuf, &lenMsg); - MsgSetHOSTNAME(pMsg, pBuf, ustrlen(pBuf)); - } - - /* APP-NAME */ - if(bContParse) { - parseRFCField(&p2parse, pBuf, &lenMsg); - MsgSetAPPNAME(pMsg, (char*)pBuf); - } - - /* PROCID */ - if(bContParse) { - parseRFCField(&p2parse, pBuf, &lenMsg); - MsgSetPROCID(pMsg, (char*)pBuf); - } - - /* MSGID */ - if(bContParse) { - parseRFCField(&p2parse, pBuf, &lenMsg); - MsgSetMSGID(pMsg, (char*)pBuf); - } - - /* STRUCTURED-DATA */ - if(bContParse) { - parseRFCStructuredData(&p2parse, pBuf, &lenMsg); - MsgSetStructuredData(pMsg, (char*)pBuf); - } - - /* MSG */ - MsgSetMSGoffs(pMsg, p2parse - pMsg->pszRawMsg); - - free(pBuf); - ENDfunc - return 0; /* all ok */ -} - - -/* parse a legay-formatted syslog message. This function returns - * 0 if processing of the message shall continue and 1 if something - * went wrong and this messe should be ignored. This function has been - * implemented in the effort to support syslog-protocol. - * rger, 2005-11-24 - * As of 2006-01-10, I am removing the logic to continue parsing only - * when a valid TIMESTAMP is detected. Validity of other fields already - * is ignored. This is due to the fact that the parser has grown smarter - * and is now more able to understand different dialects of the syslog - * message format. I do not expect any bad side effects of this change, - * but I thought I log it in this comment. - * rgerhards, 2006-01-10 - */ -int parseLegacySyslogMsg(msg_t *pMsg, int flags) -{ - uchar *p2parse; - int lenMsg; - int bTAGCharDetected; - int i; /* general index for parsing */ - uchar bufParseTAG[CONF_TAG_MAXSIZE]; - uchar bufParseHOSTNAME[CONF_TAG_HOSTNAME]; - BEGINfunc - - assert(pMsg != NULL); - assert(pMsg->pszRawMsg != NULL); - lenMsg = pMsg->iLenRawMsg - (pMsg->offAfterPRI + 1); - p2parse = pMsg->pszRawMsg + pMsg->offAfterPRI; /* point to start of text, after PRI */ - - /* Check to see if msg contains a timestamp. We start by assuming - * that the message timestamp is the time of reception (which we - * generated ourselfs and then try to actually find one inside the - * message. There we go from high-to low precison and are done - * when we find a matching one. -- rgerhards, 2008-09-16 - */ - if(datetime.ParseTIMESTAMP3339(&(pMsg->tTIMESTAMP), &p2parse, &lenMsg) == RS_RET_OK) { - /* we are done - parse pointer is moved by ParseTIMESTAMP3339 */; - } else if(datetime.ParseTIMESTAMP3164(&(pMsg->tTIMESTAMP), &p2parse, &lenMsg) == RS_RET_OK) { - /* we are done - parse pointer is moved by ParseTIMESTAMP3164 */; - } else if(*p2parse == ' ' && lenMsg > 1) { /* try to see if it is slighly malformed - HP procurve seems to do that sometimes */ - ++p2parse; /* move over space */ - --lenMsg; - if(datetime.ParseTIMESTAMP3164(&(pMsg->tTIMESTAMP), &p2parse, &lenMsg) == RS_RET_OK) { - /* indeed, we got it! */ - /* we are done - parse pointer is moved by ParseTIMESTAMP3164 */; - } else {/* parse pointer needs to be restored, as we moved it off-by-one - * for this try. - */ - --p2parse; - ++lenMsg; - } - } - - if(flags & IGNDATE) { - /* we need to ignore the msg data, so simply copy over reception date */ - memcpy(&pMsg->tTIMESTAMP, &pMsg->tRcvdAt, sizeof(struct syslogTime)); - } - - /* rgerhards, 2006-03-13: next, we parse the hostname and tag. But we - * do this only when the user has not forbidden this. I now introduce some - * code that allows a user to configure rsyslogd to treat the rest of the - * message as MSG part completely. In this case, the hostname will be the - * machine that we received the message from and the tag will be empty. This - * is meant to be an interim solution, but for now it is in the code. - */ - if(bParseHOSTNAMEandTAG && !(flags & INTERNAL_MSG)) { - /* parse HOSTNAME - but only if this is network-received! - * rger, 2005-11-14: we still have a problem with BSD messages. These messages - * do NOT include a host name. In most cases, this leads to the TAG to be treated - * as hostname and the first word of the message as the TAG. Clearly, this is not - * of advantage ;) I think I have now found a way to handle this situation: there - * are certain characters which are frequently used in TAG (e.g. ':'), which are - * *invalid* in host names. So while parsing the hostname, I check for these characters. - * If I find them, I set a simple flag but continue. After parsing, I check the flag. - * If it was set, then we most probably do not have a hostname but a TAG. Thus, I change - * the fields. I think this logic shall work with any type of syslog message. - * rgerhards, 2009-06-23: and I now have extended this logic to every character - * that is not a valid hostname. - */ - bTAGCharDetected = 0; - if(lenMsg > 0 && flags & PARSE_HOSTNAME) { - i = 0; - while(i < lenMsg && (isalnum(p2parse[i]) || p2parse[i] == '.' || p2parse[i] == '.' - || p2parse[i] == '_' || p2parse[i] == '-') && i < CONF_TAG_MAXSIZE) { - bufParseHOSTNAME[i] = p2parse[i]; - ++i; - } - - if(i > 0 && p2parse[i] == ' ' && isalnum(p2parse[i-1])) { - /* we got a hostname! */ - p2parse += i + 1; /* "eat" it (including SP delimiter) */ - lenMsg -= i + 1; - bufParseHOSTNAME[i] = '\0'; - MsgSetHOSTNAME(pMsg, bufParseHOSTNAME, i); - } - } - - /* now parse TAG - that should be present in message from all sources. - * This code is somewhat not compliant with RFC 3164. As of 3164, - * the TAG field is ended by any non-alphanumeric character. In - * practice, however, the TAG often contains dashes and other things, - * which would end the TAG. So it is not desirable. As such, we only - * accept colon and SP to be terminators. Even there is a slight difference: - * a colon is PART of the TAG, while a SP is NOT part of the tag - * (it is CONTENT). Starting 2008-04-04, we have removed the 32 character - * size limit (from RFC3164) on the tag. This had bad effects on existing - * envrionments, as sysklogd didn't obey it either (probably another bug - * in RFC3164...). We now receive the full size, but will modify the - * outputs so that only 32 characters max are used by default. - */ - i = 0; - while(lenMsg > 0 && *p2parse != ':' && *p2parse != ' ' && i < CONF_TAG_MAXSIZE) { - bufParseTAG[i++] = *p2parse++; - --lenMsg; - } - if(lenMsg > 0 && *p2parse == ':') { - ++p2parse; - --lenMsg; - bufParseTAG[i++] = ':'; - } - - /* no TAG can only be detected if the message immediatly ends, in which case an empty TAG - * is considered OK. So we do not need to check for empty TAG. -- rgerhards, 2009-06-23 - */ - bufParseTAG[i] = '\0'; /* terminate string */ - MsgSetTAG(pMsg, bufParseTAG, i); - } else {/* we enter this code area when the user has instructed rsyslog NOT - * to parse HOSTNAME and TAG - rgerhards, 2006-03-13 - */ - if(!(flags & INTERNAL_MSG)) { - DBGPRINTF("HOSTNAME and TAG not parsed by user configuraton.\n"); - } - } - - /* The rest is the actual MSG */ - MsgSetMSGoffs(pMsg, p2parse - pMsg->pszRawMsg); - - ENDfunc - return 0; /* all ok */ -} - - /* submit a message to the main message queue. This is primarily * a hook to prevent the need for callers to know about the main message queue - * (which may change in the future as we will probably have multiple rule - * sets and thus queues...). * rgerhards, 2008-02-13 */ rsRetVal submitMsg(msg_t *pMsg) { + qqueue_t *pQueue; + ruleset_t *pRuleset; DEFiRet; ISOBJ_TYPE_assert(pMsg, msg); + pRuleset = MsgGetRuleset(pMsg); + + pQueue = (pRuleset == NULL) ? pMsgQueue : ruleset.GetRulesetQueue(pRuleset); MsgPrepareEnqueue(pMsg); - qqueueEnqObj(pMsgQueue, pMsg->flowCtlType, (void*) pMsg); + qqueueEnqObj(pQueue, pMsg->flowCtlType, (void*) pMsg); RETiRet; } /* submit multiple messages at once, very similar to submitMsg, just - * for multi_submit_t. + * for multi_submit_t. All messages need to go into the SAME queue! * rgerhards, 2009-06-16 */ rsRetVal multiSubmitMsg(multi_submit_t *pMultiSub) { int i; + qqueue_t *pQueue; + ruleset_t *pRuleset; DEFiRet; assert(pMultiSub != NULL); + if(pMultiSub->nElem == 0) + FINALIZE; + for(i = 0 ; i < pMultiSub->nElem ; ++i) { MsgPrepareEnqueue(pMultiSub->ppMsgs[i]); } - iRet = qqueueMultiEnqObj(pMsgQueue, pMultiSub); + pRuleset = MsgGetRuleset(pMultiSub->ppMsgs[0]); + pQueue = (pRuleset == NULL) ? pMsgQueue : ruleset.GetRulesetQueue(pRuleset); + iRet = qqueueMultiEnqObj(pQueue, pMultiSub); pMultiSub->nElem = 0; +finalize_it: RETiRet; } -/* Log a message to the appropriate log files, users, etc. based on - * the priority. - * rgerhards 2004-11-08: actually, this also decodes all but the PRI part. - * rgerhards 2004-11-09: ... but only, if syslogd could properly be initialized - * if not, we use emergency logging to the console and in - * this case, no further decoding happens. - * changed to no longer receive a plain message but a msg object instead. - * rgerhards-2004-11-16: OK, we are now up to another change... This method - * actually needs to PARSE the message. How exactly this needs to happen depends on - * a number of things. Most importantly, it depends on the source. For example, - * locally received messages (SOURCE_UNIXAF) do NOT have a hostname in them. So - * we need to treat them differntly form network-received messages which have. - * Well, actually not all network-received message really have a hostname. We - * can just hope they do, but we can not be sure. So this method tries to find - * whatever can be found in the message and uses that... Obviously, there is some - * potential for misinterpretation, which we simply can not solve under the - * circumstances given. - */ -void -logmsg(msg_t *pMsg, int flags) -{ - char *msg; - - BEGINfunc - assert(pMsg != NULL); - assert(pMsg->pszRawMsg != NULL); - - msg = (char*) pMsg->pszRawMsg + pMsg->offAfterPRI; /* point to start of text, after PRI */ - DBGPRINTF("logmsg: flags %x, from '%s', msg %s\n", flags, getRcvFrom(pMsg), msg); - - /* rger 2005-11-24 (happy thanksgiving!): we now need to check if we have - * a traditional syslog message or one formatted according to syslog-protocol. - * We need to apply different parsers depending on that. We use the - * -protocol VERSION field for the detection. - */ - if(msg[0] == '1' && msg[1] == ' ') { - DBGPRINTF("Message has syslog-protocol format.\n"); - setProtocolVersion(pMsg, 1); - if(parseRFCSyslogMsg(pMsg, flags) == 1) { - msgDestruct(&pMsg); - return; - } - } else { /* we have legacy syslog */ - DBGPRINTF("Message has legacy syslog format.\n"); - setProtocolVersion(pMsg, 0); - if(parseLegacySyslogMsg(pMsg, flags) == 1) { - msgDestruct(&pMsg); - return; - } - } - - /* ---------------------- END PARSING ---------------- */ - - /* now submit the message to the main queue - then we are done */ - pMsg->msgFlags = flags; - MsgPrepareEnqueue(pMsg); - qqueueEnqObj(pMsgQueue, pMsg->flowCtlType, (void*) pMsg); - ENDfunc -} static void @@ -1449,7 +706,7 @@ DEFFUNC_llExecFunc(flushRptdMsgsActions) * the beginn of the llExec(). This makes it slightly less correct, but * in an acceptable way. -- rgerhards, 2008-09-16 */ - if (pAction->f_prevcount && time(NULL) >= REPEATTIME(pAction)) { + if (pAction->f_prevcount && datetime.GetTime(NULL) >= REPEATTIME(pAction)) { DBGPRINTF("flush %s: repeated %d times, %d sec.\n", module.GetStateName(pAction->pMod), pAction->f_prevcount, repeatinterval[pAction->f_repeatcount]); @@ -1474,13 +731,27 @@ doFlushRptdMsgs(void) static void debug_switch() { + time_t tTime; + struct tm tp; struct sigaction sigAct; + datetime.GetTime(&tTime); + localtime_r(&tTime, &tp); if(debugging_on == 0) { debugging_on = 1; - DBGPRINTF("Switching debugging_on to true\n"); + dbgprintf("\n"); + dbgprintf("\n"); + dbgprintf("********************************************************************************\n"); + dbgprintf("Switching debugging_on to true at %2.2d:%2.2d:%2.2d\n", + tp.tm_hour, tp.tm_min, tp.tm_sec); + dbgprintf("********************************************************************************\n"); } else { - DBGPRINTF("Switching debugging_on to false\n"); + dbgprintf("********************************************************************************\n"); + dbgprintf("Switching debugging_on to false at %2.2d:%2.2d:%2.2d\n", + tp.tm_hour, tp.tm_min, tp.tm_sec); + dbgprintf("********************************************************************************\n"); + dbgprintf("\n"); + dbgprintf("\n"); debugging_on = 0; } @@ -1495,7 +766,7 @@ void legacyOptsEnq(uchar *line) { legacyOptsLL_t *pNew; - pNew = malloc(sizeof(legacyOptsLL_t)); + pNew = MALLOC(sizeof(legacyOptsLL_t)); if(line == NULL) pNew->line = NULL; else @@ -1703,7 +974,7 @@ die(int sig) thrdTerminateAll(); /* and THEN send the termination log message (see long comment above) */ - if (sig) { + if(sig && bLogStatusMsgs) { (void) snprintf(buf, sizeof(buf) / sizeof(char), " [origin software=\"rsyslogd\" " "swVersion=\"" VERSION \ "\" x-pid=\"%d\" x-info=\"http://www.rsyslog.com\"]" " exiting on signal %d.", @@ -2084,13 +1355,6 @@ static void dbgPrintInitInfo(void) DBGPRINTF("Messages with malicious PTR DNS Records are %sdropped.\n", glbl.GetDropMalPTRMsgs() ? "" : "not "); - DBGPRINTF("Control characters are %sreplaced upon reception.\n", - bEscapeCCOnRcv? "" : "not "); - - if(bEscapeCCOnRcv) - DBGPRINTF("Control character escape sequence prefix is '%c'.\n", - cCCEscapeChar); - DBGPRINTF("Main queue size %d messages.\n", iMainMsgQueueSize); DBGPRINTF("Main queue worker threads: %d, wThread shutdown: %d, Perists every %d updates.\n", iMainMsgQueueNumWorkers, iMainMsgQtoWrkShutdown, iMainMsgQPersistUpdCnt); @@ -2168,6 +1432,70 @@ startInputModules(void) } +/* create a main message queue, now also used for ruleset queues. This function + * needs to be moved to some other module, but it is considered acceptable for + * the time being (remember that we want to restructure config processing at large!). + * rgerhards, 2009-10-27 + */ +rsRetVal createMainQueue(qqueue_t **ppQueue, uchar *pszQueueName) +{ + DEFiRet; + + /* switch the message object to threaded operation, if necessary */ + if(MainMsgQueType == QUEUETYPE_DIRECT || iMainMsgQueueNumWorkers > 1) { + MsgEnableThreadSafety(); + } + + /* create message queue */ + CHKiRet_Hdlr(qqueueConstruct(ppQueue, MainMsgQueType, iMainMsgQueueNumWorkers, iMainMsgQueueSize, msgConsumer)) { + /* no queue is fatal, we need to give up in that case... */ + errmsg.LogError(0, iRet, "could not create (ruleset) main message queue"); \ + } + /* name our main queue object (it's not fatal if it fails...) */ + obj.SetName((obj_t*) (*ppQueue), pszQueueName); + + /* ... set some properties ... */ +# define setQPROP(func, directive, data) \ + CHKiRet_Hdlr(func(*ppQueue, data)) { \ + errmsg.LogError(0, NO_ERRCODE, "Invalid " #directive ", error %d. Ignored, running with default setting", iRet); \ + } +# define setQPROPstr(func, directive, data) \ + CHKiRet_Hdlr(func(*ppQueue, data, (data == NULL)? 0 : strlen((char*) data))) { \ + errmsg.LogError(0, NO_ERRCODE, "Invalid " #directive ", error %d. Ignored, running with default setting", iRet); \ + } + + setQPROP(qqueueSetMaxFileSize, "$MainMsgQueueFileSize", iMainMsgQueMaxFileSize); + setQPROP(qqueueSetsizeOnDiskMax, "$MainMsgQueueMaxDiskSpace", iMainMsgQueMaxDiskSpace); + setQPROP(qqueueSetiDeqBatchSize, "$MainMsgQueueDequeueBatchSize", iMainMsgQueDeqBatchSize); + setQPROPstr(qqueueSetFilePrefix, "$MainMsgQueueFileName", pszMainMsgQFName); + setQPROP(qqueueSetiPersistUpdCnt, "$MainMsgQueueCheckpointInterval", iMainMsgQPersistUpdCnt); + setQPROP(qqueueSetbSyncQueueFiles, "$MainMsgQueueSyncQueueFiles", bMainMsgQSyncQeueFiles); + setQPROP(qqueueSettoQShutdown, "$MainMsgQueueTimeoutShutdown", iMainMsgQtoQShutdown ); + setQPROP(qqueueSettoActShutdown, "$MainMsgQueueTimeoutActionCompletion", iMainMsgQtoActShutdown); + setQPROP(qqueueSettoWrkShutdown, "$MainMsgQueueWorkerTimeoutThreadShutdown", iMainMsgQtoWrkShutdown); + setQPROP(qqueueSettoEnq, "$MainMsgQueueTimeoutEnqueue", iMainMsgQtoEnq); + setQPROP(qqueueSetiHighWtrMrk, "$MainMsgQueueHighWaterMark", iMainMsgQHighWtrMark); + setQPROP(qqueueSetiLowWtrMrk, "$MainMsgQueueLowWaterMark", iMainMsgQLowWtrMark); + setQPROP(qqueueSetiDiscardMrk, "$MainMsgQueueDiscardMark", iMainMsgQDiscardMark); + setQPROP(qqueueSetiDiscardSeverity, "$MainMsgQueueDiscardSeverity", iMainMsgQDiscardSeverity); + setQPROP(qqueueSetiMinMsgsPerWrkr, "$MainMsgQueueWorkerThreadMinimumMessages", iMainMsgQWrkMinMsgs); + setQPROP(qqueueSetbSaveOnShutdown, "$MainMsgQueueSaveOnShutdown", bMainMsgQSaveOnShutdown); + setQPROP(qqueueSetiDeqSlowdown, "$MainMsgQueueDequeueSlowdown", iMainMsgQDeqSlowdown); + setQPROP(qqueueSetiDeqtWinFromHr, "$MainMsgQueueDequeueTimeBegin", iMainMsgQueueDeqtWinFromHr); + setQPROP(qqueueSetiDeqtWinToHr, "$MainMsgQueueDequeueTimeEnd", iMainMsgQueueDeqtWinToHr); + +# undef setQPROP +# undef setQPROPstr + + /* ... and finally start the queue! */ + CHKiRet_Hdlr(qqueueStart(*ppQueue)) { + /* no queue is fatal, we need to give up in that case... */ + errmsg.LogError(0, iRet, "could not start (ruleset) main message queue"); \ + } + RETiRet; +} + + /* INIT -- Initialize syslogd * Note that if iConfigVerify is set, only the config file is verified but nothing * else happens. -- rgerhards, 2008-07-28 @@ -2184,11 +1512,6 @@ init(void) struct sigaction sigAct; DEFiRet; - /* initialize some static variables */ - pDfltHostnameCmp = NULL; - pDfltProgNameCmp = NULL; - eDfltHostnameCmpMode = HN_NO_COMP; - DBGPRINTF("rsyslog %s - called init()\n", VERSION); /* construct the default ruleset */ @@ -2238,17 +1561,6 @@ init(void) legacyOptsHook(); - /* we are now done with reading the configuration. This is the right time to - * free some objects that were just needed for loading it. rgerhards 2005-10-19 - */ - if(pDfltHostnameCmp != NULL) { - rsCStrDestruct(&pDfltHostnameCmp); - } - - if(pDfltProgNameCmp != NULL) { - rsCStrDestruct(&pDfltProgNameCmp); - } - /* some checks */ if(iMainMsgQueueNumWorkers < 1) { errmsg.LogError(0, NO_ERRCODE, "$MainMsgQueueNumWorkers must be at least 1! Set to 1.\n"); @@ -2284,59 +1596,20 @@ init(void) ABORT_FINALIZE(RS_RET_VALIDATION_RUN); } - /* switch the message object to threaded operation, if necessary */ - if(MainMsgQueType == QUEUETYPE_DIRECT || iMainMsgQueueNumWorkers > 1) { - MsgEnableThreadSafety(); + if(bAbortOnUncleanConfig && bHadConfigErr) { + fprintf(stderr, "rsyslogd: $AbortOnUncleanConfig is set, and config is not clean.\n" + "Check error log for details, fix errors and restart. As a last\n" + "resort, you may want to remove $AbortOnUncleanConfig to permit a\n" + "startup with a dirty config.\n"); + exit(2); } /* create message queue */ - CHKiRet_Hdlr(qqueueConstruct(&pMsgQueue, MainMsgQueType, iMainMsgQueueNumWorkers, iMainMsgQueueSize, msgConsumer)) { + CHKiRet_Hdlr(createMainQueue(&pMsgQueue, UCHAR_CONSTANT("main Q"))) { /* no queue is fatal, we need to give up in that case... */ fprintf(stderr, "fatal error %d: could not create message queue - rsyslogd can not run!\n", iRet); exit(1); } - /* name our main queue object (it's not fatal if it fails...) */ - obj.SetName((obj_t*) pMsgQueue, (uchar*) "main Q"); - - /* ... set some properties ... */ -# define setQPROP(func, directive, data) \ - CHKiRet_Hdlr(func(pMsgQueue, data)) { \ - errmsg.LogError(0, NO_ERRCODE, "Invalid " #directive ", error %d. Ignored, running with default setting", iRet); \ - } -# define setQPROPstr(func, directive, data) \ - CHKiRet_Hdlr(func(pMsgQueue, data, (data == NULL)? 0 : strlen((char*) data))) { \ - errmsg.LogError(0, NO_ERRCODE, "Invalid " #directive ", error %d. Ignored, running with default setting", iRet); \ - } - - setQPROP(qqueueSetMaxFileSize, "$MainMsgQueueFileSize", iMainMsgQueMaxFileSize); - setQPROP(qqueueSetsizeOnDiskMax, "$MainMsgQueueMaxDiskSpace", iMainMsgQueMaxDiskSpace); - setQPROP(qqueueSetiDeqBatchSize, "$MainMsgQueueDequeueBatchSize", iMainMsgQueDeqBatchSize); - setQPROPstr(qqueueSetFilePrefix, "$MainMsgQueueFileName", pszMainMsgQFName); - setQPROP(qqueueSetiPersistUpdCnt, "$MainMsgQueueCheckpointInterval", iMainMsgQPersistUpdCnt); - setQPROP(qqueueSetbSyncQueueFiles, "$MainMsgQueueSyncQueueFiles", bMainMsgQSyncQeueFiles); - setQPROP(qqueueSettoQShutdown, "$MainMsgQueueTimeoutShutdown", iMainMsgQtoQShutdown ); - setQPROP(qqueueSettoActShutdown, "$MainMsgQueueTimeoutActionCompletion", iMainMsgQtoActShutdown); - setQPROP(qqueueSettoWrkShutdown, "$MainMsgQueueWorkerTimeoutThreadShutdown", iMainMsgQtoWrkShutdown); - setQPROP(qqueueSettoEnq, "$MainMsgQueueTimeoutEnqueue", iMainMsgQtoEnq); - setQPROP(qqueueSetiHighWtrMrk, "$MainMsgQueueHighWaterMark", iMainMsgQHighWtrMark); - setQPROP(qqueueSetiLowWtrMrk, "$MainMsgQueueLowWaterMark", iMainMsgQLowWtrMark); - setQPROP(qqueueSetiDiscardMrk, "$MainMsgQueueDiscardMark", iMainMsgQDiscardMark); - setQPROP(qqueueSetiDiscardSeverity, "$MainMsgQueueDiscardSeverity", iMainMsgQDiscardSeverity); - setQPROP(qqueueSetiMinMsgsPerWrkr, "$MainMsgQueueWorkerThreadMinimumMessages", iMainMsgQWrkMinMsgs); - setQPROP(qqueueSetbSaveOnShutdown, "$MainMsgQueueSaveOnShutdown", bMainMsgQSaveOnShutdown); - setQPROP(qqueueSetiDeqSlowdown, "$MainMsgQueueDequeueSlowdown", iMainMsgQDeqSlowdown); - setQPROP(qqueueSetiDeqtWinFromHr, "$MainMsgQueueDequeueTimeBegin", iMainMsgQueueDeqtWinFromHr); - setQPROP(qqueueSetiDeqtWinToHr, "$MainMsgQueueDequeueTimeEnd", iMainMsgQueueDeqtWinToHr); - -# undef setQPROP -# undef setQPROPstr - - /* ... and finally start the queue! */ - CHKiRet_Hdlr(qqueueStart(pMsgQueue)) { - /* no queue is fatal, we need to give up in that case... */ - fprintf(stderr, "fatal error %d: could not start message queue - rsyslogd can not run!\n", iRet); - exit(1); - } bHaveMainQueue = (MainMsgQueType == QUEUETYPE_DIRECT) ? 0 : 1; DBGPRINTF("Main processing queue is initialized and running\n"); @@ -2365,11 +1638,13 @@ init(void) /* we now generate the startup message. It now includes everything to * identify this instance. -- rgerhards, 2005-08-17 */ - snprintf(bufStartUpMsg, sizeof(bufStartUpMsg)/sizeof(char), - " [origin software=\"rsyslogd\" " "swVersion=\"" VERSION \ - "\" x-pid=\"%d\" x-info=\"http://www.rsyslog.com\"] start", - (int) myPid); - logmsgInternal(NO_ERRCODE, LOG_SYSLOG|LOG_INFO, (uchar*)bufStartUpMsg, 0); + if(bLogStatusMsgs) { + snprintf(bufStartUpMsg, sizeof(bufStartUpMsg)/sizeof(char), + " [origin software=\"rsyslogd\" " "swVersion=\"" VERSION \ + "\" x-pid=\"%d\" x-info=\"http://www.rsyslog.com\"] start", + (int) myPid); + logmsgInternal(NO_ERRCODE, LOG_SYSLOG|LOG_INFO, (uchar*)bufStartUpMsg, 0); + } finalize_it: RETiRet; @@ -2479,14 +1754,13 @@ void sigttin_handler() * really help us. TODO: add error messages? * rgerhards, 2007-08-03 */ -static void processImInternal(void) +static inline void processImInternal(void) { int iPri; - int iFlags; msg_t *pMsg; - while(iminternalRemoveMsg(&iPri, &pMsg, &iFlags) == RS_RET_OK) { - logmsg(pMsg, iFlags); + while(iminternalRemoveMsg(&iPri, &pMsg) == RS_RET_OK) { + submitMsg(pMsg); } } @@ -2514,11 +1788,14 @@ doHUP(void) { char buf[512]; - snprintf(buf, sizeof(buf) / sizeof(char), - " [origin software=\"rsyslogd\" " "swVersion=\"" VERSION - "\" x-pid=\"%d\" x-info=\"http://www.rsyslog.com\"] rsyslogd was HUPed", - (int) myPid); - errno = 0; + if(bLogStatusMsgs) { + snprintf(buf, sizeof(buf) / sizeof(char), + " [origin software=\"rsyslogd\" " "swVersion=\"" VERSION + "\" x-pid=\"%d\" x-info=\"http://www.rsyslog.com\"] rsyslogd was HUPed", + (int) myPid); + errno = 0; + logmsgInternal(NO_ERRCODE, LOG_SYSLOG|LOG_INFO, (uchar*)buf, 0); + } ruleset.IterateAllActions(doHUPActions, NULL); } @@ -2625,6 +1902,14 @@ static rsRetVal loadBuildInModules(void) */ CHKiRet(module.doModInit(modInitUsrMsg, (uchar*) "builtin-usrmsg", NULL)); + /* load build-in parser modules */ + CHKiRet(module.doModInit(modInitpmrfc5424, UCHAR_CONSTANT("builtin-pmrfc5424"), NULL)); + CHKiRet(module.doModInit(modInitpmrfc3164, UCHAR_CONSTANT("builtin-pmrfc3164"), NULL)); + + /* and set default parser modules (order is *very* important, legacy (3164) parse needs to go last! */ + CHKiRet(parser.AddDfltParser(UCHAR_CONSTANT("rsyslog.rfc5424"))); + CHKiRet(parser.AddDfltParser(UCHAR_CONSTANT("rsyslog.rfc3164"))); + /* ok, initialization of the command handler probably does not 100% belong right in * this space here. However, with the current design, this is actually quite a good * place to put it. We might decide to shuffle it around later, but for the time @@ -2632,6 +1917,7 @@ static rsRetVal loadBuildInModules(void) * is that rsyslog will terminate if we can not register our built-in config commands. * This, I think, is the right thing to do. -- rgerhards, 2007-07-31 */ + CHKiRet(regCfSysLineHdlr((uchar *)"logrsyslogstatusmessages", 0, eCmdHdlrBinary, NULL, &bLogStatusMsgs, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"actionresumeretrycount", 0, eCmdHdlrInt, NULL, &glbliActionResumeRetryCount, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"defaultruleset", 0, eCmdHdlrGetWord, setDefaultRuleset, NULL, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"ruleset", 0, eCmdHdlrGetWord, setCurrRuleset, NULL, NULL)); @@ -2657,13 +1943,10 @@ static rsRetVal loadBuildInModules(void) CHKiRet(regCfSysLineHdlr((uchar *)"mainmsgqueuesaveonshutdown", 0, eCmdHdlrBinary, NULL, &bMainMsgQSaveOnShutdown, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"mainmsgqueuedequeuetimebegin", 0, eCmdHdlrInt, NULL, &iMainMsgQueueDeqtWinFromHr, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"mainmsgqueuedequeuetimeend", 0, eCmdHdlrInt, NULL, &iMainMsgQueueDeqtWinToHr, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"abortonuncleanconfig", 0, eCmdHdlrBinary, NULL, &bAbortOnUncleanConfig, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"repeatedmsgreduction", 0, eCmdHdlrBinary, NULL, &bReduceRepeatMsgs, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"actionexeconlywhenpreviousissuspended", 0, eCmdHdlrBinary, NULL, &bActExecWhenPrevSusp, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionexeconlyonceeveryinterval", 0, eCmdHdlrInt, NULL, &iActExecOnceInterval, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"actionresumeinterval", 0, eCmdHdlrInt, setActionResumeInterval, NULL, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"controlcharacterescapeprefix", 0, eCmdHdlrGetChar, NULL, &cCCEscapeChar, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"escapecontrolcharactersonreceive", 0, eCmdHdlrBinary, NULL, &bEscapeCCOnRcv, NULL)); - CHKiRet(regCfSysLineHdlr((uchar *)"droptrailinglfonreception", 0, eCmdHdlrBinary, NULL, &bDropTrailingLF, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"template", 0, eCmdHdlrCustomHandler, conf.doNameLine, (void*)DIR_TEMPLATE, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"outchannel", 0, eCmdHdlrCustomHandler, conf.doNameLine, (void*)DIR_OUTCHANNEL, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"allowedsender", 0, eCmdHdlrCustomHandler, conf.doNameLine, (void*)DIR_ALLOWEDSENDER, NULL)); @@ -2685,11 +1968,6 @@ static rsRetVal loadBuildInModules(void) CHKiRet(regCfSysLineHdlr((uchar *)"privdroptogroup", 0, eCmdHdlrGID, NULL, &gidDropPriv, NULL)); CHKiRet(regCfSysLineHdlr((uchar *)"privdroptogroupid", 0, eCmdHdlrGID, NULL, &gidDropPriv, NULL)); - /* now add other modules handlers (we should work on that to be able to do it in ClassInit(), but so far - * that is not possible). -- rgerhards, 2008-01-28 - */ - CHKiRet(actionAddCfSysLineHdrl()); - finalize_it: RETiRet; } @@ -2863,14 +2141,14 @@ InitGlobalClasses(void) CHKiRet(objUse(conf, CORE_COMPONENT)); pErrObj = "prop"; CHKiRet(objUse(prop, CORE_COMPONENT)); + pErrObj = "parser"; + CHKiRet(objUse(parser, CORE_COMPONENT)); /* intialize some dummy classes that are not part of the runtime */ pErrObj = "action"; CHKiRet(actionClassInit()); pErrObj = "template"; CHKiRet(templateInit()); - pErrObj = "parser"; - CHKiRet(parserClassInit()); /* TODO: the dependency on net shall go away! -- rgerhards, 2008-03-07 */ pErrObj = "net"; @@ -2909,6 +2187,7 @@ GlobalClassExit(void) objRelease(rule, CORE_COMPONENT); objRelease(expr, CORE_COMPONENT); vmClassExit(); /* this is hack, currently core_modules do not get this automatically called */ + parserClassExit(); /* this is hack, currently core_modules do not get this automatically called */ objRelease(datetime, CORE_COMPONENT); /* TODO: implement the rest of the deinit */ @@ -2949,7 +2228,7 @@ bufOptAdd(char opt, char *arg) DEFiRet; bufOpt_t *pBuf; - if((pBuf = malloc(sizeof(bufOpt_t))) == NULL) + if((pBuf = MALLOC(sizeof(bufOpt_t))) == NULL) ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); pBuf->optchar = opt; @@ -3042,8 +2321,6 @@ doGlblProcessInit(void) fputs(" Already running.\n", stderr); exit(1); /* "good" exit, done if syslogd is already running */ } - } else { - debugging_on = 1; } /* tuck my process id away */ @@ -3372,10 +2649,10 @@ int realMain(int argc, char **argv) fprintf(stderr, "error -p is no longer supported, use module imuxsock instead"); } case 'q': /* add hostname if DNS resolving has failed */ - *net.pACLAddHostnameOnFail = 1; + net.pACLAddHostnameOnFail = 1; break; case 'Q': /* dont resolve hostnames in ACL to IPs */ - *net.pACLDontResolve = 1; + net.pACLDontResolve = 1; break; case 'r': /* accept remote messages */ if(iCompatibilityMode < 3) { @@ -3407,7 +2684,7 @@ int realMain(int argc, char **argv) case 'u': /* misc user settings */ iHelperUOpt = atoi(arg); if(iHelperUOpt & 0x01) - bParseHOSTNAMEandTAG = 0; + glbl.SetParseHOSTNAMEandTAG(0); if(iHelperUOpt & 0x02) bChDirRoot = 0; break; diff --git a/tools/syslogd.h b/tools/syslogd.h index 3dfdbe2b..c3b99f9d 100644 --- a/tools/syslogd.h +++ b/tools/syslogd.h @@ -32,7 +32,6 @@ /* the following prototypes should go away once we have an input * module interface -- rgerhards, 2007-12-12 */ -void logmsg(msg_t *pMsg, int flags); extern int NoHops; extern int send_to_all; extern int Debug; |
