diff options
128 files changed, 5388 insertions, 350 deletions
@@ -27,7 +27,170 @@ Version 4.7.0 [v4-devel] (rgerhards), 2009-09-?? (bugs require certain non-standard settings to appear) Thanks to varmojfekoj for the patch [imported from 4.5.8] --------------------------------------------------------------------------- -Version 4.5.8 [v4-beta] (rgerhards), 2010-01-?? +Version 4.6.3 [v4-stable] (rgerhards), 2010-03-?? +- improvded testbench + - added test with truly random data received via syslog to test + robustness +- bugfix: default for $OMFileFlushOnTXEnd was wrong ("off"). + This, in default mode, caused buffered writing to be used, what + means that it looked like no output were written or partial + lines. Thanks to Michael Biebl for pointing out this bug. +- bugfix: testbench failed when not executed in UTC+1 timezone + accidently, the time zone information was kept inside some + to-be-checked-for responses +- temporary bugfix replaced by permanent one for + message-induced off-by-one error (potential segfault) (see 4.6.2) + The analysis has been completed and a better fix been crafted and + integrated. +- bugfix: the T/P/E config size specifiers did not work properly under + all 32-bit platforms +- bugfix: local unix system log socket was deleted even when it was + not configured +--------------------------------------------------------------------------- +Version 4.6.2 [v4-stable] (rgerhards), 2010-03-26 +- new feature: "." action type added to support writing files to relative + pathes (this is primarily meant as a debug aid) +- added replacements for atomic instructions on systems that do not + support them. [backport of Stefen Sledz' patch for v5) +- new feature: $OMFileAsyncWriting directive added + it permits to specifiy if asynchronous writing should be done or not +- bugfix(temporary): message-induced off-by-one error (potential segfault) + Some types of malformed messages could trigger an off-by-one error + (for example, \0 or \n as the last character, and generally control + character escaption is questionable). This is due to not strictly + following a the \0 or string counted string paradigm (during the last + optimization on the cstring class). As a temporary fix, we have + introduced a proper recalculation of the size. However, a final + patch is expected in the future. See bug tracker for further details + and when the final patch will be available: + http://bugzilla.adiscon.com/show_bug.cgi?id=184 + Note that the current patch is considered sufficient to solve the + situation, but it requires a bit more runtime than desirable. +- bugfix: potential segfault in dynafile cache + This bug was triggered by an open failure. The the cache was full and + a new entry needed to be placed inside it, a victim for eviction was + selected. That victim was freed, then the open of the new file tried. If + the open failed, the victim entry was still freed, and the function + exited. However, on next invocation and cache search, the victim entry + was used as if it were populated, most probably resulting in a segfault. +- bugfix: race condition during directory creation + If multiple files try to create a directory at (almost) the same time, + some of them may fail. This is a data race and also exists with other + processes that may create the same directory. We do now check for this + condition and gracefully handle it. +- bugfix: potential re-use of free()ed file stream object in omfile + when dynaCache is enabled, the cache is full, a new entry needs to + be allocated, thus the LRU discarded, then a new entry is opend and that + fails. In that case, it looks like the discarded stream may be reused + improperly (based on code analysis, test case and confirmation pending) +- added new property replacer option "date-rfc3164-buggyday" primarily + to ease migration from syslog-ng. See property replacer doc for + details. [backport from 5.5.3 because urgently needed by some] +- improved testbench +- bugfix: invalid buffer write in (file) stream class + currently being accessed buffer could be overwritten with new data. + While this probably did not cause access violations, it could case loss + and/or duplication of some data (definitely a race with no deterministic + outcome) +- bugfix: potential hang condition during filestream close + predicate was not properly checked when waiting for the background file + writer +- bugfix: improper synchronization when "$OMFileFlushOnTXEnd on" was used + Internal data structures were not properly protected due to missing + mutex calls. +- bugfix: potential data loss during file stream shutdown +- bugfix: potential problems during file stream shutdown + The shutdown/close sequence was not clean, what potentially (but + unlikely) could lead to some issues. We have not been able to describe + any fatal cases, but there was some bug potential. Sequence has now + been straighted out. +- bugfix: potential problem (loop, abort) when file write error occured + When a write error occured in stream.c, variable iWritten had the error + code but this was handled as if it were the actual number of bytes + written. That was used in pointer arithmetic later on, and thus could + lead to all sorts of problems. However, this could only happen if the + error was EINTR or the file in question was a tty. All other cases were + handled properly. Now, iWritten is reset to zero in such cases, resulting + in proper retries. +- bugfix: $omfileFlushOnTXEnd was turned on when set to off and vice + versa due to an invalid check +- bugfix: recent patch to fix small memory leak could cause invalid free. + This could only happen during config file parsing. +- bugfix(minor): handling of extremely large strings in dbgprintf() fixed + Previously, it could lead to garbagge output and, in extreme cases, also + to segfaults. Note: this was a problem only when debug output was + actually enabled, so it caused no problem in production use. +- bugfix(minor): BSD_SO_COMPAT query function had some global vars not + properly initialized. However, in practice the loader initializes them + with zero, the desired value, so there were no actual issue in almost + all cases. +--------------------------------------------------------------------------- +Version 4.6.1 [v4-stable] (rgerhards), 2010-03-04 +- re-enabled old pipe output (using new module ompipe, built-in) after + some problems with pipes (and especially in regard to xconsole) were + discovered. Thanks to Michael Biebl for reporting the issues. +- bugfix: potential problems with large file support could cause segfault + ... and other weird problems. This seemed to affect 32bit-platforms + only, but I can not totally outrule there were issues on other + platforms as well. The previous code could cause system data types + to be defined inconsistently, and that could lead to various + troubles. Special thanks go to the Mandriva team for identifying + an initial problem, help discussing it and ultimately a fix they + contributed. +- bugfix: fixed problem that caused compilation on FreeBSD 9.0 to fail. + bugtracker: http://bugzilla.adiscon.com/show_bug.cgi?id=181 + Thanks to Christiano for reporting. +- bugfix: potential segfault in omfile when a dynafile open failed + In that case, a partial cache entry was written, and some internal + pointers (iCurrElt) not correctly updated. In the next iteration, that + could lead to a segfault, especially if iCurrElt then points to the + then-partial record. Not very likely, but could happen in practice. +- bugfix (theoretical): potential segfault in omfile under low memory + condition. This is only a theoretical bug, because it would only + happen when strdup() fails to allocate memory - which is highly + unlikely and will probably lead to all other sorts of errors. +- bugfix: comment char ('#') in literal terminated script parsing + and thus could not be used. + but tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=119 + [merged in from v3.22.2] +--------------------------------------------------------------------------- +Version 4.6.0 [v4-stable] (rgerhards), 2010-02-24 +*************************************************************************** +* This is a new stable v4 version. It contains all fixes and enhancements * +* made during the 4.5.x phase as well as those listed below. * +* Note: this version is scheduled to conclude the v4 development process. * +* Do not expect any more new developments in v4. The focus is now * +* on v5 (what also means we have a single devel branch again). * +* ("development" means new feature development, bug fixes are of * +* course provided for v4-stable) * +*************************************************************************** +- improved testbench to contain samples for totally malformed messages + which miss parts of the message content +- bugfix: some malformed messages could lead to a missing LF inside files + or some other missing parts of the template content. +- bugfix: if a message ended immediately with a hostname, the hostname + was mistakenly interpreted as TAG, and localhost be used as hostname +- bugfix: message without MSG part could case a segfault + [backported from v5 commit 98d1ed504ec001728955a5bcd7916f64cd85f39f] + This actually was a "recent" regression, but I did not realize that it + was introduced by the performance optimization in v4-devel. Shame on + me for having two devel versions at the same time... +--------------------------------------------------------------------------- +Version 4.5.8 [v4-beta] (rgerhards), 2010-02-10 +- enhanced doc for using PostgreSQL + Thanks to Marc Schiffbauer for the new/updated doc +- bugfix: property replacer returned invalid parameters under some (unusual) + conditions. In extreme cases, this could lead to garbled logs and/or + a system failure. +- bugfix: invalid length returned (often) when using regular expressions + inside the property replacer +- bugfix: submatch regex in property replacer did not honor "return 0 on + no match" config case +- bugfix: imuxsock incorrectly stated inputname "imudp" + Thanks to Ryan Lynch for reporting this. +- (slightly) enhanced support for FreeBSD by setting _PATH_MODDIR to + the correct value on FreeBSD. + Thanks to Cristiano for the patch. - bugfix: -d did not enable display of debug messages regression from introduction of "debug on demand" mode Thanks to Michael Biebl for reporting this bug @@ -216,6 +379,8 @@ Version 4.5.0 [DEVEL] (rgerhards), 2009-07-02 an abort --------------------------------------------------------------------------- Version 4.4.3 [v4-stable] (rgerhards), 2009-10-?? +- bugfix: several smaller bugs resolved after flexelint review + Thanks to varmojfekoj for the patch. - bugfix: $ActionExecOnlyOnceEveryInterval did not work. This was a regression from the time() optimizations done in v4. Bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=143 @@ -478,6 +643,9 @@ version before switching to this one. Thanks to Ken for providing the patch --------------------------------------------------------------------------- Version 3.22.2 [v3-stable] (rgerhards), 2009-07-?? +- bugfix: comment char ('#') in literal terminated script parsing + and thus could not be used. + but tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=119 - enhance: imrelp now also provides remote peer's IP address [if librelp != 1.0.0 is used] - bugfix: sending syslog messages with zip compression did not work diff --git a/INSTALL b/INSTALL new file mode 100644 index 00000000..7d1c323b --- /dev/null +++ b/INSTALL @@ -0,0 +1,365 @@ +Installation Instructions +************************* + +Copyright (C) 1994, 1995, 1996, 1999, 2000, 2001, 2002, 2004, 2005, +2006, 2007, 2008, 2009 Free Software Foundation, Inc. + + Copying and distribution of this file, with or without modification, +are permitted in any medium without royalty provided the copyright +notice and this notice are preserved. This file is offered as-is, +without warranty of any kind. + +Basic Installation +================== + + Briefly, the shell commands `./configure; make; make install' should +configure, build, and install this package. The following +more-detailed instructions are generic; see the `README' file for +instructions specific to this package. Some packages provide this +`INSTALL' file but do not implement all of the features documented +below. The lack of an optional feature in a given package is not +necessarily a bug. More recommendations for GNU packages can be found +in *note Makefile Conventions: (standards)Makefile Conventions. + + The `configure' shell script attempts to guess correct values for +various system-dependent variables used during compilation. It uses +those values to create a `Makefile' in each directory of the package. +It may also create one or more `.h' files containing system-dependent +definitions. Finally, it creates a shell script `config.status' that +you can run in the future to recreate the current configuration, and a +file `config.log' containing compiler output (useful mainly for +debugging `configure'). + + It can also use an optional file (typically called `config.cache' +and enabled with `--cache-file=config.cache' or simply `-C') that saves +the results of its tests to speed up reconfiguring. Caching is +disabled by default to prevent problems with accidental use of stale +cache files. + + If you need to do unusual things to compile the package, please try +to figure out how `configure' could check whether to do them, and mail +diffs or instructions to the address given in the `README' so they can +be considered for the next release. If you are using the cache, and at +some point `config.cache' contains results you don't want to keep, you +may remove or edit it. + + The file `configure.ac' (or `configure.in') is used to create +`configure' by a program called `autoconf'. You need `configure.ac' if +you want to change it or regenerate `configure' using a newer version +of `autoconf'. + + The simplest way to compile this package is: + + 1. `cd' to the directory containing the package's source code and type + `./configure' to configure the package for your system. + + Running `configure' might take a while. While running, it prints + some messages telling which features it is checking for. + + 2. Type `make' to compile the package. + + 3. Optionally, type `make check' to run any self-tests that come with + the package, generally using the just-built uninstalled binaries. + + 4. Type `make install' to install the programs and any data files and + documentation. When installing into a prefix owned by root, it is + recommended that the package be configured and built as a regular + user, and only the `make install' phase executed with root + privileges. + + 5. Optionally, type `make installcheck' to repeat any self-tests, but + this time using the binaries in their final installed location. + This target does not install anything. Running this target as a + regular user, particularly if the prior `make install' required + root privileges, verifies that the installation completed + correctly. + + 6. You can remove the program binaries and object files from the + source code directory by typing `make clean'. To also remove the + files that `configure' created (so you can compile the package for + a different kind of computer), type `make distclean'. There is + also a `make maintainer-clean' target, but that is intended mainly + for the package's developers. If you use it, you may have to get + all sorts of other programs in order to regenerate files that came + with the distribution. + + 7. Often, you can also type `make uninstall' to remove the installed + files again. In practice, not all packages have tested that + uninstallation works correctly, even though it is required by the + GNU Coding Standards. + + 8. Some packages, particularly those that use Automake, provide `make + distcheck', which can by used by developers to test that all other + targets like `make install' and `make uninstall' work correctly. + This target is generally not run by end users. + +Compilers and Options +===================== + + Some systems require unusual options for compilation or linking that +the `configure' script does not know about. Run `./configure --help' +for details on some of the pertinent environment variables. + + You can give `configure' initial values for configuration parameters +by setting variables in the command line or in the environment. Here +is an example: + + ./configure CC=c99 CFLAGS=-g LIBS=-lposix + + *Note Defining Variables::, for more details. + +Compiling For Multiple Architectures +==================================== + + You can compile the package for more than one kind of computer at the +same time, by placing the object files for each architecture in their +own directory. To do this, you can use GNU `make'. `cd' to the +directory where you want the object files and executables to go and run +the `configure' script. `configure' automatically checks for the +source code in the directory that `configure' is in and in `..'. This +is known as a "VPATH" build. + + With a non-GNU `make', it is safer to compile the package for one +architecture at a time in the source code directory. After you have +installed the package for one architecture, use `make distclean' before +reconfiguring for another architecture. + + On MacOS X 10.5 and later systems, you can create libraries and +executables that work on multiple system types--known as "fat" or +"universal" binaries--by specifying multiple `-arch' options to the +compiler but only a single `-arch' option to the preprocessor. Like +this: + + ./configure CC="gcc -arch i386 -arch x86_64 -arch ppc -arch ppc64" \ + CXX="g++ -arch i386 -arch x86_64 -arch ppc -arch ppc64" \ + CPP="gcc -E" CXXCPP="g++ -E" + + This is not guaranteed to produce working output in all cases, you +may have to build one architecture at a time and combine the results +using the `lipo' tool if you have problems. + +Installation Names +================== + + By default, `make install' installs the package's commands under +`/usr/local/bin', include files under `/usr/local/include', etc. You +can specify an installation prefix other than `/usr/local' by giving +`configure' the option `--prefix=PREFIX', where PREFIX must be an +absolute file name. + + You can specify separate installation prefixes for +architecture-specific files and architecture-independent files. If you +pass the option `--exec-prefix=PREFIX' to `configure', the package uses +PREFIX as the prefix for installing programs and libraries. +Documentation and other data files still use the regular prefix. + + In addition, if you use an unusual directory layout you can give +options like `--bindir=DIR' to specify different values for particular +kinds of files. Run `configure --help' for a list of the directories +you can set and what kinds of files go in them. In general, the +default for these options is expressed in terms of `${prefix}', so that +specifying just `--prefix' will affect all of the other directory +specifications that were not explicitly provided. + + The most portable way to affect installation locations is to pass the +correct locations to `configure'; however, many packages provide one or +both of the following shortcuts of passing variable assignments to the +`make install' command line to change installation locations without +having to reconfigure or recompile. + + The first method involves providing an override variable for each +affected directory. For example, `make install +prefix=/alternate/directory' will choose an alternate location for all +directory configuration variables that were expressed in terms of +`${prefix}'. Any directories that were specified during `configure', +but not in terms of `${prefix}', must each be overridden at install +time for the entire installation to be relocated. The approach of +makefile variable overrides for each directory variable is required by +the GNU Coding Standards, and ideally causes no recompilation. +However, some platforms have known limitations with the semantics of +shared libraries that end up requiring recompilation when using this +method, particularly noticeable in packages that use GNU Libtool. + + The second method involves providing the `DESTDIR' variable. For +example, `make install DESTDIR=/alternate/directory' will prepend +`/alternate/directory' before all installation names. The approach of +`DESTDIR' overrides is not required by the GNU Coding Standards, and +does not work on platforms that have drive letters. On the other hand, +it does better at avoiding recompilation issues, and works well even +when some directory options were not specified in terms of `${prefix}' +at `configure' time. + +Optional Features +================= + + If the package supports it, you can cause programs to be installed +with an extra prefix or suffix on their names by giving `configure' the +option `--program-prefix=PREFIX' or `--program-suffix=SUFFIX'. + + Some packages pay attention to `--enable-FEATURE' options to +`configure', where FEATURE indicates an optional part of the package. +They may also pay attention to `--with-PACKAGE' options, where PACKAGE +is something like `gnu-as' or `x' (for the X Window System). The +`README' should mention any `--enable-' and `--with-' options that the +package recognizes. + + For packages that use the X Window System, `configure' can usually +find the X include and library files automatically, but if it doesn't, +you can use the `configure' options `--x-includes=DIR' and +`--x-libraries=DIR' to specify their locations. + + Some packages offer the ability to configure how verbose the +execution of `make' will be. For these packages, running `./configure +--enable-silent-rules' sets the default to minimal output, which can be +overridden with `make V=1'; while running `./configure +--disable-silent-rules' sets the default to verbose, which can be +overridden with `make V=0'. + +Particular systems +================== + + On HP-UX, the default C compiler is not ANSI C compatible. If GNU +CC is not installed, it is recommended to use the following options in +order to use an ANSI C compiler: + + ./configure CC="cc -Ae -D_XOPEN_SOURCE=500" + +and if that doesn't work, install pre-built binaries of GCC for HP-UX. + + On OSF/1 a.k.a. Tru64, some versions of the default C compiler cannot +parse its `<wchar.h>' header file. The option `-nodtk' can be used as +a workaround. If GNU CC is not installed, it is therefore recommended +to try + + ./configure CC="cc" + +and if that doesn't work, try + + ./configure CC="cc -nodtk" + + On Solaris, don't put `/usr/ucb' early in your `PATH'. This +directory contains several dysfunctional programs; working variants of +these programs are available in `/usr/bin'. So, if you need `/usr/ucb' +in your `PATH', put it _after_ `/usr/bin'. + + On Haiku, software installed for all users goes in `/boot/common', +not `/usr/local'. It is recommended to use the following options: + + ./configure --prefix=/boot/common + +Specifying the System Type +========================== + + There may be some features `configure' cannot figure out +automatically, but needs to determine by the type of machine the package +will run on. Usually, assuming the package is built to be run on the +_same_ architectures, `configure' can figure that out, but if it prints +a message saying it cannot guess the machine type, give it the +`--build=TYPE' option. TYPE can either be a short name for the system +type, such as `sun4', or a canonical name which has the form: + + CPU-COMPANY-SYSTEM + +where SYSTEM can have one of these forms: + + OS + KERNEL-OS + + See the file `config.sub' for the possible values of each field. If +`config.sub' isn't included in this package, then this package doesn't +need to know the machine type. + + If you are _building_ compiler tools for cross-compiling, you should +use the option `--target=TYPE' to select the type of system they will +produce code for. + + If you want to _use_ a cross compiler, that generates code for a +platform different from the build platform, you should specify the +"host" platform (i.e., that on which the generated programs will +eventually be run) with `--host=TYPE'. + +Sharing Defaults +================ + + If you want to set default values for `configure' scripts to share, +you can create a site shell script called `config.site' that gives +default values for variables like `CC', `cache_file', and `prefix'. +`configure' looks for `PREFIX/share/config.site' if it exists, then +`PREFIX/etc/config.site' if it exists. Or, you can set the +`CONFIG_SITE' environment variable to the location of the site script. +A warning: not all `configure' scripts look for a site script. + +Defining Variables +================== + + Variables not defined in a site shell script can be set in the +environment passed to `configure'. However, some packages may run +configure again during the build, and the customized values of these +variables may be lost. In order to avoid this problem, you should set +them in the `configure' command line, using `VAR=value'. For example: + + ./configure CC=/usr/local2/bin/gcc + +causes the specified `gcc' to be used as the C compiler (unless it is +overridden in the site shell script). + +Unfortunately, this technique does not work for `CONFIG_SHELL' due to +an Autoconf bug. Until the bug is fixed you can use this workaround: + + CONFIG_SHELL=/bin/bash /bin/bash ./configure CONFIG_SHELL=/bin/bash + +`configure' Invocation +====================== + + `configure' recognizes the following options to control how it +operates. + +`--help' +`-h' + Print a summary of all of the options to `configure', and exit. + +`--help=short' +`--help=recursive' + Print a summary of the options unique to this package's + `configure', and exit. The `short' variant lists options used + only in the top level, while the `recursive' variant lists options + also present in any nested packages. + +`--version' +`-V' + Print the version of Autoconf used to generate the `configure' + script, and exit. + +`--cache-file=FILE' + Enable the cache: use and save the results of the tests in FILE, + traditionally `config.cache'. FILE defaults to `/dev/null' to + disable caching. + +`--config-cache' +`-C' + Alias for `--cache-file=config.cache'. + +`--quiet' +`--silent' +`-q' + Do not print messages saying which checks are being made. To + suppress all normal output, redirect it to `/dev/null' (any error + messages will still be shown). + +`--srcdir=DIR' + Look for the package's source code in directory DIR. Usually + `configure' can determine that directory automatically. + +`--prefix=DIR' + Use DIR as the installation prefix. *note Installation Names:: + for more details, including other options available for fine-tuning + the installation locations. + +`--no-create' +`-n' + Run the configure checks, but stop before creating any output + files. + +`configure' also accepts some other, not widely useful, options. Run +`configure --help' for more details. + diff --git a/Makefile.am b/Makefile.am index a050e95e..e69349d8 100644 --- a/Makefile.am +++ b/Makefile.am @@ -59,6 +59,10 @@ if ENABLE_IMKLOG SUBDIRS += plugins/imklog endif +if ENABLE_IMDOOR +SUBDIRS += plugins/imdoor +endif + if ENABLE_GSSAPI SUBDIRS += plugins/omgssapi plugins/imgssapi endif @@ -4,7 +4,7 @@ * * File begun on 2007-08-06 by RGerhards (extracted from syslogd.c) * - * Copyright 2007-2009 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2010 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * diff --git a/configure.ac b/configure.ac index 8c6fc6b1..14cb2ee6 100644 --- a/configure.ac +++ b/configure.ac @@ -69,7 +69,7 @@ AC_SUBST(DL_LIBS) AC_HEADER_RESOLV AC_HEADER_STDC AC_HEADER_SYS_WAIT -AC_CHECK_HEADERS([arpa/inet.h libgen.h malloc.h fcntl.h locale.h netdb.h netinet/in.h paths.h stddef.h stdlib.h string.h sys/file.h sys/ioctl.h sys/param.h sys/socket.h sys/time.h sys/stat.h syslog.h unistd.h utmp.h sys/epoll.h sys/prctl.h]) +AC_CHECK_HEADERS([arpa/inet.h libgen.h malloc.h fcntl.h locale.h netdb.h netinet/in.h paths.h stddef.h stdlib.h string.h sys/file.h sys/ioctl.h sys/param.h sys/socket.h sys/time.h sys/stat.h syslog.h unistd.h utmp.h utmpx.h sys/epoll.h sys/prctl.h]) # Checks for typedefs, structures, and compiler characteristics. AC_C_CONST @@ -84,6 +84,7 @@ AC_TYPE_UINT8_T AC_HEADER_TIME AC_STRUCT_TM AC_C_VOLATILE +AC_C_TYPEOF sa_includes="\ $ac_includes_default @@ -126,6 +127,9 @@ AC_TRY_COMPILE([ # check for availability of atomic operations RS_ATOMIC_OPERATIONS +# fall back to POSIX sems for atomic operations (cpu expensive) +AC_CHECK_HEADERS([semaphore.h]) + # Additional module directories AC_ARG_WITH(moddirs, @@ -152,19 +156,12 @@ AC_SUBST(moddirs) # Large file support -AC_ARG_ENABLE(largefile, - [AS_HELP_STRING([--enable-largefile],[Enable large file support @<:@default=yes@:>@])], - [case "${enableval}" in - yes) enable_largefile="yes" ;; - no) enable_largefile="no" ;; - *) AC_MSG_ERROR(bad value ${enableval} for --enable-largefile) ;; - esac], - [enable_largefile="yes"] -) -if test "$enable_largefile" = "no"; then - AC_DEFINE(NOLARGEFILE, 1, [Defined when large file support is disabled.]) -fi - +# http://www.gnu.org/software/autoconf/manual/html_node/System-Services.html#index-AC_005fSYS_005fLARGEFILE-1028 +AC_SYS_LARGEFILE +case "${enable_largefile}" in + no) ;; + *) enable_largefile="yes" ;; +esac # Regular expressions AC_ARG_ENABLE(regexp, @@ -281,6 +278,7 @@ AC_ARG_ENABLE(klog, AM_CONDITIONAL(ENABLE_IMKLOG, test x$enable_klog = xyes) AM_CONDITIONAL(ENABLE_IMKLOG_BSD, test x$os_type = xbsd) AM_CONDITIONAL(ENABLE_IMKLOG_LINUX, test x$os_type = xlinux) +AM_CONDITIONAL(ENABLE_IMKLOG_SOLARIS, test x$os_type = xsolaris) # @@ -732,6 +730,19 @@ AC_ARG_ENABLE(imfile, AM_CONDITIONAL(ENABLE_IMFILE, test x$enable_imfile = xyes) +# settings for the door input module (under solaris, thus default off) +AC_ARG_ENABLE(imdoor, + [AS_HELP_STRING([--enable-imdoor],[door input module enabled @<:@default=no@:>@])], + [case "${enableval}" in + yes) enable_imdoor="yes" ;; + no) enable_imdoor="no" ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-imdoor) ;; + esac], + [enable_imdoor=no] +) +AM_CONDITIONAL(ENABLE_IMDOOR, test x$enable_imdoor = xyes) + + # settings for the omprog output module AC_ARG_ENABLE(omprog, [AS_HELP_STRING([--enable-omprog],[Compiles omprog module @<:@default=no@:>@])], @@ -833,6 +844,7 @@ AC_CONFIG_FILES([Makefile \ plugins/omprog/Makefile \ plugins/omstdout/Makefile \ plugins/imfile/Makefile \ + plugins/imdoor/Makefile \ plugins/imrelp/Makefile \ plugins/imdiag/Makefile \ plugins/omtesting/Makefile \ @@ -865,6 +877,7 @@ echo "---{ input plugins }---" echo " Klog functionality enabled: $enable_klog ($os_type)" echo " imdiag enabled: $enable_imdiag" echo " file input module enabled: $enable_imfile" +echo " Solaris door input module enabled: $enable_imdoor" echo " input template module will be compiled: $enable_imtemplate" echo echo "---{ output plugins }---" diff --git a/doc/Makefile.am b/doc/Makefile.am index 3dfc8d3a..a1f192ee 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -19,6 +19,7 @@ html_files = \ rsyslog_conf.html \ rsyslog-example.conf \ rsyslog_mysql.html \ + rsyslog_pgsql.html \ rsyslog_packages.html \ rsyslog_high_database_rate.html \ rsyslog_php_syslog_ng.html \ diff --git a/doc/manual.html b/doc/manual.html index 52a8380e..3d9a2f2d 100644 --- a/doc/manual.html +++ b/doc/manual.html @@ -45,6 +45,7 @@ if you do not read the doc, but doing so will definitely improve your experience <li><a href="generic_design.html">backgrounder on generic syslog application design</a> <li><a href="modules.html">description of rsyslog modules</a></li> +<li><a href="http://cookbook.rsyslog.com">the rsyslog "cookbook"</a> - a set of configurations ready to use</li> </ul> <p><b>We have some in-depth papers on</b></p> <ul> @@ -55,6 +56,7 @@ generic syslog application design</a> <li><a href="multi_ruleset.html">using multiple rule sets in rsyslog</a></li> <li><a href="rsyslog_stunnel.html">ssl-encrypting syslog with stunnel</a></li> <li><a href="rsyslog_mysql.html">writing syslog messages to MySQL (and other databases as well)</a></li> +<li><a href="rsyslog_pgsql.html">writing syslog messages to PostgreSQL (and other databases as well)</a></li> <li><a href="rsyslog_high_database_rate.html">writing massive amounts of syslog messages to a database</a></li> <li><a href="rsyslog_reliable_forwarding.html">reliable forwarding to a remote server</a></li> <li><a href="rsyslog_php_syslog_ng.html">using diff --git a/doc/property_replacer.html b/doc/property_replacer.html index 7b604ea0..4d242a34 100644 --- a/doc/property_replacer.html +++ b/doc/property_replacer.html @@ -335,6 +335,19 @@ Especially useful for PIX.</td> <td>format as RFC 3164 date</td> </tr> <tr> +<tr> +<td valign="top"><b>date-rfc3164-buggyday</b></td> +<td>similar to date-rfc3164, but emulates a common coding error: RFC 3164 demands +that a space is written for single-digit days. With this option, a zero is +written instead. This format seems to be used by syslog-ng and the +date-rfc3164-buggyday option can be used in migration scenarios where otherwise +lots of scripts would need to be adjusted. It is recommended <i>not</i> to use this +option when forwarding to remote hosts - they may treat the date as invalid +(especially when parsing strictly according to RFC 3164).</td> +<br><i>This feature was introduced in rsyslog 4.6.2 and v4 versions above and +5.5.3 and all versions above.</i> +</tr> +<tr> <td><b>date-rfc3339</b></td> <td>format as RFC 3339 date</td> </tr> diff --git a/doc/rsconf1_markmessageperiod.html b/doc/rsconf1_markmessageperiod.html index 2c833339..a6486ba1 100644 --- a/doc/rsconf1_markmessageperiod.html +++ b/doc/rsconf1_markmessageperiod.html @@ -7,7 +7,7 @@ <h2>$MarkMessagePeriod</h2> <p><b>Type:</b> specific to immark input module</p> -<p><b>Default:</b> 1800 (20 minutes)</p> +<p><b>Default:</b> 1200 (20 minutes)</p> <p><b>Description:</b></p> <p>This specifies when mark messages are to be written to output modules. The time specified is in seconds. Specifying 0 is possible and disables mark diff --git a/doc/rsyslog_conf_actions.html b/doc/rsyslog_conf_actions.html index 8c4b9cfc..6020dd88 100644 --- a/doc/rsyslog_conf_actions.html +++ b/doc/rsyslog_conf_actions.html @@ -37,8 +37,15 @@ compared to the otherwise-equal config directives below:</p> *.=crit /var/log/critmsgs</b></code></p> <p> </p> <h3>Regular File</h3> -<p>Typically messages are logged to real files. The file has to -be specified with full pathname, beginning with a slash "/''.<br> +<p>Typically messages are logged to real files. The file usually is +specified by full pathname, beginning with a slash "/". +Starting with version 4.6.2 and 5.4.1 (previous v5 version do NOT support this) +relative file names can also be specified. To do so, these must begin with a +dot. For example, use "./file-in-current-dir.log" to specify a file in the +current directory. Please note that rsyslogd usually changes its working +directory to the root, so relative file names must be tested with care (they +were introduced primarily as a debugging vehicle, but may have useful other applications +as well).<br> <br> <br> You may prefix each entry with the minus "-'' sign to omit syncing the diff --git a/doc/rsyslog_conf_global.html b/doc/rsyslog_conf_global.html index 76dce26d..add9765a 100644 --- a/doc/rsyslog_conf_global.html +++ b/doc/rsyslog_conf_global.html @@ -212,13 +212,21 @@ supported in order to be compliant to the upcoming new syslog RFC series. <li><a href="rsconf1_maxopenfiles.html">$MaxOpenFiles</a></li> <li><a href="rsconf1_moddir.html">$ModDir</a></li> <li><a href="rsconf1_modload.html">$ModLoad</a></li> +<li><b>$OMFileAsyncWriting</b> [on/<b>off</b>], if turned on, the files will be written +in asynchronous mode via a separate thread. In that case, double buffers will be used so +that one buffer can be filled while the other buffer is being written. Note that in order +to enable $OMFileFlushInterval, $OMFileAsyncWriting must be set to "on". Otherwise, the flush +interval will be ignored. Also note that when $OMFileFlushOnTXEnd is "on" but +$OMFileAsyncWriting is off, output will only be written when the buffer is full. This may take +several hours, or even require a rsyslog shutdown. However, a buffer flush can be forced +in that case by sending rsyslogd a HUP signal. <li><b>$OMFileZipLevel</b> 0..9 [default 0] - if greater 0, turns on gzip compression of the output file. The higher the number, the better the compression, but also the more CPU is required for zipping.</li> <li><b>$OMFileIOBufferSize</b> <size_nbr>, default 4k, size of the buffer used to writing output data. The larger the buffer, the potentially better performance is. The default of 4k is quite conservative, it is useful to go up to 64k, and 128K if you used gzip compression (then, even higher sizes may make sense)</li> -<li><b>$OMFileFlushOnTXEnd</b> <[<b>on</b>/off]>, default on. Omfile has the +<li><b>$OMFileFlushOnTXEnd</b> <[on/<b>off</b>]>, default ff. Omfile has the capability to -writes output using a buffered writer. Disk writes are only done when the buffer is +write output using a buffered writer. Disk writes are only done when the buffer is full. So if an error happens during that write, data is potentially lost. In cases where this is unacceptable, set $OMFileFlushOnTXEnd to on. Then, data is written at the end of each transaction (for pre-v5 this means after <b>each</b> log message) and the usual diff --git a/doc/rsyslog_pgsql.html b/doc/rsyslog_pgsql.html new file mode 100644 index 00000000..dcb9dc3a --- /dev/null +++ b/doc/rsyslog_pgsql.html @@ -0,0 +1,336 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> +<HTML> +<HEAD> + <META HTTP-EQUIV="CONTENT-TYPE" CONTENT="text/html; charset=utf-8"> + <TITLE></TITLE> + <META NAME="GENERATOR" CONTENT="OpenOffice.org 3.1 (Unix)"> + <META NAME="AUTHOR" CONTENT="Marc Schiffbauer"> + <META NAME="CREATED" CONTENT="20100129;15054500"> + <META NAME="CHANGEDBY" CONTENT="Marc Schiffbauer"> + <META NAME="CHANGED" CONTENT="20100129;16035000"> + <META NAME="Info 1" CONTENT=""> + <META NAME="Info 2" CONTENT=""> + <META NAME="Info 3" CONTENT=""> + <META NAME="Info 4" CONTENT=""> + <STYLE TYPE="text/css"> + <!-- + @page { size: 8.27in 11.69in; margin: 0.79in } + P { margin-bottom: 0.08in } + P.western { font-family: "Arial", sans-serif } + H1 { margin-bottom: 0.08in } + H1.western { font-family: "Times New Roman", serif } + H1.cjk { font-family: "DejaVu Sans" } + H1.ctl { font-family: "DejaVu Sans" } + H2 { margin-bottom: 0.08in } + H2.western { font-family: "Times New Roman", serif } + BLOCKQUOTE.western { font-family: "Arial", sans-serif } + H3 { margin-bottom: 0.08in } + H3.western { font-family: "Times New Roman", serif } + A:link { so-language: zxx } + --> + </STYLE> +</HEAD> +<BODY LANG="de-DE" DIR="LTR"> +<H1 CLASS="western"><SPAN LANG="en-US">Writing </SPAN>syslog messages +to MySQL, PostgreSQL or any other supported Database</H1> +<P CLASS="western"><FONT SIZE=2><I>Written by </I></FONT><A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php"><FONT SIZE=2><I>Rainer +Gerhards</I></FONT></A><FONT SIZE=2><I> with some additions by Marc +Schiffbauer (2008-02-28)</I></FONT></P> +<H2 CLASS="western">Abstract</H2> +<P CLASS="western"><SPAN LANG="en-US"><I><B>In this paper, I describe +how to write </B></I></SPAN><A HREF="http://www.monitorware.com/en/topics/syslog/">syslog</A><SPAN LANG="en-US"><I><B> +messages to a </B></I></SPAN><A HREF="http://www.mysql.com/">MySQL</A><SPAN LANG="en-US"><I><B> +or </B></I></SPAN><A HREF="http://www.postgresql.org/">PostgreSQL</A><SPAN LANG="en-US"><I><B> +database.</B></I></SPAN><SPAN LANG="en-US"><I> Having syslog messages +in a database is often handy, especially when you intend to set up a +front-end for viewing them. This paper describes an approach with +</I></SPAN><A HREF="http://www.rsyslog.com/">rsyslogd</A><SPAN LANG="en-US"><I>, +an alternative enhanced syslog daemon natively supporting MySQL and +PostgreSQL. I describe the components needed to be installed and how +to configure them. Please note that as of this writing, rsyslog +supports a variety of databases. While this guide is still MySQL- and +PostgreSQL-focused, you can probably use it together with other ones +too. You just need to modify a few settings.</I></SPAN></P> +<H2 CLASS="western">Background</H2> +<P LANG="en-US" CLASS="western">In many cases, syslog data is simply +written to text files. This approach has some advantages, most +notably it is very fast and efficient. However, data stored in text +files is not readily accessible for real-time viewing and analysis. +To do that, the messages need to be in a database. There are various +ways to store syslog messages in a database. For example, some have +the syslogd write text files which are later feed via a separate +script into the database. Others have written scripts taking the data +(via a pipe) from a non-database-aware syslogd and store them as they +appear. Some others use database-aware syslogds and make them write +the data directly to the database. In this paper, I use that "direct +write" approach. I think it is superior, because the syslogd +itself knows the status of the database connection and thus can +handle it intelligently (well ... hopefully ;)). I use rsyslogd to +acomplish this, simply because I have initiated the rsyslog project +with database-awareness as one goal.</P> +<P CLASS="western"><SPAN LANG="en-US"><B>One word of caution:</B></SPAN><SPAN LANG="en-US"> +while message storage in the database provides an excellent +foundation for interactive analysis, it comes at a cost. Database i/o +is considerably slower than text file i/o. As such, directly writing +to the database makes sense only if your message volume is low enough +to allow a) the syslogd, b) the network, and c) the database server +to catch up with it. Some time ago, I have written a paper on +</SPAN><A HREF="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php">optimizing +syslog server performance</A><SPAN LANG="en-US">. While this paper +talks about Window-based solutions, the ideas in it are generic +enough to apply here, too. So it might be worth reading if you +anticipate medium high to high traffic. If you anticipate really high +traffic (or very large traffic spikes), you should seriously consider +forgetting about direct database writes - in my opinion, such a +situation needs either a very specialized system or a different +approach (the text-file-to-database approach might work better for +you in this case). </SPAN> +</P> +<H2 CLASS="western">Overall System Setup</H2> +<P CLASS="western"><SPAN LANG="en-US">In this paper, I concentrate on +the server side. If you are thinking about interactive syslog message +review, you probably want to centralize syslog. In such a scenario, +you have multiple machines (the so-called clients) send their data to +a central machine (called server in this context). While I expect +such a setup to be typical when you are interested in storing +messages in the database, I do not describe how to set it up. This is +beyond the scope of this paper. If you search a little, you will +probably find many good descriptions on </SPAN><SPAN LANG="en-US">how +to centralize syslog. If you do that, it might be a good idea to do +it securely, so you might also be interested in my paper on +</SPAN><A HREF="http://www.rsyslog.com/doc-rsyslog_stunnel.html">ssl-encrypting +syslog message transfer</A><SPAN LANG="en-US">.</SPAN></P> +<P LANG="en-US" CLASS="western">No matter how the messages arrive at +the server, their processing is always the same. So you can use this +paper in combination with any description for centralized syslog +reporting.</P> +<P CLASS="western"><SPAN LANG="en-US">As I already said, I use +rsyslogd on the server. It has intrinsic support for talking to the +supported databases. For obvious reasons, we also need an instance of +MySQL or PostgreSQL running. To keep us focused, the setup of the +database itself is also beyond the scope of this paper. I assume that +you have successfully installed the database and also have a +front-end at hand to work with it (for example, </SPAN><A HREF="http://www.phpmyadmin.net/">phpMyAdmin</A><SPAN LANG="en-US"> +or </SPAN><A HREF="http://phppgadmin.sourceforge.net/">phpPgAdmin</A><SPAN LANG="en-US">. +Please make sure that this is installed, actually working and you +have a basic understanding of how to handle it.</SPAN></P> +<H2 CLASS="western">Setting up the system</H2> +<P CLASS="western"><SPAN LANG="en-US">You need to download and +install rsyslogd first. Obtain it from the </SPAN><A HREF="http://www.rsyslog.com/">rsyslog +site</A><SPAN LANG="en-US">. Make sure that you disable stock +syslogd, otherwise you will experience some difficulties. On some +distributions (Fedora 8 and above, for example), rsyslog may +already by the default syslogd, in which case you obviously do not +need to do anything specific. For many others, there are prebuild +packages available. If you use either, please make sure that you have +the required database plugins for your database available. It usually +is a separate package and typically </SPAN><SPAN LANG="en-US"><B>not</B></SPAN><SPAN LANG="en-US"> +installed by default.</SPAN></P> +<P CLASS="western"><SPAN LANG="en-US">It is important to understand +how rsyslogd talks to the database. In rsyslogd, there is the concept +of "templates". Basically, a template is a string that +includes some replacement characters, which are called "properties" +in rsyslog. Properties are accessed via the "</SPAN><A HREF="http://www.rsyslog.com/doc-property_replacer.html">Property +Replacer</A><SPAN LANG="en-US">". Simply said, you access +properties by including their name between percent signs inside the +template. For example, if the syslog message is "Test", the +template "%msg%" would be expanded to "Test". +Rsyslogd supports sending template text as a SQL statement to the +database. As such, the template must be a valid SQL statement. There +is no limit in what the statement might be, but there are some +obvious and not so obvious choices. For example, a template "drop +table xxx" is possible, but does not make an awful lot of sense. +In practice, you will always use an "insert" statement +inside the template.</SPAN></P> +<P LANG="en-US" CLASS="western">An example: if you would just like to +store the msg part of the full syslog message, you have probably +created a table "syslog" with a single column "message". +In such a case, a good template would be "insert into +syslog(message) values ('%msg%')". With the example above, that +would be expanded to "insert into syslog(message) +values('Test')". This expanded string is then sent to the +database. It's that easy, no special magic. The only thing you must +ensure is that your template expands to a proper SQL statement and +that this statement matches your database design.</P> +<P CLASS="western"><SPAN LANG="en-US">Does that mean you need to +create database schema yourself and also must fully understand +rsyslogd's properties? No, that's not needed. Because we anticipated +that folks are probably more interested in getting things going +instead of designing them from scratch. So we have provided a default +schema as well as build-in support for it. This schema also offers an +additional benefit: rsyslog is part of </SPAN><A HREF="http://www.adiscon.com/en/">Adiscon</A><SPAN LANG="en-US">'s +</SPAN><A HREF="http://www.monitorware.com/en/">MonitorWare product +line</A><SPAN LANG="en-US"> (which includes open source and closed +source members). All of these tools share the same default schema and +know how to operate on it. For this reason, the default schema is +also called the "MonitorWare Schema". If you use it, you +can simply add </SPAN><A HREF="http://www.phplogcon.org/">phpLogCon, +a GPLed syslog web interface</A><SPAN LANG="en-US">, to your system +and have instant interactive access to your database. So there are +some benefits in using the provided schema.</SPAN></P> +<P LANG="en-US" CLASS="western">The schema definition is contained in +the file "createDB.sql". It comes with the rsyslog package +and one can be found for each supported database type (in the plugins +directory). Review it to check that the database name is acceptable +for you. Be sure to leave the table and field names unmodified, +because otherwise you need to customize rsyslogd's default sql +template, which we do not do in this paper. Then, run the script with +your favorite SQL client. Double-check that the table was +successfully created.</P> +<P LANG="en-US" CLASS="western">It is important to note that the +correct database encoding must be used so that the database will +accept strings independend of the string encoding. This is an +important part because it can not be guarantied that all syslog +messages will have a defined character encoding. This is especially +true if the rsyslog-Server will collect messages from different +clients and different products. +</P> +<P LANG="en-US" CLASS="western">For example PostgreSQL may refuse to +accept messages if you would set the database encoding to “UTF8†+while a client is sending invalid byte sequences for that encoding. +</P> +<P LANG="en-US" CLASS="western">Database support in rsyslog is +integrated via loadable plugin modules. To use the database +functionality, the database plugin must be enabled in the config file +BEFORE the first database table action is used. This is done by +placing the</P> +<BLOCKQUOTE CLASS="western"><CODE>$ModLoad ommysql</CODE></BLOCKQUOTE> +<P CLASS="western">directive at the begining of /etc/rsyslog.conf for +MySQL and</P> +<BLOCKQUOTE CLASS="western"><CODE>$ModLoad ompgsql</CODE></BLOCKQUOTE> +<P CLASS="western"><CODE><FONT FACE="Arial, sans-serif">for +PostgreSQL.</FONT></CODE></P> +<P LANG="en-US" CLASS="western"><FONT FACE="Arial, sans-serif">For +other databases, use their plugin name (e.g. omoracle).</FONT></P> +<P CLASS="western">Next, we need to tell rsyslogd to write data to +the database. As we use the default schema, we do NOT need to define +a template for this. We can use the hardcoded one (rsyslogd handles +the proper template linking). So all we need to do e.g. for MySQL is +add a simple selector line to /etc/rsyslog.conf:</P> +<BLOCKQUOTE CLASS="western"><CODE>*.* +:ommysql:database-server,database-name,database-userid,database-password</CODE></BLOCKQUOTE> +<P CLASS="western">Again, other databases have other selector names, +e.g. ":ompgsql:" instead of ":ommysql:". See the +output plugin's documentation for details.</P> +<P LANG="en-US" CLASS="western">In many cases, the database will run +on the local machine. In this case, you can simply use "127.0.0.1" +for <I>database-server</I>. This can be especially advisable, if you +do not need to expose the database to any process outside of the +local machine. In this case, you can simply bind it to 127.0.0.1, +which provides a quite secure setup. Of course, rsyslog also supports +remote database instances. In that case, use the remote server name +(e.g. mydb.example.com) or IP-address. The <I>database-name</I> by +default is "Syslog". If you have modified the default, use +your name here. <I>Database-userid</I> and <I>-password</I> are the +credentials used to connect to the database. As they are stored in +clear text in rsyslog.conf, that user should have only the least +possible privileges. It is sufficient to grant it INSERT privileges +to the systemevents table, only. As a side note, it is strongly +advisable to make the rsyslog.conf file readable by root only - if +you make it world-readable, everybody could obtain the password (and +eventually other vital information from it). In our example, let's +assume you have created a database user named "syslogwriter" +with a password of "topsecret" (just to say it bluntly: +such a password is NOT a good idea...). If your database is on the +local machine, your rsyslog.conf line might look like in this sample:</P> +<BLOCKQUOTE CLASS="western"><CODE>*.* +:ommysql:127.0.0.1,Syslog,syslogwriter,topsecret</CODE></BLOCKQUOTE> +<P CLASS="western">Save rsyslog.conf, restart rsyslogd - and you +should see syslog messages being stored in the "systemevents" +table!</P> +<P LANG="en-US" CLASS="western">The example line stores every message +to the database. Especially if you have a high traffic volume, you +will probably limit the amount of messages being logged. This is easy +to accomplish: the "write database" action is just a +regular selector line. As such, you can apply normal selector-line +filtering. If, for example, you are only interested in messages from +the mail subsystem, you can use the following selector line:</P> +<BLOCKQUOTE CLASS="western"><CODE>mail.* :ommysql:127.0.0.1,syslog,syslogwriter,topsecret</CODE></BLOCKQUOTE> +<P CLASS="western">Review the <A HREF="http://www.rsyslog.com/doc-rsyslog_conf.html">rsyslog.conf</A> +documentation for details on selector lines and their filtering.</P> +<P CLASS="western"><SPAN LANG="en-US"><B>You have now completed +everything necessary to store syslog messages to the a database.</B></SPAN><SPAN LANG="en-US"> +If you would like to try out a front-end, you might want to look at +</SPAN><A HREF="http://www.phplogcon.org/">phpLogCon</A><SPAN LANG="en-US">, +which displays syslog data in a browser. As of this writing, +phpLogCon is not yet a powerful tool, but it's open source, so it +might be a starting point for your own solution.</SPAN></P> +<H2 CLASS="western">On Reliability...</H2> +<P LANG="en-US" CLASS="western">Rsyslogd writes syslog messages +directly to the database. This implies that the database must be +available at the time of message arrival. If the database is offline, +no space is left or something else goes wrong - rsyslogd can not +write the database record. If rsyslogd is unable to store a message, +it performs one retry. This is helpful if the database server was +restarted. In this case, the previous connection was broken but a +reconnect immediately succeeds. However, if the database is down for +an extended period of time, an immediate retry does not help.</P> +<P CLASS="western"><SPAN LANG="en-US">Message loss in this scenario +can easily be prevented with rsyslog. All you need to do is run the +database writer in queued mode. This is now described in a generic +way and I do not intend to duplicate it here. So please be sure to +read "</SPAN><A HREF="http://www.rsyslog.com/doc-rsyslog_high_database_rate.html">Handling +a massive syslog database insert rate with Rsyslog</A><SPAN LANG="en-US">", +which describes the scenario and also includes configuration +examples.</SPAN></P> +<H2 CLASS="western">Conclusion</H2> +<P LANG="en-US" CLASS="western">With minimal effort, you can use +rsyslogd to write syslog messages to a database. You can even make it +absolutely fail-safe and protect it against database server downtime. +Once the messages are arrived there, you can interactively review and +analyze them. In practice, the messages are also stored in text files +for longer-term archival and the databases are cleared out after some +time (to avoid becoming too slow). If you expect an extremely high +syslog message volume, storing it in real-time to the database may +outperform your database server. In such cases, either filter out +some messages or used queued mode (which in general is recommended +with databases).</P> +<P LANG="en-US" CLASS="western">The method outlined in this paper +provides an easy to setup and maintain solution for most use cases.</P> +<H3 CLASS="western">Feedback Requested</H3> +<P CLASS="western">I would appreciate feedback on this paper. If you +have additional ideas, comments or find bugs, please <A HREF="mailto:rgerhards@adiscon.com">let +me know</A>.</P> +<H2 CLASS="western">References and Additional Material</H2> +<UL> + <LI><P CLASS="western" STYLE="margin-bottom: 0in"><A HREF="http://www.rsyslog.com/">www.rsyslog.com</A> + - the rsyslog site + </P> + <LI><P CLASS="western"><A HREF="http://www.monitorware.com/Common/en/Articles/performance-optimizing-syslog-server.php">Paper + on Syslog Server Optimization</A> + </P> +</UL> +<H2 CLASS="western">Revision History</H2> +<UL> + <LI><P CLASS="western" STYLE="margin-bottom: 0in">2005-08-02 * + <A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer + Gerhards</A> * initial version created + </P> + <LI><P CLASS="western" STYLE="margin-bottom: 0in">2005-08-03 * + <A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer + Gerhards</A> * added references to demo site + </P> + <LI><P CLASS="western" STYLE="margin-bottom: 0in">2007-06-13 * + <A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer + Gerhards</A> * removed demo site - was torn down because too + expensive for usage count + </P> + <LI><P CLASS="western" STYLE="margin-bottom: 0in">2008-02-21 * + <A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer + Gerhards</A> * updated reliability section, can now be done with + on-demand disk queues</P> + <LI><P CLASS="western">2008-02-28 * <A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer + Gerhards</A> * added info on other databases, updated syntax to more + recent one + </P> + <LI><P CLASS="western">2010-01-29 * Marc Schiffbauer * added some + PostgreSQL stuff, made wording more database generic, fixed some + typos</P> +</UL> +<H2 CLASS="western">Copyright</H2> +<P CLASS="western">Copyright (c) 2005-2010 <A HREF="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer +Gerhards</A>, Marc Schiffbauer and <A HREF="http://www.adiscon.com/en/">Adiscon</A>.</P> +<P CLASS="western"><BR><BR> +</P> +</BODY> +</HTML>
\ No newline at end of file diff --git a/doc/rsyslog_php_syslog_ng.html b/doc/rsyslog_php_syslog_ng.html index bf48a1eb..ed4d72fc 100644 --- a/doc/rsyslog_php_syslog_ng.html +++ b/doc/rsyslog_php_syslog_ng.html @@ -7,8 +7,10 @@ <P><small><i>Written by <a href="http://www.adiscon.com/en/people/rainer-gerhards.php">Rainer Gerhards</a> (2005-08-04)</i></small></P> -<p><b>Note: it has been reported that this guide is somewhat outdated. Please -use with care. </b></p> +<p>Note: it has been reported that this guide is somewhat outdated. Please +use with care. Also, please note that <b>rsyslog's "native" web frontend is +<a href="http://www.phplogcon.org">phpLogCon</a></b>, which provides best integration +and a lot of extra functionality.</p> <h2>Abstract</h2> <p><i><b>In this paper, I describe how to use <a href="http://www.vermeer.org/projects/php-syslog-ng">php-syslog-ng</a> with @@ -116,11 +118,11 @@ those unfamiliar with syslog-ng, this configuration is probably easier to set up then switching to syslog-ng. For existing rsyslogd users, php-syslog-ng might be a nice add-on to their logging infrastructure.</P> <P>Please note that the <a href="http://www.monitorware.com/en/">MonitorWare family</a> (to which rsyslog belongs) also -offers a web-interface: <a href="http://www.phplogcon.org/">phpLogCon</a>. At the time of this writing, phpLogCon's code -is by far not as clean as I would like it to be. Also the user-interface is -definitely not as intutive as pp-syslog-ng. From a functionality point of view, -however, I think it already is a bit ahead. So you might -consider using it. I have set up a <a href="http://demo.rsyslog.com/">demo server</a>., +offers a web-interface: <a href="http://www.phplogcon.org/">phpLogCon</a>. +From my point of view, obviously, <b>phpLogCon is the more natural choice for a web interface +to be used together with rsyslog</b>. It also offers superb functionality and provides, +for example,native display of Windows event log entries. +I have set up a <a href="http://demo.phplogcon.org/">demo server</a>., You can have a peek at it without installing anything.</P> <h2>Feedback Requested</h2> diff --git a/doc/src/rsyslog_pgsql.odt b/doc/src/rsyslog_pgsql.odt Binary files differnew file mode 100644 index 00000000..5034c5fb --- /dev/null +++ b/doc/src/rsyslog_pgsql.odt diff --git a/plugins/imdoor/Makefile.am b/plugins/imdoor/Makefile.am new file mode 100644 index 00000000..6ad7a904 --- /dev/null +++ b/plugins/imdoor/Makefile.am @@ -0,0 +1,6 @@ +pkglib_LTLIBRARIES = imdoor.la + +imdoor_la_SOURCES = imdoor.c sun_cddl.c +imdoor_la_CPPFLAGS = -I$(top_srcdir) $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) +imdoor_la_LDFLAGS = -module -avoid-version +imdoor_la_LIBADD = diff --git a/plugins/imdoor/imdoor.c b/plugins/imdoor/imdoor.c new file mode 100644 index 00000000..83890d03 --- /dev/null +++ b/plugins/imdoor/imdoor.c @@ -0,0 +1,430 @@ +/* imdoor.c + * This input module is used to receive syslog messages via the Solaris + * door mechanism. Not surprisingly, it most probably can not be built + * on other platforms. + * + * NOTE: read comments in module-template.h to understand how this file + * works! + * + * File begun on 2010-03-26 by RGerhards + * + * Copyright 2010 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdlib.h> +#include <stdio.h> +#include <assert.h> +#include <string.h> +#include <errno.h> +#include <unistd.h> +#include <sys/stat.h> +#include <sys/un.h> +#include "dirty.h" +#include "cfsysline.h" +#include "unicode-helper.h" +#include "module-template.h" +#include "srUtils.h" +#include "errmsg.h" +#include "net.h" +#include "glbl.h" +#include "msg.h" +#include "prop.h" + +MODULE_TYPE_INPUT + +/* defines */ +#define MAXFUNIX 20 +#ifndef _PATH_LOG +#ifdef BSD +#define _PATH_LOG "/var/run/log" +#else +#define _PATH_LOG "/dev/log" +#endif +#endif + + +/* handle some defines missing on more than one platform */ +#ifndef SUN_LEN +#define SUN_LEN(su) \ + (sizeof(*(su)) - sizeof((su)->sun_path) + strlen((su)->sun_path)) +#endif +/* Module static data */ +DEF_IMOD_STATIC_DATA +DEFobjCurrIf(errmsg) +DEFobjCurrIf(glbl) +DEFobjCurrIf(prop) + +static prop_t *pInputName = NULL; /* our inputName currently is always "imuxsock", and this will hold it */ +static int startIndexUxLocalSockets; /* process funix from that index on (used to + * suppress local logging. rgerhards 2005-08-01 + * read-only after startup + */ +static int funixParseHost[MAXFUNIX] = { 0, }; /* should parser parse host name? read-only after startup */ +static int funixFlags[MAXFUNIX] = { IGNDATE, }; /* should parser parse host name? read-only after startup */ +static uchar *funixn[MAXFUNIX] = { (uchar*) _PATH_LOG }; /* read-only after startup */ +static uchar *funixHName[MAXFUNIX] = { NULL, }; /* host-name override - if set, use this instead of actual name */ +static int funixFlowCtl[MAXFUNIX] = { eFLOWCTL_NO_DELAY, }; /* flow control settings for this socket */ +static int funix[MAXFUNIX] = { -1, }; /* read-only after startup */ +static int nfunix = 1; /* number of Unix sockets open / read-only after startup */ + +/* config settings */ +static int bOmitLocalLogging = 0; +static uchar *pLogSockName = NULL; +static uchar *pLogHostName = NULL; /* host name to use with this socket */ +static int bUseFlowCtl = 0; /* use flow control or not (if yes, only LIGHT is used! */ +static int bIgnoreTimestamp = 1; /* ignore timestamps present in the incoming message? */ + + +/* set the timestamp ignore / not ignore option for the system + * log socket. This must be done separtely, as it is not added via a command + * but present by default. -- rgerhards, 2008-03-06 + */ +static rsRetVal setSystemLogTimestampIgnore(void __attribute__((unused)) *pVal, int iNewVal) +{ + DEFiRet; + funixFlags[0] = iNewVal ? IGNDATE : NOFLAG; + RETiRet; +} + +/* set flowcontrol for the system log socket + */ +static rsRetVal setSystemLogFlowControl(void __attribute__((unused)) *pVal, int iNewVal) +{ + DEFiRet; + funixFlowCtl[0] = iNewVal ? eFLOWCTL_LIGHT_DELAY : eFLOWCTL_NO_DELAY; + RETiRet; +} + +/* add an additional listen socket. Socket names are added + * until the array is filled up. It is never reset, only at + * module unload. + * TODO: we should change the array to a list so that we + * can support any number of listen socket names. + * rgerhards, 2007-12-20 + * added capability to specify hostname for socket -- rgerhards, 2008-08-01 + */ +static rsRetVal addLstnSocketName(void __attribute__((unused)) *pVal, uchar *pNewVal) +{ + if(nfunix < MAXFUNIX) { + if(*pNewVal == ':') { + funixParseHost[nfunix] = 1; + } + else { + funixParseHost[nfunix] = 0; + } + funixHName[nfunix] = pLogHostName; + pLogHostName = NULL; /* re-init for next, not freed because funixHName[] now owns it */ + funixFlowCtl[nfunix] = bUseFlowCtl ? eFLOWCTL_LIGHT_DELAY : eFLOWCTL_NO_DELAY; + funixFlags[nfunix] = bIgnoreTimestamp ? IGNDATE : NOFLAG; + funixn[nfunix++] = pNewVal; + } + else { + errmsg.LogError(0, NO_ERRCODE, "Out of unix socket name descriptors, ignoring %s\n", + pNewVal); + } + + return RS_RET_OK; +} + +/* free the funixn[] socket names - needed as cleanup on several places + * note that nfunix is NOT reset! funixn[0] is never freed, as it comes from + * the constant memory pool - and if not, it is freeed via some other pointer. + */ +static rsRetVal discardFunixn(void) +{ + int i; + + for (i = 1; i < nfunix; i++) { + if(funixn[i] != NULL) { + free(funixn[i]); + funixn[i] = NULL; + } + if(funixHName[i] != NULL) { + free(funixHName[i]); + funixHName[i] = NULL; + } + } + + return RS_RET_OK; +} + + +static int create_unix_socket(const char *path) +{ + struct sockaddr_un sunx; + int fd; + + if (path[0] == '\0') + return -1; + + unlink(path); + + memset(&sunx, 0, sizeof(sunx)); + sunx.sun_family = AF_UNIX; + (void) strncpy(sunx.sun_path, path, sizeof(sunx.sun_path)); + fd = socket(AF_UNIX, SOCK_DGRAM, 0); + if (fd < 0 || bind(fd, (struct sockaddr *) &sunx, SUN_LEN(&sunx)) < 0 || + chmod(path, 0666) < 0) { + errmsg.LogError(errno, NO_ERRCODE, "connot create '%s'", path); + dbgprintf("cannot create %s (%d).\n", path, errno); + close(fd); + return -1; + } + return fd; +} + + +/* This function receives data from a socket indicated to be ready + * to receive and submits the message received for processing. + * rgerhards, 2007-12-20 + * Interface changed so that this function is passed the array index + * of the socket which is to be processed. This eases access to the + * growing number of properties. -- rgerhards, 2008-08-01 + */ +static rsRetVal readSocket(int fd, int iSock) +{ + DEFiRet; + int iRcvd; + int iMaxLine; + uchar bufRcv[4096+1]; + uchar *pRcv = NULL; /* receive buffer */ + + assert(iSock >= 0); + + iMaxLine = glbl.GetMaxLine(); + + /* we optimize performance: if iMaxLine is below 4K (which it is in almost all + * cases, we use a fixed buffer on the stack. Only if it is higher, heap memory + * is used. We could use alloca() to achive a similar aspect, but there are so + * many issues with alloca() that I do not want to take that route. + * rgerhards, 2008-09-02 + */ + if((size_t) iMaxLine < sizeof(bufRcv) - 1) { + pRcv = bufRcv; + } else { + CHKmalloc(pRcv = (uchar*) malloc(sizeof(uchar) * (iMaxLine + 1))); + } + + iRcvd = recv(fd, pRcv, iMaxLine, 0); + dbgprintf("Message from UNIX socket: #%d\n", fd); + if (iRcvd > 0) { + parseAndSubmitMessage(funixHName[iSock] == NULL ? glbl.GetLocalHostName() : funixHName[iSock], + (uchar*)"127.0.0.1", pRcv, + iRcvd, funixParseHost[iSock] ? (funixFlags[iSock] | PARSE_HOSTNAME) : funixFlags[iSock], + funixFlowCtl[iSock], pInputName, NULL, 0); + } else if (iRcvd < 0 && errno != EINTR) { + char errStr[1024]; + rs_strerror_r(errno, errStr, sizeof(errStr)); + dbgprintf("UNIX socket error: %d = %s.\n", errno, errStr); + errmsg.LogError(errno, NO_ERRCODE, "recvfrom UNIX"); + } + +finalize_it: + if(pRcv != NULL && (size_t) iMaxLine >= sizeof(bufRcv) - 1) + free(pRcv); + + RETiRet; +} + + +/* This function is called to gather input. */ +BEGINrunInput + int maxfds; + int nfds; + int i; + int fd; + fd_set readfds; +CODESTARTrunInput + /* this is an endless loop - it is terminated when the thread is + * signalled to do so. This, however, is handled by the framework, + * right into the sleep below. + */ + while(1) { + /* Add the Unix Domain Sockets to the list of read + * descriptors. + * rgerhards 2005-08-01: we must now check if there are + * any local sockets to listen to at all. If the -o option + * is given without -a, we do not need to listen at all.. + */ + maxfds = 0; + FD_ZERO (&readfds); + /* Copy master connections */ + for (i = startIndexUxLocalSockets; i < nfunix; i++) { + if (funix[i] != -1) { + FD_SET(funix[i], &readfds); + if (funix[i]>maxfds) maxfds=funix[i]; + } + } + + if(Debug) { + dbgprintf("--------imuxsock calling select, active file descriptors (max %d): ", maxfds); + for (nfds= 0; nfds <= maxfds; ++nfds) + if ( FD_ISSET(nfds, &readfds) ) + dbgprintf("%d ", nfds); + dbgprintf("\n"); + } + + /* wait for io to become ready */ + nfds = select(maxfds+1, (fd_set *) &readfds, NULL, NULL, NULL); + + for (i = 0; i < nfunix && nfds > 0; i++) { + if ((fd = funix[i]) != -1 && FD_ISSET(fd, &readfds)) { + readSocket(fd, i); + --nfds; /* indicate we have processed one */ + } + } + } + + RETiRet; +ENDrunInput + + +BEGINwillRun +CODESTARTwillRun + register int i; + + /* first apply some config settings */ + startIndexUxLocalSockets = bOmitLocalLogging ? 1 : 0; + if(pLogSockName != NULL) + funixn[0] = pLogSockName; + + /* initialize and return if will run or not */ + for (i = startIndexUxLocalSockets ; i < nfunix ; i++) { + if ((funix[i] = create_unix_socket((char*) funixn[i])) != -1) + dbgprintf("Opened UNIX socket '%s' (fd %d).\n", funixn[i], funix[i]); + } + + /* we need to create the inputName property (only once during our lifetime) */ + CHKiRet(prop.Construct(&pInputName)); + CHKiRet(prop.SetString(pInputName, UCHAR_CONSTANT("imuxsock"), sizeof("imuxsock") - 1)); + CHKiRet(prop.ConstructFinalize(pInputName)); + +finalize_it: +ENDwillRun + + +BEGINafterRun +CODESTARTafterRun + int i; + /* do cleanup here */ + /* Close the UNIX sockets. */ + for (i = 0; i < nfunix; i++) + if (funix[i] != -1) + close(funix[i]); + + /* Clean-up files. */ + for (i = 0; i < nfunix; i++) + if (funixn[i] && funix[i] != -1) + unlink((char*) funixn[i]); + /* free no longer needed string */ + if(pLogSockName != NULL) + free(pLogSockName); + if(pLogHostName != NULL) { + free(pLogHostName); + } + + discardFunixn(); + nfunix = 1; + + if(pInputName != NULL) + prop.Destruct(&pInputName); +ENDafterRun + + +BEGINmodExit +CODESTARTmodExit + objRelease(glbl, CORE_COMPONENT); + objRelease(errmsg, CORE_COMPONENT); + objRelease(prop, CORE_COMPONENT); +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_IMOD_QUERIES +ENDqueryEtryPt + +static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __attribute__((unused)) *pVal) +{ + bOmitLocalLogging = 0; + if(pLogSockName != NULL) { + free(pLogSockName); + pLogSockName = NULL; + } + if(pLogHostName != NULL) { + free(pLogHostName); + pLogHostName = NULL; + } + + discardFunixn(); + nfunix = 1; + bIgnoreTimestamp = 1; + bUseFlowCtl = 0; + + return RS_RET_OK; +} + + +BEGINmodInit() + int i; +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(glbl, CORE_COMPONENT)); + CHKiRet(objUse(prop, CORE_COMPONENT)); + + dbgprintf("imuxsock version %s initializing\n", PACKAGE_VERSION); + + /* initialize funixn[] array */ + for(i = 1 ; i < MAXFUNIX ; ++i) { + funixn[i] = NULL; + funix[i] = -1; + } + + /* register config file handlers */ + CHKiRet(omsdRegCFSLineHdlr((uchar *)"omitlocallogging", 0, eCmdHdlrBinary, + NULL, &bOmitLocalLogging, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputunixlistensocketignoremsgtimestamp", 0, eCmdHdlrBinary, + NULL, &bIgnoreTimestamp, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"systemlogsocketname", 0, eCmdHdlrGetWord, + NULL, &pLogSockName, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputunixlistensockethostname", 0, eCmdHdlrGetWord, + NULL, &pLogHostName, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"inputunixlistensocketflowcontrol", 0, eCmdHdlrBinary, + NULL, &bUseFlowCtl, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"addunixlistensocket", 0, eCmdHdlrGetWord, + addLstnSocketName, NULL, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, + resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID)); + /* the following one is a (dirty) trick: the system log socket is not added via + * an "addUnixListenSocket" config format. As such, it's properties can not be modified + * via $InputUnixListenSocket*". So we need to add a special directive + * for that. We should revisit all of that once we have the new config format... + * rgerhards, 2008-03-06 + */ + CHKiRet(omsdRegCFSLineHdlr((uchar *)"systemlogsocketignoremsgtimestamp", 0, eCmdHdlrBinary, + setSystemLogTimestampIgnore, NULL, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"systemlogsocketflowcontrol", 0, eCmdHdlrBinary, + setSystemLogFlowControl, NULL, STD_LOADABLE_MODULE_ID)); +ENDmodInit +/* vim:set ai: + */ diff --git a/plugins/imdoor/sun_cddl.c b/plugins/imdoor/sun_cddl.c new file mode 100644 index 00000000..8e9714d9 --- /dev/null +++ b/plugins/imdoor/sun_cddl.c @@ -0,0 +1,592 @@ +#define MAXLINE 4096 +/* + * CDDL HEADER START + * + * The contents of this file are subject to the terms of the + * Common Development and Distribution License (the "License"). + * You may not use this file except in compliance with the License. + * + * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE + * or http://www.opensolaris.org/os/licensing. + * See the License for the specific language governing permissions + * and limitations under the License. + * + * When distributing Covered Code, include this CDDL HEADER in each + * file and include the License file at usr/src/OPENSOLARIS.LICENSE. + * If applicable, add the following below this CDDL HEADER, with the + * fields enclosed by brackets "[]" replaced with your own identifying + * information: Portions Copyright [yyyy] [name of copyright owner] + * + * CDDL HEADER END + */ +/* Portions Copyright 2010 by Rainer Gerhards and Adiscon + */ +/* + * Copyright 2008 Sun Microsystems, Inc. All rights reserved. + * Use is subject to license terms. + */ + +/* + * Copyright (c) 1984, 1986, 1987, 1988, 1989 AT&T + * All Rights Reserved + */ + +/* + * University Copyright- Copyright (c) 1982, 1986, 1988 + * The Regents of the University of California + * All Rights Reserved + * + * University Acknowledgment- Portions of this document are derived from + * software developed by the University of California, Berkeley, and its + * contributors. + */ +#include <unistd.h> +#include <note.h> +#include <errno.h> +#include <sys/types.h> +#include <stdio.h> +#include <stdio_ext.h> +#include <stdlib.h> +#include <ctype.h> +#include <signal.h> +#include <string.h> +#include <strings.h> +#include <libscf.h> +#include <netconfig.h> +#include <netdir.h> +#include <pwd.h> +#include <sys/socket.h> +#include <tiuser.h> +#include <utmpx.h> +#include <limits.h> +#include <pthread.h> +#include <fcntl.h> +#include <stropts.h> +#include <assert.h> +#include <sys/statvfs.h> + +#include <sys/param.h> +#include <sys/sysmacros.h> +#include <sys/syslog.h> +#include <sys/strlog.h> +#include <sys/stat.h> +#include <sys/time.h> +#include <sys/utsname.h> +#include <sys/poll.h> +#include <sys/wait.h> +#include <sys/resource.h> +#include <sys/mman.h> +#include <sys/note.h> +#include <door.h> +#include <sys/door.h> + +#include "rsyslog.h" +#include "debug.h" + +#define DOORFILE "/var/run/syslog_door" +#define RELATIVE_DOORFILE "../var/run/syslog_door" +#define OLD_DOORFILE "/etc/.syslog_door" + +static int DoorFd = -1; +static int DoorCreated = 0; +static char *DoorFileName = DOORFILE; + +/* for managing door server threads */ +static pthread_mutex_t door_server_cnt_lock = PTHREAD_MUTEX_INITIALIZER; +static uint_t door_server_cnt = 0; +static pthread_attr_t door_thr_attr; + +/* + * the 'server' function that we export via the door. It does + * nothing but return. + */ +/*ARGSUSED*/ +static void +server(void *cookie, char *argp, size_t arg_size, + door_desc_t *dp, uint_t n) +{ + (void) door_return(NULL, 0, NULL, 0); + /* NOTREACHED */ +} + +/*ARGSUSED*/ +static void * +create_door_thr(void *arg) +{ + (void) pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, NULL); + (void) door_return(NULL, 0, NULL, 0); + + /* + * If there is an error in door_return(), it will return here and + * the thread will exit. Hence we need to decrement door_server_cnt. + */ + (void) pthread_mutex_lock(&door_server_cnt_lock); + door_server_cnt--; + (void) pthread_mutex_unlock(&door_server_cnt_lock); + return (NULL); +} + +/* + * Max number of door server threads for syslogd. Since door is used + * to check the health of syslogd, we don't need large number of + * server threads. + */ +#define MAX_DOOR_SERVER_THR 3 + +/* + * Manage door server thread pool. + */ +/*ARGSUSED*/ +static void +door_server_pool(door_info_t *dip) +{ + (void) pthread_mutex_lock(&door_server_cnt_lock); + if (door_server_cnt <= MAX_DOOR_SERVER_THR && + pthread_create(NULL, &door_thr_attr, create_door_thr, NULL) == 0) { + door_server_cnt++; + (void) pthread_mutex_unlock(&door_server_cnt_lock); + return; + } + + (void) pthread_mutex_unlock(&door_server_cnt_lock); +} + +static void +delete_doorfiles(void) +{ + pthread_t mythreadno; + struct stat sb; + int err; + char line[MAXLINE+1]; + + if (Debug) { + mythreadno = pthread_self(); + } + + + if (lstat(DoorFileName, &sb) == 0 && !S_ISDIR(sb.st_mode)) { + if (unlink(DoorFileName) < 0) { + err = errno; + (void) snprintf(line, sizeof (line), + "unlink() of %s failed - fatal", DoorFileName); + errno = err; + DBGPRINTF("%s", line);//logerror(line); + DBGPRINTF("delete_doorfiles(%u): error: %s, " + "errno=%d\n", mythreadno, line, err); + exit(1); + } + + DBGPRINTF("delete_doorfiles(%u): deleted %s\n", + mythreadno, DoorFileName); + } + + if (strcmp(DoorFileName, DOORFILE) == 0) { + if (lstat(OLD_DOORFILE, &sb) == 0 && !S_ISDIR(sb.st_mode)) { + if (unlink(OLD_DOORFILE) < 0) { + err = errno; + (void) snprintf(line, sizeof (line), + "unlink() of %s failed", OLD_DOORFILE); + DBGPRINTF("delete_doorfiles(%u): %s\n", + mythreadno, line); + + if (err != EROFS) { + errno = err; + (void) strlcat(line, " - fatal", + sizeof (line)); + logerror(line); + DBGPRINTF("delete_doorfiles(%u): " + "error: %s, errno=%d\n", + mythreadno, line, err); + exit(1); + } + + DBGPRINTF("delete_doorfiles(%u): unlink() " + "failure OK on RO file system\n", + mythreadno); + } + + DBGPRINTF("delete_doorfiles(%u): deleted %s\n", + mythreadno, OLD_DOORFILE); + } + } + +#if 0 + if (lstat(PidFileName, &sb) == 0 && !S_ISDIR(sb.st_mode)) { + if (unlink(PidFileName) < 0) { + err = errno; + (void) snprintf(line, sizeof (line), + "unlink() of %s failed - fatal", PidFileName); + errno = err; + logerror(line); + DBGPRINTF("delete_doorfiles(%u): error: %s, " + "errno=%d\n", mythreadno, line, err); + exit(1); + } + + DBGPRINTF("delete_doorfiles(%u): deleted %s\n", mythreadno, + PidFileName); + } + + if (strcmp(PidFileName, PIDFILE) == 0) { + if (lstat(OLD_PIDFILE, &sb) == 0 && !S_ISDIR(sb.st_mode)) { + if (unlink(OLD_PIDFILE) < 0) { + err = errno; + (void) snprintf(line, sizeof (line), + "unlink() of %s failed", OLD_PIDFILE); + DBGPRINTF(5, "delete_doorfiles(%u): %s, \n", + mythreadno, line); + + if (err != EROFS) { + errno = err; + (void) strlcat(line, " - fatal", + sizeof (line)); + logerror(line); + DBGPRINTF(1, "delete_doorfiles(%u): " + "error: %s, errno=%d\n", + mythreadno, line, err); + exit(1); + } + + DBGPRINTF(5, "delete_doorfiles(%u): unlink " + "failure OK on RO file system\n", + mythreadno); + } + + DBGPRINTF(5, "delete_doorfiles(%u): deleted %s\n", + mythreadno, OLD_PIDFILE); + } + } +#endif + + if (DoorFd != -1) { + (void) door_revoke(DoorFd); + } + + DBGPRINTF("delete_doorfiles(%u): revoked door: DoorFd=%d\n", + mythreadno, DoorFd); +} + + +/* + * Create the door file and the pid file in /var/run. If the filesystem + * containing /etc is writable, create symlinks /etc/.syslog_door and + * /etc/syslog.pid to them. On systems that do not support /var/run, create + * /etc/.syslog_door and /etc/syslog.pid directly. + * + * Note: it is not considered fatal to fail to create the pid file or its + * symlink. Attempts to use them in the usual way will fail, of course, but + * syslogd will function nicely without it (not so for the door file). + */ + +static void +sun_open_door(void) +{ + struct stat buf; + door_info_t info; + char line[MAXLINE+1]; + pthread_t mythreadno; + int err; + + if (Debug) { + mythreadno = pthread_self(); + } + + /* + * first see if another syslogd is running by trying + * a door call - if it succeeds, there is already + * a syslogd process active + */ + + if (!DoorCreated) { + int door; + + if ((door = open(DoorFileName, O_RDONLY)) >= 0) { + DBGPRINTF("open_door(%u): %s opened " + "successfully\n", mythreadno, DoorFileName); + + if (door_info(door, &info) >= 0) { + DBGPRINTF("open_door(%u): " + "door_info:info.di_target = %ld\n", + mythreadno, info.di_target); + + if (info.di_target > 0) { + (void) sprintf(line, "syslogd pid %ld" + " already running. Cannot " + "start another syslogd pid %ld", + info.di_target, getpid()); + DBGPRINTF("open_door(%u): error: " + "%s\n", mythreadno, line); + errno = 0; + //logerror(line); + exit(1); + } + } + + (void) close(door); + } else { + if (lstat(DoorFileName, &buf) < 0) { + err = errno; + + DBGPRINTF("open_door(%u): lstat() of %s " + "failed, errno=%d\n", + mythreadno, DoorFileName, err); + + if ((door = creat(DoorFileName, 0644)) < 0) { + err = errno; + (void) snprintf(line, sizeof (line), + "creat() of %s failed - fatal", + DoorFileName); + DBGPRINTF("open_door(%u): error: %s, " + "errno=%d\n", mythreadno, line, + err); + errno = err; + //logerror(line); + delete_doorfiles(); + exit(1); + } + + (void) fchmod(door, + S_IRUSR|S_IWUSR|S_IRGRP|S_IROTH); + + DBGPRINTF("open_door(%u): creat() of %s " + "succeeded\n", mythreadno, + DoorFileName); + + (void) close(door); + } + } + + if (strcmp(DoorFileName, DOORFILE) == 0) { + if (lstat(OLD_DOORFILE, &buf) == 0) { + DBGPRINTF("open_door(%u): lstat() of %s " + "succeeded\n", mythreadno, + OLD_DOORFILE); + + if (S_ISDIR(buf.st_mode)) { + (void) snprintf(line, sizeof (line), + "%s is a directory - fatal", + OLD_DOORFILE); + DBGPRINTF("open_door(%u): error: " + "%s\n", mythreadno, line); + errno = 0; + //logerror(line); + delete_doorfiles(); + exit(1); + } + + DBGPRINTF("open_door(%u): %s is not a " + "directory\n", + mythreadno, OLD_DOORFILE); + + if (unlink(OLD_DOORFILE) < 0) { + err = errno; + (void) snprintf(line, sizeof (line), + "unlink() of %s failed", + OLD_DOORFILE); + DBGPRINTF("open_door(%u): %s\n", + mythreadno, line); + + if (err != EROFS) { + DBGPRINTF("open_door(%u): " + "error: %s, " + "errno=%d\n", + mythreadno, line, err); + (void) strcat(line, " - fatal"); + errno = err; + //logerror(line); + delete_doorfiles(); + exit(1); + } + + DBGPRINTF("open_door(%u): unlink " + "failure OK on RO file " + "system\n", mythreadno); + } + } else { + DBGPRINTF("open_door(%u): file %s doesn't " + "exist\n", mythreadno, OLD_DOORFILE); + } + + if (symlink(RELATIVE_DOORFILE, OLD_DOORFILE) < 0) { + err = errno; + (void) snprintf(line, sizeof (line), + "symlink %s -> %s failed", OLD_DOORFILE, + RELATIVE_DOORFILE); + DBGPRINTF("open_door(%u): %s\n", mythreadno, + line); + + if (err != EROFS) { + DBGPRINTF("open_door(%u): error: %s, " + "errno=%d\n", mythreadno, line, + err); + errno = err; + (void) strcat(line, " - fatal"); + //logerror(line); + delete_doorfiles(); + exit(1); + } + + DBGPRINTF("open_door(%u): symlink failure OK " + "on RO file system\n", mythreadno); + } else { + DBGPRINTF("open_door(%u): symlink %s -> %s " + "succeeded\n", mythreadno, + OLD_DOORFILE, RELATIVE_DOORFILE); + } + } + + if ((DoorFd = door_create(server, 0, + DOOR_REFUSE_DESC)) < 0) { + //???? DOOR_NO_CANEL requires newer libs??? DOOR_REFUSE_DESC | DOOR_NO_CANCEL)) < 0) { + err = errno; + (void) sprintf(line, "door_create() failed - fatal"); + DBGPRINTF("open_door(%u): error: %s, errno=%d\n", + mythreadno, line, err); + errno = err; + //logerror(line); + delete_doorfiles(); + exit(1); + } + //???? (void) door_setparam(DoorFd, DOOR_PARAM_DATA_MAX, 0); + DBGPRINTF("open_door(%u): door_create() succeeded, " + "DoorFd=%d\n", mythreadno, DoorFd); + + DoorCreated = 1; + } + + (void) fdetach(DoorFileName); /* just in case... */ + + (void) door_server_create(door_server_pool); + + if (fattach(DoorFd, DoorFileName) < 0) { + err = errno; + (void) snprintf(line, sizeof (line), "fattach() of fd" + " %d to %s failed - fatal", DoorFd, DoorFileName); + DBGPRINTF("open_door(%u): error: %s, errno=%d\n", mythreadno, + line, err); + errno = err; + //logerror(line); + delete_doorfiles(); + exit(1); + } + + DBGPRINTF("open_door(%u): attached server() to %s\n", mythreadno, + DoorFileName); + +#if 0 + /* + * create pidfile anyway, so those using it to control + * syslogd (with kill `cat /etc/syslog.pid` perhaps) + * don't get broken. + */ + + if (!PidfileCreated) { + int pidfd; + + PidfileCreated = 1; + + if ((pidfd = open(PidFileName, O_RDWR|O_CREAT|O_TRUNC, 0644)) + < 0) { + err = errno; + (void) snprintf(line, sizeof (line), + "open() of %s failed", PidFileName); + DBGPRINTF(1, "open_door(%u): warning: %s, errno=%d\n", + mythreadno, line, err); + errno = err; + //logerror(line); + return; + } + + (void) fchmod(pidfd, S_IRUSR|S_IWUSR|S_IRGRP|S_IROTH); + (void) sprintf(line, "%ld\n", getpid()); + + if (write(pidfd, line, strlen(line)) < 0) { + err = errno; + (void) snprintf(line, sizeof (line), + "write to %s on fd %d failed", PidFileName, pidfd); + DBGPRINTF(1, "open_door(%u): warning: %s, errno=%d\n", + mythreadno, line, err); + errno = err; + //logerror(line); + return; + } + + (void) close(pidfd); + + DBGPRINTF("open_door(%u): %s created\n", + mythreadno, PidFileName); + + if (strcmp(PidFileName, PIDFILE) == 0) { + if (lstat(OLD_PIDFILE, &buf) == 0) { + DBGPRINTF("open_door(%u): lstat() of %s " + "succeded\n", mythreadno, OLD_PIDFILE); + + if (S_ISDIR(buf.st_mode)) { + (void) snprintf(line, sizeof (line), + "file %s is a directory", + OLD_PIDFILE); + DBGPRINTF("open_door(%u): warning: " + "%s\n", mythreadno, line); + errno = 0; + //logerror(line); + return; + } + + if (unlink(OLD_PIDFILE) < 0) { + err = errno; + (void) snprintf(line, sizeof (line), + "unlink() of %s failed", + OLD_PIDFILE); + DBGPRINTF("open_door(%u): %s\n", + mythreadno, line); + + if (err != EROFS) { + DBGPRINTF(1, "open_door (%u): " + "warning: %s, " + "errno=%d\n", + mythreadno, line, err); + errno = err; + //logerror(line); + return; + } + + DBGPRINTF(5, "open_door(%u): unlink " + "failure OK on RO file " + "system\n", mythreadno); + } + } else { + DBGPRINTF("open_door(%u): file %s doesn't " + "exist\n", mythreadno, OLD_PIDFILE); + } + + if (symlink(RELATIVE_PIDFILE, OLD_PIDFILE) < 0) { + err = errno; + (void) snprintf(line, sizeof (line), + "symlink %s -> %s failed", OLD_PIDFILE, + RELATIVE_PIDFILE); + DBGPRINTF("open_door(%u): %s\n", mythreadno, + line); + + if (err != EROFS) { + DBGPRINTF(1, "open_door(%u): warning: " + "%s, errno=%d\n", mythreadno, + line, err); + errno = err; + //logerror(line); + return; + } + + DBGPRINTF(5, "open_door(%u): symlink failure OK " + "on RO file system\n", mythreadno); + return; + } + + DBGPRINTF(5, "open_door(%u): symlink %s -> %s " + "succeeded\n", mythreadno, OLD_PIDFILE, + RELATIVE_PIDFILE); + } + } +#endif +} + + diff --git a/plugins/imklog/Makefile.am b/plugins/imklog/Makefile.am index 5d4d0465..06d4013c 100644 --- a/plugins/imklog/Makefile.am +++ b/plugins/imklog/Makefile.am @@ -11,6 +11,10 @@ if ENABLE_IMKLOG_LINUX imklog_la_SOURCES += linux.c module.h ksym.c ksyms.h ksym_mod.c endif +if ENABLE_IMKLOG_SOLARIS +imklog_la_SOURCES += solaris.c solaris_cddl.c +endif + imklog_la_CPPFLAGS = -I$(top_srcdir) $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) imklog_la_LDFLAGS = -module -avoid-version imklog_la_LIBADD = diff --git a/plugins/imklog/solaris.c b/plugins/imklog/solaris.c new file mode 100644 index 00000000..c2aec30a --- /dev/null +++ b/plugins/imklog/solaris.c @@ -0,0 +1,181 @@ +/* klog driver for solaris + * + * This contains OS-specific functionality to read the + * kernel log. For a general overview, see head comment in + * imklog.c. + * + * This file relies on Sun code in solaris_cddl.c. We have split + * it from Sun's code to keep the copyright issue as simple as possible. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * If that may be required, an exception is granted to permit linking + * this code to the code in solaris_cddl.c that is under the cddl license. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ + +#ifdef HAVE_CONFIG_H +# include "config.h" +#endif +#include <stdlib.h> +#include <unistd.h> +#include <fcntl.h> +#include <errno.h> +#include <string.h> +#include <sys/socket.h> + + + +#include "rsyslog.h" +#include "imklog.h" +#include "unicode-helper.h" +#include "solaris_cddl.h" + +/* globals */ +static int fklog; // TODO: remove +#ifndef _PATH_KLOG +# define _PATH_KLOG "/dev/log" +#endif + + +static uchar *GetPath(void) +{ + return pszPath ? pszPath : UCHAR_CONSTANT(_PATH_KLOG); +} + +/* open the kernel log - will be called inside the willRun() imklog + * entry point. -- rgerhards, 2008-04-09 + */ +rsRetVal +klogWillRun(void) +{ + DEFiRet; + + fklog = sun_openklog((char*) GetPath(), O_RDONLY); + if (fklog < 0) { + char errStr[1024]; + int err = errno; + rs_strerror_r(err, errStr, sizeof(errStr)); + DBGPRINTF("error %d opening log socket: %s\n", + GetPath(), errStr); + iRet = RS_RET_ERR; // TODO: better error code + } + + RETiRet; +} + + +/* Read /dev/klog while data are available, split into lines. + * Contrary to standard BSD syslogd, we do a blocking read. We can + * afford this as imklog is running on its own threads. So if we have + * a single file, it really doesn't matter if we wait inside a 1-file + * select or the read() directly. + */ +static void +readklog(void) +{ + char *p, *q; + int len, i; + int iMaxLine; + uchar bufRcv[4096+1]; + uchar *pRcv = NULL; /* receive buffer */ + + iMaxLine = klog_getMaxLine(); + + /* we optimize performance: if iMaxLine is below 4K (which it is in almost all + * cases, we use a fixed buffer on the stack. Only if it is higher, heap memory + * is used. We could use alloca() to achive a similar aspect, but there are so + * many issues with alloca() that I do not want to take that route. + * rgerhards, 2008-09-02 + */ + if((size_t) iMaxLine < sizeof(bufRcv) - 1) { + pRcv = bufRcv; + } else { + if((pRcv = (uchar*) malloc(sizeof(uchar) * (iMaxLine + 1))) == NULL) + iMaxLine = sizeof(bufRcv) - 1; /* better this than noting */ + } + + len = 0; + for (;;) { + dbgprintf("----------imklog(BSD) waiting for kernel log line\n"); + i = read(fklog, pRcv + len, iMaxLine - len); + if (i > 0) { + pRcv[i + len] = '\0'; + } else { + if (i < 0 && errno != EINTR && errno != EAGAIN) { + imklogLogIntMsg(LOG_ERR, + "imklog error %d reading kernel log - shutting down imklog", + errno); + fklog = -1; + } + break; + } + + for (p = pRcv; (q = strchr(p, '\n')) != NULL; p = q + 1) { + *q = '\0'; + Syslog(LOG_INFO, (uchar*) p); + } + len = strlen(p); + if (len >= iMaxLine - 1) { + Syslog(LOG_INFO, (uchar*)p); + len = 0; + } + if (len > 0) + memmove(pRcv, p, len + 1); + } + if (len > 0) + Syslog(LOG_INFO, pRcv); + + if(pRcv != NULL && (size_t) iMaxLine >= sizeof(bufRcv) - 1) + free(pRcv); +} + + +/* to be called in the module's AfterRun entry point + * rgerhards, 2008-04-09 + */ +rsRetVal klogAfterRun(void) +{ + DEFiRet; + if(fklog != -1) + close(fklog); + RETiRet; +} + + + +/* to be called in the module's WillRun entry point, this is the main + * "message pull" mechanism. + * rgerhards, 2008-04-09 + */ +rsRetVal klogLogKMsg(void) +{ + DEFiRet; + sun_sys_poll(); + RETiRet; +} + + +/* provide the (system-specific) default facility for internal messages + * rgerhards, 2008-04-14 + */ +int +klogFacilIntMsg(void) +{ + return LOG_SYSLOG; +} + diff --git a/plugins/imklog/solaris_cddl.c b/plugins/imklog/solaris_cddl.c new file mode 100644 index 00000000..1053de66 --- /dev/null +++ b/plugins/imklog/solaris_cddl.c @@ -0,0 +1,323 @@ +#define MAXLINE 4096 +/* + * CDDL HEADER START + * + * The contents of this file are subject to the terms of the + * Common Development and Distribution License (the "License"). + * You may not use this file except in compliance with the License. + * + * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE + * or http://www.opensolaris.org/os/licensing. + * See the License for the specific language governing permissions + * and limitations under the License. + * + * When distributing Covered Code, include this CDDL HEADER in each + * file and include the License file at usr/src/OPENSOLARIS.LICENSE. + * If applicable, add the following below this CDDL HEADER, with the + * fields enclosed by brackets "[]" replaced with your own identifying + * information: Portions Copyright [yyyy] [name of copyright owner] + * + * CDDL HEADER END + */ +/* Portions Copyright 2010 by Rainer Gerhards and Adiscon + */ +/* + * Copyright 2008 Sun Microsystems, Inc. All rights reserved. + * Use is subject to license terms. + */ + +/* + * Copyright (c) 1984, 1986, 1987, 1988, 1989 AT&T + * All Rights Reserved + */ + +/* + * University Copyright- Copyright (c) 1982, 1986, 1988 + * The Regents of the University of California + * All Rights Reserved + * + * University Acknowledgment- Portions of this document are derived from + * software developed by the University of California, Berkeley, and its + * contributors. + */ +#include "config.h" +#include <stdlib.h> +#include <unistd.h> +#include <errno.h> +#include <pthread.h> +#include <sys/poll.h> +#include <pthread.h> +#include <fcntl.h> +#include <stropts.h> +#include <assert.h> +#include <sys/strlog.h> + +#include "rsyslog.h" + +static struct pollfd Pfd; /* Pollfd for local the log device */ + + +/* findnl_bkwd: + * Scans each character in buf until it finds the last newline in buf, + * or the scanned character becomes the last COMPLETE character in buf. + * Returns the number of scanned bytes. + * + * buf - pointer to a buffer containing the message string + * len - the length of the buffer + */ +size_t +findnl_bkwd(const char *buf, const size_t len) +{ + const char *p; + size_t mb_cur_max; + pthread_t mythreadno; + + if (Debug) { + mythreadno = pthread_self(); + } + + if (len == 0) { + return (0); + } + + mb_cur_max = MB_CUR_MAX; + + if (mb_cur_max == 1) { + /* single-byte locale */ + for (p = buf + len - 1; p != buf; p--) { + if (*p == '\n') { + return ((size_t)(p - buf)); + } + } + return ((size_t)len); + } else { + /* multi-byte locale */ + int mlen; + const char *nl; + size_t rem; + + p = buf; + nl = NULL; + for (rem = len; rem >= mb_cur_max; ) { + mlen = mblen(p, mb_cur_max); + if (mlen == -1) { + /* + * Invalid character found. + */ + dbgprintf("klog:findnl_bkwd(%u): Invalid MB " + "sequence\n", mythreadno); + /* + * handle as a single byte character. + */ + p++; + rem--; + } else { + /* + * It's guaranteed that *p points to + * the 1st byte of a multibyte character. + */ + if (*p == '\n') { + nl = p; + } + p += mlen; + rem -= mlen; + } + } + if (nl) { + return ((size_t)(nl - buf)); + } + /* + * no newline nor null byte found. + * Also it's guaranteed that *p points to + * the 1st byte of a (multibyte) character + * at this point. + */ + return (len - rem); + } +} +//___ end + + +/* Attempts to open the local log device + * and return a file descriptor. + */ +int +sun_openklog(char *name, int mode) +{ + int fd; + struct strioctl str; + + if ((fd = open(name, mode)) < 0) { + //logerror("cannot open %s", name); + dbgprintf("klog:openklog: cannot open %s (%d)\n", + name, errno); + return (-1); + } + str.ic_cmd = I_CONSLOG; + str.ic_timout = 0; + str.ic_len = 0; + str.ic_dp = NULL; + if (ioctl(fd, I_STR, &str) < 0) { + //logerror("cannot register to log console messages"); + dbgprintf("klog:openklog: cannot register to log " + "console messages (%d)\n", errno); + return (-1); + } + Pfd.fd = fd; + Pfd.events = POLLIN; + return (fd); +} + + +/* + * Pull up one message from log driver. + */ +void +sun_getkmsg() +{ + int flags = 0, i; + char *lastline; + struct strbuf ctl, dat; + struct log_ctl hdr; + char buf[MAXLINE+1]; + size_t buflen; + size_t len; + char tmpbuf[MAXLINE+1]; + + dat.maxlen = MAXLINE; + dat.buf = buf; + ctl.maxlen = sizeof (struct log_ctl); + ctl.buf = (caddr_t)&hdr; + + while ((i = getmsg(Pfd.fd, &ctl, &dat, &flags)) == MOREDATA) { + lastline = &dat.buf[dat.len]; + *lastline = '\0'; + + dbgprintf("klog:sys_poll: getmsg: dat.len = %d\n", dat.len); + buflen = strlen(buf); + len = findnl_bkwd(buf, buflen); + + (void) memcpy(tmpbuf, buf, len); + tmpbuf[len] = '\0'; + + /* Format sys will enqueue the log message. + * Set the sync flag if timeout != 0, which + * means that we're done handling all the + * initial messages ready during startup. + */ + Syslog(LOG_INFO, buf); + /*if (timeout == 0) { + formatsys(&hdr, tmpbuf, 0); + //sys_init_msg_count++; + } else { + formatsys(&hdr, tmpbuf, 1); + }*/ + + if (len != buflen) { + /* If anything remains in buf */ + size_t remlen; + + if (buf[len] == '\n') { + /* skip newline */ + len++; + } + + /* Move the remaining bytes to + * the beginnning of buf. + */ + + remlen = buflen - len; + (void) memcpy(buf, &buf[len], remlen); + dat.maxlen = MAXLINE - remlen; + dat.buf = &buf[remlen]; + } else { + dat.maxlen = MAXLINE; + dat.buf = buf; + } + } + + if (i == 0 && dat.len > 0) { + dat.buf[dat.len] = '\0'; + /* + * Format sys will enqueue the log message. + * Set the sync flag if timeout != 0, which + * means that we're done handling all the + * initial messages ready during startup. + */ + dbgprintf("klog:getkmsg: getmsg: dat.maxlen = %d\n", dat.maxlen); + dbgprintf("klog:getkmsg: getmsg: dat.len = %d\n", dat.len); + dbgprintf("klog:getkmsg: getmsg: strlen(dat.buf) = %d\n", strlen(dat.buf)); + dbgprintf("klog:getkmsg: getmsg: dat.buf = \"%s\"\n", dat.buf); + dbgprintf("klog:getkmsg: buf len = %d\n", strlen(buf)); + //if (timeout == 0) { + //formatsys(&hdr, buf, 0); + //--sys_init_msg_count++; + //} else { + //formatsys(&hdr, buf, 1); + //} + Syslog(LOG_INFO, buf); + } else if (i < 0 && errno != EINTR) { + if(1){ // (!shutting_down) { + dbgprintf("klog:kernel log driver read error"); + } + // TODO trigger retry logic + //(void) close(Pfd.fd); + //Pfd.fd = -1; + } +} + + +/* this thread listens to the local stream log driver for log messages + * generated by this host, formats them, and queues them to the logger + * thread. + */ +/*ARGSUSED*/ +void * +sun_sys_poll() +{ + int nfds; + + dbgprintf("klog:sys_poll: sys_thread started\n"); + + /* + * Try to process as many messages as we can without blocking on poll. + * We count such "initial" messages with sys_init_msg_count and + * enqueue them without the SYNC_FILE flag. When no more data is + * waiting on the local log device, we set timeout to INFTIM, + * clear sys_init_msg_count, and generate a flush message to sync + * the previously counted initial messages out to disk. + */ + + for (;;) { + errno = 0; + + nfds = poll(&Pfd, 1, INFTIM); + + if (nfds == 0) + continue; + + if (nfds < 0) { + if (errno != EINTR) + dbgprintf("klog:poll error");// logerror("poll"); + continue; + } + if (Pfd.revents & POLLIN) { + sun_getkmsg(); + } else { + // TODO: shutdown, the rsyslog way (in v5!) + //if (shutting_down) { + //pthread_exit(0); + //} + if (Pfd.revents & (POLLNVAL|POLLHUP|POLLERR)) { + // TODO: trigger retry logic +/* logerror("kernel log driver poll error"); + (void) close(Pfd.fd); + Pfd.fd = -1; + */ + } + } + + } + /*NOTREACHED*/ + return (NULL); +} diff --git a/plugins/imklog/solaris_cddl.h b/plugins/imklog/solaris_cddl.h new file mode 100644 index 00000000..22295658 --- /dev/null +++ b/plugins/imklog/solaris_cddl.h @@ -0,0 +1 @@ +int sun_openklog(char *name, int mode); diff --git a/plugins/imudp/imudp.c b/plugins/imudp/imudp.c index ecc36c28..d76f3544 100644 --- a/plugins/imudp/imudp.c +++ b/plugins/imudp/imudp.c @@ -242,7 +242,7 @@ processSocket(int fd, struct sockaddr_storage *frominetPrev, int *pbIsPermitted, } } - DBGPRINTF("recv(%d,%d)/%s,acl:%d,msg:%.80s\n", fd, (int) lenRcvBuf, fromHost, *pbIsPermitted, pRcvBuf); + DBGPRINTF("recv(%d,%d)/%s,acl:%d,msg:%s\n", fd, (int) lenRcvBuf, fromHost, *pbIsPermitted, pRcvBuf); if(*pbIsPermitted) { if((iTimeRequery == 0) || (iNbrTimeUsed++ % iTimeRequery) == 0) { diff --git a/plugins/imuxsock/imuxsock.c b/plugins/imuxsock/imuxsock.c index 5aa3c25b..daa3bb47 100644 --- a/plugins/imuxsock/imuxsock.c +++ b/plugins/imuxsock/imuxsock.c @@ -71,7 +71,7 @@ DEFobjCurrIf(errmsg) DEFobjCurrIf(glbl) DEFobjCurrIf(prop) -static prop_t *pInputName = NULL; /* our inputName currently is always "imudp", and this will hold it */ +static prop_t *pInputName = NULL; /* our inputName currently is always "imuxsock", and this will hold it */ static int startIndexUxLocalSockets; /* process funix from that index on (used to * suppress local logging. rgerhards 2005-08-01 * read-only after startup @@ -315,7 +315,16 @@ CODESTARTwillRun register int i; /* first apply some config settings */ - startIndexUxLocalSockets = bOmitLocalLogging ? 1 : 0; +# ifdef OS_SOLARIS + /* under solaris, we must NEVER process the local log socket, because + * it is implemented there differently. If we used it, we would actually + * delete it and render the system partly unusable. So don't do that. + * rgerhards, 2010-03-26 + */ + startIndexUxLocalSockets = 1; +# else + startIndexUxLocalSockets = bOmitLocalLogging ? 1 : 0; +# endif if(pLogSockName != NULL) funixn[0] = pLogSockName; @@ -327,7 +336,7 @@ CODESTARTwillRun /* we need to create the inputName property (only once during our lifetime) */ CHKiRet(prop.Construct(&pInputName)); - CHKiRet(prop.SetString(pInputName, UCHAR_CONSTANT("imudp"), sizeof("imudp") - 1)); + CHKiRet(prop.SetString(pInputName, UCHAR_CONSTANT("imuxsock"), sizeof("imuxsock") - 1)); CHKiRet(prop.ConstructFinalize(pInputName)); finalize_it: @@ -344,7 +353,7 @@ CODESTARTafterRun close(funix[i]); /* Clean-up files. */ - for (i = 0; i < nfunix; i++) + for(i = startIndexUxLocalSockets; i < nfunix; i++) if (funixn[i] && funix[i] != -1) unlink((char*) funixn[i]); /* free no longer needed string */ diff --git a/plugins/omgssapi/omgssapi.c b/plugins/omgssapi/omgssapi.c index 7b5a46e1..782ac22f 100644 --- a/plugins/omgssapi/omgssapi.c +++ b/plugins/omgssapi/omgssapi.c @@ -377,7 +377,7 @@ CODESTARTtryResume ENDtryResume BEGINdoAction - char *psz; /* temporary buffering */ + char *psz = NULL; /* temporary buffering */ register unsigned l; int iMaxLine; CODESTARTdoAction @@ -431,11 +431,14 @@ CODESTARTdoAction * rgerhards, 2006-11-30 */ dbgprintf("Compression failed, sending uncompressed message\n"); + free(out); } else if(destLen+1 < l) { /* only use compression if there is a gain in using it! */ dbgprintf("there is gain in compression, so we do it\n"); psz = (char*) out; l = destLen + 1; /* take care for the "z" at message start! */ + } else { + free(out); } ++destLen; } @@ -450,6 +453,12 @@ CODESTARTdoAction break; } finalize_it: +# ifdef USE_NETZIP + if((psz != NULL) && (psz != (char*) ppString[0])) { + /* we need to free temporary buffer, alloced above - Naoya Nakazawa, 2010-01-11 */ + free(psz); + } +# endif ENDdoAction diff --git a/plugins/ompgsql/createDB.sql b/plugins/ompgsql/createDB.sql index facdcaa0..2f72a0a6 100644 --- a/plugins/ompgsql/createDB.sql +++ b/plugins/ompgsql/createDB.sql @@ -1,4 +1,4 @@ -CREATE DATABASE "Syslog"; +CREATE DATABASE 'Syslog' WITH ENCODING 'SQL_ASCII'; \c Syslog; CREATE TABLE SystemEvents ( diff --git a/runtime/Makefile.am b/runtime/Makefile.am index c1a15198..ac006bca 100644 --- a/runtime/Makefile.am +++ b/runtime/Makefile.am @@ -9,6 +9,7 @@ librsyslog_la_SOURCES = \ rsyslog.h \ unicode-helper.h \ atomic.h \ + atomic-posix-sem.c \ syslogd-types.h \ module-template.h \ obj-types.h \ diff --git a/runtime/atomic-posix-sem.c b/runtime/atomic-posix-sem.c new file mode 100644 index 00000000..979fae02 --- /dev/null +++ b/runtime/atomic-posix-sem.c @@ -0,0 +1,70 @@ +/* atomic_posix_sem.c: This file supplies an emulation for atomic operations using + * POSIX semaphores. + * + * Copyright 2010 DResearch Digital Media Systems GmbH + * + * This file is part of the rsyslog runtime library. + * + * The rsyslog runtime library is free software: you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * The rsyslog runtime library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with the rsyslog runtime library. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + * A copy of the LGPL can be found in the file "COPYING.LESSER" in this distribution. + */ + +#include "config.h" +#ifndef HAVE_ATOMIC_BUILTINS +#ifdef HAVE_SEMAPHORE_H +#include <semaphore.h> +#include <errno.h> + +#include "atomic.h" +#include "rsyslog.h" +#include "srUtils.h" + +sem_t atomicSem; + +rsRetVal +atomicSemInit(void) +{ + DEFiRet; + + dbgprintf("init posix semaphore for atomics emulation\n"); + if(sem_init(&atomicSem, 0, 1) == -1) + { + char errStr[1024]; + rs_strerror_r(errno, errStr, sizeof(errStr)); + dbgprintf("init posix semaphore for atomics emulation failed: %s\n", errStr); + iRet = RS_RET_SYS_ERR; /* the right error code ??? */ + } + + RETiRet; +} + +void +atomicSemExit(void) +{ + dbgprintf("destroy posix semaphore for atomics emulation\n"); + if(sem_destroy(&atomicSem) == -1) + { + char errStr[1024]; + rs_strerror_r(errno, errStr, sizeof(errStr)); + dbgprintf("destroy posix semaphore for atomics emulation failed: %s\n", errStr); + } +} + +#endif /* HAVE_SEMAPHORE_H */ +#endif /* !defined(HAVE_ATOMIC_BUILTINS) */ + +/* vim:set ai: + */ diff --git a/runtime/atomic.h b/runtime/atomic.h index d5aaf56b..ea3be37a 100644 --- a/runtime/atomic.h +++ b/runtime/atomic.h @@ -53,6 +53,122 @@ # define ATOMIC_CAS(data, oldVal, newVal) __sync_bool_compare_and_swap(&(data), (oldVal), (newVal)); # define ATOMIC_CAS_VAL(data, oldVal, newVal) __sync_val_compare_and_swap(&(data), (oldVal), (newVal)); #else +#ifdef HAVE_SEMAPHORE_H + /* we use POSIX semaphores instead */ + +#include "rsyslog.h" +#include <semaphore.h> + +extern sem_t atomicSem; +rsRetVal atomicSemInit(void); +void atomicSemExit(void); + +#if HAVE_TYPEOF +#define my_typeof(x) typeof(x) +#else /* sorry, can't determine types, using 'int' */ +#define my_typeof(x) int +#endif + +# define ATOMIC_SUB(data, val) \ +({ \ + my_typeof(data) tmp; \ + sem_wait(&atomicSem); \ + tmp = data; \ + data -= val; \ + sem_post(&atomicSem); \ + tmp; \ +}) + +# define ATOMIC_ADD(data, val) \ +({ \ + my_typeof(data) tmp; \ + sem_wait(&atomicSem); \ + tmp = data; \ + data += val; \ + sem_post(&atomicSem); \ + tmp; \ +}) + +# define ATOMIC_INC_AND_FETCH(data) \ +({ \ + my_typeof(data) tmp; \ + sem_wait(&atomicSem); \ + tmp = data; \ + data += 1; \ + sem_post(&atomicSem); \ + tmp; \ +}) + +# define ATOMIC_INC(data) ((void) ATOMIC_INC_AND_FETCH(data)) + +# define ATOMIC_DEC_AND_FETCH(data) \ +({ \ + sem_wait(&atomicSem); \ + data -= 1; \ + sem_post(&atomicSem); \ + data; \ +}) + +# define ATOMIC_DEC(data) ((void) ATOMIC_DEC_AND_FETCH(data)) + +# define ATOMIC_FETCH_32BIT(data) ((unsigned) ATOMIC_ADD((data), 0xffffffff)) + +# define ATOMIC_STORE_1_TO_32BIT(data) \ +({ \ + my_typeof(data) tmp; \ + sem_wait(&atomicSem); \ + tmp = data; \ + data = 1; \ + sem_post(&atomicSem); \ + tmp; \ +}) + +# define ATOMIC_STORE_0_TO_INT(data) \ +({ \ + my_typeof(data) tmp; \ + sem_wait(&atomicSem); \ + tmp = data; \ + data = 0; \ + sem_post(&atomicSem); \ + tmp; \ +}) + +# define ATOMIC_STORE_1_TO_INT(data) \ +({ \ + my_typeof(data) tmp; \ + sem_wait(&atomicSem); \ + tmp = data; \ + data = 1; \ + sem_post(&atomicSem); \ + tmp; \ +}) + +# define ATOMIC_CAS(data, oldVal, newVal) \ +({ \ + int ret; \ + sem_wait(&atomicSem); \ + if(data != oldVal) ret = 0; \ + else \ + { \ + data = newVal; \ + ret = 1; \ + } \ + sem_post(&atomicSem); \ + ret; \ +}) + +# define ATOMIC_CAS_VAL(data, oldVal, newVal) \ +({ \ + sem_wait(&atomicSem); \ + if(data == oldVal) \ + { \ + data = newVal; \ + } \ + sem_post(&atomicSem); \ + data; \ +}) + +#else /* not HAVE_SEMAPHORE_H */ /* note that we gained parctical proof that theoretical problems DO occur * if we do not properly address them. See this blog post for details: * http://blog.gerhards.net/2009/01/rsyslog-data-race-analysis.html @@ -66,6 +182,10 @@ # define ATOMIC_DEC_AND_FETCH(data) (--(data)) # define ATOMIC_FETCH_32BIT(data) (data) # define ATOMIC_STORE_1_TO_32BIT(data) (data) = 1 +# define ATOMIC_STORE_1_TO_INT(data) (data) = 1 +# define ATOMIC_STORE_0_TO_INT(data) (data) = 0 +# define ATOMIC_CAS_VAL(data, oldVal, newVal) (data) = (newVal) +#endif #endif #endif /* #ifndef INCLUDED_ATOMIC_H */ diff --git a/runtime/cfsysline.c b/runtime/cfsysline.c index 184c0d87..5df8e64c 100644 --- a/runtime/cfsysline.c +++ b/runtime/cfsysline.c @@ -217,9 +217,11 @@ static rsRetVal doGetSize(uchar **pp, rsRetVal (*pSetHdlr)(void*, uid_t), void * case 'K': i *= 1000; ++(*pp); break; case 'M': i *= 1000000; ++(*pp); break; case 'G': i *= 1000000000; ++(*pp); break; - case 'T': i *= 1000000000000; ++(*pp); break; /* tera */ - case 'P': i *= 1000000000000000; ++(*pp); break; /* peta */ - case 'E': i *= 1000000000000000000; ++(*pp); break; /* exa */ + /* we need to use the multiplication below because otherwise + * the compiler gets an error during constant parsing */ + case 'T': i *= (int64) 1000 * 1000000000; ++(*pp); break; /* tera */ + case 'P': i *= (int64) 1000000 * 1000000000; ++(*pp); break; /* peta */ + case 'E': i *= (int64) 1000000000 * 1000000000; ++(*pp); break; /* exa */ } /* done */ diff --git a/runtime/conf.c b/runtime/conf.c index b92664a1..ef795237 100644 --- a/runtime/conf.c +++ b/runtime/conf.c @@ -513,7 +513,7 @@ finalize_it: rsRetVal cflineParseTemplateName(uchar** pp, omodStringRequest_t *pOMSR, int iEntry, int iTplOpts, uchar *dfltTplName) { uchar *p; - uchar *tplName; + uchar *tplName = NULL; cstr_t *pStrB; DEFiRet; @@ -550,10 +550,12 @@ rsRetVal cflineParseTemplateName(uchar** pp, omodStringRequest_t *pOMSR, int iEn CHKiRet(cstrConvSzStrAndDestruct(pStrB, &tplName, 0)); } - iRet = OMSRsetEntry(pOMSR, iEntry, tplName, iTplOpts); - if(iRet != RS_RET_OK) goto finalize_it; + CHKiRet(OMSRsetEntry(pOMSR, iEntry, tplName, iTplOpts)); finalize_it: + if(iRet != RS_RET_OK) + free(tplName); + *pp = p; RETiRet; diff --git a/runtime/ctok.c b/runtime/ctok.c index 6f5f0273..18ddaed2 100644 --- a/runtime/ctok.c +++ b/runtime/ctok.c @@ -87,11 +87,12 @@ ctokUngetCharFromStream(ctok_t *pThis, uchar __attribute__((unused)) c) } -/* get the next character from the input "stream" (currently just a in-memory - * string...) -- rgerhards, 2008-02-19 +/* get the next character from the input "stream". Note that this version + * does NOT look for comment characters as end-of-stream, so it is suitable + * when building constant strings! -- rgerhards, 2010-03-01 */ -static rsRetVal -ctokGetCharFromStream(ctok_t *pThis, uchar *pc) +static inline rsRetVal +ctokGetCharFromStreamNoComment(ctok_t *pThis, uchar *pc) { DEFiRet; @@ -99,7 +100,7 @@ ctokGetCharFromStream(ctok_t *pThis, uchar *pc) ASSERT(pc != NULL); /* end of string or begin of comment terminates the "stream" */ - if(*pThis->pp == '\0' || *pThis->pp == '#') { + if(*pThis->pp == '\0') { ABORT_FINALIZE(RS_RET_EOS); } else { *pc = *pThis->pp; @@ -111,6 +112,28 @@ finalize_it: } +/* get the next character from the input "stream" (currently just a in-memory + * string...) -- rgerhards, 2008-02-19 + */ +static rsRetVal +ctokGetCharFromStream(ctok_t *pThis, uchar *pc) +{ + DEFiRet; + + ISOBJ_TYPE_assert(pThis, ctok); + ASSERT(pc != NULL); + + CHKiRet(ctokGetCharFromStreamNoComment(pThis, pc)); + /* begin of comment terminates the "stream"! */ + if(*pc == '#') { + ABORT_FINALIZE(RS_RET_EOS); + } + +finalize_it: + RETiRet; +} + + /* skip whitespace in the input "stream". * rgerhards, 2008-02-19 */ @@ -302,7 +325,7 @@ ctokGetSimpStr(ctok_t *pThis, ctok_token_t *pToken) pToken->tok = ctok_SIMPSTR; CHKiRet(cstrConstruct(&pstrVal)); - CHKiRet(ctokGetCharFromStream(pThis, &c)); + CHKiRet(ctokGetCharFromStreamNoComment(pThis, &c)); /* while we are in escape mode (had a backslash), no sequence * terminates the loop. If outside, it is terminated by a single quote. */ @@ -317,7 +340,7 @@ ctokGetSimpStr(ctok_t *pThis, ctok_token_t *pToken) CHKiRet(cstrAppendChar(pstrVal, c)); } } - CHKiRet(ctokGetCharFromStream(pThis, &c)); + CHKiRet(ctokGetCharFromStreamNoComment(pThis, &c)); } CHKiRet(cstrFinalize(pstrVal)); diff --git a/runtime/datetime.c b/runtime/datetime.c index 6160bd7c..eff72f91 100644 --- a/runtime/datetime.c +++ b/runtime/datetime.c @@ -291,11 +291,11 @@ ParseTIMESTAMP3339(struct syslogTime *pTime, uchar** ppszTS, int *pLenStr) } /* OK, we actually have a 3339 timestamp, so let's indicated this */ - if(lenStr > 0 && *pszTS == ' ') { + if(lenStr > 0) { + if(*pszTS != ' ') /* if it is not a space, it can not be a "good" time - 2010-02-22 rgerhards */ + ABORT_FINALIZE(RS_RET_INVLD_TIME); + ++pszTS; /* just skip past it */ --lenStr; - ++pszTS; - } else { - ABORT_FINALIZE(RS_RET_INVLD_TIME); } /* we had success, so update parse pointer and caller-provided timestamp */ @@ -510,6 +510,7 @@ ParseTIMESTAMP3164(struct syslogTime *pTime, uchar** ppszTS, int *pLenStr) if(lenStr == 0 || *pszTS++ != ' ') ABORT_FINALIZE(RS_RET_INVLD_TIME); + --lenStr; /* we accept a slightly malformed timestamp when receiving. This is * we accept one-digit days @@ -565,7 +566,13 @@ ParseTIMESTAMP3164(struct syslogTime *pTime, uchar** ppszTS, int *pLenStr) * invalid format, it occurs frequently enough (e.g. with Cisco devices) * to permit it as a valid case. -- rgerhards, 2008-09-12 */ - if(lenStr == 0 || *pszTS++ == ':') { + if(lenStr > 0 && *pszTS == ':') { + ++pszTS; /* just skip past it */ + --lenStr; + } + if(lenStr > 0) { + if(*pszTS != ' ') /* if it is not a space, it can not be a "good" time - 2010-02-22 rgerhards */ + ABORT_FINALIZE(RS_RET_INVLD_TIME); ++pszTS; /* just skip past it */ --lenStr; } @@ -786,8 +793,12 @@ int formatTimestamp3339(struct syslogTime *ts, char* pBuf) * buffer that will receive the resulting string. The function * returns the size of the timestamp written in bytes (without * the string termnator). If 0 is returend, an error occured. + * rgerhards, 2010-03-05: Added support to for buggy 3164 dates, + * where a zero-digit is written instead of a space for the first + * day character if day < 10. syslog-ng seems to do that, and some + * parsing scripts (in migration cases) rely on that. */ -int formatTimestamp3164(struct syslogTime *ts, char* pBuf) +int formatTimestamp3164(struct syslogTime *ts, char* pBuf, int bBuggyDay) { static char* monthNames[12] = { "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" }; @@ -800,7 +811,7 @@ int formatTimestamp3164(struct syslogTime *ts, char* pBuf) pBuf[2] = monthNames[(ts->month - 1) % 12][2]; pBuf[3] = ' '; iDay = (ts->day / 10) % 10; /* we need to write a space if the first digit is 0 */ - pBuf[4] = iDay ? iDay + '0' : ' '; + pBuf[4] = (bBuggyDay || iDay > 0) ? iDay + '0' : ' '; pBuf[5] = ts->day % 10 + '0'; pBuf[6] = ' '; pBuf[7] = (ts->hour / 10) % 10 + '0'; diff --git a/runtime/datetime.h b/runtime/datetime.h index 8140eb71..9dcce3c5 100644 --- a/runtime/datetime.h +++ b/runtime/datetime.h @@ -39,15 +39,17 @@ BEGINinterface(datetime) /* name must also be changed in ENDinterface macro! */ int (*formatTimestampToMySQL)(struct syslogTime *ts, char* pDst); int (*formatTimestampToPgSQL)(struct syslogTime *ts, char *pDst); int (*formatTimestamp3339)(struct syslogTime *ts, char* pBuf); - int (*formatTimestamp3164)(struct syslogTime *ts, char* pBuf); + int (*formatTimestamp3164)(struct syslogTime *ts, char* pBuf, int); int (*formatTimestampSecFrac)(struct syslogTime *ts, char* pBuf); ENDinterface(datetime) -#define datetimeCURR_IF_VERSION 2 /* increment whenever you change the interface structure! */ +#define datetimeCURR_IF_VERSION 4 /* increment whenever you change the interface structure! */ /* interface changes: * 1 - initial version * 2 - not compatible to 1 - bugfix required ParseTIMESTAMP3164 to accept char ** as * last parameter. Did not try to remain compatible as this is not something any * third-party module should call. -- rgerhards, 2008.-09-12 + * 3 - taken by v5 branch! + * 4 - formatTimestamp3164 takes a third int parameter */ /* prototypes */ diff --git a/runtime/debug.c b/runtime/debug.c index c23dec3b..c82a411d 100644 --- a/runtime/debug.c +++ b/runtime/debug.c @@ -1,4 +1,3 @@ -#include <sys/syscall.h> /* debug.c * * This file proides debug and run time error analysis support. Some of the @@ -550,6 +549,7 @@ if(pLog == NULL) { return; /* if we don't know it yet, we can not clean up... */ } #endif +#include <sys/syscall.h> /* we found the last lock entry. We now need to see from which FuncDB we need to * remove it. This is recorded inside the mutex log entry. @@ -922,7 +922,7 @@ void dbgoprint(obj_t *pObj, char *fmt, ...) { va_list ap; - char pszWriteBuf[1024]; + char pszWriteBuf[32*1024]; size_t lenWriteBuf; if(!(Debug && debugging_on)) @@ -952,7 +952,7 @@ void dbgprintf(char *fmt, ...) { va_list ap; - char pszWriteBuf[1024]; + char pszWriteBuf[20480]; size_t lenWriteBuf; if(!(Debug && debugging_on)) @@ -961,6 +961,15 @@ dbgprintf(char *fmt, ...) va_start(ap, fmt); lenWriteBuf = vsnprintf(pszWriteBuf, sizeof(pszWriteBuf), fmt, ap); va_end(ap); + if(lenWriteBuf >= sizeof(pszWriteBuf)) { + /* prevent buffer overrruns and garbagge display */ + pszWriteBuf[sizeof(pszWriteBuf) - 5] = '.'; + pszWriteBuf[sizeof(pszWriteBuf) - 4] = '.'; + pszWriteBuf[sizeof(pszWriteBuf) - 3] = '.'; + pszWriteBuf[sizeof(pszWriteBuf) - 2] = '\n'; + pszWriteBuf[sizeof(pszWriteBuf) - 1] = '\0'; + lenWriteBuf = sizeof(pszWriteBuf); + } dbgprint(NULL, pszWriteBuf, lenWriteBuf); } diff --git a/runtime/msg.c b/runtime/msg.c index b45775b6..91057f97 100644 --- a/runtime/msg.c +++ b/runtime/msg.c @@ -1171,7 +1171,7 @@ uchar *getMSG(msg_t *pM) if(pM == NULL) ret = UCHAR_CONSTANT(""); else { - if(pM->offMSG == -1) + if(pM->iLenMSG == 0) ret = UCHAR_CONSTANT(""); else ret = pM->pszRawMsg + pM->offMSG; @@ -1213,10 +1213,12 @@ static inline char *getTimeReported(msg_t *pM, enum tplFormatTypes eFmt) switch(eFmt) { case tplFmtDefault: case tplFmtRFC3164Date: + case tplFmtRFC3164BuggyDate: MsgLock(pM); if(pM->pszTIMESTAMP3164 == NULL) { pM->pszTIMESTAMP3164 = pM->pszTimestamp3164; - datetime.formatTimestamp3164(&pM->tTIMESTAMP, pM->pszTIMESTAMP3164); + datetime.formatTimestamp3164(&pM->tTIMESTAMP, pM->pszTIMESTAMP3164, + (eFmt == tplFmtRFC3164BuggyDate)); } MsgUnlock(pM); return(pM->pszTIMESTAMP3164); @@ -1279,7 +1281,7 @@ static inline char *getTimeGenerated(msg_t *pM, enum tplFormatTypes eFmt) MsgUnlock(pM); return ""; } - datetime.formatTimestamp3164(&pM->tRcvdAt, pM->pszRcvdAt3164); + datetime.formatTimestamp3164(&pM->tRcvdAt, pM->pszRcvdAt3164, 0); } MsgUnlock(pM); return(pM->pszRcvdAt3164); @@ -1306,13 +1308,15 @@ static inline char *getTimeGenerated(msg_t *pM, enum tplFormatTypes eFmt) MsgUnlock(pM); return(pM->pszRcvdAt_PgSQL); case tplFmtRFC3164Date: + case tplFmtRFC3164BuggyDate: MsgLock(pM); if(pM->pszRcvdAt3164 == NULL) { if((pM->pszRcvdAt3164 = malloc(16)) == NULL) { MsgUnlock(pM); return ""; } - datetime.formatTimestamp3164(&pM->tRcvdAt, pM->pszRcvdAt3164); + datetime.formatTimestamp3164(&pM->tRcvdAt, pM->pszRcvdAt3164, + (eFmt == tplFmtRFC3164BuggyDate)); } MsgUnlock(pM); return(pM->pszRcvdAt3164); @@ -1947,12 +1951,20 @@ void MsgSetHOSTNAME(msg_t *pThis, uchar* pszHOSTNAME, int lenHOSTNAME) /* set the offset of the MSG part into the raw msg buffer + * Note that the offset may be higher than the length of the raw message + * (exactly by one). This can happen if we have a message that does not + * contain any MSG part. */ void MsgSetMSGoffs(msg_t *pMsg, short offs) { ISOBJ_TYPE_assert(pMsg, msg); - pMsg->iLenMSG = pMsg->iLenRawMsg - offs; pMsg->offMSG = offs; + if(offs > pMsg->iLenRawMsg) { + assert(offs - 1 == pMsg->iLenRawMsg); + pMsg->iLenMSG = 0; + } else { + pMsg->iLenMSG = pMsg->iLenRawMsg - offs; + } } @@ -1986,7 +1998,8 @@ rsRetVal MsgReplaceMSG(msg_t *pThis, uchar* pszMSG, int lenMSG) pThis->pszRawMsg = bufNew; } - memcpy(pThis->pszRawMsg + pThis->offMSG, pszMSG, lenMSG); + if(lenMSG > 0) + memcpy(pThis->pszRawMsg + pThis->offMSG, pszMSG, lenMSG); pThis->pszRawMsg[lenNew] = '\0'; /* this also works with truncation! */ pThis->iLenRawMsg = lenNew; pThis->iLenMSG = lenMSG; @@ -1997,6 +2010,8 @@ finalize_it: /* set raw message in message object. Size of message is provided. + * The function makes sure that the stored rawmsg is properly + * terminated by '\0'. * rgerhards, 2009-06-16 */ void MsgSetRawMsg(msg_t *pThis, char* pszRawMsg, size_t lenMsg) @@ -2139,6 +2154,10 @@ static uchar *getNOW(eNOWType eNow) * be used in selector line processing. * rgerhards 2005-09-15 */ +/* a quick helper to save some writing: */ +#define RET_OUT_OF_MEMORY { *pbMustBeFreed = 0;\ + *pPropLen = sizeof("**OUT OF MEMORY**") - 1; \ + return(UCHAR_CONSTANT("**OUT OF MEMORY**"));} uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, propid_t propID, size_t *pPropLen, unsigned short *pbMustBeFreed) @@ -2200,8 +2219,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, case PROP_PRI_TEXT: pBuf = malloc(20 * sizeof(uchar)); if(pBuf == NULL) { - *pbMustBeFreed = 0; - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } else { *pbMustBeFreed = 1; pRes = (uchar*)textpri((char*)pBuf, 20, getPRIi(pMsg)); @@ -2245,49 +2263,49 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, break; case PROP_SYS_NOW: if((pRes = getNOW(NOW_NOW)) == NULL) { - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } else *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ break; case PROP_SYS_YEAR: if((pRes = getNOW(NOW_YEAR)) == NULL) { - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } else *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ break; case PROP_SYS_MONTH: if((pRes = getNOW(NOW_MONTH)) == NULL) { - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } else *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ break; case PROP_SYS_DAY: if((pRes = getNOW(NOW_DAY)) == NULL) { - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } else *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ break; case PROP_SYS_HOUR: if((pRes = getNOW(NOW_HOUR)) == NULL) { - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } else *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ break; case PROP_SYS_HHOUR: if((pRes = getNOW(NOW_HHOUR)) == NULL) { - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } else *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ break; case PROP_SYS_QHOUR: if((pRes = getNOW(NOW_QHOUR)) == NULL) { - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } else *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ break; case PROP_SYS_MINUTE: if((pRes = getNOW(NOW_MINUTE)) == NULL) { - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } else *pbMustBeFreed = 1; /* all of these functions allocate dyn. memory */ break; @@ -2299,10 +2317,11 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, * error message unreadable. rgerhards, 2007-07-10 */ dbgprintf("invalid property id: '%d'\n", propID); + *pbMustBeFreed = 0; + *pPropLen = sizeof("**INVALID PROPERTY NAME**") - 1; return UCHAR_CONSTANT("**INVALID PROPERTY NAME**"); } - /* If we did not receive a template pointer, we are already done... */ if(pTpe == NULL) { return pRes; @@ -2357,8 +2376,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(pBuf == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } /* now copy */ memcpy(pBuf, pFld, iLen); @@ -2375,6 +2393,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(*pbMustBeFreed == 1) free(pRes); *pbMustBeFreed = 0; + *pPropLen = sizeof("**FIELD NOT FOUND**") - 1; return UCHAR_CONSTANT("**FIELD NOT FOUND**"); } } else if(pTpe->data.field.iFromPos != 0 || pTpe->data.field.iToPos != 0) { @@ -2403,8 +2422,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(pBuf == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } pSb = pRes; if(iFrom) { @@ -2434,9 +2452,15 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, } else { /* Check for regular expressions */ if (pTpe->data.field.has_regex != 0) { - if (pTpe->data.field.has_regex == 2) + if (pTpe->data.field.has_regex == 2) { /* Could not compile regex before! */ + if (*pbMustBeFreed == 1) { + free(pRes); + *pbMustBeFreed = 0; + } + *pPropLen = sizeof("**NO MATCH** **BAD REGULAR EXPRESSION**") - 1; return UCHAR_CONSTANT("**NO MATCH** **BAD REGULAR EXPRESSION**"); + } dbgprintf("string to match for regex is: %s\n", pRes); @@ -2476,12 +2500,16 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, free(pRes); *pbMustBeFreed = 0; } - if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_DFLTSTR) - return UCHAR_CONSTANT("**NO MATCH**"); - else if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_ZERO) - return UCHAR_CONSTANT("0"); - else - return UCHAR_CONSTANT(""); + if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_DFLTSTR) { + bufLen = sizeof("**NO MATCH**") - 1; + pRes = UCHAR_CONSTANT("**NO MATCH**"); + } else if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_ZERO) { + bufLen = 1; + pRes = UCHAR_CONSTANT("0"); + } else { + bufLen = 0; + pRes = UCHAR_CONSTANT(""); + } } } else { /* Match- but did it match the one we wanted? */ @@ -2492,10 +2520,16 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, free(pRes); *pbMustBeFreed = 0; } - if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_DFLTSTR) - return UCHAR_CONSTANT("**NO MATCH**"); - else - return UCHAR_CONSTANT(""); + if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_DFLTSTR) { + bufLen = sizeof("**NO MATCH**") - 1; + pRes = UCHAR_CONSTANT("**NO MATCH**"); + } else if(pTpe->data.field.nomatchAction == TPL_REGEX_NOMATCH_USE_ZERO) { + bufLen = 1; + pRes = UCHAR_CONSTANT("0"); + } else { + bufLen = 0; + pRes = UCHAR_CONSTANT(""); + } } } /* OK, we have a usable match - we now need to malloc pB */ @@ -2509,13 +2543,12 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if (pB == NULL) { if (*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } /* Lets copy the matched substring to the buffer */ memcpy(pB, pRes + iOffs + pmatch[pTpe->data.field.iSubMatchToUse].rm_so, iLenBuf); - bufLen = iLenBuf - 1; + bufLen = iLenBuf; pB[iLenBuf] = '\0';/* terminate string, did not happen before */ if (*pbMustBeFreed == 1) @@ -2533,6 +2566,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, free(pRes); *pbMustBeFreed = 0; } + *pPropLen = sizeof("***REGEXP NOT AVAILABLE***") - 1; return UCHAR_CONSTANT("***REGEXP NOT AVAILABLE***"); } } @@ -2565,8 +2599,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(pB == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } pSrc = pRes; while(*pSrc) { @@ -2612,8 +2645,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(pDst == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } for(pSrc = pRes; *pSrc; pSrc++) { if(!iscntrl((int) *pSrc)) @@ -2648,8 +2680,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(pDst == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } for(pSrc = pRes; *pSrc; pSrc++) { if(iscntrl((int) *pSrc)) @@ -2688,8 +2719,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(pB == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } while(*pRes) { if(iscntrl((int) *pRes)) { @@ -2734,8 +2764,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(pDst == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } for(pSrc = pRes; *pSrc; pSrc++) { if(*pSrc != '/') @@ -2770,8 +2799,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(pDst == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } for(pSrc = pRes; *pSrc; pSrc++) { if(*pSrc == '/') @@ -2825,8 +2853,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, /* ok, original copy, need a private one */ pB = malloc((iLn + 1) * sizeof(uchar)); if(pB == NULL) { - *pbMustBeFreed = 0; - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } memcpy(pB, pRes, iLn - 1); pRes = pB; @@ -2845,6 +2872,7 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(pTpe->data.field.options.bCSV) { /* we need to obtain a private copy, as we need to at least add the double quotes */ int iBufLen; + int i; uchar *pBStart; uchar *pDst; uchar *pSrc; @@ -2856,10 +2884,10 @@ uchar *MsgGetProp(msg_t *pMsg, struct templateEntry *pTpe, if(pDst == NULL) { if(*pbMustBeFreed == 1) free(pRes); - *pbMustBeFreed = 0; - return UCHAR_CONSTANT("**OUT OF MEMORY**"); + RET_OUT_OF_MEMORY; } pSrc = pRes; + i = 0; *pDst++ = '"'; /* starting quote */ while(*pSrc) { if(*pSrc == '"') diff --git a/runtime/net.c b/runtime/net.c index e91c8a7f..fe6eef5b 100644 --- a/runtime/net.c +++ b/runtime/net.c @@ -1010,8 +1010,8 @@ static int should_use_so_bsdcompat(void) { #ifndef OS_BSD - static int init_done; - static int so_bsdcompat_is_obsolete; + static int init_done = 0; + static int so_bsdcompat_is_obsolete = 0; if (!init_done) { struct utsname myutsname; diff --git a/runtime/parser.c b/runtime/parser.c index 466066e7..36e88ebd 100644 --- a/runtime/parser.c +++ b/runtime/parser.c @@ -176,7 +176,10 @@ sanitizeMessage(msg_t *pMsg) pszMsg = pMsg->pszRawMsg; lenMsg = pMsg->iLenRawMsg; - /* remove NUL character at end of message (see comment in function header) */ + /* remove NUL character at end of message (see comment in function header) + * Note that we do not need to add a NUL character in this case, because it + * is already present ;) + */ if(pszMsg[lenMsg-1] == '\0') { DBGPRINTF("dropped NUL at very end of message\n"); bUpdatedLen = TRUE; @@ -190,8 +193,9 @@ sanitizeMessage(msg_t *pMsg) */ if(bDropTrailingLF && pszMsg[lenMsg-1] == '\n') { DBGPRINTF("dropped LF at very end of message (DropTrailingLF is set)\n"); - bUpdatedLen = TRUE; lenMsg--; + pszMsg[lenMsg] = '\0'; + bUpdatedLen = TRUE; } /* it is much quicker to sweep over the message and see if it actually @@ -245,6 +249,7 @@ sanitizeMessage(msg_t *pMsg) } ++iSrc; } + pDst[iDst] = '\0'; MsgSetRawMsg(pMsg, (char*)pDst, iDst); /* save sanitized string */ diff --git a/runtime/rsyslog.c b/runtime/rsyslog.c index 443d0f41..5750ca76 100644 --- a/runtime/rsyslog.c +++ b/runtime/rsyslog.c @@ -80,6 +80,7 @@ #include "prop.h" #include "rule.h" #include "ruleset.h" +#include "atomic.h" /* forward definitions */ static rsRetVal dfltErrLogger(int, uchar *errMsg); @@ -139,6 +140,12 @@ rsrtInit(char **ppErrObj, obj_if_t *pObjIF) CHKiRet(objClassInit(NULL)); /* *THIS* *MUST* always be the first class initilizer being called! */ CHKiRet(objGetObjInterface(pObjIF)); /* this provides the root pointer for all other queries */ +#ifndef HAVE_ATOMIC_BUILTINS +#ifdef HAVE_SEMAPHORE_H + CHKiRet(atomicSemInit()); +#endif /* HAVE_SEMAPHORE_H */ +#endif /* !defined(HAVE_ATOMIC_BUILTINS) */ + /* initialize core classes. We must be very careful with the order of events. Some * classes use others and if we do not initialize them in the right order, we may end * up with an invalid call. The most important thing that can happen is that an error @@ -215,6 +222,13 @@ rsrtExit(void) glblClassExit(); rulesetClassExit(); ruleClassExit(); + +#ifndef HAVE_ATOMIC_BUILTINS +#ifdef HAVE_SEMAPHORE_H + atomicSemExit(); +#endif /* HAVE_SEMAPHORE_H */ +#endif /* !defined(HAVE_ATOMIC_BUILTINS) */ + objClassExit(); /* *THIS* *MUST/SHOULD?* always be the first class initilizer being called (except debug)! */ } diff --git a/runtime/rsyslog.h b/runtime/rsyslog.h index 27bea6bc..8979893a 100644 --- a/runtime/rsyslog.h +++ b/runtime/rsyslog.h @@ -35,7 +35,7 @@ * value to the fixed size of the message object. */ #define CONF_TAG_MAXSIZE 512 /* a value that is deemed far too large for any valid TAG */ -#define CONF_TAG_HOSTNAME 512 /* a value that is deemed far too large for any valid HOSTNAME */ +#define CONF_HOSTNAME_MAXSIZE 512 /* a value that is deemed far too large for any valid HOSTNAME */ #define CONF_RAWMSG_BUFSIZE 101 #define CONF_TAG_BUFSIZE 32 #define CONF_HOSTNAME_BUFSIZE 32 @@ -46,15 +46,6 @@ * # End Config Settings # * * ############################################################# */ -#ifndef NOLARGEFILE -# undef _LARGEFILE_SOURCE -# undef _LARGEFILE64_SOURCE -# undef _FILE_OFFSET_BITS -# define _LARGEFILE_SOURCE -# define _LARGEFILE64_SOURCE -# define _FILE_OFFSET_BITS 64 -#endif - /* portability: not all platforms have these defines, so we * define them here if they are missing. -- rgerhards, 2008-03-04 */ diff --git a/runtime/rule.c b/runtime/rule.c index 182d616a..4c2c9edb 100644 --- a/runtime/rule.c +++ b/runtime/rule.c @@ -164,6 +164,7 @@ shouldProcessThisMessage(rule_t *pRule, msg_t *pMsg, int *bProcessMsg) if(pRule->f_filter_type == FILTER_PRI) { /* skip messages that are incorrect priority */ +dbgprintf("testing filter, f_pmask %d\n", pRule->f_filterData.f_pmask[pMsg->iFacility]); if ( (pRule->f_filterData.f_pmask[pMsg->iFacility] == TABLE_NOPRI) || \ ((pRule->f_filterData.f_pmask[pMsg->iFacility] & (1<<pMsg->iSeverity)) == 0) ) bRet = 0; diff --git a/runtime/srutils.c b/runtime/srutils.c index c403b312..1452c9b7 100644 --- a/runtime/srutils.c +++ b/runtime/srutils.c @@ -166,10 +166,22 @@ uchar *srUtilStrDup(uchar *pOld, size_t len) /* creates a path recursively - * Return 0 on success, -1 otherwise. On failure, errno - * hold the last OS error. - * Param "mode" holds the mode that all non-existing directories - * are to be created with. + * Return 0 on success, -1 otherwise. On failure, errno * hold the last OS error. + * Param "mode" holds the mode that all non-existing directories are to be + * created with. + * Note that we have a potential race inside that code, a race that even exists + * outside of the rsyslog process (if multiple instances run, or other programs + * generate directories): If the directory does not exist, a context switch happens, + * at that moment another process creates it, then our creation on the context + * switch back fails. This actually happened in practice, and depending on the + * configuration it is even likely to happen. We can not solve this situation + * with a mutex, as that works only within out process space. So the solution + * is that we take the optimistic approach, try the creation, and if it fails + * with "already exists" we go back and do one retry of the check/create + * sequence. That should then succeed. If the directory is still not found but + * the creation fails in the similar way, we return an error on that second + * try because otherwise we would potentially run into an endless loop. + * loop. -- rgerhards, 2010-03-25 */ int makeFileParentDirs(uchar *szFile, size_t lenFile, mode_t mode, uid_t uid, gid_t gid, int bFailOnChownFail) @@ -177,6 +189,8 @@ int makeFileParentDirs(uchar *szFile, size_t lenFile, mode_t mode, uchar *p; uchar *pszWork; size_t len; + int err; + int iTry = 0; int bErr = 0; assert(szFile != NULL); @@ -190,8 +204,9 @@ int makeFileParentDirs(uchar *szFile, size_t lenFile, mode_t mode, if(*p == '/') { /* temporarily terminate string, create dir and go on */ *p = '\0'; +again: if(access((char*)pszWork, F_OK)) { - if(mkdir((char*)pszWork, mode) == 0) { + if((err = mkdir((char*)pszWork, mode)) == 0) { if(uid != (uid_t) -1 || gid != (gid_t) -1) { /* we need to set owner/group */ if(chown((char*)pszWork, uid, gid) != 0) @@ -201,8 +216,13 @@ int makeFileParentDirs(uchar *szFile, size_t lenFile, mode_t mode, * to do so. */ } - } else + } else { + if(err == EEXIST && iTry == 0) { + iTry = 1; + goto again; + } bErr = 1; + } if(bErr) { int eSave = errno; free(pszWork); diff --git a/runtime/stream.c b/runtime/stream.c index 36f44003..e8805a40 100644 --- a/runtime/stream.c +++ b/runtime/stream.c @@ -67,7 +67,7 @@ DEFobjStaticHelpers DEFobjCurrIf(zlibw) /* forward definitions */ -static rsRetVal strmFlush(strm_t *pThis); +static rsRetVal strmFlushInternal(strm_t *pThis); static rsRetVal strmWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf); static rsRetVal strmCloseFile(strm_t *pThis); static void *asyncWriterThread(void *pPtr); @@ -163,7 +163,7 @@ doSizeLimitProcessing(strm_t *pThis) ASSERT(pThis->fd != -1); if(pThis->iCurrOffs >= pThis->iSizeLimit) { - /* strmClosefile() destroys the current file name, so we + /* strmCloseFile() destroys the current file name, so we * need to preserve it. */ CHKmalloc(pszCurrFName = ustrdup(pThis->pszCurrFName)); @@ -220,7 +220,7 @@ doPhysOpen(strm_t *pThis) char errStr[1024]; int err = errno; rs_strerror_r(err, errStr, sizeof(errStr)); - dbgoprint((obj_t*) pThis, "open error %d, file '%s': %s\n", errno, pThis->pszCurrFName, errStr); + DBGOPRINT((obj_t*) pThis, "open error %d, file '%s': %s\n", errno, pThis->pszCurrFName, errStr); if(err == ENOENT) ABORT_FINALIZE(RS_RET_FILE_NOT_FOUND); else @@ -278,7 +278,7 @@ static rsRetVal strmOpenFile(strm_t *pThis) pThis->iCurrOffs = offset; } - dbgoprint((obj_t*) pThis, "opened file '%s' for %s as %d\n", pThis->pszCurrFName, + DBGOPRINT((obj_t*) pThis, "opened file '%s' for %s as %d\n", pThis->pszCurrFName, (pThis->tOperationsMode == STREAMMODE_READ) ? "READ" : "WRITE", pThis->fd); finalize_it: @@ -296,8 +296,10 @@ strmWaitAsyncWriterDone(strm_t *pThis) BEGINfunc if(pThis->bAsyncWrite) { /* awake writer thread and make it write out everything */ - pthread_cond_signal(&pThis->notEmpty); - d_pthread_cond_wait(&pThis->isEmpty, &pThis->mut); + while(pThis->iCnt > 0) { + pthread_cond_signal(&pThis->notEmpty); + d_pthread_cond_wait(&pThis->isEmpty, &pThis->mut); + } } ENDfunc } @@ -306,27 +308,33 @@ strmWaitAsyncWriterDone(strm_t *pThis) /* close a strm file * Note that the bDeleteOnClose flag is honored. If it is set, the file will be * deleted after close. This is in support for the qRead thread. + * Note: it is valid to call this function when the physical file is closed. If so, + * strmCloseFile() will still check if there is any unwritten data inside buffers + * (this may be the case) and, if so, will open the file, write the data, and then + * close it again (this is done via strmFlushInternal and friends). */ static rsRetVal strmCloseFile(strm_t *pThis) { DEFiRet; ASSERT(pThis != NULL); - ASSERT(pThis->fd != -1); - dbgoprint((obj_t*) pThis, "file %d closing\n", pThis->fd); + DBGOPRINT((obj_t*) pThis, "file %d(%s) closing\n", pThis->fd, + (pThis->pszFName == NULL) ? "N/A" : (char*)pThis->pszFName); - if(!pThis->bInClose && pThis->tOperationsMode != STREAMMODE_READ) { - pThis->bInClose = 1; + if(pThis->tOperationsMode != STREAMMODE_READ) { + strmFlushInternal(pThis); if(pThis->bAsyncWrite) { - strmFlush(pThis); - } else { strmWaitAsyncWriterDone(pThis); } - pThis->bInClose = 0; } - close(pThis->fd); - pThis->fd = -1; + /* the file may already be closed (or never have opened), so guard + * against this. -- rgerhards, 2010-03-19 + */ + if(pThis->fd != -1) { + close(pThis->fd); + pThis->fd = -1; + } if(pThis->fdDir != -1) { /* close associated directory handle, if it is open */ @@ -441,7 +449,7 @@ strmHandleEOF(strm_t *pThis) case STREAMTYPE_FILE_CIRCULAR: /* we have multiple files and need to switch to the next one */ /* TODO: think about emulating EOF in this case (not yet needed) */ - dbgoprint((obj_t*) pThis, "file %d EOF\n", pThis->fd); + DBGOPRINT((obj_t*) pThis, "file %d EOF\n", pThis->fd); CHKiRet(strmNextFile(pThis)); break; case STREAMTYPE_FILE_MONITOR: @@ -473,7 +481,7 @@ strmReadBuf(strm_t *pThis) */ CHKiRet(strmOpenFile(pThis)); iLenRead = read(pThis->fd, pThis->pIOBuf, pThis->sIOBufSize); - dbgoprint((obj_t*) pThis, "file %d read %ld bytes\n", pThis->fd, iLenRead); + DBGOPRINT((obj_t*) pThis, "file %d read %ld bytes\n", pThis->fd, iLenRead); if(iLenRead == 0) { CHKiRet(strmHandleEOF(pThis)); } else if(iLenRead < 0) @@ -505,7 +513,7 @@ static rsRetVal strmReadChar(strm_t *pThis, uchar *pC) ASSERT(pThis != NULL); ASSERT(pC != NULL); - /* DEV debug only: dbgoprint((obj_t*) pThis, "strmRead index %d, max %d\n", pThis->iBufPtr, pThis->iBufPtrMax); */ + /* DEV debug only: DBGOPRINT((obj_t*) pThis, "strmRead index %d, max %d\n", pThis->iBufPtr, pThis->iBufPtrMax); */ if(pThis->iUngetC != -1) { /* do we have an "unread" char that we need to provide? */ *pC = pThis->iUngetC; ++pThis->iCurrOffs; /* one more octet read */ @@ -617,11 +625,11 @@ static rsRetVal strmConstructFinalize(strm_t *pThis) * to make sure we can write out everything with a SINGLE api call! * We add another 128 bytes to take care of the gzip header and "all eventualities". */ - CHKmalloc(pThis->pZipBuf = (Bytef*) malloc(sizeof(uchar) * pThis->sIOBufSize + 128)); + CHKmalloc(pThis->pZipBuf = (Bytef*) malloc(sizeof(uchar) * (pThis->sIOBufSize + 128))); } } - /* if we are aset to sync, we must obtain a file handle to the directory for fsync() purposes */ + /* if we are set to sync, we must obtain a file handle to the directory for fsync() purposes */ if(pThis->bSync && !pThis->bIsTTY) { pThis->fdDir = open((char*)pThis->pszDir, O_RDONLY | O_CLOEXEC | O_NOCTTY); if(pThis->fdDir == -1) { @@ -633,6 +641,9 @@ static rsRetVal strmConstructFinalize(strm_t *pThis) } } + DBGPRINTF("file stream %s params: flush interval %d, async write %d\n", + (pThis->pszFName == NULL) ? "N/A" : (char*)pThis->pszFName, + pThis->iFlushInterval, pThis->bAsyncWrite); /* if we have a flush interval, we need to do async writes in any case */ if(pThis->iFlushInterval != 0) { pThis->bAsyncWrite = 1; @@ -685,8 +696,10 @@ CODESTARTobjDestruct(strm) /* Note: mutex will be unlocked in stopWriter! */ d_pthread_mutex_lock(&pThis->mut); - if(pThis->tOperationsMode != STREAMMODE_READ) - strmFlush(pThis); + /* strmClose() will handle read-only files as well as need to open + * files that have unwritten buffers. -- rgerhards, 2010-03-09 + */ + strmCloseFile(pThis); if(pThis->bAsyncWrite) { stopWriter(pThis); @@ -705,14 +718,11 @@ CODESTARTobjDestruct(strm) * IMPORTANT: we MUST free this only AFTER the ansyncWriter has been stopped, else * we get random errors... */ - if(pThis->fd != -1) - strmCloseFile(pThis); - free(pThis->pszDir); free(pThis->pZipBuf); free(pThis->pszCurrFName); free(pThis->pszFName); - + pThis->bStopWriter = 2; /* RG: use as flag for destruction */ ENDobjDestruct(strm) @@ -732,7 +742,7 @@ static rsRetVal strmCheckNextOutputFile(strm_t *pThis) strmWaitAsyncWriterDone(pThis); if(pThis->iCurrOffs >= pThis->iMaxFileSize) { - dbgoprint((obj_t*) pThis, "max file size %ld reached for %d, now %ld - starting new file\n", + DBGOPRINT((obj_t*) pThis, "max file size %ld reached for %d, now %ld - starting new file\n", (long) pThis->iMaxFileSize, pThis->fd, (long) pThis->iCurrOffs); CHKiRet(strmNextFile(pThis)); } @@ -790,6 +800,7 @@ doWriteCall(strm_t *pThis, uchar *pBuf, size_t *pLenBuf) if(iWritten < 0) { char errStr[1024]; int err = errno; + iWritten = 0; /* we have written NO bytes! */ rs_strerror_r(err, errStr, sizeof(errStr)); DBGPRINTF("log file (%d) write error %d: %s\n", pThis->fd, err, errStr); if(err == EINTR) { @@ -811,7 +822,7 @@ doWriteCall(strm_t *pThis, uchar *pBuf, size_t *pLenBuf) pWriteBuf += iWritten; } while(lenBuf > 0); /* Warning: do..while()! */ - dbgoprint((obj_t*) pThis, "file %d write wrote %d bytes\n", pThis->fd, (int) iWritten); + DBGOPRINT((obj_t*) pThis, "file %d write wrote %d bytes\n", pThis->fd, (int) iWritten); finalize_it: *pLenBuf = iTotalWritten; @@ -855,7 +866,8 @@ doAsyncWriteInternal(strm_t *pThis, size_t lenBuf) DEFiRet; ISOBJ_TYPE_assert(pThis, strm); - while(pThis->iCnt >= STREAM_ASYNC_NUMBUFS) + /* the -1 below is important, because we need one buffer for the main thread! */ + while(pThis->iCnt >= STREAM_ASYNC_NUMBUFS - 1) d_pthread_cond_wait(&pThis->notFull, &pThis->mut); pThis->asyncBuf[pThis->iEnq % STREAM_ASYNC_NUMBUFS].lenBuf = lenBuf; @@ -880,13 +892,22 @@ strmSchedWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) ASSERT(pThis != NULL); + /* we need to reset the buffer pointer BEFORE calling the actual write + * function. Otherwise, in circular mode, the write function will + * potentially close the file, then close will flush and as the + * buffer pointer is nonzero, will re-call into this code here. In + * the end result, we than have a problem (and things are screwed + * up). So we reset the buffer pointer first, and all this can + * not happen. It is safe to do so, because that pointer is NOT + * used inside the write functions. -- rgerhads, 2010-03-10 + */ + pThis->iBufPtr = 0; /* we are at the begin of a new buffer */ if(pThis->bAsyncWrite) { CHKiRet(doAsyncWriteInternal(pThis, lenBuf)); } else { CHKiRet(doWriteInternal(pThis, pBuf, lenBuf)); } - pThis->iBufPtr = 0; /* we are at the begin of a new buffer */ finalize_it: RETiRet; @@ -911,10 +932,11 @@ asyncWriterThread(void *pPtr) if(prctl(PR_SET_NAME, "rs:asyn strmwr", 0, 0, 0) != 0) { DBGPRINTF("prctl failed, not setting thread name for '%s'\n", "stream writer"); } -#endif +# endif while(1) { /* loop broken inside */ d_pthread_mutex_lock(&pThis->mut); +dbgprintf("XXX: asyncWriterThread iterating %s\n", pThis->pszFName); while(pThis->iCnt == 0) { if(pThis->bStopWriter) { pthread_cond_broadcast(&pThis->isEmpty); @@ -923,13 +945,14 @@ asyncWriterThread(void *pPtr) } if(bTimedOut && pThis->iBufPtr > 0) { /* if we timed out, we need to flush pending data */ - strmFlush(pThis); + strmFlushInternal(pThis); bTimedOut = 0; continue; /* now we should have data */ } bTimedOut = 0; - timeoutComp(&t, pThis->iFlushInterval * 2000); /* *1000 millisconds */ + timeoutComp(&t, pThis->iFlushInterval * 2000); /* *1000 millisconds */ // TODO: check the 2000?!? if(pThis->bDoTimedWait) { +dbgprintf("asyncWriter thread going to timeout sleep\n"); if(pthread_cond_timedwait(&pThis->notEmpty, &pThis->mut, &t) != 0) { int err = errno; if(err == ETIMEDOUT) { @@ -943,13 +966,16 @@ asyncWriterThread(void *pPtr) } } } else { +dbgprintf("asyncWriter thread going to eternal sleep\n"); d_pthread_cond_wait(&pThis->notEmpty, &pThis->mut); } +dbgprintf("asyncWriter woke up\n"); } bTimedOut = 0; /* we may have timed out, but there *is* work to do... */ iDeq = pThis->iDeq++ % STREAM_ASYNC_NUMBUFS; +dbgprintf("asyncWriter writes data\n"); doWriteInternal(pThis, pThis->asyncBuf[iDeq].pBuf, pThis->asyncBuf[iDeq].lenBuf); // TODO: error check????? 2009-07-06 @@ -1058,10 +1084,6 @@ finalize_it: * add a config switch so that the user can decide the risk he is ready * to take, but so far this is not yet implemented (not even requested ;)). * rgerhards, 2009-06-04 - * For the time being, we take a very conservative approach and do not run this - * method multithreaded. This is done in an effort to solve a segfault condition - * that seems to be related to the zip code. -- rgerhards, 2009-09-22 - * TODO: make multithreaded again! */ static rsRetVal doZipWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) @@ -1120,12 +1142,14 @@ finalize_it: * rgerhards, 2008-01-10 */ static rsRetVal -strmFlush(strm_t *pThis) +strmFlushInternal(strm_t *pThis) { DEFiRet; ASSERT(pThis != NULL); - dbgoprint((obj_t*) pThis, "file %d flush, buflen %ld\n", pThis->fd, (long) pThis->iBufPtr); + DBGOPRINT((obj_t*) pThis, "file %d(%s) flush, buflen %ld%s\n", pThis->fd, + (pThis->pszFName == NULL) ? "N/A" : (char*)pThis->pszFName, + (long) pThis->iBufPtr, (pThis->iBufPtr == 0) ? " (no need to flush)" : ""); if(pThis->tOperationsMode != STREAMMODE_READ && pThis->iBufPtr > 0) { iRet = strmSchedWrite(pThis, pThis->pIOBuf, pThis->iBufPtr); @@ -1135,6 +1159,31 @@ strmFlush(strm_t *pThis) } +/* flush stream output buffer to persistent storage. This can be called at any time + * and is automatically called when the output buffer is full. This function is for + * use by EXTERNAL callers. Do NOT use it internally. It locks the async writer + * mutex if ther is need to do so. + * rgerhards, 2010-03-18 + */ +static rsRetVal +strmFlush(strm_t *pThis) +{ + DEFiRet; + + ASSERT(pThis != NULL); + + if(pThis->bAsyncWrite) + d_pthread_mutex_lock(&pThis->mut); + CHKiRet(strmFlushInternal(pThis)); + +finalize_it: + if(pThis->bAsyncWrite) + d_pthread_mutex_unlock(&pThis->mut); + + RETiRet; +} + + /* seek a stream to a specific location. Pending writes are flushed, read data * is invalidated. * rgerhards, 2008-01-12 @@ -1148,9 +1197,9 @@ static rsRetVal strmSeek(strm_t *pThis, off_t offs) if(pThis->fd == -1) strmOpenFile(pThis); else - strmFlush(pThis); + strmFlushInternal(pThis); int i; - dbgoprint((obj_t*) pThis, "file %d seek, pos %ld\n", pThis->fd, (long) offs); + DBGOPRINT((obj_t*) pThis, "file %d seek, pos %ld\n", pThis->fd, (long) offs); i = lseek(pThis->fd, offs, SEEK_SET); // TODO: check error! pThis->iCurrOffs = offs; /* we are now at *this* offset */ pThis->iBufPtr = 0; /* buffer invalidated */ @@ -1189,7 +1238,7 @@ static rsRetVal strmWriteChar(strm_t *pThis, uchar c) /* if the buffer is full, we need to flush before we can write */ if(pThis->iBufPtr == pThis->sIOBufSize) { - CHKiRet(strmFlush(pThis)); + CHKiRet(strmFlushInternal(pThis)); } /* we now always have space for one character, so we simply copy it */ *(pThis->pIOBuf + pThis->iBufPtr) = c; @@ -1233,6 +1282,11 @@ finalize_it: * caller-provided buffer is larger than our one. So instead of optimizing a case * which normally does not exist, we expect some degradation in its case but make us * perform better in the regular cases. -- rgerhards, 2009-07-07 + * Note: the pThis->iBufPtr == pThis->sIOBufSize logic below looks a bit like an + * on-off error. In fact, it is not, because iBufPtr always points to the next + * *free* byte in the buffer. So if it is sIOBufSize - 1, there actually is one + * free byte left. This came up during a code walkthrough and was considered + * worth nothing. -- rgerhards, 2010-03-10 */ static rsRetVal strmWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) @@ -1254,7 +1308,7 @@ strmWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) iOffset = 0; do { if(pThis->iBufPtr == pThis->sIOBufSize) { - CHKiRet(strmFlush(pThis)); /* get a new buffer for rest of data */ + CHKiRet(strmFlushInternal(pThis)); /* get a new buffer for rest of data */ } iWrite = pThis->sIOBufSize - pThis->iBufPtr; /* this fits in current buf */ if(iWrite > lenBuf) @@ -1269,7 +1323,7 @@ strmWrite(strm_t *pThis, uchar *pBuf, size_t lenBuf) * write it. This seems more natural than waiting (hours?) for the next message... */ if(pThis->iBufPtr == pThis->sIOBufSize) { - CHKiRet(strmFlush(pThis)); /* get a new buffer for rest of data */ + CHKiRet(strmFlushInternal(pThis)); /* get a new buffer for rest of data */ } finalize_it: @@ -1357,8 +1411,7 @@ strmSetDir(strm_t *pThis, uchar *pszDir, size_t iLenDir) if(iLenDir < 1) ABORT_FINALIZE(RS_RET_FILE_PREFIX_MISSING); - if((pThis->pszDir = malloc(sizeof(uchar) * iLenDir + 1)) == NULL) - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + CHKmalloc(pThis->pszDir = malloc(sizeof(uchar) * iLenDir + 1)); memcpy(pThis->pszDir, pszDir, iLenDir + 1); /* always think about the \0! */ pThis->lenDir = iLenDir; @@ -1429,7 +1482,7 @@ static rsRetVal strmSerialize(strm_t *pThis, strm_t *pStrm) ISOBJ_TYPE_assert(pThis, strm); ISOBJ_TYPE_assert(pStrm, strm); - strmFlush(pThis); + strmFlushInternal(pThis); CHKiRet(obj.BeginSerialize(pStrm, (obj_t*) pThis)); objSerializeSCALAR(pStrm, iCurrFNum, INT); diff --git a/runtime/stream.h b/runtime/stream.h index 89175b0f..369d5a0f 100644 --- a/runtime/stream.h +++ b/runtime/stream.h @@ -119,7 +119,6 @@ typedef struct strm_s { size_t iBufPtr; /* pointer into current buffer */ int iUngetC; /* char set via UngetChar() call or -1 if none set */ bool bInRecord; /* if 1, indicates that we are currently writing a not-yet complete record */ - bool bInClose; /* used to break "deadly close loops", tells us we are already inside a close */ int iZipLevel; /* zip level (0..9). If 0, zip is completely disabled */ Bytef *pZipBuf; /* support for async flush procesing */ diff --git a/runtime/vm.c b/runtime/vm.c index a1d992c3..0ed174d1 100644 --- a/runtime/vm.c +++ b/runtime/vm.c @@ -91,6 +91,9 @@ rsfrAddFunction(uchar *szName, prsf_t rsf) funcRegRoot = pEntry; finalize_it: + if(iRet != RS_RET_OK && iRet != RS_RET_DUP_FUNC_NAME) + free(pEntry); + RETiRet; } diff --git a/solaris/README b/solaris/README new file mode 100644 index 00000000..3f88431d --- /dev/null +++ b/solaris/README @@ -0,0 +1,38 @@ +Notes for Solaris + +Rsyslog will be fully supported on Solaris in the future. To build it, the GNU build +tools (and most of the GNU environment) is needed. This software can be +found at the excellent http://www.blastwave.org site. + +PREQUISITES +It is strongly recommended to use GCC4 with support for +atomic instructions (if available for the platform). While rsyslog can +be built without atomic instructin support (and will work well then), +it then falls back to POSIX semaphores, which require much more CPU +time than atomic instructions. Note that even on intel platforms the +(current, as of 2010-03-25) blastwave gcc4 version targets too-old +processors by default. To change that, use "-imarch=I686" in your +CFLAGS. + +CONFIGURE OPTIONS +A number of GNU tools are renamed g* so that they not conflict with +the native Solaris tools. As we need the GNU replacements, this +must be specified on the ./configure line. +Also, we must tell the linker where to find the glibc library when +building the plugins. This is done via the LDFLAGS variable as +shown below (based on the good information availabe at +http://prefetch.net/articles/linkers.badldlibrary.html + +The working sample configure sequence I use is: + +export LDFLAGS="-R/opt/csw/gcc4/lib" +./configure AR=gar ...other options... + +As a "quick and dirty" fix, one may set the following library +path before executing rsyslog (may be useful to avoid recompile): + +export LD_LIBRARY_PATH=/opt/csw/gcc4/lib + +NOT YET SUPPORTED +* local log socket +* kernel log diff --git a/solaris/cddllicense.txt b/solaris/cddllicense.txt new file mode 100644 index 00000000..a10bba27 --- /dev/null +++ b/solaris/cddllicense.txt @@ -0,0 +1,242 @@ + COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL)
+ Version 1.0
+
+
+1. Definitions.
+
+ 1.1. “Contributor” means each individual or entity that creates or contributes to the creation of Modifications.
+
+ 1.2. “Contributor Version” means the combination of the Original Software, prior Modifications used by a Contributor
+ (if any), and the Modifications made by that particular Contributor.
+
+ 1.3. “Covered Software” means (a) the Original Software, or (b) Modifications, or (c) the combination
+ of files containing Original Software with files containing Modifications, in each case including portions thereof.
+
+ 1.4. “Executable” means the Covered Software in any form other than Source Code.
+
+ 1.5. “Initial Developer” means the individual or entity that first makes Original Software available under this License.
+
+ 1.6. “Larger Work” means a work which combines Covered Software or portions thereof with code not governed by the terms
+ of this License.
+
+ 1.7. “License” means this document.
+
+ 1.8. “Licensable” means having the right to grant, to the maximum extent possible, whether at the time of the
+ initial grant or subsequently acquired, any and all of the rights conveyed herein.
+
+ 1.9. “Modifications” means the Source Code and Executable form of any of the following:
+ A. Any file that results from an addition to, deletion from or modification of the contents of a file
+ containing Original Software or previous Modifications;
+ B. Any new file that contains any part of the Original Software or previous Modification; or
+ C. Any new file that is contributed or otherwise made available under the terms of this License.
+
+ 1.10. “Original Software” means the Source Code and Executable form of computer software code that is originally
+ released under this License.
+
+ 1.11. “Patent Claims” means any patent claim(s), now owned or hereafter acquired, including without limitation,
+ method, process, and apparatus claims, in any patent Licensable by grantor.
+
+ 1.12. “Source Code” means (a) the common form of computer software code in which modifications are made and
+ (b) associated documentation included in or with such code.
+
+ 1.13. “You” (or “Your”) means an individual or a legal entity exercising rights under, and complying with all of
+ the terms of, this License. For legal entities, “You” includes any entity which controls, is controlled by, or is
+ under common control with You. For purposes of this definition, “control” means (a) the power, direct or indirect,
+ to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more
+ than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity.
+
+2. License Grants.
+
+ 2.1. The Initial Developer Grant.
+ Conditioned upon Your compliance with Section 3.1 below and subject to third party intellectual property claims,
+ the Initial Developer hereby grants You a world-wide, royalty-free, non-exclusive license:
+
+ (a) under intellectual property rights (other than patent or trademark) Licensable by Initial Developer,
+ to use, reproduce, modify, display, perform, sublicense and distribute the Original Software (or portions
+ thereof), with or without Modifications, and/or as part of a Larger Work; and
+
+ (b) under Patent Claims infringed by the making, using or selling of Original Software, to make, have made,
+ use, practice, sell, and offer for sale, and/or otherwise dispose of the Original Software (or portions thereof).
+
+ (c) The licenses granted in Sections 2.1(a) and (b) are effective on the date Initial Developer first
+ distributes or otherwise makes the Original Software available to a third party under the terms of this License.
+
+ (d) Notwithstanding Section 2.1(b) above, no patent license is granted: (1) for code that You delete from
+ the Original Software, or (2) for infringements caused by: (i) the modification of the Original Software,
+ or (ii) the combination of the Original Software with other software or devices.
+
+ 2.2. Contributor Grant.
+ Conditioned upon Your compliance with Section 3.1 below and subject to third party intellectual property claims,
+ each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license:
+
+ (a) under intellectual property rights (other than patent or trademark) Licensable by Contributor to use,
+ reproduce, modify, display, perform, sublicense and distribute the Modifications created by such Contributor
+ (or portions thereof), either on an unmodified basis, with other Modifications, as Covered Software and/or
+ as part of a Larger Work; and
+
+ (b) under Patent Claims infringed by the making, using, or selling of Modifications made by that Contributor
+ either alone and/or in combination with its Contributor Version (or portions of such combination), to make,
+ use, sell, offer for sale, have made, and/or otherwise dispose of: (1) Modifications made by that Contributor
+ (or portions thereof); and (2) the combination of Modifications made by that Contributor with its Contributor
+ Version (or portions of such combination).
+
+ (c) The licenses granted in Sections 2.2(a) and 2.2(b) are effective on the date Contributor first distributes
+ or otherwise makes the Modifications available to a third party.
+
+ (d) Notwithstanding Section 2.2(b) above, no patent license is granted: (1) for any code that Contributor has
+ deleted from the Contributor Version; (2) for infringements caused by: (i) third party modifications of
+ Contributor Version, or (ii) the combination of Modifications made by that Contributor with other software
+ (except as part of the Contributor Version) or other devices; or (3) under Patent Claims infringed by Covered
+ Software in the absence of Modifications made by that Contributor.
+
+3. Distribution Obligations.
+
+ 3.1. Availability of Source Code.
+ Any Covered Software that You distribute or otherwise make available in Executable form must also be made available
+ in Source Code form and that Source Code form must be distributed only under the terms of this License. You must
+ include a copy of this License with every copy of the Source Code form of the Covered Software You distribute or
+ otherwise make available. You must inform recipients of any such Covered Software in Executable form as to how they
+ can obtain such Covered Software in Source Code form in a reasonable manner on or through a medium customarily used
+ for software exchange.
+
+ 3.2. Modifications.
+ The Modifications that You create or to which You contribute are governed by the terms of this License. You
+ represent that You believe Your Modifications are Your original creation(s) and/or You have sufficient rights to
+ grant the rights conveyed by this License.
+
+ 3.3. Required Notices.
+ You must include a notice in each of Your Modifications that identifies You as the Contributor of the Modification.
+ You may not remove or alter any copyright, patent or trademark notices contained within the Covered Software, or
+ any notices of licensing or any descriptive text giving attribution to any Contributor or the Initial Developer.
+
+ 3.4. Application of Additional Terms.
+ You may not offer or impose any terms on any Covered Software in Source Code form that alters or restricts the
+ applicable version of this License or the recipients’ rights hereunder. You may choose to offer, and to charge a
+ fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software.
+ However, you may do so only on Your own behalf, and not on behalf of the Initial Developer or any Contributor.
+ You must make it absolutely clear that any such warranty, support, indemnity or liability obligation is offered
+ by You alone, and You hereby agree to indemnify the Initial Developer and every Contributor for any liability
+ incurred by the Initial Developer or such Contributor as a result of warranty, support, indemnity or liability
+ terms You offer.
+
+ 3.5. Distribution of Executable Versions.
+ You may distribute the Executable form of the Covered Software under the terms of this License or under the terms
+ of a license of Your choice, which may contain terms different from this License, provided that You are in compliance
+ with the terms of this License and that the license for the Executable form does not attempt to limit or alter
+ the recipient’s rights in the Source Code form from the rights set forth in this License. If You distribute the
+ Covered Software in Executable form under a different license, You must make it absolutely clear that any terms
+ which differ from this License are offered by You alone, not by the Initial Developer or Contributor. You hereby
+ agree to indemnify the Initial Developer and every Contributor for any liability incurred by the Initial Developer
+ or such Contributor as a result of any such terms You offer.
+
+ 3.6. Larger Works.
+ You may create a Larger Work by combining Covered Software with other code not governed by the terms of this License
+ and distribute the Larger Work as a single product. In such a case, You must make sure the requirements of this
+ License are fulfilled for the Covered Software.
+
+4. Versions of the License.
+
+ 4.1. New Versions.
+ Sun Microsystems, Inc. is the initial license steward and may publish revised and/or new versions of this License
+ from time to time. Each version will be given a distinguishing version number. Except as provided in Section 4.3,
+ no one other than the license steward has the right to modify this License.
+
+ 4.2. Effect of New Versions.
+ You may always continue to use, distribute or otherwise make the Covered Software available under the terms of
+ the version of the License under which You originally received the Covered Software. If the Initial Developer
+ includes a notice in the Original Software prohibiting it from being distributed or otherwise made available under
+ any subsequent version of the License, You must distribute and make the Covered Software available under the terms
+ of the version of the License under which You originally received the Covered Software. Otherwise, You may also
+ choose to use, distribute or otherwise make the Covered Software available under the terms of any subsequent version
+ of the License published by the license steward.
+
+ 4.3. Modified Versions.
+ When You are an Initial Developer and You want to create a new license for Your Original Software, You may create
+ and use a modified version of this License if You: (a) rename the license and remove any references to the name of
+ the license steward (except to note that the license differs from this License); and (b) otherwise make it clear that
+ the license contains terms which differ from this License.
+
+5. DISCLAIMER OF WARRANTY.
+
+COVERED SOFTWARE IS PROVIDED UNDER THIS LICENSE ON AN “AS IS” BASIS, WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR
+IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES THAT THE COVERED SOFTWARE IS FREE OF DEFECTS, MERCHANTABLE, FIT FOR A
+PARTICULAR PURPOSE OR NON-INFRINGING. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE COVERED SOFTWARE IS WITH YOU.
+SHOULD ANY COVERED SOFTWARE PROVE DEFECTIVE IN ANY RESPECT, YOU (NOT THE INITIAL DEVELOPER OR ANY OTHER CONTRIBUTOR) ASSUME THE
+COST OF ANY NECESSARY SERVICING, REPAIR OR CORRECTION. THIS DISCLAIMER OF WARRANTY CONSTITUTES AN ESSENTIAL PART OF THIS LICENSE.
+NO USE OF ANY COVERED SOFTWARE IS AUTHORIZED HEREUNDER EXCEPT UNDER THIS DISCLAIMER.
+
+6. TERMINATION.
+
+ 6.1. This License and the rights granted hereunder will terminate automatically if You fail to comply with terms
+ herein and fail to cure such breach within 30 days of becoming aware of the breach. Provisions which, by their
+ nature, must remain in effect beyond the termination of this License shall survive.
+
+ 6.2. If You assert a patent infringement claim (excluding declaratory judgment actions) against Initial Developer
+ or a Contributor (the Initial Developer or Contributor against whom You assert such claim is referred to as
+ “Participant”) alleging that the Participant Software (meaning the Contributor Version where the Participant
+ is a Contributor or the Original Software where the Participant is the Initial Developer) directly or
+ indirectly infringes any patent, then any and all rights granted directly or indirectly to You by such Participant,
+ the Initial Developer (if the Initial Developer is not the Participant) and all Contributors under Sections 2.1
+ and/or 2.2 of this License shall, upon 60 days notice from Participant terminate prospectively and automatically
+ at the expiration of such 60 day notice period, unless if within such 60 day period You withdraw Your claim with
+ respect to the Participant Software against such Participant either unilaterally or pursuant to a written agreement
+ with Participant.
+
+ 6.3. If You assert a patent infringement claim against Participant alleging that the Participant Software directly
+ or indirectly infringes any patent where such claim is resolved (such as by license or settlement) prior to the
+ initiation of patent infringement litigation, then the reasonable value of the licenses granted by such Participant
+ under Sections 2.1 or 2.2 shall be taken into account in determining the amount or value of any payment or license.
+
+ 6.4. In the event of termination under Sections 6.1 or 6.2 above, all end user licenses that have been validly granted
+ by You or any distributor hereunder prior to termination (excluding licenses granted to You by any distributor)
+ shall survive termination.
+
+7. LIMITATION OF LIABILITY.
+
+UNDER NO CIRCUMSTANCES AND UNDER NO LEGAL THEORY, WHETHER TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE, SHALL YOU, THE
+INITIAL DEVELOPER, ANY OTHER CONTRIBUTOR, OR ANY DISTRIBUTOR OF COVERED SOFTWARE, OR ANY SUPPLIER OF ANY OF SUCH PARTIES, BE LIABLE TO
+ANY PERSON FOR ANY INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES OF ANY CHARACTER INCLUDING, WITHOUT LIMITATION, DAMAGES FOR
+LOSS OF GOODWILL, WORK STOPPAGE, COMPUTER FAILURE OR MALFUNCTION, OR ANY AND ALL OTHER COMMERCIAL DAMAGES OR LOSSES, EVEN IF SUCH PARTY
+SHALL HAVE BEEN INFORMED OF THE POSSIBILITY OF SUCH DAMAGES. THIS LIMITATION OF LIABILITY SHALL NOT APPLY TO LIABILITY FOR DEATH OR PERSONAL
+INJURY RESULTING FROM SUCH PARTY’S NEGLIGENCE TO THE EXTENT APPLICABLE LAW PROHIBITS SUCH LIMITATION. SOME JURISDICTIONS DO NOT ALLOW THE
+EXCLUSION OR LIMITATION OF INCIDENTAL OR CONSEQUENTIAL DAMAGES, SO THIS EXCLUSION AND LIMITATION MAY NOT APPLY TO YOU.
+
+8. U.S. GOVERNMENT END USERS.
+
+The Covered Software is a “commercial item,” as that term is defined in 48 C.F.R. 2.101 (Oct. 1995), consisting of
+“commercial computer software” (as that term is defined at 48 C.F.R. § 252.227-7014(a)(1)) and “commercial computer software documentation”
+as such terms are used in 48 C.F.R. 12.212 Sept. 1995). Consistent with 48 C.F.R. 12.212 and 48 C.F.R. 227.7202-1 through 227.7202-4 (June 1995),
+all U.S. Government End Users acquire Covered Software with only those rights set forth herein. This U.S. Government Rights clause is in lieu of,
+and supersedes, any other FAR, DFAR, or other clause or provision that addresses Government rights in computer software under this License.
+
+9. MISCELLANEOUS.
+
+This License represents the complete agreement concerning subject matter hereof. If any provision of this License is held to be
+unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. This License shall be governed
+by the law of the jurisdiction specified in a notice contained within the Original Software (except to the extent applicable law, if any,
+provides otherwise), excluding such jurisdiction’s conflict-of-law provisions. Any litigation relating to this License shall be subject
+to the jurisdiction of the courts located in the jurisdiction and venue specified in a notice contained within the Original Software, with the
+losing party responsible for costs, including, without limitation, court costs and reasonable attorneys’ fees and expenses. The application of
+the United Nations Convention on Contracts for the International Sale of Goods is expressly excluded. Any law or regulation which provides that
+the language of a contract shall be construed against the drafter shall not apply to this License. You agree that You alone are responsible for
+compliance with the United States export administration regulations (and the export control laws and regulation of any other countries) when You use,
+distribute or otherwise make available any Covered Software.
+
+10. RESPONSIBILITY FOR CLAIMS.
+
+As between Initial Developer and the Contributors, each party is responsible for claims and damages arising, directly or indirectly, out of
+its utilization of rights under this License and You agree to work with Initial Developer and Contributors to distribute such responsibility on
+an equitable basis. Nothing herein is intended or shall be deemed to constitute any admission of liability. + +-------- + +NOTICE PURSUANT TO SECTION 9 OF THE COMMON DEVELOPMENT AND DISTRIBUTION +LICENSE (CDDL) + +The OpenSolaris code released under the CDDL shall be governed by the laws +of the State of California (excluding conflict-of-law provisions). Any +litigation relating to this License shall be subject to the jurisdiction of +the Federal Courts of the Northern District of California and the state +courts of the State of California, with venue lying in Santa Clara County, +California. + diff --git a/tcps_sess.c b/tcps_sess.c index 8d307380..dec14b5a 100644 --- a/tcps_sess.c +++ b/tcps_sess.c @@ -59,8 +59,6 @@ DEFobjCurrIf(datetime) static int iMaxLine; /* maximum size of a single message */ -static int iNbrTimeUsed = 0; /* how often has previous time been used so far? */ - /* forward definitions */ static rsRetVal Close(tcps_sess_t *pThis); @@ -475,7 +473,6 @@ DataRcvd(tcps_sess_t *pThis, char *pData, size_t iLen) /* We now copy the message to the session buffer. */ pEnd = pData + iLen; /* this is one off, which is intensional */ - iNbrTimeUsed = 0; /* full time query */ while(pData < pEnd) { CHKiRet(processDataRcvd(pThis, *pData++, &stTime, ttGenTime, &multiSub)); } @@ -427,10 +427,7 @@ static int do_Constant(unsigned char **pp, struct template *pTpl) } if((pTpe = tpeConstruct(pTpl)) == NULL) { - /* OK, we are out of luck. Let's invalidate the - * entry and that's it. - */ - pTpe->eEntryType = UNDEFINED; + rsCStrDestruct(&pStrB); return 1; } pTpe->eEntryType = CONSTANT; @@ -490,6 +487,8 @@ static void doOptions(unsigned char **pp, struct templateEntry *pTpe) pTpe->data.field.eDateFormat = tplFmtPgSQLDate; } else if(!strcmp((char*)Buf, "date-rfc3164")) { pTpe->data.field.eDateFormat = tplFmtRFC3164Date; + } else if(!strcmp((char*)Buf, "date-rfc3164-buggyday")) { + pTpe->data.field.eDateFormat = tplFmtRFC3164BuggyDate; } else if(!strcmp((char*)Buf, "date-rfc3339")) { pTpe->data.field.eDateFormat = tplFmtRFC3339Date; } else if(!strcmp((char*)Buf, "date-subseconds")) { @@ -566,8 +565,11 @@ static int do_Parameter(unsigned char **pp, struct template *pTpl) /* got the name */ cstrFinalize(pStrB); - if(propNameToID(pStrB, &pTpe->data.field.propid) != RS_RET_OK) + if(propNameToID(pStrB, &pTpe->data.field.propid) != RS_RET_OK) { + cstrDestruct(&pStrB); return 1; + } + cstrDestruct(&pStrB); /* no longer needed, now use ID */ /* Check frompos, if it has an R, then topos should be a regex */ if(*p == ':') { @@ -48,7 +48,7 @@ struct template { enum EntryTypes { UNDEFINED = 0, CONSTANT = 1, FIELD = 2 }; enum tplFormatTypes { tplFmtDefault = 0, tplFmtMySQLDate = 1, tplFmtRFC3164Date = 2, tplFmtRFC3339Date = 3, tplFmtPgSQLDate = 4, - tplFmtSecFrac = 5}; + tplFmtSecFrac = 5, tplFmtRFC3164BuggyDate = 6}; enum tplFormatCaseConvTypes { tplCaseConvNo = 0, tplCaseConvUpper = 1, tplCaseConvLower = 2 }; #include "msg.h" diff --git a/tests/Makefile.am b/tests/Makefile.am index 6091896b..051a9403 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -1,6 +1,6 @@ if ENABLE_TESTBENCH TESTRUNS = rt_init rscript -check_PROGRAMS = $(TESTRUNS) ourtail nettester tcpflood chkseq +check_PROGRAMS = $(TESTRUNS) ourtail nettester tcpflood chkseq msleep randomgen TESTS = $(TESTRUNS) cfg.sh \ validation-run.sh \ imtcp-multiport.sh \ @@ -10,6 +10,22 @@ TESTS = $(TESTRUNS) cfg.sh \ rsf_getenv.sh \ sndrcv.sh \ sndrcv_gzip.sh \ + asynwr_simple.sh \ + asynwr_timeout.sh \ + asynwr_small.sh \ + asynwr_tinybuf.sh \ + wr_large_async.sh \ + wr_large_sync.sh \ + asynwr_deadlock.sh \ + asynwr_deadlock2.sh \ + asynwr_deadlock4.sh \ + gzipwr_large.sh \ + gzipwr_large_dynfile.sh \ + dynfile_invld_async.sh \ + dynfile_invld_sync.sh \ + dynfile_invalid2.sh \ + complex1.sh \ + random.sh \ queue-persist.sh \ pipeaction.sh execonlyonce.sh \ @@ -79,7 +95,9 @@ EXTRA_DIST= 1.rstest 2.rstest 3.rstest err1.rstest \ testsuites/1.parse1 \ testsuites/2.parse1 \ testsuites/3.parse1 \ + testsuites/snare.parse1 \ testsuites/oversizeTag-1.parse1 \ + testsuites/weird.parse1 \ testsuites/date1.parse1 \ testsuites/date2.parse1 \ testsuites/date3.parse1 \ @@ -90,6 +108,17 @@ EXTRA_DIST= 1.rstest 2.rstest 3.rstest err1.rstest \ testsuites/rfc5424-2.parse1 \ testsuites/rfc5424-3.parse1 \ testsuites/rfc5424-4.parse1 \ + testsuites/parse3.conf \ + testsuites/reallife.parse3 \ + testsuites/parse-nodate.conf \ + testsuites/parse_invld_regex.conf \ + testsuites/samples.parse_invld_regex \ + testsuites/parse-3164-buggyday.conf \ + testsuites/samples.parse-3164-buggyday \ + testsuites/snare_ccoff_udp.conf \ + testsuites/samples.snare_ccoff_udp \ + testsuites/snare_ccoff_udp2.conf \ + testsuites/samples.snare_ccoff_udp2 \ testsuites/omod-if-array.conf \ testsuites/1.omod-if-array \ testsuites/1.field1 \ @@ -131,6 +160,38 @@ EXTRA_DIST= 1.rstest 2.rstest 3.rstest err1.rstest \ testsuites/sndrcv_gzip_rcvr.conf \ pipeaction.sh \ testsuites/pipeaction.conf \ + asynwr_simple.sh \ + testsuites/asynwr_simple.conf \ + asynwr_timeout.sh \ + testsuites/asynwr_timeout.conf \ + asynwr_small.sh \ + testsuites/asynwr_small.conf \ + asynwr_tinybuf.sh \ + testsuites/asynwr_tinybuf.conf \ + wr_large_async.sh \ + wr_large_sync.sh \ + wr_large.sh \ + testsuites/wr_large.conf \ + asynwr_deadlock.sh \ + testsuites/asynwr_deadlock.conf \ + asynwr_deadlock2.sh \ + testsuites/asynwr_deadlock2.conf \ + asynwr_deadlock4.sh \ + testsuites/asynwr_deadlock4.conf \ + gzipwr_large.sh \ + testsuites/gzipwr_large.conf \ + gzipwr_large_dynfile.sh \ + testsuites/gzipwr_large_dynfile.conf \ + complex1.sh \ + testsuites/complex1.conf \ + random.sh \ + testsuites/random.conf \ + dynfile_invld_async.sh \ + dynfile_invld_sync.sh \ + dynfile_cachemiss.sh \ + testsuites/dynfile_cachemiss.conf \ + dynfile_invalid2.sh \ + testsuites/dynfile_invalid2.conf \ proprepltest.sh \ testsuites/rfctag.conf \ testsuites/master.rfctag \ @@ -143,11 +204,15 @@ EXTRA_DIST= 1.rstest 2.rstest 3.rstest err1.rstest \ cfg.sh ourtail_SOURCES = ourtail.c +msleep_SOURCES = msleep.c chkseq_SOURCES = chkseq.c tcpflood_SOURCES = tcpflood.c tcpflood_LDADD = $(SOL_LIBS) +randomgen_SOURCES = randomgen.c +randomgen_LDADD = $(SOL_LIBS) + nettester_SOURCES = nettester.c getline.c nettester_LDADD = $(SOL_LIBS) diff --git a/tests/asynwr_deadlock.sh b/tests/asynwr_deadlock.sh new file mode 100755 index 00000000..dc08355e --- /dev/null +++ b/tests/asynwr_deadlock.sh @@ -0,0 +1,23 @@ +# This is test case from practice, with the version we introduced it, it +# caused a deadlock on shutdown. I have added it to the test suite to automatically +# detect such things in the future. +# +# added 2010-03-17 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo ================================================================================ +echo TEST: \[asynwr_deadlock.sh\]: a case known to have caused a deadlock in the past +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup asynwr_deadlock.conf +# just send one message +source $srcdir/diag.sh tcpflood -m1 +# sleep is important! need to make sure the instance is inactive +sleep 1 +# now try shutdown. The actual test is if the process does hang here! +echo "processing must continue soon" +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +source $srcdir/diag.sh seq-check 0 0 +source $srcdir/diag.sh exit diff --git a/tests/asynwr_deadlock2.sh b/tests/asynwr_deadlock2.sh new file mode 100755 index 00000000..1190f67a --- /dev/null +++ b/tests/asynwr_deadlock2.sh @@ -0,0 +1,69 @@ +# This is test case from practice, with the version we introduced it, it +# caused a deadlock during processing (when the a stream was purged from the +# dynafile cache). +# We added this as a standard test in the hopes that iw will help +# detect such things in the future. +# +# The problem that originally caused this test to fail was: +# We write to files asynchronously (with the async writer thread). There is +# no signaling done when the file stream is closed. That can lead to the writer +# process hanging in memory, that in turn leads to the main thread waiting on a +# condition that never occurs (because it would need to be signalled by the +# async writer). Even worse, in that case, the async writer was signalled invalid +# in such a way that when it received a wakeup, it thought it shall not terminate, +# but received a spurios wakeup due to timeout and no data to write. In that case +# it (correctly) concluded that it would not need to timeout until a new buffer write +# was done (in which case it would receive a wakeup). As such, it went into an eternal +# wait. However, the invalid signaling did not take into account that it did not +# signal the async writer to shut down. So the main thread went into a condition +# wait - and thus we had a deadlock. That situation occured only under very specific +# cirumstances. As far as the analysis goes, the following need to happen: +# 1. buffers on that file are being flushed +# 2. no new data arrives +# 3. the inactivity timeout has not yet expired +# 4. *then* (and only then) the stream is closed or destructed +# In that, 1 to 4 are prequisites for the deadlock which will happen in 4. However, +# for it to happen, we also need the right "timing". There is a race between the +# main thread and the async writer thread. The deadlock will only happen under +# the "right" circumstances, which basically means it will not happen always. +# In order to create this case as reliable as possible, I have used +# the "$OMFileFlushOnTXEnd on" directive +# inside my test case. It makes sure that #1 above happens. The test uses a dynafile +# cache size of 4, and the load generator generates data for 5 different dynafiles. +# So over time, we will hit a spot where 4 dynafiles are open and the 5th file name +# is generated. As such, one file needs to be discarded. Thanks to FlushOnTXEnd, we +# now likely have #2 in place and thanks to the load pattern generated, we most +# probably have #3 in place. During the dynafile cache displacement of the oldest +# entry, #4 is generated. At this point, we have the deadlock we are testing for. +# Note that this deadlock does not necessarily lead to a total lockup of rsyslogd. +# Parts of it continue to operate. But in our test setup, this means data is +# received and placed into the main queue. Once it's high water mark is hit, data +# is still being enqueued, but at a slow rate. So if one is patient enough, the load +# generator will be able to finish. However, rsyslogd will never process the data +# it received because it is locked in the deadlock caused by #4 above. +# Note that "$OMFileFlushOnTXEnd on" is not causing this behaviour. We just use it +# to (quite) reliably cause the failure condition. The failure described above +# (in version 4.6.1) was also present when the setting was set to "off", but its +# occurence was very much less probable - because the perquisites are then much +# harder to hit. without it, the test may need to run for several hours before +# we hit all failure conditions. +# +# added 2010-03-17 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo ================================================================================= +echo TEST: \[asynwr_deadlock2.sh\]: a case known to have caused a deadlock in the past +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup asynwr_deadlock2.conf +# send 20000 messages, each close to 2K (non-randomized!), so that we can fill +# the buffers and hopefully run into the "deadlock". +source $srcdir/diag.sh tcpflood -m20000 -d1800 -P129 -i1 -f5 +# the sleep below is needed to prevent too-early termination of the tcp listener +sleep 1 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +cat rsyslog.out.*.log > rsyslog.out.log +source $srcdir/diag.sh seq-check 1 20000 -E +source $srcdir/diag.sh exit diff --git a/tests/asynwr_deadlock4.sh b/tests/asynwr_deadlock4.sh new file mode 100755 index 00000000..a3452f5b --- /dev/null +++ b/tests/asynwr_deadlock4.sh @@ -0,0 +1,25 @@ +# This is test case from practice, with the version we introduced it, it +# caused a deadlock during processing. +# We added this as a standard test in the hopes that iw will help +# detect such things in the future. +# +# This is a test that is constructed similar to asynwr_deadlock2.sh, but +# can produce problems in a simpler way. +# +# added 2010-03-18 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo ================================================================================= +echo TEST: \[asynwr_deadlock4.sh\]: a case known to have caused a deadlock in the past +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup asynwr_deadlock4.conf +# send 20000 messages, each close to 2K (non-randomized!), so that we can fill +# the buffers and hopefully run into the "deadlock". +source $srcdir/diag.sh tcpflood -m20000 -d18 -P129 -i1 -f5 +# sleep is important! need to make sure the instance is inactive +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +source $srcdir/diag.sh seq-check 1 20000 -E +source $srcdir/diag.sh exit diff --git a/tests/asynwr_simple.sh b/tests/asynwr_simple.sh new file mode 100755 index 00000000..eb87443c --- /dev/null +++ b/tests/asynwr_simple.sh @@ -0,0 +1,18 @@ +# This is test driver for testing asynchronous file output. +# +# added 2010-03-09 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[asynwr_simple.sh\]: simple test for async file writing +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup asynwr_simple.conf +# send 35555 messages, make sure file size is not a multiple of +# 10K, the buffer size! +source $srcdir/diag.sh tcpflood -m35555 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +source $srcdir/diag.sh seq-check 0 35554 +source $srcdir/diag.sh exit diff --git a/tests/asynwr_small.sh b/tests/asynwr_small.sh new file mode 100755 index 00000000..97818f6e --- /dev/null +++ b/tests/asynwr_small.sh @@ -0,0 +1,26 @@ +# This tests async writing with only a small set of data. That +# shall result in data staying in buffers until shutdown, what +# then will trigger some somewhat complex logic in the stream +# writer (open, write, close all during the stream close +# opertion). It is vital that only few messages be sent. +# +# The main effort of this test is not (only) to see if we +# receive the data, but rather to see if we get into an abort +# condition. +# +# added 2010-03-09 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[asynwr_small.sh\]: test for async file writing for few messages +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup asynwr_small.conf +# send 4000 messages +source $srcdir/diag.sh tcpflood -m2 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +source $srcdir/diag.sh seq-check 0 1 +source $srcdir/diag.sh exit diff --git a/tests/asynwr_timeout.sh b/tests/asynwr_timeout.sh new file mode 100755 index 00000000..b0bc5c28 --- /dev/null +++ b/tests/asynwr_timeout.sh @@ -0,0 +1,21 @@ +# This test writes to the output buffers, let's the output +# write timeout (and write data) and then continue. The conf file +# has a 2 second timeout, so we wait 4 seconds to be on the save side. +# +# added 2010-03-09 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[asynwr_timeout.sh\]: test async file writing timeout writes +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup asynwr_timeout.conf +# send 35555 messages, make sure file size is not a multiple of +# 10K, the buffer size! +source $srcdir/diag.sh tcpflood -m 35555 +sleep 4 # wait for output writer to write and empty buffer +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +source $srcdir/diag.sh seq-check 0 35554 +source $srcdir/diag.sh exit diff --git a/tests/asynwr_tinybuf.sh b/tests/asynwr_tinybuf.sh new file mode 100755 index 00000000..8eae1e26 --- /dev/null +++ b/tests/asynwr_tinybuf.sh @@ -0,0 +1,19 @@ +# This tests async writing with a very small output buffer (1 byte!), +# so it stresses output buffer handling. This also means operations will +# be somewhat slow, so we send only a small amounts of data. +# +# added 2010-03-09 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[asynwr_tinybuf.sh\]: test async file writing with 1-byte buffer +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup asynwr_tinybuf.conf +# send 1000 messages, fairly enough to trigger problems +source $srcdir/diag.sh tcpflood -m1000 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh seq-check 0 999 +source $srcdir/diag.sh exit diff --git a/tests/chkseq.c b/tests/chkseq.c index 6334d787..b22c8992 100644 --- a/tests/chkseq.c +++ b/tests/chkseq.c @@ -32,6 +32,7 @@ #include "config.h" #include <stdio.h> #include <stdlib.h> +#include <string.h> #include <getopt.h> int main(int argc, char *argv[]) @@ -40,14 +41,18 @@ int main(int argc, char *argv[]) int val; int i; int ret = 0; + int scanfOK; int verbose = 0; + int bHaveExtraData = 0; int dupsPermitted = 0; int start = 0, end = 0; int opt; int nDups = 0; + int edLen; /* length of extra data */ + static char edBuf[500*1024]; /* buffer for extra data (pretty large to be on the save side...) */ char *file = NULL; - while((opt = getopt(argc, argv, "e:f:ds:v")) != EOF) { + while((opt = getopt(argc, argv, "e:f:ds:vE")) != EOF) { switch((char)opt) { case 'f': file = optarg; @@ -64,8 +69,11 @@ int main(int argc, char *argv[]) case 'v': ++verbose; break; - default:printf("Invalid call of chkseq\n"); - printf("Usage: chkseq file -sstart -eend -d\n"); + case 'E': + bHaveExtraData = 1; + break; + default:printf("Invalid call of chkseq, optchar='%c'\n", opt); + printf("Usage: chkseq file -sstart -eend -d -E\n"); exit(1); } } @@ -93,7 +101,17 @@ int main(int argc, char *argv[]) } for(i = start ; i < end+1 ; ++i) { - if(fscanf(fp, "%d\n", &val) != 1) { + if(bHaveExtraData) { + scanfOK = fscanf(fp, "%d,%d,%s\n", &val, &edLen, edBuf) == 3 ? 1 : 0; + if(edLen != (int) strlen(edBuf)) { + printf("extra data length specified %d, but actually is %ld in record %d\n", + edLen, (long) strlen(edBuf), i); + exit(1); + } + } else { + scanfOK = fscanf(fp, "%d\n", &val) == 1 ? 1 : 0; + } + if(!scanfOK) { printf("scanf error in index i=%d\n", i); exit(1); } @@ -116,5 +134,10 @@ int main(int argc, char *argv[]) exit(1); } + if(!feof(fp)) { + printf("end of processing, but NOT end of file!\n"); + exit(1); + } + exit(ret); } diff --git a/tests/complex1.sh b/tests/complex1.sh new file mode 100755 index 00000000..7395bf21 --- /dev/null +++ b/tests/complex1.sh @@ -0,0 +1,22 @@ +# This is a rather complex test that runs a number of features together. +# +# added 2010-03-16 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo ==================================================================================== +echo TEST: \[complex1.sh\]: complex test with gzip and multiple action queues +source $srcdir/diag.sh init +# uncomment for debugging support: +export RSYSLOG_DEBUG="debug nostdout" +export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup complex1.conf +# send 40,000 messages of 400 bytes plus header max, via three dest ports +source $srcdir/diag.sh tcpflood -m40000 -rd400 -P129 -f5 -n3 -c15 -i1 +sleep 2 # due to large messages, we need this time for the tcp receiver to settle... +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +ls rsyslog.out.*.log +source $srcdir/diag.sh setzcat # find out which zcat to use +$ZCAT rsyslog.out.*.log > rsyslog.out.log +source $srcdir/diag.sh seq-check 1 40000 -E +source $srcdir/diag.sh exit diff --git a/tests/diag.sh b/tests/diag.sh index 3198d7d4..f65d197b 100755 --- a/tests/diag.sh +++ b/tests/diag.sh @@ -8,6 +8,7 @@ #valgrind="valgrind --log-fd=1" #valgrind="valgrind --tool=drd --log-fd=1" #valgrind="valgrind --tool=helgrind --log-fd=1" +#valgrind="valgrind --tool=exp-ptrcheck --log-fd=1" #set -o xtrace #export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" #export RSYSLOG_DEBUGLOG="log" @@ -15,16 +16,19 @@ case $1 in 'init') $srcdir/killrsyslog.sh # kill rsyslogd if it runs for some reason cp $srcdir/testsuites/diag-common.conf diag-common.conf cp $srcdir/testsuites/diag-common2.conf diag-common2.conf - rm -f rsyslogd.started work-*.conf + rm -f rsyslog.action.*.include + rm -f rsyslogd.started work-*.conf rsyslog.random.data rm -f rsyslogd2.started work-*.conf rm -f work rsyslog.out.log rsyslog.out.log.save # common work files + rm -f rsyslog.out.*.log rm -rf test-spool rm -f core.* vgcore.* mkdir test-spool ;; 'exit') rm -f rsyslogd.started work-*.conf diag-common.conf - rm -f rsyslogd2.started diag-common2.conf + rm -f rsyslogd2.started diag-common2.conf rsyslog.action.*.include rm -f work rsyslog.out.log rsyslog.out.log.save # common work files + rm -f rsyslog.out.*.log rsyslog.random.data rm -rf test-spool ;; 'startup') # start rsyslogd with default params. $2 is the config file name to use @@ -34,14 +38,14 @@ case $1 in ;; 'wait-startup') # wait for rsyslogd startup ($2 is the instance) while test ! -f rsyslogd$2.started; do - true + ./msleep 100 # wait 100 milliseconds done echo "rsyslogd$2 started with pid " `cat rsyslog$2.pid` ;; 'wait-shutdown') # actually, we wait for rsyslog.pid to be deleted. $2 is the # instance while test -f rsyslog$2.pid; do - true + ./msleep 100 # wait 100 milliseconds done ;; 'wait-queueempty') # wait for main message queue to be empty. $2 is the instance. @@ -62,7 +66,7 @@ case $1 in # note: we do not wait for the actual termination! ;; 'tcpflood') # do a tcpflood run and check if it worked params are passed to tcpflood - ./tcpflood $2 $3 $4 $5 $6 $7 $8 + ./tcpflood $2 $3 $4 $5 $6 $7 $8 $9 if [ "$?" -ne "0" ]; then echo "error during tcpflood! see rsyslog.out.log.save for what was written" cp rsyslog.out.log rsyslog.out.log.save @@ -85,6 +89,7 @@ case $1 in ;; 'seq-check') # do the usual sequence check to see if everything was properly received. $2 is the instance. rm -f work + cp rsyslog.out.log work-presort sort < rsyslog.out.log > work # $4... are just to have the abilit to pass in more options... ./chkseq -fwork -v -s$2 -e$3 $4 $5 $6 $7 @@ -105,6 +110,18 @@ case $1 in exit 1 fi ;; + 'gzip-seq-check') # do the usual sequence check, but for gzip files + rm -f work + ls -l rsyslog.out.log + gunzip < rsyslog.out.log | sort > work + ls -l work + # $4... are just to have the abilit to pass in more options... + ./chkseq -fwork -v -s$2 -e$3 $4 $5 $6 $7 + if [ "$?" -ne "0" ]; then + echo "sequence error detected" + exit 1 + fi + ;; 'nettester') # perform nettester-based tests # use -v for verbose output! ./nettester -t$2 -i$3 @@ -112,5 +129,12 @@ case $1 in exit 1 fi ;; + 'setzcat') # find out name of zcat tool + if [ `uname` == SunOS ]; then + ZCAT=gzcat + else + ZCAT=zcat + fi + ;; *) echo "invalid argument" $1 esac diff --git a/tests/diskqueue-fsync.sh b/tests/diskqueue-fsync.sh index 0282202d..c7745930 100755 --- a/tests/diskqueue-fsync.sh +++ b/tests/diskqueue-fsync.sh @@ -9,7 +9,7 @@ echo testing queue disk-only mode, fsync case source $srcdir/diag.sh init source $srcdir/diag.sh startup diskqueue-fsync.conf # 1000 messages should be enough - the disk fsync test is very slow! -source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 1000 +source $srcdir/diag.sh tcpflood -m1000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 999 diff --git a/tests/diskqueue.sh b/tests/diskqueue.sh index 8233d569..7a50d82e 100755 --- a/tests/diskqueue.sh +++ b/tests/diskqueue.sh @@ -6,11 +6,14 @@ # This file is part of the rsyslog project, released under GPLv3 # uncomment for debugging support: echo diskqueue.sh: testing queue disk-only mode +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" source $srcdir/diag.sh init source $srcdir/diag.sh startup diskqueue.conf # 20000 messages should be enough - the disk test is slow enough ;) sleep 4 -source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 20000 +source $srcdir/diag.sh tcpflood -m20000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 19999 diff --git a/tests/dynfile_cachemiss.sh b/tests/dynfile_cachemiss.sh new file mode 100755 index 00000000..6e2d9cca --- /dev/null +++ b/tests/dynfile_cachemiss.sh @@ -0,0 +1,34 @@ +# This test checks if omfile segfaults when a file open() in dynacache mode fails. +# The test is mimiced after a real-life scenario (which, of course, was much more +# complex). +# +# added 2010-03-09 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[dynfile_cachemiss.sh\]: test open fail for dynafiles with `cat rsyslog.action.1.include` +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup dynfile_cachemiss.conf +# we send handcrafted message. We have a dynafile cache of 4, and now send one message +# each to fill up the cache. +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.0.log:0" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.1.log:1" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.2.log:2" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.3.log:3" +# the next one has caused a segfault in practice +# note that /proc/rsyslog.error.file must not be creatable +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:/proc/rsyslog.error.file:boom" +# some more writes +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.0.log:4" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.1.log:5" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.2.log:6" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.3.log:7" +# done message generation +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +cat rsyslog.out.*.log > rsyslog.out.log +source $srcdir/diag.sh seq-check 0 7 +source $srcdir/diag.sh exit diff --git a/tests/dynfile_invalid2.sh b/tests/dynfile_invalid2.sh new file mode 100755 index 00000000..b6fb3945 --- /dev/null +++ b/tests/dynfile_invalid2.sh @@ -0,0 +1,34 @@ +# This test checks if omfile segfaults when a file open() in dynacache mode fails. +# The test is mimiced after a real-life scenario (which, of course, was much more +# complex). +# +# added 2010-03-22 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[dynfile_invalid2.sh\]: test open fail for dynafiles +source $srcdir/diag.sh init +# uncomment for debugging support: +export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup dynfile_invalid2.conf +# we send handcrafted message. We have a dynafile cache of 4, and now send one message +# each to fill up the cache. +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.0.log:0" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.1.log:1" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.2.log:2" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.3.log:3" +# the next one has caused a segfault in practice +# note that /proc/rsyslog.error.file must not be creatable +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:/proc/rsyslog.error.file:boom" +# some more writes +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.0.log:4" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.1.log:5" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.2.log:6" +./tcpflood -m1 -M "<129>Mar 10 01:00:00 172.20.245.8 tag msg:rsyslog.out.3.log:7" +# done message generation +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +cat rsyslog.out.*.log > rsyslog.out.log +source $srcdir/diag.sh seq-check 0 7 +source $srcdir/diag.sh exit diff --git a/tests/dynfile_invld_async.sh b/tests/dynfile_invld_async.sh new file mode 100755 index 00000000..3c9b2045 --- /dev/null +++ b/tests/dynfile_invld_async.sh @@ -0,0 +1,2 @@ +echo "\$OMFileAsyncWriting on" > rsyslog.action.1.include +source $srcdir/dynfile_cachemiss.sh diff --git a/tests/dynfile_invld_sync.sh b/tests/dynfile_invld_sync.sh new file mode 100755 index 00000000..cc6b6451 --- /dev/null +++ b/tests/dynfile_invld_sync.sh @@ -0,0 +1,2 @@ +echo "\$OMFileAsyncWriting off" > rsyslog.action.1.include +source $srcdir/dynfile_cachemiss.sh diff --git a/tests/execonlyonce.sh b/tests/execonlyonce.sh index a9b9d932..b7f60849 100755 --- a/tests/execonlyonce.sh +++ b/tests/execonlyonce.sh @@ -10,11 +10,11 @@ echo =========================================================================== echo \[execonlyonce.sh\]: test for the $ActionExecOnlyOnceEveryInterval directive source $srcdir/diag.sh init source $srcdir/diag.sh startup execonlyonce.conf -source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 10 1 +source $srcdir/diag.sh tcpflood -m10 -i1 # now wait until the interval definitely expires sleep 4 # one more than the once inerval! # and inject another couple of messages -source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 10 100 +source $srcdir/diag.sh tcpflood -m10 -i100 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages # now we need your custom logic to see if the result is equal to the diff --git a/tests/gzipwr_large.sh b/tests/gzipwr_large.sh new file mode 100755 index 00000000..ffce06f6 --- /dev/null +++ b/tests/gzipwr_large.sh @@ -0,0 +1,20 @@ +# This tests writing large data records in gzip mode. We use up to 10K +# record size. +# +# added 2010-03-10 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[gzipwr_large.sh\]: test for gzip file writing for large message sets +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup gzipwr_large.conf +# send 4000 messages of 10.000bytes plus header max, randomized +source $srcdir/diag.sh tcpflood -m4000 -r -d10000 -P129 +sleep 1 # due to large messages, we need this time for the tcp receiver to settle... +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +source $srcdir/diag.sh gzip-seq-check 0 3999 -E +source $srcdir/diag.sh exit diff --git a/tests/gzipwr_large_dynfile.sh b/tests/gzipwr_large_dynfile.sh new file mode 100755 index 00000000..ebd4c558 --- /dev/null +++ b/tests/gzipwr_large_dynfile.sh @@ -0,0 +1,36 @@ +# This tests writing large data records in gzip mode. We also write it to +# 5 different dynafiles, with a dynafile cache size set to 4. So this stresses +# both the input side, as well as zip writing, async writing and the dynafile +# cache logic. +# +# This test is a bit timing-dependent on the tcp reception side, so if it fails +# one may look into the timing first. The main issue is that the testbench +# currently has no good way to know if the tcp receiver is finished. This is NOT +# a problem in rsyslogd, but only of the testbench. +# +# Note that we do not yet have sufficient support for dynafiles in diag.sh, +# so we mangle some files here manually. +# +# added 2010-03-10 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo ==================================================================================== +echo TEST: \[gzipwr_large_dynfile.sh\]: test for gzip file writing for large message sets +source $srcdir/diag.sh init +# uncomment for debugging support: +export RSYSLOG_DEBUG="debug nostdout" +export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup gzipwr_large_dynfile.conf +# send 4000 messages of 10.000bytes plus header max, randomized +source $srcdir/diag.sh tcpflood -m4000 -r -d10000 -P129 -f5 +sleep 2 # due to large messages, we need this time for the tcp receiver to settle... +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +gunzip < rsyslog.out.0.log > rsyslog.out.log +gunzip < rsyslog.out.1.log >> rsyslog.out.log +gunzip < rsyslog.out.2.log >> rsyslog.out.log +gunzip < rsyslog.out.3.log >> rsyslog.out.log +gunzip < rsyslog.out.4.log >> rsyslog.out.log +#cat rsyslog.out.* > rsyslog.out.log +source $srcdir/diag.sh seq-check 0 3999 -E +source $srcdir/diag.sh exit diff --git a/tests/imtcp-multiport.sh b/tests/imtcp-multiport.sh index 47a33cb9..ad2b44f8 100755 --- a/tests/imtcp-multiport.sh +++ b/tests/imtcp-multiport.sh @@ -11,7 +11,7 @@ echo =========================================================================== echo \[imtcp-multiport.sh\]: testing imtcp multiple listeners source $srcdir/diag.sh init source $srcdir/diag.sh startup imtcp-multiport.conf -source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 10000 +source $srcdir/diag.sh tcpflood -p13514 -m10000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 9999 @@ -23,7 +23,7 @@ source $srcdir/diag.sh exit # source $srcdir/diag.sh init source $srcdir/diag.sh startup imtcp-multiport.conf -source $srcdir/diag.sh tcpflood 127.0.0.1 13515 1 10000 +source $srcdir/diag.sh tcpflood -p13515 -m10000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 9999 @@ -35,7 +35,7 @@ source $srcdir/diag.sh exit # source $srcdir/diag.sh init source $srcdir/diag.sh startup imtcp-multiport.conf -source $srcdir/diag.sh tcpflood 127.0.0.1 13516 1 10000 +source $srcdir/diag.sh tcpflood -p13516 -m10000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages source $srcdir/diag.sh wait-shutdown source $srcdir/diag.sh seq-check 0 9999 diff --git a/tests/longrun.sh b/tests/longrun.sh new file mode 100755 index 00000000..407e9fa9 --- /dev/null +++ b/tests/longrun.sh @@ -0,0 +1,30 @@ +# This is a test-aid script to try running some tests through many iterations. +# It is not yet used in the automated testbench, but I keep this file so that +# I can use it whenever there is need to. As such, it currently does not have +# parameters but is expected to be edited as needed. -- rgerhards, 2010-03-10 +# +# use: ./longrun.sh testname.sh +# +# where testname.sh is the test to be run +# to change other params, you need to edit the settings here below: +MAXRUNS=10 +DISPLAYALIVE=100 +LOGFILE=runlog + +echo "logfile is $LOGFILE" +echo "executing test $1" + +date > $LOGFILE + +for (( i=0; $i < 10000; i++ )) + do + if [ $(( i % DISPLAYALIVE )) -eq 0 ]; then + echo "$i iterations done" + fi + $1 >> $LOGFILE + if [ "$?" -ne "0" ]; then + echo "Test failed in iteration $i, review $LOGFILE for details!" + exit 1 + fi + done +echo "No failure in $i iterations." diff --git a/tests/manytcp.sh b/tests/manytcp.sh index c55eb9c0..94a5c035 100755 --- a/tests/manytcp.sh +++ b/tests/manytcp.sh @@ -2,7 +2,10 @@ source $srcdir/diag.sh init source $srcdir/diag.sh startup manytcp.conf # the config file specifies exactly 1100 connections -source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1000 40000 +source $srcdir/diag.sh tcpflood -c1000 -m40000 +# the sleep below is needed to prevent too-early termination of the tcp listener +sleep 1 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # we need to wait until rsyslogd is finished! source $srcdir/diag.sh seq-check 0 39999 source $srcdir/diag.sh exit diff --git a/tests/msleep.c b/tests/msleep.c new file mode 100644 index 00000000..6fa57b79 --- /dev/null +++ b/tests/msleep.c @@ -0,0 +1,50 @@ +/* sleeps for the specified number of MILLIseconds. + * Primarily meant as a portable tool available everywhere for the + * testbench (sleep 0.1 does not work on all platforms). + * + * Part of the testbench for rsyslog. + * + * Copyright 2010 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +#include "config.h" +#include <stdio.h> +#include <stdlib.h> + +int main(int argc, char *argv[]) +{ + struct timeval tvSelectTimeout; + long sleepTime; + + if(argc != 2) { + fprintf(stderr, "usage: msleep <milliseconds>\n"); + exit(1); + } + + sleepTime = atoi(argv[1]); + tvSelectTimeout.tv_sec = sleepTime / 1000; + tvSelectTimeout.tv_usec = (sleepTime % 1000) * 1000; /* micro seconds */ + if(select(0, NULL, NULL, NULL, &tvSelectTimeout) == -1) { + perror("select"); + exit(1); + } + + return 0; +} + diff --git a/tests/nettester.c b/tests/nettester.c index 209c2a6f..eff5929b 100644 --- a/tests/nettester.c +++ b/tests/nettester.c @@ -47,6 +47,7 @@ #include <signal.h> #include <netinet/in.h> #include <getopt.h> +#include <ctype.h> #define EXIT_FAILURE 1 #define INVALID_SOCKET -1 @@ -61,6 +62,7 @@ static char *testSuite = NULL; /* name of current test suite */ static int iPort = 12514; /* port which shall be used for sending data */ static char* pszCustomConf = NULL; /* custom config file, use -c conf to specify */ static int verbose = 0; /* verbose output? -v option */ +static char **ourEnvp; /* these two are quick hacks... */ int iFailed = 0; @@ -216,11 +218,6 @@ int openPipe(char *configFile, pid_t *pid, int *pfd) char *newargv[] = {"../tools/rsyslogd", "dummy", "-c4", "-u2", "-n", "-irsyslog.pid", "-M../runtime/.libs:../.libs", NULL }; char confFile[1024]; - char *newenviron[] = { NULL }; - /* debug aide... - char *newenviron[] = { "RSYSLOG_DEBUG=debug nostdout", - "RSYSLOG_DEBUGLOG=log", NULL }; - */ sprintf(confFile, "-f%s/testsuites/%s.conf", srcdir, (pszCustomConf == NULL) ? configFile : pszCustomConf); @@ -243,7 +240,7 @@ int openPipe(char *configFile, pid_t *pid, int *pfd) close(pipefd[1]); close(pipefd[0]); fclose(stdin); - execve("../tools/rsyslogd", newargv, newenviron); + execve("../tools/rsyslogd", newargv, ourEnvp); } else { close(pipefd[1]); *pid = cpid; @@ -254,6 +251,62 @@ int openPipe(char *configFile, pid_t *pid, int *pfd) } +/* This function unescapes a string of testdata. That it, escape sequences + * are converted into their one-character equivalent. While doing so, it applies + * C-like semantics. This was made necessary for easy integration of control + * characters inside test cases. -- rgerhards, 2009-03-11 + * Currently supported: + * \\ single backslash + * \n, \t, \r as in C + * \nnn where nnn is a 1 to 3 character octal sequence + * Note that when a problem occurs, the end result is undefined. After all, this + * is for a testsuite generatort, it must not be 100% bullet proof (so do not + * copy this code into something that must be!). Also note that we do in-memory + * unescaping and assume that the string gets shorter but NEVER longer! + */ +void unescapeTestdata(char *testdata) +{ + char *pDst; + char *pSrc; + int i; + int c; + + pDst = pSrc = testdata; + while(*pSrc) { + if(*pSrc == '\\') { + switch(*++pSrc) { + case '\\': *pDst++ = *pSrc++; + break; + case 'n': *pDst++ = '\n'; + ++pSrc; + break; + case 'r': *pDst++ = '\r'; + ++pSrc; + break; + case 't': *pDst++ = '\t'; + ++pSrc; + break; + case '0': + case '1': + case '2': + case '3': c = *pSrc++ - '0'; + i = 1; /* we already processed one digit! */ + while(i < 3 && isdigit(*pSrc)) { + c = c * 8 + *pSrc++ - '0'; + ++i; + } + *pDst++ = c; + break; + default: break; + } + } else { + *pDst++ = *pSrc++; + } + } + *pDst = '\0'; +} + + /* Process a specific test case. File name is provided. * Needs to return 0 if all is OK, something else otherwise. */ @@ -291,6 +344,7 @@ processTestFile(int fd, char *pszFileName) testdata[strlen(testdata)-1] = '\0'; /* remove \n */ /* now we have the test data to send (we could use function pointers here...) */ + unescapeTestdata(testdata); if(inputMode == inputUDP) { if(udpSend(testdata, strlen(testdata)) != 0) return(2); @@ -314,10 +368,13 @@ processTestFile(int fd, char *pszFileName) expected, buf); ret = 1; } - + /* we need to free buffers, as we have potentially modified them! */ + free(testdata); + testdata = NULL; + free(expected); + expected = NULL; } - free(testdata); free(expected); fclose(fp); return(ret); @@ -390,7 +447,7 @@ void doAtExit(void) * of this file. * rgerhards, 2009-04-03 */ -int main(int argc, char *argv[]) +int main(int argc, char *argv[], char *envp[]) { int fd; int opt; @@ -399,7 +456,8 @@ int main(int argc, char *argv[]) char buf[4096]; char testcases[4096]; - while((opt = getopt(argc, argv, "c:i:p:t:v")) != EOF) { + ourEnvp = envp; + while((opt = getopt(argc, argv, "dc:i:p:t:v")) != EOF) { switch((char)opt) { case 'c': pszCustomConf = optarg; @@ -424,7 +482,7 @@ int main(int argc, char *argv[]) verbose = 1; break; default:printf("Invalid call of nettester, invalid option '%c'.\n", opt); - printf("Usage: nettester -ttestsuite-name -iudp|tcp [-pport] [-ccustomConfFile] \n"); + printf("Usage: nettester -d -ttestsuite-name -iudp|tcp [-pport] [-ccustomConfFile] \n"); exit(1); } } diff --git a/tests/ourtail.c b/tests/ourtail.c index 4e8a6412..c31babb9 100644 --- a/tests/ourtail.c +++ b/tests/ourtail.c @@ -26,6 +26,7 @@ * * A copy of the GPL can be found in the file "COPYING" in this distribution. */ +#include "config.h" #include <stdio.h> int main(int __attribute__((unused)) argc, char __attribute__((unused)) *argv[]) diff --git a/tests/parsertest.sh b/tests/parsertest.sh index ef33256e..fc68ab84 100755 --- a/tests/parsertest.sh +++ b/tests/parsertest.sh @@ -1,5 +1,33 @@ -echo TEST: parsertest.sh - various parser tests +echo TEST: \[parsertest.sh\]: various parser tests source $srcdir/diag.sh init source $srcdir/diag.sh nettester parse1 udp source $srcdir/diag.sh nettester parse1 tcp -source $srcdir/diag.sh init +source $srcdir/diag.sh nettester parse3 udp +source $srcdir/diag.sh nettester parse3 tcp +source $srcdir/diag.sh nettester parse_invld_regex udp +source $srcdir/diag.sh nettester parse_invld_regex tcp +source $srcdir/diag.sh nettester parse-3164-buggyday udp +source $srcdir/diag.sh nettester parse-3164-buggyday tcp +source $srcdir/diag.sh nettester parse-nodate udp +source $srcdir/diag.sh nettester parse-nodate tcp +# the following samples can only be run over UDP as they are so +# malformed they break traditional syslog/tcp framing... +source $srcdir/diag.sh nettester snare_ccoff_udp udp +source $srcdir/diag.sh nettester snare_ccoff_udp2 udp + +echo \[parsertest.sh]: redoing tests in IPv4-only mode +source $srcdir/diag.sh nettester parse1 udp -4 +source $srcdir/diag.sh nettester parse1 tcp -4 +source $srcdir/diag.sh nettester parse3 udp -4 +source $srcdir/diag.sh nettester parse3 tcp -4 +source $srcdir/diag.sh nettester parse_invld_regex udp -4 +source $srcdir/diag.sh nettester parse_invld_regex tcp -4 +source $srcdir/diag.sh nettester parse-3164-buggyday udp -4 +source $srcdir/diag.sh nettester parse-3164-buggyday tcp -4 +source $srcdir/diag.sh nettester parse-nodate udp -4 +source $srcdir/diag.sh nettester parse-nodate tcp -4 +# UDP-only tests +source $srcdir/diag.sh nettester snare_ccoff_udp udp -4 +source $srcdir/diag.sh nettester snare_ccoff_udp2 udp -4 + +source $srcdir/diag.sh exit diff --git a/tests/pipeaction.sh b/tests/pipeaction.sh index 26a4c22a..c2201011 100755 --- a/tests/pipeaction.sh +++ b/tests/pipeaction.sh @@ -17,7 +17,7 @@ echo background cp process id is $CPPROCESS # now do the usual run source $srcdir/diag.sh startup pipeaction.conf # 20000 messages should be enough -#source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 20000 +#source $srcdir/diag.sh tcpflood -m20000 source $srcdir/diag.sh injectmsg 0 20000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages source $srcdir/diag.sh wait-shutdown diff --git a/tests/random.sh b/tests/random.sh new file mode 100755 index 00000000..79f704c7 --- /dev/null +++ b/tests/random.sh @@ -0,0 +1,20 @@ +# Test if rsyslog survives sending truely random data to it... +# +# added 2010-04-01 by Rgerhards +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[random.sh\]: testing random data +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +source $srcdir/diag.sh startup random.conf +# generate random data +./randomgen -f rsyslog.random.data -s 100000 +ls -l rsyslog.random.data +source $srcdir/diag.sh tcpflood -B -I rsyslog.random.data -c5 -C10 +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +# we do not check anything yet, the point is if rsyslog survived ;) +# TODO: check for exit message, but we'll notice an abort anyhow, so not that important +#source $srcdir/diag.sh exit diff --git a/tests/randomgen.c b/tests/randomgen.c new file mode 100644 index 00000000..9ba56954 --- /dev/null +++ b/tests/randomgen.c @@ -0,0 +1,130 @@ +/* generates random data for later use in test cases. Of course, + * we could generate random data during the testcase itself, but + * the core idea is that we record the random data so that we have + * a chance to reproduce a problem should it occur. IMHO this + * provides the best compromise, by a) having randomness but + * b) knowing what was used during the test. + * + * Params + * -f output file name (stdout if not given) + * -s size of test data, plain number is size in k, 1MB default + * -u uses /dev/urandom instead of libc random number generator + * (when available). Note that this is usually much slower. + * + * Part of the testbench for rsyslog. + * + * Copyright 2010 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +#include "config.h" +#include <stdio.h> +#include <stdlib.h> +#include <time.h> +#include <signal.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <arpa/inet.h> +#include <assert.h> +#include <unistd.h> +#include <string.h> +#include <netinet/in.h> + +#define EXIT_FAILURE 1 + +static char *fileName = NULL; /* name of output file */ +static int tryUseURandom = 0; /* try to use /dev/urandom? */ +static long long fileSize = 1024*1024; /* file size in K, 1MB default */ + + +/* generate the random file. This code really can be improved (e.g. read /dev/urandom + * when available) + */ +static inline void +genFile() +{ + long i; + FILE *fp; + FILE *rfp = NULL; + + if(fileName == NULL) { + fp = stdout; + } else { + if((fp = fopen(fileName, "w")) == NULL) { + perror(fileName); + } + } + + /* try to use /dev/urandom, if available */ + if(tryUseURandom) + rfp = fopen("/dev/urandom", "r"); + + if(rfp == NULL) { + /* fallback, use libc random number generator */ + for(i = 0 ; i < fileSize ; ++i) { + if(fputc((char) rand(), fp) == EOF) { + perror(fileName); + exit(1); + } + } + } else { + /* use /dev/urandom */ + printf("using /dev/urandom"); + for(i = 0 ; i < fileSize ; ++i) { + if(fputc(fgetc(rfp), fp) == EOF) { + perror(fileName); + exit(1); + } + } + } + + if(fileName != NULL) + fclose(fp); +} + + +/* Run the test. + * rgerhards, 2009-04-03 + */ +int main(int argc, char *argv[]) +{ + int ret = 0; + int opt; + + srand(time(NULL)); /* seed is good enough for our needs */ + + while((opt = getopt(argc, argv, "f:s:u")) != -1) { + switch (opt) { + case 'f': fileName = optarg; + break; + case 's': fileSize = atol(optarg) * 1024; + break; + case 'u': tryUseURandom = 1; + break; + default: printf("invalid option '%c' or value missing - terminating...\n", opt); + exit (1); + break; + } + } + + printf("generating random data file '%s' of %ldkb - may take a short while...\n", + fileName, (long) (fileSize / 1024)); + genFile(); + + exit(ret); +} diff --git a/tests/rt-init.c b/tests/rt-init.c index aaac7ed1..dbe94b4a 100644 --- a/tests/rt-init.c +++ b/tests/rt-init.c @@ -21,10 +21,10 @@ * * A copy of the GPL can be found in the file "COPYING" in this distribution. */ -#include <stdio.h> - +#include "config.h" #include "rsyslog.h" #include "testbench.h" +#include <stdio.h> /* must be last, else we get a zlib compile error on some platforms */ MODULE_TYPE_TESTBENCH @@ -39,6 +39,6 @@ ENDExit BEGINTest CODESTARTTest -finalize_it: +/*finalize_it:*/ /* room for custom error reporter, leave blank if not needed */ ENDTest diff --git a/tests/sndrcv_drvr.sh b/tests/sndrcv_drvr.sh index 63afd6b5..3d613069 100755 --- a/tests/sndrcv_drvr.sh +++ b/tests/sndrcv_drvr.sh @@ -34,7 +34,7 @@ source $srcdir/diag.sh wait-startup 2 # now inject the messages into instance 2. It will connect to instance 1, # and that instance will record the data. -source $srcdir/diag.sh tcpflood 127.0.0.1 13514 1 $2 1 +source $srcdir/diag.sh tcpflood -m$2 -i1 sleep 2 # make sure all data is received in input buffers # shut down sender when everything is sent, receiver continues to run concurrently # may be needed by TLS (once we do it): sleep 60 diff --git a/tests/tcpflood.c b/tests/tcpflood.c index 0439e33e..e92d1308 100644 --- a/tests/tcpflood.c +++ b/tests/tcpflood.c @@ -2,15 +2,36 @@ * messages over them. This is used for stress-testing. * * Params - * argv[1] target address - * argv[2] target port - * argv[3] number of connections - * argv[4] number of messages to send (connection is random) - * argv[5] initial message number (optional) + * -t target address (default 127.0.0.1) + * -p target port (default 13514) + * -n number of target ports (targets are in range -p..(-p+-n-1) + * -c number of connections (default 1) + * -m number of messages to send (connection is random) + * -i initial message number (optional) + * -P PRI to be used for generated messages (default is 167). + * Specify the plain number without leading zeros + * -d amount of extra data to add to message. If present, the + * number itself will be added as third field, and the data + * bytes as forth. Add -r to randomize the amount of extra + * data included in the range 1..(value of -d). + * -r randomize amount of extra data added (-d must be > 0) + * -f support for testing dynafiles. If given, include a dynafile ID + * in the range 0..(f-1) as the SECOND field, shifting all field values + * one field to the right. Zero (default) disables this functionality. + * -M the message to be sent. Disables all message format options, as + * only that exact same message is sent. + * -I read specified input file, do NOT generate own test data. The test + * completes when eof is reached. + * -B The specified file (-I) is binary. No data processing is done by + * tcpflood. If multiple connections are specified, data is read in + * chunks and spread across the connections without taking any record + * delemiters into account. + * -C when input from a file is read, this file is transmitted -C times + * (C like cycle, running out of meaningful option switches ;)) * * Part of the testbench for rsyslog. * - * Copyright 2009 Rainer Gerhards and Adiscon GmbH. + * Copyright 2009, 2010 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -41,18 +62,32 @@ #include <unistd.h> #include <string.h> #include <netinet/in.h> +#include <sys/resource.h> #define EXIT_FAILURE 1 #define INVALID_SOCKET -1 /* Name of input file, must match $IncludeConfig in test suite .conf files */ #define NETTEST_INPUT_CONF_FILE "nettest.input.conf" /* name of input file, must match $IncludeConfig in .conf files */ -static char *targetIP; -static int targetPort; +#define MAX_EXTRADATA_LEN 100*1024 + +static char *targetIP = "127.0.0.1"; +static char *msgPRI = "167"; +static int targetPort = 13514; +static int numTargetPorts = 1; +static int dynFileIDs = 0; +static int extraDataLen = 0; /* amount of extra data to add to message */ +static int bRandomizeExtraData = 0; /* randomize amount of extra data added */ static int numMsgsToSend; /* number of messages to send */ -static int numConnections; /* number of connections to create */ +static int numConnections = 1; /* number of connections to create */ static int *sockArray; /* array of sockets to use */ static int msgNum = 0; /* initial message number to start with */ +static int bShowProgress = 1; /* show progress messages */ +static char *MsgToSend = NULL; /* if non-null, this is the actual message to send */ +static int bBinaryFile = 0; /* is -I file binary */ +static char *dataFile = NULL; /* name of data file, if NULL, generate own data */ +static int numFileIterations = 1;/* how often is file data to be sent? */ +FILE *dataFP = NULL; /* file pointer for data file, if used */ /* open a single tcp connection @@ -61,16 +96,25 @@ int openConn(int *fd) { int sock; struct sockaddr_in addr; + int port; int retries = 0; + int rnd; if((sock=socket(AF_INET, SOCK_STREAM, 0))==-1) { perror("socket()"); return(1); } + /* randomize port if required */ + if(numTargetPorts > 1) { + rnd = rand(); /* easier if we need value for debug messages ;) */ + port = targetPort + (rnd % numTargetPorts); + } else { + port = targetPort; + } memset((char *) &addr, 0, sizeof(addr)); addr.sin_family = AF_INET; - addr.sin_port = htons(targetPort); + addr.sin_port = htons(port); if(inet_aton(targetIP, &addr.sin_addr)==0) { fprintf(stderr, "inet_aton() failed\n"); return(1); @@ -103,11 +147,13 @@ int openConnections(void) char msgBuf[128]; size_t lenMsg; - write(1, " open connections", sizeof(" open connections")-1); + if(bShowProgress) + write(1, " open connections", sizeof(" open connections")-1); sockArray = calloc(numConnections, sizeof(int)); for(i = 0 ; i < numConnections ; ++i) { if(i % 10 == 0) { - printf("\r%5.5d", i); + if(bShowProgress) + printf("\r%5.5d", i); //lenMsg = sprintf(msgBuf, "\r%5.5d", i); //write(1, msgBuf, lenMsg); } @@ -133,14 +179,17 @@ int openConnections(void) void closeConnections(void) { int i; - char msgBuf[128]; size_t lenMsg; + char msgBuf[128]; - write(1, " close connections", sizeof(" close connections")-1); + if(bShowProgress) + write(1, " close connections", sizeof(" close connections")-1); for(i = 0 ; i < numConnections ; ++i) { if(i % 10 == 0) { - lenMsg = sprintf(msgBuf, "\r%5.5d", i); - write(1, msgBuf, lenMsg); + if(bShowProgress) { + lenMsg = sprintf(msgBuf, "\r%5.5d", i); + write(1, msgBuf, lenMsg); + } } close(sockArray[i]); } @@ -150,6 +199,62 @@ void closeConnections(void) } +/* generate the message to be sent according to program command line parameters. + * this has been moved to its own function as we now have various different ways + * of constructing test messages. -- rgerhards, 2010-03-31 + */ +static inline void +genMsg(char *buf, size_t maxBuf, int *pLenBuf) +{ + int edLen; /* actual extra data length to use */ + char extraData[MAX_EXTRADATA_LEN + 1]; + char dynFileIDBuf[128] = ""; + static int numMsgsGen = 0; + int done; + + if(dataFP != NULL) { + /* get message from file */ + do { + done = 1; + *pLenBuf = fread(buf, 1, 1024, dataFP); + if(feof(dataFP)) { + if(--numFileIterations > 0) { + rewind(dataFP); + done = 0; /* need new iteration */ + } else { + *pLenBuf = 0; + goto finalize_it; + } + } + } while(!done); /* Attention: do..while()! */ + } else if(MsgToSend == NULL) { + if(dynFileIDs > 0) { + snprintf(dynFileIDBuf, maxBuf, "%d:", rand() % dynFileIDs); + } + if(extraDataLen == 0) { + *pLenBuf = snprintf(buf, maxBuf, "<%s>Mar 1 01:00:00 172.20.245.8 tag msgnum:%s%8.8d:\n", + msgPRI, dynFileIDBuf, msgNum); + } else { + if(bRandomizeExtraData) + edLen = ((long) rand() + extraDataLen) % extraDataLen + 1; + else + edLen = extraDataLen; + memset(extraData, 'X', edLen); + extraData[edLen] = '\0'; + *pLenBuf = snprintf(buf, maxBuf, "<%s>Mar 1 01:00:00 172.20.245.8 tag msgnum:%s%8.8d:%d:%s\n", + msgPRI, dynFileIDBuf, msgNum, edLen, extraData); + } + } else { + /* use fixed message format from command line */ + *pLenBuf = snprintf(buf, maxBuf, "%s\n", MsgToSend); + } + + if(numMsgsGen++ >= numMsgsToSend) + *pLenBuf = 0; /* indicate end of run */ + +finalize_it: ; +} + /* send messages to the tcp connections we keep open. We use * a very basic format that helps identify the message * (via msgnum:<number>: e.g. msgnum:00000001:). This format is suitable @@ -160,39 +265,53 @@ void closeConnections(void) */ int sendMessages(void) { - int i; + int i = 0; int socknum; int lenBuf; int lenSend; - char buf[2048]; - - srand(time(NULL)); /* seed is good enough for our needs */ - - printf("Sending %d messages.\n", numMsgsToSend); - printf("\r%5.5d messages sent", 0); - for(i = 0 ; i < numMsgsToSend ; ++i) { + char *statusText; + char buf[MAX_EXTRADATA_LEN + 1024]; + + if(dataFile == NULL) { + printf("Sending %d messages.\n", numMsgsToSend); + statusText = "messages"; + } else { + printf("Sending file '%s' %d times.\n", dataFile, numFileIterations); + statusText = "kb"; + } + if(bShowProgress) + printf("\r%8.8d %s sent", 0, statusText); + while(1) { /* broken inside loop! */ if(i < numConnections) socknum = i; else if(i >= numMsgsToSend - numConnections) socknum = i - (numMsgsToSend - numConnections); - else - socknum = rand() % numConnections; - lenBuf = sprintf(buf, "<167>Mar 1 01:00:00 172.20.245.8 tag msgnum:%8.8d:\n", msgNum); + else { + int rnd = rand(); + //socknum = rand() % numConnections; + socknum = rnd % numConnections; + } + genMsg(buf, sizeof(buf), &lenBuf); /* generate the message to send according to params */ + if(lenBuf == 0) + break; /* end of processing! */ lenSend = send(sockArray[socknum], buf, lenBuf, 0); if(lenSend != lenBuf) { printf("\r%5.5d\n", i); fflush(stdout); perror("send test data"); - printf("send() failed at socket %d, index %d, msgNum %d\n", socknum, i, msgNum); + printf("send() failed at socket %d, index %d, msgNum %d\n", + sockArray[socknum], i, msgNum); fflush(stderr); return(1); } if(i % 100 == 0) { - printf("\r%5.5d", i); + if(bShowProgress) + printf("\r%8.8d", i); } ++msgNum; + ++i; } - printf("\r%5.5d messages sent\n", i); + printf("\r%8.8d %s sent\n", i, statusText); return 0; } @@ -251,17 +370,18 @@ tcpSend(char *buf, int lenBuf) } -/* Run the test suite. This must be called with exactly one parameter, the - * name of the test suite. For details, see file header comment at the top - * of this file. +/* Run the test. * rgerhards, 2009-04-03 */ int main(int argc, char *argv[]) { int ret = 0; + int opt; struct sigaction sigAct; static char buf[1024]; + srand(time(NULL)); /* seed is good enough for our needs */ + /* on Solaris, we do not HAVE MSG_NOSIGNAL, so for this reason * we block SIGPIPE (not an issue for this program) */ @@ -271,19 +391,75 @@ int main(int argc, char *argv[]) sigaction(SIGPIPE, &sigAct, NULL); setvbuf(stdout, buf, _IONBF, 48); + + if(!isatty(1)) + bShowProgress = 0; + + while((opt = getopt(argc, argv, "f:t:p:c:C:m:i:I:P:d:n:M:rB")) != -1) { + switch (opt) { + case 't': targetIP = optarg; + break; + case 'p': targetPort = atoi(optarg); + break; + case 'n': numTargetPorts = atoi(optarg); + break; + case 'c': numConnections = atoi(optarg); + break; + case 'C': numFileIterations = atoi(optarg); + break; + case 'm': numMsgsToSend = atoi(optarg); + break; + case 'i': msgNum = atoi(optarg); + break; + case 'P': msgPRI = optarg; + break; + case 'd': extraDataLen = atoi(optarg); + if(extraDataLen > MAX_EXTRADATA_LEN) { + fprintf(stderr, "-d max is %d!\n", + MAX_EXTRADATA_LEN); + exit(1); + } + break; + case 'r': bRandomizeExtraData = 1; + break; + case 'f': dynFileIDs = atoi(optarg); + break; + case 'M': MsgToSend = optarg; + break; + case 'I': dataFile = optarg; + /* in this mode, we do not know the num messages to send, so + * we set a (high) number to keep the code happy. + */ + numMsgsToSend = 1000000; + break; + case 'B': bBinaryFile = 1; + break; + default: printf("invalid option '%c' or value missing - terminating...\n", opt); + exit (1); + break; + } + } - if(argc != 5 && argc != 6) { - printf("Invalid call of tcpflood\n"); - printf("Usage: tcpflood target-host target-port num-connections num-messages [initial msgnum]\n"); - exit(1); + if(numConnections > 20) { + /* if we use many (whatever this means, 20 is randomly picked) + * connections, we need to make sure we have a high enough + * limit. -- rgerhards, 2010-03-25 + */ + struct rlimit maxFiles; + maxFiles.rlim_cur = numConnections + 20; + maxFiles.rlim_max = numConnections + 20; + if(setrlimit(RLIMIT_NOFILE, &maxFiles) < 0) { + perror("setrlimit to increase file handles failed"); + exit(1); + } } - targetIP = argv[1]; - targetPort = atoi(argv[2]); - numConnections = atoi(argv[3]); - numMsgsToSend = atoi(argv[4]); - if(argc == 6) - msgNum = atoi(argv[5]); + if(dataFile != NULL) { + if((dataFP = fopen(dataFile, "r")) == NULL) { + perror(dataFile); + exit(1); + } + } if(openConnections() != 0) { printf("error opening connections\n"); @@ -295,7 +471,6 @@ int main(int argc, char *argv[]) exit(1); } - //closeConnections(); printf("End of tcpflood Run\n"); exit(ret); diff --git a/tests/testsuites/asynwr_deadlock.conf b/tests/testsuites/asynwr_deadlock.conf new file mode 100644 index 00000000..dc4045b0 --- /dev/null +++ b/tests/testsuites/asynwr_deadlock.conf @@ -0,0 +1,14 @@ +# rgerhards, 2010-03-09 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%\n" + +$OMFileFlushOnTXEnd on +$OMFileFlushInterval 10 +$OMFileFlushIOBufferSize 10k +$OMFileAsyncWriting on +:msg, contains, "msgnum:" ./rsyslog.out.log;outfmt diff --git a/tests/testsuites/asynwr_deadlock2.conf b/tests/testsuites/asynwr_deadlock2.conf new file mode 100644 index 00000000..07811613 --- /dev/null +++ b/tests/testsuites/asynwr_deadlock2.conf @@ -0,0 +1,16 @@ +# rgerhards, 2010-03-17 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:3%,%msg:F,58:4%,%msg:F,58:5%\n" +$template dynfile,"rsyslog.out.%msg:F,58:2%.log" # use multiple dynafiles + +$OMFileFlushOnTXEnd on +$OMFileFlushInterval 10 +$OMFileIOBufferSize 10k +$OMFileAsyncWriting on +$DynaFileCacheSize 4 +local0.* ?dynfile;outfmt diff --git a/tests/testsuites/asynwr_deadlock4.conf b/tests/testsuites/asynwr_deadlock4.conf new file mode 100644 index 00000000..f4308ff1 --- /dev/null +++ b/tests/testsuites/asynwr_deadlock4.conf @@ -0,0 +1,16 @@ +# rgerhards, 2010-03-17 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:3%,%msg:F,58:4%,%msg:F,58:5%\n" +$template dynfile,"rsyslog.out.log" # use multiple dynafiles + +$OMFileFlushOnTXEnd on +$OMFileFlushInterval 10 +$OMFileIOBufferSize 10k +$OMFileAsyncWriting on +$DynaFileCacheSize 4 +local0.* ?dynfile;outfmt diff --git a/tests/testsuites/asynwr_simple.conf b/tests/testsuites/asynwr_simple.conf new file mode 100644 index 00000000..44b03f2b --- /dev/null +++ b/tests/testsuites/asynwr_simple.conf @@ -0,0 +1,15 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +$OMFileFlushOnTXEnd off +$OMFileFlushInterval 2 +$OMFileFlushIOBufferSize 10k +$OMFileAsyncWriting on +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/asynwr_small.conf b/tests/testsuites/asynwr_small.conf new file mode 100644 index 00000000..f04ce962 --- /dev/null +++ b/tests/testsuites/asynwr_small.conf @@ -0,0 +1,14 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +$OMFileFlushOnTXEnd off +$OMFileFlushInterval 2 +$OMFileAsyncWriting on +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/asynwr_timeout.conf b/tests/testsuites/asynwr_timeout.conf new file mode 100644 index 00000000..44b03f2b --- /dev/null +++ b/tests/testsuites/asynwr_timeout.conf @@ -0,0 +1,15 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +$OMFileFlushOnTXEnd off +$OMFileFlushInterval 2 +$OMFileFlushIOBufferSize 10k +$OMFileAsyncWriting on +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/asynwr_tinybuf.conf b/tests/testsuites/asynwr_tinybuf.conf new file mode 100644 index 00000000..01dec4d8 --- /dev/null +++ b/tests/testsuites/asynwr_tinybuf.conf @@ -0,0 +1,15 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +$OMFileFlushOnTXEnd off +$OMFileFlushInterval 2 +$OMFileIOBufferSize 1 +$OMFileAsyncWriting on +:msg, contains, "msgnum:" ?dynfile;outfmt diff --git a/tests/testsuites/complex1.conf b/tests/testsuites/complex1.conf new file mode 100644 index 00000000..9e2441d4 --- /dev/null +++ b/tests/testsuites/complex1.conf @@ -0,0 +1,81 @@ +# complex test case with multiple actions in gzip mode +# rgerhards, 2009-05-22 +$MaxMessageSize 10k +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 + +$template outfmt,"%msg:F,58:3%,%msg:F,58:4%,%msg:F,58:5%\n" +$template dynfile,"rsyslog.out.%inputname%.%msg:F,58:2%.log" + +## RULESET with listener +$Ruleset R13514 +# queue params: +$ActionQueueTimeoutShutdown 60000 +$ActionQueueSize 5000 +$ActionQueueSaveOnShutdown on +$ActionQueueHighWaterMark 4900 +$ActionQueueLowWaterMark 3500 +$ActionQueueType FixedArray +$ActionQueueWorkerThreads 1 +# action params: +$OMFileFlushOnTXEnd off +$OMFileZipLevel 6 +#$OMFileIOBufferSize 256k +$DynaFileCacheSize 4 +$omfileFlushInterval 1 +*.* ?dynfile;outfmt +# listener +$InputTCPServerInputName 13514 +$InputTCPServerBindRuleset R13514 +$InputTCPServerRun 13514 + + +## RULESET with listener +$Ruleset R13515 +# queue params: +$ActionQueueTimeoutShutdown 60000 +$ActionQueueSize 5000 +$ActionQueueSaveOnShutdown on +$ActionQueueHighWaterMark 4900 +$ActionQueueLowWaterMark 3500 +$ActionQueueType FixedArray +$ActionQueueWorkerThreads 1 +# action params: +$OMFileFlushOnTXEnd off +$OMFileZipLevel 6 +$OMFileIOBufferSize 256k +$DynaFileCacheSize 4 +$omfileFlushInterval 1 +*.* ?dynfile;outfmt +# listener +$InputTCPServerInputName 13515 +$InputTCPServerBindRuleset R13515 +$InputTCPServerRun 13515 + + + +## RULESET with listener +$Ruleset R13516 +# queue params: +$ActionQueueTimeoutShutdown 60000 +$ActionQueueSize 5000 +$ActionQueueSaveOnShutdown on +$ActionQueueHighWaterMark 4900 +$ActionQueueLowWaterMark 3500 +$ActionQueueType FixedArray +$ActionQueueWorkerThreads 1 +# action params: +$OMFileFlushOnTXEnd off +$OMFileZipLevel 6 +$OMFileIOBufferSize 256k +$DynaFileCacheSize 4 +$omfileFlushInterval 1 +*.* ?dynfile;outfmt +# listener +$InputTCPServerInputName 13516 +$InputTCPServerBindRuleset R13516 +$InputTCPServerRun 13516 + + diff --git a/tests/testsuites/diag-common2.conf b/tests/testsuites/diag-common2.conf new file mode 100644 index 00000000..94f7e87f --- /dev/null +++ b/tests/testsuites/diag-common2.conf @@ -0,0 +1,16 @@ +# This is a config include file. It sets up rsyslog so that the +# diag system can successfully be used. Also, it generates a file +# "rsyslogd.started" after rsyslogd is initialized. This config file +# should be included in all tests that intend to use common code for +# controlling the daemon. +# NOTE: we assume that rsyslogd's current working directory is +# ./tests (or the distcheck equivalent), in particlular that this +# config file resides in the testsuites subdirectory. +# rgerhards, 2009-05-27 +$ModLoad ../plugins/imdiag/.libs/imdiag +$IMDiagServerRun 13501 + +$template startupfile,"rsyslogd2.started" # trick to use relative path names! +:syslogtag, contains, "rsyslogd" ?startupfile + +$ErrorMessagesToStderr off diff --git a/tests/testsuites/dynfile_cachemiss.conf b/tests/testsuites/dynfile_cachemiss.conf new file mode 100644 index 00000000..273ff176 --- /dev/null +++ b/tests/testsuites/dynfile_cachemiss.conf @@ -0,0 +1,14 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:3%\n" +$template dynfile,"%msg:F,58:2%.log" # complete name is in message +$OMFileFlushOnTXEnd on +$DynaFileCacheSize 4 +$IncludeConfig rsyslog.action.1.include +local0.* ?dynfile;outfmt diff --git a/tests/testsuites/dynfile_invalid2.conf b/tests/testsuites/dynfile_invalid2.conf new file mode 100644 index 00000000..6d94c40d --- /dev/null +++ b/tests/testsuites/dynfile_invalid2.conf @@ -0,0 +1,14 @@ +# simple async writing test +# rgerhards, 2010-03-22 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:3%\n" +$template dynfile,"%msg:F,58:2%.log" # complete name is in message +$OMFileFlushOnTXEnd off +$DynaFileCacheSize 4 +$omfileFlushInterval 1 +local0.* ?dynfile;outfmt diff --git a/tests/testsuites/gzipwr_large.conf b/tests/testsuites/gzipwr_large.conf new file mode 100644 index 00000000..54ad3bb3 --- /dev/null +++ b/tests/testsuites/gzipwr_large.conf @@ -0,0 +1,15 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$MaxMessageSize 10k +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%,%msg:F,58:3%,%msg:F,58:4%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +$OMFileFlushOnTXEnd off +$OMFileZipLevel 6 +$OMFileIOBufferSize 256k +local0.* ?dynfile;outfmt diff --git a/tests/testsuites/gzipwr_large_dynfile.conf b/tests/testsuites/gzipwr_large_dynfile.conf new file mode 100644 index 00000000..3a1b255a --- /dev/null +++ b/tests/testsuites/gzipwr_large_dynfile.conf @@ -0,0 +1,17 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$MaxMessageSize 10k +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:3%,%msg:F,58:4%,%msg:F,58:5%\n" +$template dynfile,"rsyslog.out.%msg:F,58:2%.log" # use multiple dynafiles +$OMFileFlushOnTXEnd off +$OMFileZipLevel 6 +$OMFileIOBufferSize 256k +$DynaFileCacheSize 4 +$omfileFlushInterval 1 +local0.* ?dynfile;outfmt diff --git a/tests/testsuites/oversizeTag-1.parse1 b/tests/testsuites/oversizeTag-1.parse1 index 56510c63..d45ba1f2 100644 --- a/tests/testsuites/oversizeTag-1.parse1 +++ b/tests/testsuites/oversizeTag-1.parse1 @@ -1,3 +1,2 @@ -<38>Mar 27 19:06:53 source_server 0123456780123456780123456780123456789: MSG part -38,auth,info,Mar 27 19:06:53,source_server,0123456780123456780123456780123456789,0123456780123456780123456780123456789:, MSG part -# yet another real-life sample where we had some issues with +<38>Mar 27 19:06:53 source_server 0123456789012345678901234567890123456789: MSG part +38,auth,info,Mar 27 19:06:53,source_server,0123456789012345678901234567890123456789,0123456789012345678901234567890123456789:, MSG part diff --git a/tests/testsuites/parse-3164-buggyday.conf b/tests/testsuites/parse-3164-buggyday.conf new file mode 100644 index 00000000..937f423a --- /dev/null +++ b/tests/testsuites/parse-3164-buggyday.conf @@ -0,0 +1,8 @@ +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format that we can easily parse in expect +$template expect,"%PRI%,%syslogfacility-text%,%syslogseverity-text%,%timestamp:::date-rfc3164-buggyday%,%hostname%,%programname%,%syslogtag%,%msg%\n" +*.* :omstdout:;expect diff --git a/tests/testsuites/parse-nodate.conf b/tests/testsuites/parse-nodate.conf new file mode 100644 index 00000000..570638d9 --- /dev/null +++ b/tests/testsuites/parse-nodate.conf @@ -0,0 +1,14 @@ +# test is a test config that does not include the timestamp. This is necessary to +# test some illformed messages that do not contain a date. In that case, the system's +# current timestamp is used, and that of course is a bit hard to compare against +# a fixed template. So the solution in this case is to use a format that does +# not contain any timestamp. Maybe not optimal, but it works ;) +# rgerhards, 2010-03-19 +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format that we can easily parse +$template fmt,"%PRI%,%syslogfacility-text%,%syslogseverity-text%,%hostname%,%programname%,%syslogtag%,%msg%\n" +*.* :omstdout:;fmt diff --git a/tests/testsuites/parse3.conf b/tests/testsuites/parse3.conf new file mode 100644 index 00000000..8a3cb317 --- /dev/null +++ b/tests/testsuites/parse3.conf @@ -0,0 +1,10 @@ +# note: we need to strip off the TZ designator in the rfc3339 timestamp +# as this test otherwise fails in different timezones! +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format that we can easily parse in expect +$Template output,"%timereported:1:19:date-rfc3339,csv%, %hostname:::csv%, %programname:::csv%, %syslogtag:R,ERE,0,BLANK:[0-9]+--end:csv%, %syslogseverity:::csv%, %msg:::drop-last-lf,csv%\n" +*.* :omstdout:;output diff --git a/tests/testsuites/parse_invld_regex.conf b/tests/testsuites/parse_invld_regex.conf new file mode 100644 index 00000000..d18a2b3c --- /dev/null +++ b/tests/testsuites/parse_invld_regex.conf @@ -0,0 +1,10 @@ +# note: we need to strip off the TZ designator in the rfc3339 timestamp +# as this test otherwise fails in different timezones! +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# use a special format that we can easily parse in expect +$Template output,"%timereported:1:19:date-rfc3339,csv%, %hostname:::csv%, %programname:::csv%, %syslogtag:R,ERE,0,BLANK:[0-9+--end:csv%, %syslogseverity:::csv%, %msg:::drop-last-lf,csv%\n" +*.* :omstdout:;output diff --git a/tests/testsuites/random.conf b/tests/testsuites/random.conf new file mode 100644 index 00000000..a7079df1 --- /dev/null +++ b/tests/testsuites/random.conf @@ -0,0 +1,13 @@ +# we write to /dev/null, as we have no chance to verify the output +# in any case. What we really check is that rsyslogd does not +# segfault or otherwise abort. +# rgerhards, 2010-04-01 +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%rawmsg%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +*.* /dev/null diff --git a/tests/testsuites/reallife.parse3 b/tests/testsuites/reallife.parse3 new file mode 100644 index 00000000..dad3f56e --- /dev/null +++ b/tests/testsuites/reallife.parse3 @@ -0,0 +1,15 @@ +# New tests should be added to this file if there is no specific +# reason for not doing that. Initially, we could only handle one test +# case per file, but this restriction has been removed some time ago. +# So it is less troublesome (and easier to overlook) to have all related +# tests in a single file. +# This file contains a lot of real-life samples (of course mangled so +# that they can not be traced back to the original submitter). Note +# that IP addr 192.0.2.1 is specifically set aside for testing and +# documentation by IANA. +# rgerhards, 2009-10-19 +<175>Oct 16 2009 23:47:31 hostname tag This is a message +"2009-10-16T23:47:31", "hostname", "tag", "", "7", " This is a message" +# +<175>Oct 16 2009 23:47:31 hostname tag[1234] This is a message +"2009-10-16T23:47:31", "hostname", "tag", "1234", "7", " This is a message" diff --git a/tests/testsuites/samples.parse-3164-buggyday b/tests/testsuites/samples.parse-3164-buggyday new file mode 100644 index 00000000..e21df980 --- /dev/null +++ b/tests/testsuites/samples.parse-3164-buggyday @@ -0,0 +1,6 @@ +# in 3164-buggyday mode, we need to have a leading zero in front of the day +<38> Mar 7 19:06:53 example tag: testmessage (only date actually tested) +38,auth,info,Mar 07 19:06:53,example,tag,tag:, testmessage (only date actually tested) +# and now one with a complete date: +<38> Mar 17 19:06:53 example tag: testmessage (only date actually tested) +38,auth,info,Mar 17 19:06:53,example,tag,tag:, testmessage (only date actually tested) diff --git a/tests/testsuites/samples.parse-nodate b/tests/testsuites/samples.parse-nodate new file mode 100644 index 00000000..7f16181c --- /dev/null +++ b/tests/testsuites/samples.parse-nodate @@ -0,0 +1,6 @@ +<27>xapi: [error|xen3|15|Guest liveness monitor D:bca30ab3f1c1|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message) +27,daemon,err,localhost,xapi,xapi:, [error|xen3|15|Guest liveness monitor D:bca30ab3f1c1|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message) +# a message with just text (as permitted by rfc 3164) +# it is questionable if the current sample result is really correct as of 3164! +This is a message! +13,user,notice,This,is,is, a message! diff --git a/tests/testsuites/samples.parse_invld_regex b/tests/testsuites/samples.parse_invld_regex new file mode 100644 index 00000000..0d0e4ce3 --- /dev/null +++ b/tests/testsuites/samples.parse_invld_regex @@ -0,0 +1,16 @@ +# New tests should be added to this file if there is no specific +# reason for not doing that. Initially, we could only handle one test +# case per file, but this restriction has been removed some time ago. +# So it is less troublesome (and easier to overlook) to have all related +# tests in a single file. +# the actual message is not important. There is an error inside the conf +# file, and all messages will trigger the same problem. +# NOTE: it is correct that the "BAD REGULAR EXPRESSION" error message is +# *NOT* run through the rest of the propety replace, in specific through +# the CSV escaper. We do not do this because it could potentially lead +# to an obfuscated error message, and thus making problems hard to find. As +# this is a real error case, there is no problem in not obeying to the +# configured format. +# rgerhards, 2010-02-08 +<175>Feb 08 2008 23:47:31 hostname tag This is a message +"2008-02-08T23:47:31", "hostname", "tag", **NO MATCH** **BAD REGULAR EXPRESSION**, "7", " This is a message" diff --git a/tests/testsuites/samples.snare_ccoff_udp b/tests/testsuites/samples.snare_ccoff_udp new file mode 100644 index 00000000..010e44d5 --- /dev/null +++ b/tests/testsuites/samples.snare_ccoff_udp @@ -0,0 +1,14 @@ +# see comments in snare_ccoff_udp.conf +# note that some of these samples look pretty wild, but they are +# *real* cases (just mangled to anonymize them...) +# Sample 1 - note the absence of PRI! +windowsserver MSWinEventLog 1 Security 1167 Fri Mar 19 15:33:30 2010 540 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Successful Network Logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {79b6eb79-7bcc-8a2e-7dad-953c51dc00fd} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 10.11.11.3 Source Port: 3306 733\n +13,user,notice,localhost,windowsserver,windowsserver MSWinEventLog 1 Security 1167 Fri, Mar 19 15:33:30 2010 540 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Successful Network Logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {79b6eb79-7bcc-8a2e-7dad-953c51dc00fd} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 10.11.11.3 Source Port: 3306 733 +# Sample 2 +# the samples below need to be disabled for the "workaround patch" for the message +# parser to work. They need to be re-enabled once a final solution has been crafted +#windowsserver MSWinEventLog 1 Security 1166 Fri Mar 19 15:33:30 2010 576 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Special privileges assigned to new logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Privileges: SeSecurityPrivilege SeBackupPrivilege SeRestorePrivilege SeTakeOwnershipPrivilege SeDebugPrivilege SeSystemEnvironmentPrivilege SeLoadDriverPrivilege SeImpersonatePrivilege SeEnableDelegationPrivilege 732\n +#13,user,notice,localhost,windowsserver,windowsserver MSWinEventLog 1 Security 1166 Fri, Mar 19 15:33:30 2010 576 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Special privileges assigned to new logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Privileges: SeSecurityPrivilege SeBackupPrivilege SeRestorePrivilege SeTakeOwnershipPrivilege SeDebugPrivilege SeSystemEnvironmentPrivilege SeLoadDriverPrivilege SeImpersonatePrivilege SeEnableDelegationPrivilege 732 +# Sample 3 +#windowsserver MSWinEventLog 1 Security 1165 Fri Mar 19 15:33:30 2010 538 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff User Logoff: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF8830B) Logon Type: 3 731\n +#13,user,notice,localhost,windowsserver,windowsserver MSWinEventLog 1 Security 1165 Fri, Mar 19 15:33:30 2010 538 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff User Logoff: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF8830B) Logon Type: 3 731 diff --git a/tests/testsuites/samples.snare_ccoff_udp2 b/tests/testsuites/samples.snare_ccoff_udp2 new file mode 100644 index 00000000..337cd97c --- /dev/null +++ b/tests/testsuites/samples.snare_ccoff_udp2 @@ -0,0 +1,26 @@ +# see comments in snare_ccoff_udp.conf +# note that some of these samples look pretty wild, but they are +# *real* cases (just mangled to anonymize them...) +# +# NOTE +# The current responses are probably not correct (handling of messages without PRI). +# However, we keep them inside the test to be consistent. We should look at how +# PRI-less messages are handled and once we have fixed that, the test cases may need +# to be adapted. We do NOT try to preserve misbehaviour on such seriously malformed +# messages. +# +# this is a very simple test, though not snare-based +test +insert into windows (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('', 1, 'test',5, '20100321185328', '20100321185328', 1, '') +# and yet another one we have seen in practice +UX=Abcd-efg-hij-klmno; XXXXX=1111111111, Z123=192.12.231.245:11111, S1234=123456789, XXXXXX=111111111 +insert into windows (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values (' XXXXX=1111111111, Z123=192.12.231.245:11111, S1234=123456789, XXXXXX=111111111', 1, 'localhost',5, '20100321185328', '20100321185328', 1, 'UX=Abcd-efg-hij-klmno;') +# Sample 1 - note the absence of PRI! +windowsserver MSWinEventLog 1 Security 1167 Fri Mar 19 15:33:30 2010 540 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Successful Network Logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {79b6eb79-7bcc-8a2e-7dad-953c51dc00fd} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 10.11.11.3 Source Port: 3306 733\n +insert into windows (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values (' Mar 19 15:33:30 2010 540 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Successful Network Logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {79b6eb79-7bcc-8a2e-7dad-953c51dc00fd} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 10.11.11.3 Source Port: 3306 733', 1, 'localhost',5, '20100321185328', '20100321185328', 1, 'windowsserver MSWinEventLog 1 Security 1167 Fri') +# Sample 2 +windowsserver MSWinEventLog 1 Security 1166 Fri Mar 19 15:33:30 2010 576 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Special privileges assigned to new logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Privileges: SeSecurityPrivilege SeBackupPrivilege SeRestorePrivilege SeTakeOwnershipPrivilege SeDebugPrivilege SeSystemEnvironmentPrivilege SeLoadDriverPrivilege SeImpersonatePrivilege SeEnableDelegationPrivilege 732\n +insert into windows (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values (' Mar 19 15:33:30 2010 576 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff Special privileges assigned to new logon: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF88396) Privileges: SeSecurityPrivilege SeBackupPrivilege SeRestorePrivilege SeTakeOwnershipPrivilege SeDebugPrivilege SeSystemEnvironmentPrivilege SeLoadDriverPrivilege SeImpersonatePrivilege SeEnableDelegationPrivilege 732', 1, 'localhost',5, '20100321185328', '20100321185328', 1, 'windowsserver MSWinEventLog 1 Security 1166 Fri') +# Sample 3 +windowsserver MSWinEventLog 1 Security 1165 Fri Mar 19 15:33:30 2010 538 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff User Logoff: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF8830B) Logon Type: 3 731\n +insert into windows (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values (' Mar 19 15:33:30 2010 538 Security SYSTEM User Success Audit WINDOWSSERVER Logon/Logoff User Logoff: User Name: WINDOWSSERVER$ Domain: DOMX Logon ID: (0x0,0xF8830B) Logon Type: 3 731', 1, 'localhost',5, '20100321185328', '20100321185328', 1, 'windowsserver MSWinEventLog 1 Security 1165 Fri') diff --git a/tests/testsuites/snare.parse1 b/tests/testsuites/snare.parse1 new file mode 100644 index 00000000..550b0703 --- /dev/null +++ b/tests/testsuites/snare.parse1 @@ -0,0 +1,83 @@ +# some parse test build around data in snare-format +<141>Mar 10 09:30:20 zuse.xysystems.local MSWinEventLog\0111\011Security\011563\011Wed Mar 10 09:30:15 2010\011538\011Security\011XYWS011$\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011User Logoff: User Name: XYWS011$ Domain: XYZSYSTEMS Logon ID: (0x0,0x5984789C) Logon Type: 3 \011552 +141,local1,notice,Mar 10 09:30:20,zuse.xysystems.local,MSWinEventLog#0111#011Security#011563#011Wed,MSWinEventLog#0111#011Security#011563#011Wed, Mar 10 09:30:15 2010#011538#011Security#011XYWS011$#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011User Logoff: User Name: XYWS011$ Domain: XYZSYSTEMS Logon ID: (0x0,0x5984789C) Logon Type: 3 #011552 +# +# NEXT MESSAGE +# +Mar 10 09:30:20 zuse.xysystems.local MSWinEventLog\0111\011Security\011564\011Wed Mar 10 09:30:19 2010\011540\011Security\011BACKUP1$\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: BACKUP1$ Domain: XYZSYSTEMS Logon ID: (0x0,0x59848DB4) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {f6f65903-1932-d229-4b75-64816121d569} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.31 Source Port: 0 \011553 +13,user,notice,Mar 10 09:30:20,zuse.xysystems.local,MSWinEventLog#0111#011Security#011564#011Wed,MSWinEventLog#0111#011Security#011564#011Wed, Mar 10 09:30:19 2010#011540#011Security#011BACKUP1$#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: BACKUP1$ Domain: XYZSYSTEMS Logon ID: (0x0,0x59848DB4) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {f6f65903-1932-d229-4b75-64816121d569} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.31 Source Port: 0 #011553 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011566\011Wed Mar 10 09:30:21 2010\011540\011Security\011aadminps\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: aadminps Domain: XYSYSTEMS Logon ID: (0x0,0x5984973C) Logon Type: 3 Logon Process: Authz Authentication Package: Kerberos Workstation Name: ZUSE Logon GUID: - Caller User Name: ZUSE$ Caller Domain: XYSYSTEMS Caller Logon ID: (0x0,0x3E7) Caller Process ID: 1004 Transited Services: - Source Network Address: - Source Port: - \011555 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011566#011Wed,MSWinEventLog#0111#011Security#011566#011Wed, Mar 10 09:30:21 2010#011540#011Security#011aadminps#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: aadminps Domain: XYSYSTEMS Logon ID: (0x0,0x5984973C) Logon Type: 3 Logon Process: Authz Authentication Package: Kerberos Workstation Name: ZUSE Logon GUID: - Caller User Name: ZUSE$ Caller Domain: XYSYSTEMS Caller Logon ID: (0x0,0x3E7) Caller Process ID: 1004 Transited Services: - Source Network Address: - Source Port: - #011555 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011567\011Wed Mar 10 09:30:21 2010\011538\011Security\011aadminps\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011User Logoff: User Name: aadminps Domain: XYSYSTEMS Logon ID: (0x0,0x5984973C) Logon Type: 3 \011556 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011567#011Wed,MSWinEventLog#0111#011Security#011567#011Wed, Mar 10 09:30:21 2010#011538#011Security#011aadminps#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011User Logoff: User Name: aadminps Domain: XYSYSTEMS Logon ID: (0x0,0x5984973C) Logon Type: 3 #011556 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011568\011Wed Mar 10 09:30:25 2010\011540\011Security\011ANONYMOUS LOGON\011Well Known Group\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: Domain: Logon ID: (0x0,0x5984AB6F) Logon Type: 3 Logon Process: NtLmSsp Authentication Package: NTLM Workstation Name: XYWS083 Logon GUID: - Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.3.91 Source Port: 0 \011557 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011568#011Wed,MSWinEventLog#0111#011Security#011568#011Wed, Mar 10 09:30:25 2010#011540#011Security#011ANONYMOUS LOGON#011Well Known Group#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: Domain: Logon ID: (0x0,0x5984AB6F) Logon Type: 3 Logon Process: NtLmSsp Authentication Package: NTLM Workstation Name: XYWS083 Logon GUID: - Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.3.91 Source Port: 0 #011557 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011569\011Wed Mar 10 09:30:25 2010\011540\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984ACA7) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2318 \011558 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011569#011Wed,MSWinEventLog#0111#011Security#011569#011Wed, Mar 10 09:30:25 2010#011540#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984ACA7) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2318 #011558 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011570\011Wed Mar 10 09:30:25 2010\011538\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984ACA7) Logon Type: 3 \011559 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011570#011Wed,MSWinEventLog#0111#011Security#011570#011Wed, Mar 10 09:30:25 2010#011538#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984ACA7) Logon Type: 3 #011559 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011571\011Wed Mar 10 09:30:25 2010\011540\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AD7C) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2319 \011560\ +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011571#011Wed,MSWinEventLog#0111#011Security#011571#011Wed, Mar 10 09:30:25 2010#011540#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AD7C) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2319 #011560 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011572\011Wed Mar 10 09:30:25 2010\011538\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AD7C) Logon Type: 3 \011561 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011572#011Wed,MSWinEventLog#0111#011Security#011572#011Wed, Mar 10 09:30:25 2010#011538#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AD7C) Logon Type: 3 #011561 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011573\011Wed Mar 10 09:30:25 2010\011680\011Security\011ettore.trezzani\011User\011Success Audit\011ZUSE\011Account Logon\011\011Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon account: ettore.trezzani Source Workstation: XYWS083 Error Code: 0x0 \011562 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011573#011Wed,MSWinEventLog#0111#011Security#011573#011Wed, Mar 10 09:30:25 2010#011680#011Security#011ettore.trezzani#011User#011Success Audit#011ZUSE#011Account Logon#011#011Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon account: ettore.trezzani Source Workstation: XYWS083 Error Code: 0x0 #011562 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011574\011Wed Mar 10 09:30:25 2010\011540\011Security\011ettore.trezzani\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: ettore.trezzani Domain: XYSYSTEMS Logon ID: (0x0,0x5984ADD5) Logon Type: 3 Logon Process: NtLmSsp Authentication Package: NTLM Workstation Name: XYWS083 Logon GUID: - Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.3.91 Source Port: 0 \011563 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011574#011Wed,MSWinEventLog#0111#011Security#011574#011Wed, Mar 10 09:30:25 2010#011540#011Security#011ettore.trezzani#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: ettore.trezzani Domain: XYSYSTEMS Logon ID: (0x0,0x5984ADD5) Logon Type: 3 Logon Process: NtLmSsp Authentication Package: NTLM Workstation Name: XYWS083 Logon GUID: - Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.3.91 Source Port: 0 #011563 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011575\011Wed Mar 10 09:30:25 2010\011540\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AE49) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2320 \011564 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011575#011Wed,MSWinEventLog#0111#011Security#011575#011Wed, Mar 10 09:30:25 2010#011540#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AE49) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2320 #011564 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011576\011Wed Mar 10 09:30:25 2010\011538\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AE49) Logon Type: 3 \011565 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011576#011Wed,MSWinEventLog#0111#011Security#011576#011Wed, Mar 10 09:30:25 2010#011538#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AE49) Logon Type: 3 #011565 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011577\011Wed Mar 10 09:30:25 2010\011540\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AF00) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2321 \011566 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011577#011Wed,MSWinEventLog#0111#011Security#011577#011Wed, Mar 10 09:30:25 2010#011540#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AF00) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {20014d9a-ce6c-6834-d1ed-607c08f0b6a7} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.0.15 Source Port: 2321 #011566 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011578\011Wed Mar 10 09:30:25 2010\011538\011Security\011SYSTEM\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AF00) Logon Type: 3 \011567 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011578#011Wed,MSWinEventLog#0111#011Security#011578#011Wed, Mar 10 09:30:25 2010#011538#011Security#011SYSTEM#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011User Logoff: User Name: ZUSE$ Domain: XYSYSTEMS Logon ID: (0x0,0x5984AF00) Logon Type: 3 #011567 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:25 zuse.xysystems.local MSWinEventLog\0111\011Security\011579\011Wed Mar 10 09:30:25 2010\011538\011Security\011ANONYMOUS LOGON\011Well Known Group\011Success Audit\011ZUSE\011Logon/Logoff\011\011User Logoff: User Name: ANONYMOUS LOGON Domain: NT AUTHORITY Logon ID: (0x0,0x5984AB6F) Logon Type: 3 \011568 +141,local1,notice,Mar 10 09:30:25,zuse.xysystems.local,MSWinEventLog#0111#011Security#011579#011Wed,MSWinEventLog#0111#011Security#011579#011Wed, Mar 10 09:30:25 2010#011538#011Security#011ANONYMOUS LOGON#011Well Known Group#011Success Audit#011ZUSE#011Logon/Logoff#011#011User Logoff: User Name: ANONYMOUS LOGON Domain: NT AUTHORITY Logon ID: (0x0,0x5984AB6F) Logon Type: 3 #011568 +# +# NEXT MESSAGE +# +<141>Mar 10 09:30:30 zuse.xysystems.local MSWinEventLog\0111\011Security\011580\011Wed Mar 10 09:30:29 2010\011540\011Security\011XYWSBADGE$\011User\011Success Audit\011ZUSE\011Logon/Logoff\011\011Successful Network Logon: User Name: XYWSBADGE$ Domain: XYSYSTEMS Logon ID: (0x0,0x59852D73) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {4bc3c075-5a77-4648-5822-bfdf88b4c211} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.3.18 Source Port: 0 \011569 +141,local1,notice,Mar 10 09:30:30,zuse.xysystems.local,MSWinEventLog#0111#011Security#011580#011Wed,MSWinEventLog#0111#011Security#011580#011Wed, Mar 10 09:30:29 2010#011540#011Security#011XYWSBADGE$#011User#011Success Audit#011ZUSE#011Logon/Logoff#011#011Successful Network Logon: User Name: XYWSBADGE$ Domain: XYSYSTEMS Logon ID: (0x0,0x59852D73) Logon Type: 3 Logon Process: Kerberos Authentication Package: Kerberos Workstation Name: Logon GUID: {4bc3c075-5a77-4648-5822-bfdf88b4c211} Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 172.16.3.18 Source Port: 0 #011569 diff --git a/tests/testsuites/snare_ccoff_udp.conf b/tests/testsuites/snare_ccoff_udp.conf new file mode 100644 index 00000000..6abbedf4 --- /dev/null +++ b/tests/testsuites/snare_ccoff_udp.conf @@ -0,0 +1,21 @@ +# This test some real-world snare cases. I don't like snare (no wonder +# as I have written EventReporter, the ultimate Windows-to-Syslog tool), +# but besides that snare generates severely malformed messages that +# really stress-test the rsyslog engine. They deserve to be beaten by someone ;) +# This test needs to be run over UDP only, as snare puts LF INTO some of the messages, +# which makes it impossible to try these out via traditional syslog/tcp +# added 2010-03-21 rgerhards +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# snare usses HT as field delimiter, so many users have turned off +# control character escaping to make any sense at all from these messages... +$EscapeControlCharactersOnReceive off + +# use a special format that we can easily check. We do NOT include a timestamp because +# the malformed snare messages usually do not contain one (and we can not check against +# the system time in our test cases). +$template fmt,"%PRI%,%syslogfacility-text%,%syslogseverity-text%,%hostname%,%programname%,%syslogtag%,%msg%\n" +*.* :omstdout:;fmt diff --git a/tests/testsuites/snare_ccoff_udp2.conf b/tests/testsuites/snare_ccoff_udp2.conf new file mode 100644 index 00000000..9115c14f --- /dev/null +++ b/tests/testsuites/snare_ccoff_udp2.conf @@ -0,0 +1,17 @@ +# Similar to snare_ccoff_udp_2, but with a different template. This template +# has triggered problems in the past, thus a test is granted. +# added 2010-03-21 rgerhards +$ModLoad ../plugins/omstdout/.libs/omstdout +$IncludeConfig nettest.input.conf # This picks the to be tested input from the test driver! + +$ErrorMessagesToStderr off + +# snare usses HT as field delimiter, so many users have turned off +# control character escaping to make any sense at all from these messages... +$EscapeControlCharactersOnReceive off + +# we need to use a fixed timestamp, as otherwise we can not compare :( +# This could be improved in later versions of the testing tools, but requires +# modification to the rsyslog core... +$template fmt,"insert into windows (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg:::space-cc%', %syslogfacility%, '%HOSTNAME%',%syslogpriority%, '20100321185328', '20100321185328', %iut%, '%syslogtag:::space-cc%')\n",sql +*.* :omstdout:;fmt diff --git a/tests/testsuites/weird.parse1 b/tests/testsuites/weird.parse1 new file mode 100644 index 00000000..e8b90c74 --- /dev/null +++ b/tests/testsuites/weird.parse1 @@ -0,0 +1,37 @@ +# some really weird samples, some of them seen in practice, +# some other deliberately generated. The main point is that they +# should not cause an abort... +<14>Aug 30 23:00:05 X4711 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA +14,user,info,Aug 30 23:00:05,X4711,AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA,AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA, +# important: the following line has a SP at the end of the line! +<14>Aug 30 23:00:05 X4711 +14,user,info,Aug 30 23:00:05,X4711,,, +# and this one NOT +<14>Aug 30 23:00:05 X4711 +14,user,info,Aug 30 23:00:05,X4711,,, +# there is a SP at the end of the line +<14>Aug 30 23:00:05 +14,user,info,Aug 30 23:00:05,localhost,,, +# and here is no SP at the end of the line +<14>Aug 30 23:00:05 +14,user,info,Aug 30 23:00:05,localhost,,, +# unfortunately, I can not test missing dates with this test suite, because +# we would have the current date in the response, which we can not check against +# +# and now the same tests with RFC3339 data - this can make a difference +# as a different date parser is involved. +# +<14>2010-08-30T23:00:05Z X4711 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA +14,user,info,Aug 30 23:00:05,X4711,AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA,AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA, +# important: the following line has a SP at the end of the line! +<14>2010-08-30T23:00:05Z X4711 +14,user,info,Aug 30 23:00:05,X4711,,, +# and this one NOT +<14>2010-08-30T23:00:05Z X4711 +14,user,info,Aug 30 23:00:05,X4711,,, +# there is a SP at the end of the line +<14>2010-08-30T23:00:05Z +14,user,info,Aug 30 23:00:05,localhost,,, +# and here is no SP at the end of the line +<14>2010-08-30T23:00:05Z +14,user,info,Aug 30 23:00:05,localhost,,, diff --git a/tests/testsuites/wr_large.conf b/tests/testsuites/wr_large.conf new file mode 100644 index 00000000..b64f132b --- /dev/null +++ b/tests/testsuites/wr_large.conf @@ -0,0 +1,16 @@ +# simple async writing test +# rgerhards, 2010-03-09 +$MaxMessageSize 10k +$IncludeConfig diag-common.conf + +$ModLoad ../plugins/imtcp/.libs/imtcp +$MainMsgQueueTimeoutShutdown 10000 +$InputTCPServerRun 13514 + +$template outfmt,"%msg:F,58:2%,%msg:F,58:3%,%msg:F,58:4%\n" +$template dynfile,"rsyslog.out.log" # trick to use relative path names! +$OMFileFlushOnTXEnd off +$OMFileFlushInterval 2 +$OMFileIOBufferSize 256k +$IncludeConfig rsyslog.action.1.include +local0.* ?dynfile;outfmt diff --git a/tests/threadingmqaq.sh b/tests/threadingmqaq.sh index 9f2ea793..b7764821 100755 --- a/tests/threadingmqaq.sh +++ b/tests/threadingmqaq.sh @@ -9,7 +9,7 @@ echo TEST: threadingmqaq.sh - main/action queue concurrency source $srcdir/diag.sh init source $srcdir/diag.sh startup threadingmqaq.conf -#source $srcdir/diag.sh tcpflood 127.0.0.1 13514 2 100000 +#source $srcdir/diag.sh tcpflood -c2 -m100000 #source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages source $srcdir/diag.sh injectmsg 0 100000 source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages diff --git a/tests/wr_large.sh b/tests/wr_large.sh new file mode 100755 index 00000000..84f12989 --- /dev/null +++ b/tests/wr_large.sh @@ -0,0 +1,16 @@ +# This tests async writing large data records. We use up to 10K +# record size. + +# added 2010-03-10 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +cat rsyslog.action.1.include +source $srcdir/diag.sh init +source $srcdir/diag.sh startup wr_large.conf +# send 4000 messages of 10.000bytes plus header max, randomized +source $srcdir/diag.sh tcpflood -m4000 -r -d10000 -P129 +sleep 1 # due to large messages, we need this time for the tcp receiver to settle... +source $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +source $srcdir/diag.sh wait-shutdown # and wait for it to terminate +source $srcdir/diag.sh seq-check 0 3999 -E +source $srcdir/diag.sh exit diff --git a/tests/wr_large_async.sh b/tests/wr_large_async.sh new file mode 100755 index 00000000..88a1acf8 --- /dev/null +++ b/tests/wr_large_async.sh @@ -0,0 +1,14 @@ +# This tests async writing large data records. We use up to 10K +# record size. + +# added 2010-03-10 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[wr_large_async.sh\]: test for file writing for large message sets +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +echo "\$OMFileAsyncWriting on" > rsyslog.action.1.include +source $srcdir/wr_large.sh diff --git a/tests/wr_large_sync.sh b/tests/wr_large_sync.sh new file mode 100755 index 00000000..a1c4fd77 --- /dev/null +++ b/tests/wr_large_sync.sh @@ -0,0 +1,14 @@ +# This tests async writing large data records. We use up to 10K +# record size. + +# added 2010-03-10 by Rgerhards +# +# This file is part of the rsyslog project, released under GPLv3 +echo =============================================================================== +echo TEST: \[wr_large_sync.sh\]: test for file writing for large message sets +source $srcdir/diag.sh init +# uncomment for debugging support: +#export RSYSLOG_DEBUG="debug nostdout noprintmutexaction" +#export RSYSLOG_DEBUGLOG="log" +echo "\$OMFileAsyncWriting off" > rsyslog.action.1.include +source $srcdir/wr_large.sh diff --git a/tools/Makefile.am b/tools/Makefile.am index f0f9afab..1497d3be 100644 --- a/tools/Makefile.am +++ b/tools/Makefile.am @@ -13,6 +13,8 @@ rsyslogd_SOURCES = \ omfwd.h \ omfile.c \ omfile.h \ + ompipe.c \ + ompipe.h \ omdiscard.c \ omdiscard.h \ iminternal.c \ diff --git a/tools/msggen.c b/tools/msggen.c index 06244c18..29ade3a7 100644 --- a/tools/msggen.c +++ b/tools/msggen.c @@ -21,6 +21,7 @@ * A copy of the GPL can be found in the file "COPYING" in this distribution. */ +#include "config.h" #include <stdio.h> #include <syslog.h> diff --git a/tools/omfile.c b/tools/omfile.c index efdaead4..74b22019 100644 --- a/tools/omfile.c +++ b/tools/omfile.c @@ -5,10 +5,6 @@ * works! * * File begun on 2007-07-21 by RGerhards (extracted from syslogd.c) - * This file is under development and has not yet arrived at being fully - * self-contained and a real object. So far, it is mostly an excerpt - * of the "old" message code without any modifications. However, it - * helps to have things at the right place one we go to the meat of it. * * A large re-write of this file was done in June, 2009. The focus was * to introduce many more features (like zipped writing), clean up the code @@ -16,6 +12,10 @@ * solid basis for the next three to five years to come. During it, bugs * may have been introduced ;) -- rgerhards, 2009-06-04 * + * Note that as of 2010-02-28 this module does no longer handle + * pipes. These have been moved to ompipe, to reduced the entanglement + * between the two different functionalities. -- rgerhards + * * Copyright 2007-2009 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. @@ -79,13 +79,15 @@ DEFobjCurrIf(strm) struct s_dynaFileCacheEntry { uchar *pName; /* name currently open, if dynamic name */ strm_t *pStrm; /* our output stream */ - time_t lastUsed; /* for LRU - last access */ // TODO: perforamcne change to counter (see other comment!) + time_t lastUsed; /* for LRU - last access */ }; typedef struct s_dynaFileCacheEntry dynaFileCacheEntry; #define IOBUF_DFLT_SIZE 1024 /* default size for io buffers */ #define FLUSH_INTRVL_DFLT 1 /* default buffer flush interval (in seconds) */ +#define USE_ASYNCWRITER_DFLT 0 /* default buffer use async writer */ +#define FLUSHONTX_DFLT 1 /* default for flush on TX end */ #define DFLT_bForceChown 0 /* globals for default values */ @@ -101,10 +103,11 @@ static uid_t dirGID; /* GID to be used for newly created directories */ static int bCreateDirs = 1;/* auto-create directories for dynaFiles: 0 - no, 1 - yes */ static int bEnableSync = 0;/* enable syncing of files (no dash in front of pathname in conf): 0 - no, 1 - yes */ static int iZipLevel = 0; /* zip compression mode (0..9 as usual) */ -static bool bFlushOnTXEnd = 1;/* flush write buffers when transaction has ended? */ +static bool bFlushOnTXEnd = FLUSHONTX_DFLT;/* flush write buffers when transaction has ended? */ static int64 iIOBufSize = IOBUF_DFLT_SIZE; /* size of an io buffer */ static int iFlushInterval = FLUSH_INTRVL_DFLT; /* how often flush the output buffer on inactivity? */ -static uchar *pszTplName = NULL; /* name of the default template to use */ +static int bUseAsyncWriter = USE_ASYNCWRITER_DFLT; /* should we enable asynchronous writing? */ +uchar *pszFileDfltTplName = NULL; /* name of the default template to use */ /* end globals for default values */ @@ -137,6 +140,7 @@ typedef struct _instanceData { int iIOBufSize; /* size of associated io buffer */ int iFlushInterval; /* how fast flush buffer on inactivity? */ bool bFlushOnTXEnd; /* flush write buffers when transaction has ended? */ + bool bUseAsyncWriter; /* use async stream writer? */ } instanceData; @@ -150,28 +154,24 @@ ENDisCompatibleWithFeature BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo if(pData->bDynamicName) { - dbgprintf("[dynamic]\n\ttemplate='%s'" - "\tfile cache size=%d\n" - "\tcreate directories: %s\n" - "\tfile owner %d, group %d\n" - "\tdirectory owner %d, group %d\n" - "\tforce chown() for all files: %s\n" - "\tdir create mode 0%3.3o, file create mode 0%3.3o\n" - "\tfail if owner/group can not be set: %s\n", - pData->f_fname, - pData->iDynaFileCacheSize, - pData->bCreateDirs ? "yes" : "no", - pData->fileUID, pData->fileGID, - pData->dirUID, pData->dirGID, - pData->bForceChown ? "yes" : "no", - pData->fDirCreateMode, pData->fCreateMode, - pData->bFailOnChown ? "yes" : "no" - ); + dbgprintf("[dynamic]\n"); } else { /* regular file */ - dbgprintf("%s", pData->f_fname); - if (pData->pStrm == NULL) - dbgprintf(" (unused)"); + dbgprintf("%s%s\n", pData->f_fname, + (pData->pStrm == NULL) ? " (unused)" : ""); } + + dbgprintf("\ttemplate='%s'\n", pData->f_fname); + dbgprintf("\tuse async writer=%d\n", pData->bUseAsyncWriter); + dbgprintf("\tflush on TX end=%d\n", pData->bFlushOnTXEnd); + dbgprintf("\tflush interval=%d\n", pData->iFlushInterval); + dbgprintf("\tfile cache size=%d\n", pData->iDynaFileCacheSize); + dbgprintf("\tcreate directories: %s\n", pData->bCreateDirs ? "yes" : "no"); + dbgprintf("\tfile owner %d, group %d\n", pData->fileUID, pData->fileGID); + dbgprintf("\tforce chown() for all files: %s\n", pData->bForceChown ? "yes" : "no"); + dbgprintf("\tdirectory owner %d, group %d\n", pData->dirUID, pData->dirGID); + dbgprintf("\tdir create mode 0%3.3o, file create mode 0%3.3o\n", + pData->fDirCreateMode, pData->fCreateMode); + dbgprintf("\tfail if owner/group can not be set: %s\n", pData->bFailOnChown ? "yes" : "no"); ENDdbgPrintInstInfo @@ -263,7 +263,7 @@ static rsRetVal cflineParseOutchannel(instanceData *pData, uchar* p, omodStringR pData->pszSizeLimitCmd = pOch->cmdOnSizeLimit; iRet = cflineParseTemplateName(&p, pOMSR, iEntry, iTplOpts, - (pszTplName == NULL) ? (uchar*)"RSYSLOG_FileFormat" : pszTplName); + (pszFileDfltTplName == NULL) ? (uchar*)"RSYSLOG_FileFormat" : pszFileDfltTplName); finalize_it: RETiRet; @@ -288,17 +288,18 @@ dynaFileDelCacheEntry(dynaFileCacheEntry **pCache, int iEntry, int bFreeEntry) DBGPRINTF("Removed entry %d for file '%s' from dynaCache.\n", iEntry, pCache[iEntry]->pName == NULL ? UCHAR_CONSTANT("[OPEN FAILED]") : pCache[iEntry]->pName); - /* if the name is NULL, this is an improperly initilized entry which - * needs to be discarded. In this case, neither the file is to be closed - * not the name to be freed. - */ + if(pCache[iEntry]->pName != NULL) { - if(pCache[iEntry]->pStrm != NULL) - strm.Destruct(&pCache[iEntry]->pStrm); d_free(pCache[iEntry]->pName); pCache[iEntry]->pName = NULL; } + if(pCache[iEntry]->pStrm != NULL) { + strm.Destruct(&pCache[iEntry]->pStrm); + if(pCache[iEntry]->pStrm != NULL) /* safety check -- TODO: remove if no longer necessary */ + abort(); + } + if(bFreeEntry) { d_free(pCache[iEntry]); pCache[iEntry] = NULL; @@ -323,6 +324,7 @@ dynaFileFreeCacheEntries(instanceData *pData) for(i = 0 ; i < pData->iCurrCacheSize ; ++i) { dynaFileDelCacheEntry(pData->dynCache, i, 1); } + pData->iCurrElt = -1; /* invalidate current element */ ENDfunc; } @@ -432,7 +434,7 @@ prepareFile(instanceData *pData, uchar *newFileName) * async processing, which is a real performance waste if we do not do buffered * writes! -- rgerhards, 2009-07-06 */ - if(!pData->bFlushOnTXEnd) + if(pData->bUseAsyncWriter) CHKiRet(strm.SetiFlushInterval(pData->pStrm, pData->iFlushInterval)); if(pData->pszSizeLimitCmd != NULL) CHKiRet(strm.SetpszSizeLimitCmd(pData->pStrm, ustrdup(pData->pszSizeLimitCmd))); @@ -486,11 +488,11 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) iOldest = 0; /* we assume the first element to be the oldest - that will change as we loop */ ttOldest = time(NULL) + 1; /* there must always be an older one */ for(i = 0 ; i < pData->iCurrCacheSize ; ++i) { - if(pCache[i] == NULL) { + if(pCache[i] == NULL || pCache[i]->pName == NULL) { if(iFirstFree == -1) iFirstFree = i; } else { /* got an element, let's see if it matches */ - if(!ustrcmp(newFileName, pCache[i]->pName)) { + if(!ustrcmp(newFileName, pCache[i]->pName)) { // RG: name == NULL? /* we found our element! */ pData->pStrm = pCache[i]->pStrm; pData->iCurrElt = i; @@ -506,23 +508,45 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) } /* we have not found an entry */ + + /* invalidate iCurrElt as we may error-exit out of this function when the currrent + * iCurrElt has been freed or otherwise become unusable. This is a precaution, and + * performance-wise it may be better to do that in each of the exits. However, that + * is error-prone, so I prefer to do it here. -- rgerhards, 2010-03-02 + */ + pData->iCurrElt = -1; + /* similarly, we need to set the current pStrm to NULL, because otherwise, if prepareFile() fails, + * we may end up using an old stream. This bug depends on how exactly prepareFile fails, + * but it* could be triggered in the common case of a failed open() system call. + * rgerhards, 2010-03-22 + */ + pData->pStrm = NULL; + if(iFirstFree == -1 && (pData->iCurrCacheSize < pData->iDynaFileCacheSize)) { /* there is space left, so set it to that index */ iFirstFree = pData->iCurrCacheSize++; } +// RG: this is the begin of a potential problem area + /* Note that the following code sequence does not work with the cache entry itself, + * but rather with pData->pStrm, the (sole) stream pointer in the non-dynafile case. + * The cache array is only updated after the open was successful. -- rgerhards, 2010-03-21 + */ if(iFirstFree == -1) { dynaFileDelCacheEntry(pCache, iOldest, 0); iFirstFree = iOldest; /* this one *is* now free ;) */ } else { /* we need to allocate memory for the cache structure */ + /* TODO: performance note: we could alloc all entries on startup, thus saving malloc + * overhead -- this may be something to consider in v5... + */ CHKmalloc(pCache[iFirstFree] = (dynaFileCacheEntry*) calloc(1, sizeof(dynaFileCacheEntry))); } /* Ok, we finally can open the file */ localRet = prepareFile(pData, newFileName); /* ignore exact error, we check fd below */ - /* file is either open now or an error state set */ + /* file is either open now or an error state set */ // RG: better check localRet? if(pData->pStrm == NULL) { /* do not report anything if the message is an internally-generated * message. Otherwise, we could run into a never-ending loop. The bad @@ -533,13 +557,15 @@ prepareDynFile(instanceData *pData, uchar *newFileName, unsigned iMsgOpts) } else { errmsg.LogError(0, NO_ERRCODE, "Could not open dynamic file '%s' - discarding message", newFileName); } - dynaFileDelCacheEntry(pCache, iFirstFree, 1); ABORT_FINALIZE(localRet); } - CHKmalloc(pCache[iFirstFree]->pName = ustrdup(newFileName)); + if((pCache[iFirstFree]->pName = ustrdup(newFileName)) == NULL) { + strm.Destruct(&pData->pStrm); /* need to free failed entry! */ + ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); + } pCache[iFirstFree]->pStrm = pData->pStrm; - pCache[iFirstFree]->lastUsed = time(NULL); // monotonically increasing value! TODO: performance + pCache[iFirstFree]->lastUsed = time(NULL); pData->iCurrElt = iFirstFree; DBGPRINTF("Added new entry %d for file cache, file '%s'.\n", iFirstFree, newFileName); @@ -640,7 +666,7 @@ ENDdoAction BEGINparseSelectorAct CODESTARTparseSelectorAct - if(!(*p == '$' || *p == '?' || *p == '|' || *p == '/' || *p == '-')) + if(!(*p == '$' || *p == '?' || *p == '/' || *p == '.' || *p == '-')) ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); CHKiRet(createInstance(&pData)); @@ -673,7 +699,7 @@ CODESTARTparseSelectorAct CODE_STD_STRING_REQUESTparseSelectorAct(2) ++p; /* eat '?' */ CHKiRet(cflineParseFileName(p, (uchar*) pData->f_fname, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, - (pszTplName == NULL) ? (uchar*)"RSYSLOG_FileFormat" : pszTplName)); + (pszFileDfltTplName == NULL) ? (uchar*)"RSYSLOG_FileFormat" : pszFileDfltTplName)); /* "filename" is actually a template name, we need this as string 1. So let's add it * to the pOMSR. -- rgerhards, 2007-07-27 */ @@ -686,8 +712,11 @@ CODESTARTparseSelectorAct calloc(iDynaFileCacheSize, sizeof(dynaFileCacheEntry*))); break; - case '|': + /* case '|': while pipe support has been removed, I leave the code in in case we + * need high-performance pipes at a later stage (unlikely). -- rgerhards, 2010-02-28 + */ case '/': + case '.': CODE_STD_STRING_REQUESTparseSelectorAct(1) /* we now have *almost* the same semantics for files and pipes, but we still need * to know we deal with a pipe, because we must do non-blocking opens in that case @@ -701,7 +730,7 @@ CODESTARTparseSelectorAct } CHKiRet(cflineParseFileName(p, (uchar*) pData->f_fname, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, - (pszTplName == NULL) ? (uchar*)"RSYSLOG_FileFormat" : pszTplName)); + (pszFileDfltTplName == NULL) ? (uchar*)"RSYSLOG_FileFormat" : pszFileDfltTplName)); pData->bDynamicName = 0; break; default: @@ -723,6 +752,7 @@ CODESTARTparseSelectorAct pData->bFlushOnTXEnd = bFlushOnTXEnd; pData->iIOBufSize = (int) iIOBufSize; pData->iFlushInterval = iFlushInterval; + pData->bUseAsyncWriter = bUseAsyncWriter; if(pData->bDynamicName == 0) { /* try open and emit error message if not possible. At this stage, we ignore the @@ -756,12 +786,13 @@ static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __a bCreateDirs = 1; bEnableSync = 0; iZipLevel = 0; - bFlushOnTXEnd = 1; + bFlushOnTXEnd = FLUSHONTX_DFLT; iIOBufSize = IOBUF_DFLT_SIZE; iFlushInterval = FLUSH_INTRVL_DFLT; - if(pszTplName != NULL) { - free(pszTplName); - pszTplName = NULL; + bUseAsyncWriter = USE_ASYNCWRITER_DFLT; + if(pszFileDfltTplName != NULL) { + free(pszFileDfltTplName); + pszFileDfltTplName = NULL; } return RS_RET_OK; @@ -772,7 +803,6 @@ BEGINdoHUP CODESTARTdoHUP if(pData->bDynamicName) { dynaFileFreeCacheEntries(pData); - pData->iCurrElt = -1; /* invalidate current element */ } else { if(pData->pStrm != NULL) { strm.Destruct(&pData->pStrm); @@ -786,7 +816,7 @@ BEGINmodExit CODESTARTmodExit objRelease(errmsg, CORE_COMPONENT); objRelease(strm, CORE_COMPONENT); - free(pszTplName); + free(pszFileDfltTplName); ENDmodExit @@ -806,6 +836,7 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(omsdRegCFSLineHdlr((uchar *)"dynafilecachesize", 0, eCmdHdlrInt, (void*) setDynaFileCacheSize, NULL, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"omfileziplevel", 0, eCmdHdlrInt, NULL, &iZipLevel, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"omfileflushinterval", 0, eCmdHdlrInt, NULL, &iFlushInterval, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"omfileasyncwriting", 0, eCmdHdlrBinary, NULL, &bUseAsyncWriter, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"omfileflushontxend", 0, eCmdHdlrBinary, NULL, &bFlushOnTXEnd, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"omfileiobuffersize", 0, eCmdHdlrSize, NULL, &iIOBufSize, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"dirowner", 0, eCmdHdlrUID, NULL, &dirUID, STD_LOADABLE_MODULE_ID)); @@ -818,7 +849,7 @@ CODEmodInit_QueryRegCFSLineHdlr CHKiRet(omsdRegCFSLineHdlr((uchar *)"failonchownfailure", 0, eCmdHdlrBinary, NULL, &bFailOnChown, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"omfileForceChown", 0, eCmdHdlrBinary, NULL, &bForceChown, STD_LOADABLE_MODULE_ID)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"actionfileenablesync", 0, eCmdHdlrBinary, NULL, &bEnableSync, STD_LOADABLE_MODULE_ID)); - CHKiRet(regCfSysLineHdlr((uchar *)"actionfiledefaulttemplate", 0, eCmdHdlrGetWord, NULL, &pszTplName, NULL)); + CHKiRet(regCfSysLineHdlr((uchar *)"actionfiledefaulttemplate", 0, eCmdHdlrGetWord, NULL, &pszFileDfltTplName, NULL)); CHKiRet(omsdRegCFSLineHdlr((uchar *)"resetconfigvariables", 1, eCmdHdlrCustomHandler, resetConfigVariables, NULL, STD_LOADABLE_MODULE_ID)); ENDmodInit /* vi:set ai: diff --git a/tools/omfile.h b/tools/omfile.h index 03e081f3..8dca6a88 100644 --- a/tools/omfile.h +++ b/tools/omfile.h @@ -3,7 +3,7 @@ * * File begun on 2007-07-21 by RGerhards (extracted from syslogd.c) * - * Copyright 2007 Rainer Gerhards and Adiscon GmbH. + * Copyright 2007-2010 Rainer Gerhards and Adiscon GmbH. * * This file is part of rsyslog. * @@ -28,7 +28,11 @@ /* prototypes */ rsRetVal modInitFile(int iIFVersRequested __attribute__((unused)), int *ipIFVersProvided, rsRetVal (**pQueryEtryPt)(), rsRetVal (*pHostQueryEtryPt)(uchar*, rsRetVal (**)()), modInfo_t*); +/* the define below is dirty, but we need it for ompipe integration. There is no + * other way to have the functionality (well, one way would be to go through the + * globals, but that seems not yet justified. -- rgerhards, 2010-03-01 + */ +uchar *pszFileDfltTplName; #endif /* #ifndef OMFILE_H_INCLUDED */ -/* - * vi:set ai: +/* vi:set ai: */ diff --git a/tools/omfwd.c b/tools/omfwd.c index 02f19eac..cbfc36a4 100644 --- a/tools/omfwd.c +++ b/tools/omfwd.c @@ -411,7 +411,7 @@ CODESTARTtryResume ENDtryResume BEGINdoAction - char *psz; /* temporary buffering */ + char *psz = NULL; /* temporary buffering */ register unsigned l; int iMaxLine; CODESTARTdoAction @@ -458,11 +458,14 @@ CODESTARTdoAction * rgerhards, 2006-11-30 */ dbgprintf("Compression failed, sending uncompressed message\n"); + free(out); } else if(destLen+1 < l) { /* only use compression if there is a gain in using it! */ dbgprintf("there is gain in compression, so we do it\n"); psz = (char*) out; l = destLen + 1; /* take care for the "z" at message start! */ + } else { + free(out); } ++destLen; } @@ -484,7 +487,7 @@ CODESTARTdoAction } finalize_it: # ifdef USE_NETZIP - if(psz != (char*) ppString[0]) { + if((psz != NULL) && (psz != (char*) ppString[0])) { /* we need to free temporary buffer, alloced above - Naoya Nakazawa, 2010-01-11 */ free(psz); } diff --git a/tools/ompipe.c b/tools/ompipe.c new file mode 100644 index 00000000..7cf61822 --- /dev/null +++ b/tools/ompipe.c @@ -0,0 +1,240 @@ +/* ompipe.c + * This is the implementation of the build-in pipe output module. + * Note that this module stems back to the "old" (4.4.2 and below) + * omfile. There were some issues with the new omfile code and pipes + * (namely in regard to xconsole), so we took out the pipe code and moved + * that to a separate module. That a) immediately solves the issue for a + * less common use case and probably makes it much easier to enhance + * file and pipe support (now independently) in the future (we always + * needed to think about pipes in omfile so far, what we now no longer + * need to, hopefully resulting in reduction of complexity). + * + * NOTE: read comments in module-template.h to understand how this pipe + * works! + * + * Copyright 2007-2010 Rainer Gerhards and Adiscon GmbH. + * + * This file is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the file "COPYING" in this distribution. + */ +#include "config.h" +#include "rsyslog.h" +#include <stdio.h> +#include <stdarg.h> +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <errno.h> +#include <fcntl.h> +#include <sys/file.h> + +#include "syslogd.h" +#include "syslogd-types.h" +#include "srUtils.h" +#include "template.h" +#include "ompipe.h" +#include "omfile.h" /* for dirty trick: access to $ActionFileDefaultTemplate value */ +#include "cfsysline.h" +#include "module-template.h" +#include "conf.h" +#include "errmsg.h" + +MODULE_TYPE_OUTPUT + +/* internal structures + */ +DEF_OMOD_STATIC_DATA +DEFobjCurrIf(errmsg) + + +/* globals for default values */ +/* end globals for default values */ + + +typedef struct _instanceData { + uchar f_fname[MAXFNAME];/* pipe or template name (display only) */ + short fd; /* pipe descriptor for (current) pipe */ +} instanceData; + + +BEGINisCompatibleWithFeature +CODESTARTisCompatibleWithFeature + if(eFeat == sFEATURERepeatedMsgReduction) + iRet = RS_RET_OK; +ENDisCompatibleWithFeature + + +BEGINdbgPrintInstInfo +CODESTARTdbgPrintInstInfo + dbgprintf("pipe %s", pData->f_fname); + if (pData->fd == -1) + dbgprintf(" (unused)"); +ENDdbgPrintInstInfo + + +/* This is now shared code for all types of files. It simply prepares + * pipe access, which, among others, means the the pipe wil be opened + * and any directories in between will be created (based on config, of + * course). -- rgerhards, 2008-10-22 + * changed to iRet interface - 2009-03-19 + */ +static inline rsRetVal +preparePipe(instanceData *pData) +{ + DEFiRet; + pData->fd = open((char*) pData->f_fname, O_RDWR|O_NONBLOCK|O_CLOEXEC); + RETiRet; +} + + +/* rgerhards 2004-11-11: write to a pipe output. This + * will be called for all outputs using pipe semantics, + * for example also for pipes. + */ +static rsRetVal writePipe(uchar **ppString, instanceData *pData) +{ + int iLenWritten; + DEFiRet; + + ASSERT(pData != NULL); + + if(pData->fd == -1) { + rsRetVal iRetLocal; + iRetLocal = preparePipe(pData); + if((iRetLocal != RS_RET_OK) || (pData->fd == -1)) + ABORT_FINALIZE(RS_RET_SUSPENDED); /* whatever the failure was, we need to retry */ + } + + /* create the message based on format specified */ + iLenWritten = write(pData->fd, ppString[0], strlen((char*)ppString[0])); + if(iLenWritten < 0) { + int e = errno; + char errStr[1024]; + rs_strerror_r(errno, errStr, sizeof(errStr)); + DBGPRINTF("pipe (%d) write error %d: %s\n", pData->fd, e, errStr); + + /* If a named pipe is full, we suspend this action for a while */ + if(e == EAGAIN) + ABORT_FINALIZE(RS_RET_SUSPENDED); + + close(pData->fd); + pData->fd = -1; /* tell that fd is no longer open! */ + iRet = RS_RET_SUSPENDED; + errno = e; + errmsg.LogError(0, NO_ERRCODE, "%s", pData->f_fname); + } + +finalize_it: + RETiRet; +} + + +BEGINcreateInstance +CODESTARTcreateInstance + pData->fd = -1; +ENDcreateInstance + + +BEGINfreeInstance +CODESTARTfreeInstance + if(pData->fd != -1) + close(pData->fd); +ENDfreeInstance + + +BEGINtryResume +CODESTARTtryResume +ENDtryResume + +BEGINdoAction +CODESTARTdoAction + DBGPRINTF(" (%s)\n", pData->f_fname); + iRet = writePipe(ppString, pData); +ENDdoAction + + +BEGINparseSelectorAct +CODESTARTparseSelectorAct + /* yes, the if below is redundant, but I need it now. Will go away as + * the code further changes. -- rgerhards, 2007-07-25 + */ + if(*p == '|') { + if((iRet = createInstance(&pData)) != RS_RET_OK) { + ENDfunc + return iRet; /* this can not use RET_iRet! */ + } + } else { + /* this is not clean, but we need it for the time being + * TODO: remove when cleaning up modularization + */ + ENDfunc + return RS_RET_CONFLINE_UNPROCESSED; + } + + CODE_STD_STRING_REQUESTparseSelectorAct(1) + ++p; + /* rgerhards 2004-11-17: from now, we need to have different + * processing, because after the first comma, the template name + * to use is specified. So we need to scan for the first coma first + * and then look at the rest of the line. + */ + CHKiRet(cflineParseFileName(p, (uchar*) pData->f_fname, *ppOMSR, 0, OMSR_NO_RQD_TPL_OPTS, + (pszFileDfltTplName == NULL) ? (uchar*)"RSYSLOG_FileFormat" : pszFileDfltTplName)); + + /* at this stage, we ignore the return value of preparePipe, this is taken + * care of in later steps. -- rgerhards, 2009-03-19 + */ + preparePipe(pData); + + if(pData->fd < 0 ) { + pData->fd = -1; + DBGPRINTF("Error opening log pipe: %s\n", pData->f_fname); + errmsg.LogError(0, RS_RET_NO_FILE_ACCESS, "Could no open output pipe '%s'", pData->f_fname); + } +CODE_STD_FINALIZERparseSelectorAct +ENDparseSelectorAct + + +BEGINdoHUP +CODESTARTdoHUP + if(pData->fd != -1) { + close(pData->fd); + pData->fd = -1; + } +ENDdoHUP + + +BEGINmodExit +CODESTARTmodExit +ENDmodExit + + +BEGINqueryEtryPt +CODESTARTqueryEtryPt +CODEqueryEtryPt_STD_OMOD_QUERIES +CODEqueryEtryPt_doHUP +ENDqueryEtryPt + + +BEGINmodInit(Pipe) +CODESTARTmodInit + *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ +CODEmodInit_QueryRegCFSLineHdlr + CHKiRet(objUse(errmsg, CORE_COMPONENT)); +ENDmodInit +/* vi:set ai: + */ diff --git a/tools/ompipe.h b/tools/ompipe.h new file mode 100644 index 00000000..d17346c5 --- /dev/null +++ b/tools/ompipe.h @@ -0,0 +1,31 @@ +/* ompipe.h + * These are the definitions for the build-in pipe output module. + * + * Copyright 2007-2010 Rainer Gerhards and Adiscon GmbH. + * + * This pipe is part of rsyslog. + * + * Rsyslog is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Rsyslog is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with Rsyslog. If not, see <http://www.gnu.org/licenses/>. + * + * A copy of the GPL can be found in the pipe "COPYING" in this distribution. + */ +#ifndef OMPIPE_H_INCLUDED +#define OMPIPE_H_INCLUDED 1 + +/* prototypes */ +rsRetVal modInitPipe(int iIFVersRequested __attribute__((unused)), int *ipIFVersProvided, rsRetVal (**pQueryEtryPt)(), rsRetVal (*pHostQueryEtryPt)(uchar*, rsRetVal (**)()), modInfo_t*); + +#endif /* #ifndef OMPIPE_H_INCLUDED */ +/* vi:set ai: + */ diff --git a/tools/omusrmsg.c b/tools/omusrmsg.c index 499a11dd..e788a006 100644 --- a/tools/omusrmsg.c +++ b/tools/omusrmsg.c @@ -50,7 +50,15 @@ #include <assert.h> #include <signal.h> #include <sys/param.h> -#include <utmp.h> +#ifdef HAVE_UTMP_H +# include <utmp.h> +# define STRUCTUTMP struct utmp +# define UTNAME ut_name +#else +# include <utmpx.h> +# define STRUCTUTMP struct utmpx +# define UTNAME ut_user +#endif #include <unistd.h> #include <sys/uio.h> #include <sys/stat.h> @@ -124,6 +132,12 @@ ENDdbgPrintInstInfo * need! rgerhards 2005-03-18 */ #ifdef OS_BSD +/* Since version 900007, FreeBSD has a POSIX compliant <utmpx.h> */ +#if defined(__FreeBSD__) && (__FreeBSD_version >= 900007) +# define setutent(void) setutxent(void) +# define getutent(void) getutxent(void) +# define endutent(void) endutxent(void) +#else static FILE *BSD_uf = NULL; void setutent(void) { @@ -134,9 +148,9 @@ void setutent(void) } } -struct utmp* getutent(void) +STRUCTUTMP* getutent(void) { - static struct utmp st_utmp; + static STRUCTUTMP st_utmp; if(fread((char *)&st_utmp, sizeof(st_utmp), 1, BSD_uf) != 1) return NULL; @@ -149,6 +163,7 @@ void endutent(void) fclose(BSD_uf); BSD_uf = NULL; } +#endif /* if defined(__FreeBSD__) */ #endif /* #ifdef OS_BSD */ @@ -173,8 +188,8 @@ static rsRetVal wallmsg(uchar* pMsg, instanceData *pData) int errnoSave; int ttyf; int wrRet; - struct utmp ut; - struct utmp *uptr; + STRUCTUTMP ut; + STRUCTUTMP *uptr; struct stat statb; DEFiRet; @@ -187,13 +202,13 @@ static rsRetVal wallmsg(uchar* pMsg, instanceData *pData) while((uptr = getutent())) { memcpy(&ut, uptr, sizeof(ut)); /* is this slot used? */ - if(ut.ut_name[0] == '\0') + if(ut.UTNAME[0] == '\0') continue; #ifndef OS_BSD if(ut.ut_type != USER_PROCESS) continue; #endif - if(!(strncmp (ut.ut_name,"LOGIN", 6))) /* paranoia */ + if(!(strncmp (ut.UTNAME,"LOGIN", 6))) /* paranoia */ continue; /* should we send the message to this user? */ @@ -203,7 +218,7 @@ static rsRetVal wallmsg(uchar* pMsg, instanceData *pData) i = MAXUNAMES; break; } - if(strncmp(pData->uname[i], ut.ut_name, UNAMESZ) == 0) + if(strncmp(pData->uname[i], ut.UTNAME, UNAMESZ) == 0) break; } if(i == MAXUNAMES) /* user not found? */ diff --git a/tools/regexp.c b/tools/regexp.c index c8e4c681..e8bba4f4 100644 --- a/tools/regexp.c +++ b/tools/regexp.c @@ -26,6 +26,7 @@ * * A copy of the GPL can be found in the file "COPYING" in this distribution. */ +#include "config.h" #include <stdio.h> #include <stdlib.h> #include <string.h> diff --git a/tools/rsyslog.conf.5 b/tools/rsyslog.conf.5 index f2b915e2..e8a4ab92 100644 --- a/tools/rsyslog.conf.5 +++ b/tools/rsyslog.conf.5 @@ -80,7 +80,7 @@ used like this: .IP $ModLoad imudp .IP -$InputUDPServerRun 514 +$UDPServerRun 514 .TP .I imtcp Input plugin for plain TCP syslog. Replaces the deprecated -t diff --git a/tools/syslogd.c b/tools/syslogd.c index 8fcb8dd0..e318fd8e 100644 --- a/tools/syslogd.c +++ b/tools/syslogd.c @@ -127,6 +127,7 @@ #include "omusrmsg.h" #include "omfwd.h" #include "omfile.h" +#include "ompipe.h" #include "omdiscard.h" #include "threads.h" #include "queue.h" @@ -177,7 +178,11 @@ static rsRetVal GlobalClassExit(void); #endif #ifndef _PATH_MODDIR -#define _PATH_MODDIR "/lib/rsyslog/" +# if defined(__FreeBSD__) +# define _PATH_MODDIR "/usr/local/lib/rsyslog/" +# else +# define _PATH_MODDIR "/lib/rsyslog/" +# endif #endif #if defined(SYSLOGD_PIDNAME) @@ -792,7 +797,7 @@ parseAndSubmitMessage(uchar *hname, uchar *hnameIP, uchar *msg, int len, int fla * (I couldn't do any more smart things anyway...). * rgerhards, 2007-9-20 */ - DBGPRINTF("internal error: iMsg > max msg size in printchopped()\n"); + DBGPRINTF("internal error: iMsg > max msg size in parseAndSubmitMessage()\n"); } FINALIZE; /* in this case, we are done... nothing left we can do */ } @@ -1193,13 +1198,12 @@ int parseLegacySyslogMsg(msg_t *pMsg, int flags) int bTAGCharDetected; int i; /* general index for parsing */ uchar bufParseTAG[CONF_TAG_MAXSIZE]; - uchar bufParseHOSTNAME[CONF_TAG_HOSTNAME]; + uchar bufParseHOSTNAME[CONF_HOSTNAME_MAXSIZE]; BEGINfunc assert(pMsg != NULL); assert(pMsg->pszRawMsg != NULL); - lenMsg = pMsg->iLenRawMsg - (pMsg->offAfterPRI + 1); - + lenMsg = pMsg->iLenRawMsg - pMsg->offAfterPRI; /* note: offAfterPRI is already the number of PRI chars (do not add one!) */ p2parse = pMsg->pszRawMsg + pMsg->offAfterPRI; /* point to start of text, after PRI */ /* Check to see if msg contains a timestamp. We start by assuming @@ -1255,16 +1259,24 @@ int parseLegacySyslogMsg(msg_t *pMsg, int flags) bTAGCharDetected = 0; if(lenMsg > 0 && flags & PARSE_HOSTNAME) { i = 0; - while(lenMsg > 0 && (isalnum(p2parse[i]) || p2parse[i] == '.' || p2parse[i] == '.' - || p2parse[i] == '_' || p2parse[i] == '-') && i < CONF_TAG_MAXSIZE) { + while(i < lenMsg && (isalnum(p2parse[i]) || p2parse[i] == '.' || p2parse[i] == '.' + || p2parse[i] == '_' || p2parse[i] == '-') && i < (CONF_HOSTNAME_MAXSIZE - 1)) { bufParseHOSTNAME[i] = p2parse[i]; ++i; - --lenMsg; } - if(i > 0 && p2parse[i] == ' ' && isalnum(p2parse[i-1])) { + if(i == lenMsg) { + /* we have a message that is empty immediately after the hostname, + * but the hostname thus is valid! -- rgerhards, 2010-02-22 + */ + p2parse += i; + lenMsg -= i; + bufParseHOSTNAME[i] = '\0'; + MsgSetHOSTNAME(pMsg, bufParseHOSTNAME, i); + } else if(i > 0 && p2parse[i] == ' ' && isalnum(p2parse[i-1])) { /* we got a hostname! */ p2parse += i + 1; /* "eat" it (including SP delimiter) */ + lenMsg -= i + 1; bufParseHOSTNAME[i] = '\0'; MsgSetHOSTNAME(pMsg, bufParseHOSTNAME, i); } @@ -2664,6 +2676,9 @@ static rsRetVal loadBuildInModules(void) if((iRet = module.doModInit(modInitFile, UCHAR_CONSTANT("builtin-file"), NULL)) != RS_RET_OK) { RETiRet; } + if((iRet = module.doModInit(modInitPipe, UCHAR_CONSTANT("builtin-pipe"), NULL)) != RS_RET_OK) { + RETiRet; + } #ifdef SYSLOG_INET if((iRet = module.doModInit(modInitFwd, UCHAR_CONSTANT("builtin-fwd"), NULL)) != RS_RET_OK) { RETiRet; @@ -2769,7 +2784,7 @@ static void printVersion(void) #else printf("\tFEATURE_REGEXP:\t\t\t\tNo\n"); #endif -#ifndef NOLARGEFILE +#if defined(_LARGE_FILES) || (defined (_FILE_OFFSET_BITS) && _FILE_OFFSET_BITS >= 64) printf("\tFEATURE_LARGEFILE:\t\t\tYes\n"); #else printf("\tFEATURE_LARGEFILE:\t\t\tNo\n"); @@ -3173,6 +3188,7 @@ int realMain(int argc, char **argv) uchar *LocalHostName; uchar *LocalDomain; uchar *LocalFQDNName; + char cwdbuf[128]; /* buffer to obtain/display current working directory */ /* first, parse the command line options. We do not carry out any actual work, just * see what we should do. This relieves us from certain anomalies and we can process @@ -3259,8 +3275,9 @@ int realMain(int argc, char **argv) if ((argc -= optind)) usage(); - DBGPRINTF("rsyslogd %s startup, compatibility mode %d, module path '%s'\n", - VERSION, iCompatibilityMode, glblModPath == NULL ? "" : (char*)glblModPath); + DBGPRINTF("rsyslogd %s startup, compatibility mode %d, module path '%s', cwd:%s\n", + VERSION, iCompatibilityMode, glblModPath == NULL ? "" : (char*)glblModPath, + getcwd(cwdbuf, sizeof(cwdbuf))); /* we are done with the initial option parsing and processing. Now we init the system. */ diff --git a/tools/zpipe.c b/tools/zpipe.c index bde6c5c1..38069425 100644 --- a/tools/zpipe.c +++ b/tools/zpipe.c @@ -4,6 +4,13 @@ Version 2.0 03 June 2009 Rainer Gerhards */ /* RSYSLOG NOTE: + * This file is primarily been used as a testing aid for rsyslog. We do NOT + * properly maintain it and it has been brought to our attention that it may + * have some security issues. However, we prefer not to remove the file as it + * may turn out to be useful for further testing. All users are advised NOT + * to base any development on this version here, but rather look for the + * original zpipe.c by the authors mentioned above. + * * This file is beeing distributed as part of rsyslog, but is just an * add-on. Most importantly, rsyslog's copyright does not apply but * rather the (non-) copyright stated above. @@ -22,6 +29,7 @@ files created by rsyslog's zip output writer. */ +#include "config.h" #include <stdio.h> #include <string.h> #include <assert.h> |