path: root/docs/source/configuration.rst

.. Copyright 2012 - 2015 David Sommerseth <dazo@eurephia.org>

   This is free software: you can redistribute it and/or modify it
   under the terms of the GNU General Public License as published by
   the Free Software Foundation, version 2 of the License.

   This program is distributed in the hope that it will be useful, but
   WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
   General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this program.  If not, see
   <http://www.gnu.org/licenses/>.

.. For notes on how to document Python in RST form, see e.g.:
.. http://sphinx.pocoo.org/domains.html#the-python-domain

Configuring logactio
====================

Logactio makes use of a single ini-styled configuration file.  The default
location for this file is */etc/logactio.cfg*.  This file uses a separate
section, identified by *[section-name]*.  There are three different groups
of sections which you need to configure, *Reporter*, *Logfile* and *Rule*.
Each of these group names have a *label*, where the group name and the label
are separated by a colon (:)


.. index::
   pair: Configuration; Reporters

Configuring Reporters
*********************

There are three different reporters available in logactio, and each of them take
different configuration parameters.  If no reporter is configured, a built-in
reporter called *Default* will be used.

* The Default reporter

   This reporter will only write data via the configured logging methods.  This
   reporter also does not have any configuration settings and does not require
   any specific declaration.

* :ref:`lnk-httpreporter`

   This reporter will send the extracted log data to a web server.  Both
   HTTP and HTTPS may be used.

* :ref:`lnk-smtpreporter`

   This reporter will send the extracted log data via SMTP to one or more
   e-mail recipients.  Support for SSL and STARTTLS in addition to SMTP-AUTH
   are availble as well.

* :ref:`lnk-IPTipset`

   This "reporter" can add extracted IP addresses from a log file and add
   it to an ipset table.  This table can be used by iptables to for example
   block access.

* :ref:`lnk-qpidreporter`

   This reporter will send the extracted log data to an
   `Apache Qpid <http://qpid.apache.org>` AMQP message broker.  One
   or more AMQP consumer clients may then act upon the messages recieved
   in the message queue.  In the *examples/* directory a simple Qpid
   consumer client is available.


.. index::
   pair: Configuration; HTTPreporter
.. _lnk-httpreporter:

HTTPreporter
------------

This reporter takes takes two configuration variables

* method

   This defaults to **GET** if this is not set.  But can be set to **POST**
   if you want the reports to be sent via HTTP POST instead of HTTP GET.

* url

   This is a required setting.  This is the URL where to submit the reports.

.. index::
   pair: Examples; HTTPreporter

HTTPreporter example
~~~~~~~~~~~~~~~~~~~~~~

.. code-block :: ini

  [Reporter:HTTP-DEMO-1]
  module: HTTPreporter
  method: POST
  url: http://logactio.example.com/alert/

  [Reporter:HTTP-DEMO-2]
  module: HTTPreporter
  url: http://logactio.example.com/info/

Here two reporters are configured, *HTTP-DEMO-1* and *HTTP-DEMO-2*.  That's the
labels which you will need to use later on.  The *module* option is to tell
logactio to load the *HTTPreporter* module.

The first reporter will use HTTP POST when submitting reports to the provided
URL, while the second reporter will use HTTP GET.

.. index::
   pair: Configuration; SMTPreporter
.. _lnk-smtpreporter:

SMTPreporter
------------
This reporter requires the following configuration variables:

* sender

  The e-mail address which will be used in the "From:" field when sending
  mails

* recipients

  This contains a comma separated list with all e-mail addresses who will
  get the logactio reports.

* smtp_host

  This declares which SMTP server to use when sending the reports.

In addition the SMTPreporter supports these optional variables:

* subject_prefix

   The default subject prefix is set to 'LogActio Alert: '.  By setting
   this variable, the subject prefix will be changed accordingly.

* smtp_port

   The default value is set to port 25.

* smtp_username

   If the SMTP server requires authentication to relay messages, this
   variable sets the SMTP user name.  To use this feature, you must also
   set the smtp_password.

* smtp_password

   This sets the SMTP password to use for the authentication

* sslmode

   This is not set by default, so everything goes in clear text.  If your SMTP
   server supports either SSL or STARTTLS, you can set it to SSL or STARTTLS.
   In SSL mode the SMTP library expects to the server to do the SSL handshake
   before the SMTP commands can be sent.  In STARTTLS mode, the SMTP library
   will connect to the SMTP server in clear text and if the server supports
   STARTTLS, it will send the STARTTLS command and start the SSL handshake.

.. index::
   pair: Examples; SMTPreporter

SMTPreporter example
~~~~~~~~~~~~~~~~~~~~~~

.. code-block :: ini

   [Reporter:SMTP-DEMO-1]
   module: SMTPreporter
   sender: logactio@example.com
   recipients: john.doe@example.com, jane.doe@example.com
   smtp_host: localhost

   [Reporter:SMTP-DEMO-2]
   module: SMTPreporter
   sender: john.doe@example.com
   recipients: bob.external@acme.com
   smtp_host: smtp.example.com
   smtp_port: 587
   sslmode: STARTTLS
   smtp_user: logactioSMTP
   smtp_password: S3cretP4ssw0rd
   subject_prefix: Issues at Example Corp:

Here two more reporters are configured.  SMTP-DEMO-1 will use the SMTP MTA
running locally on the system, and this will not require any authentication
or SSL functionality.  When this reporter is triggered, it will send mails
to john.doe and jane.doe with the From field set to logactio@example.com.

The SMTP-DEMO-2 reporter will send mails using an external SMTP server using
port 587 and which requires STARTTLS and authentication.  The subject line
is also prefixed differently.


.. index::
   pair: Configuration; IPTipset
.. _lnk-IPTipset:

IPTipset
--------
To use this module, you need ipset and iptables installed and available.  This
will also require logactio to run with root privileges, otherwise it lacks
privileges to update the ipset and iptables rules.

.. IMPORTANT::
   This module expects only a single regex group to be extracted.  This value
   **has** to be an IP address, otherwise the IPTipset module will fail when
   executing the ipset command.

The required configuration variables are:

* ipset-name
  This is the name of the ipset table to this specific section will use.

Optional settings are:

* ipset-create

  Boolean variable which tells wether the IPTipset module should create
  the ipset table when starting or not.  The values of "true" (case
  or "1" will be understood as True, all other values as False.

* ipset-hashsize, ipset-timeout, ipset-counters

  These variables will be sent further to the 'ipset create' command and will
  set hashsize, timeout and the counters flag.  ipset-counters is a boolean
  flag, where the value of "true" (case insensitive) or "1" will be understood
  as True, all other values as False.  The default value for ipset-hashsize
  is 1024, ipset-timeout is not defined and ipset-counters is False by default.

* ipset-save

  This will enable the IPTipset module to save the state of the ipset table
  when exiting.  If this file exists when LogActio is started again later on,
  this state will be loaded automatically.

* iptables-chains

  If set, LogActio will insert the ipset checks into the listed iptables
  chains.  Multiple chains can be listed, separated by comma.

* iptables-insert-points

  This is optional but only available if iptables-chains are used.   This
  defines where in the different iptables chains the ipset rules will be
  inserted.  By default it will be inserted at the top of the chain.  The
  syntax is CHAIN:NUM, where CHAIN is the name provided in *iptables-chains*
  and NUM is the value used when calling iptables.  The result is something
  like 'iptables -I CHAIN NUM ...'.  Multiple chains can be defined,
  separated by comma.

* iptables-jump

  This is required when using *iptables-chains*.  This defines what to do
  when iptables get a positive match against the ipset table.  This sets the
  '-j' options when iptables commands are executed.

.. index::
   pair: Examples; IPTipset

IPTipset example
~~~~~~~~~~~~~~~~

.. code-block :: ini

    [Reporter:ipsetblock]
    module: IPTipset
    ipset-name: BlockList
    ipset-create: True
    ipset-hashsize: 2048
    ipset-timeout: 3600
    ipset-counters: False
    ipset-save: /var/lib/ipset/logactio-ipset.save
    iptables-chains: INPUT,OUTPUT,FORWARD
    iptables-insert-points: INPUT:2,FORWARD:4
    iptables-jump: DROP

This will configure a reported named ipsetblock.  Whenever this reporter
is called, it will add an IP address to an ipset table named 'Blocklist'.
This list will be created when LogActio starts, with a hashsize of 2048
and a timeout of 1 hour (3600 seconds).  The ipset table state will be
saved to /var/lib/ipset/logactio-ipset.save.  When starting LogActio
the following commands will be run:

.. code-block :: shell

    # ipset --exist create BlockList hash:ipset hashsize 2048 timeout 3600
    # iptables -I INPUT 2 -m set --match-set BlockList src -j DROP
    # iptables -I OUTPUT -m set --match-set BlockList src -j DROP
    # iptables -I FORWARD 4 -m set --match-set BlockList src -j DROP

.. index::
   pair: Configuration; QpidReporter
.. _lnk-qpidreporter:

QpidReporter
------------
To use this reporter, you must have a running AMQP server available for
logactio. You must also have configured a topic exchange which this reporter
can use.  It also requires the python-qpid module to be installed

The required configuration variables which must be set are:

* broker

  This is the hostname or IP address of the AMQP broker to connect to.

* exchange

  This is the AMQP exchange logactio will use when sending reports

* routing_key

  This contains the "topic queue" where subscribers can receive reports
  sent by logactio.

Optional settings are:

* port

  If the broker is not running on the default port 5672, this can be changed
  with this variable.

.. index::
   pair: Examples; QpidReporter

QpidReporter example
~~~~~~~~~~~~~~~~~~~~~~

.. code-block :: ini

   [Reporter:QPID-DEMO-1]
   module: QpidReporter
   broker: amqp.example.com
   exchange: amq.topic
   routing_key: logactio.demo1

This configures the reporter labelled QPID-DEMO-1 to use QpidReporter to
send reports via the *amqp.example.com* AMQP server.  All messages goes
to all the *logactio.demo1* subscribers in the *amq.topic* exchange.

If you have an AMQP broker running, you can use the demo script
*examples/qpid-alert-watcher* and run it like this:

.. code-block :: bash

  [user@host: ~/logactio/example] $ ./qpid-alert-watcher --broker amqp.example.com --exchange amq.topic --bind-key logactio.demo1

This will start up a message consumer which will dump all the reports sent by logactio to stdout.

.. index::
   single: Log files

Configuring log files
*********************
The configuration sections for the log files are similar to the reporters.  You
declare a section for each log file you want logactio to watch and how often
you want logactio to check the file for changes.  For each configured logfile
section logactio will start a separate worker thread for the event processing.

There are three configuration variables logactio supports for logfiles:

* logfile

  This is mandatory, and declares the log file it should watch

* reporters

  This is optional, but declares the default reporter module(s) to use if an
  event happens to this file.  You may list more reporter modules, separated
  by comma.

* polltime

  This is optional.  The default is 30 seconds.  This declares how often
  logactio should check the file for changes, which indirectly defines
  how quickly logactio would react to and report events.

Logfile example
---------------

A typical configuration for log file sections would look something like this:

.. code-block :: ini

   [Logfile:messages]
   logfile: /var/log/messages
   reporters: SMTP-DEMO-1

   [Logfile:maillog]
   logfile: /var/log/maillog
   reporters: HTTP-DEMO-1, QPID-DEMO-1
   polltime: 15

In this example we have configured two Logfile groups, *messages* and *maillog*.
Any event happening in /var/log/messages will by default be reported using the
configured SMTP-DEMO-1 reporter setup.  The *maillog* will be checked every
15th second and by default both the HTTP-DEMO-1 and QPID-DEMO-1 reporters will
be used when reporting events.

.. index::
   single: Rules

Configuring watch rules
***********************
If you only configure Logfile and Reporter sections, logactio will not trigger
at all.  You need to configure some rules what logactio should react to.

The rules are based on regular expressions.  And if there is a match on the
log lines received, each of these lines will be acted upon separately.

A Rules section consists of two required configuration variables:

* logfile

  This is the log file this rule is to be used against

* regex

  This is the regular expression which needs to match to cause the reporter
  to be triggered.  You can also use regex groups, like (.*), to extract
  information from the log line which will be sent to the reporter.  If
  you use multiple groups, all of them will be sent to reporter.

* threshold:

  This sets how many times this event should match before triggering the
  reporter.

* threshold-type:

  This defines how the threshold counter works.  By default, it is set to
  *rule*.  This will increase the "hit counter" each time this watch rule
  is triggered.  By setting the *threshold-type* parameter to *exact* it
  will also consider the regex groups defined in the *regex* parameter.
  When using the *exact* type, it will have individual threshold counters
  per group contents for each time this rule is triggered.

The optional settings are:

* reporters

  This overrides the default reporters configured in the log file's Logfile
  section.  This can be used to add exceptions or report an event differently
  in special cases.  For example you might want developers to get an automatic
  mail if their program causes an exception which is logged.  While a system-
  administrator might only want reports if someone tries to log into a system
  unsuccessfully more than 3 times.  Setting up different Rule sections with
  different reporters and thresholds brings you this power, even if everything
  is logged to the same file.

* time-frame

  This extends the threshold trigger to also consider a time frame before
  trigging an action.  If the threshold is set to 3 and time-frame is set
  to 10, logaction will not trigger an action unless there are 3 events
  within the last 10 seconds.

* rate-limit

  This will restrict logactio from any flood actions.  If this value is set to
  10 and you have log changes which matches this rule once every second,
  logactio will only perform the configured action once per 10 second.

* reset-rule-rate-limits

  This takes a comma separated list of rule names, but only for the same log
  file this rule uses.  This can be used to "unlock" another rule's rate-limit
  restriction.

  This is useful where you might report connection issues only once an hour,
  even though the failed reconnection attempts are logged every minute.  But
  in the moment the connection really is restored you can trigger a logactio
  action informing the connection is back again.  But if this connection drops
  after a few minutes again - it might be the "connection-failed" rule won't
  trigger before an hour later.  By adding reset-rule-rate-limits on the
  "connection-is-back" rule, it can reset the "connection-failed"'s rate-limit
  check, so it that rule will trigger instantly.

Rule examples
-------------
.. code-block :: ini

   [Rule:iptables]
   logfile: messages
   regex: .* (.*) kernel: .* IN=(\S*) OUT=.* MAC=.* SRC=(\S* )DST=(\S* ).* PROTO=(\S* )SPT=(\d* )DPT=(\d* ).*
   threshold: 1

   [Rule:lost-connection]
   logfile: maillog
   regex: lost connection after (.*) from (.*)
   reporters: HTTP-DEMO-2
   threshold: 5

We declare two rules here, one which looks for a certain pattern which matches
iptables and uses the default reporters.  Each time log line in /var/log/messages
matches this rule, the reporters are triggered.  It will report the hostname of
the server this happened, the input device, source and destination IP addresses
as well as protocol, source and destination ports.

The second rule looks for connection issues in /var/maillog and will report
these every 5th issue using only the configured HTTP-DEMO-2 reporter.