Tuesday, January 15, 2013

Splunk Howto - Splunk for Fail2ban, get a the Fail2ban Multi-host frontend with Splunk!








*** Updated June 9, 2013  ***


Current Version = 2.02

Splunk (if you don't yet know it) is an incredibly powerful solution that collects, indexes and exploits any kind of data from any system, offering you as many solution as you need and even the possibility to create custom applications with graphical front-ends. (dashboards, reports, saved searches...)

In a few words, i am really impressed by Splunk, i think i've been looking for this for many many years!

Don't hesitate to take a look at main Splunk Website, you will easily find a lot of information and great documentations: http://www.splunk.com/

Splunk can be used for free with some little restrictions. (not more than 500Mb of input data per day)


I developed my first Splunk application "Splunk For Fail2ban" to provide a cool frontend and log managing tool associated with the well known and powerful Fail2ban tool. (take a look at my older post: http://youresuchageek.blogspot.fr/2012/11/howto-fail2ban-secure-your-network.html)

To install this addon, follow this link on Splunkbase or install it through the standard Splunk application process search online: 


Splunk pre-requirements:

Ensure to install requirements Splunk addons:



Splunk For Fail2ban provides:

A complete Dashboard Overview of Fail2ban activity for all managed systems: 


Home page with realtime quick summary activity overview and links to interfaces:






A complete Dashboard Overview of Fail2ban activity for all managed systems: 

Activity overview:


Activity and Alert Trend:




Various Top 10 Charts and stats:










Google Maps Dashboard, identify the source of connexion attempts


A Fail2ban Event search interface with selection per kind of data (IPs, ID, Jail...)




Pre-defined major searches to get all the most important information



System view: Index activity





Installation and utilization

Introduction

Installing and configuring Splunk is out of the scope of this post, still installing Splunk is really easy and well done, in 10 minutes you'll be done ^^


As a brieve description, here is how Splunk for Fail2ban works:

- We modify Fail2ban to add a specific message for each ban action and containing fields Splunk will analyse
- Through Syslog, we can manage as many Fail2ban servers as required
- Splunk collects our data and produces the IT intelligency

Installation and configuration will be done in a few steps:

1. Modifying Fail2ban configuration files related to the ban action (the goal is send fields we will analyse with Splunk)
2. Setting up Fail2ban to log to Syslog system
3. Setting up Syslog to trap custom Fail2ban events into a specific log file (can be local or remote Syslog if numerous Fail2ban hosts)
4. Installation and configuring Splunk for Fail2ban


Part 1: Configure Fail2ban

1. Set Fail2ban output to Syslog

I recommend the use of "rsyslog" as your main Syslog management, it comes with much more improvement than the standard Syslog. (http://www.rsyslog.com/)

First, we need to set Fail2ban to log its messages into Syslog instead of a standard log file.

To do so, edit "/etc/fail2ban/fail2ban.conf" and set:
logtarget = SYSLOG

2. Add a new action.d configuration file for events logging

Note: 
See this configuration sample if required: splunk.conf.example


Create a new file: "/etc/fail2ban/actions.d/splunk.conf" with the following content:
[Definition]
actionban = logger -i "[fail2ban.banevent]: fail2ban_host: [`hostname`] \
Banhost: [<ip>] jailname: [<name>] numberoffailures: [<failures>] \
logmessage: [ `grep '\<<ip>\>' <logpath> | tail -1` ] "

[Init]


3. Configure "/etc/fail2ban/jail.conf":

Depending of your wish, you can set Fail2ban to use 1 of these 3 actions: (by editing /etc/fail2ban/jail.conf)
  • action_ = Fail2ban will temporarely ban the IP source host
  • action_mw = Fail2ban will temporarely ban the IP host and send a warning mail including whois result request
  • action_mwl = Fail2ban will temporarely ban the IP host and send a warning mail including whois result request and log traces
All you need is to modify jail.conf for all these action level to include our specific logging for Splunk.

Note: 
See this configuration sample if required: jail.conf.example


In jail.conf, add the following line just before the 3 action definition lines (action_, action_mw, action_mwl)
# Name of Splunk config file
splunkconf = splunk

Then, add a new line related splunk underneath each action level, your configuration file will looks like:
#
# Action shortcuts. To be used to define action parameter

# The simplest action to take: ban only
action_ = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
            %(splunkconf)s[name=%(__name__)s, logpath=%(logpath)s]

# ban & send an e-mail with whois report to the destemail.
action_mw = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
              %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s", protocol="%(protocol)s", chain="%(chain)s"]
                %(splunkconf)s[name=%(__name__)s, logpath=%(logpath)s]

# ban & send an e-mail with whois report and relevant log lines
# to the destemail.
action_mwl = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
               %(mta)s-whois-lines[name=%(__name__)s, dest="%(destemail)s", logpath=%(logpath)s, chain="%(chain)s"]
                %(splunkconf)s[name=%(__name__)s, logpath=%(logpath)s]



3. Restart Fail2ban, check logging to Syslog:

Now let's test your system, generate a ban event (try to log in through SSH with bad credentials) and check

your Syslog file to find the generated event. (look for the pattern "fail2ban.banevent")

You should find a ban event like this:
Jan 11 20:24:34 myhostname logger[30720]: [fail2ban.banevent]: fail2ban_host: [myfail2ban] Banhost: [xx.xx.xx.xx] jailname: [ssh] numberoffailures: [6] logmessage: [ Jan 11 20:24:32 myhostname sshd[30706]: Received disconnect from xx.xx.xx.xx: 11: Bye Bye [preauth] ] 

Now you're done with Fail2ban, let's configure Syslog ^^


Part 2: Configure Syslog - Standalone and Multi-Hosts


In 2 steps:
  • if you want to manage different Fail2ban servers from Splunk, then read the Multiple Fail2ban client configuration note
  • If you just one host to manage (Fail2ban and Splunk are installed in the same host), then just follow the common configuration section


MULTIPLE FAIL2BAN CLIENT CONFIGURATION NOTE: Remote and centralized Syslog configuration

Configuring Syslog to send events from a Syslog host to a remote Syslog server is out of the scope of this guide.

Therefore, if you want to collect fail2ban events from different hosts, you can choose between different solutions, as:
  • Sending events using Syslog to a remote centralized Syslog
  • Sending events from local log file using Splunk forwarder module
  • Others (homemade scripts, file sharing...)
I would recommend using Rsyslog (default enhanced Syslog for many Linux systems) to achieve this, which is in deed easy enough, robust and efficient.

Here is in 2 steps a quick rsyslog centralized configuration: (remember to restart rsyslog after each modification)

1. In each client rsyslog host, modify "/etc/rsyslog.conf" and add a section to send any events to your Syslog server: (adapt the example IP)

"/etc/rsyslog.conf"
*.* @192.168.1.254:514 

2. In syslog server configuration, create a configuration file that will trapp any remote client Syslog events and put then into a dedicated per host log file:

Ensure your configuration name will be read after the fail2ban syslog config file you will create after. (see above, this is very )

Create "/etc/rsyslog.d/10-fail2ban.conf" with the following content: (Note: The fail2ban config we will create after will be called 08 to be read before this and intercept messages)

"/etc/rsyslog.d/10-fail2ban.conf"
$template RemoteHostFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogfacility-text% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::space-cc,drop-last-lf%\n" 
:inputname, isequal, "imudp" ?PerHostLog;RemoteHostFileFormat
& ~

Restart rsyslog after any config modification.


COMMON CONFIGURATION for Single and Multiple (for the centralized rsyslog server) Fail2ban installation: 

1. Set Syslog to trap ban events to a dedicated logfile

This configuration part will depend on your system and needs, i recommend the use of "rsyslog"

The goal is to configure syslog to trap any event containing a key word "[fail2ban.banevent]" into a dedicated log file

In Debian/Ubuntu systeprintfms for example, create an rsyslog configuration file, example:
Create "/etc/rsyslog.d/08-fail2ban.conf" with the following content: 

"/etc/rsyslog.d/08-fail2ban.conf"
:msg, contains, "[fail2ban.banevent]" /var/log/fail2ban_banevent.log
& ~

Restart rsyslog to take effect:
sudo service rsyslog restart

2. Generate a ban event and check your logfile

Generate a new ban event and check your log file, you should see a new ban event message! 

If you are ok with that, then you're done with system configuration ^^ 



Part 3: Configuration of Splunk (the easy part!)

Here comes the easier part with no doubts :-)

1. Configure Input file
Go to "manager", "Data Input" and configure MANUALLY a new input file pointing to your Fail2ban log file, with following settings:


Host:

You can let the default settings, it does not mind as we don't use it to recognize the fail2ban reporting server.

Source type:

- Set the source Type: Manual
- Source type: fail2ban_banevent

Index:

- Set the destination Index: fail2ban_index





Good news, you're done!!!
Just wait a few minutes to let Splunk get the content of your fail2ban log file, then go to the splunk application Splunk for Fail2ban






Don't hesitate to share any comment with me, this is my very first Splunk application and it may still needs some improvement :-)


11 comments:

  1. I cant handle fail2ban.banevent event. Just it wont show up. Still get:

    fail2ban.actions.action: DEBUG iptables -n -L INPUT | grep -q fail2ban-ssh
    2013-03-18 16:56:55,469 fail2ban.actions.action: DEBUG iptables -n -L INPUT | grep -q fail2ban-ssh returned successfully
    2013-03-18 16:56:55,469 fail2ban.actions.action: DEBUG iptables -I fail2ban-ssh 1 -s 192.168.103.3 -j DROP
    2013-03-18 16:56:55,472 fail2ban.actions.action: DEBUG iptables -I fail2ban-ssh 1 -s 192.168.103.3 -j DROP returned successfully

    Could you help me with this - a I have no idea why it looks like this. I have added splunk.conf and add proper lines to jail.conf

    ReplyDelete
    Replies
    1. Hi,

      For sure, can you check / confirm:

      - Is your log extract coming from your syslog ? In other words, are you logging to syslog instead of a dedicated fail2ban logfile (which is the default config for fail2ban)

      - Can you test to log to Syslog using the command logger, as in splunk.conf if you do such command:

      logger -i "[fail2ban.TEST]:"

      --> Ensure you get this message in your syslog file
      If you don't have this message in your Syslog, then your problem is somewhere else

      If that's ok, there may be a typo somewhere, can you check these 2 config files (2 config files of mine), and copy / paste if required the interesting lines:

      https://dl.dropbox.com/u/63061887/Splunk/SplunkForFail2ban/config_example/config_example.zip

      Please let me know ^^

      Delete
  2. A couple things, your 10-fail2ban.conf for rsyslog is missing a $ in front of template.

    $template RemoteHostFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogfacility-text% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::space-cc,drop-last-lf%\n"
    :inputname, isequal, "imudp" ?RemoteHostFileFormat

    that took me a bit to figure out.

    Next, for some reason my reporting host is getting reported as the failing rule and not the host name, in my case "ssh". Any idea whats messed up there?

    ReplyDelete
    Replies
    1. Here is my raw syslog line

      Apr 27 15:59:40 abq-wordpress-hosted-vm01 logger[4019]: [fail2ban.banevent]: fail2ban_host: [abq-wordpress-hosted-vm01] Banhost: [67.0.72.94] jailname: [ssh] numberoffailures: [3] logmessage: [ Apr 27 15:59:38 abq-wordpress-hosted-vm01 sshd[4003]: Failed password for invalid user admin from 67.0.72.94 port 1418 ssh2 ]

      Delete
    2. Hi,

      My apologies for that!

      I will review that section, can you confirm me that your remote syslog file is being logged into a seperated log file into your rsyslog server ?

      My config file to log remove events into seperated files:

      # Log remote hosts to separate log file
      $template PerHostLog,"/var/log/remote-hosts/%HOSTNAME%/%HOSTNAME%.log"
      $template RemoteHostFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogfacility-text% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::space-cc,drop-last-lf%\n"
      :inputname, isequal, "imudp" ?PerHostLog;RemoteHostFileFormat
      & ~

      Indeed a "$" was missing anyway...

      About this issue, i think something is getting bad in your context with the $HOME_SPLUNK/etc/apps/splunkforfail2ban/default/props.conf which defines events fields:

      ############################################################
      # Rules for fail2ban
      ############################################################

      [fail2ban_banevent]
      FIELDALIAS-realip = srcip AS clientip
      EXTRACT-id = (?i) logger\[(?P[^\]]+)
      EXTRACT-message = (?i) .*?: \[(?P\d+)(?=\])
      EXTRACT-srcip = (?i) Banhost: \[(?P[^\]]+)
      EXTRACT-fail2ban_host = (?i).*?: \[(?P\w+)(?=\])
      EXTRACT-jailname = (?i) jailname: \[(?P[^\]]+)
      EXTRACT-number_of_failures = (?i) .*?: \[(?P\d+)(?=\])

      I have tested your raw file and the rex definition, can change the line:

      EXTRACT-fail2ban_host = (?i).*?: \[(?P\w+)(?=\])

      to:

      EXTRACT-fail2ban_host = (?i)fail2ban_host: \[(?P[^\]]+)

      Then restart Splunk and check, the fail2ban host field should be recognized with success.

      Please let me know, i will review this application...




      Delete
    3. Hi,

      Copy / Paste under comments prevent me from giving you the correct code line, could you contact me by mail ?

      Guilhem

      Delete
    4. yah, contact me at brians at xynergy dot. com

      with

      EXTRACT-fail2ban_host = (?i)fail2ban_host: \[(?P[^\]]+)

      no data shows in the field now.

      Brian

      Delete
  3. Would you know how to define default actions on RHEL/CentOS? It seems action_, action_mw & action_mwl have disappeard (http://centoshelp.org/security/fail2ban/). I'm running fail2ban 0.8.8-3.el6 on CentOS 6.4 X86_64.

    ReplyDelete
    Replies
    1. Hi,

      I think this configuration (the one you linked in centos) is minimalist, do you have any mail advertising when client are being rejected in this config ?

      Check within the file config_examples/jail.conf.example

      You could start copying all the ACTION part (overwritting what could already exist in your own config), restart fail2ban and check

      I think the centos config hard coded default action

      Let me know!

      Note: I recently updated the apps to 2.01, please upgrade

      Delete
    2. I've recompiled fail2ban from the latest source, replaced the original jail.conf with your example and now it works.

      I had to install the latest version of Sideview Utils (http://sideviewapps.com/apps/sideview-utils/) to be able to view the Spunk for Fail2Ban app without any problems. This version isn't available on the official Splunk site.

      Thanks for the great write up.

      ps. You have a typo regarding the latest update, the date should be june 4th instead of july 4th :-)

      Delete
    3. Hi,

      Happy to hear you made it work.
      I've updated various link to specify the V2 of Sideview Utils is required.

      And corrected the typo thanks ^^

      Don't hesitate if you have any comment about the application :-)

      Delete

Please feel free to comment ^^