Splunk Howto - Splunk for Fail2ban, get a the Fail2ban Multi-host frontend with Splunk!

*** Updated June 9, 2013  ***

Current Version = 2.02

Splunk (if you don't yet know it) is an incredibly powerful solution that collects, indexes and exploits any kind of data from any system, offering you as many solution as you need and even the possibility to create custom applications with graphical front-ends. (dashboards, reports, saved searches...)

In a few words, i am really impressed by Splunk, i think i've been looking for this for many many years!

Don't hesitate to take a look at main Splunk Website, you will easily find a lot of information and great documentations: http://www.splunk.com/

Splunk can be used for free with some little restrictions. (not more than 500Mb of input data per day)

Details: http://docs.splunk.com/Documentation/Splunk/5.0.1/Admin/TypesofSplunklicenses

I developed my first Splunk application "Splunk For Fail2ban" to provide a cool frontend and log managing tool associated with the well known and powerful Fail2ban tool. (take a look at my older post: http://youresuchageek.blogspot.fr/2012/11/howto-fail2ban-secure-your-network.html)

To install this addon, follow this link on Splunkbase or install it through the standard Splunk application process search online: 

http://splunk-base.splunk.com/apps/72087/splunk-for-fail2ban

Splunk pre-requirements:

Ensure to install requirements Splunk addons:

• Sideview utils addon V2.x (not the older version available in Splunk Base)
• Google Maps addon
• Maxmind geo location addon

Splunk For Fail2ban provides:

A complete Dashboard Overview of Fail2ban activity for all managed systems: 

Home page with realtime quick summary activity overview and links to interfaces:

A complete Dashboard Overview of Fail2ban activity for all managed systems: 

Activity overview:

Activity and Alert Trend:

Various Top 10 Charts and stats:

Google Maps Dashboard, identify the source of connexion attempts

A Fail2ban Event search interface with selection per kind of data (IPs, ID, Jail...)

Pre-defined major searches to get all the most important information

System view: Index activity

Installation and utilization

Introduction

Installing and configuring Splunk is out of the scope of this post, still installing Splunk is really easy and well done, in 10 minutes you'll be done ^^

Download Splunk !

As a brieve description, here is how Splunk for Fail2ban works:

- We modify Fail2ban to add a specific message for each ban action and containing fields Splunk will analyse

- Through Syslog, we can manage as many Fail2ban servers as required

- Splunk collects our data and produces the IT intelligency

Installation and configuration will be done in a few steps:

1. Modifying Fail2ban configuration files related to the ban action (the goal is send fields we will analyse with Splunk)

2. Setting up Fail2ban to log to Syslog system

3. Setting up Syslog to trap custom Fail2ban events into a specific log file (can be local or remote Syslog if numerous Fail2ban hosts)

4. Installation and configuring Splunk for Fail2ban

Part 1: Configure Fail2ban

1. Set Fail2ban output to Syslog

I recommend the use of "rsyslog" as your main Syslog management, it comes with much more improvement than the standard Syslog. (http://www.rsyslog.com/)

First, we need to set Fail2ban to log its messages into Syslog instead of a standard log file.

To do so, edit "/etc/fail2ban/fail2ban.conf" and set:

logtarget = SYSLOG

2. Add a new action.d configuration file for events logging

Note: 
See this configuration sample if required: splunk.conf.example

Create a new file: "/etc/fail2ban/actions.d/splunk.conf" with the following content:

[Definition]
actionban = logger -i "[fail2ban.banevent]: fail2ban_host: [`hostname`] \
Banhost: [<ip>] jailname: [<name>] numberoffailures: [<failures>] \
logmessage: [ `grep '\<<ip>\>' <logpath> | tail -1` ] "

[Init]

3. Configure "/etc/fail2ban/jail.conf":

Depending of your wish, you can set Fail2ban to use 1 of these 3 actions: (by editing /etc/fail2ban/jail.conf)

• action_ = Fail2ban will temporarely ban the IP source host
• action_mw = Fail2ban will temporarely ban the IP host and send a warning mail including whois result request
• action_mwl = Fail2ban will temporarely ban the IP host and send a warning mail including whois result request and log traces

All you need is to modify jail.conf for all these action level to include our specific logging for Splunk.

Note: 
See this configuration sample if required: jail.conf.example

In jail.conf, add the following line just before the 3 action definition lines (action_, action_mw, action_mwl)

# Name of Splunk config file
splunkconf = splunk

Then, add a new line related splunk underneath each action level, your configuration file will looks like:

#
# Action shortcuts. To be used to define action parameter

# The simplest action to take: ban only
action_ = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
            %(splunkconf)s[name=%(__name__)s, logpath=%(logpath)s]

# ban & send an e-mail with whois report to the destemail.
action_mw = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
              %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s", protocol="%(protocol)s", chain="%(chain)s"]
                %(splunkconf)s[name=%(__name__)s, logpath=%(logpath)s]

# ban & send an e-mail with whois report and relevant log lines
# to the destemail.
action_mwl = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
               %(mta)s-whois-lines[name=%(__name__)s, dest="%(destemail)s", logpath=%(logpath)s, chain="%(chain)s"]
                %(splunkconf)s[name=%(__name__)s, logpath=%(logpath)s]

3. Restart Fail2ban, check logging to Syslog:

Now let's test your system, generate a ban event (try to log in through SSH with bad credentials) and check

your Syslog file to find the generated event. (look for the pattern "fail2ban.banevent")

You should find a ban event like this:

Jan 11 20:24:34 myhostname logger[30720]: [fail2ban.banevent]: fail2ban_host: [myfail2ban] Banhost: [xx.xx.xx.xx] jailname: [ssh] numberoffailures: [6] logmessage: [ Jan 11 20:24:32 myhostname sshd[30706]: Received disconnect from xx.xx.xx.xx: 11: Bye Bye [preauth] ] 

Now you're done with Fail2ban, let's configure Syslog ^^

Part 2: Configure Syslog - Standalone and Multi-Hosts

In 2 steps:

• if you want to manage different Fail2ban servers from Splunk, then read the Multiple Fail2ban client configuration note
• If you just one host to manage (Fail2ban and Splunk are installed in the same host), then just follow the common configuration section

MULTIPLE FAIL2BAN CLIENT CONFIGURATION NOTE: Remote and centralized Syslog configuration

Configuring Syslog to send events from a Syslog host to a remote Syslog server is out of the scope of this guide.

Therefore, if you want to collect fail2ban events from different hosts, you can choose between different solutions, as:

• Sending events using Syslog to a remote centralized Syslog
• Sending events from local log file using Splunk forwarder module
• Others (homemade scripts, file sharing...)

I would recommend using Rsyslog (default enhanced Syslog for many Linux systems) to achieve this, which is in deed easy enough, robust and efficient.

Here is in 2 steps a quick rsyslog centralized configuration: (remember to restart rsyslog after each modification)

1. In each client rsyslog host, modify "/etc/rsyslog.conf" and add a section to send any events to your Syslog server: (adapt the example IP)

"/etc/rsyslog.conf"

*.* @192.168.1.254:514 

2. In syslog server configuration, create a configuration file that will trapp any remote client Syslog events and put then into a dedicated per host log file:

Ensure your configuration name will be read after the fail2ban syslog config file you will create after. (see above, this is very )

Create "/etc/rsyslog.d/10-fail2ban.conf" with the following content: (Note: The fail2ban config we will create after will be called 08 to be read before this and intercept messages)

"/etc/rsyslog.d/10-fail2ban.conf"

$template RemoteHostFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogfacility-text% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::space-cc,drop-last-lf%\n" 
:inputname, isequal, "imudp" ?PerHostLog;RemoteHostFileFormat
& ~

Restart rsyslog after any config modification.

COMMON CONFIGURATION for Single and Multiple (for the centralized rsyslog server) Fail2ban installation: 

1. Set Syslog to trap ban events to a dedicated logfile

This configuration part will depend on your system and needs, i recommend the use of "rsyslog"

The goal is to configure syslog to trap any event containing a key word "[fail2ban.banevent]" into a dedicated log file

In Debian/Ubuntu systeprintfms for example, create an rsyslog configuration file, example:

Create "/etc/rsyslog.d/08-fail2ban.conf" with the following content: 

"/etc/rsyslog.d/08-fail2ban.conf"

:msg, contains, "[fail2ban.banevent]" /var/log/fail2ban_banevent.log
& ~

Restart rsyslog to take effect:

sudo service rsyslog restart

2. Generate a ban event and check your logfile

Generate a new ban event and check your log file, you should see a new ban event message! 

If you are ok with that, then you're done with system configuration ^^ 

Part 3: Configuration of Splunk (the easy part!)

Here comes the easier part with no doubts :-)

1. Configure Input file

Go to "manager", "Data Input" and configure MANUALLY a new input file pointing to your Fail2ban log file, with following settings:

Host:

You can let the default settings, it does not mind as we don't use it to recognize the fail2ban reporting server.

Source type:

- Set the source Type: Manual
- Source type: fail2ban_banevent

Index:

- Set the destination Index: fail2ban_index

Good news, you're done!!!
Just wait a few minutes to let Splunk get the content of your fail2ban log file, then go to the splunk application Splunk for Fail2ban

Don't hesitate to share any comment with me, this is my very first Splunk application and it may still needs some improvement :-)