Splunk Howto - Splunk for Fail2ban, get a the Fail2ban Multi-host frontend with Splunk!

*** Updated June 9, 2013  ***

Current Version = 2.02

Splunk (if you don't yet know it) is an incredibly powerful solution that collects, indexes and exploits any kind of data from any system, offering you as many solution as you need and even the possibility to create custom applications with graphical front-ends. (dashboards, reports, saved searches...)

In a few words, i am really impressed by Splunk, i think i've been looking for this for many many years!

Don't hesitate to take a look at main Splunk Website, you will easily find a lot of information and great documentations: http://www.splunk.com/

Splunk can be used for free with some little restrictions. (not more than 500Mb of input data per day)

Details: http://docs.splunk.com/Documentation/Splunk/5.0.1/Admin/TypesofSplunklicenses

I developed my first Splunk application "Splunk For Fail2ban" to provide a cool frontend and log managing tool associated with the well known and powerful Fail2ban tool. (take a look at my older post: http://youresuchageek.blogspot.fr/2012/11/howto-fail2ban-secure-your-network.html)

To install this addon, follow this link on Splunkbase or install it through the standard Splunk application process search online: 

http://splunk-base.splunk.com/apps/72087/splunk-for-fail2ban

Splunk pre-requirements:

Ensure to install requirements Splunk addons:

• Sideview utils addon V2.x (not the older version available in Splunk Base)
• Google Maps addon
• Maxmind geo location addon

Splunk For Fail2ban provides:

A complete Dashboard Overview of Fail2ban activity for all managed systems: 

Home page with realtime quick summary activity overview and links to interfaces:

A complete Dashboard Overview of Fail2ban activity for all managed systems: 

Activity overview:

Activity and Alert Trend:

Various Top 10 Charts and stats:

Google Maps Dashboard, identify the source of connexion attempts

A Fail2ban Event search interface with selection per kind of data (IPs, ID, Jail...)

Pre-defined major searches to get all the most important information

System view: Index activity

Installation and utilization

Introduction

Installing and configuring Splunk is out of the scope of this post, still installing Splunk is really easy and well done, in 10 minutes you'll be done ^^

Download Splunk !

As a brieve description, here is how Splunk for Fail2ban works:

- We modify Fail2ban to add a specific message for each ban action and containing fields Splunk will analyse

- Through Syslog, we can manage as many Fail2ban servers as required

- Splunk collects our data and produces the IT intelligency

Installation and configuration will be done in a few steps:

1. Modifying Fail2ban configuration files related to the ban action (the goal is send fields we will analyse with Splunk)

2. Setting up Fail2ban to log to Syslog system

3. Setting up Syslog to trap custom Fail2ban events into a specific log file (can be local or remote Syslog if numerous Fail2ban hosts)

4. Installation and configuring Splunk for Fail2ban

Part 1: Configure Fail2ban

1. Set Fail2ban output to Syslog

I recommend the use of "rsyslog" as your main Syslog management, it comes with much more improvement than the standard Syslog. (http://www.rsyslog.com/)

First, we need to set Fail2ban to log its messages into Syslog instead of a standard log file.

To do so, edit "/etc/fail2ban/fail2ban.conf" and set:

logtarget = SYSLOG

2. Add a new action.d configuration file for events logging

Note: 
See this configuration sample if required: splunk.conf.example

Create a new file: "/etc/fail2ban/actions.d/splunk.conf" with the following content:

[Definition]
actionban = logger -i "[fail2ban.banevent]: fail2ban_host: [`hostname`] \
Banhost: [<ip>] jailname: [<name>] numberoffailures: [<failures>] \
logmessage: [ `grep '\<<ip>\>' <logpath> | tail -1` ] "

[Init]

3. Configure "/etc/fail2ban/jail.conf":

Depending of your wish, you can set Fail2ban to use 1 of these 3 actions: (by editing /etc/fail2ban/jail.conf)

• action_ = Fail2ban will temporarely ban the IP source host
• action_mw = Fail2ban will temporarely ban the IP host and send a warning mail including whois result request
• action_mwl = Fail2ban will temporarely ban the IP host and send a warning mail including whois result request and log traces

All you need is to modify jail.conf for all these action level to include our specific logging for Splunk.

Note: 
See this configuration sample if required: jail.conf.example

In jail.conf, add the following line just before the 3 action definition lines (action_, action_mw, action_mwl)

# Name of Splunk config file
splunkconf = splunk

Then, add a new line related splunk underneath each action level, your configuration file will looks like:

#
# Action shortcuts. To be used to define action parameter

# The simplest action to take: ban only
action_ = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
            %(splunkconf)s[name=%(__name__)s, logpath=%(logpath)s]

# ban & send an e-mail with whois report to the destemail.
action_mw = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
              %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s", protocol="%(protocol)s", chain="%(chain)s"]
                %(splunkconf)s[name=%(__name__)s, logpath=%(logpath)s]

# ban & send an e-mail with whois report and relevant log lines
# to the destemail.
action_mwl = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
               %(mta)s-whois-lines[name=%(__name__)s, dest="%(destemail)s", logpath=%(logpath)s, chain="%(chain)s"]
                %(splunkconf)s[name=%(__name__)s, logpath=%(logpath)s]

3. Restart Fail2ban, check logging to Syslog:

Now let's test your system, generate a ban event (try to log in through SSH with bad credentials) and check

your Syslog file to find the generated event. (look for the pattern "fail2ban.banevent")

You should find a ban event like this:

Jan 11 20:24:34 myhostname logger[30720]: [fail2ban.banevent]: fail2ban_host: [myfail2ban] Banhost: [xx.xx.xx.xx] jailname: [ssh] numberoffailures: [6] logmessage: [ Jan 11 20:24:32 myhostname sshd[30706]: Received disconnect from xx.xx.xx.xx: 11: Bye Bye [preauth] ] 

Now you're done with Fail2ban, let's configure Syslog ^^

Part 2: Configure Syslog - Standalone and Multi-Hosts

In 2 steps:

• if you want to manage different Fail2ban servers from Splunk, then read the Multiple Fail2ban client configuration note
• If you just one host to manage (Fail2ban and Splunk are installed in the same host), then just follow the common configuration section

MULTIPLE FAIL2BAN CLIENT CONFIGURATION NOTE: Remote and centralized Syslog configuration

Configuring Syslog to send events from a Syslog host to a remote Syslog server is out of the scope of this guide.

Therefore, if you want to collect fail2ban events from different hosts, you can choose between different solutions, as:

• Sending events using Syslog to a remote centralized Syslog
• Sending events from local log file using Splunk forwarder module
• Others (homemade scripts, file sharing...)

I would recommend using Rsyslog (default enhanced Syslog for many Linux systems) to achieve this, which is in deed easy enough, robust and efficient.

Here is in 2 steps a quick rsyslog centralized configuration: (remember to restart rsyslog after each modification)

1. In each client rsyslog host, modify "/etc/rsyslog.conf" and add a section to send any events to your Syslog server: (adapt the example IP)

"/etc/rsyslog.conf"

*.* @192.168.1.254:514 

2. In syslog server configuration, create a configuration file that will trapp any remote client Syslog events and put then into a dedicated per host log file:

Ensure your configuration name will be read after the fail2ban syslog config file you will create after. (see above, this is very )

Create "/etc/rsyslog.d/10-fail2ban.conf" with the following content: (Note: The fail2ban config we will create after will be called 08 to be read before this and intercept messages)

"/etc/rsyslog.d/10-fail2ban.conf"

$template RemoteHostFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogfacility-text% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::space-cc,drop-last-lf%\n" 
:inputname, isequal, "imudp" ?PerHostLog;RemoteHostFileFormat
& ~

Restart rsyslog after any config modification.

COMMON CONFIGURATION for Single and Multiple (for the centralized rsyslog server) Fail2ban installation: 

1. Set Syslog to trap ban events to a dedicated logfile

This configuration part will depend on your system and needs, i recommend the use of "rsyslog"

The goal is to configure syslog to trap any event containing a key word "[fail2ban.banevent]" into a dedicated log file

In Debian/Ubuntu systeprintfms for example, create an rsyslog configuration file, example:

Create "/etc/rsyslog.d/08-fail2ban.conf" with the following content: 

"/etc/rsyslog.d/08-fail2ban.conf"

:msg, contains, "[fail2ban.banevent]" /var/log/fail2ban_banevent.log
& ~

Restart rsyslog to take effect:

sudo service rsyslog restart

2. Generate a ban event and check your logfile

Generate a new ban event and check your log file, you should see a new ban event message! 

If you are ok with that, then you're done with system configuration ^^ 

Part 3: Configuration of Splunk (the easy part!)

Here comes the easier part with no doubts :-)

1. Configure Input file

Go to "manager", "Data Input" and configure MANUALLY a new input file pointing to your Fail2ban log file, with following settings:

Host:

You can let the default settings, it does not mind as we don't use it to recognize the fail2ban reporting server.

Source type:

- Set the source Type: Manual
- Source type: fail2ban_banevent

Index:

- Set the destination Index: fail2ban_index

Good news, you're done!!!
Just wait a few minutes to let Splunk get the content of your fail2ban log file, then go to the splunk application Splunk for Fail2ban

Don't hesitate to share any comment with me, this is my very first Splunk application and it may still needs some improvement :-)

SSH / Google 2-Step Authentication How-To : Enhance your SSH security with Google Two factor Authentication Service

*** Updated March 9, 2013  ***

Major changes:
03/09/2013 - Added missing pam settings upon user comment

The Goal:

Google provides for free a great service to enhance your Google account security called "Google 2-Step Authentication"  (also called two factor authentication) and offers a real strong authentication mechanism.

This service can also easily be used to enhance your SSH access security.

In a few words, you will be able to protect your SSH access with strong authentication using your smartphone as a software token.

Do not hesitate to read official Google page if you need more information:

https://support.google.com/accounts/bin/topic.py?hl=en&topic=28786   

You may also read my article about configuring it to protect your Google account access:

http://youresuchageek.blogspot.fr/2012/07/google-account-howto-protect-and-secure.html

Other useful sources (thanks to various authors):

http://www.mnxsolutions.com/security/two-factor-ssh-with-google-authenticator.html

What you need:

• A running Linux Box with SSH installed and accessible
• A smartphone : Iphone, Android or RIM

Step 1: Install Google Authenticator

Tested under Ubuntu 12.04 TLS:

sudo apt-get install libpam-google-authenticator

Step 2: Configure SSH to use Google Authenticator

Edit "/etc/pam.d/sshd" with your favorite text editor and add:

auth required pam_google_authenticator.so

Edit "/etc/ssh/sshd_config" and set:

ChallengeResponseAuthentication yes

Edit "/etc/pam.d/common-auth" and set:


auth required pam_google_authenticator.so
auth [success=1 default=ignore] pam_unix.so nullok_secure


As the user you want to connect with, configure your Google two factors authentication:

$ google-authenticator

https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/user@host%3Fsecret%3DZDTR6VU5FR5OIZ3G

<BAR CODE>
       
Your new secret key is: ZDTR6VU5FR5OIZ3G
Your verification code is 843231
Your emergency scratch codes are:
  31043901
  75807840
  98606066
  42902460
  31208347

Do you want me to update your "~/.google_authenticator" file (y/n)

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, tokens are good for 30 seconds and in order to compensate for
possible time-skew between the client and the server, we allow an extra
token before and after the current time. If you experience problems with poor
time synchronization, you can increase the window from its default
size of 1:30min to about 4min. Do you want to do so (y/n) y

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting (y/n) y

Note: 
Emergency codes are provided in case of your phone would be unavailable, you should keep it somewhere 

Open your Google Authenticator application on your phone, click on "+" and "read bar code", get the bar code provided by the terminal, it will be added automatically in the application.


Restart ssh:

sudo service ssh restart

Note:
I recommend you to keep your opened terminal up in case you would be unable to connect 

Step 3: Check authentication

Try to connect to your host using Google code provided by your phone:

ssh user@host
Password: 
Verification code: 
Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-26-generic x86_64)

Last login: Sun Jul 15 11:28:17 2012 from XXX.XXX.X.XX
user@host:~$ 

Ajaxterm - Howto: SSH access to your host through an SSL secured Web page

Ajaxterm - Howto: SSH access to your host through an SSL secured Web page

Author's official page:

http://antony.lesuisse.org/software/ajaxterm/

The Goal:

Ajaxterm will you provide you a way to access with SSH to your server through a Web server page secured with SSL. (recommended)

In a few words, you will be able to access to your SSH session without the need of an SSH client and as if it was any simple web Page :)

What you need:

- An SSH running server
- Opened and / or redirected ports to allow connection from outside to your Web SSH page
- Apache Web server and Openssl
- Optionally a third party server you may use as an SSH gateway to access to your final SSH server (improves security by avoiding direct connection to your real system, see my previous post: http://youresuchageek.blogspot.fr/2012/06/apache-2-reverse-proxy-howoto-protect.html)

Step 1: Install Ajaxterm

Nothing more simple, on Debian based system:

sudo apt-get install install ajaxterm

Step 2: Base configuration

Configuration is really easy, you will find 2 configurations files.

"/etc/default/ajaxterm":

• Change Web server listening port if needed, by default it will listen to 8022 : 

# Allow to change the default port used by Ajaxterm                                                                                         
#PORT="8022"                                              

• Change SSH server listening port if needed, if you your SSH server isn't listening to standard port, you have to change it :

# Allow to use a different port than 22 to connect to the ssh server                                                                        
#SERVERPORT="22"                    

"/etc/ajaxterm.conf":

Adapt your Width and Height preferences:

// Sets the terminal width (default: 80)                                                                                                    
width=140;                                                                                                                                  
                                                                                                                                            
// Sets the terminal height (default: 25)                                                                                                   
height=50;   

After installation, Ajaxterm will immediately be available accessing your localhost : http://localhost:8022

Step 3: Apache configuration

You may have or not a third party server running Apache and acting as a reverse proxy.

In both cases (third party or not), configure an apache instance secured by SSL:

If not yet installed and configured, in the example we will use 443 as the standard SSL port but you can change it to whatever you want:

Install Apache 2 on Debian and derived systems:

sudo apt-get install apache2 openssl

Activate required Apache modules:

sudo a2enmod proxy proxy_http proxy_connect ssl

Deactivate defaults http and https sites (we don't need it and don't want it):

sudo a2dissite default

Configure Apache to listen to required ports:
edit "/etc/apache2/ports.conf" as follows:

NameVirtualHost *:443
Listen 443

Create your auto signed certificate to encrypt and secure Web traffic with SSL (use whatever you want when asked by openssl) :

sudo openssl req -x509 -nodes -days 3650 -newkey rsa:1024 -out /etc/apache2/server.crt -keyout /etc/apache2/server.key

Configure an "htpassword" file for simple authentication (at least recommended)

generate an .htpasswd file to protect your site by authentication (adapt your username) :
NB:

• "-c" option will create a new file
• "-m" option will use MD5 to secure password, by default htpasswd uses DES which will only consider first 8 characters 

sudo htpasswd -c -m /etc/apache2/.htpasswd username

Create your ajaxterm reverse proxy site:

create a new file "/etc/apache2/sites-available/ajaxterm:

NB: 

• If your are using a third party server, adapt HOSTNAME to match SSH running host or IP
• If not, change HOSTNAME to localhost

<VirtualHost *:443>                                                                                                                         
  ServerName XXXXXXXXXXXXXX                                                                                                              
  ProxyRequests Off                                                                                                                         
  ProxyVia Off                                                                                                                              
    <Proxy *>                                                                                                                               
     Order deny,allow                                                                                                                       
     Allow from all                                                                                                                         
    </Proxy>                                                                                                                                
  ProxyPass / http://HOSTNAME:8022/                                                                                                        
  ProxyPassReverse / http://HOSTNAME:8022/                                                                                                  
  <Location />                                                                                                                              
    Order allow,deny                                                                                                                        
    Allow from all                                                                                                                          
    AuthName "Access Restricted"                                                                                                            
    AuthType Basic                                                                                                                          
    AuthUserFile "/etc/apache2/.htpasswd"                                                                                                   
    Require valid-user                                                                                                                      
  </Location>                                                                                                                               
  LogLevel info                                                                                                                             
  CustomLog /var/log/apache2/access_ajaxterm.log combined                                                                                   
  ErrorLog /var/log/apache2/error_ajaxterm.log                                                                                              
  SSLEngine on                                                                                                                              
  SSLCertificateFile /etc/apache2/server.crt                                                                                                
  SSLCertificateKeyFile /etc/apache2/server.key                                                                                             
</VirtualHost>                                        

Enable the site:

sudo a2ensite ajaxterm

Restart Apache:

sudo service apache2 restart (or "sudo apachectl restart" if you prefer)

Test your ajaxterm by accessing https://<reverse_proxy_ip>

You're done and should have now access to your with SSH through an SSL secured Web page :)