Howto - Apache2 ModSecurity - Enhance your Web Server and Applications security with an Opensource Web Application Firewall (WAF)

*** Updated March 9, 2013  ***

03/09/2013: Various typo corrections


The Goal: 

ModSecurity is an OpenSource Web Application Firewall, today stable when used with Apache Web server, it will allow to drastically increase your Web security by analyzing in real time any kind of traffic and applying various analysis rules.

This tutorial is based on Ubuntu and Debian based system, bu can be easily adapted to any Unix / Linux based OS.

Major sources: 

Official ModSecurity site:
http://www.modsecurity.org/

ModSecurity Rules:
http://spiderlabs.github.com/owasp-modsecurity-crs/

Various interesting sources:
http://www.thefanclub.co.za/how-to/how-install-apache2-modsecurity-and-modevasive-ubuntu-1204-lts-server
http://www.ebelair.fr/2011/06/07/installer-et-configurer-modsecurity/



Summary of steps: 

Step 1: Prepare your systemStep 2: Download components - ModSecurity Sources and Rules

Step 3: Compile and install ModSecurity Module
Step 4: Configure ModSecurity module and rules
Step 5: Main ModSecurity Configuration
Step 6: Test your configuration, Simulate a generic attack and malicious code upload
Step 7: Whitelist configuration, test your Web sites and add whitelist exceptions if required
Step 8: Advanced vulnerabilities and attack tools

FAQ & issues

I will assume you already have a running OS and Apache installation.

To implement ModSecurity for Apache on Debian / Ubuntu based systems, you can use official OS Repositories or download, compile and configure latest official versions of ModSecurity sources and rules.

This tutorial is based on the second solution because we wan't to take benefit of very last versions.


Step 1: Prepare your system

Install various dependencies:

$ sudo apt-get install g++ make autoconf automake libtool flex bison gcc apache2-threaded-dev libxml2-dev libcurl4-gnutls-dev libapr1 libapr1-dev libpcre3 libpcre3-dev libxml2 libxml2-dev

ModSecurity can be combined with an Antivirus if you want to scan for malicious code any incoming file, this can be a great idea to secure your Web Server.
Clamav is an Opensource Antivirus and will be used to achieve this.

Install Clamav:

$ sudo apt-get install clamav clamav-base clamav-freshclam

An Apache module called "unique_id" is required to use ModSecurity.
This module should be installed by default.

$ sudo a2enmod unique_id

Step 2: Download components - ModSecurity Sources and Rules

I would advise to download and keep files into "/usr/local/src/modsecurity" but this is up to you to change it.

Download required components:

• ModSecurity sources:

Go to http://www.modsecurity.org/download/ and download last version  "modsecurity*apache*.tar.gz"

• ModSecurity Core Rules:

Go to http://spiderlabs.github.com/owasp-modsecurity-crs/ and download last modsecurity*apache*.tar.gz


Step 3: Compile and install ModSecurity Module

Extract Modsecurity Sources:

$ tar -xvzf <replace with Modsecurity Sources archive version name>
$ cd <extracted directory>

Configure, compile and install:

$ ./autogen.sh
$ ./configure
$ make
$ sudo make install

The module core will be installed by default in "/usr/local/modsecurity/lib".


Step 4: Configure ModSecurity module and rules

Create module loading file in Apache:

Create a new file "/etc/apache2/mods-available/security2_module.load" and add:

LoadFile libxml2.so
LoadFile liblua5.1.so
LoadModule security2_module /usr/local/modsecurity/lib/mod_security2.so

Extract Modsecurity Core Rules:

$ tar -xvzf <replace with Modsecurity Core Rules archive version name>
$ sudo mv <extracted directory> /usr/local/src/modsecurity-crs

Prepare your Apache configuration: 

$ sudo mv <extracted directory> /usr/local/src/modsecurity-crs
$ sudo mkdir -p /etc/apache2/modsecurity/conf
$ sudo mkdir -p /etc/apache2/modsecurity/activated_rules
$ sudo cp /usr/local/src/modsecurity-crs/*.conf.example /usr/local/src/modsecurity-crs/modsecurity.conf
$ sudo ln -s /usr/local/src/modsecurity-crs/modsecurity.conf /etc/apache2/modsecurity/conf/modsecurity.conf

Activate recommended rules by creating symlinks:

$ for f in `ls base_rules/`; do sudo ln -s /usr/local/src/modsecurity-crs/base_rules/$f /etc/apache2/modsecurity/activated_rules/$f; done
$ for f in `ls optional_rules/ | grep comment_spam`; do sudo ln -s /usr/local/src/modsecurity-crs/optional_rules/$f /etc/apache2/modsecurity/activated_rules/$f; done

Add symlink to "util" directory:

$ sudo ln -s /usr/local/src/modsecurity-crs/util /etc/apache2/modsecurity/util

Add module launch configuration file:

Create a new file "/etc/apache2/conf.d/modsecurity.conf" and add:

<IfModule security2_module>
 Include /etc/apache2/modsecurity/conf/*.conf
 Include /etc/apache2/modsecurity/activated_rules/*.conf
<IfModule>

Activate modsecurity module:

$ sudo a2enmod security2_module

Step 5: Main ModSecurity Configuration


Now, let's create the module configuration file, based on recommended version by modsecurity:
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual

Create the main Modsecurity configuration file:

/etc/apache2/mods-available/security2_module.conf :

Modifications i recommend:

# Line 7: Change "SecRuleEngine DetectionOnly" to "SecRuleEngine On" if you want active response from the module.
In detection mode only, no actions but logging will be done by modsecurity module.

# Line 167: Comment out and change debug log file, change "#SecDebugLog /opt/modsecurity/var/log/debug.log" to "SecDebugLog /var/log/apache2/modsec_debug.log"

# Line 187: Change main modsec log from"SecAuditLog /var/log/modsec_audit.log" to "SecAuditLog /var/log/apache2/modsec_audit.log"

# Line 147: Change "#SecUploadDir /opt/modsecurity/var/upload/" to "/var/cache/www-upload" (will be created later)

# Line 32/33: Depending on your needs, you have set the maximum allowed file size for uploads in your sites, there is an hard limit 1GB, if you want this, then set:
SecRequestBodyLimit 1073741824
SecRequestBodyNoFilesLimit 1073741824

If you want to use Clamav to scan uploaded files:


#Line 160: add the following line:
SecRule FILES_TMPNAMES "@inspectFile /etc/apache2/modsecurity/util/runav.pl" "id:159,phase:2,t:none,log,deny,msg:'Malicious Code Detected, access denied'"


/etc/apache2/mods-available/security2_module.conf :

# -- Rule engine initialization ----------------------------------------------

# Enable ModSecurity, attaching it to every transaction. Use detection
# only to start with, because that minimises the chances of post-installation
# disruption.
#
SecRuleEngine On


# -- Request body handling ---------------------------------------------------

# Allow ModSecurity to access request bodies. If you don't, ModSecurity
# won't be able to see any POST parameters, which opens a large security
# hole for attackers to exploit.
#
SecRequestBodyAccess On


# Enable XML request body parser.
# Initiate XML Processor in case of xml content-type
#
SecRule REQUEST_HEADERS:Content-Type "text/xml" \
     "id:'200000',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=XML"


# Maximum request body size we will accept for buffering. If you support
# file uploads then the value given on the first line has to be as large
# as the largest file you are willing to accept. The second value refers
# to the size of data, with files excluded. You want to keep that value as
# low as practical.
#
SecRequestBodyLimit 1073741824
SecRequestBodyNoFilesLimit 1073741824

# Store up to 128 KB of request body data in memory. When the multipart
# parser reachers this limit, it will start using your hard disk for
# storage. That is slow, but unavoidable.
#
SecRequestBodyInMemoryLimit 131072

# What to do if the request body size is above our configured limit.
# Keep in mind that this setting will automatically be set to ProcessPartial
# when SecRuleEngine is set to DetectionOnly mode in order to minimize
# disruptions when initially deploying ModSecurity.
#
SecRequestBodyLimitAction Reject

# Verify that we've correctly processed the request body.
# As a rule of thumb, when failing to process a request body
# you should reject the request (when deployed in blocking mode)
# or log a high-severity alert (when deployed in detection-only mode).
#
SecRule REQBODY_ERROR "!@eq 0" "id:'200001', phase:2,t:none,log,deny, \
  status:400,msg:'Failed to parse request body.',logdata:'%{reqbody_error_msg}',severity:2"

# By default be strict with what we accept in the multipart/form-data
# request body. If the rule below proves to be too strict for your
# environment consider changing it to detection-only. You are encouraged
# _not_ to remove it altogether.
#
SecRule MULTIPART_STRICT_ERROR "!@eq 0" \
  "id:'200002',phase:2,t:none,log,deny,status:400, \
  msg:'Multipart request body failed strict validation: \
  PE %{REQBODY_PROCESSOR_ERROR}, \
  BQ %{MULTIPART_BOUNDARY_QUOTED}, \
  BW %{MULTIPART_BOUNDARY_WHITESPACE}, \
  DB %{MULTIPART_DATA_BEFORE}, \
  DA %{MULTIPART_DATA_AFTER}, \
  HF %{MULTIPART_HEADER_FOLDING}, \
  LF %{MULTIPART_LF_LINE}, \
  SM %{MULTIPART_MISSING_SEMICOLON}, \
  IQ %{MULTIPART_INVALID_QUOTING}, \
  IP %{MULTIPART_INVALID_PART}, \
  IH %{MULTIPART_INVALID_HEADER_FOLDING}, \
  FL %{MULTIPART_FILE_LIMIT_EXCEEDED}'"

# Did we see anything that might be a boundary?
#
SecRule MULTIPART_UNMATCHED_BOUNDARY "!@eq 0" "id:'200003',phase:2,t:none,log,deny, \
  status:400,msg:'Multipart parser detected a possible unmatched boundary.'"

# PCRE Tuning
# We want to avoid a potential RegEx DoS condition
#
SecPcreMatchLimit 1000
SecPcreMatchLimitRecursion 1000

# Some internal errors will set flags in TX and we will need to look for these.
# All of these are prefixed with "MSC_".  The following flags currently exist:
#
# MSC_PCRE_LIMITS_EXCEEDED: PCRE match limits were exceeded.
#
SecRule TX:/^MSC_/ "!@streq 0" \
        "id:'200004',phase:2,t:none,deny,msg:'ModSecurity internal error flagged: %{MATCHED_VAR_NAME}'"


# -- Response body handling --------------------------------------------------

# Allow ModSecurity to access response bodies. 
# You should have this directive enabled in order to identify errors
# and data leakage issues.
# 
# Do keep in mind that enabling this directive does increases both
# memory consumption and response latency.
#
SecResponseBodyAccess On

# Which response MIME types do you want to inspect? You should adjust the
# configuration below to catch documents but avoid static files
# (e.g., images and archives).
#
SecResponseBodyMimeType text/plain text/html text/xml

# Buffer response bodies of up to 512 KB in length.
SecResponseBodyLimit 524288

# What happens when we encounter a response body larger than the configured
# limit? By default, we process what we have and let the rest through.
# That's somewhat less secure, but does not break any legitimate pages.
#
SecResponseBodyLimitAction ProcessPartial


# -- Filesystem configuration ------------------------------------------------

# The location where ModSecurity stores temporary files (for example, when
# it needs to handle a file upload that is larger than the configured limit).
# 
# This default setting is chosen due to all systems have /tmp available however, 
# this is less than ideal. It is recommended that you specify a location that's private.
#
SecTmpDir /tmp/

# The location where ModSecurity will keep its persistent data.  This default setting 
# is chosen due to all systems have /tmp available however, it
# too should be updated to a place that other users can't access.
#
SecDataDir /tmp/


# -- File uploads handling configuration -------------------------------------

# The location where ModSecurity stores intercepted uploaded files. This
# location must be private to ModSecurity. You don't want other users on
# the server to access the files, do you?
#
SecUploadDir /var/cache/www-upload

# By default, only keep the files that were determined to be unusual
# in some way (by an external inspection script). For this to work you
# will also need at least one file inspection rule.
#
#SecUploadKeepFiles RelevantOnly

# Uploaded files are by default created with permissions that do not allow
# any other user to access them. You may need to relax that if you want to
# interface ModSecurity to an external program (e.g., an anti-virus).
#
#SecUploadFileMode 0600
SecRule FILES_TMPNAMES "@inspectFile /etc/apache2/modsecurity/util/runav.pl" "id:159,phase:2,t:none,log,deny,msg:'Malicious Code Detected, access denied'"

# -- Debug log configuration -------------------------------------------------

# The default debug log configuration is to duplicate the error, warning
# and notice messages from the error log.
#
SecDebugLog /var/log/apache2/modsec_debug.log
#SecDebugLogLevel 3


# -- Audit log configuration -------------------------------------------------

# Log the transactions that are marked by a rule, as well as those that
# trigger a server error (determined by a 5xx or 4xx, excluding 404,  
# level response status codes).
#
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"

# Log everything we know about a transaction.
SecAuditLogParts ABIJDEFHZ

# Use a single file for logging. This is much easier to look at, but
# assumes that you will use the audit log only occasionally.
#
SecAuditLogType Serial
SecAuditLog /var/log/apache2/modsec_audit.log

# Use concurrent logging
#SecAuditLogType Concurrent
#SecAuditLog "|/opt/modsecurity/bin/mlogc /opt/modsecurity/etc/mlogc.conf"

# Specify the path for concurrent audit logging.
#SecAuditLogStorageDir /opt/modsecurity/var/audit/


# -- Miscellaneous -----------------------------------------------------------

# Use the most commonly used application/x-www-form-urlencoded parameter
# separator. There's probably only one application somewhere that uses
# something else so don't expect to change this value.
#
SecArgumentSeparator &

# Settle on version 0 (zero) cookies, as that is what most applications
# use. Using an incorrect cookie version may open your installation to
# evasion attacks (against the rules that examine named cookies).
#
SecCookieFormat 0

# Specify your Unicode Code Point.
# This mapping is used by the t:urlDecodeUni transformation function
# to properly map encoded data to your language. Properly setting
# these directives helps to reduce false positives and negatives.
#
#SecUnicodeCodePage 20127
#SecUnicodeMapFile unicode.mapping

Create upload dir cache:

$ sudo mkdir /var/cache/www-upload
$ sudo chown www-data:wwwdata /var/cache/www-upload

Create whitelist file (blank for now):

$ sudo touch /etc/apache2/modsecurity/conf/whitelist.conf

Were're done with configuration, restart Apache2:

$ sudo service apache2 restart

Step 6: Test your configuration, Simulate a generic attack and malicious code upload


Test your configuration:

1. Main log file

Upon Apache2 start-up, you should see a new file "/var/log/apache2/modsec_audit.log", to see its content in real time:

$ tail -f /var/log/apache2/modsec_audit.log

2. Simulate a simple attack

Open your Web Browser and simulate a generic attack (adapt to your configuration, name and port):
http://localhost/test.php?secret_file=/etc/passwd

If you have activated SecRuleEngine to On, then your get an refused error from Apache.
Otherwise, an attack should have been logged in main audit log.

If ok:

Forbidden

You don't have permission to access / on this server.

In log you should see something like this:

Message: Access denied with code 403 (phase 2). Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:secret_file. [file "/etc/apache2/modsecurity/activated_rules/modsecurity_crs_40_generic_attacks.conf"] [line "205"] [id "950005"] [rev "2"] [msg "Remote File Access Attempt"] [data "Matched Data: /etc/ found within ARGS:secret_file: /etc/passwd"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.6"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/WEB_ATTACK/FILE_INJECTION"] [tag "WASCTC/WASC-33"] [tag "OWASP_TOP_10/A4"] [tag "PCI/6.5.4"]
Action: Intercepted (phase 2)
Apache-Handler: application/x-httpd-php
Stopwatch: 1352556406954555 3620 (- - -)
Stopwatch2: 1352556406954555 3620; combined=2104, p1=546, p2=1462, p3=0, p4=0, p5=95, sr=121, sw=1, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.0 (http://www.modsecurity.org/); OWASP_CRS/2.2.6.
Server: Apache
Engine-Mode: "ENABLED"

3. Try to upload a malicious test file

Create locally a file "testeicar.txt" with following content and try to upload it to your Web Server:

X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*

Upload should fail and you should see this kind of message in modsec log:

Message: Access denied with code 403 (phase 2). File "/tmp//20121110-143758-UJ5Ytn8AAQEAAFjiESIAAAAI-file-pCUZqH" rejected by the approver script "/etc/apache2/modsecurity/util/runav.pl": 0 clamscan: Eicar-Test-Signature [file "/etc/apache2/mods-enabled/security2_module.conf"] [line "160"] [id "159"] [msg "Malicious Code Detected, access denied"]
Action: Intercepted (phase 2)
Apache-Handler: application/x-httpd-php
Stopwatch: 1352554678179590 9450540 (- - -)
Stopwatch2: 1352554678179590 9450540; combined=9447023, p1=562, p2=9446396, p3=0, p4=0, p5=64, sr=117, sw=1, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.0 (http://www.modsecurity.org/); OWASP_CRS/2.2.6.
Server: Apache
Engine-Mode: "ENABLED"

Step 7: Whitelist configuration, test your Web sites and add whitelist exceptions if required


You could probably encounter actions refused based on rules policies with your Web Sites and Applications, so it is recommended to test and add exceptions when required.

As far en example, in my case using Ajaxplorer and uploading files won't possible in default configuration because some rules policies will reject it.

The way to deal with that is almost simple, keep a terminal view on modsec log and test normal user action, each rejected action will be associated to a unique id, we will add exceptions to these ids to allow normal users actions.

Example with ajaxplorer and file upload:

Without whitelist exception, upload fails with following message log: 

Message: Access denied with code 400 (phase 2). Match of "eq 0" against "MULTIPART_UNMATCHED_BOUNDARY" required. [file "/etc/apache2/mods-enabled/security2_module.conf"] [line "80"] [id "200003"] [msg "Multipart parser detected a possible unmatched boundary."]
Action: Intercepted (phase 2)
Apache-Handler: application/x-httpd-php
Stopwatch: 1352555512125594 4755100 (- - -)
Stopwatch2: 1352555512125594 4755100; combined=1441, p1=1276, p2=65, p3=0, p4=0, p5=99, sr=336, sw=1, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.0 (http://www.modsecurity.org/); OWASP_CRS/2.2.6.
Server: Apache
Engine-Mode: "ENABLED"

This is the first policy blocking our normal user action, so we will had an exception associated with this id:

Edit "/etc/apache2/modsecurity/conf/whitelist.conf" and add:

<LocationMatch "/ajaxplorer/">
SecRuleRemoveById 200003
</LocationMatch>

Then reload or restart Apache2 (sudo service apache2 reload) and restart your normal user action, if it still fails then re-check your modsec log, note the id and add it to exception.

Finally, for Ajaxplorer/owncloud example, i had to deal with these exceptions:

<LocationMatch "/ajaxplorer/">
SecRuleRemoveById 200003 960024 960915
</LocationMatch>

Proceed the same way with all your sites and applications, and you will be done!

Note:

If you have any problem with whitelist by location like exception not been working, try to apply your exceptions directly in the Apache configuration file.

Behind the directive DocumentRoot, append your exceptions like this:

SecRuleRemoveById 200003 960024 960915

 And restart/reload Apache.


Step 8: Advanced vulnerabilities and attack tools

Now that your Apache installation has been secured, you should think about testing your site with advanced vulnerabilities tools.

Here are some tools i recommend:

1. Nessus

Nessus is a well known security scan tool, you can get for a free a personal user key and use the software to scan your sites or hosts.

See:
http://www.tenable.com/


Installation is really simple, then simply launch a Web Application scan and wait for the report, you should see many output in log showing ModSecurity activity.

2. Nikto

Also very powerful and simple to use, Nikto will carefully scan and launch various attacks/exploit against your Web Server.

The result will instantly visible in ModSecurity main log with thousands of events logs, i would recommend to to launch 2 scan, a first scan with ModSecurity activated and second with ModSecurity deactivated and just compare results ^^

See:
http://cirt.net/

Example of use:

Launching a scan in http:

perl nikto.pl -h HOST -p PORT

Launching a scan in https:

perl nikto.pl -h HOST -p PORT -ssl

3. Web-sorrow

Another tool, under hard tests :-)

http://code.google.com/p/web-sorrow/

FAQ & Issues

• How to activate / deactivate ModSecurity Module

Nothing more simple:

Activating:

$ sudo a2enmod security2_module
$ sudo service apache2 restart

Deactivating:

$ sudo a2dismod security2_module
$ sudo service apache2 restart

• How to partially deactivate ModSecurity for a location

You may want to partially deactivate ModSecurity for a specific location, to do this add a such section into your whitelist:

<LocationMatch "/application/">
SecRuleEngine Off
</LocationMatch>

And restart / reload Apache to apply.

• How to deactivate ModSecurity for a full Apache instance

You may also want to deactivate ModSecurity by instance, to do so edit the instance Apache configuration file and this behind the DocumentRoot line:

SecRuleEngine Off

And restart / reload Apache to apply.

• How to update ModSecurity Rules

An auto update tool is provided with ModSecurity but the provided repository does not seem to be as up to date as the spiderlabs link... at this writing time the modsecurity contains v2.2.5 where spiderlabs provides 2.2.6 ^^

The update tool "rules-updater.pl" is located in "util" directory. 

You can also simply update rules by extracting and overwriting content in /usr/local/src/modsecurity-crs