Using Fail2Ban to protect your LAMP application

From the Fail2Ban Homepage:

Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the  malicious signs — too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, courier, ssh, etc).

The Problem

When a web site comes under attack, occasionally, the resources of the machine become exhausted creating a denial of service.  This, possibly, is not the aim of the attacker but a side effect of their Good Work.  Without an investigation that would find the attacks the application’s developer might ask for the hosting machine to be given more resources.  More of a resource, like RAM, would help the application to avoid the denial of service but would afford the attack more time in which to be successful.  And, of course, this would be expensive.  The increase in RAM could be used elsewhere and the machine would not always need the extra resource because the attacker will move on if the application is esoteric or reasonably well written.  It is easy to give extra RAM to a machine but it is harder to take it away.

These attacks are attempts to penetrate a system through a flaw in the application or operating system.  Most of the attacks we see are against popular applications: phpMyAdmin, WordPress, and their Windows equivalents.  As long as good practices, when developing a LAMP application, are used e.g. using prepared SQL statements to prevent SQL injection, the main application might be safer from attack than are the more popular well understood applications.  This situation is weakened when a utility like phpMyAdmin is used to provide functionality not yet present in the main application.

So, instead of adding more RAM to mitigate against an accidental denial of service and to protect well known applications as well as the main application, we could instead monitor for attacks and proactively protect the host machine accordingly.  This is where we might use Fail2Ban.

The Attack

As most attacks are against the application and most Internet connected application are web applications we are probably talking about Apache or NGINX for delivery of content.  We will spot attacks through the logs files, for example, error_log:

[Sun Nov 05 09:34:27 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/admin
[Sun Nov 05 09:34:27 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/admin
[Sun Nov 05 09:34:27 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/admin
[Sun Nov 05 09:34:27 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/apache-default
[Sun Nov 05 09:34:27 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/cpanelphpmyadmin
[Sun Nov 05 09:34:27 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/cpphpmyadmin
[Sun Nov 05 09:34:28 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/dbadmin
[Sun Nov 05 09:34:28 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/db

This shows one IP address (bad machine on the Internet) running through obvious URLs trying to find phpMyAdmin but failing (404).  It shows that there are many attempts in a short amount of time.  Fail2Ban could be set up to ban the bad machine for a period of time after the first three failures preventing the rest of the attempts draining the resource of the machine.  This log does not show actions that might exhaust a machine’s resources but if phpMyAdmin was found a series of tests could then be run against it and that might create a denial of service.

A Solution for this Attack

Fail2Ban is available to CentOS:

$ yum install fail2ban

The interesting aspects of this package for us are:

/etc/fail2ban/jail.d/
/etc/fail2ban/filter.d

I’ll create this in /etc/fail2ban/jail.d/apache-fourofour.conf

[apache-fourofour]
enabled = true
port = http,https
filter = apache-fourofour
logpath = /home/apache/p*/logs/error_log
maxretry = 20
findtime = 120
bantime = 16400

The above isn’t the complete truth…because that would be telling!

And in /etc/fail2ban/filter.d/apache-fourofour.conf

[Definition]
failregex = [[]client <HOST>[]] File does not exist: *
ignoreregex = .*(robots.txt|favicon.ico|jpg|png)

The inspiration for this filter can be found here, created by Dominic Derdau.

So, a quick check to see if we get any hits (against a file we know has some):

fail2ban-regex home/apache/public/logs/error_log-20171112 /etc/fail2ban/filter.d/apache-fourofour.conf

reveals 184 hits.

Let’s enable the fail2ban service and start it.  This machine is CentOS 6:

# chkconfig fail2ban on
# service fail2ban start

and check that the firewall has Fail2Ban gaols set up:

# iptables -L -v -n
.
.
.
 0 0 f2b-apache-fourofour tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443
.
.
.
Chain f2b-apache-fourofour (1 references)

Now, how to test?  A quick script that quits once it is blocked at the firewall:

n=0
while :
do
  echo "Attempt $n"
  let n=$n+1
  curl http://xxx.xxx.xxx.xxx/bob
  if [ $? -eq 7 ]
  then
    echo 'Firewall block!'
    break
  fi
done

The script above may surprise you.  It did surprise me.  I got many more attempts in before I was blocked.  This is because Apache, in this case, will serve pages as fast as it can but will write to the logs like it’s a hot lazy Sunday afternoon.  You can tune /etc/fail2ban/jail.d/apache-fourofour.conf to suit your needs.

Eeek, I’m in gaol:

Chain f2b-apache-fourofour (1 references)
 pkts bytes target prot opt in out source destination 
 17 1020 REJECT all -- * * 10.3.232.4 0.0.0.0/0 reject-with icmp-port-unreachable 
 190 16313 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0

So, this works.  Check that it works, too, after a reboot.  Fail2Ban should start running after iptables otherwise you’ll miss out on the gaol creation.

Well done you.

About c3iq

Opensource, Linux, Unix, Fish, Family
This entry was posted in ITMS, Linux SysAdmin, Secuity and tagged , , , , , . Bookmark the permalink.