From the Fail2Ban Homepage:
Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs — too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, courier, ssh, etc).
When a web site comes under attack, occasionally, the resources of the machine become exhausted creating a denial of service. This, possibly, is not the aim of the attacker but a side effect of their Good Work. Without an investigation that would find the attacks the application’s developer might ask for the hosting machine to be given more resources. More of a resource, like RAM, would help the application to avoid the denial of service but would afford the attack more time in which to be successful. And, of course, this would be expensive. The increase in RAM could be used elsewhere and the machine would not always need the extra resource because the attacker will move on if the application is esoteric or reasonably well written. It is easy to give extra RAM to a machine but it is harder to take it away.
These attacks are attempts to penetrate a system through a flaw in the application or operating system. Most of the attacks we see are against popular applications: phpMyAdmin, WordPress, and their Windows equivalents. As long as good practices, when developing a LAMP application, are used e.g. using prepared SQL statements to prevent SQL injection, the main application might be safer from attack than are the more popular well understood applications. This situation is weakened when a utility like phpMyAdmin is used to provide functionality not yet present in the main application.
So, instead of adding more RAM to mitigate against an accidental denial of service and to protect well known applications as well as the main application, we could instead monitor for attacks and proactively protect the host machine accordingly. This is where we might use Fail2Ban.
As most attacks are against the application and most Internet connected application are web applications we are probably talking about Apache or NGINX for delivery of content. We will spot attacks through the logs files, for example, error_log:
[Sun Nov 05 09:34:27 2017] [error] [client 22.214.171.124] File does not exist: /home/apache/public/htdocs/admin [Sun Nov 05 09:34:27 2017] [error] [client 126.96.36.199] File does not exist: /home/apache/public/htdocs/admin [Sun Nov 05 09:34:27 2017] [error] [client 188.8.131.52] File does not exist: /home/apache/public/htdocs/admin [Sun Nov 05 09:34:27 2017] [error] [client 184.108.40.206] File does not exist: /home/apache/public/htdocs/apache-default [Sun Nov 05 09:34:27 2017] [error] [client 220.127.116.11] File does not exist: /home/apache/public/htdocs/cpanelphpmyadmin [Sun Nov 05 09:34:27 2017] [error] [client 18.104.22.168] File does not exist: /home/apache/public/htdocs/cpphpmyadmin [Sun Nov 05 09:34:28 2017] [error] [client 22.214.171.124] File does not exist: /home/apache/public/htdocs/dbadmin [Sun Nov 05 09:34:28 2017] [error] [client 126.96.36.199] File does not exist: /home/apache/public/htdocs/db
This shows one IP address (bad machine on the Internet) running through obvious URLs trying to find phpMyAdmin but failing (404). It shows that there are many attempts in a short amount of time. Fail2Ban could be set up to ban the bad machine for a period of time after the first three failures preventing the rest of the attempts draining the resource of the machine. This log does not show actions that might exhaust a machine’s resources but if phpMyAdmin was found a series of tests could then be run against it and that might create a denial of service.
A Solution for this Attack
Fail2Ban is available to CentOS:
$ yum install fail2ban
The interesting aspects of this package for us are:
I’ll create this in /etc/fail2ban/jail.d/apache-fourofour.conf
[apache-fourofour] enabled = true port = http,https filter = apache-fourofour logpath = /home/apache/p*/logs/error_log maxretry = 20 findtime = 120 bantime = 16400
The above isn’t the complete truth…because that would be telling!
And in /etc/fail2ban/filter.d/apache-fourofour.conf
[Definition] failregex = [client <HOST>] File does not exist: * ignoreregex = .*(robots.txt|favicon.ico|jpg|png)
The inspiration for this filter can be found here, created by Dominic Derdau.
So, a quick check to see if we get any hits (against a file we know has some):
fail2ban-regex home/apache/public/logs/error_log-20171112 /etc/fail2ban/filter.d/apache-fourofour.conf
reveals 184 hits.
Let’s enable the fail2ban service and start it. This machine is CentOS 6:
# chkconfig fail2ban on # service fail2ban start
and check that the firewall has Fail2Ban gaols set up:
# iptables -L -v -n . . . 0 0 f2b-apache-fourofour tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443 . . . Chain f2b-apache-fourofour (1 references)
Now, how to test? A quick script that quits once it is blocked at the firewall:
n=0 while : do echo "Attempt $n" let n=$n+1 curl http://xxx.xxx.xxx.xxx/bob if [ $? -eq 7 ] then echo 'Firewall block!' break fi done
The script above may surprise you. It did surprise me. I got many more attempts in before I was blocked. This is because Apache, in this case, will serve pages as fast as it can but will write to the logs like it’s a hot lazy Sunday afternoon. You can tune /etc/fail2ban/jail.d/apache-fourofour.conf to suit your needs.
Eeek, I’m in gaol:
Chain f2b-apache-fourofour (1 references) pkts bytes target prot opt in out source destination 17 1020 REJECT all -- * * 10.3.232.4 0.0.0.0/0 reject-with icmp-port-unreachable 190 16313 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
So, this works. Check that it works, too, after a reboot. Fail2Ban should start running after iptables otherwise you’ll miss out on the gaol creation.