Blocking IP on MITIGATOR with Nginx and fail2ban

The following web server security configuration is described:

  • Nginx module ngx_http_limit_req detects the excess of the request limit;
  • fail2ban analyzes error.log, which Nginx uses to report on the excesses;
  • IP is added to the list of blocked by MITIGATOR API.

MITIGATOR API client

There is a script mitigator.py (download) to manage MITIGATOR, in particular to temporarily block an IP address via the MITIGATOR API. If necessary, the script can be modified to perform any other actions on MITIGATOR. The script uses only the standard modules and works with Python 2.7+ and Python 3.

To run the script, you need the MITIGATOR account and the protection policy ID (for example, 42 in the URL .../policies/42 when accessing MITIGATOR). The script also accepts IP and blocking time in seconds. The description of all parameters is printed with the --help (-h) option.

Nginx setup

The ngx_http_limit_req module is built into Nginx and allows you to limit the number of requests per second (RPS) via the Nginx configuration.

Request limit refers to zone. Typically, the zone is the client’s IP (eg. requests from it are limited) or the URI on the site (requests to it are limited), but more complex combinations of request parameters are also possible

Zones are described in the http context. Let’s describe a perip (per IP) zone that can track a 10 RPS limit violation for any one of 10 million IP addresses. To do this, add the line to /etc/nginx.conf through an intermediate file:

mkdir -p /etc/nginx/nginx.conf.d

cat > /etc/nginx/conf.d/limit-req.conf <<'END'
limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;
END

To apply this limit to a part of the site (the location context) and to prevent access from one IP to the page more than 10 times per second, use the following directive in /etc/nginx/sites-available/default (if requests are limited for the default site):

server {
    ...
    location / {
        ...
        limit_req zone=perip burst=20 nodelay;
    }
}

Here burst=20 allows bursts of activity up to 20 RPS (but no more than 10 RPS on average, as described for the zone), and nodelay means that requests over the limit are dropped, and do not wait in line (relevant for DDoS).

Let’s update Nginx config:

nginx -s reload

When the limit is exceeded, lines like this appear in error.log:

2019/01/11 09:27:12 [error] 155#155: *142 limiting requests, excess: 20.200 by zone "perip", client: 10.0.1.254, server: _, request: "GET / HTTP/1.1", host: "10.0.2.254"

fail2ban setup

The fail2ban utility analyzes the logs and, if it finds the specified signs, performs blocking actions. Fail2ban also allows you to manage lists of blocked addresses.

Let’s place the blocking script:

install mitigator.py /usr/local/bin

Let’s create a new fail2ban action that will call the script:

cat >/etc/fail2ban/action.d/mitigator.conf <<'END'
[Definition]
actionban = \
        /usr/local/bin/mitigator.py \
            --server "<server>" \
            --user "<user>" --password "<password>" \
            --no-verify \
            --policy <policy> \
            tbl block --ip <ip> --time <bantime>
actionunban = \
        /usr/local/bin/mitigator.py \
            --server "<server>" \
            --user "<user>" --password "<password>" \
            --no-verify \
            --policy <policy> \
            tbl unblock --ip <ip>
END

Labels <server>, <user>, <password>, <policy>, <ip> and <bantime> must be written as is - these are parameters that will be automatically replaced with correct values. The --no-verify key is needed if MITIGATOR works with a self-signed certificate and you need to disable its verification.

Let’s create a restriction (jail) that will block IP based on entries in error.log:

cat >>/etc/fail2ban/jail.local <<END

[nginx-limit-req]
enabled = true
logpath = %(nginx_error_log)s
banaction = mitigator[server=mitigator.local, user=admin, password=admin, policy=42, bantime=600]
END

The nginx-limit-req module comes with fail2ban and finds the required lines. The banaction=... line specifies the actual parameters of the action: server, username, password, policy ID. For example, the blocking time is 10 minutes.

Let’s reboot fail2ban rules:

fail2ban-client reload

Checking work

Traffic must go through MITIGATOR, general protection and protection of the selected policy must be enabled.

You can simulate an attack with the httperf utility:

httperf --server 192.0.2.20 --num-conn 100

With a successful setup, a few seconds after running httperf on the server, you can see that fail2ban has worked:

$ fail2ban-client status nginx-limit-req
Status for the jail: nginx-limit-req
|- Filter
|  |- Currently failed:	1
|  |- Total failed:	65
|  `- File list:	/var/log/nginx/error.log
`- Actions
   |- Currently banned:	1
   |- Total banned:	1
   `- Banned IP list:	192.0.2.10

On MITIGATOR in the TBL policy card you can check that the attacker’s IP (192.0.2.10 in the example) is on the list of temporarily blocked.

You can unblock IP either through MITIGATOR or fail2ban:

fail2ban-client set nginx-limit-req unbanip 192.0.2.10

If you do not need to block an IP through fail2ban, and it is undesirable to add it to the «White list» (WL or TWL) on MITIGATOR, you can ignore the IP at the fail2ban level:

fail2ban-client set nginx-limit-req addignoreip 192.0.2.10

If fail2ban did not work, errors can be observed in its log:

tail -f /var/log/fail2ban/fail2ban.log