Dynamic IP Blacklisting with NGINX Plus and fail2ban
You may not realize it, but your website is under constant threat. If it’s running WordPress, there will be bots trying to spam you. If it has a login page, there will be password brute-force attacks. You may also consider search engine spiders as unwanted visitors.
Defending your site from unwanted, suspicious, and malicious activity is no easy task. Web Application Firewalls, such as ModSecurity and Wallarm, are an effective tool and should be considered as part of your security stack. For most environments, there’s no such thing as too much security, and a multi-layered approach is invariably the most effective.
In this blog post, we’ll discuss the use of fail2ban as another layer of the web security stack. Fail2ban is an intrusion detection system (IDS) which continually monitors log files for suspicious activity, and then takes one or more preconfigured actions. Typically fail2ban will monitor for failed login attempts and then block (ban) the offending IP address for a period of time. This is a simple, yet effective defense against password brute-force attacks.
With NGINX Plus R13, we introduced a native Key-Value Store and an API. This allows for a wealth of on-the-fly reconfiguration solutions, so that NGINX Plus configuration and behavior can be driven by an external system. In this blog post, we’ll also show how fail2ban can be used to automatically reconfigure NGINX Plus to ignore requests from IP addresses which have caused multiple failed authentication events.
Using a Key-Value Store for IP Blacklisting
The NGINX Plus Key-Value Store is a native, in-memory store with three primary characteristics.
- Key-Value pairs are represented as a JSON object
- Key-Value pairs are managed entirely through an API
- The values are available to NGINX Plus as regular configuration variables
The Key-Value Store is defined by creating a shared memory zone named by the keyval_zone directive. This Key-Value Store can then be populated with an initial set of values by using the HTTP POST method to submit a JSON object to the API. The keyval directive then defines which existing variable will be used as the lookup key ($remote_addr), and the name of a new variable ($num_failures) that will be evaluated from that key’s corresponding value.
The API is enabled by designating a location block as the NGINX Plus API endpoint.
Examining the contents of the blacklist Key-Value Store returns an empty JSON object.
$ curl http://localhost:1111/api/1/http/keyvals/blacklist
{}The Key-Value store can now be populated with an initial key-value pair using the HTTP POST method to submit a new JSON object. Multiple key-value pairs can be POST-ed to an empty Key-Value Store and singularly thereafter.
$ curl -id '{"10.0.0.1":"1"}' http://localhost:1111/api/1/http/keyvals/blacklist
HTTP/1.1 201 Created
…A Key-Value pair is removed by PATCH-ing the key with a value of null.
$ curl -iX PATCH -d '{"10.0.0.1":null}' http://localhost:1111/api/1/http/keyvals/blacklist
HTTP/1.1 204 No Content
…All key-value pairs can be removed from the Key-Value Store by sending the DELETE method.
$ curl -iX DELETE http://localhost:1111/api/1/http/keyvals/blacklist
HTTP/1.1 204 No Content
…A simple implementation of IP blacklisting can now be configured.
This configuration snippet configures port 80 as a web server. Line 18 evaluates the $num_failures variable to the value part of the key-value pair in the blacklist shared memory zone that matches the key that corresponds to the $remote_addr variable (client IP address). This process of evaluating $num_failures is more clearly expressed sequentially:
- Pass the value of
$remote_addrto theblacklistKey-Value Store - If
$remote_addris an exact match for a key then obtain the value of that key-value pair - Return the value as
$num_failures.
And so, if the client IP address has been POST-ed to the Key-Value Store then all requests will result in a HTTP 403 error (forbidden). The advantage of this approach, as opposed to using a map for the same purpose, is that the IP blacklist can be controlled by an external system.
Using fail2ban to Dynamically Manage the IP Blacklist
As mentioned above, fail2ban is commonly used to detect suspicious and/or malicious activity in log files, and then take action to protect the system. The default action is to configure iptables to drop all packets originating from the IP address that was logged. This approach, although effective in blocking a malicious actor, provides a very poor user experience for genuine users, especially if they’ve forgotten their password. To these users, it appears as if the website is simply unavailable.
The following configuration describes a fail2ban action that submits the offending IP address to the NGINX Plus blacklist Key-Value Store. NGINX Plus then displays a more helpful error page and simultaneously applies a request rate limit to that IP address so that if the actions are part of an attack, then that attack is effectively neutralized.
We assume a default installation of fail2ban on the same host as NGINX Plus, with all configuration under the /etc/fail2ban directory.
Note: The pathnames under each section of code describe the file’s location within the /etc/fail2ban directory.
This fail2ban action uses the NGINX Plus API to add and remove “banned” IP addresses within the blacklist Key-Value Store in the same way as our simple example above.
This fail2ban jail enables the built-in filter that detects failed login attempts using NGINX’s HTTP basic authentication module and applies the action defined in nginx-plus-blacklist.conf. Many other built-in filters are available, and can be easily created to detect unwanted activity in NGINX access logs.
The NGINX Plus configuration applies HTTP basic authentication to the default Welcome to NGINX page.
Line 11 defines the keyval_zone as before and with the addition of the state parameter, which specifies a file that is used to persist the state of the Key-Value Store so that NGINX Plus can be stopped and restarted without emptying the Key-Value Store. The Key-Value Store is persisted across regular configuration reloads without specifying a state file.
Line 14 specifies a maximum request rate of 20 requests per minute for each client IP address and is known by the shared memory zone of 20permin. This rate limit is then applied when an IP address is added to the blacklist by fail2ban (line 30).
The server block defines a web server listening on the default port of 80, where all requests are matched by the location / block (line 20). The root directive (line 18) specifies where our web content resides. Lines 21 and 22 specify that HTTP basic authentication is required for this site and that the password database of authorized users is contained within the users.htpasswd file. The documentation for the auth_basic_user_file directive describes the format of that file. Note that this configuration is unencrypted for testing purposes and any production web site using HTTP basic authentication should use SSL/TLS for transport security.
When the client IP address has been blacklisted (line 24), instead of returning the HTTP 403 error code, we rewrite the request to be /banned.html which continues processing in the location block that exactly matches that URI (line 29). This returns a static web page that explains that the user’s IP address has been blocked due to excessive login failures. It also applies a restrictive rate limit (line 30) that prevents a malicious client from unnecessarily consuming system resources.
We can simulate successive failed login attempts and the corresponding fail2ban activity as follows.
$ curl -i http://admin:password@www.example.com/
HTTP/1.1 401 Unauthorized …
$ curl -i http://admin:admin@www.example.com/
HTTP/1.1 401 Unauthorized …
$ curl -i http://admin:pass1234@www.example.com/
HTTP/1.1 401 Unauthorized …
$ curl -i http://admin:letmein@www.example.com/
HTTP/1.1 401 Unauthorized …
$ curl -i http://admin:fred@www.example.com/
HTTP/1.1 401 Unauthorized …
$ curl http://admin:P@ssw0rd@www.example.com/
Banned
…
$ tail -f /var/log/fail2ban.log
2017-08-17 13:55:18,903 fail2ban.filter [28498]: INFO [nginx-http-auth] Found 172.16.52.1
2017-08-17 13:55:28,836 fail2ban.filter [28498]: INFO [nginx-http-auth] Found 172.16.52.1
2017-08-17 13:57:49,228 fail2ban.filter [28498]: INFO [nginx-http-auth] Found 172.16.52.1
2017-08-17 13:57:50,286 fail2ban.filter [28498]: INFO [nginx-http-auth] Found 172.16.52.1
2017-08-17 13:57:52,015 fail2ban.filter [28498]: INFO [nginx-http-auth] Found 172.16.52.1
2017-08-17 13:57:52,088 fail2ban.actions [28498]: NOTICE [nginx-http-auth] Ban 172.16.52.1
2017-08-17 13:59:52,379 fail2ban.actions [28498]: NOTICE [nginx-http-auth] Unban 172.16.52.1This dynamic IP blacklisting solution can now run without any further configuration changes. Fail2ban watches the NGINX log files and adds banned IP addresses to the NGINX Plus Key-Value Store using the API. After the configured bantime of 120 seconds, the offending IP address is then removed from the blacklist, again using the NGINX Plus API.
The NGINX Plus API and Key-Value Store make this type of integration possible. Advanced configuration solutions can now be achieved without requiring the construction of NGINX configuration files and performing reloads. We’d love to hear about how you’re using these features to create your own on-the-fly configuration solutions.
To try out NGINX Plus for yourself: start your free 30‑day trial today or contact us for a live demo.
The post Dynamic IP Blacklisting with NGINX Plus and fail2ban appeared first on NGINX.
Source: Dynamic IP Blacklisting with NGINX Plus and fail2ban
