Dynamic A/B Testing with NGINX Plus
The key-value store feature was introduced in NGINX Plus R13 for HTTP traffic and extended to Stream traffic in NGINX Plus R14. This feature provides an API to dynamically maintain values that can be used as part of the NGINX Plus configuration, without requiring a reload of the configuration. There are many possible use cases for this feature and I have no doubt that our customers will find a variety of ways to take advantage it.
This blog post describes one use case, dynamically altering how the split_clients module is used to do A/B testing.
The Key-Value Store
The NGINX Plus API can be used to maintain a set of key-value pairs that can be accessed at runtime by NGINX Plus. For example, let’s look at the use case where you want to keep a list of client IP addresses that are blacklisted. The key would be the client IP address, which is available in the $remote_addr variable. The value would be something to indicate that an IP address is blacklisted.
We can give the value variable a name of $blacklist_status and set it to 1 to indicate that the client IP address is blacklisted. To configure this, we follow these steps:
- Set up a
key_zoneto contain the key-value pairs - Give the
key_zonea name - Specify the maximum amount of memory to allocate for it
- Optionally, specify a state file to store the entries so they persist across NGINX Plus restarts
For the state file, we have created the directory /etc/nginx/state_files and made it writable by the unprivileged user that runs the NGINX worker processes (as defined by the user directive):
keyval_zone zone=blacklist:64k;
state=/etc/nginx/state_files/blacklist.json;Then, you define the key-value pair:
keyval $remote_addr;
$blacklist_status zone=blacklist;Key-value pairs are set initially with an HTTP POST request. For example:
# curl -id '{"10.11.12.13":1}' http://localhost/api/2/http/keyvals/blacklistAnd key-value pair values can be modified using an HTTP PATCH request. For example:
# curl -iX PATCH -d '{"10.11.12.13":0}' http://localhost/api/2/http/keyvals/blacklistKey-value pairs can be removed using an HTTP PATCH request and a null value. For example:
# curl -iX PATCH -d '{"10.11.12.13":null}' http://localhost/api/2/http/keyvals/blacklistThe value of a key-value pair will be in the $blacklist_status variable based on using $remote_addr as the key.
Split Clients for A/B Testing
The split_clients module allows you to split incoming traffic by any value in a request. The module sets a value depending on the percentages you specify. For example, let’s say you have two upstream groups, appversion1 and appversion2, and you want to send 5% of the traffic to appversion2 and 95% of the traffic to appversion1.
For this, we can use the client IP address, as defined by the $remote_addr variable, as the value to do the split on. We configure the split_clients module to set the variable $upstream to the name of the upstream group.
The following would be a basic NGINX configuration:
map $remote_addr $upstream {
5% appversion2;
* appversion1;
}
upstream appversion1 {
...
}
upstream appversion2 {
...
}
server {
listen 80;
location / {
proxy_pass http://$upstream;
}
}
Prior to NGINX Plus R13, if you wanted to change the percentages for the distribution, you would edit the configuration file and reload the configuration. But now, you simply change the percentage value stored in the key-value pair and the distribution changes to fit, without the need for a reload.
Using the Key-Value Store with Split Clients
Using the key-value store feature and the API, you can set a value that specifies what percentage to use, without needing to do a configuration reload. To add to the example use case, let’s say we have decided that we want NGINX Plus to support the following options for how much traffic gets sent to appversion2: 0%, 5%, 10%, 25%, 50% and 100%.
Additionally, let’s say that we want to be able to set the split percentage based on the Host header. The following NGINX Plus configuration implements this functionality.
First we set up the key-value store:
keyval_zone zone=split:64k;
state=/etc/nginx/state_files/split.json;
keyval $host $split_level zone=split;As mentioned when discussing the initial use case, we would likely use the client IP address, $remote_addr, as the value to drive the split from. However, when doing simple testing using a tool like curl, all the requests will come from a single IP address, so the effects of the split are not visible.
For testing, we want to use a value that is more random, so we use $request_id. To make the configuration easily switchable from test to production, we set a new variable in the server block, $client_ip, that is set to either $remote_addr or $request_id. Then we set up the split_clients configuration.
The resulting variables, eg. split0, split5, etc., will be set the name of the upstream group:
split_clients $client_ip $split0 {
* appversion1;
}
split_clients $client_ip $split5 {
5% appversion2;
* appversion1;
}
split_clients $client_ip $split10 {
10% appversion2;
* appversion1;
}
split_clients $client_ip $split25 {
25% appversion2;
* appversion1;
}
split_clients $client_ip $split50 {
50% appversion2;
* appversion1;
}
split_clients $client_ip $split100 {
* appversion2;
}Now that we have the key-value store and split_clients configured, we can set up a map to set the $upstream variable to the upstream group specified in the appropriate split variable:
map $split_level $upstream {
0 $split0;
5 $split5;
10 $split10;
25 $split25;
50 $split50;
100 $split100;
default $split0;
}Finally, we have the rest of the configuration for the upstream groups and the virtual server. Note that we have also configured the API which is used for the key-value store and the status dashboard. This is the new status dashboard in NGINX Plus R14:
upstream appversion1 {
zone appversion1 64k;
server 192.168.50.100;
server 192.168.50.101;
}
upstream appversion2 {
zone appversion2 64k;
server 192.168.50.102;
server 192.168.50.103;
}
server {
listen 80;
status_zone test;
#set $client_ip $remote_addr; # Production
set $client_ip $request_id; # For testing only
location / {
proxy_pass http://$upstream;
}
location = /dashboard.html {
root /usr/share/nginx/html;
}
location /api {
api write=on;
}
}
Using this configuration, we can now control how the traffic is split between the appversion1 and appversion2 upstream groups by sending an API request to NGINX Plus and setting the $split_level value for a host name. For example, the following two requests can be sent to NGINX Plus so that 5% of the traffic for www.example.com is sent to the appversion2 upstream group and 25% of the traffic for www2.example.com is sent to the appversion2 upstream group:
# curl -id '{"www.example.com":5}' http://localhost/api/2/http/keyvals/split
# curl -id '{"www2.example.com":25}' http://localhost/api/2/http/keyvals/splitTo change the value for www.example.com to 10:
# curl -iX PATCH -d '{"www.example.com":10}' http://localhost/api/2/http/keyvals/splitTo clear a value:
# curl -iX PATCH -d '{"www.example.com":null}' http://localhost/api/2/http/keyvals/splitAfter each one of these requests, you will see that NGINX Plus immediately starts using the new split value.
Here is the full configuration file:
# Set up a key-value store to specify the percentage to send to # each upstream group based on the host header.
keyval_zone zone=split:64k; state=/etc/nginx/state_files/split.json;
keyval $host $split_level zone=split;
# For a real application you would probably use $remote_addr
# with split_clients. But, if testing with one client,
# $remote_addr will always be the same, so to get some
# randomness use $request_id instead. In the server block,
# $client_ip is set to either $remote_addr or $client_ip.
split_clients $client_ip $split0 {
* appversion1;
}
split_clients $client_ip $split5 {
5% appversion2;
* appversion1;
}
split_clients $client_ip $split10 {
10% appversion2;
* appversion1;
}
split_clients $client_ip $split25 {
25% appversion2;
* appversion1;
}
split_clients $client_ip $split50 {
50% appversion2;
* appversion1;
}
split_clients $client_ip $split100 {
* appversion2;
}
map $split_level $upstream {
0 $split0;
5 $split5;
10 $split10;
25 $split25;
50 $split50;
100 $split100;
default $split0;
}
upstream appversion1 {
zone appversion1 64k;
server 192.168.50.100;
server 192.168.50.101;
}
upstream appversion2 {
zone appversion2 64k;
server 192.168.50.102;
server 192.168.50.103;
}
server {
listen 80;
status_zone test;
#set $client_ip $remote_addr; # Production
set $client_ip $request_id; # For testing only
location / {
proxy_pass http://$upstream;
}
# Configure the API and Status Dashboard. For production,
# access can be restricted with the allow and deny directives.
location = /dashboard.html {
root /usr/share/nginx/html;
}
location /api {
api write=on;
}
}
Conclusion
This is just one example of what you can do with the key-value store feature. You can use a similar approach for request-rate limiting, bandwidth limiting, or connection limiting. Dynamic IP Blacklisting with NGINX Plus and fail2ban discusses a detailed IP blacklisting use case, and there are many other possibilities.
If you don’t already have NGINX Plus, you can get the free trial and give it a try.
The post Dynamic A/B Testing with NGINX Plus appeared first on NGINX.
Leave a Reply