Service Discovery for NGINX Plus Using DNS SRV Records from Consul

KENNETH

9년 ago

Our previous blog post on service discovery with Consul discussed the importance of service discovery in distributed systems, including service-oriented and microservices architectures. It described how to use NGINX Plus’ on‑the‑fly reconfiguration API to dynamically add or remove load‑balanced servers that are registered with Consul.

In this blog post, we will go over another method to dynamically reconfigure your upstream servers in NGINX Plus using Consul’s DNS interface. With this method, NGINX Plus periodically re‑resolves the service name using Consul DNS. If the list of IP addresses associated with the service has changed, NGINX Plus immediately starts load balancing across the updated group of servers.

To make it easier to combine upstream reconfiguration in NGINX Plus with Consul DNS, we’ve created a sample demo, consul-dns-srv-demo, with step‑by‑step instructions for creating the configuration described in this blog post. In this post, we will walk you through the proof of concept. Using tools like Docker, Docker Compose, and Homebrew, you can spin up a Docker‑based environment in which NGINX Plus load balances HTTP traffic across a group of open source NGINX web servers, with all components running in separate Docker containers.

How the Demo Works

First we spin up a separate Docker container for each of the following apps:

Consul – Performs service discovery and provides DNS service.
Registrator – Registers services with Consul. Registrator monitors for containers being started and stopped and updates Consul when a container changes state.
hello – Simulates a backend server that can be scaled up or down. This is another project from NGINX, Inc., an NGINX web server that serves an HTML page containing the web server’s hostname, IP address, and port.
NGINX Plus R9 – Load balances the backend servers listed above.

The NGINX Plus container listens on the public port 80, and the built‑in NGINX Plus dashboard on port 8080. The Consul container listens on ports 8300, 8400, 8500, and 53 (the last mapped to port 8600 on the Docker host, which listens for DNS queries over both TCP and UDP). Consul advertises the IP address of the Docker host (configured with the environment variable HOST_IP before starting the containers) for intercontainer communication.

Each time a new Docker container that has exposed ports is launched, Registrator registers the associated service with Consul. When a container quits or is removed, Registrator removes it from the Consul service catalog automatically.

By setting environment variables within the containers, we can be more explicit about how to register the services with Consul. For the hello container, we set the environment variable SERVICE_NAME to http to override the default service name of the service object created within Registrator, which later gets proxied to the registry backend (Consul in our case). This tells Consul to return the information (IP address, port, priority, and weight) for all the hello containers identified by service name http when NGINX Plus sends a DNS SRV query using the service=http parameter discussed in the next paragraph.

The NGINX Plus configuration file app.conf included in the demo configures DNS‑based load balancing with these directives:

The resolver directive designates the container named consul, which is the Consul DNS service listening on port 53, as the DNS server. NGINX Plus obtains the IP address of the consul container from the /etc/hosts file in the NGINX Plus container.

The valid parameter means that NGINX Plus ignores the TTL in the records provided by Consul DNS and instead re‑resolves them every 10 seconds.
The service parameter to the server directive, introduced in NGINX Plus R9, enables NGINX Plus to read DNS SRV records and get not only the IP addresses for service.consul from the name server, but also port numbers, weights, and priorities. Consul doesn’t currently support weights and priorities, so in the demo NGINX Plus uses just the port numbers of the backend servers registered with Consul.

resolver consul:53 valid=10s;
upstream backend {

    zone upstream_backend 64k;

    server service.consul service=http resolve;

}
server {

    ...

    location / {

        proxy_pass http://backend;

    }

}

Another feature introduced in NGINX Plus R9 facilitates the use of SRV records – support for DNS lookups over TCP. SRV records are larger than other types of DNS records, making it more likely that the complete DNS response won’t fit in a single 512 byte UDP datagram. The DNS server flags an incomplete response as “truncated”, and NGINX Plus immediately retries the request over TCP. (This feature also makes it easy for NGINX Plus to handle the very large deployments of servers that are common in microservices applications.) Note that you must also configure Consul to set the truncated flag when appropriate, by setting the enable_truncate flag. In the demo, we do this in the consul_dns_config.json file.

Summary

Using Consul’s DNS interface to dynamically reconfigure your upstream servers in NGINX Plus is an alternative to using the NGINX Plus’ on‑the‑fly reconfiguration API, and is useful for organizations that frequently need to change the configuration of upstream server groups, such as those that use autoscaling. A major benefit is that you can manage the servers in the upstream group without modifying the NGINX Plus configuration directly.

Check out our other blogs about service discovery for NGINX Plus (not using SRV records):

And try out for yourself automated reconfiguration of NGINX Plus upstream groups using Consul DNS for service discovery: