Service Discovery in NGINX Plus via etcd

KENNETH

9년 ago

Service discovery is one of the big challenges for applications with a microservices architecture and for other distributed or service-oriented systems. The network locations of service instances can change frequently due to autoscaling, failures, or upgrades, so they are usually dynamically assigned. That means applications need a sophisticated service discovery mechanism for tracking the existence and current location of a service instance.

This blog post describes how to dynamically add or remove servers in an NGINX Plus upstream group by registering them with etcd, a popular open source tool for service discovery. The solution uses NGINX Plus’ on-the-fly reconfiguration API, which eliminates the need to reload the NGINX Plus configuration when changing the set of load-balanced servers. With NGINX Plus Release 8 (R8), changes you make with the API can be configured to persist across restarts and configuration reloads.

CoreOS has developed etcd as a distributed, consistent key-value store for shared configuration and service discovery. Its service discovery protocol helps any new etcd member to discover all other members during the cluster bootstrap phase, using a shared discovery URL.

To make it easier to combine the NGINX Plus on-the-fly reconfiguration API with etcd, we’ve created a GitHub project called etcd-demo, which includes step-by-step instructions for setting up the demo deployment described here. Using tools like Docker, Docker Compose, Homebrew, and jq, you can spin up a Docker–based environment in which NGINX Plus load balances HTTP traffic to a couple of hello-world applications, with all components running in separate Docker containers.

How the Demo Works

First we spin up a separate Docker container for each of the following apps:

etcd – Performs service discovery.
Registrator – Registers services with Consul. Registrator monitors for containers being started and stopped and updates Consul when a container changes state.
hello – Simulates a backend server. This is another GitHub project from NGINX, Inc., an NGINX web server that serves an HTML page containing the web server’s hostname, IP address, and port.
A second instance of hello – Simulates another backend server.
NGINX Plus R8 – Load balances the other listed services.

The NGINX Plus container listens on the public port 80, and the built-in NGINX Plus dashboard on port 8080. The etcd container advertises a fixed IP address of 10.0.2.15 for intercontainer communication and listens on ports 2379, 2380, and 4001.

Registrator monitors Docker for new containers that are launched with exposed ports, and registers the associated service with etcd. By setting environment variables within the containers, we can be more explicit about how to register the services with etcd. For each hello (web server) container, we set the SERVICE_TAGS environment variable to production to identify the container as an upstream server for load balancing by NGINX Plus. When a container quits or is removed, Registrator removes its corresponding key from etcd automatically.

Finally, we use the script included in the demo, etcd_exec_watch.sh, to invoke an external handler, script.sh, every time there is an update to the list of registered services (that is, a change to etcd keys). The external handler uses environment variables set by etcd to determine which servers to add and remove from the NGINX Plus upstream group. If ETCD_WATCH_ACTION=set, the handler adds servers named by ETCD_WATCH_VALUE, and if ETCD_WATCH_ACTION=delete it removes servers that are not also present in etcd.

Summary

Using a script and handler like the ones in etcd-demo to dynamically reconfigure your upstream groups based on the services registered with etcd automates the process of upstream reconfiguration in NGINX Plus. This automation also frees you from having to figure out how to issue the API calls correctly and reduces the amount of time between the state change of a service in etcd and its addition or removal in the NGINX Plus upstream group.

Try out automated reconfiguration of NGINX Plus upstream groups using etcd for yourself: