Service Discovery for NGINX Plus with ZooKeeper
Our previous blog posts about service discovery with Consul and etcd discussed the importance of service discovery in distributed systems, including service-oriented and microservices architectures. In such systems it’s common to assign network locations to service instances dynamically, because their locations can change over time due to autoscaling, failures, or upgrades. To track the current network location of a service instance, clients need a sophisticated service discovery mechanism.
In this blog, we’ll explain how to dynamically add or remove load-balanced servers that are registered with Apache ZooKeeperTM, a tool used for service discovery, in combination with NGINX Plus’ on-the-fly reconfiguration API. With this solution you can automate service discovery and change the set of load-balanced servers without having to reload the NGINX Plus configuration.
Apache ZooKeeper is a software project of the Apache Software Foundation that provides an open source distributed configuration service, synchronization service, and naming registry for large distributed systems. ZooKeeper nodes – or Znodes, as they are popularly called – store their data in a hierarchical name space, much like a file system or a tree data structure. All clients can read from and write to the nodes, making ZooKeeper a shared configuration service.
To make it easier to combine the on-the-fly reconfiguration API with ZooKeeper, we’ve created a sample zookeeper-demo, with step-by-step instructions for creating the configuration described in this blog post. In this post, we will walk you through the proof of concept. Using tools like Docker, Docker Compose, and Homebrew, you can spin up a Docker-based environment in which NGINX Plus load balances HTTP traffic to a couple of hello-world applications, with all components running in separate Docker containers.
How the Demo Works
First we spin up a separate Docker container for each of the following apps:
- ZooKeeper – Performs service discovery.
- Registrator – Registers services with ZooKeeper. Registrator monitors the starting and stopping of containers and updates ZooKeeper about the state changes.
- hello – Simulates a backend server. This is another project from NGINX, Inc., an NGINX web server that serves an HTML page containing the web server’s hostname, IP address, and port.
- A second instance of the hello app – Simulates another backend server.
- NGINX Plus – Load balances the above services.
The NGINX Plus container listens on the public port 80, and the built-in NGINX Plus dashboard on port 8080. The ZooKeeper container listens on ports 2181, 2888, and 3888.
Registrator monitors Docker for new containers that are launched with exposed ports, and registers the associated services with ZooKeeper. By setting environment variables within the containers, we can be more explicit about how to register the services with ZooKeeper. For each hello-world container, we set the
SERVICE_TAGS environment variable to
production to identify the container as an upstream server for NGINX Plus to load balance. When a container quits or is removed, Registrator removes its corresponding Znode entry from ZooKeeper automatically.
Finally, with a tool written in Ruby and included in the sample demo,
zk-tool, we use ZooKeeper watches to trigger an external handler (script.sh) every time there is a change in the list of registered service containers. This
bash script gets the list of all current NGINX Plus upstream servers, uses
zk-tool to loop through all the containers registered with ZooKeeper that are tagged
production, and uses the on-the-fly reconfiguration (
upstream_conf) API to add them to the NGINX Plus upstream group if they’re not listed already. It also then removes from the NGINX Plus upstream group any
production-tagged containers that are not registered with ZooKeeper.
Using a script like the one in our demo to dynamically reconfigure upstream groups based on the services registered with ZooKeeper automates upstream reconfiguration in NGINX Plus. This automation also frees you from having to figure out how to issue the API calls correctly and reduces the amount of time between the state change of a service in ZooKeeper and its addition or removal within the upstream group in NGINX Plus.
Check out our other blogs about server discovery for NGINX Plus:
And try out automated reconfiguration of NGINX Plus upstream groups using ZooKeeper for yourself:
- Download zookeeper-demo from our GitHub repository
- Start a free 30-day trial of NGINX Plus today or contact us for a live demo