\n
![](\"https:\/\/jirak.net\/wp\/wp-content\/uploads\/2016\/06\/dashboard-demoapp-upstream-2-servers-up.png\")
<\/p>\nUsing NGINX\u00a0Plus for Backend Upgrades with Zero Downtime, Part\u00a02\u00a0\u2013\u00a0Individual Servers
\n<\/p>\n
This is the second of three articles in our series about using NGINX Plus to upgrade backend servers with zero downtime. In the first article<\/a>, we describe the two NGINX Plus<\/a> features you can use for backend upgrades with zero downtime – the on-the-fly reconfiguration API<\/a> and application health checks<\/a> – and discuss the advantages of each method.<\/p>\n In this second article, we explore use cases around upgrading the software or hardware on an individual server, which is one of the most common reasons to take servers offline. We could just take the server offline with no preparation, but that kills all the current client connections, making for a bad user experience. What we want is to stop sending any new requests or connections to the server, while letting it finish off any outstanding work. Then we can safely take it offline without impacting clients. NGINX Plus provides a few methods for achieving this outcome.<\/p>\n For use cases around upgrading the version of an application on a group of upstream servers, see the third article, Using NGINX Plus for Backend Upgrades with Zero Downtime, Part 3 – Application Version<\/a>.<\/p>\n For the API examples we will be making the API calls from the NGINX Plus instance, so they will be sent to The base configuration for the use cases starts with two servers in a single We\u2019re configuring an application health check, which is a best practice for reducing the impact of backend server errors on the user experience and for improving monitoring. Here we configure the health check to succeed if the server returns the file healthcheck.html<\/strong> with an HTTP Though it’s not strictly necessary for basic health checks, we’re putting the server { location \/ { location @healthcheck { <\/code><\/p>\n We also configure a second virtual server that listens on port 8080 for requests to locations corresponding to the dynamic reconfiguration API (\/upstream_conf<\/strong>), the NGINX Plus status dashboard (\/status.html<\/strong>), and the NGINX Plus status API (\/status<\/strong>). Note that these location names are the conventional ones, but you can choose different names if you wish.<\/p>\n It is a best practice to secure all traffic to the reconfiguration and status APIs and the dashboard, which we do here by granting access only to users on internal IP addresses in the range 192.168.100.0 to 192.168.100.255. For stronger security, use client certificates, HTTP Basic authentication, or the Auth Request<\/a> module to integrate with external authorization systems like LDAP<\/a>.<\/p>\n location = \/ { location \/upstream_conf { location = \/status.html { location \/status { <\/code><\/p>\n With this configuration in place, the base command for the API commands in this article is <\/p>\n <\/code><\/p>\n To verify that the two servers in the demoapp<\/strong> upstream group (configured in the previous section) are active, we look at the Upstreams<\/strong> tab on the NGINX Plus live activity monitoring dashboard. The numbers in the Requests<\/strong> and Conns<\/strong> columns confirm that the servers are processing traffic:<\/p>\n Now we take server 172.16.210.82 offline for maintence. To see the ID number assigned to it, we send the base API command. The response tells us the server has <\/code><\/p>\n To mark the server as <\/code><\/p>\n Now the dashboard shows that the active connection count (Conns > A<\/strong>) for 172.16.210.82 is zero, so it is safe to take it offline for maintenance.<\/p>\n When maintenance is complete, we can bring the server back online by appending this string to the base command:<\/p>\n <\/code><\/p>\n Note that you can also use the editing interface on the dashboard’s Upstreams<\/strong> tab to change server state (mark a server as When we enable session persistence<\/a>, clients are directed to the same backend server for all requests during a session. NGINX Plus supports several session persistence methods with the <\/code><\/p>\n Session persistence is required for any application that keeps state information for users (such as a shopping cart), but it complicates upgrades because now it is not enough just to wait until there are no active connections to our server before taking it offline. There might be clients that aren’t sending requests right now but haven’t ended their session with the server. For the best user experience, we want to keep the active sessions open – the amount of time depends on the application – but don\u2019t want any new sessions to start. <\/p>\n Fortunately, the NGINX Plus <\/code><\/p>\n In this case, before taking the server offline we don’t only want the number of active connections to be zero, but also for all sessions to end. That translates to the server being idle for some amount of time, which depends on the application. We can periodically check the dashboard or use the status API to determine that the server is idle, but we can also automate the process of marking a server I\u2019ve created the following Python program called server-drain-down.py<\/strong><\/span> as an example. It takes the upstream group name and the IP address and port of the server as input, and marks the specified server with import requests if len(sys.argv) != 3: upstream=sys.argv[1] # The URL for the NGINX Plus status API # The time the server needs to be idle before being marked down, in seconds # The total elapsed time before marking the server down even if it isn't idle, sleepInterval = 1<\/p>\n client = requests.Session() # Create a session for making HTTP requests<\/p>\n ################################################################################ ################################################################################ startTime = int(time.time()) <\/code><\/p>\n Whether we use the program or verify manually that the server is idle, after it is marked Recall that we set up a health check with the <\/code><\/p>\n When we want to take a server offline, we simply rename the file to fail‑healthcheck.html<\/strong> and health checks fail. NGINX Plus stops sending any new requests to the server, but allows existing requests to complete (equivalent to the As previously mentioned, with health checks we can make use of the slow-start<\/a> feature if the server requires some warm-up time before it is ready to receive its full share of traffic. Here we modify the servers in the upstream group so that NGINX Plus ramps up traffic gradually during the 30 seconds after they come up:<\/p>\n <\/code><\/p>\n NGINX Plus provides operations and DevOps engineers with several options for managing software and hardware upgrades on individual servers while continuing to provide a good customer experience by avoiding downtime. <\/p>\n Check out the other two articles in this series:<\/p>\n Try NGINX Plus out for yourself and see how it makes upgrades easier and more efficient – start a 30‑day free trial<\/a> today or contact us<\/a> for a live demo.<\/p>\n The post Using NGINX Plus for Backend Upgrades with Zero Downtime, Part 2 – Individual Servers<\/a> appeared first on NGINX<\/a>.<\/p>\n Source: Using NGINX\u00a0Plus for Backend Upgrades with Zero Downtime, Part\u00a02\u00a0\u2013\u00a0Individual Servers<\/a><\/p>\n","protected":false},"excerpt":{"rendered":" Using NGINX\u00a0Plus for Backend Upgrades with Zero Downtime, Part\u00a02\u00a0\u2013\u00a0Individual Servers This is the second of three articles in our series about using NGINX Plus to upgrade backend servers with zero downtime. In the first article, we describe the two NGINX Plus features you can use for backend upgrades with zero downtime – the on-the-fly reconfiguration API and application health checks – and discuss the advantages of each method. In this second article, we explore use cases around upgrading the software or hardware on an individual server, which is one of the most common reasons to take servers offline. We could just take the server offline with no preparation, but that kills all the current client connections, making for a bad user experience. What we want is to stop sending any new requests or connections to the server, while letting it finish off any outstanding work. Then [ more… ]<\/a><\/p>\n<\/div>","protected":false},"author":1,"featured_media":8193,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[169],"tags":[652],"class_list":["post-8192","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news","tag-nginx"],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/posts\/8192","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/comments?post=8192"}],"version-history":[{"count":1,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/posts\/8192\/revisions"}],"predecessor-version":[{"id":8194,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/posts\/8192\/revisions\/8194"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/media\/8193"}],"wp:attachment":[{"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/media?parent=8192"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/categories?post=8192"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/tags?post=8192"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
Base Configuration for the Use Cases<\/h2>\n
localhost<\/code>. <\/p>\nupstream<\/code><\/a> configuration block called demoapp<\/strong>. In the first server<\/code><\/a> configuration block, we configure a virtual server listening on port 80 that load balances all requests to the demoapp<\/strong> upstream group. <\/p>\n2xx<\/code> or 3xx<\/code> response code (the default success criterion for health checks). <\/p>\nhealth_check<\/code><\/a> directive in its own location<\/code><\/a> block. This is a good practice as it allows us to configure different settings, such as timeouts and headers, for health checks versus regular traffic. For a use case where a separate location for the health_check<\/code> directive is required, see Doing a Dark Launch<\/a>.<\/p>\n# In the HTTP context
\nupstream demoapp {
\n zone demoapp 64k;
\n server 172.16.210.81:80;
\n server 172.16.210.82:80;
\n}<\/p>\n
\n listen 80;
\n status_zone demoapp;<\/p>\n
\n proxy_pass http:\/\/demoapp;
\n }<\/p>\n
\n internal;
\n proxy_pass http:\/\/demoapp;
\n health_check uri=\/healthcheck.html;
\n }
\n}<\/pre>\n# In the HTTP context
\nserver {
\n listen 8080;
\n allow 192.168.100.0\/24;
\n deny all;<\/p>\n
\n return 301 \/status.html;
\n }<\/p>\n
\n upstream_conf;
\n }<\/p>\n
\n root \/usr\/share\/nginx\/html;
\n }<\/p>\n
\n status;
\n }
\n}<\/pre>\nhttp:\/\/localhost:8080\/upstream_conf?upstream=demoapp<\/pre>\nUsing the API to Upgrade an Individual Server<\/h2>\n
![\"Screenshot](\"https:\/\/assets.wp.nginx.com\/wp-content\/uploads\/2016\/06\/dashboard-demoapp-upstream-2-servers-up.png\")
<\/p>\nid=1<\/code>:<\/p>\nhttp:\/\/localhost:8080\/upstream_conf?upstream=demoapp
\nserver 172.16.210.81:80; # id=0
\nserver 172.16.211.82:80; # id=1<\/pre>\ndown<\/code>, we append this string to the base command:<\/p>\n...&id=1&down=<\/pre>\n![\"Screenshot](\"https:\/\/assets.wp.nginx.com\/wp-content\/uploads\/2016\/06\/dashboard-demoapp-upstream-1-server-down.png\")
<\/p>\n...&id=1&up=<\/pre>\nup<\/code>, down<\/code>, or drain<\/code>) rather than sending commands to the API. For instructions, see our blog<\/a>.<\/p>\nUsing the API to Upgrade an Individual Server with Session Persistence Configured<\/h2>\n
sticky<\/code><\/a> directive to the upstream<\/code> block; here we use the sticky cookie<\/em> method: <\/p>\n# In the HTTP context
\nupstream demoapp {
\n zone demoapp 64k;
\n server 172.16.210.81:80;
\n server 172.16.210.82:80;
\n sticky cookie srv_id expires=1h domain=.example.com path=\/<\/strong>;
\n}<\/pre>\ndrain<\/code> state does exactly this. Session draining adds one more step to the process outlined in the previous section. Instead of immediately marking the server as down<\/code>, we mark it as drain<\/code> by appending the following string to the base command:<\/p>\n...&id=1&drain=<\/pre>\ndrain<\/code> and verifying it is idle before marking it down<\/code>. <\/p>\ndrain<\/code>. It then marks the server down<\/code> after either it has been idle for 60 seconds, or 5 minutes have elapsed since session draining began (even if the server isn\u2019t idle). The program uses the status API to get the timestamp of the last request sent to the server and the number of active connections. It uses the configuration API to mark the server with drain<\/code> and then down<\/code>.<\/p>\n#!\/usr\/bin\/env python
\n################################################################################
\n# Copyright (C) 2016 NGINX, Inc.
\n#
\n# This program is provided for demonstration purposes only and is not covered
\n# by your NGINX Plus support agreement.
\n#
\n# It is a proof of concept for automating the process of taking a server offline
\n# when it is configured for session persistence.
\n#
\n# This program takes two command line-arguments:
\n# - upstream group name
\n# - server IP address and port
\n#
\n# It uses the NGINX Plus status API to get the server's ID and the
\n# upstream_conf API to set the state of the server to 'drain'. It then loops,
\n# waiting to mark the server down until either it has been idle for a
\n# configured period of time or a configured maximum time has elapsed (even if
\n# the server is not idle).
\n################################################################################<\/p>\n
\nimport json
\nimport sys
\nimport time<\/p>\n
\n print \"Error: Wrong number of arguments. Usage is:\"
\n print \" server-drain-down.py \"
\n sys.exit(1)<\/p>\n
\nserver=sys.argv[2]<\/p>\n
\nstatusURL = 'http:\/\/localhost:8080\/status'
\n# The URL for the NGINX Plus reconfiguration API
\nconfURL = 'http:\/\/localhost:8080\/upstream_conf' <\/p>\n
\nmaxIdleTime = 60 <\/p>\n
\n# in seconds
\nmaxTime = 300 <\/p>\n
\n# Function sendRequest
\n#
\n# Send an HTTP request. Status 200 is expected for all requests.
\n################################################################################
\ndef sendRequest(url):
\n try:
\n response = client.get(url) # Make an NGINX Plus status API call
\n if response.status_code == 200:
\n return response
\n else:
\n print (\"Error: Response code %d\") %(response.status_code)
\n sys.exit(1)
\n except requests.exceptions.ConnectionError:
\n print \"Error: Unable to connect to \" + url
\n sys.exit(1)<\/p>\n
\n# Main
\n################################################################################
\nurl = statusURL + '\/upstreams\/' + upstream + '\/peers'
\nresponse = sendRequest(url)
\nnginxstats = json.loads(response.content) # Convert JSON to dict
\nid = \"\"
\nstate = \"\"
\nserverFound = False
\nfor stats in nginxstats:
\n if stats['server'] == server:
\n serverFound = True
\n id = stats['id']
\n state = stats['state']
\n # The last time a request was sent to this server, converted to seconds
\n lastSelected = stats['selected'] \/ 1000
\n break
\nif not serverFound:
\n print(\"Server %s not found in Upstream Group %s\") %(server, upstream)
\n sys.exit(1)
\nif state == 'down':
\n print \"The server is already marked as down\"
\n sys.exit(0)
\nelif state == 'unhealthy' or state == 'unavailable':
\n # The server is not healthy so it won't be receiving requests and can be
\n # marked down
\n url = confURL + '?upstream=' + upstream + '&id=' + str(id) + '&down='
\n response = sendRequest(url)
\n print \"The server was unhealthy or unavailable and has been marked down\"
\n sys.exit(0)
\nif state == 'up':
\n print \"Set server to drain\"
\n url = confURL + '?upstream=' + upstream + '&id=' + str(id) + '&drain='
\n response = sendRequest(url)<\/p>\n
\nwhile True: # Loop forever
\n now = int(time.time())
\n totalTime = now - startTime
\n if totalTime >= maxTime:
\n print \"Max time has expired. Mark server as down\"
\n url = confURL + '?upstream=' + upstream + '&id=' + str(id) + '&down='
\n response = sendRequest(url)
\n break
\n idleTime = now - lastSelected
\n if idleTime >= maxIdleTime:
\n if nginxstats['active'] == 0:
\n print \"Idle time has expired. Mark server as down\"
\n url = confURL + '?upstream=' + upstream + '&id=' + str(id) + '&down='
\n response = sendRequest(url)
\n break
\n else:
\n print(\"Idle time has expired but there are still active \"
\n \"connections. %d max seconds\") %(totalTime)
\n else:
\n print(\"Server idle for %d seconds. %d max seconds\") %(idleTime, totalTime)
\n url = statusURL + '\/upstreams\/' + upstream + '\/peers\/' + str(id)
\n response = sendRequest(url)
\n nginxstats = json.loads(response.content)
\n lastSelected = nginxstats['selected'] \/ 1000
\n time.sleep(sleepInterval)<\/pre>\ndown<\/code> we proceed as in the previous section: take the server offline, do the upgrade, and mark it as up<\/code> to return it to service.<\/p>\nUsing Health Checks to Upgrade an Individual Server<\/h2>\n
health_check<\/code> directive in the first server<\/code> block we configured in Base Configuration for the Use Cases<\/a>. Now we use it to control server state. The health check succeeds if the server returns the file healthcheck.html<\/strong> with an HTTP 2xx<\/code> or 3xx<\/code> response code. <\/p>\n# In the first server block
\nlocation @healthcheck {
\n internal;
\n proxy_pass http:\/\/demoapp;
\n health_check uri=\/healthcheck.html;
\n}<\/pre>\ndown<\/code> state set with the API). After making the health check fail, we use the dashboard or the status API to monitor the server as we did when using the API<\/a> to mark the server down<\/code>. We wait for connections to go to zero before taking the server offline to do the upgrade. When the server is ready to return to service, we rename the file back to healthcheck.html<\/strong> and health checks once again succeed. <\/p>\n# In the HTTP context
\nupstream demoapp {
\n zone demoapp 64K;
\n server 172.16.210.81:80 slow_start=30s<\/strong>;
\n server 172.16.210.82:80 slow_start=30s<\/strong>;
\n sticky cookie srv_id expires=1h domain=.example.com path=\/;
\n}<\/pre>\nConclusion<\/h2>\n
\n