\n
![](\"https:\/\/jirak.net\/wp\/wp-content\/uploads\/2017\/12\/Autoscaling1-1024x512.png\")
<\/p>\nAutoscaling and Orchestration with NGINX Plus and Chef
\n<\/p>\n
There are many solutions for handling autoscaling in cloud environments, but they\u2019re usually dependent on the specific infrastructure of a given cloud provider. Leveraging the flexibility of NGINX Plus with the functionality of Chef<\/a>, we can build an autoscaling system that can be used on most cloud providers.<\/p>\n Chef has a tool, knife<\/a>, which you can use at the command line to act on objects such as cookbooks, nodes, data bags, and more. Knife plugins help you extend knife. So we use knife plugins to help abstract out functionality specific to one specific cloud, enabling knife commands to work the same way across clouds. <\/p>\n For this setup, we\u2019ll be leveraging our NGINX Chef cookbook. The installation and a basic overview of this cookbook can be found here<\/a>. Also, we\u2019ll be utilizing Hosted Chef, to make switching between clouds more straightforward. This setup is currently configured to work with AWS<\/a>, Azure<\/a>, and OpenStack<\/a>. It\u2019s possible to extend it to cover all of the Knife cloud plug-ins<\/a>, but they haven\u2019t been tested.<\/p>\n This configuration heavily relies on role membership to look up information about the different nodes that are part of the cluster. You\u2019ll need to have three basic roles; one for the NGINXPlus servers, one for the autoscaler server, and one for the upstream application servers. The autoscaler server is a node that monitors the NGINX Plus status page, and will make the API calls to scale up or down servers based on NGINX Plus statistics. Here are the example roles:<\/p>\n <\/code><\/p>\n <\/code><\/p>\n <\/code><\/p>\n Here\u2019s a quick breakdown of the different attributes leveraged in the roles:<\/p>\n You\u2019ll also need to configure your knife.rb<\/strong> file to have the necessary credentials to access the different cloud providers you would like to leverage. Here\u2019s an example knife.rb<\/strong> with the different supported cloud provider details:<\/p>\n <\/code><\/p>\n Now we can check out the few scripts that make the autoscaling happen. First is the script that runs on the NGINX Plus nodes to watch for changes in the nodes that are online:<\/p>\n for i in $ADD_NODE; do rm $NGINX_NODES $CONFIG_NODES $DIFF_OUT<\/pre>\n <\/code><\/p>\n The script is a little harder to read because it\u2019s still in the Chef template version. It\u2019s comparing the running config from the extended status page to the upstream config file managed by Chef. Here\u2019s the logic used to generate the upstream configs:<\/p>\n <\/code><\/p>\n Finally, we can take a look at the actual script that will handle the autoscaling:<\/p>\n Chef::Config.from_file(File.expand_path(\"~\/.chef\/knife.rb\")) def get_nginx_active_servers(nginx_status_data, nginx_upstream) def get_nginx_server_conns(nginx_status_data, nginx_server_zone) def add_backend_node(create_args) unless firstnum.nil? && lastnum.nil? def del_backend_node(nginx_status_data, nginx_node, active_nodes, cloud_provider, nginx_upstream) last_conns_count = -1<\/p>\n while true active_nodes = get_nginx_active_servers(nginx_status_data, nginx_upstream) conns_per_server = current_conns \/ server_count.to_f<\/p>\n puts \"Current connections = #{current_conns}\" if server_count max_conns end<\/p>\n last_conns_count = current_conns <\/code><\/p>\n The primary roles of this script are to monitor the NGINX Plus status page and to add and remove nodes based on statistics to and from the NGINX Plus node. In its current state, the script is making decisions based on the number of active connections, divided by the amount of active servers in the load-balanced pool. You can easily modify this to use any of the other statistics available from the NGINX Plus status page.<\/p>\n First, we\u2019re going to start an autoscaler instance. We\u2019ll use the Waiting for EC2 to create the instance...... Waiting for sshd access to become available SSH Target Address: ec2-35-164-35-19.us-west-2.compute.amazonaws.com() <\/code><\/p>\n Now let\u2019s take a look at the script that will actually handle the autoscaling that runs on this node, \/usr\/bin\/autoscale_nginx.rb<\/strong>:<\/p>\n Chef::Config.from_file(File.expand_path(\"~\/.chef\/knife.rb\")) def get_nginx_active_servers(nginx_status_data, nginx_upstream) def get_nginx_server_conns(nginx_status_data, nginx_server_zone) def add_backend_node(create_args) def del_backend_node(nginx_status_data, nginx_node, active_nodes, cloud_provider, nginx_upstream) last_conns_count = -1<\/p>\n while true active_nodes = get_nginx_active_servers(nginx_status_data, nginx_upstream) conns_per_server = current_conns \/ server_count.to_f<\/p>\n puts \"Current connections = #{current_conns}\" if server_count max_conns end<\/p>\n last_conns_count = current_conns root@ip-172-31-27-162:~#<\/pre>\n <\/code><\/p>\n You can see that the Now we can start our NGINX Plus server:<\/p>\n Waiting for EC2 to create the instance....... Waiting for sshd access to become available SSH Target Address: ec2-35-165-171-46.us-west-2.compute.amazonaws.com() <\/code><\/p>\n Now we can check our autoscaler script on the autoscaler instancer to make sure the script was updated with the new node’s IP address:<\/p>\n <\/code><\/p>\n You should now be able to hit the status page for your NGINX Plus node, and it should look like below:<\/p>\n You can also hit the NGINX Plus server on port 80, and you\u2019ll see a Before we fire up the autoscaler script and get some application nodes started, let\u2019s take a look at the script that will add these new nodes to the running NGINX config, \/tmp\/api_update.sh<\/strong>:<\/p>\n for i in $ADD_NODE; do rm $NGINX_NODES $CONFIG_NODES $DIFF_OUT <\/code><\/p>\n This script will be called every time Chef runs, and it\u2019ll compare the existing running config to the upstream config file defined for the autoscaling group. As you can see from the recipe snippet below, Chef manages the config file, but doesn\u2019t reload NGINX when it is updated. Instead, it calls the <\/code><\/p>\n Now we can start the autoscaler script and get some application servers brought online. Because we need to utilize the Ruby binary shipped with the Chef client, we have to fully qualify the path to the Ruby binary for the script to run properly:<\/p>\n Waiting for EC2 to create the instance... <\/code><\/p>\n Now that you have one application node up, you can go back to your NGINX Plus node. You should see this demo page now, instead of the And if you go back to the status page, you now have an upstream defined:<\/p>\n Next, we can use a tool like And on the autoscaler node, you can see the script catch the increase in connections and start a new instance:<\/p>\n [terminal]Current connections = 0 Waiting for EC2 to create the instance...... <\/code><\/p>\n You should now be able to see two upstream nodes in your dashboard. Now the script will stay at this point, because it's configured to scale up when the nodes have more than 20 active connections on average. If you go back and refresh your browser pointed at port 80 of your NGINX server, you should see the data change as it switches between the different backend nodes. If we stop the traffic from being generated, you can see the script take one of the nodes offline, as it is configured to always have a minimum of one server running.<\/p>\n <\/code><\/p>\n This is a rather basic script at this point, but it should provide a starting point for you to build a customized autoscaling solution that fits your environment. And if you ever want to migrate to a different cloud provider, it\u2019s as simple as changing one attribute in the Chef configuration.<\/p>\n The post Autoscaling and Orchestration with NGINX Plus and Chef<\/a> appeared first on NGINX<\/a>.<\/p>\n Source: Autoscaling and Orchestration with NGINX Plus and Chef<\/a><\/p>\n","protected":false},"excerpt":{"rendered":" Autoscaling and Orchestration with NGINX Plus and Chef Introduction There are many solutions for handling autoscaling in cloud environments, but they\u2019re usually dependent on the specific infrastructure of a given cloud provider. Leveraging the flexibility of NGINX Plus with the functionality of Chef, we can build an autoscaling system that can be used on most cloud providers. Chef has a tool, knife, which you can use at the command line to act on objects such as cookbooks, nodes, data bags, and more. Knife plugins help you extend knife. So we use knife plugins to help abstract out functionality specific to one specific cloud, enabling knife commands to work the same way across clouds. Requirements For this setup, we\u2019ll be leveraging our NGINX Chef cookbook. The installation and a basic overview of this cookbook can be found here. Also, we\u2019ll be utilizing [ more… ]<\/a><\/p>\n<\/div>","protected":false},"author":1,"featured_media":21394,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[169],"tags":[652],"class_list":["post-21393","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news","tag-nginx"],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/posts\/21393","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/comments?post=21393"}],"version-history":[{"count":1,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/posts\/21393\/revisions"}],"predecessor-version":[{"id":21395,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/posts\/21393\/revisions\/21395"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/media\/21394"}],"wp:attachment":[{"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/media?parent=21393"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/categories?post=21393"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/tags?post=21393"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Requirements<\/h2>\n
Basic Setup<\/h2>\n
NGINX Plus server role:<\/h3>\n
name \"nginx_plus_autoscale\"
\ndescription \"An example role to install nginx plus\"
\nrun_list \"recipe[nginx]\",\"recipe[nginx::autoscale]\"
\ndefault_attributes \"nginx\" => { \"install_source\" => \"plus\",
\n \"plus_status_enable\" => true,
\n \"enable_upstream_conf\" => true,
\n \"plus_status_allowed_ips\" => ['104.245.19.144', '172.31.0.0\/16', '127.0.0.1'],
\n \"server_name\" => \"test.local\",
\n \"upstream\" => \"test\",
\n \"nginx_repo_key\" => \"-----BEGIN PRIVATE KEY-----nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCbYwum24BwEYDf4P4x0\/KZjkKN7\/EE\/gg0qAU3ebG5kY8gWb8NpQ2itj\/DfmwPAEnvI6In86c6YFokAZxeo6HbkKkeQKBgQDGQEHp2lCON9FLgRNMjtcp4S2VYjxdAMVinDLkIgVb9qgh6BvTDt5hRY\/Vhcx8xV70+BCnoMSzbvLWhZbpSrdmD9nOj1KkPcWn4ArSv6prlYItUwWbNtFLw\/E=n-----END PRIVATE KEY-----\",
\n \"nginx_repo_crt\" => \"-----BEGIN CERTIFICATE-----nMIIDrDCCApSgAwIBAgICBs8wDQYJKoZIhvcNAQEFBQAwXjELMAkGA1UI2pLoSbonYiEvivb4Cg7POn+cQBwurcYUH\/jB9zLPPSwlqcUiG2hScuEeaBiEoK\/ixHIRuMV9nyp3xTi3b0ZKvOFjEZpBHB8WIdQVneTNRvaFLbiwznhiAe7D4uMaAEYqF96GTgX2XnbovinLlYPfdi7BhlXTI9u78+tqbo0YVsSBiDV49hcIA=n-----END CERTIFICATE-----\" }<\/pre>\nUpstream application server role:<\/h3>\n
name \"test-upstream\"
\ndescription \"An example role to install nginx plus hello demo\"
\nrun_list \"recipe[nginx::hello-demo]\"
\ndefault_attributes \"nginx\" => { \"application_port\" => \"80\"}<\/pre>\nAutoscaler server role:<\/h3>\n
name \"autoscaler\"
\ndescription \"An example role to install autoscaler script\"
\nrun_list \"recipe[nginx::autoscale_script]\"
\ndefault_attributes \"nginx\" => { \"server_name\" => \"test.local\",
\n\t\t\t\t\"upstream\" => \"test\",
\n\t\t\t\t\"cloud_provider\" => \"ec2\" }<\/pre>\n\n
install_source<\/code> – Tells the NGINX cookbook to install NGINX Plus instead of open source<\/li>\nplus_status_enable<\/code> – Enables the NGINX Plus status page<\/li>\nenable_upstream_conf<\/code> – Enables the dynamic reconfiguration API<\/li>\nplus_status_allowed_ips<\/code> – List of IPs or IP ranges that are allowed to access the status page and reconfiguration API<\/li>\nserver_name<\/code> – Defines a server directive in the NGINX configuration <\/li>\nupstream<\/code> – Defines an upstream group to be used with the server_name<\/code> configuration above<\/li>\nnginx_repo_key<\/code> – Defines a certificate key to be used to access the NGINX Plus repositories<\/li>\nnginx_repo_crt<\/code> – Defines a certificate to be used to access the NGINX Plus repositories<\/li>\napplication_port<\/code> – The port that the upstream application servers will listen on<\/li>\ncloud_provider<\/code> – Defines the cloud provider(ec2\/azure\/google\/openstack) to be used for the autoscale_nginx<\/code> script\n<\/ul>\ncurrent_dir = File.dirname(__FILE__)
\nlog_level :info
\nlog_location STDOUT
\nnode_name \"damiancurry\"
\nclient_key \"#{current_dir}\/damiancurry.pem\"
\nchef_server_url \"https:\/\/api.chef.io\/organizations\/nginx\"
\ncookbook_path [\"#{current_dir}\/..\/cookbooks\"]
\n#AWS variables
\nknife[:aws_access_key_id] =
\nknife[:aws_secret_access_key] =
\n#azure variables
\nknife[:azure_tenant_id] =
\nknife[:azure_subscription_id] =
\nknife[:azure_client_id] =
\nknife[:azure_client_secret] =
\n#openstack variables
\nknife[:openstack_auth_url] =
\nknife[:openstack_username] =
\nknife[:openstack_password] =
\nknife[:openstack_tenant] =
\nknife[:openstack_image] =
\nknife[:openstack_ssh_key_id] = \"demo_key\"<\/pre>\n#!\/bin\/bash
\nNGINX_NODES=\"$(mktemp)\"
\n\/usr\/bin\/curl -s \"http:\/\/localhost:\/upstream_conf?upstream=\"| \/usr\/bin\/awk '{print $2}' | \/bin\/sed -r 's\/;\/\/g' | \/usr\/bin\/sort > $NGINX_NODES
\nCONFIG_NODES=\"$(mktemp)\"
\n\/bin\/grep -E '[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}' \/etc\/nginx\/conf.d\/-upstream.conf | \/usr\/bin\/awk '{print $2}' | \/bin\/sed -r 's\/;\/\/g' | \/usr\/bin\/sort > $CONFIG_NODES
\nDIFF_OUT=\"$(mktemp)\"
\n\/usr\/bin\/diff $CONFIG_NODES $NGINX_NODES > $DIFF_OUT
\nADD_NODE=`\/usr\/bin\/diff ${CONFIG_NODES} ${NGINX_NODES} | \/bin\/grep \"\" | \/usr\/bin\/awk '{print $2}'`<\/p>\n
\n echo \"adding node ${i}\";
\n \/usr\/bin\/curl -s \"http:\/\/localhost:\/upstream_conf?add=&upstream=&server=${i}&max_fails=0\"
\ndone
\nfor i in $DEL_NODE; do
\n echo \"removing node ${i}\";
\n #NODE_ID=`\/usr\/bin\/curl -s \"http:\/\/localhost:\/upstream_conf?upstream=\" | \/bin\/grep ${i} | \/usr\/bin\/awk '{print $4}' | \/bin\/sed -r 's\/id=\/\/g'`
\n NODE_ID=`\/usr\/bin\/curl -s \"http:\/\/localhost:\/upstream_conf?upstream=\" | \/bin\/grep ${i} | \/bin\/grep -oP 'id=Kd+'`
\n NODE_COUNT=`\/usr\/bin\/curl -s \"http:\/\/localhost:\/upstream_conf?upstream=\" | \/bin\/grep -n ${i} | \/bin\/grep -oP 'd+:server' | \/bin\/sed -r 's\/:server\/\/g'`
\n JSON_NODE_NUM=$(expr $NODE_COUNT - 1)
\n NODE_CONNS=`\/usr\/bin\/curl -s \"http:\/\/localhost:\/status\" | \/usr\/bin\/jq \".upstreams..peers[${JSON_NODE_NUM}].active\"`
\n NODE_STATE=`\/usr\/bin\/curl -s \"http:\/\/localhost:\/status\" | \/usr\/bin\/jq \".upstreams..peers[${JSON_NODE_NUM}].state\"`
\n if [[ ${NODE_STATE} == '\"up\"' ]] && [[ ${NODE_CONNS} == 0 ]]; then
\n\techo \"nodes is up with no active connections, removing ${i}\"
\n\t\/usr\/bin\/curl -s \"http:\/\/localhost:\/upstream_conf?remove=&upstream=&id=${NODE_ID}\"
\n elif [[ ${NODE_STATE} == '\"draining\"' ]] && [[ ${NODE_CONNS} == 0 ]]; then
\n echo \"nodes is draining with no active connections, removing ${i}\"
\n \/usr\/bin\/curl -s \"http:\/\/localhost:\/upstream_conf?remove=&upstream=&id=${NODE_ID}\"
\n elif [[ ${NODE_STATE} == '\"down\"' ]]; then
\n\techo \"node state is down, removing ${i}\":
\n\t\/usr\/bin\/curl -s \"http:\/\/localhost:\/upstream_conf?remove=&upstream=&id=${NODE_ID}\"
\n elif [[ ${NODE_STATE} == '\"unhealthy\"' ]]; then
\n\techo \"node state is down, removing ${i}\":
\n\t\/usr\/bin\/curl -s \"http:\/\/localhost:\/upstream_conf?remove=&upstream=&id=${NODE_ID}\"
\n elif [[ ${NODE_STATE} == '\"up\"' ]] && [[ ${NODE_CONNS} != 0 ]]; then
\n\techo \"node has active connections, draining connections on ${i}\"
\n fi
\ndone<\/p>\nupstream_node_ips = []
\nupstream_role = (node[:nginx][:upstream]).to_s
\nsearch(:node, \"role:#{node[:nginx][:upstream]}-upstream\") do |nodes|
\n host_ip = nodes['ipaddress']
\n unless host_ip.to_s.strip.empty?
\n host_port = nodes['nginx']['application_port']
\n upstream_node_ips << "#{host_ip}:#{host_port}" # if value.has_key?("broadcast")\n end\nend\n\ntemplate "\/etc\/nginx\/conf.d\/#{node[:nginx][:upstream]}-upstream.conf" do\n source 'upstreams.conf.erb'\n owner 'root'\n group node['root_group']\n mode 0644\n variables(\n hosts: upstream_node_ips\n )\n # notifies :reload, 'service[nginx]', :delayed\n notifies :run, 'execute[run_api_update_script]', :delayed\nend[\/config]\n\nYou can see that we\u2019re using the Chef search functionality to find nodes that are currently assigned to the upstream role you defined for this application. It then extracts the IP and application port for the node, and passes it to the template as an array. Here\u2019s the templated version of the upstream configuration:\n\n[config]upstream {
\n zone 64k;
\n
\n server ;
\n
\n }<\/pre>\nrequire 'chef\/api_client'
\nrequire 'chef\/config'
\nrequire 'chef\/knife'
\nrequire 'chef\/node'
\nrequire 'chef\/search\/query'
\nrequire 'net\/http'
\nrequire 'json'
\nclass MyCLI
\n include Mixlib::CLI
\nend<\/p>\n
\nnginx_node = \"\"
\ncloud_provider = \"\"
\nnginx_upstream = \"\"
\nnginx_server_zone = \"\"
\nif cloud_provider == \"ec2\"
\n create_args = [\"#{cloud_provider}\", 'server', 'create', '-r', \"role[#{nginx_upstream}-upstream]\", '-S', 'chef-demo', '-I', 'ami-93d80ff3', '--region', 'us-west-2', '-f', 'm1.medium', '-g', 'chef-demo', '--ssh-user', 'ubuntu', '-i', '~\/.ssh\/chef-demo.pem']
\nelsif cloud_provider == \"openstack\"
\n create_args = [\"#{cloud_provider}\", 'server', 'create', '-i', '~\/.ssh\/demo_key.pem', '--ssh-user', 'ubuntu', '-f', 'demo_flavor', '--openstack-private-network', '-Z', 'nova', '-r', \"role[#{nginx_upstream}-upstream]\"]
\nelse
\n puts \"Please specify a valid cloud provider\"
\n exit
\nend
\nsleep_interval_in_seconds = 10
\nmin_server_count = 1
\nmax_server_count = 10
\nmin_conns = 10
\nmax_conns = 20
\nnginx_status_url = \"http:\/\/#{nginx_node}:8080\/status\"<\/p>\n
\n active_nodes = Array.new
\n peers = nginx_status_data[\"upstreams\"][\"#{nginx_upstream}\"][\"peers\"]
\n peers.each do |node|
\n if node[\"state\"] == \"up\"
\n active_nodes.push node[\"server\"]
\n end
\n end
\n return active_nodes
\nend<\/p>\n
\n return nginx_status_data[\"server_zones\"][\"#{nginx_server_zone}\"][\"processing\"]
\nend<\/p>\n
\n #search for existing hostnames to pick a new one
\n query = Chef::Search::Query.new
\n #nodes = query.search('node', 'role:#{nginx_upstream}-upstream').first rescue []
\n nodes = query.search('node', 'role:-upstream').first rescue []
\n hosts = Array.new
\n used_num = Array.new
\n nodes.each do |node|
\n node_name = node.name
\n hosts.push node_name
\n num = node_name.scan(\/d+\/)
\n used_num.push num
\n end
\n used_num.sort!
\n fixed1 = used_num.flatten.collect do |num| num.to_i end
\n fixed_num = fixed1.sort!
\n firstnum = fixed_num.first
\n lastnum = fixed_num.last
\n firsthost = hosts.sort[0].to_i
\n lasthost = hosts.sort[-1].to_i<\/p>\n
\n total = (1..lastnum).to_a
\n missingnum = total-fixed_num
\n end
\n newhostname = \"\"
\n if missingnum.nil?
\n puts \"No existing hosts\"
\n fixnum = \"1\"
\n newnum = fixnum.to_i
\n newhostname = \"-app-#{newnum}\"
\n elsif missingnum.any?
\n puts \"Missing numbers are #{missingnum}\"
\n newnum = missingnum.first
\n newhostname = \"-app-#{newnum}\"
\n else
\n newnum = lastnum + 1
\n puts \"new number is n\"
\n newhostname = \"-app-#{newnum}\"
\n end
\n new_create_args = create_args + ['--node-name', newhostname]
\n knife = Chef::Knife.new
\n knife.options=MyCLI.options
\n Chef::Knife.run(new_create_args, MyCLI.options)
\n #sleep to wait for chef run
\n 1.upto(10) do |n|
\n puts \".\"
\n sleep 1 # second
\n end
\nend<\/p>\n
\n #lookup hostnames\/ips and pick a backend at random
\n query = Chef::Search::Query.new
\n #nodes = query.search('node', 'role:#{nginx_upstream}-upstream').first rescue []
\n nodes = query.search('node', 'role:-upstream').first rescue []
\n hosts = Array.new
\n nodes.each do |node|
\n node_name = node.name
\n node_ip = node['ipaddress']
\n if active_nodes.any? { |val| \/#{node_ip}\/ =~ val }
\n hosts.push \"#{node_name}:#{node_ip}\"
\n end
\n end
\n del_node = hosts.sample
\n node_name = del_node.rpartition(\":\").first
\n node_ip = del_node.rpartition(\":\").last
\n puts \"Removing #{node_name}\"
\n nginx_url = \"http:\/\/#{nginx_node}:8080\/upstream_conf?upstream=#{nginx_upstream}\"
\n response = Net::HTTP.get(URI(nginx_url))
\n node_id = response.lines.grep(\/#{node_ip}\/).first.split('id=').last.chomp
\n drain_url = \"http:\/\/#{nginx_node}:8080\/upstream_conf?upstream=#{nginx_upstream}&id=#{node_id}&drain=1\"
\n Net::HTTP.get(URI(drain_url))
\n sleep(5)
\n knife = Chef::Knife.new
\n knife.options=MyCLI.options
\n #delete_args = [\"#{cloud_provider}\", 'server', 'delete', \"#{node_name}\", '--purge', '-y']
\n #Chef::Knife.run(delete_args, MyCLI.options)
\n delete_args = \"#{cloud_provider} server delete -N #{node_name} -P -y\"
\n `knife #{delete_args}`
\nend<\/p>\n
\n response = Net::HTTP.get(URI(nginx_status_url))
\n nginx_status_data = JSON.parse(response)<\/p>\n
\n server_count = active_nodes.length
\n current_conns = get_nginx_server_conns(nginx_status_data, nginx_server_zone)<\/p>\n
\n puts \"connections per server = #{conns_per_server}\"<\/p>\n
\n if server_count < max_server_count\n puts "Creating new #{cloud_provider} Instance"\n add_backend_node(create_args)\n end\n elsif conns_per_server min_server_count
\n del_backend_node(nginx_status_data, nginx_node, active_nodes, cloud_provider, nginx_upstream)
\n end<\/p>\n
\n sleep(sleep_interval_in_seconds)
\nend<\/pre>\nDeploying an Autoscaling Stack<\/h2>\n
knife-ec2<\/code> plug-in:<\/p>\nDamians-MacBook-Pro:chef-repo damiancurry$ knife ec2 server create -r \"role[autoscaler]\" -g sg-1f285866 -I ami-93d80ff3 -f m1.medium -S chef-demo --region us-west-2 --node-name autoscaler-test --ssh-user ubuntu -i ~\/.ssh\/chef-demo.pem<\/span>
\nInstance ID: i-0c359f3a443d18d64
\nFlavor: m1.medium
\nImage: ami-93d80ff3
\nRegion: us-west-2
\nAvailability Zone: us-west-2a
\nSecurity Group Ids: sg-1f285866
\nTags: Name: autoscaler-test
\nSSH Key: chef-demo<\/p>\n
\nPublic DNS Name: ec2-35-164-35-19.us-west-2.compute.amazonaws.com
\nPublic IP Address: 35.164.35.19
\nPrivate DNS Name: ip-172-31-27-162.us-west-2.compute.internal
\nPrivate IP Address: 172.31.27.162<\/p>\n
\nSSH Target Address: ec2-35-164-35-19.us-west-2.compute.amazonaws.com(dns_name)
\ndone<\/p>\n
\nCreating new client for autoscaler-test
\nCreating new node for autoscaler-test
\nConnecting to ec2-35-164-35-19.us-west-2.compute.amazonaws.com
\nec2-35-164-35-19.us-west-2.compute.amazonaws.com -----> Installing Chef Omnibus (-v 12)
\n\u2026
\nec2-35-164-35-19.us-west-2.compute.amazonaws.com Chef Client finished, 6\/6 resources updated in 13 seconds<\/pre>\nrequire 'chef\/api_client'
\nrequire 'chef\/config'
\nrequire 'chef\/knife'
\nrequire 'chef\/node'
\nrequire 'chef\/search\/query'
\nrequire 'net\/http'
\nrequire 'json'
\nclass MyCLI
\n include Mixlib::CLI
\nend<\/p>\n
\nnginx_node = \"[]\"
\ncloud_provider = \"ec2\"
\nnginx_upstream = \"test\"
\nnginx_server_zone = \"test.local\"
\nif cloud_provider == \"ec2\"
\n create_args = [\"#{cloud_provider}\", 'server', 'create', '-r', \"role[#{nginx_upstream}-upstream]\", '-S', 'damiancurry', '-I', 'ami-93d80ff3', '--region', 'us-west-2', '-f', 'm1.medium', '--ssh-user', 'ubuntu', '-i', '~\/.ssh\/damiancurry.pem']
\nelsif cloud_provider == \"openstack\"
\n create_args = [\"#{cloud_provider}\", 'server', 'create', '-i', '~\/.ssh\/demo_key.pem', '--ssh-user', 'ubuntu', '-f', 'demo_flavor', '--openstack-private-network', '-Z', 'nova', '-r', \"role[#{nginx_upstream}-upstream]\"]
\nelse
\n puts \"Please specify a valid cloud provider\"
\n exit
\nend
\nsleep_interval_in_seconds = 10
\nmin_server_count = 1
\nmax_server_count = 10
\nmin_conns = 10
\nmax_conns = 20
\nnginx_status_url = \"http:\/\/#{nginx_node}:8080\/status\"<\/p>\n
\n active_nodes = Array.new
\n peers = nginx_status_data[\"upstreams\"][\"#{nginx_upstream}\"][\"peers\"]
\n peers.each do |node|
\n if node[\"state\"] == \"up\"
\n active_nodes.push node[\"server\"]
\n end
\n end
\n return active_nodes
\nend<\/p>\n
\n return nginx_status_data[\"server_zones\"][\"#{nginx_server_zone}\"][\"processing\"]
\nend<\/p>\n
\n knife = Chef::Knife.new
\n knife.options=MyCLI.options
\n Chef::Knife.run(create_args, MyCLI.options)
\n #sleep to wait for chef run
\n 1.upto(10) do |n|
\n puts \".\"
\n sleep 1 # second
\n end
\nend<\/p>\n
\n #lookup hostnames\/ips and pick a backend at random
\n query = Chef::Search::Query.new
\n #nodes = query.search('node', 'role:#{nginx_upstream}-upstream').first rescue []
\n nodes = query.search('node', 'role:test-upstream').first rescue []
\n hosts = Array.new
\n nodes.each do |node|
\n node_name = node.name
\n node_ip = node['ipaddress']
\n if active_nodes.any? { |val| \/#{node_ip}\/ =~ val }
\n hosts.push \"#{node_name}:#{node_ip}\"
\n end
\n end
\n del_node = hosts.sample
\n node_name = del_node.rpartition(\":\").first
\n node_ip = del_node.rpartition(\":\").last
\n puts \"Removing #{node_name}\"
\n nginx_url = \"http:\/\/#{nginx_node}:8080\/upstream_conf?upstream=#{nginx_upstream}\"
\n response = Net::HTTP.get(URI(nginx_url))
\n node_id = response.lines.grep(\/#{node_ip}\/).first.split('id=').last.chomp
\n drain_url = \"http:\/\/#{nginx_node}:8080\/upstream_conf?upstream=#{nginx_upstream}&id=#{node_id}&drain=1\"
\n Net::HTTP.get(URI(drain_url))
\n sleep(5)
\n knife = Chef::Knife.new
\n knife.options=MyCLI.options
\n #delete_args = [\"#{cloud_provider}\", 'server', 'delete', \"#{node_name}\", '--purge', '-y']
\n #Chef::Knife.run(delete_args, MyCLI.options)
\n delete_args = \"#{cloud_provider} server delete #{node_name} -P -y\"
\n `knife #{delete_args}`
\nend<\/p>\n
\n response = Net::HTTP.get(URI(nginx_status_url))
\n nginx_status_data = JSON.parse(response)<\/p>\n
\n server_count = active_nodes.length
\n current_conns = get_nginx_server_conns(nginx_status_data, nginx_server_zone)<\/p>\n
\n puts \"connections per server = #{conns_per_server}\"<\/p>\n
\n if server_count < max_server_count\n puts "Creating new #{cloud_provider} Instance"\n add_backend_node(create_args)\n end\n elsif conns_per_server min_server_count
\n del_backend_node(nginx_status_data, nginx_node, active_nodes, cloud_provider, nginx_upstream)
\n end<\/p>\n
\n sleep(sleep_interval_in_seconds)
\nend<\/p>\nnginx_node<\/code> variable at the top of the script does not have an IP address associated with it yet. This is because we haven\u2019t created an NGINX Plus server yet, but Chef will update the script with that information once it has been created.<\/p>\nDamians-MacBook-Pro:default damiancurry$ knife ec2 server create -r \"role[nginx_plus_autoscale]\" -g sg-1f285866 -I ami-93d80ff3 -f m1.medium -S chef-demo --region us-west-2 --ssh-user ubuntu -i ~\/.ssh\/chef-demo.pem --node-name nginx-autoscale
\nInstance ID: i-0856ee80f54c8f3e6
\nFlavor: m1.medium
\nImage: ami-93d80ff3
\nRegion: us-west-2
\nAvailability Zone: us-west-2b
\nSecurity Group Ids: sg-1f285866
\nTags: Name: nginx-autoscale
\nSSH Key: chef-demo<\/p>\n
\nPublic DNS Name: ec2-35-165-171-46.us-west-2.compute.amazonaws.com
\nPublic IP Address: 35.165.171.46
\nPrivate DNS Name: ip-172-31-38-163.us-west-2.compute.internal
\nPrivate IP Address: 172.31.38.163<\/p>\n
\nSSH Target Address: ec2-35-165-171-46.us-west-2.compute.amazonaws.com(dns_name)
\ndone<\/p>\n
\nCreating new client for nginx-autoscale
\nCreating new node for nginx-autoscale
\nConnecting to ec2-35-165-171-46.us-west-2.compute.amazonaws.com
\nec2-35-165-171-46.us-west-2.compute.amazonaws.com -----> Installing Chef Omnibus (-v 12)
\n\u2026
\nec2-35-165-171-46.us-west-2.compute.amazonaws.com Chef Client finished, 24\/34 resources updated in 43 seconds<\/pre>\nroot@ip-172-31-27-162:~# grep 'nginx_node =' \/usr\/bin\/autoscale_nginx.rb<\/span>
\nnginx_node = \"172.31.38.163\"
\nroot@ip-172-31-27-162:~#<\/pre>\n![\"\"](\"https:\/\/cdn-1.wp.nginx.com\/wp-content\/uploads\/2017\/10\/Autoscaling1-1024x512.png\")
<\/p>\n502<\/code> Bad<\/code> Gateway<\/code><\/span> error page because you haven\u2019t started any backend application servers yet.<\/p>\n
\n#!\/bin\/bash
\nNGINX_NODES=\"$(mktemp)\"
\n\/usr\/bin\/curl -s \"http:\/\/localhost:8080\/upstream_conf?upstream=test\"| \/usr\/bin\/awk '{print $2}' | \/bin\/sed -r 's\/;\/\/g' | \/usr\/bin\/sort > $NGINX_NODES
\nCONFIG_NODES=\"$(mktemp)\"
\n\/bin\/grep -E '[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}' \/etc\/nginx\/conf.d\/test-upstream.conf | \/usr\/bin\/awk '{print $2}' | \/bin\/sed -r 's\/;\/\/g' | \/usr\/bin\/sort > $CONFIG_NODES
\nDIFF_OUT=\"$(mktemp)\"
\n\/usr\/bin\/diff $CONFIG_NODES $NGINX_NODES > $DIFF_OUT
\nADD_NODE=`\/usr\/bin\/diff ${CONFIG_NODES} ${NGINX_NODES} | \/bin\/grep \"\" | \/usr\/bin\/awk '{print $2}'`<\/p>\n
\n echo \"adding node ${i}\";
\n \/usr\/bin\/curl -s \"http:\/\/localhost:8080\/upstream_conf?add=&upstream=test&server=${i}&max_fails=0\"
\ndone
\nfor i in $DEL_NODE; do
\n echo \"removing node ${i}\";
\n #NODE_ID=`\/usr\/bin\/curl -s \"http:\/\/localhost:8080\/upstream_conf?upstream=test\" | \/bin\/grep ${i} | \/usr\/bin\/awk '{print $4}' | \/bin\/sed -r 's\/id=\/\/g'`
\n NODE_ID=`\/usr\/bin\/curl -s \"http:\/\/localhost:8080\/upstream_conf?upstream=test\" | \/bin\/grep ${i} | \/bin\/grep -oP 'id=Kd+'`
\n NODE_COUNT=`\/usr\/bin\/curl -s \"http:\/\/localhost:8080\/upstream_conf?upstream=test\" | \/bin\/grep -n ${i} | \/bin\/grep -oP 'd+:server' | \/bin\/sed -r 's\/:server\/\/g'`
\n JSON_NODE_NUM=$(expr $NODE_COUNT - 1)
\n NODE_CONNS=`\/usr\/bin\/curl -s \"http:\/\/localhost:8080\/status\" | \/usr\/bin\/jq \".upstreams.test.peers[${JSON_NODE_NUM}].active\"`
\n NODE_STATE=`\/usr\/bin\/curl -s \"http:\/\/localhost:8080\/status\" | \/usr\/bin\/jq \".upstreams.test.peers[${JSON_NODE_NUM}].state\"`
\n if [[ ${NODE_STATE} == '\"up\"' ]] && [[ ${NODE_CONNS} == 0 ]]; then
\n\techo \"nodes is up with no active connections, removing ${i}\"
\n\t\/usr\/bin\/curl -s \"http:\/\/localhost:8080\/upstream_conf?remove=&upstream=test&id=${NODE_ID}\"
\n elif [[ ${NODE_STATE} == '\"draining\"' ]] && [[ ${NODE_CONNS} == 0 ]]; then
\n echo \"nodes is draining with no active connections, removing ${i}\"
\n \/usr\/bin\/curl -s \"http:\/\/localhost:8080\/upstream_conf?remove=&upstream=test&id=${NODE_ID}\"
\n elif [[ ${NODE_STATE} == '\"down\"' ]]; then
\n\techo \"node state is down, removing ${i}\":
\n\t\/usr\/bin\/curl -s \"http:\/\/localhost:8080\/upstream_conf?remove=&upstream=test&id=${NODE_ID}\"
\n elif [[ ${NODE_STATE} == '\"unhealthy\"' ]]; then
\n\techo \"node state is down, removing ${i}\":
\n\t\/usr\/bin\/curl -s \"http:\/\/localhost:8080\/upstream_conf?remove=&upstream=test&id=${NODE_ID}\"
\n elif [[ ${NODE_STATE} == '\"up\"' ]] && [[ ${NODE_CONNS} != 0 ]]; then
\n\techo \"node has active connections, draining connections on ${i}\"
\n fi
\ndone<\/p>\n
\nubuntu@ip-172-31-38-163:~$<\/pre>\napt_update<\/code> script:<\/p>\ntemplate \"\/etc\/nginx\/conf.d\/#{node[:nginx][:upstream]}-upstream.conf\" do
\n source 'upstreams.conf.erb'
\n owner 'root'
\n group node['root_group']
\n mode 0644
\n variables(
\n hosts: upstream_node_ips
\n )
\n # notifies :reload, 'service[nginx]', :delayed
\n notifies :run, 'execute[run_api_update_script]', :delayed
\nend<\/pre>\nubuntu@ip-172-31-27-162:~$ \/opt\/chef\/embedded\/bin\/ruby \/usr\/bin\/autoscale_nginx.rb<\/span>
\nCurrent connections = 0
\nconnections per server = NaN
\nCreating new ec2 Instance
\nNo existing hosts
\ntest-app-1
\nInstance ID: i-0c671d851a1c5e6d0
\nFlavor: m1.medium
\nImage: ami-93d80ff3
\nRegion: us-west-2
\nAvailability Zone: us-west-2b
\nSecurity Group Ids: chef-demo
\nTags: Name: test-app-1
\nSSH Key: chef-demo<\/p>\n
\n\u2026
\nec2-35-165-4-158.us-west-2.compute.amazonaws.com Chef Client finished, 16\/26 resources updated in 34 seconds
\n\u2026
\nPrivate IP Address: 172.31.40.186
\nEnvironment: _default
\nRun List: role[test-upstream]
\n.
\n.
\n.
\n.
\n.
\n.
\n.
\n.
\n.
\n.
\nCurrent connections = 0
\nconnections per server = 0.0
\nCurrent connections = 0
\nconnections per server = 0.0<\/pre>\n502<\/code> error page:<\/p>\n![\"\"](\"https:\/\/cdn-1.wp.nginx.com\/wp-content\/uploads\/2017\/10\/Autoscaling2.png\")
<\/p>\n![\"\"](\"https:\/\/cdn-1.wp.nginx.com\/wp-content\/uploads\/2017\/10\/Autoscaling3.png\")
<\/p>\nwrk<\/code> to generate some load against the site:<\/p>\nDamians-MacBook-Pro:wrk damiancurry$ .\/wrk -c 25 -t 2 -d 10m http:\/\/ec2-35-165-171-46.us-west-2.compute.amazonaws.com\/<\/span>
\nRunning 10m test @ http:\/\/ec2-35-165-171-46.us-west-2.compute.amazonaws.com\/
\n 2 threads and 25 connections<\/p>\n
\nconnections per server = 0.0
\nCurrent connections = 24
\nconnections per server = 24.0
\nCreating new ec2 Instance
\nnew number is
\n2
\ntest-app-2
\nInstance ID: i-07186f5451c7d9e77
\nFlavor: m1.medium
\nImage: ami-93d80ff3
\nRegion: us-west-2
\nAvailability Zone: us-west-2b
\nSecurity Group Ids: chef-demo
\nTags: Name: test-app-2
\nSSH Key: chef-demo<\/p>\n
\n\u2026.
\nec2-35-166-214-136.us-west-2.compute.amazonaws.com Chef Client finished, 16\/26 resources updated in 35 seconds
\nCurrent connections = 24
\nconnections per server = 12.0
\nCurrent connections = 24
\nconnections per server = 12.0<\/pre>\nCurrent connections = 24
\nconnections per server = 12.0
\nCurrent connections = 0
\nconnections per server = 0.0
\nRemoving test-app-2
\nno instance id is specific, trying to retrieve it from node name
\nWARNING: Deleted server i-0dcf4740c1b34417f
\nWARNING: Deleted node test-app-2
\nWARNING: Deleted client test-app-2
\nCurrent connections = 0
\nconnections per server = 0.0<\/pre>\nConclusion<\/h2>\n