Deploying NGINX Plus for High Availability with Chef
In a previous blog post, we went over how to deploy NGINX Plus using Chef. In this blog post we will build on the Chef configs from the last blog post and use Chef to deploy a highly available (HA) NGINX Plus active/passive cluster.
This post assumes that you have already run through the previous post and have a working Chef installation. If not, please review at least the first two sections in the previous post, Preparing Your Chef Environment and Downloading and Configuring the NGINX Cookbook.
To set up the HA active/passive cluster, we’re using the HA solution provided by NGINX, Inc., which is based on keepalived and the Virtual Router Redundancy Protocol (VRRP). The solution comes with an interactive script for creating the keepalived configuration file, but here we’re using Chef to automate the complete setup process.
Modifying the NGINX Cookbook
First we set up the Chef files for installing of the NGINX Plus HA package (nginx-ha-keepalived) and creating the keepalived configuration file, keepalive.conf.
-
Modify the existing plus_package recipe to include package and configuration templates for the HA solution, by adding the following code to the bottom of the plus_package.rb file (per the instructions in the previous post, the file is in the ~/chef-zero/playground/cookbooks/nginx/recipes directory).
We are using the eth1 interface on each NGINX host, which makes the code a bit more complicated than if we used eth0. In case you are using eth0, the relevant code appears near the top of the file, commented out.
This code does three things:
- It looks up the IP address of the eth1 interface on the node where NGINX Plus is being installed, and assigns the value to the
origipvariable so it can be passed to the template. - It finds the other node in the HA pair by using Chef’s
searchfunction to iterate through all Chef nodes, then looks up the IP address for that node’s eth1 interface and assigns the address to theha_pair_ipsvariable. - It installs the nginx-ha-keepalived package, registers the
keepalivedservice with Chef, and generates the keepalived.conf configuration file as a template, passing in the values of theorigipandha_pair_ipsvariables.
if node['nginx']['enable_ha_mode'] == 'true'
ha_pair_ips = Array.new
origip = "#{node[:network][:interfaces][:eth1][:addresses].detect{|k,v| v[:family] == 'inet'}.first}"
# The code for finding the IP address of the eth0 interface
# follows, commented out.
#origip = "#{node[:ipaddress]}"
#search(:node, "role:nginx_plus_ha") do |nodes|
# ha_pair_ips << nodes["ipaddress"] #end # This is a workaround for getting the IP address for the eth1 # that VMs need search(:node, "role:nginx_plus_ha AND enable_ha_mode:true NOT name:#{node.name}") do |nodes| nodes["network"]["interfaces"]["eth1"]["addresses"].each_pair do |address,value| ha_pair_ips < true, :restart => true, :reload => true
action :enable
end
template '/etc/keepalived/keepalived.conf' do
source 'nginx_plus_keepalived.conf.erb'
owner 'root'
group node['root_group']
mode '0644'
variables(
:myip => origip,
:ha_pair_ip => ha_pair_ips
)
notifies :reload, 'service[keepalived]', :delayed
end
endYou can download the full recipe file from the NGINX, Inc. website.
- It looks up the IP address of the eth1 interface on the node where NGINX Plus is being installed, and assigns the value to the
-
Create the Chef template for creating keepalived.conf, by copying the following content to a new template file, nginx_plus_keepalived.conf.erb, in the ~/chef-zero/playground/cookbooks/nginx/templates/default directory.
We’re using a combination of variables and attributes to pass the necessary information to keepalived.conf. We’ll set the attributes in the next step. Here we set the two variables in the template file to the host IP addresses that were set with the
variablesdirective in the plus_package.rb recipe (modified in the previous step):myip– The primary IP address used bykeepalivedto communicate with the other HA nodes. Corresponds to theorigipvariable in the plus_package recipe.ha_pair_ip– An array containing the IP address of each host that has the nginx_plus_ha role in its run list; it is used to set the IP address of the peer (secondary) host in keepalived.conf. Corresponds to theha_pair_ipsarray in the plus_package recipe.
vrrp_script chk_nginx_service {
script "/usr/lib/keepalived/nginx-ha-check"
interval 3
weight 50
}vrrp_instance VI_1 {
interface eth1
state MASTER
priority 151
state BACKUP
priority 150
virtual_router_id 51
advert_int 1
unicast_src_ip
unicast_peer {
}
authentication {
auth_type PASS
auth_pass
}
virtual_ipaddress {
}
track_script {
chk_nginx_service
}
notify "/usr/lib/keepalived/nginx-ha-notify"
} -
Create a role that sets attributes used in the recipe and template files created in the previous steps, by copying the following contents to a new role file, nginx_plus_ha.rb in the ~/chef-zero/playground/roles directory.
Four attributes need to be set, and in the role we set the following three:
[nginx][ha_keepalived_key]– The authentication key used bykeepalivedto encrypt communication with the other HA nodes. Used in the template file.[nginx][ha_vip]– The virtual IP address (VIP) advertised to clients as the address for NGINX Plus. Thekeepalivedprocess assigns it to the primary NGINX Plus instance, and is is responsible for transferring it to the other instance in case of failover. Used in the template file.[nginx][enable_ha_mode]– Triggers the HA section of the plus_package recipe when set totrue. Used in the recipe file.
We don’t set the fourth attribute,
ha_primary, in the role because it must be set on a per-host basis. It is used in the template file to set thestateandpriorityvalues in keepalived.conf; those values in turn determine which NGINX Plus instance is marked as primary. Ifha_primaryistruefor a node, it becomes the primary NGINX Plus instance in the HA pair; it becomes the secondary instance if the value isfalse.It is also possible to set the primary instance based on hostname matching instead of the
ha_primaryattribute, as we’ll cover in Setting the Primary Node Based on Hostname Matching. Also, you can set all four attributes on a per-host basis, but I prefer to keep as many of the shared attributes in a single location as possible.The
run_listdirective in the role references the nginx_plus role we created in the previous blog post, to save us from having to duplicate everything already defined in that role.name "nginx_plus_ha"
description "An example role to install nginx plus in an HA cluster"
run_list "role[nginx_plus]"
default_attributes "nginx" => { "enable_ha_mode" => "true",
"ha_keepalived_key" => "a0cf476cf069ea3dfa8940ff6d6bd885",
"ha_vip" => "10.100.10.50"
}
Preparing Nodes for Installation
Now we bootstrap the nodes and get them ready for the installation. Note that the longer outputs have been truncated, leaving only the most important output.
Setting Up the First Node
- Upload all of the files and bootstrap the first node:
root@chef-server:~# cd chef-zero/playground/cookbooks/
root@chef-server:~/chef-zero/playground/cookbooks# knife cookbook upload *
Uploading apache2 [1.0.0]
Uploading apt [2.8.2]
...
Uploading nginx [2.7.6]
...
Uploading yum [3.8.1]
Uploading yum-epel [0.6.3]
Uploaded 18 cookbooks.root@chef-server:~/chef-zero/playground/cookbooks# cd ../
root@chef-server:~/chef-zero/playground# knife role from file roles/nginx_plus.rb
Updated Role nginx_plus!root@chef-server:~/chef-zero/playground# knife role from file roles/nginx_plus_ha.rb
Updated Role nginx_plus_ha!root@chef-server:~/chef-zero/playground# knife bootstrap -N chef-test-1 -x username --sudo 10.100.10.100
Creating new client for chef-test-1
Creating new node for chef-test-1
Connecting to 10.100.10.100
username@10.100.10.100's password:
10.100.10.100 knife sudo password:
Enter your password:
10.100.10.100
10.100.10.100 -----> Existing Chef installation detected
10.100.10.100 Starting first Chef Client run...
10.100.10.100 Starting Chef Client, version 12.6.0
10.100.10.100 resolving cookbooks for run list: []
10.100.10.100 Synchronizing Cookbooks:
10.100.10.100 Compiling Cookbooks...
10.100.10.100 [2016-02-07T06:17:13-08:00] WARN: Node chef-test-1 has an empty run list.
10.100.10.100 Converging 0 resources
10.100.10.100
10.100.10.100 Running handlers:
10.100.10.100 Running handlers complete
10.100.10.100 Chef Client finished, 0/0 resources updated in 01 seconds -
Create a local copy of the node definition file, which we’ll edit as appropriate for the node we bootstrapped in the previous step, chef-test-1:
root@chef-server:~/chef-zero/playground# knife node show chef-test-1 --format json > nodes/chef-test-1.json -
Edit chef-test-1.json to have the following contents. In particular, we’re updating the run list and setting the
ha_primaryattribute, as required for the HA deployment.{
"name": "chef-test-1",
"chef_environment": "_default",
"run_list": [
"role[nginx_plus_ha]"
]
,
"normal": {
"nginx": {
"ha_primary": "true"
},
"tags": []
}
} -
Push the changed node definition to Chef:
root@chef-server:~/chef-zero/playground# knife node from file nodes/chef-test-1.json
Updated Node chef-test-1! -
Log in on the chef-test-1 node and run the
chef‑clientcommand to get everything configured:username@chef-test-1:~$ sudo chef-client
Starting Chef Client, version 12.6.0
resolving cookbooks for run list: ["nginx"]
Synchronizing Cookbooks:
- ohai (2.0.1)
- build-essential (2.2.4)
- nginx (2.7.6)
- yum-epel (0.6.3)
- apt (2.8.2)
- bluepill (2.4.0)
- runit (1.7.2)
- rsyslog (2.1.0)
- packagecloud (0.1.0)
- yum (3.8.1)
Compiling Cookbooks...
….
* template[/etc/keepalived/keepalived.conf] action create
- create new file /etc/keepalived/keepalived.conf
- update content in file /etc/keepalived/keepalived.conf from none to 9816fd
--- /etc/keepalived/keepalived.conf 2016-02-07 06:34:18.117013010 -0800
+++ /etc/keepalived/.keepalived.conf20160207-1079-sjg8xn 2016-02-07 06:34:18.117013010 -0800
@@ -1 +1,28 @@
+vrrp_script chk_nginx_service {
+ script "/usr/lib/keepalived/nginx-ha-check"
+ interval 3
+ weight 50
+}
+
+vrrp_instance VI_1 {
+ interface eth1
+ state MASTER
+ priority 151
+ virtual_router_id 51
+ advert_int 1
+ unicast_src_ip 10.100.10.100
+ unicast_peer {
+ }
+ authentication {
+ auth_type PASS
+ auth_pass a0cf476cf069ea3dfa8940ff6d6bd885
+ }
+ virtual_ipaddress {
+ 10.100.10.50
+ }
+ track_script {
+ chk_nginx_service
+ }
+ notify "/usr/lib/keepalived/nginx-ha-notify"
+}
- change mode from '' to '0644'
- change owner from '' to 'root'
- change group from '' to 'root'
Recipe: nginx::default
* service[nginx] action start (up to date)
* service[nginx] action reload
- reload service service[nginx]
Recipe: nginx::plus_package
* service[keepalived] action reload (up to date)Running handlers:
Running handlers complete
Chef Client finished, 18/50 resources updated in 07 seconds
If we look at keepalived.conf at this point, we see the template hasn’t set any values in the unicast_peer section, because we’ve registered just this one node with Chef. The following command shows that the VIP specified in the role file, 10.100.10.50, is assigned to eth1 on this node, making it the primary HA node:
username@chef-test-1:~$ ip addr show eth1
3: eth1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:c8:66:ee brd ff:ff:ff:ff:ff:ff
inet 10.100.10.100/24 brd 10.100.10.255 scope global eth1
valid_lft forever preferred_lft forever
inet 10.100.10.50/32 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fec8:66ee/64 scope link
valid_lft forever preferred_lft foreverSetting Up the Second Node
Now, let’s get the second node bootstrapped and make sure all of the values are being propagated properly. We know from the first node that the run list works as expected, so we can combine some steps and set the run list and the ha_primary attribute directly with the knife bootstrap command, which we run on the Chef server.
root@chef-server:~/chef-zero/playground# knife bootstrap -N chef-test-2 -x username --json-attributes "{"nginx": {"ha_primary": "false"}}" --sudo --run-list "role[nginx_plus_ha]" 10.100.10.102
Creating new client for chef-test-2
Creating new node for chef-test-2
Connecting to 10.100.10.102
username@10.100.10.102's password:
10.100.10.102 sudo: unable to resolve host chef-test
10.100.10.102 knife sudo password:
Enter your password:
10.100.10.102
10.100.10.102 -----> Existing Chef installation detected
10.100.10.102 Starting first Chef Client run...
10.100.10.102 Starting Chef Client, version 12.6.0
10.100.10.102 resolving cookbooks for run list: ["nginx"]
10.100.10.102 Synchronizing Cookbooks:
10.100.10.102 - bluepill (2.4.0)
10.100.10.102 - apt (2.8.2)
10.100.10.102 - build-essential (2.2.4)
10.100.10.102 - ohai (2.0.1)
10.100.10.102 - nginx (2.7.6)
10.100.10.102 - runit (1.7.2)
10.100.10.102 - yum-epel (0.6.3)
10.100.10.102 - rsyslog (2.1.0)
10.100.10.102 - packagecloud (0.1.0)
10.100.10.102 - yum (3.8.1)
10.100.10.102 Compiling Cookbooks...
…
10.100.10.102 * service[keepalived] action enable (up to date)
10.100.10.102 * template[/etc/keepalived/keepalived.conf] action create
10.100.10.102 - create new file /etc/keepalived/keepalived.conf
10.100.10.102 - update content in file /etc/keepalived/keepalived.conf from none to a9363c
10.100.10.102 --- /etc/keepalived/keepalived.conf 2016-02-07 06:45:10.529976825 -0800
10.100.10.102 +++ /etc/keepalived/.keepalived.conf20160207-11317-1env6hu 2016-02-07 06:45:10.529976825 -0800
10.100.10.102 @@ -1 +1,29 @@
10.100.10.102 +vrrp_script chk_nginx_service {
10.100.10.102 + script "/usr/lib/keepalived/nginx-ha-check"
10.100.10.102 + interval 3
10.100.10.102 + weight 50
10.100.10.102 +}
10.100.10.102 +
10.100.10.102 +vrrp_instance VI_1 {
10.100.10.102 + interface eth1
10.100.10.102 + state BACKUP
10.100.10.102 + priority 150
10.100.10.102 + virtual_router_id 51
10.100.10.102 + advert_int 1
10.100.10.102 + unicast_src_ip 10.100.10.102
10.100.10.102 + unicast_peer {
10.100.10.102 + 10.100.10.100
10.100.10.102 + }
10.100.10.102 + authentication {
10.100.10.102 + auth_type PASS
10.100.10.102 + auth_pass a0cf476cf069ea3dfa8940ff6d6bd885
10.100.10.102 + }
10.100.10.102 + virtual_ipaddress {
10.100.10.102 + 10.100.10.50
10.100.10.102 + }
10.100.10.102 + track_script {
10.100.10.102 + chk_nginx_service
10.100.10.102 + }
10.100.10.102 + notify "/usr/lib/keepalived/nginx-ha-notify"
10.100.10.102 +}
10.100.10.102 - change mode from '' to '0644'
10.100.10.102 - change owner from '' to 'root'
10.100.10.102 - change group from '' to 'root'
10.100.10.102 Recipe: nginx::default
10.100.10.102 * service[nginx] action start (up to date)
10.100.10.102 * service[nginx] action reload
10.100.10.102 - reload service service[nginx]
10.100.10.102 Recipe: nginx::plus_package
10.100.10.102 * service[keepalived] action reload
10.100.10.102 - reload service service[keepalived]
10.100.10.102
10.100.10.102 Running handlers:
10.100.10.102 Running handlers complete
10.100.10.102 Chef Client finished, 18/50 resources updated in 10 secondsIf we look at keepalived.conf at this point, we see that there is a peer set in the unicast_peer section. But the following command shows that chef‑test‑2, which we intend to be the secondary node, is also assigned the VIP (10.100.10.50). This is because we haven’t yet updated the Chef configuration on chef‑test‑1 to make its keepalived aware of the secondary node.
username@chef-test-2:~$ ip addr show eth1
3: eth1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:6d:d9:64 brd ff:ff:ff:ff:ff:ff
inet 10.100.10.102/24 brd 10.100.10.255 scope global eth1
valid_lft forever preferred_lft forever
inet 10.100.10.50/32 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe6d:d964/64 scope link
valid_lft forever preferred_lft foreverSynchronizing the Nodes
To make keepalived on chef-test-1 aware of chef-test-2 and its IP address, we rerun the chef‑client command on chef‑test‑1:
username@chef-test-1:~$ sudo chef-client
Starting Chef Client, version 12.6.0
resolving cookbooks for run list: ["nginx"]
Synchronizing Cookbooks:
- ohai (2.0.1)
- build-essential (2.2.4)
- nginx (2.7.6)
- yum-epel (0.6.3)
- apt (2.8.2)
- bluepill (2.4.0)
- runit (1.7.2)
- rsyslog (2.1.0)
- packagecloud (0.1.0)
- yum (3.8.1)
Compiling Cookbooks...
….
* template[/etc/keepalived/keepalived.conf] action create
- update content in file /etc/keepalived/keepalived.conf from 9816fd to af7ce0
--- /etc/keepalived/keepalived.conf 2016-02-07 06:34:18.117013010 -0800
+++ /etc/keepalived/.keepalived.conf20160207-3369-73qgm3 2016-02-07 06:53:04.593013010 -0800
@@ -12,6 +12,7 @@
advert_int 1
unicast_src_ip 10.100.10.100
unicast_peer {
+ 10.100.10.102
}
authentication {
auth_type PASS
Recipe: nginx::default
* service[nginx] action start (up to date)
* service[nginx] action reload
- reload service service[nginx]
Recipe: nginx::plus_package
* service[keepalived] action reload (up to date)
Running handlers:
Running handlers complete
Chef Client finished, 2/47 resources updated in 05 seconds
We see that chef-test-1 is still assigned the VIP:
username@chef-test-1:~$ ip addr show eth1
3: eth1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:c8:66:ee brd ff:ff:ff:ff:ff:ff
inet 10.100.10.100/24 brd 10.100.10.255 scope global eth1
valid_lft forever preferred_lft forever
inet 10.100.10.50/32 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fec8:66ee/64 scope link
valid_lft forever preferred_lft foreverAnd chef-test-2, as the secondary node, is now assigned only its physical IP address:
username@chef-test-2:~$ ip addr show eth1
3: eth1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:6d:d9:64 brd ff:ff:ff:ff:ff:ff
inet 10.100.10.102/24 brd 10.100.10.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe6d:d964/64 scope link
valid_lft forever preferred_lft foreverIf you want to test the keepalived failover at this point, stop the NGINX Plus service on the primary node.
Setting the Primary Node Based on Hostname Matching
You can automate your HA setup even further if you indicate a host’s usual role (primary or secondary) in its hostname. Then, instead of setting the ha_primary attribute in the node definition file (as in Step 3 of Setting Up the First Node), you can have Chef set the attribute based on the hostname. This takes advantage of Chef’s flexibility in allowing you to set variables inside of recipes, which you do by using slightly modified Ruby code to inspect any of the attributes and membership information the Chef server is aware of. One example is Step 1 of Modifying the NGINX Cookbook, where we added code to the plus_package recipe to look up eth1‘s IP address.
Here we extend that code further by adding an if statement that inspects the hostname and sets ha_primary appropriately. It relies on the presence of the word primary or standby in the hostname. Put it after the code in plus_package.rb that assigns a value to the origip variable (the complete code appears in Step 1 of Modifying the NGINX Cookbook):
...
origip = "#{node[:network][:interfaces][:eth1][:addresses].detect{|k,v| v[:family] == 'inet'}.first}"
search(:node, "role:nginx_plus_ha AND enable_ha_mode:#{node.nginx.enable_ha_mode} NOT name:#{node.name}") do |nodes|
nodes["network"]["interfaces"]["eth1"]["addresses"].each_pair do |address,value|
ha_pair_ips << address if value.has_key?("broadcast")
end
end
if node['name'].include? "primary"
ha_primary = "true"
elsif node['name'].include? "standby"
ha_primary = "false"
end
package 'nginx-ha-keepalived' do
action :install
end
...
Of course you can match on terms in the hostname other than primary and standby, such as lb1 and lb2. You can also put this logic in the nginx_plus_keepalived.conf.erb template file instead, but that is less portable.
Summary
We have covered some more complex examples of what is possible using the Chef search function, as well as Ruby code in the cookbook recipes. It is possible to apply these techniques in many ways, using attributes and the search function to build very portable and flexible Chef configurations. We will cover some of these concepts in future blog posts.
Try out the NGINX Plus HA solution with Chef for yourself – start your free 30-day trial today or contact us for a live demo.
The post Deploying NGINX Plus for High Availability with Chef appeared first on NGINX.
Source: Deploying NGINX Plus for High Availability with Chef
