Secure Distribution of SSL Private Keys with NGINX<\/p>\n
This blog post describes several methods for securely distributing the SSL private keys that NGINX uses when hosting SSL‑encrypted websites. It explains:<\/p>\n
For many deployments, the standard approach is sufficient. The two more sophisticated approaches discussed in this post block other ways an attacker can obtain SSL private keys. We’ll also look at a couple more techniques in follow‑up posts:<\/p>\n
The approaches presented in this post apply to users who need to manage their own keys and create their own secure key‑distribution strategy. They are not necessary for users who are running NGINX in environments that already integrate with a secret store, such as Kubernetes<\/a>.<\/p>\n This post applies to both NGINX Open Source and NGINX Plus. For ease of reading, we’ll refer to NGINX throughout.<\/p>\n SSL\/TLS is used to authenticate, encrypt, and verify the integrity of network transactions. Websites authenticate themselves using a public certificate<\/strong> signed by a Certificate Authority (CA), and demonstrate they own the certificate by performing calculations using the corresponding private key<\/strong> (which must be kept secret).<\/p>\n If the private key is compromised (disclosed to another entity), there are two main risks.<\/p>\n If the private key is compromised, your only recourse is to contact the CA and request that your certificate be revoked; you must then rely on clients to check and honor the revocation status.<\/p>\n In addition, it is good practice to use certificates with short expiry times (for example, Let’s Encrypt<\/a> certificates expire after 90 days). Shortly before a certificate expires, you need to generate a new private key and obtain a new certificate from the CA. This reduces your exposure in the event the private key is compromised.<\/p>\n Which people and processes can access SSL private keys in NGINX? <\/p>\n First of all, any user who gains Therefore, no matter how the private key is stored and distributed, it\u2019s not possible to protect the private key from an attacker with Next, any user who can modify and commit NGINX configuration can use that power in many ways \u2013 to open proxy access to internal services, to bypass authentication measures, etc. He or she can modify NGINX configuration to obtain Therefore, it is generally not possible to protect the private key from an attacker who can modify and commit NGINX configuration.<\/p>\n Fortunately, any competent organization has sound security processes to make it difficult for an attacker to gain However, there are two other ways that a less privileged attacker might obtain access to the private key:<\/p>\n The processes described in this document explain how to seal these two disclosure methods.<\/p>\n We begin by reviewing what a typical NGINX configuration with SSL\/TLS looks like:<\/p>\n The SSL public certificate (a.dev0.crt<\/strong>) and private key (a.dev0.key<\/strong>) are stored in the filesystem, at \/etc\/nginx\/ssl\/<\/strong>. The private key is only read by the NGINX master process, which typically runs as The private key must be available at all times; the NGINX master process reads it whenever the NGINX software starts, configuration is reloaded, or a syntax check is performed ( For more information on configuring SSL\/TLS, see the NGINX Plus Admin Guide<\/a>.<\/p>\n As noted above, the SSL private key can be read by an attacker who gains NGINX supports encrypted private keys, using secure algorithms such as AES256:<\/p>\n When you then start NGINX, or reload or test NGINX configuration, NGINX requests the decryption password interactively:<\/p>\n Entering passwords interactively is inconvenient and difficult to automate, but you can configure NGINX to use the passwords stored in a separate file named by the The This method reduces the attack surface by making the NGINX configuration alone useless to an attacker. The attacker must also obtain the contents of the If an attacker does gain You can reduce this risk by storing the The process below describes a more secure way to distribute lists of SSL passwords, from a central distribution point. <\/p>\n Whenever NGINX needs to decrypt an SSL key, it queries the central distribution point and uses the passwords without ever storing them on the local disk. To authenticate itself with the central password server, the NGINX instance uses a token which you can revoke at any time to cut off access to the passwords.<\/p>\n Begin by creating a password distribution point (PDP). For this simple implementation, we’re using an HTTPS service to deliver the password list, authenticated by username and password:<\/p>\n You can then enable or revoke access by adding or removing authentication tokens at the PDP as needed. You can implement the password distribution server using a web server such as NGINX, and use whatever kind of authentication tokens is appropriate.<\/p>\n Next, we need to set up NGINX to retrieve the passwords from the PDP. We start by creating a shell script called connector.sh<\/strong> with the following contents:<\/p>\n The script needs to run as a background process, invoked as follows:<\/p>\n The connector attaches to the specified local path (\/var\/run\/nginx\/ssl_passwords<\/strong>), and you use the Test the connector by reading from the connector path:<\/p>\n Verify that NGINX can read the password and decrypt the SSL keys:<\/p>\n You can use the central PDP approach to securely distribute any resource that NGINX normally reads from disk, for example individual private keys or other sensitive data.<\/p>\n This solution has several benefits compared to storing SSL passwords on disk:<\/p>\n Note that a user who has access to the filesystem can potentially extract the credentials used to access the PDP. It is important to revoke these credentials when they are no longer needed.<\/p>\n There are many ways to protect SSL private keys from disclosure, with increasing security and complexity.<\/p>\n For the large majority of organizations, it is sufficient to restrict access to the environments running NGINX so that unauthorized users cannot gain For some environments, it might not be possible to fully restrict access to NGINX configuration, so an SSL password file can be used.<\/p>\n In limited cases, organizations may wish to ensure that keys and passwords are never stored on disk. The password distribution point<\/a> process illustrates a proof of concept for this solution.<\/p>\n The following blog posts in this series will show additional steps that can be taken:<\/p>\n The post Secure Distribution of SSL Private Keys with NGINX<\/a> appeared first on NGINX<\/a>.<\/p>\n Source: Secure Distribution of SSL Private Keys with NGINX<\/a><\/p>\n","protected":false},"excerpt":{"rendered":" Secure Distribution of SSL Private Keys with NGINX This blog post describes several methods for securely distributing the SSL private keys that NGINX uses when hosting SSL‑encrypted websites. It explains: The standard approach for configuring SSL with NGINX, and the potential security limitations How to encrypt the keys using passwords that are stored separately from the NGINX configuration How to distribute the encryption passwords securely, avoiding disk storage, and then revoke them when needed For many deployments, the standard approach is sufficient. The two more sophisticated approaches discussed in this post block other ways an attacker can obtain SSL private keys. We’ll also look at a couple more techniques in follow‑up posts: Using third‑party secret stores such as Hashicorp Vault to securely distribute passwords Automating the provisioning of certificates from Vault to NGINX Plus\u2019s key‑value store, so that private key material [ more… ]<\/a><\/p>\n<\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[169],"tags":[652],"class_list":["post-30267","post","type-post","status-publish","format-standard","hentry","category-news","tag-nginx"],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/posts\/30267","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/comments?post=30267"}],"version-history":[{"count":1,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/posts\/30267\/revisions"}],"predecessor-version":[{"id":30268,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/posts\/30267\/revisions\/30268"}],"wp:attachment":[{"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/media?parent=30267"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/categories?post=30267"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jirak.net\/wp\/wp-json\/wp\/v2\/tags?post=30267"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Why Protect the SSL Private Key?<\/h2>\n
\n
The NGINX Security Boundary<\/h2>\n
root<\/code> access to the server running NGINX is able to read and use all resources that NGINX itself uses. For example, there are known methods to extract the SSL private key from the memory of a running process. <\/p>\nroot<\/code> privileges on the host server.<\/p>\nroot<\/code> access (or equivalent) to the server, although tools like SELinux<\/a> and AppArmor<\/a> help mitigate against that possibility.<\/p>\nroot<\/code> privileges or to modify NGINX configuration.<\/p>\n\n
Standard NGINX Configuration<\/h2>\n
server {\r\n listen 443 ssl;\r\n\r\n server_name a.dev0; \r\n\r\n ssl_certificate ssl\/a.dev0.crt;<\/strong>\r\n ssl_certificate_key ssl\/a.dev0.key;<\/strong>\r\n\r\n location \/ {\r\n return 200 \"Hello from service An\";\r\n }\r\n}<\/code><\/pre>\nroot<\/code>, so you can set the strictest possible access permissions on it:<\/p>\nroot@web1:\/etc\/nginx\/ssl# ls -l a.dev0.key<\/span>\r\n-r-------- 1 root root 1766 Aug 15 16:32 a.dev0.key<\/code><\/pre>\nnginx<\/code> -t<\/code><\/span>).<\/p>\nSecurity Implications of the Standard Configuration<\/h3>\n
root<\/code> access to the running container, virtual machine, or server that is running the NGINX software.<\/p>\nEncrypting SSL Private Keys<\/h2>\n
root@web1:\/etc\/nginx\/ssl# mv a.dev0.key a.dev0.key.plain<\/span>\r\nroot@web1:\/etc\/nginx\/ssl# openssl rsa -aes256 -in a.dev0.key.plain -out a.dev0.key<\/span>\r\nwriting RSA key\r\nEnter PEM pass phrase: secure password<\/em><\/span>\r\nVerifying - Enter PEM pass phrase: secure password again<\/em><\/span><\/code><\/pre>\nroot@web1:\/etc\/nginx# nginx -t<\/span>\r\nEnter PEM pass phrase: secure password<\/em><\/span>\r\nnginx: the configuration file \/etc\/nginx\/nginx.conf syntax is ok\r\nnginx: configuration file \/etc\/nginx\/nginx.conf test is successful<\/code><\/pre>\nUsing an SSL Password File<\/h3>\n
ssl_password_file<\/code><\/a> directive. When NGINX needs to read a private key, it attempts to decrypt the key using each of the passwords in the file in turn. If none of the passwords is valid, NGINX refuses to start.<\/p>\nssl_password_file \/var\/lib\/nginx\/ssl_passwords.txt;<\/code><\/pre>\nssl_password_file<\/code> must be distributed separately from the configuration, and be readable only by the root<\/code> user. You can regard it as an authorization token that is placed on trusted servers. NGINX can only decrypt the private keys when it is running on a server with the authorization token.<\/p>\nSecurity Implications of Encrypted Keys<\/h3>\n
ssl_password_file<\/code>.<\/p>\nroot<\/code> access to the filesystem where the ssl_password_file<\/code> is stored (for example, from a backup or through the host system), he or she can read the file and use the passwords to decrypt SSL private keys.<\/p>\nssl_password_file<\/code> on a RAM disk or tmpfs<\/strong>. This storage is generally less accessible to an external attacker (for example, it\u2019s cleared when the server is restarted) and can be excluded from system backups. You need to ensure that the password file is initialized on system boot.<\/p>\nDistributing SSL Password Lists More Securely<\/h2>\n
Creating a Central Password Distribution Point<\/h3>\n
$ curl -u dev0:mypassword https:\/\/pdpserver.local\/ssl_passwords.txt<\/span>\r\npassword1\r\npassword2<\/code><\/pre>\n#!\/bin\/sh\r\n\r\n# Usage: connector.sh \r\n\r\nCONNECTOR=$1\r\nCREDS=$2\r\nPDP_URL=$3\r\n\r\n[ -e $CONNECTOR ] && \/bin\/rm -f $CONNECTOR\r\n\r\nmkfifo $CONNECTOR; chmod 600 $CONNECTOR\r\n\r\nwhile true; do\r\n curl -s -u $CREDS -k $PDP_URL -o $CONNECTOR\r\ndone<\/code><\/pre>\nroot@web1:~# .\/connector.sh \/var\/run\/nginx\/ssl_passwords \r\ndev0:mypassword https:\/\/pdpserver.local\/ssl_passwords.txt &<\/span><\/code><\/pre>\nssl_password_file<\/code> directive to configure NGINX to access that path:<\/p>\nssl_password_file \/var\/run\/nginx\/ssl_passwords;<\/code><\/pre>\nroot@web1:~# cat \/var\/run\/nginx\/ssl_passwords<\/span>\r\npassword1\r\npassword2<\/code><\/pre>\nroot@web1:~# nginx -t<\/span>\r\nnginx: the configuration file \/etc\/nginx\/nginx.conf syntax is ok\r\nnginx: configuration file \/etc\/nginx\/nginx.conf test is successful<\/code><\/pre>\nSecurity Implications of a PDP<\/h3>\n
\n
Summary<\/h2>\n
root<\/code> access and cannot look at NGINX configuration.<\/p>\n\n