Avoiding the Top 10 NGINX Configuration Mistakes<\/p>\n
When we help NGINX users who are having problems, we often see the same configuration mistakes we\u2019ve seen over and over in other users\u2019 configurations – sometimes even in configurations written by fellow NGINX engineers! In this blog we look at 10 of the most common errors, explaining what\u2019s wrong and how to fix it.<\/p>\n
error_log<\/code> off<\/code> directive<\/a><\/li>\n- Not enabling keepalive connections to upstream servers<\/a><\/li>\n
- Forgetting how directive inheritance works<\/a><\/li>\n
- The
proxy_buffering<\/code> off<\/code> directive<\/a><\/li>\n- Improper use of the
if<\/code> directive<\/a><\/li>\n- Excessive health checks<\/a><\/li>\n
- Unsecured access to metrics<\/a><\/li>\n
- Using
ip_hash<\/code> when all traffic comes from the same \/24 CIDR block<\/a><\/li>\n- Not taking advantage of upstream groups<\/a><\/li>\n<\/ol>\n
Mistake 1: Not Enough File Descriptors per Worker<\/h2>\n
The worker_connections<\/code><\/a> directive sets the maximum number of simultaneous connections that a NGINX worker process can have open (the default is 512). All types of connections (for example, connections with proxied servers) count against the maximum, not just client connections. But it\u2019s important to keep in mind that ultimately there is another limit on the number of simultaneous connections per worker: the operating system limit on the maximum number of file descriptors (FDs) allocated to each process. In modern UNIX distributions, the default limit is 1024.<\/p>\nFor all but the smallest NGINX deployments, a limit of 512 connections per worker is probably too small. Indeed, the default nginx.conf<\/strong> file we distribute with NGINX Open Source binaries and NGINX Plus increases it to 1024. <\/p>\nThe common configuration mistake is not increasing the limit on FDs to at least twice the value of worker_connections<\/code>. The fix is to set that value with the worker_rlimit_nofile<\/code><\/a> directive in the main configuration context. <\/p>\nHere\u2019s why more FDs are needed: each connection from an NGINX worker process to a client or upstream server consumes an FD. When NGINX acts as a web server, it uses one FD for the client connection and one FD per served file, for a minimum of two FDs per client (but most web pages are built from many files). When it acts as a proxy server, NGINX uses one FD each for the connection to the client and upstream server, and potentially a third FD for the file used to store the server\u2019s response temporarily. As a caching server, NGINX behaves like a web server for cached responses and like a proxy server if the cache is empty or expired.<\/p>\n
NGINX also uses an FD per log file and a couple FDs to communicate with master process, but usually these numbers are small compared to the number of FDs used for connections and files.<\/p>\n
UNIX offers several ways to set the number of FDs per process:<\/p>\n
\n- The
ulimit<\/code> command if you start NGINX from a shell<\/li>\n- The
init<\/code> script or systemd<\/code> service manifest variables if you start NGINX as a service<\/li>\n- The \/etc\/security\/limits.conf<\/strong> file<\/li>\n<\/ul>\n
However, the method to use depends on how you start NGINX, whereas worker_rlimit_nofile<\/code> works no matter how you start NGINX.<\/p>\nThere is also a system‑wide limit on the number of FDs, which you can set with the OS\u2019s sysctl<\/code> fs.file-max<\/code><\/span> command. It is usually large enough, but it is worth verifying that the maximum number of file descriptors all NGINX worker processes might use (worker_rlimit_nofile<\/code> *<\/code> worker_connections<\/code><\/span>) is significantly less than fs.file‑max<\/code>. If NGINX somehow uses all available FDs (for example, during a DoS attack), it becomes impossible even to log in to the machine to fix the issue.<\/p>\nMistake 2: The error_log<\/code> off<\/code> Directive<\/h2>\nThe common mistake is thinking that the error_log<\/code> off<\/code><\/span> directive disables logging. In fact, unlike the access_log<\/code><\/a> directive, error_log<\/code><\/a> does not take an off<\/code> parameter. If you include the error_log<\/code> off<\/code><\/span> directive in the configuration, NGINX creates an error log file named off<\/strong> in the default directory for NGINX configuration files (usually \/etc\/nginx<\/strong>).<\/p>\nWe don\u2019t recommend disabling the error log, because it is a vital source of information when debugging any problems with NGINX. However, if storage is so limited that it might be possible to log enough data to exhaust the available disk space, it might make sense to disable error logging. Include this directive in the main configuration context:<\/p>\n
error_log \/dev\/null emerg;<\/code><\/pre>\nNote that this directive doesn\u2019t apply until NGINX reads and validates the configuration. So each time NGINX starts up or the configuration is reloaded, it might log to the default error log location (usually \/var\/log\/nginx\/error.log<\/strong>) until the configuration is validated. To change the log directory, include the -e<\/code> <error_log_location<\/em>><\/code><\/span> parameter on the nginx<\/code><\/a> command. <\/p>\nMistake 3: Not Enabling Keepalive Connections to Upstream Servers<\/h2>\n
By default, NGINX opens a new connection to an upstream (backend) server for every new incoming request. This is safe but inefficient, because NGINX and the server must exchange three packets to establish a connection and three or four to terminate it.<\/p>\n
At high traffic volumes, opening a new connection for every request can exhaust system resources and make it impossible to open connections at all. Here\u2019s why: for each connection the 4-tuple<\/span> of source address, source port, destination address, and destination port must be unique. For connections from NGINX to an upstream server, three of the elements (the first, third, and fourth) are fixed, leaving only the source port as a variable. When a connection is closed, the Linux socket sits in the TIME‑WAIT<\/code> state for two minutes, which at high traffic volumes increases the possibility of exhausting the pool of available source ports. If that happens, NGINX cannot open new connections to upstream servers.<\/p>\nThe fix is to enable keepalive connections<\/a> between NGINX and upstream servers – instead of being closed when a request completes, the connection stays open to be used for additional requests. This both reduces the possibility of running out of source ports and improves performance<\/a>.<\/p>\nTo enable keepalive connections:<\/p>\n
\n- \n
Include the keepalive<\/code><\/a> directive in every upstream{}<\/code><\/a> block, to set the number of idle keepalive connections to upstream servers preserved in the cache of each worker process.<\/p>\nNote that the keepalive<\/code> directive does not limit the total number of connections to upstream servers that an NGINX worker process can open – this is a common misconception. So the parameter to keepalive<\/code> does not need to be as large as you might think.<\/p>\nWe recommend setting the parameter to twice the number of servers listed in the upstream{}<\/code> block. This is large enough for NGINX to maintain keepalive connections with all the servers, but small enough that upstream servers can process new incoming connections as well.<\/p>\nNote also that when you specify a load‑balancing algorithm in the upstream{}<\/code> block – with the hash<\/code><\/a>, ip_hash<\/code><\/a>, least_conn<\/code><\/a>, least_time<\/code><\/a>, or random<\/code><\/a> directive – the directive must appear above the keepalive<\/code> directive. This is one of the rare exceptions to the general rule that the order of directives in the NGINX configuration doesn\u2019t matter.<\/p>\n<\/li>\n- \n
In the location{}<\/code><\/a> block that forwards requests to an upstream group, include the following directives along with the proxy_pass<\/code><\/a> directive:<\/p>\nproxy_http_version 1.1;\r\nproxy_set_header \"Connection\" \"\";<\/code><\/pre>\nBy default NGINX uses HTTP\/1.0 for connections to upstream servers and accordingly adds the Connection:<\/code> close<\/code><\/span> header to the requests that it forwards to the servers. The result is that each connection gets closed when the request completes, despite the presence of the keepalive<\/code> directive in the upstream{}<\/code> block.<\/p>\nThe proxy_http_version<\/code><\/a> directive tells NGINX to use HTTP\/1.1 instead, and the proxy_set_header<\/code><\/a> directive removes the close<\/code> value from the Connection<\/code> header.<\/p>\n<\/li>\n<\/ul>\nMistake 4: Forgetting How Directive Inheritance Works<\/h2>\n
NGINX directives are inherited downwards, or \u201coutside‑in\u201d: a child<\/em> context – one nested within another context (its parent<\/em>) – inherits the settings of directives included at the parent level. For example, all server{}<\/code><\/a> and location{}<\/code><\/a> blocks in the http{}<\/code> context inherit the value of directives included at the http<\/code> level, and a directive in a server{}<\/code> block is inherited by all the child location{}<\/code> blocks in it. However, when the same directive is included in both a parent context and its child context, the values are not added together – instead, the value in the child context overrides the parent value.<\/p>\nThe mistake is to forget this \u201coverride rule\u201d for array directives<\/em>, which can be included not only in multiple contexts but also multiple times within a given context. Examples include proxy_set_header<\/code><\/a> and add_header<\/code><\/a> – having \u201cadd\u201d in the name of second makes it particularly easy to forget about the override rule. <\/p>\nWe can illustrate how inheritance works with this example for add_header<\/code>:<\/p>\nhttp {\r\n add_header X-HTTP-LEVEL-HEADER 1;\r\n add_header X-ANOTHER-HTTP-LEVEL-HEADER 1;\r\n\r\n server {\r\n listen 8080;\r\n location \/ {\r\n return 200 \"OK\";\r\n } \r\n }\r\n\r\n server {\r\n listen 8081;\r\n add_header X-SERVER-LEVEL-HEADER 1;\r\n\r\n location \/ {\r\n return 200 \"OK\";\r\n }\r\n\r\n location \/test {\r\n add_header X-LOCATION-LEVEL-HEADER 1;\r\n return 200 \"OK\";\r\n }\r\n\r\n location \/correct {\r\n add_header X-HTTP-LEVEL-HEADER 1;\r\n add_header X-ANOTHER-HTTP-LEVEL-HEADER 1;\r\n\r\n add_header X-SERVER-LEVEL-HEADER 1;\r\n add_header X-LOCATION-LEVEL-HEADER 1;\r\n return 200 \"OK\";\r\n } \r\n }\r\n}<\/code><\/pre>\nFor the server listening on port 8080, there are no add_header<\/code> directives in either the server{}<\/code> or location{}<\/code> blocks. So inheritance is straightforward and we see the two headers defined in the http{}<\/code> context: <\/p>\n% curl -is localhost:8081<\/strong>\r\nHTTP\/1.1 200 OK\r\nServer: nginx\/1.21.5\r\nDate: Mon, 21 Feb 2022 10:12:15 GMT\r\nContent-Type: text\/plain\r\nContent-Length: 2\r\nConnection: keep-alive\r\nX-HTTP-LEVEL-HEADER: 1\r\nX-ANOTHER-HTTP-LEVEL-HEADER: 1<\/strong>\r\nOK<\/code><\/pre>\nFor the server listening on port 8081, there is an add_header<\/code> directive in the server{}<\/code> block but not in its child location<\/code> \/<\/code><\/span> block. The header defined in the server{}<\/code> block overrides the two headers defined in the http{}<\/code> context: <\/p>\n% curl -is localhost:8080<\/strong>\r\nHTTP\/1.1 200 OK\r\nServer: nginx\/1.21.5\r\nDate: Mon, 21 Feb 2022 10:12:20 GMT\r\nContent-Type: text\/plain\r\nContent-Length: 2\r\nConnection: keep-alive\r\nX-SERVER-LEVEL-HEADER: 1<\/strong>\r\nOK<\/code><\/pre>\nIn the child location<\/code> \/test<\/code><\/span> block, there is an add_header<\/code> directive and it overrides both the header from its parent server{}<\/code> block and the two headers from the http{}<\/code> context:<\/p>\n% curl -is localhost:8080\/test<\/strong>\r\nHTTP\/1.1 200 OK\r\nServer: nginx\/1.21.5\r\nDate: Mon, 21 Feb 2022 10:12:25 GMT\r\nContent-Type: text\/plain\r\nContent-Length: 2\r\nConnection: keep-alive\r\nX-LOCATION-LEVEL-HEADER: 1<\/strong>\r\nOK<\/code><\/pre>\nIf we want a location{}<\/code> block to preserve the headers defined in its parent contexts along with any headers defined locally, we must redefine the parent headers within the location{}<\/code> block. That\u2019s what we\u2019ve done in the location<\/code> \/correct<\/code><\/span> block:<\/p>\n% curl -is localhost:8080\/correct<\/strong>\r\nHTTP\/1.1 200 OK\r\nServer: nginx\/1.21.5\r\nDate: Mon, 21 Feb 2022 10:12:30 GMT\r\nContent-Type: text\/plain\r\nContent-Length: 2\r\nConnection: keep-alive\r\nX-HTTP-LEVEL-HEADER: 1\r\nX-ANOTHER-HTTP-LEVEL-HEADER: 1\r\nX-SERVER-LEVEL-HEADER: 1\r\nX-LOCATION-LEVEL-HEADER: 1<\/strong>\r\nOK<\/code><\/pre>\nMistake 5: The proxy_buffering<\/code> off<\/code> Directive<\/h2>\nProxy buffering is enabled by default in NGINX (the proxy_buffering<\/code><\/a> directive is set to on<\/code>). Proxy buffering means that NGINX stores the response from a server in internal buffers as it comes in, and doesn\u2019t start sending data to the client until the entire response is buffered. Buffering helps to optimize performance with slow clients – because NGINX buffers the response for as long as it takes for the client to retrieve all of it, the proxied server can return its response as quickly as possible and return to being available to serve other requests. <\/p>\nWhen proxy buffering is disabled, NGINX buffers only the first part of a server\u2019s response before starting to send it to the client, in a buffer that by default is one memory page in size (4 KB<\/span> or 8 KB<\/span> depending on the operating system). This is usually just enough space for the response header. NGINX then sends the response to the client synchronously as it receives it, forcing the server to sit idle as it waits until NGINX can accept the next response segment. <\/p>\nSo we\u2019re surprised by how often we see proxy_buffering<\/code> off<\/code><\/span> in configurations. Perhaps it is intended to reduce the latency experienced by clients, but the effect is negligible while the side effects are numerous: with proxy buffering disabled, rate limiting and caching don\u2019t work even if configured, performance suffers, and so on.<\/p>\nThere are only a small number of use cases where disabling proxy buffering might make sense (such as long polling), so we strongly discourage changing the default. For more information, see the NGINX Plus Admin Guide<\/a>. <\/p>\nMistake 6: Improper Use of the if<\/code> Directive<\/h2>\nThe if<\/code><\/a> directive is tricky to use, especially in location{}<\/code><\/a> blocks. It often doesn\u2019t do what you expect and can even cause segfaults. In fact, it\u2019s so tricky that there\u2019s an article titled If is Evil<\/a> in the NGINX Wiki, and we direct you there for a detailed discussion of the problems and how to avoid them. <\/p>\nIn general, the only directives you can always use safely within an if{}<\/code> block are return<\/code><\/a> and rewrite<\/code><\/a>. The following example uses if<\/code> to detect requests that include the X‑Test<\/code> header (but this can be any condition you want to test for). NGINX returns the 430<\/code> (Request<\/code> Header<\/code> Fields<\/code> Too<\/code> Large)<\/code><\/span> error, intercepts it at the named location @error_430<\/strong> and proxies the request to the upstream group named b<\/strong>.<\/p>\nlocation \/ {\r\n error_page 430 = @error_430;\r\n if ($http_x_test) {\r\n return 430; \r\n }\r\n\r\n proxy_pass http:\/\/a;\r\n}\r\n\r\nlocation @error_430 {\r\n proxy_pass b;\r\n}<\/code><\/pre>\nFor this and many other uses of if<\/code>, it\u2019s often possible to avoid the directive altogether. In the following example, when the request includes the X‑Test<\/code> header the map{}<\/code><\/a> block sets the $upstream_name<\/code> variable to b<\/code> and the request is proxied to the upstream group with that name. <\/p>\nmap $http_x_test $upstream_name {\r\n default \"b\";\r\n \"\" \"a\";\r\n}\r\n\r\n# ...\r\n\r\nlocation \/ {\r\n proxy_pass http:\/\/$upstream_name;\r\n}<\/code><\/pre>\nMistake 7: Excessive Health Checks<\/h2>\n
It is quite common to configure multiple virtual servers to proxy requests to the same upstream group (in other words, to include the identical proxy_pass<\/code><\/a> directive in multiple server{}<\/code><\/a> blocks). The mistake in this situation is to include a