Fix Nginx "upstream prematurely closed connection" While Reading Response Header
Quick Answer
- The "upstream prematurely closed connection while reading response header from upstream" error means Nginx's connection to your backend server was terminated before the full response header could be read, typically due to timeouts, backend crashes, or buffer exhaustion.
- First check: verify your upstream server is healthy and responding, then increase
proxy_read_timeout(start at 300s) and ensureproxy_buffer_sizeis large enough for your response headers. - Minimal fix: add
proxy_read_timeout 300;andproxy_buffer_size 8k;to your location block, then restart Nginx. - This error commonly appears with slow backend APIs, large response headers, or upstream servers that close idle connections prematurely.
What Problem It Solves
When Nginx acts as a reverse proxy, it maintains a connection to your upstream server (application server, API, or database). The "upstream prematurely closed connection" error occurs when that connection is severed while Nginx is still reading the response header from the upstream. This can happen for several reasons:
- The upstream server times out or crashes mid-response
- Response headers exceed the configured buffer size
- The upstream server closes idle connections too aggressively
- Network issues between Nginx and the upstream
This guide provides practical configuration adjustments to prevent this error and ensure reliable proxy communication.
Root Cause Analysis
The error message "upstream prematurely closed connection while reading response header from upstream" specifically indicates the failure occurs during the header reading phase, not during body transmission. Common root causes:
- Timeout expiration: The upstream takes longer than
proxy_read_timeoutto send the first byte of the response header - Buffer overflow: The response header exceeds
proxy_buffer_size, causing Nginx to discard the connection - Upstream server crash: The backend process dies or restarts while generating the response
- Connection reuse issues: Keepalive connections between Nginx and upstream are closed by the upstream while Nginx still considers them valid
Minimal Working Configuration
Create or modify your Nginx configuration file (typically /etc/nginx/nginx.conf or a site-specific file in /etc/nginx/sites-available/):
NGINXhttp { upstream backend { server 127.0.0.1:8080; keepalive 32; } server { listen 80; server_name example.com; location / { proxy_pass http://backend; proxy_http_version 1.1; proxy_set_header Connection ""; # Critical timeout settings proxy_connect_timeout 60s; proxy_read_timeout 300s; proxy_send_timeout 60s; # Buffer settings for large headers proxy_buffer_size 8k; proxy_buffers 8 8k; proxy_busy_buffers_size 16k; # Large client headers large_client_header_buffers 4 16k; } } }
After making changes, test and reload:
BASHnginx -t systemctl reload nginx
Parameters and Environment Variables
| Parameter | Default | Recommended | Description |
|---|---|---|---|
proxy_read_timeout | 60s | 300s | Maximum time Nginx waits for upstream to send data. Increase for slow backends. |
proxy_connect_timeout | 60s | 60s-120s | Time to establish connection with upstream. Increase if upstream is slow to accept. |
proxy_send_timeout | 60s | 60s-120s | Time to send request data to upstream. |
proxy_buffer_size | 4k | 8k-16k | Buffer for reading the first part of upstream response (headers). Increase if headers are large. |
proxy_buffers | 8 4k | 8 8k | Number and size of buffers for reading upstream response. |
proxy_busy_buffers_size | 8k | 16k | Limit for buffers that can be busy sending to client while response is not fully read. |
large_client_header_buffers | 4 8k | 4 16k | Max number and size of buffers for large client request headers. |
Common Errors and Fixes
Error: "upstream prematurely closed connection"
Solution: Check upstream server health first. Then increase proxy_read_timeout and proxy_buffer_size:
NGINXlocation /api/ { proxy_pass http://backend; proxy_read_timeout 300s; proxy_buffer_size 16k; proxy_buffers 16 16k; }
Error: "Connection timeout"
Solution: Increase both proxy_connect_timeout and proxy_read_timeout:
NGINXproxy_connect_timeout 120s; proxy_read_timeout 600s;
Error: "Too many open files"
Solution: Increase system file descriptor limits:
BASH# Check current limit ulimit -n # Increase system-wide limit echo "fs.file-max = 100000" >> /etc/sysctl.conf sysctl -p # Increase Nginx worker limit echo "worker_rlimit_nofile 65535;" >> /etc/nginx/nginx.conf
Error: Upstream server health check failure
Solution: Implement health checks and ensure upstream is reachable:
BASH# Test upstream connectivity curl -I http://127.0.0.1:8080/health # Check Nginx error logs tail -f /var/log/nginx/error.log
Production Notes and Security Checks
-
Connection limits: Monitor Nginx worker connections with
nginx -Vand adjustworker_connectionsin the events block:NGINXevents { worker_connections 4096; multi_accept on; } -
File descriptor limits: Ensure OS limits are sufficient for expected concurrent connections:
BASH# Check current limits cat /proc/sys/fs/file-max ulimit -n -
TLS encryption: Always use HTTPS between Nginx and upstream in production:
NGINXupstream backend { server 127.0.0.1:8443; } location / { proxy_pass https://backend; proxy_ssl_verify on; proxy_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt; } -
Keepalive optimization: Enable keepalive connections to reduce connection overhead:
NGINXupstream backend { server 127.0.0.1:8080; keepalive 32; keepalive_requests 100; keepalive_timeout 60s; } -
Monitoring: Enable Nginx status module for real-time metrics:
NGINXlocation /nginx_status { stub_status on; allow 127.0.0.1; deny all; }
FAQ
Q: How do I optimize Nginx for high concurrent requests?
A: Increase proxy_buffers and proxy_buffer_size, adjust timeout settings, and optimize OS network parameters. Also increase worker_connections and file descriptor limits. For example: proxy_buffers 16 16k; proxy_buffer_size 16k; worker_connections 4096;.
Q: What should I set for proxy_read_timeout?
A: Start with 300 seconds and adjust based on your upstream's actual response time. Monitor error logs and increase if you still see timeout errors. For very slow APIs, values up to 600s may be necessary.
Q: How can I monitor Nginx performance?
A: Use the ngx_http_stub_status_module to track active connections, requests, and response times. Combine with external tools like Prometheus and Grafana for comprehensive monitoring. Enable access and error logs with appropriate log formats.