vLLM + Nginx: Fixing Proxy Timeout Issues

The Timeout Symptoms

You put Nginx in front of your vLLM server and requests start failing with:

504 Gateway Timeout

Or streaming responses cut off mid-generation. Or the first token takes 60 seconds even though direct requests to vLLM respond in under a second. Nginx is a solid reverse proxy, but its defaults are designed for web pages that respond in milliseconds — not LLM inference that can run for tens of seconds generating long outputs.

Why Nginx Times Out on LLM Requests

Three Nginx defaults collide with LLM inference behaviour:

1. proxy_read_timeout (60s default): If vLLM takes longer than 60 seconds to produce the first byte (during prefill of a long prompt), Nginx sends a 504.
2. proxy_buffering (on by default): Nginx buffers the entire response before sending it to the client. For streaming SSE responses, this prevents tokens from arriving incrementally.
3. proxy_connect_timeout (60s default): If vLLM is busy and its accept queue is full, new connections time out.

The Fix: Nginx Configuration for vLLM

upstream vllm_backend {
    server 127.0.0.1:8000;
    keepalive 32;
}

server {
    listen 443 ssl http2;
    server_name api.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/api.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.yourdomain.com/privkey.pem;

    location /v1/ {
        proxy_pass http://vllm_backend;

        # Timeout settings for LLM inference
        proxy_connect_timeout 300s;
        proxy_send_timeout 300s;
        proxy_read_timeout 300s;

        # Critical: disable buffering for SSE streaming
        proxy_buffering off;
        proxy_cache off;

        # SSE-specific headers
        proxy_set_header Connection '';
        proxy_http_version 1.1;
        chunked_transfer_encoding off;

        # Forward client info
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Request size for long prompts
        client_max_body_size 10m;
    }
}

Key Settings Explained

proxy_buffering off

This is the single most important setting. Without it, Nginx collects the entire streamed response before forwarding it, defeating the purpose of server-sent events. Clients would wait for the full generation to complete before seeing any tokens.

proxy_read_timeout 300s

Five minutes allows for long-context prefill and lengthy generations. For production GPU servers processing large documents, you may need even more. Adjust based on your maximum expected generation time.

keepalive 32

Maintains persistent connections to vLLM, avoiding TCP handshake overhead per request. This improves throughput noticeably at high request rates.

Testing the Fix

# Reload Nginx
sudo nginx -t && sudo nginx -s reload

# Test non-streaming request
curl -X POST https://api.yourdomain.com/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Meta-Llama-3.1-8B-Instruct","prompt":"Hello","max_tokens":100}'

# Test streaming request (should show tokens incrementally)
curl -X POST https://api.yourdomain.com/v1/completions \
  -H "Content-Type: application/json" \
  -N \
  -d '{"model":"meta-llama/Meta-Llama-3.1-8B-Instruct","prompt":"Write a long story","max_tokens":500,"stream":true}'

The streaming test should display tokens as they arrive, not all at once. If tokens still arrive in a burst, check that no CDN or downstream proxy is buffering the response.

Load Balancing Multiple vLLM Instances

For multi-GPU servers running separate vLLM instances per GPU:

upstream vllm_backend {
    least_conn;
    server 127.0.0.1:8000;  # GPU 0
    server 127.0.0.1:8001;  # GPU 1
    keepalive 64;
}

least_conn routes requests to the instance with fewest active connections, which balances load better than round-robin for variable-length LLM requests.

Security and Monitoring

With the proxy working, add rate limiting and access control:

limit_req_zone $binary_remote_addr zone=llm:10m rate=10r/s;

location /v1/ {
    limit_req zone=llm burst=20 nodelay;
    # ... rest of proxy config ...
}

Our API security guide covers API key authentication, IP whitelisting, and additional hardening. For SSL setup, see the tutorials section for Let’s Encrypt configuration. Monitor both Nginx access logs and vLLM metrics with your monitoring setup to track latency at each layer.