How to Configure Nginx Reverse Proxy for AI Inference APIs

Running an AI inference server on a dedicated GPU server is only half the battle. You need a reverse proxy in front of it to handle TLS, manage connections, and provide a stable public endpoint. Nginx is the industry standard for this, and configuring it for AI workloads requires specific tuning for long-lived connections and streaming responses. This tutorial walks through a complete API hosting setup with Nginx fronting vLLM or Ollama.

Install Nginx on Ubuntu

Install the latest stable Nginx from the official repository:

# Add Nginx official repository
sudo apt update
sudo apt install -y curl gnupg2 ca-certificates lsb-release

echo "deb http://nginx.org/packages/ubuntu $(lsb_release -cs) nginx" \
    | sudo tee /etc/apt/sources.list.d/nginx.list

curl -fsSL https://nginx.org/keys/nginx_signing.key | sudo apt-key add -
sudo apt update
sudo apt install -y nginx

# Enable and start Nginx
sudo systemctl enable --now nginx
sudo systemctl status nginx

Before configuring Nginx, ensure your inference backend is running. Follow the vLLM production setup guide or Ollama setup guide to get your backend ready.

Basic Reverse Proxy Configuration

Create a server block that forwards requests to your inference backend. This example proxies to a vLLM server on port 8000:

# /etc/nginx/conf.d/ai-api.conf
server {
    listen 80;
    server_name api.yourdomain.com;

    # Max request body size for large prompts
    client_max_body_size 10m;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeouts for long inference requests
        proxy_connect_timeout 60s;
        proxy_send_timeout 300s;
        proxy_read_timeout 300s;
    }

    location /health {
        proxy_pass http://127.0.0.1:8000/health;
        access_log off;
    }
}

# Test and reload
sudo nginx -t
sudo systemctl reload nginx

TLS with Let’s Encrypt

Secure your API endpoint with a free TLS certificate from Let’s Encrypt using Certbot:

# Install Certbot
sudo apt install -y certbot python3-certbot-nginx

# Obtain and install certificate
sudo certbot --nginx -d api.yourdomain.com --non-interactive --agree-tos -m [email protected]

# Verify auto-renewal
sudo certbot renew --dry-run

# Check certificate
sudo certbot certificates

Certbot automatically modifies your Nginx config to redirect HTTP to HTTPS and adds the certificate paths. If you are running multiple inference endpoints on the same server, consider our production AI inference server guide for a multi-domain setup.

Streaming Response Support

LLM inference APIs return streaming responses using server-sent events (SSE). The default Nginx configuration buffers these, causing the client to receive the entire response at once instead of token by token. Disable buffering for streaming endpoints:

# Add to your location block in /etc/nginx/conf.d/ai-api.conf
location /v1/chat/completions {
    proxy_pass http://127.0.0.1:8000;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Connection '';
    proxy_http_version 1.1;

    # Critical for SSE streaming
    proxy_buffering off;
    proxy_cache off;
    chunked_transfer_encoding on;

    # Long timeout for streaming generation
    proxy_read_timeout 600s;
    proxy_send_timeout 600s;
}

location /v1/completions {
    proxy_pass http://127.0.0.1:8000;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Connection '';
    proxy_http_version 1.1;
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 600s;
}

Test streaming is working:

curl -N https://api.yourdomain.com/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "messages": [{"role": "user", "content": "Hello"}],
        "stream": true
    }'

Load Balancing Multiple GPU Backends

If you run multiple inference servers across a multi-GPU cluster, Nginx can distribute traffic across them:

# /etc/nginx/conf.d/ai-api.conf
upstream ai_backends {
    least_conn;  # Route to the backend with fewest active connections

    server 10.0.1.10:8000 weight=3;  # 4x RTX 6000 Pro server
    server 10.0.1.11:8000 weight=2;  # 2x RTX 6000 Pro server
    server 10.0.1.12:8000 weight=1;  # 1x RTX 6000 Pro server

    keepalive 32;
}

server {
    listen 443 ssl http2;
    server_name api.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/api.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.yourdomain.com/privkey.pem;

    location / {
        proxy_pass http://ai_backends;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffering off;
        proxy_read_timeout 600s;
    }
}

For auto-scaling based on GPU load, see our auto-scaling AI inference guide. To monitor backend health, follow the GPU monitoring tutorial.

Rate Limiting and Connection Control

Protect your GPU resources from abuse with Nginx rate limiting:

# Add to /etc/nginx/nginx.conf (http block)
http {
    # Rate limit zone: 10 requests per second per IP
    limit_req_zone $binary_remote_addr zone=ai_api:10m rate=10r/s;

    # Connection limit: max 20 concurrent connections per IP
    limit_conn_zone $binary_remote_addr zone=ai_conn:10m;
}

# Add to your server block
location /v1/ {
    limit_req zone=ai_api burst=20 nodelay;
    limit_conn ai_conn 20;
    limit_req_status 429;

    proxy_pass http://127.0.0.1:8000;
    proxy_buffering off;
    proxy_read_timeout 600s;
}

For API key authentication on top of rate limiting, see the secure AI inference API guide. This is essential for any public-facing LLM hosting endpoint.

Full Production Configuration

Here is a complete production-ready Nginx configuration combining all the above:

# /etc/nginx/conf.d/ai-api.conf
upstream vllm_backend {
    least_conn;
    server 127.0.0.1:8000;
    keepalive 32;
}

server {
    listen 80;
    server_name api.yourdomain.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name api.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/api.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.yourdomain.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    client_max_body_size 10m;

    # API endpoints with streaming support
    location /v1/ {
        limit_req zone=ai_api burst=20 nodelay;
        limit_conn ai_conn 20;

        proxy_pass http://vllm_backend;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Connection "";

        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
    }

    # Health check (no rate limiting)
    location /health {
        proxy_pass http://vllm_backend/health;
        access_log off;
    }

    # Metrics (restrict to internal)
    location /metrics {
        allow 10.0.0.0/8;
        allow 127.0.0.1;
        deny all;
        proxy_pass http://vllm_backend/metrics;
    }
}

# Test and apply
sudo nginx -t && sudo systemctl reload nginx

# Verify HTTPS is working
curl -v https://api.yourdomain.com/health

Browse more deployment guides in the tutorials section, and explore the private AI infrastructure guide for end-to-end security architecture.