Set Up vLLM for Production LLM Serving

Prerequisites & Hardware Requirements

vLLM is the fastest open source LLM inference engine available, and running it on a dedicated GPU server gives you the lowest possible latency with no virtualisation overhead. This guide walks you through a full production deployment — from install to systemd service to monitoring. For a comparison of inference engines, see our vLLM vs Ollama breakdown.

You need a server with NVIDIA drivers and CUDA already configured. If you haven’t done that yet, follow our PyTorch GPU server setup tutorial first.

Install vLLM

Create a dedicated Python environment and install vLLM with pip:

conda create -n vllm python=3.11 -y
conda activate vllm
pip install vllm

This pulls in PyTorch, Triton, and all CUDA dependencies automatically. On a server with local NVMe, the install completes in about two minutes. Verify the install:

python -c "import vllm; print(vllm.__version__)"

To serve a model, you first need to download it. For this guide we’ll use Meta’s Llama 3 8B Instruct:

pip install huggingface_hub
huggingface-cli download meta-llama/Meta-Llama-3-8B-Instruct --local-dir /models/llama-3-8b-instruct

For other model options, our best GPU for LLM inference guide covers which models fit on which GPUs.

Run the OpenAI-Compatible API Server

vLLM includes a built-in API server that is drop-in compatible with the OpenAI Chat Completions format. Start it with:

python -m vllm.entrypoints.openai.api_server \
  --model /models/llama-3-8b-instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 8192 \
  --dtype auto

Test the endpoint with curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/models/llama-3-8b-instruct",
    "messages": [{"role": "user", "content": "Explain GPU memory in one sentence."}],
    "max_tokens": 100
  }'

If you get a JSON response with a completion, vLLM is serving correctly. The server is now OpenAI-compatible, so any client library or application that works with the OpenAI API can point at your server instead.

Configure vLLM as a systemd Service

For production, vLLM must start automatically on boot and restart on failure. Create a systemd unit file:

sudo tee /etc/systemd/system/vllm.service > /dev/null <<'EOF'
[Unit]
Description=vLLM OpenAI API Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/root
Environment="PATH=/root/miniconda3/envs/vllm/bin:/usr/local/cuda-12.4/bin:/usr/bin"
ExecStart=/root/miniconda3/envs/vllm/bin/python -m vllm.entrypoints.openai.api_server \
  --model /models/llama-3-8b-instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 8192 \
  --dtype auto \
  --gpu-memory-utilization 0.90
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable vllm.service
sudo systemctl start vllm.service
sudo systemctl status vllm.service

Check the logs if anything goes wrong:

journalctl -u vllm.service -f --no-pager

The service will now survive reboots and automatically restart if vLLM crashes — essential for production vLLM hosting deployments.

Memory & Batching Optimisation

Default vLLM settings are conservative. For production throughput on dedicated hardware, tune these parameters:

An optimised ExecStart line for an RTX 3090 serving Llama 3 8B:

ExecStart=/root/miniconda3/envs/vllm/bin/python -m vllm.entrypoints.openai.api_server \
  --model /models/llama-3-8b-instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 8192 \
  --dtype auto \
  --gpu-memory-utilization 0.93 \
  --max-num-batched-tokens 32768 \
  --max-num-seqs 96 \
  --enable-prefix-caching \
  --swap-space 8

After editing the service file, reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart vllm.service

Prefix caching is particularly effective for chatbot workloads where system prompts repeat across requests. On our benchmarks, it improves throughput by 15-25% for multi-turn conversations. See the full data on our tokens per second benchmark page.

Monitoring & Health Checks

vLLM exposes a Prometheus-compatible metrics endpoint. Add a health check and metrics scrape to your setup:

# Health check — returns 200 when model is loaded
curl -s http://localhost:8000/health

# Prometheus metrics
curl -s http://localhost:8000/metrics | head -30

Key metrics to monitor:

• vllm:num_requests_running — active requests (watch for sustained max)
• vllm:num_requests_waiting — queued requests (should stay near zero)
• vllm:gpu_cache_usage_perc — KV cache utilisation (alert above 95%)
• vllm:avg_generation_throughput_toks_per_s — tokens per second

Set up a basic watchdog script that restarts vLLM if the health check fails:

sudo tee /usr/local/bin/vllm-watchdog.sh > /dev/null <<'SCRIPT'
#!/bin/bash
if ! curl -sf http://localhost:8000/health > /dev/null 2>&1; then
  echo "$(date): vLLM health check failed, restarting..." >> /var/log/vllm-watchdog.log
  systemctl restart vllm.service
fi
SCRIPT
sudo chmod +x /usr/local/bin/vllm-watchdog.sh

Add it to cron to run every minute:

(crontab -l 2>/dev/null; echo "* * * * * /usr/local/bin/vllm-watchdog.sh") | crontab -

For GPU temperature and utilisation monitoring, nvidia-smi can log to CSV:

nvidia-smi --query-gpu=timestamp,gpu_name,temperature.gpu,utilization.gpu,memory.used,memory.total \
  --format=csv -l 10 >> /var/log/gpu-stats.csv &

Load Balancing Multiple GPUs

For higher throughput, run multiple vLLM instances behind an Nginx load balancer. This is the standard pattern for scaling open source LLM hosting beyond a single GPU.

If your server has two GPUs, run one vLLM instance per GPU on different ports:

# Instance 1 on GPU 0
CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server \
  --model /models/llama-3-8b-instruct --port 8000 ...

# Instance 2 on GPU 1
CUDA_VISIBLE_DEVICES=1 python -m vllm.entrypoints.openai.api_server \
  --model /models/llama-3-8b-instruct --port 8001 ...

Add an Nginx upstream to distribute requests:

sudo tee /etc/nginx/sites-available/vllm-lb > /dev/null <<'NGINX'
upstream vllm_backends {
    least_conn;
    server 127.0.0.1:8000;
    server 127.0.0.1:8001;
}

server {
    listen 80;
    server_name your-server-ip;

    location /v1/ {
        proxy_pass http://vllm_backends;
        proxy_set_header Host $host;
        proxy_read_timeout 120s;
    }

    location /health {
        proxy_pass http://vllm_backends;
    }
}
NGINX
sudo ln -sf /etc/nginx/sites-available/vllm-lb /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

The least_conn directive routes each request to the instance with fewer active connections, which works well for variable-length LLM generation. For deploying specific models like DeepSeek on this stack, follow our deploy DeepSeek server guide.

What to Do Next

Your vLLM production deployment is now running with automatic restarts, monitoring, and load balancing. To go further:

• Estimate costs — Use our cost per 1M tokens analysis to forecast your monthly spend
• Self-host more models — Our self-host LLM guide covers model selection and deployment patterns
• Try Ollama for dev — For simpler local testing, Ollama hosting is easier to set up alongside your production vLLM stack
• More tutorials — Browse the tutorials category for additional GPU server setup guides

All GigaGPU dedicated GPU servers include technical support for vLLM configuration — reach out if you need help tuning for your specific model and traffic pattern.