vLLM API Returns 500 Error: Debug Guide

The 500 Error from vLLM

Your vLLM server is running and accepting connections, but some or all requests return:

HTTP/1.1 500 Internal Server Error
{"object":"error","message":"RuntimeError","type":"server_error","code":500}

A 500 from vLLM means something went wrong during inference — not during HTTP handling. The request reached the engine, but the engine could not produce a response. The HTTP error alone gives almost no diagnostic information. The real clues are in the server-side logs.

Getting the Real Error Message

The first step is always to check vLLM’s server output. If you launched vLLM in the foreground, the error is printed to stderr. If it runs as a systemd service on your GPU server:

journalctl -u vllm.service -n 200 --no-pager | grep -i error

Common errors behind the 500 response:

• RuntimeError: CUDA error: device-side assert triggered — an input triggered an assertion in a CUDA kernel.
• torch.cuda.OutOfMemoryError — the request exceeded available KV cache capacity.
• ValueError: token_ids must not be empty — the tokenizer produced empty output for the input.
• KeyError: 'choices' — an internal formatting failure in the response builder.

Fix: CUDA Errors During Inference

If the log shows a CUDA error, the vLLM process is likely in a corrupted state. A single bad request can poison the entire CUDA context, causing all subsequent requests to fail.

# Restart the vLLM server
sudo systemctl restart vllm.service

# Or kill and relaunch manually
pkill -f "vllm.entrypoints"
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --max-model-len 4096 &

To prevent this, add input validation in your application layer before forwarding requests to vLLM. Extreme token counts, malformed prompts, or unsupported special characters can trigger these errors. Our API security guide covers input validation patterns.

Fix: OOM During Active Serving

If vLLM starts fine but 500s appear under load, the KV cache is being exhausted by concurrent requests:

# Reduce concurrent capacity to prevent OOM
--max-num-seqs 64 \
--max-model-len 2048 \
--gpu-memory-utilization 0.95

Lower max-num-seqs limits concurrency, preventing the cache from overflowing. See our vLLM memory optimization guide for detailed capacity planning.

Fix: Tokenizer and Input Formatting Errors

When the 500 error is intermittent and correlates with specific inputs, the tokenizer may be failing on certain content:

# Test tokenization directly
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")

test_input = "Your problematic prompt here"
tokens = tok.encode(test_input)
print(f"Token count: {len(tokens)}")
print(f"Max token ID: {max(tokens)}")

If max(tokens) exceeds the model’s vocabulary size, that request will crash the engine. If the token count exceeds max-model-len, vLLM should return a 400 error, but edge cases can trigger 500s instead.

Fix: Chat Template Mismatches

Using the /v1/chat/completions endpoint requires a properly configured chat template. If the model does not have one:

--chat-template ./my_template.jinja

Without a chat template, vLLM may produce 500 errors when chat-format requests arrive. Test with the completions endpoint first to isolate template issues from model issues.

Automatic Recovery from 500 Errors

For production GPU servers, configure automatic restart:

# In your systemd service file
[Service]
Restart=always
RestartSec=10
WatchdogSec=300

And add a health check script:

#!/bin/bash
response=$(curl -s -o /dev/null -w "%{http_code}" \
  -X POST http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Meta-Llama-3.1-8B-Instruct","prompt":"Hi","max_tokens":1}')

if [ "$response" != "200" ]; then
    sudo systemctl restart vllm.service
    echo "vLLM restarted due to health check failure"
fi

Follow our vLLM production setup guide for the complete systemd configuration. Monitor error rates with GPU server monitoring to catch degradation early. For workloads requiring higher reliability, consider load-balanced multi-instance deployments where a failing instance is automatically drained.