vLLM Out of Memory: How to Fix KV Cache OOM

The vLLM OOM Error You Are Seeing

You launch vLLM and it crashes during startup or shortly after with:

ValueError: The model's max seq len (32768) is larger than the maximum number of
tokens that can be stored in KV cache (2048). Try increasing `gpu_memory_utilization`
or decreasing `max_model_len` when initializing the engine.

Or, during serving:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.50 GiB

vLLM’s architecture pre-allocates a KV cache at startup that consumes a fixed portion of VRAM. If the model weights plus the KV cache exceed your GPU’s total memory, the engine cannot start. This is the most common issue on vLLM hosting setups.

Understanding vLLM Memory Layout

vLLM divides GPU memory into three regions:

• Model weights: Fixed cost. A 7B parameter model in FP16 needs roughly 14 GB.
• KV cache: Stores attention key-value pairs for all active sequences. Size depends on max_model_len, number of layers, number of heads, and head dimension.
• Overhead: CUDA context, temporary activations, and fragmentation buffer.

The gpu_memory_utilization parameter (default 0.90) tells vLLM what fraction of total VRAM it may use. The KV cache fills whatever space remains after loading the model. If there is not enough room for even a minimal KV cache, vLLM refuses to start.

Fix 1: Reduce max_model_len

The fastest fix. If your use case does not need the full context window:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90

Cutting max-model-len from 32768 to 4096 reduces KV cache memory by roughly 8x. For chatbot workloads where most conversations stay under 4K tokens, this is the right trade-off.

Fix 2: Increase gpu_memory_utilization

--gpu-memory-utilization 0.95

This gives vLLM 95 percent of VRAM instead of 90 percent. The extra 5 percent can make the difference between a functional KV cache and an OOM. Be cautious going above 0.95 — the CUDA context needs some headroom, and going too high causes sporadic OOM during spiky workloads on your GPU server.

Fix 3: Use a Quantized Model

Quantization shrinks model weights, freeing more VRAM for the KV cache:

# Use a GPTQ 4-bit quantized model
python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Meta-Llama-3.1-8B-Instruct-GPTQ \
  --quantization gptq \
  --max-model-len 16384

A 4-bit quantized 8B model uses roughly 4 GB instead of 14 GB, tripling the available space for KV cache. Quality impact is minimal for instruction-following tasks. Our vLLM memory optimization guide covers quantization methods in detail.

Fix 4: Spread Across Multiple GPUs

If your dedicated server has multiple GPUs:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 8192

Tensor parallelism distributes model weights and KV cache across GPUs. A 70B model that cannot fit on a single 24 GB card runs comfortably across four cards with room for a reasonable context length.

GPU Sizing Guide for vLLM

Model Size → Minimum VRAM (FP16, 4K context)
7-8B       → 16 GB (RTX 5080/5090)
13B        → 28 GB (RTX 6000 Pro or 2x RTX 5090)
34B        → 70 GB (RTX 6000 Pro 96 GB or 4x RTX 5090)
70B        → 140 GB (2x RTX 6000 Pro 96 GB or 4x RTX 6000 Pro)

These are minimum figures. Production deployments with high concurrency need more KV cache space. Browse GigaGPU’s GPU configurations to find hardware that matches your model.

Verifying the Fix

After adjusting parameters, confirm vLLM starts and serves requests:

# Check startup logs for KV cache allocation
# Look for: "# GPU blocks: XXXX, # CPU blocks: XXXX"
# More GPU blocks = more concurrent capacity

# Test with a request
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Meta-Llama-3.1-8B-Instruct", "prompt": "Hello", "max_tokens": 50}'

Monitor VRAM during load testing with GPU monitoring tools. For production deployment, follow our vLLM production setup guide and protect endpoints with API security.