vLLM max-model-len and GPU Memory Utilisation Tradeoff

The most common over-provisioning mistake on a dedicated GPU server is setting --max-model-len too high. Every concurrent sequence reserves KV cache up to that length. If you set 32,000 and your real prompts average 2,000, you are budgeting for ghosts and starving live traffic.

Contents

• What each flag controls
• The KV cache math
• How to pick the right value
• Real examples

What Each Flag Does

--max-model-len: the longest total sequence (prompt + output) any single request can use. Caps context window at serve time. Per-sequence KV cache allocation scales with this.

--gpu-memory-utilization: fraction of VRAM vLLM uses in total. Default 0.9. The rest is reserved for CUDA context, PyTorch allocator overhead, and any other processes.

The KV Cache Math

KV cache per token is roughly 2 × num_layers × hidden_size × 2 bytes for FP16. For Llama 3 8B that is ~128 KB per token. At max-model-len=8192, each concurrent sequence pre-allocates up to 1 GB. At 32,768 it is 4 GB. On a 16 GB 4060 Ti with the model taking 8 GB, you have ~7 GB of KV cache capacity – that is 7 concurrent 8k sequences or only 1.75 concurrent 32k sequences.

Picking the Value

Measure actual input and output lengths in production. Pick max-model-len to cover the 95th percentile, not the theoretical max. Most chat workloads live under 4,000 total tokens. RAG sits at 8,000-16,000. Document processing sits at 32,000+.

Set gpu-memory-utilization as high as you can without crashing. 0.92 is common. 0.95 works if nothing else lives on the card.

Real Examples

Llama 3 8B INT8 on a 5090 (32 GB):

--max-model-len 8192 --gpu-memory-utilization 0.92
# ~18 GB available for KV cache
# ~18 concurrent 8k sequences

Llama 3 70B INT4 on a 6000 Pro (96 GB):

--max-model-len 16384 --gpu-memory-utilization 0.93
# ~48 GB available for KV cache
# ~24 concurrent 16k sequences

See continuous batching tuning and block size tuning.