How Context Length Affects VRAM: A Visual Guide

Why Context Length Eats VRAM

Every token in your prompt and output needs dedicated memory on your dedicated GPU server. The model weights are fixed — a 7B model always uses ~14 GB at FP16 regardless of whether you send 100 or 100,000 tokens. What changes is the KV cache: the key-value pairs stored for every token across every attention layer. This guide compares VRAM scaling across five popular open-weight LLMs and shows you exactly where the memory goes.

Understanding this relationship is essential for anyone deploying open-source LLMs in production, where you must balance context window size, concurrency, and hardware cost.

The VRAM Formula

Total VRAM for a single request can be approximated as:

Total VRAM = Model Weights + KV Cache + Runtime Overhead

The KV cache for a standard multi-head attention (MHA) model is calculated as:

KV Cache (bytes) = 2 x num_layers x num_kv_heads x head_dim x seq_length x bytes_per_element

The “2” accounts for both keys and values. Models using Grouped-Query Attention (GQA) have fewer KV heads than attention heads, reducing cache size proportionally. For a full explanation of KV cache mechanics, see our KV cache explainer.

For each additional concurrent request, multiply the KV cache allocation by the number of active users. A model that runs fine for a single user at 32K context may need double the VRAM to serve two simultaneous requests.

Cross-Model VRAM Comparison

The table below compares total VRAM at FP16 for five popular models across context lengths from 4K to 128K tokens (single request, batch size 1). All values include model weights, KV cache, and ~0.5 GB runtime overhead.

Key takeaways from this comparison:

• Mistral 7B stays flat thanks to sliding window attention. VRAM is identical from 4K to 32K.
• LLaMA 3 8B and Qwen 2.5 7B scale linearly — 128K context nearly doubles the FP16 VRAM footprint.
• LLaMA 3 70B KV cache growth adds over 150 GB between 4K and 128K.
• DeepSeek V3 benefits from MLA compression — context growth adds only ~45 GB across the full range despite the enormous model size.

Models That Break the Pattern

Not all models follow the standard linear KV cache scaling:

• Sliding window attention (Mistral, Mixtral): KV cache is capped at the window size per layer, so memory remains constant regardless of total context length.
• Multi-head Latent Attention (DeepSeek V3/R1): compresses KV representations into a latent space, reducing per-token cache by 60-70%.
• Multi-Query Attention (older models): uses a single KV head shared across all attention heads, dramatically reducing cache size but potentially affecting quality.
• GQA (LLaMA 3, Qwen 2.5): a middle ground — fewer KV heads than attention heads, reducing cache by 4-8x compared to full MHA while preserving quality.

When evaluating a new model for deployment, check its attention architecture. It determines how aggressively VRAM scales with context length.

Strategies to Reduce Context VRAM

Several techniques can extend your effective context window on existing hardware:

• Model quantisation (INT4/INT8): reduces weight memory by 50-75%, freeing VRAM for KV cache. See our GPTQ vs AWQ vs GGUF guide for format selection.
• FP8 KV cache: vLLM supports quantised KV cache, halving cache memory with negligible quality loss.
• FlashAttention: eliminates the O(n squared) peak memory spike during attention computation — essential above 16K tokens.
• PagedAttention: eliminates memory fragmentation from variable-length requests. Built into vLLM and dramatically improves effective memory utilisation.
• Multi-GPU parallelism: tensor parallelism distributes both weights and KV cache across GPUs, linearly scaling available VRAM.
• Continuous batching: dynamically manages KV cache allocation across requests, preventing memory waste from padding.

Conclusion

Context length is the hidden multiplier in LLM VRAM budgets. A model that fits comfortably on a single GPU at 4K tokens may need a multi-GPU cluster at 128K. Always plan for your maximum context length, multiply by peak concurrency, and apply quantisation to keep costs manageable. For speed data at each configuration, consult our best GPU for LLM inference benchmarks and the cost per million tokens calculator.