How Much VRAM Do You Actually Need? (Cost Optimisation Guide)

VRAM Requirements: The Basics

The single biggest factor in GPU hosting cost is VRAM. Choose too much and you overpay. Choose too little and your model will not fit. This guide helps you find the sweet spot: the minimum VRAM that runs your workload efficiently, so you can pick the cheapest dedicated GPU server that gets the job done.

The rule of thumb: a model needs approximately 2x its parameter count in GB at FP16 precision, plus overhead for KV cache and inference. A 7B model needs ~14GB, a 70B model needs ~140GB. But with quantisation, you can slash these requirements dramatically.

VRAM by Model Size

Notice the cost jump between model sizes. Going from 7B to 70B increases your hosting cost from $99 to $149-$599. That is why right-sizing your model is the single most impactful cost optimisation you can make. Use our cost per million tokens calculator to compare.

Quantisation: Cut VRAM Costs in Half

Quantisation reduces model precision from FP16 (16-bit) to INT8 (8-bit) or INT4 (4-bit). This directly translates to lower VRAM requirements and therefore cheaper GPU servers:

INT8 quantisation saves $300/month (50% cost reduction) with negligible quality impact. INT4 saves $450/month (75% reduction) with minor quality loss acceptable for many production use cases.

For most workloads, INT8 is the sweet spot. Reserve FP16 for tasks requiring maximum accuracy (medical, legal, financial). See detailed quantisation benchmarks in our best GPU for inference guide.

KV Cache: The Hidden VRAM Cost

Model weights are only part of the VRAM equation. The KV (key-value) cache stores attention state for each active request and grows with:

• Sequence length: longer conversations or documents use more KV cache
• Concurrent users: each simultaneous request needs its own KV cache
• Model architecture: models with more attention heads use more KV cache

Rule of thumb: reserve 20-40% of VRAM beyond model weights for KV cache and overhead. A 70B INT8 model uses ~70GB for weights but needs ~85-90GB total for comfortable production operation.

This is why an RTX 6000 Pro 96 GB (not 40GB) is recommended for 70B INT8: the extra 40GB provides ample KV cache room for concurrent users. vLLM’s PagedAttention optimises KV cache memory, maximising the number of concurrent requests your GPU can handle.

GPU Cost Tiers: What You Get at Each Price

See our cheapest GPU for AI inference guide and RTX 3090 vs RTX 5090 comparison for detailed hardware analysis.

Match Your Workload to the Right GPU

Cost Optimisation Tips

1. Start with the smallest model that meets your quality bar. A fine-tuned 7B model often outperforms a generic 70B model on specific tasks.
2. Use INT8 quantisation by default. The quality loss is negligible for most applications and it halves your VRAM (and cost).
3. Run multiple small models on one GPU. A 24GB GPU can host a 7B chat model AND an embedding model simultaneously.
4. Use vLLM for production. Its PagedAttention mechanism maximises concurrent users per GB of VRAM.
5. Consider MoE models. DeepSeek-V2 has 236B parameters but only activates 21B, giving large-model quality at small-model VRAM usage.
6. Benchmark before committing. Use our tokens per second benchmark to verify throughput.

For per-model cost breakdowns, see our guides for LLaMA 3, DeepSeek, Mistral, Qwen, and Phi-3. For the complete self-hosting economics, read our complete cost guide and ROI analysis.