Benchmark Overview
GPU VRAM is the most constrained resource in LLM inference. Understanding exactly how memory is consumed, split between model weights, KV cache, and system overhead, determines how many users you can serve and which models fit on your hardware. We measured real VRAM utilisation across model sizes and user counts on dedicated GPU hosting.
Test Configuration
Models: Llama 3 8B, Llama 3 70B (FP16 and INT4). GPU: RTX 6000 Pro 96 GB. Engine: vLLM with gpu_memory_utilization=0.90. Context: 4K tokens per request. Measured at 0, 1, 10, and 25 concurrent users via nvidia-smi. See vLLM production guide for memory tuning.
VRAM Utilisation Breakdown (RTX 6000 Pro 96 GB)
| Model | Model Weights | System/CUDA Overhead | KV Cache (10 users, 4K) | Total Used |
|---|---|---|---|---|
| Llama 3 8B FP16 | 16 GB | 1.5 GB | 3.2 GB | 20.7 GB |
| Llama 3 8B INT4 | 5 GB | 1.2 GB | 3.2 GB | 9.4 GB |
| Llama 3 70B FP16 | 140 GB (2 GPUs) | 3 GB | 20 GB | 163 GB |
| Llama 3 70B INT4 | 38 GB | 2 GB | 20 GB | 60 GB |
VRAM Growth with Concurrent Users (70B INT4, 4K Context)
| Concurrent Users | Model Weights | KV Cache Total | Total VRAM | Remaining (80GB) |
|---|---|---|---|---|
| 0 (idle) | 38 GB | 0 GB | 40 GB | 40 GB |
| 1 | 38 GB | 2 GB | 42 GB | 38 GB |
| 5 | 38 GB | 10 GB | 50 GB | 30 GB |
| 10 | 38 GB | 20 GB | 60 GB | 20 GB |
| 20 | 38 GB | 40 GB | 80 GB | 0 GB (OOM) |
Memory Component Analysis
Model weights are the static, predictable component. INT4 quantisation reduces them by 75%. KV cache is the dynamic, variable component that grows with both user count and context length. At 4K context with FP16 KV cache, each user consumes approximately 2GB for a 70B model. System overhead (CUDA context, vLLM buffers) is 1.5-3GB and remains constant. See GPU memory guides and benchmarks for capacity planning.
vLLM’s PagedAttention allocates the remaining VRAM after model loading as a KV cache pool. The gpu_memory_utilization parameter controls what percentage of total VRAM is available. Setting it to 0.90 reserves 8GB on an 80GB GPU for safety. For Ollama, memory management is less configurable.
Optimisation Strategies
Enable FP8 KV cache to halve per-user VRAM (1GB instead of 2GB at 4K context for 70B models). This doubles your maximum concurrent users. Quantise the model to INT4 to free 100+ GB of VRAM on 70B models. Limit maximum context length to the minimum your application needs. Deploy on multi-GPU clusters when single-GPU VRAM is insufficient.
Recommendations
Calculate your VRAM budget as: model weights + (KV cache per user x max concurrent users) + 3GB overhead. If this exceeds your GPU VRAM, either quantise the model, compress the KV cache, reduce max users, or add GPUs. Deploy on GigaGPU dedicated servers with private hosting. Visit the benchmarks section and LLM hosting guide for detailed sizing.
