Table of Contents
8B parameters has become the "default chatbot size" in open-weight LLMs. Llama 3.1 8B, Qwen 2.5 7B, Mistral 7B, Phi-3 Medium 14B (similar profile) — all sit in this band. This is the precise VRAM reference.
8B model needs ~16 GB at FP16, ~8 GB at FP8, ~5 GB at AWQ-INT4. Plus 2-10 GB of KV cache depending on context length and concurrent users. A 24 GB GPU is comfortable; 16 GB tight at FP16; 12 GB INT4-only.
Base weights by precision
| Precision | Bytes per param | 8B weights size |
|---|---|---|
| FP32 | 4 | 32 GB |
| BF16 / FP16 | 2 | ~16 GB |
| FP8 | 1 | ~8 GB |
| INT8 | 1 | ~8 GB |
| AWQ-INT4 | 0.5 + scales | ~5 GB |
| GGUF Q5_K_M | mixed | ~5.5 GB |
| GGUF Q4_K_M | mixed | ~5 GB |
KV cache scaling
For an 8B model with 32 attention heads and 128 head dim:
- Per-token KV (FP16): ~150 KB
- Per-token KV (FP8): ~75 KB
- 32K context, 1 stream, FP16 KV: ~5 GB
- 32K context, 16 streams, FP16 KV: ~80 GB (impractical)
- 32K context, 16 streams, FP8 KV: ~40 GB (still tight)
Specific 8B models
| Model | Params | FP16 | FP8 | INT4 |
|---|---|---|---|---|
| Llama 3.1 8B Instruct | 8B | 16 GB | 8 GB | 5 GB |
| Llama 3.2 3B | 3B | 6 GB | 3 GB | 2 GB |
| Mistral 7B Instruct v0.3 | 7B | 14 GB | 7 GB | 4.5 GB |
| Qwen 2.5 7B | 7B | 14 GB | 7 GB | 4.5 GB |
| Phi-3 Mini (3.8B) | 3.8B | 8 GB | 4 GB | 2.5 GB |
| Gemma 2 9B | 9B | 18 GB | 9 GB | 6 GB |
Verdict
For comfortable 8B deployment with concurrent batching, target a 16+ GB GPU at FP8 or 24+ GB at FP16. For latency-tight workloads even at FP8, aim for 32 GB to fit KV cache for high concurrency.
Bottom line
The right hardware tier for 8B models is RTX 5060 Ti / 5080 (FP8) or RTX 3090 / 5090 (FP16). See Llama 3 8B on 5060 Ti benchmark.
