Table of Contents
Llama 3 70B INT4 is the most-asked-about deployment. Here is the precise sizing.
TL;DR
Llama 3 70B AWQ-INT4 weights: ~40 GB. Plus 5-20 GB KV cache by context. Total: 45-60 GB for typical production. Fits: 2× RTX 5090 cluster (64 GB), single RTX 6000 Pro 96 GB. Does NOT fit a single 32 GB card.
The math
- 70B params × 0.5 bytes (INT4) = 35 GB
- + AWQ scales / zero points: ~5 GB → 40 GB total weights
- + KV cache: depends on context
- + 1-2 GB activations
Configurations
| Config | Combined VRAM | Fits Llama 3 70B INT4? |
|---|---|---|
| Single RTX 5090 32 GB | 32 GB | No (over by 8 GB) |
| Single RTX 6000 Pro 96 GB | 96 GB | Yes, 50 GB headroom |
| 2× RTX 5090 (TP=2) | 64 GB | Yes, ~20 GB headroom |
| Single A100 80 GB | 80 GB | Yes |
| 2× A100 40 GB (TP=2) | 80 GB | Yes |
Verdict
For Llama 3 70B INT4 single-server: 2× RTX 5090 (cheapest, £399/mo) or single RTX 6000 Pro (simplest, £899/mo).
Bottom line
Single 32 GB card does not fit. See can RTX 5090 run Llama 3 70B INT4? for the full answer.
