Table of Contents
Llama 3.3 70B (December 2024) is Meta’s text-only refresh of the 70B class — improved reasoning over 3.1, same architecture, same VRAM profile. This is the deployment runbook.
Three viable single-server configs: 1× RTX 6000 Pro 96 GB FP8 (cleanest, £899/mo), 2× RTX 5090 INT4 (cheapest, £899/mo), or 2× A100 80 GB FP16 (reference quality). For most production workloads, the 6000 Pro is the right pick.
Three single-server hardware options
| Config | Combined VRAM | Llama 3.3 70B precision | Monthly |
|---|---|---|---|
| 1× RTX 6000 Pro 96 GB | 96 GB ECC | FP8 single-card | £399 |
| 2× RTX 5090 (TP=2) | 64 GB combined | AWQ-INT4 split | £899 |
| 2× A100 80 GB SXM4 | 160 GB + NVLink | FP16 split | POA |
| 1× RTX 5090 (INT3 only) | 32 GB | INT3 only — quality risk | £399 |
vLLM config for each
RTX 6000 Pro single-card FP8:
vllm serve neuralmagic/Llama-3.3-70B-Instruct-FP8-dynamic \
--quantization fp8 \
--max-model-len 32768 \
--gpu-memory-utilization 0.92 \
--enable-prefix-caching2× RTX 5090 cluster INT4:
CUDA_VISIBLE_DEVICES=0,1 vllm serve hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4 \
--tensor-parallel-size 2 \
--quantization awq_marlin \
--max-model-len 32768Benchmarks
| Config | Aggregate tok/s | Single-stream | TTFT median | Cost per 1M |
|---|---|---|---|---|
| 1× RTX 6000 Pro FP8 | 220 | 32 | 320 ms | £1.61 |
| 2× RTX 5090 INT4 | 240 | 28 | 380 ms | £0.95 |
| 2× A100 80 GB FP16 | 180 | 26 | 340 ms | POA |
vs Llama 3.1 70B
Same hardware profile. Llama 3.3 has stronger reasoning on hard tasks (~3-5% improvement on MATH, GSM8K) and identical inference cost. For new deployments, prefer 3.3 unless you’ve already validated 3.1 in your eval harness.
Verdict
For most teams the RTX 6000 Pro 96 GB at FP8 is the right Llama 3.3 70B host: single card, full FP8 quality, comfortable headroom. For cost-anchored deployments, the 2× RTX 5090 cluster is meaningfully cheaper at the cost of slightly more operational complexity.
Bottom line
Single-card 70B = 6000 Pro. Multi-card 70B = 2× 5090. For the broader Llama 3 sizing see best GPU for Llama.
