Table of Contents
Two of the most-asked-about server configurations for serving 70B-class models on consumer-Blackwell hardware. RTX 6000 Pro 96 GB is one card with all the VRAM. 2× RTX 5090 is two cards with 64 GB combined. Cost is similar (£399 vs £399/mo). This page is the head-to-head.
The RTX 6000 Pro 96 GB is simpler to operate (single card, no NCCL) and the right pick for compliance-bound workloads needing ECC. The 2× RTX 5090 cluster is faster on most 70B workloads and cheaper monthly. For new deployments without specific compliance requirements, 2× 5090 wins on cost-per-token.
What each setup looks like
RTX 6000 Pro 96 GB single card
One Blackwell card, 96 GB GDDR7 ECC. Llama 3 70B at FP8 fits with comfortable KV cache headroom. Single vLLM process, no tensor parallelism, no NCCL.
2× RTX 5090 cluster
Two Blackwell cards, 64 GB combined GDDR7 (no ECC). PCIe Gen 4 x16 to each card, no NVLink. Llama 3 70B AWQ-INT4 with tensor-parallel-size=2.
Real benchmarks
| Workload | RTX 6000 Pro FP8 | 2× RTX 5090 INT4 (TP=2) | Winner |
|---|---|---|---|
| Llama 3.3 70B aggregate tok/s | 220 | 240 | 2× 5090 |
| Llama 3.3 70B single-stream | 32 | 28 | 6000 Pro |
| Llama 3.3 70B median TTFT | 320 ms | 380 ms | 6000 Pro |
| Llama 3.3 70B p99 TTFT | 820 ms | 950 ms | 6000 Pro |
| Multi-model (70B + Whisper + embeddings) | Yes | Tight (32 GB per card) | 6000 Pro |
| 64K context inference | Comfortable | Tight on KV | 6000 Pro |
| Cost per 1M tokens | £1.61 | £0.95 | 2× 5090 |
The 2× 5090 wins on aggregate throughput and cost-per-token. The 6000 Pro wins on TTFT, single-stream latency, and headroom.
Operational complexity
What works
- RTX 6000 Pro — single card, single vLLM process, no NCCL tuning, no tensor parallel orchestration
- RTX 6000 Pro — ECC + certified drivers (compliance simpler)
- RTX 6000 Pro — 96 GB headroom for multi-model deployments
- RTX 6000 Pro — full FP8 (vs INT4 on the cluster) with tighter quality
Where it breaks
- 2× 5090 — requires NCCL tuning, multi-process orchestration, careful GPU memory management
- 2× 5090 — INT4 quantisation has 1–3% quality regression vs FP8
- 2× 5090 — no ECC, consumer drivers (rare procurement issues)
- 2× 5090 — 5090s are not NVLink-equipped; PCIe is the bottleneck
Which one for which workload
| Workload | Recommended |
|---|---|
| Production 70B chatbot, FP8 quality required | RTX 6000 Pro |
| High-volume 70B inference, cost-anchored | 2× RTX 5090 |
| Compliance-bound (healthcare, finance) | RTX 6000 Pro |
| Multi-model stacks (70B + Whisper + embeddings) | RTX 6000 Pro |
| Long context (32K+) at 70B | RTX 6000 Pro |
| Research / ablation studies | Either; 2× 5090 if cost-driven |
| 64K+ context production | RTX 6000 Pro |
Verdict
For most production teams the RTX 6000 Pro 96 GB is the better operational choice — single card, full FP8, no NCCL, ECC. The £200/mo premium pays back in operational time.
For teams whose primary KPI is cost-per-token and who can absorb the operational complexity, 2× RTX 5090 is the cheaper deployment.
Bottom line
If you have to ask, the 6000 Pro is the right answer. If you already operate multi-GPU clusters comfortably and want every penny of cost savings, the 2× 5090 cluster is. See multi-GPU clusters and RTX 6000 Pro hosting.
