Multi-GPU Clusters for 70B+ Inference and Training
Tensor-parallel inference, ZeRO-3 fine-tunes, and high-throughput batch jobs need more than one card. Rent dedicated 2-, 4- and 8-GPU servers with PCIe Gen 4, NVLink (where supported), and 100 Gbps east-west networking — single tenant, single bill.
Why Self-Host Instead of Using a Hosted API
The cost-benefit shifts dramatically once you pass a few hundred pounds a month in API spend or have any data-residency obligation.
Hyperscaler GPU clouds
GigaGPU multi-GPU clusters
Cluster Hardware Spec
Everything you need to run production AI workloads on dedicated hardware in the UK.
Modern Blackwell & Ada GPUs
Mix-and-match RTX 3090, 5080, 5090, 6000 Pro 96 GB and A100 80 GB. We avoid mixing architectures inside one node so collective ops stay fast.
PCIe Gen 4 x16 to every card
Dual-socket EPYC platforms give you 128 PCIe Gen 4 lanes — every GPU lands on its own x16 slot. No CPU-bottlenecked all-reduce.
NVLink where the GPU supports it
RTX 6000 Pro Workstation pairs link via NVLink C2C; A100 SXM cluster nodes deliver 600 GB/s GPU-to-GPU. We document the topology before you buy.
NVMe scratch + bulk SSD
1.92 TB NVMe per GPU for Hugging Face cache and dataset shards, plus 8 TB SATA SSD bulk storage. Optional 100 TB Ceph mount for shared corpora.
100 Gbps east-west
Cluster-internal 100 GbE on every node, dedicated VLAN. Optional InfiniBand HDR upgrade for 8× node training jobs.
Power & cooling sized for 100% load
Each rack runs N+1 power feeds and a 30°C cold-aisle target. Rendered 8× RTX 5090 nodes draw 4.2 kW continuous — and we don’t throttle them.
Standard Cluster Configurations
These are the SKUs we keep in stock. Custom builds (8× A100, 4× H100, mixed precision rigs) ship in ~2 weeks.
| Config | GPUs | Combined VRAM | Best for | Price |
|---|---|---|---|---|
| Cluster S — 2× RTX 3090 | 2× RTX 3090 | 48 GB | 13B inference at scale, FLUX.1 batches | from £499/mo |
| Cluster M — 4× RTX 3090 | 4× RTX 3090 | 96 GB | Mixtral 8x7B INT4, ZeRO-2 fine-tune of 13B | from £899/mo |
| Cluster M+ — 2× RTX 5090 | 2× RTX 5090 | 64 GB | Llama 3 13B FP16 with TP=2, FLUX.1 batches | from £1,299/mo |
| Cluster L — 4× RTX 5090 | 4× RTX 5090 | 128 GB | Llama 3 70B INT4 / 33B FP16 inference | from £2,599/mo |
| Cluster L+ — 2× RTX 6000 Pro | 2× RTX 6000 Pro | 192 GB | Llama 3 70B FP16 inference, ECC research | from £3,499/mo |
| Cluster XL — 4× RTX 6000 Pro | 4× RTX 6000 Pro | 384 GB | DeepSeek V3 quantised, large fine-tunes | from £6,499/mo |
| Cluster A100 — 4× A100 80 GB | 4× A100 80 GB | 320 GB | Production training, BF16 fine-tune of 70B | POA |
| Cluster XL2 — 8× RTX 5090 | 8× RTX 5090 | 256 GB | High-throughput inference farm, batch SDXL | POA |
What People Run on Multi-GPU Clusters
Real customer workloads we run on this hardware every day.
Tensor-parallel 70B serving
Llama 3 70B and Qwen 2.5 72B do not fit on a single consumer GPU at FP16. Split across 4× RTX 5090 with vLLM tensor-parallel and you get production-grade throughput.
Distributed fine-tuning
ZeRO-3 with DeepSpeed or FSDP on PyTorch lets you fine-tune 13B–70B models on 4–8 cards. Long jobs that would cost £5K on AWS run for £3K/mo flat.
Batch experimentation
Sweep hyperparameters across multiple GPUs. Run 4 inference variants in parallel, A/B test prompt strategies, or measure latency across precisions.
Image & video generation farms
FLUX.1, SDXL, Wan and HunyuanVideo all benefit from running multiple workers on independent GPUs. One 4× RTX 5090 box is a small image-generation API.
Embedding pipelines
Indexing tens of millions of documents on a single GPU takes weeks. Spread the load across 8 cards, run BGE or nomic-embed in parallel, and ship in 24 hours.
Mixture-of-Experts hosting
MoE models like Mixtral 8x7B and DeepSeek-V2 expand to ~94 GB FP16. A 4× RTX 3090 box can serve them at INT4 with expert sharding.
From Order to First Token
From order to live inference in under 24 hours.
Pick the topology
Tell us the model, the precision, and the throughput target. We size 2/4/8 GPU and recommend the right interconnect.
Bare-metal build
We rack the chassis, populate the GPUs, validate PCIe topology and power draw, and image the OS.
Driver + framework
NVIDIA driver, CUDA toolkit, cuDNN, NCCL, and either vLLM/Ray or PyTorch + DeepSpeed pre-installed on request.
Cluster smoke test
We run NCCL all-reduce, GPU burn, and a sample tensor-parallel inference before handing over root credentials.
Deep Dive
Tensor parallel vs pipeline parallel — pick the right one
Two GPUs aren’t always faster than one. The wrong parallelism strategy will leave half your VRAM idle and double your latency. The rule of thumb:
- Tensor parallel (TP) splits each layer’s weights across N GPUs. Best for low-latency single-stream inference (chatbots). Needs fast GPU-to-GPU bandwidth — NVLink helps a lot, PCIe Gen 4 x16 is the floor.
- Pipeline parallel (PP) assigns whole layers to different GPUs. Best for very large models on cheaper interconnect, but adds "bubble" latency.
- Data parallel (DP) replicates the model and shards the batch. Best for training and high-throughput batch inference. Almost no inter-GPU traffic at inference time.
vLLM lets you set --tensor-parallel-size and --pipeline-parallel-size independently. For a typical 4× RTX 5090 cluster running Llama 3 70B INT4, the right answer is TP=4, PP=1.
How we tune NCCL for consumer-grade interconnect
The one thing that goes wrong on a budget cluster is collective ops. NCCL defaults are tuned for NVLink-rich datacenter cards. On RTX 3090/5090 boxes connected over PCIe, you need to:
- Disable peer-to-peer if topology forces traffic across the QPI link (
NCCL_P2P_DISABLE=1on rare combos) - Pin the NIC and the GPUs to the same NUMA node — we do this by default in BIOS
- Set
NCCL_SOCKET_IFNAMEexplicitly on multi-node jobs so it picks the 100 GbE NIC, not the management NIC - Tune ring vs tree algorithm based on world size — ring for 2–4 GPU, tree above
We ship a tuned NCCL config with every cluster. Our GPU server networking guide covers the full reasoning.
When to scale across nodes vs scale up the box
An 8× GPU single node beats two 4× GPU nodes for almost every workload because all communication stays on the same PCIe fabric. We recommend cross-node training only when:
- You need more than 8 GPUs (e.g., serving multiple 70B variants concurrently)
- Your fine-tune dataset is larger than what fits in a single node’s NVMe scratch
- You’re running a research workload that explicitly needs InfiniBand for AllReduce
If single-node 8× covers your need, take it. The price/performance is hard to beat.
Frequently Asked Questions
The questions buyers actually ask before committing to a GPU server.
Do you support NVLink?
Yes, on cards that support it. RTX 6000 Pro Workstation pairs link at NVLink speed; A100 SXM4 nodes hit 600 GB/s. RTX 5090 and 5080 do not have an NVLink connector — they communicate over PCIe Gen 4 x16, which is sufficient for 2–4 GPU TP.
Can I run cross-node training?
Yes. We can put 2–4 nodes on a dedicated 100 GbE fabric inside your VLAN. For ultra-low-latency AllReduce, ask about our InfiniBand HDR upgrade.
What about H100 or H200?
Available by request as a custom build. Lead time is 4–6 weeks and pricing is "contact sales" — H-class cards are not in our standard catalogue.
How does pricing compare to AWS?
Roughly 60–75% cheaper at full utilisation. A 4× A100 reservation on AWS is ~£25K/mo; ours is ~£7-9K/mo. See our RunPod alternatives and cloud vs dedicated breakdowns.
Can I bring my own SSDs / GPUs / firmware?
Yes for SSDs and firmware (within reason). For GPUs we generally provision our own — bring-your-own-card is a custom contract starting at 12 months.
How do you handle thermals on 8× cards in one box?
Custom rear-mounted blower configuration in a 4U Supermicro chassis with 30°C cold-aisle target. We thermally validate every node at 100% sustained load before shipping.
Do you offer InfiniBand?
Yes — HDR (200 Gbps) is available on dedicated cluster lanes for an additional fee. NDR (400 Gbps) on request.
What if I outgrow the cluster?
Add a second node on the same VLAN. Storage moves to a shared Ceph mount and training picks up across the new fabric. Most customers grow from 4× to 8× to a 2-node before talking colocation.
Related Pages
Pages our visitors typically read next.
Need more than one GPU? Let’s build the right cluster.
Tell us the model, the latency budget, and the precision target. We’ll come back with a topology, a price, and a delivery date — usually inside 24 hours.
