Multi-GPU NCCL Tuning on Dedicated Servers

NCCL is the library that moves tensors between GPUs during parallel inference and training. On dedicated GPU servers without NVLink, NCCL runs over PCIe and the defaults are rarely optimal. A handful of environment variables reliably improve throughput by 10-25%.

Topics

• What NCCL does by default
• Environment variables that matter
• How to measure improvement
• Common mistakes

Default Behaviour

NCCL auto-detects topology at init, picks a collective algorithm (ring, tree, or direct), and chooses a protocol (simple, LL, LL128). On consumer-class multi-GPU servers without NVLink, its choices are conservative – often correct but sometimes leaving performance on the table, particularly for all-reduce heavy workloads like tensor-parallel inference.

Variables That Matter

export NCCL_P2P_DISABLE=0
export NCCL_SHM_DISABLE=0
export NCCL_IB_DISABLE=1
export NCCL_DEBUG=WARN
export NCCL_ALGO=Ring
export NCCL_PROTO=Simple
export NCCL_NTHREADS=64

NCCL_P2P_DISABLE=0 allows GPU-to-GPU peer-to-peer over PCIe where supported. NCCL_SHM_DISABLE=0 lets NCCL fall back to shared memory for very small transfers. NCCL_IB_DISABLE=1 prevents NCCL from looking for InfiniBand when you do not have it – saves startup time. NCCL_ALGO=Ring and NCCL_PROTO=Simple are reliable on 2-card dedicated servers; try Tree if you move to 4+ GPUs.

Measuring Impact

Before tuning, benchmark your all-reduce bandwidth with nccl-tests:

./all_reduce_perf -b 8 -e 256M -f 2 -g 2

On a 2x RTX 5090 server over PCIe 5.0, expect roughly 30-50 GB/s peer-to-peer bandwidth with optimal settings. Anything below 20 GB/s suggests P2P is disabled or the CPU/chipset is bottlenecking the link.

Common Mistakes

First: do not leave NCCL_DEBUG=INFO on in production. It is useful during bring-up, noisy at scale. Use WARN once things work.

Second: do not disable shared memory. Some guides recommend NCCL_SHM_DISABLE=1 to force peer-to-peer – that backfires on servers with IOMMU restrictions where shared memory is the only reliable path for small transfers.

Third: verify the BIOS exposes ACS (Access Control Services) correctly. If ACS is on, peer-to-peer is blocked between some GPU pairs and NCCL falls back to host-staged transfers. Our dedicated servers ship with ACS configured for GPU workloads by default.

For the strategic view on when tensor parallel even pays back, see tensor vs pipeline parallelism. For production serving configuration see vLLM batching tuning.