Memory Bandwidth vs TFLOPS: Why It Matters

Your 300 TFLOPS GPU Generates Tokens No Faster Than a 100 TFLOPS Card

You upgraded to a GPU with triple the theoretical TFLOPS, expecting 3x faster inference. Token generation speed barely changed. The bottleneck was never compute — it was memory bandwidth. LLM inference during the decode phase reads the entire model weight matrix for every single token generated. A GPU with 2TB/s bandwidth feeds weights to its cores faster than one with 1TB/s, regardless of TFLOPS count. Understanding this relationship is essential when selecting a GPU server for AI workloads.

Why Bandwidth Dominates LLM Inference

Inference has two distinct phases with different bottlenecks:

# Prefill phase (processing the input prompt):
# - Highly parallel matrix multiplications
# - Compute-bound: benefits from high TFLOPS
# - All input tokens processed simultaneously
# - Arithmetic intensity: ~100-200 FLOPS/byte

# Decode phase (generating tokens one at a time):
# - Sequential: one token per forward pass
# - Must read ALL model weights for each token
# - Memory-bound: limited by bandwidth, not FLOPS
# - Arithmetic intensity: ~1-2 FLOPS/byte

# For a 70B parameter model in FP16 (140GB):
# Each decode step reads ~140GB from VRAM
# At 2TB/s bandwidth (RTX 6000 Pro 96 GB): 140/2000 = 70ms per token = ~14 tok/s
# At 3.35TB/s bandwidth (RTX 6000 Pro): 140/3350 = 42ms per token = ~24 tok/s

# TFLOPS barely matters here because the GPU cores are idle
# waiting for data from memory most of the time

Roofline Model: Visualizing the Bottleneck

The roofline model shows when a workload is memory-bound vs compute-bound:

# Calculate operational intensity (FLOPS per byte of memory traffic)
# If OI < ridge point, you are memory-bound

# Ridge point = Peak TFLOPS / Peak Bandwidth
# RTX 6000 Pro: 312 TFLOPS FP16 / 2.0 TB/s = 156 FLOPS/byte
# RTX 6000 Pro: 990 TFLOPS FP16 / 3.35 TB/s = 296 FLOPS/byte
# RTX 5090: 330 TFLOPS FP16 / 1.0 TB/s = 330 FLOPS/byte

# Workload operational intensities:
# LLM decode (batch=1):    ~1-2 FLOPS/byte  → memory-bound (always)
# LLM prefill (batch=1):   ~50-100 FLOPS/byte → memory-bound on RTX 6000 Pro
# LLM prefill (batch=32):  ~200+ FLOPS/byte → compute-bound
# Training forward pass:   ~100-300 FLOPS/byte → compute-bound
# Image generation:        ~50-150 FLOPS/byte → depends on batch size

# Python: estimate your workload's bottleneck
import torch, time

def measure_bandwidth(device="cuda"):
    """Measure effective memory bandwidth"""
    size = 1024 * 1024 * 256  # 1GB
    x = torch.randn(size, device=device, dtype=torch.float16)
    torch.cuda.synchronize()
    start = time.perf_counter()
    for _ in range(100):
        y = x.clone()
    torch.cuda.synchronize()
    elapsed = time.perf_counter() - start
    gb_transferred = 2 * size * 2 * 100 / 1e9  # read + write
    print(f"Effective bandwidth: {gb_transferred/elapsed:.0f} GB/s")

measure_bandwidth()

GPU Bandwidth Comparison for AI

# Memory bandwidth comparison (key spec for inference)
#
# GPU               VRAM    Bandwidth   FP16 TFLOPS   Bandwidth/TFLOP
# ---------------------------------------------------------------
# RTX 6000 Pro SXM          80GB    3,350 GB/s   990           3.38 GB/s
# RTX 6000 Pro PCIe         80GB    2,000 GB/s   756           2.65 GB/s
# RTX 6000 Pro SXM          80GB    2,039 GB/s   312           6.54 GB/s
# RTX 6000 Pro PCIe         80GB    2,039 GB/s   312           6.54 GB/s
# RTX 6000 Pro              48GB    864 GB/s     362           2.39 GB/s
# RTX 5090 32 GB    1,008 GB/s   330           3.05 GB/s
# RTX 3090          24GB    936 GB/s     142           6.59 GB/s
# RTX 6000 Pro             48GB    768 GB/s     155           4.95 GB/s
#
# For single-stream LLM inference: higher bandwidth = faster tokens
# For batched inference or training: TFLOPS matters more
# Bandwidth-per-TFLOP ratio shows decode efficiency

# Quick benchmark: tokens/second correlates with bandwidth
# RTX 6000 Pro 96 GB on Llama-3-70B (FP16):
#   Batch 1:  ~14 tok/s  (bandwidth-bound)
#   Batch 32: ~350 tok/s total (becoming compute-bound)
# RTX 6000 Pro SXM on Llama-3-70B (FP16):
#   Batch 1:  ~24 tok/s  (bandwidth-bound, 1.7x RTX 6000 Pro)
#   Batch 32: ~800 tok/s total (compute advantage shows)

Optimizing for Bandwidth-Bound Workloads

# Strategy 1: Quantization reduces bytes read per token
# FP16: 140GB for 70B model → ~14 tok/s on RTX 6000 Pro
# INT8: 70GB for 70B model  → ~28 tok/s on RTX 6000 Pro (2x faster)
# INT4: 35GB for 70B model  → ~50 tok/s on RTX 6000 Pro (3.5x faster)

# Strategy 2: Batching amortizes weight reads
# Batch 1:  read 140GB, generate 1 token  → 1 FLOP/byte
# Batch 16: read 140GB, generate 16 tokens → 16 FLOPS/byte
# Batching shifts workload toward compute-bound territory

# Strategy 3: Tensor parallelism aggregates bandwidth
# 2x RTX 6000 Pro: combined bandwidth = 4 TB/s → ~28 tok/s on 70B FP16
# 4x RTX 6000 Pro: combined bandwidth = 8 TB/s → ~50 tok/s on 70B FP16

# Strategy 4: Speculative decoding
# Draft model generates candidate tokens cheaply
# Main model verifies in parallel (batched = compute-bound)
# Effective 2-3x speedup for single-stream generation

# Measure actual memory bandwidth utilization during inference
nvidia-smi dmon -s mu -d 1
# Watch "fb" (framebuffer) utilization — high % means bandwidth-bound

Choosing a GPU: Bandwidth First for Inference

# Decision framework:
#
# Primarily single-user LLM inference:
#   Priority: Bandwidth > VRAM > TFLOPS
#   Best: RTX 6000 Pro SXM, RTX 6000 Pro SXM
#
# High-throughput batched inference (API serving):
#   Priority: TFLOPS ≈ Bandwidth > VRAM
#   Best: RTX 6000 Pro SXM (both are excellent)
#
# Training / Fine-tuning:
#   Priority: TFLOPS > VRAM > Bandwidth
#   Best: RTX 6000 Pro, then RTX 6000 Pro
#
# Image generation:
#   Priority: VRAM > TFLOPS > Bandwidth
#   Best: RTX 6000 Pro (48GB), RTX 5090 (fast at smaller batches)

# Cost efficiency: compare $/token not $/TFLOP
# A card with 2x bandwidth at 1.5x cost is still cheaper per token

Memory bandwidth determines single-stream token generation speed more than any other specification. Choose your GPU server based on the workload profile. See real throughput numbers in our tokens-per-second benchmarks. Deploy with vLLM using the production guide. Monitor bandwidth utilization with our monitoring setup. Explore more benchmarks, infrastructure guides, and tutorials.