CUDA Graphs in vLLM

CUDA Graphs capture sequences of GPU operations into a single executable graph. For vLLM's decode loop — many tiny kernel launches per token — this eliminates per-call CPU↔GPU overhead. ~10-20% throughput improvement on small-batch decode.

How CUDA Graphs help

Without CUDA Graphs:

• Each decode step launches ~10-50 GPU kernels
• Each launch costs ~5-10 microseconds CPU↔GPU coordination
• Total overhead per token: ~50-500 microseconds — significant fraction of small-model decode time

With CUDA Graphs:

• Capture once: full sequence of decode-step kernels recorded into graph
• Replay per token: single launch invokes the entire graph
• Per-token overhead drops to ~5-10 microseconds

Net throughput improvement: ~10-20% on small-batch decode; less impact at high batch sizes (where compute dominates).

Config

vLLM uses CUDA Graphs by default. Verify with the start-up log message about "capturing CUDA graphs". Capture takes ~5-15 seconds at vLLM start — part of cold-start time.

Tunable via --max-cuda-graph-batch-size (default 256): controls which batch sizes get pre-captured graphs. Larger batches use eager mode.

When to disable

• Debugging: CUDA Graphs hide some debug info
• Capture failure: rare; some kernel patterns don't capture cleanly
• Custom kernels: if adding non-standard ops, may need eager mode
• Don't disable for performance: graphs are essentially always faster on supported hardware

Verdict

CUDA Graphs are the silent default that makes vLLM fast. Don't turn them off without reason. The ~5-15 seconds of capture time at start is part of cold-start budget. The runtime payoff — ~10-20% throughput on small-batch decode — is essentially free.