CUDA Graph Optimization for Inference

Kernel Launch Overhead Consumes 30% of Your Inference Time

Each CUDA kernel launch takes 5-10 microseconds on the CPU side. A single LLM forward pass executes hundreds of kernels — attention, linear projections, layer norms, activations. At batch size 1, each kernel completes in microseconds on the GPU, but the CPU spends milliseconds sequentially dispatching them. This launch overhead can account for 20-40% of total inference time on small batches. CUDA Graphs capture an entire sequence of kernel launches once, then replay them as a single GPU operation, eliminating per-kernel CPU overhead on your dedicated GPU server.

How CUDA Graphs Work

CUDA Graphs record a fixed sequence of GPU operations and replay them without CPU intervention:

# Normal execution (without CUDA Graphs):
# For each forward pass:
#   CPU: launch kernel 1 → GPU: execute kernel 1
#   CPU: launch kernel 2 → GPU: execute kernel 2
#   CPU: launch kernel 3 → GPU: execute kernel 3
#   ... (hundreds of sequential launch-execute pairs)
#   CPU overhead: ~5μs × 300 kernels = 1.5ms per forward pass

# CUDA Graph execution:
# Capture phase (once):
#   CPU: record kernel 1, kernel 2, kernel 3, ... into a graph
# Replay phase (every forward pass):
#   CPU: single command to replay entire graph
#   GPU: executes all kernels as a single unit
#   CPU overhead: ~10μs total per forward pass

# The GPU executes the same operations either way
# The difference is eliminating CPU-side dispatch latency
# Savings: 1-3ms per forward pass at small batch sizes

# This matters most when:
# - Batch size is small (1-4)
# - Model has many layers (more kernels per pass)
# - Individual kernels are fast (CPU overhead is proportionally large)
# - Latency matters more than throughput

PyTorch CUDA Graph Implementation

import torch

# Step 1: Warm up the model (allocates memory, JIT compiles)
model = model.cuda().eval()
static_input = torch.randn(1, 128, device="cuda", dtype=torch.float16)

with torch.no_grad():
    for _ in range(3):
        _ = model(static_input)
torch.cuda.synchronize()

# Step 2: Capture the graph
# IMPORTANT: Input tensors must be pre-allocated and reused
static_input = torch.randn(1, 128, device="cuda", dtype=torch.float16)
static_output = torch.empty(1, 128, device="cuda", dtype=torch.float16)

graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph):
    static_output = model(static_input)

# Step 3: Replay the graph with new data
# Copy new input into the static buffer (not a new allocation)
new_data = torch.randn(1, 128, device="cuda", dtype=torch.float16)
static_input.copy_(new_data)

# Replay — executes the entire forward pass as one GPU command
graph.replay()

# Read output from the static buffer
result = static_output.clone()

# Full inference function with CUDA Graph
class CUDAGraphInference:
    def __init__(self, model, input_shape, dtype=torch.float16):
        self.model = model.cuda().eval()
        self.static_input = torch.zeros(input_shape, device="cuda", dtype=dtype)

        # Warmup
        with torch.no_grad():
            for _ in range(3):
                self.model(self.static_input)
        torch.cuda.synchronize()

        # Capture
        self.graph = torch.cuda.CUDAGraph()
        with torch.cuda.graph(self.graph):
            self.static_output = self.model(self.static_input)

    def __call__(self, input_tensor):
        self.static_input.copy_(input_tensor)
        self.graph.replay()
        return self.static_output.clone()

CUDA Graphs in vLLM

vLLM uses CUDA Graphs automatically for the decode phase:

# vLLM enables CUDA Graphs by default for decode
# It captures graphs for different batch sizes

# Start vLLM with CUDA Graphs (default behavior)
vllm serve meta-llama/Llama-3-8B-Instruct \
    --enforce-eager false

# Disable CUDA Graphs if debugging
vllm serve meta-llama/Llama-3-8B-Instruct \
    --enforce-eager true
# --enforce-eager disables both torch.compile and CUDA Graphs

# How vLLM uses CUDA Graphs:
# 1. Captures separate graphs for batch sizes: 1, 2, 4, 8, 16, ...
# 2. During decode, selects the graph matching current batch size
# 3. Pads to nearest captured batch size if needed
# 4. Prefill phase does NOT use CUDA Graphs (variable length)

# Monitor CUDA Graph usage in vLLM logs
# Look for: "CUDAGraph captured for batch size X"
# Graph capture happens during the first few requests (warmup)

# Memory overhead of CUDA Graphs
# Each captured graph uses additional VRAM
# ~50-200MB per batch size captured
# vLLM manages this within --gpu-memory-utilization budget

# Check if CUDA Graphs are active
curl -s http://localhost:8000/metrics | grep cuda_graph
# Healthy: most decode steps use graph replay

Limitations and Workarounds

# CUDA Graphs have strict requirements:

# Limitation 1: Fixed input shapes
# Graphs are captured for specific tensor dimensions
# Different sequence lengths need different graphs
# Workaround: Pad inputs to fixed sizes, capture multiple graphs
graph_cache = {}
for seq_len in [128, 256, 512, 1024, 2048]:
    static_input = torch.zeros(1, seq_len, device="cuda")
    g = torch.cuda.CUDAGraph()
    with torch.cuda.graph(g):
        output = model(static_input)
    graph_cache[seq_len] = (g, static_input, output)

# Limitation 2: No dynamic control flow
# if/else, variable loops inside the graph are not allowed
# Workaround: Capture separate graphs for each code path

# Limitation 3: No CPU-GPU synchronization inside graph
# torch.cuda.synchronize() inside capture will fail
# Workaround: Synchronize before and after graph replay only

# Limitation 4: Memory allocations during capture
# new tensor allocations inside the graph are forbidden
# Workaround: Pre-allocate all tensors before capture
# Use static buffers and copy_ instead of creating new tensors

# Limitation 5: Not all operations are supported
# Some custom CUDA kernels may not be graph-capturable
# Workaround: Use --enforce-eager for those models, or
# split the model: graph for compatible parts, eager for rest

Benchmark CUDA Graph Speedup

import torch
import time

model = model.cuda().eval().half()
input_shape = (1, 128)

# Benchmark without CUDA Graph
inp = torch.randn(input_shape, device="cuda", dtype=torch.float16)
torch.cuda.synchronize()
start = time.perf_counter()
with torch.no_grad():
    for _ in range(1000):
        out = model(inp)
torch.cuda.synchronize()
eager_time = time.perf_counter() - start

# Benchmark with CUDA Graph
static_inp = torch.randn(input_shape, device="cuda", dtype=torch.float16)
with torch.no_grad():
    for _ in range(3):
        model(static_inp)
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph):
    static_out = model(static_inp)

torch.cuda.synchronize()
start = time.perf_counter()
for _ in range(1000):
    graph.replay()
torch.cuda.synchronize()
graph_time = time.perf_counter() - start

print(f"Eager: {eager_time/1000*1000:.2f}ms per iteration")
print(f"Graph: {graph_time/1000*1000:.2f}ms per iteration")
print(f"Speedup: {eager_time/graph_time:.2f}x")

# Expected results (varies by model and GPU):
# Small model, batch=1: 1.3-1.8x speedup
# Large model, batch=1: 1.1-1.3x speedup
# Large model, batch=32: 1.02-1.05x speedup (less overhead to eliminate)

CUDA Graphs are a free performance boost for latency-sensitive inference on your GPU server. vLLM enables them by default — deploy with the production guide. Measure gains against our token benchmarks. Profile kernel overhead with our monitoring setup. Browse more benchmarks, infrastructure guides, and tutorials.