CUDA Error: Device-Side Assert Triggered (Fix)

The Error You Are Seeing

Your training run or inference script crashes with:

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,
so the stacktrace below might be incorrect.

This is one of the most frustrating CUDA errors because it gives almost no information about what actually went wrong. The assert fired inside a GPU kernel, but by the time Python catches it, the original context is lost. The traceback points to the wrong line, and every subsequent CUDA call also fails until you restart the process.

Why the Stacktrace Is Misleading

CUDA executes kernels asynchronously. When a kernel triggers an assert on the GPU, the CPU has already moved on to later operations. The error surfaces when the next CUDA synchronisation point occurs — which could be lines or even functions away from the actual problem. This is by design in CUDA’s execution model, but it makes debugging on a GPU server genuinely difficult without the right approach.

Getting the Real Error Message

The single most effective debugging step is forcing CUDA to run synchronously. Set this environment variable before running your script:

CUDA_LAUNCH_BLOCKING=1 python your_script.py

With synchronous execution, the error will now point to the exact line that caused the assert. The traceback becomes accurate, and you will usually see a more specific error message such as:

• index out of range in self — an embedding layer received an index greater than its vocabulary size
• invalid argument — a tensor shape or type was wrong for the operation
• misaligned address — memory corruption, often from incorrect custom CUDA kernels

Common Causes and Their Fixes

Cause 1: Index exceeds vocabulary size

This is the most frequent trigger. Your tokenizer produces IDs larger than your model’s embedding table.

# Check the max token ID in your data
max_id = input_ids.max().item()
vocab_size = model.config.vocab_size
print(f"Max token ID: {max_id}, Vocab size: {vocab_size}")
assert max_id < vocab_size, f"Token ID {max_id} exceeds vocab {vocab_size}"

Fix: resize the embedding layer or fix the tokenizer. If you are using Hugging Face models on a PyTorch GPU server, ensure the tokenizer and model came from the same checkpoint.

Cause 2: Label values outside expected range

Cross-entropy loss in PyTorch requires labels in the range [0, num_classes) or -100 (the ignore index). A label of -1 or a label equal to num_classes triggers the assert.

# Validate labels before passing to loss
assert labels.min() >= -100, f"Invalid label: {labels.min()}"
assert labels.max() < num_classes, f"Label {labels.max()} >= {num_classes}"

Cause 3: NaN propagation into integer casts

When a floating-point NaN is cast to an integer (for example, as an index), the result is undefined and can trigger asserts.

# Check for NaN in your tensors
if torch.isnan(logits).any():
    print("NaN detected in logits — check loss scaling or learning rate")

On dedicated GPU servers running long training jobs, NaN propagation from learning rate spikes is a common culprit.

Recovering After the Assert

Once a device-side assert fires, the CUDA context is corrupted. Every subsequent CUDA call will fail with the same error. You must restart the Python process entirely. There is no way to recover within the same process.

For vLLM or other long-running inference servers, configure automatic restarts with systemd or a process supervisor so that a single bad request does not permanently crash the service. Our vLLM production setup guide covers this.

Preventing Device-Side Asserts in Production

• Always validate input tensors before sending them to the model. Check bounds, shapes, and dtypes.
• Add assertion checks in your data pipeline that run on CPU before the GPU sees the data.
• Use GPU monitoring to detect NaN in loss values early — a sudden NaN usually precedes the assert by a few steps.
• During development, run with CUDA_LAUNCH_BLOCKING=1 to catch problems at their source. Disable it for production since it significantly slows execution.
• When fine-tuning models from third-party sources, verify that the tokenizer vocabulary size matches the model's embedding dimension.

For multi-GPU setups, device-side asserts on one GPU can cascade. If you are running distributed training via Docker or direct multi-process launching, ensure each process has independent error handling.