vLLM Memory Fragmentation: Defragmentation Guide

How Memory Fragmentation Manifests

Your vLLM server is running on a dedicated GPU with plenty of VRAM allocated to the KV cache, yet you see request rejections and throughput degradation that suggest the cache is full. nvidia-smi shows 20 percent of reserved cache memory as unused. The cache has capacity on paper but cannot use it in practice.

This is memory fragmentation — the KV cache has free blocks scattered throughout that are too small or poorly positioned to be allocated to new sequences. It is the GPU memory equivalent of a hard drive that needs defragmenting.

How vLLM’s PagedAttention Manages Memory

vLLM’s key innovation is PagedAttention, which divides the KV cache into fixed-size blocks (like pages in an operating system). Each block stores KV pairs for a fixed number of tokens. When a sequence grows, vLLM allocates additional blocks — they do not need to be contiguous.

Fragmentation occurs when:

• Short sequences complete and leave gaps. A burst of short-prompt requests fills many blocks partially, then frees them. The remaining free blocks are scattered.
• Variable-length sequences create uneven allocation. Sequences of very different lengths create a patchwork of allocated and free blocks.
• Block size does not match workload patterns. If the block size is 16 tokens and most sequences use 17 tokens, half of the second block is wasted.

Diagnosing Fragmentation

Check vLLM’s internal metrics to see block utilisation:

# If using the metrics endpoint
curl http://localhost:8000/metrics 2>/dev/null | grep -i block

# Look for:
# vllm:num_gpu_blocks_total
# vllm:num_gpu_blocks_used
# vllm:gpu_cache_usage_perc

If gpu_cache_usage_perc is consistently below 80 percent but requests are being queued or rejected, fragmentation is the issue. The cache has space but cannot allocate it efficiently.

Fix 1: Adjust Block Size

--block-size 32

The default block size is 16 tokens. If your typical generation length is short (under 100 tokens), a smaller block size of 8 reduces waste per block. If generations are long (1000+ tokens), a larger block size of 32 reduces allocation overhead. Match the block size to your median output length divided by a reasonable number of blocks per sequence.

Fix 2: Configure Preemption Policy

--preemption-mode recompute

When the cache is full, vLLM can either swap preempted sequences to CPU memory or recompute them later. The recompute mode frees blocks cleanly without fragmented swap state. For inference-heavy workloads on GPU servers with fast GPUs but limited CPU memory, recompute is often better.

Fix 3: Limit Maximum Concurrent Sequences

--max-num-seqs 128

Fewer concurrent sequences means less fragmentation. Each active sequence holds at least one block, and many short sequences fragment the cache more than fewer long ones. If your workload can tolerate slightly lower concurrency, reducing max-num-seqs from the default 256 improves block utilisation.

Fix 4: Scheduled Restarts

The most pragmatic approach for production systems: schedule periodic vLLM restarts during low-traffic windows. A fresh start resets all memory pools to a clean state.

# Cron job for daily restart at 4 AM
0 4 * * * systemctl restart vllm.service

This works well for servers that run continuously for days. Fragmentation accumulates gradually, and a restart eliminates it entirely.

Monitoring Fragmentation Over Time

#!/bin/bash
# Track cache efficiency hourly
while true; do
    USAGE=$(curl -s http://localhost:8000/metrics | \
            grep "gpu_cache_usage_perc" | awk '{print $2}')
    echo "$(date '+%Y-%m-%d %H:%M') Cache usage: ${USAGE}"
    sleep 3600
done

If cache usage drops steadily over hours despite consistent request rates, fragmentation is accumulating. Correlate with your GPU monitoring data for a complete picture.

Production Configuration for Minimal Fragmentation

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.92 \
  --max-model-len 4096 \
  --max-num-seqs 128 \
  --block-size 16 \
  --preemption-mode recompute \
  --enable-prefix-caching

Prefix caching actually helps with fragmentation because shared prefix blocks are allocated once and reused, reducing per-sequence block allocation. Combine with the settings from our vLLM optimization guide. For the complete production deployment setup including service management, see our vLLM production guide. Review other tutorials for related configuration topics.