Continuous Batching: Maximise GPU Utilisation for LLMs

Static Batching vs Continuous Batching

Traditional LLM serving on a dedicated GPU server uses static batching: collect N requests, process them all together, and wait until every request in the batch finishes before starting the next batch. This wastes GPU cycles because short responses finish early but the GPU waits for the longest response in the batch.

Continuous batching (also called iteration-level batching) solves this by inserting new requests into the batch as soon as existing requests complete. The GPU never idles waiting for stragglers. For teams hosting open-source LLMs at scale, this is one of the highest-impact optimisations available.

How Continuous Batching Works

In continuous batching, the scheduler evaluates the batch at every iteration (every token generation step). When a request finishes generating, the scheduler immediately fills that slot with a waiting request. This keeps the GPU running at maximum throughput regardless of output length variance.

The key insight: LLM requests have wildly different output lengths. A batch of 8 requests might produce outputs ranging from 20 to 500 tokens. With static batching, the GPU is 96% idle by the time the 500-token response finishes (only 1 of 8 slots still active). With continuous batching, those 7 freed slots are immediately filled with new work.

Before and After Benchmarks

We compared static and continuous batching on an RTX 3090 running Llama 3 8B with vLLM, using realistic variable-length requests.

The throughput nearly doubles and tail latency drops by 75%. This is effectively free performance gained purely from better scheduling. Check throughput expectations on our tokens per second benchmark.

Enabling Continuous Batching in vLLM

vLLM uses continuous batching by default. The key parameters to tune are batch size limits and scheduling policy.

# vLLM with continuous batching (default) and tuned parameters
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8B \
  --max-num-seqs 16 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.90 \
  --scheduler-delay-factor 0.1 \
  --port 8000

Key parameters explained:

• --max-num-seqs: Maximum concurrent sequences in the batch. Higher values increase throughput but require more KV cache VRAM.
• --max-num-batched-tokens: Maximum total tokens processed per iteration across all sequences.
• --scheduler-delay-factor: Small delay before scheduling to accumulate requests for better batching. Set to 0 for minimum latency.

For complete vLLM configuration, see our vLLM production setup guide.

VRAM Impact and Scaling

Continuous batching works best when you can maintain a large active batch. This requires sufficient VRAM for KV cache across all concurrent sequences.

The gains plateau around 16 sequences for this model and GPU. More sequences require proportionally more VRAM. Read our vLLM memory optimisation guide for KV cache sizing.

Production Tuning Guide

Match max-num-seqs to your VRAM budget. Calculate available VRAM after model weights, then divide by per-sequence KV cache size. Leave a 10% margin for safety.

Use quantised KV cache. FP8 KV cache halves memory per sequence, letting you double the batch size. Enable with --kv-cache-dtype fp8 in vLLM.

Set context length appropriately. If your application uses 2K context maximum, set --max-model-len 2048. Shorter context means smaller KV cache per sequence and more concurrent requests.

Monitor batch fill rate. Track the average number of active sequences in each iteration. If it is consistently below max-num-seqs, your traffic is too low to benefit from larger batch limits. Optimise traffic routing or reduce infrastructure. See our GPU monitoring guide for setup.

Continuous batching combined with quantisation and proper memory planning lets you extract maximum throughput from your dedicated GPU hosting. Use the cost per million tokens calculator to see how these improvements affect your per-token economics. For batch size effects, review our batch size impact analysis.