Speculative Decoding: Speed Up LLM Inference 2-3x

How Speculative Decoding Works

Standard LLM inference generates one token at a time. Each token requires a full forward pass through the model on your dedicated GPU server, and the GPU is often underutilised during single-token generation due to memory bandwidth limitations.

Speculative decoding uses a small, fast “draft” model to predict multiple tokens ahead, then verifies those predictions with the large “target” model in a single batched forward pass. If the draft model predicted correctly (which happens frequently for common patterns), you get multiple tokens from one target model pass. This can speed up inference by 2-3x without any quality loss. The technique is mathematically guaranteed to produce identical output to standard decoding. For more on inference optimisation, see our LLM hosting guides.

Before and After Benchmarks

We benchmarked speculative decoding on an RTX 3090 using vLLM.

The speedup is most pronounced at batch size 1, where the GPU is most underutilised. At larger batch sizes, the benefit diminishes because the GPU is already well-utilised. Verify these numbers on our tokens per second benchmark.

Setting Up Speculative Decoding in vLLM

vLLM supports speculative decoding natively. Enable it by specifying a draft model and the number of speculative tokens.

# Basic speculative decoding setup
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8B \
  --speculative-model meta-llama/Llama-3-1B \
  --num-speculative-tokens 5 \
  --gpu-memory-utilization 0.90 \
  --port 8000

# With quantised target model
python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Llama-3-13B-GPTQ \
  --speculative-model meta-llama/Llama-3-1B \
  --num-speculative-tokens 4 \
  --quantization gptq \
  --gpu-memory-utilization 0.90

The draft model should be from the same model family as the target for best acceptance rates. A Llama draft works best with a Llama target. For full vLLM setup, read our vLLM production setup guide.

VRAM Impact and Memory Planning

Speculative decoding requires loading both models into VRAM simultaneously. Here is the VRAM budget impact.

The draft model adds 2 GB but you gain 2x+ throughput. The trade-off is a slightly smaller KV cache budget, meaning slightly fewer concurrent requests. For memory optimisation strategies, see our vLLM memory guide.

Tuning Parameters for Best Results

Number of speculative tokens (k). Higher k means more tokens per verification step, but also higher rejection probability. Start with k=4-5 and adjust based on acceptance rate. Monitor with vLLM’s metrics endpoint.

Draft model selection. Choose the smallest model from the same family that maintains reasonable quality. For Llama 3, the 1B variant works well. For Mistral, use the smallest available checkpoint.

Batch size interaction. Speculative decoding benefits single-request latency the most. At batch sizes above 8, the overhead of running the draft model can negate the gains. For high-throughput batch workloads, standard continuous batching may be more effective. Read our batch size impact analysis for guidance.

When to Use Speculative Decoding

Use speculative decoding when: You are serving single-user or low-concurrency workloads, latency per request matters more than total throughput, and you have VRAM headroom for the draft model. It is ideal for real-time chatbots and interactive applications on private AI hosting.

Skip speculative decoding when: You are running high-batch workloads where the GPU is already saturated, VRAM is too tight for the draft model, or your use case is batch processing where total throughput matters more than per-request speed.

Speculative decoding pairs well with other optimisations like quantisation and open-source model selection. Combine techniques to get the most from your dedicated GPU server. Compare your optimised costs against API providers using the GPU vs API cost comparison.