Table of Contents
Prefix caching reuses KV state across requests with shared prompt prefixes — system prompts, RAG contexts, few-shot examples. On a memory-tight 5060 Ti, the throughput uplift is meaningful.
TL;DR
Add --enable-prefix-caching to your 5060 Ti vLLM launch. On chatbot workloads with 1.5K-token shared system prompts, expect ~50% throughput uplift at 80%+ cache hit rate. Free.
What prefix caching buys
| Workload | Without prefix caching | With prefix caching | Uplift |
|---|---|---|---|
| Chatbot, 1.5K system prompt | 580 tok/s | 880 tok/s | +52% |
| RAG, 3K shared context | 410 tok/s | 620 tok/s | +51% |
| Multi-turn, turn 5 | 350 ms TTFT | 120 ms TTFT | -65% latency |
| Random unique prompts | 720 tok/s | 720 tok/s | 0% |
Setup
Single flag:
vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
--quantization fp8 \
--enable-prefix-caching \
--max-model-len 16384Verifying it works
vLLM’s Prometheus metrics include cache hit rate:
curl http://localhost:8000/metrics | grep prefix_cache
# Expect to see vllm:gpu_prefix_cache_hit_rate_perc
# After warmup, should be 60-90% for chatbot workloadsVerdict
Prefix caching is the highest-leverage tuning flag on memory-tight cards. Always enable. Verify hit rate after launch.
Bottom line
Free throughput. Always enable. See prefix caching deep dive.
