Table of Contents
vLLM's prefix caching is one of five caching layers worth using.
TL;DR
Layered caching: 1) Prefix/KV cache (vLLM, free), 2) Response cache (Redis, exact prompt match), 3) Semantic cache (Redis + embeddings, similar prompt), 4) Embedding cache (avoid re-embedding), 5) Retrieval cache (Qdrant query results).
Five caching layers
- KV / prefix cache (vLLM): reuses prompt prefix KV state — see prefix caching
- Response cache: Redis keyed on prompt hash → cached response (exact match only)
- Semantic cache: embed query, find similar past query, reuse response (similarity threshold)
- Embedding cache: hash document content → cached embedding, avoid re-embedding
- Retrieval cache: cache top-K Qdrant results for popular queries
Verdict
For high-volume RAG: enable all five. Each one saves a different chunk of compute.
Bottom line
Caching is the cheapest cost optimisation in AI. See prompt caching economics.
