Prompt Caching Deep Dive

Caching for LLM workloads has multiple distinct layers. Each addresses different cache hit patterns. Understood and combined, they compound for substantial cost / latency savings. Standard production setup uses 2-3 layers.

Layers

• vLLM prefix caching: KV cache for shared prompt prefixes (system prompt, RAG context). 2-5× TTFT win on cache hit. Free with --enable-prefix-caching.
• Semantic caching: embed query; vector lookup; return cached response if similarity > threshold. ~20-40% hit rate typical.
• Literal response caching: exact-match cache (Redis with prompt hash key). Highest hit rate on FAQ / repeat queries.
• Hosted-API prompt caching (OpenAI / Anthropic): provider-side caching of system prompts; cost reduction on shared prefixes.

Composition

Standard production stack:

1. Request arrives
2. Literal cache lookup (Redis); if exact-match, return
3. Semantic cache lookup (Qdrant + BGE); if similarity > 0.95, return
4. vLLM serves: prefix caching reuses KV cache for shared parts of prompt
5. Response returned + cached at literal + semantic layers

Net hit rate at all-layer composition: typically 40-60% for FAQ-style workloads, 20-30% for general chatbots. Each hit saves the full LLM call cost.

Setup

• vLLM prefix caching: --enable-prefix-caching — default on in production
• Semantic cache: BGE-large embeddings + Qdrant collection + 0.95 similarity threshold
• Literal cache: Redis with SHA256(prompt) key + 24-hour TTL
• Order matters: cheapest cache lookup first (literal > semantic > vLLM)

Verdict

Composed caching is one of the highest-ROI optimisations for production LLM workloads. Three layers; each cheap; together they capture 30-60% of requests at near-zero cost. Build day-one of production; the cost saving compounds with traffic growth.