Model Warm-up and Cold Start Patterns

Cold-start latency on LLM servers comes from three sources: weight loading from disk, CUDA graph compilation, and KV cache initialisation. Combined: ~30-90 seconds from process start to ready-to-serve. For production, always-on serving avoids it; for elastic scale-to-zero, it's a real cost.

Cold-start causes

• Weight loading from disk: 7B FP8 ~7 GB; on NVMe ~5-10 seconds, on SATA SSD ~30-60 seconds
• CUDA graph compilation: ~10-30 seconds for vLLM to compile attention kernels for the model + GPU combination
• KV cache initialisation: pre-allocates PagedAttention blocks, ~5 seconds
• First-request warm-up: cuBLAS heuristic search, ~1-2 seconds on first request after start

Mitigation

• Always-on: dedicated GPU + systemd auto-start avoids cold-start except after server reboot
• NVMe for weights: 5-10× faster than SATA SSD for weight loading
• Pre-pull models: download once at provision time, not on first request
• CUDA graph caching: vLLM caches compiled graphs; second start is faster
• Health-check warm-up: send dummy request after start; subsequent first-real-request is hot
• Avoid scale-to-zero: cold-start cost outweighs idle savings for any meaningful traffic

Ops impact

• Deployment / restart: budget 60-120 seconds for service to become ready
• Rolling deploys: blue-green ensures zero-downtime despite cold start
• Auto-scaling: pre-warm replicas before they're needed; never scale-to-zero on hot paths
• Monitoring: distinguish "starting" from "ready" in health checks; routes shouldn't hit warming instances

Verdict

For production LLM serving, the right answer is "always on, never scale to zero". Dedicated GPU pricing makes this economically viable; the operational simplicity and predictable latency are worth it. Cold-start mitigation is mostly about getting deployment / restart fast rather than about handling user-facing cold starts.