Table of Contents
Customer SLAs for AI inference often get inherited from non-AI services. They're sometimes unrealistic.
TL;DR
Realistic SLA targets for single-server self-hosted AI: 99.5% uptime, p99 TTFT < 1.5s, availability during NVIDIA driver updates. For 99.9% uptime, multi-server is required.
Realistic targets
| Tier | Uptime | p99 TTFT | Architecture |
|---|---|---|---|
| Single server | 99.5% | < 1.5s | Dedicated + monitoring |
| Single server + fallback | 99.7% | < 1.5s | + hosted-API fallback via LiteLLM |
| Multi-server | 99.9% | < 1.0s | 2+ servers + load balancer |
| Multi-region | 99.95% | < 800ms regional | Geo-load-balanced |
Architecture for SLAs
- 99.5%: pin versions, automated systemd restart, monitoring
- 99.7%: + LiteLLM fallback to hosted API on failure
- 99.9%: + second dedicated server with load balancer
- 99.95%: + multi-region (UK + EU)
Verdict
Don't promise 99.99% on a single server. Five nines requires multi-server multi-region architecture.
Bottom line
Match SLA to architecture. See multi-server load balancing.
