Table of Contents
Production AI deployments fail in specific recurring patterns. A documented runbook turns "3am panicked diagnosis" into "follow the steps". The five common failure modes cover ~90% of incidents.
Five common AI incident classes: (1) GPU thermal / hardware, (2) OOM / VRAM exhaustion, (3) p99 latency spike, (4) eval score regression / quality drop, (5) external API fallback failure. Runbook: triage by symptom, mitigate via traffic routing, diagnose via metrics, recover via rollback or hot fix. Document the post-mortem.
Incident classes
- Hardware: GPU thermal throttling, VRAM ECC errors, driver crash
- Resource: OOM, vLLM queue overflow, KV cache exhaustion
- Latency: p99 TTFT spike, decoding stall, network
- Quality: eval score regression, hallucination spike, unexpected output format
- External: hosted-API fallback fails, dependency down
Response
Standard response sequence:
- Triage: which incident class? Check Grafana dashboards (GPU health, vLLM metrics, app-level errors)
- Mitigate first: route traffic to fallback (hosted API, secondary model) before deep diagnosis
- Diagnose: dive into structured logs, dmesg for hardware, vLLM logs for serving issues
- Fix: hot fix (config change, restart), rollback (model / prompt version), or scale-out (add replica)
- Verify: confirm metrics return to baseline; eval harness re-runs cleanly
- Post-mortem: document timeline, root cause, action items within 48 hours
Recovery
Recovery time targets:
- Mitigation (traffic routed away): < 5 minutes
- Hot fix or rollback: < 30 minutes
- Full root cause + fix: < 24 hours
- Post-mortem complete: < 48 hours
Verdict
Production AI incidents follow patterns. A documented runbook + practiced response cuts MTTR dramatically. The biggest predictor of bad incident outcomes is having no runbook — every team that runs AI in production should write theirs before the first incident.
Bottom line
Document the runbook before you need it. See logging.
