Table of Contents
Canary deployment for AI features ramps a small percentage of traffic to the new version, monitors for regressions, and either promotes or rolls back. For AI specifically, eval-driven gating beats pure metric gating — generative quality regressions are subtle.
Standard canary: 1% → 5% → 25% → 75% → 100% over 1-2 weeks. Eval-gated: each step requires eval score holds within tolerance + production metrics nominal. Roll back via feature flag flip if any gate fails. Always keep previous version warm during canary window.
Pattern
- Deploy new version (model / prompt / RAG strategy) to canary pool
- Run eval harness against canary; must hold within 1-2% of baseline
- Route 1% traffic via feature flag; monitor for 24-48 hours
- If metrics + eval hold: ramp to 5%; monitor 1-2 days
- Continue ramp: 5% → 25% → 75% → 100% over 1-2 weeks
- Each step: eval gate + production metric gate
- If any gate fails: feature flag flip to roll back; investigate
Eval gates
- Eval harness score: representative prompts; new score ≥ baseline minus tolerance (e.g., 1-2%)
- Safety eval: harmful-output regression check; new version passes
- Cost / latency: within acceptable envelope
- Per-segment scores: critical workloads (regulated tenants, premium features) hold individually, not just in aggregate
Monitoring
During canary ramp, watch:
- p99 TTFT / TPOT vs baseline
- Error rate
- User feedback (thumbs / rating) — segmented by canary vs control
- Hosted-API fallback rate (canary triggering more fallbacks signals quality drop)
- Cost per request
Verdict
For AI feature changes, canary is the right deploy pattern. Eval-driven gating + gradual ramp + always-warm rollback path catches regressions before they reach all users. Skip canary and you'll learn the lesson when an "identical-looking" prompt change degrades quality on a workload nobody anticipated.
Bottom line
Eval-gated canary for AI changes. See model rollout.
