Rate Limiting and Fairness for AI APIs

Rate limiting AI APIs has both standard (per-tenant fairness, prevent abuse) and AI-specific dimensions (per-tenant token budgets, context-length caps, expensive-model gating). Get this right at deployment day-one; retrofitting is painful.

Why rate limit

• Prevent runaway tests breaking production: one buggy script shouldn't saturate the GPU
• Per-tenant fairness: one tenant shouldn't starve others
• Cost control: token-budgets per tenant enforce billing tier
• Abuse prevention: prevent compromised API keys causing massive bills
• SLO protection: cap concurrent requests to maintain p99 latency

Algorithms

• Token bucket: refill at rate R; allow burst up to capacity C. Standard for request-rate limits.
• Leaky bucket: smooths out bursts more strictly. Useful for backend protection.
• Sliding window: counts requests in last N seconds; more accurate than fixed window.
• Distributed: Redis-backed for multi-instance deployments

AI-specific

• Token-budget per tenant: track input + output tokens per day / month per tenant; reject when budget exceeded
• Context-length cap: per-tenant max context length (e.g., free tier 4K, premium 32K)
• Model access: per-tenant allowlist of which models they can call
• Concurrent-request cap: max in-flight per tenant to prevent monopolising
• Cost-aware: bigger model = lower per-tenant rate limit (since each request is more expensive)

Verdict

For production AI APIs, rate limiting is essential and AI-specific. nginx handles request-rate; application layer handles token-budget + context-length + model-allowlist. Build day one; retrofitting after launch is painful. Track usage continuously to inform billing tier design.