LLM Routing Rules

For hybrid AI architectures (self-hosted + frontier API), routing rules determine which model handles which request. Smart routing captures cost saving while preserving quality where it matters. Several rule patterns work; pick by your specific cost-quality-latency priorities.

Routing rules

• Tier-based: free tier → cheap self-hosted; paid tier → premium model
• Complexity-based: small classifier predicts query difficulty; route accordingly
• Confidence-based: self-hosted runs first; if low-confidence output, escalate to frontier
• Error-based: primary fails / rate-limited → fallback to alternative
• Budget-based: monthly token budget per tenant; route to cheaper model when approaching cap
• Latency-based: time-sensitive requests → faster model; batch → cheaper / slower
• Region-based: route to GPU in user's region for latency / residency

Implementation

• LiteLLM config: declarative routing rules via YAML
• Custom Python wrapper: more flexible; complex routing logic
• Application-level router: tightest integration with your business logic
• Service mesh routing: for very large multi-tenant deployments

Examples

Concrete routing rule examples:

• If tenant.tier == "free": route to Llama 3.1 8B
• Else if query.complexity_score > 0.8: route to Claude 3.5 Sonnet
• Else if tenant.budget_remaining < 0.1: route to cheapest available
• Else: route to default Mistral 7B
• On 5xx error: retry once on alternative; if both fail, return cached response if available

Verdict

For hybrid AI architectures, smart routing rules capture cost saving while preserving quality where it matters. Start with tier-based + error-based as the simplest production rules; add complexity-based when you have signal to drive it. LiteLLM is the standard implementation primitive in 2026.