1,000 Posts on Self-Hosted AI: What We've Learnt

This is post 1,000 in this series on self-hosted AI infrastructure. Across the corpus, certain trends and patterns repeat. The takeaways have stabilised.

Trends

• Open-weight quality caught frontier on ~90% of tasks by April 2026; gap continues to narrow
• Cost economics decisively favour self-hosted above ~30M tokens/month; trajectory accelerating
• UK / EU residency driving adoption in regulated industries (financial services, healthcare, public sector)
• Hybrid architecture (self-hosted bulk + frontier API for hardest 5-10%) is the dominant production pattern
• Multi-LoRA serving turning per-tenant fine-tuning from uneconomic to standard
• Blackwell hardware + native FP8 making consumer-card AI production-grade

Dominant patterns

• Hardware: 5060 Ti 16GB for SMB 7B; 4090 24GB for 13B / mid-market; 5090 32GB for premium / 70B INT4; 6000 Pro 96GB for 70B FP8
• Stack: vLLM + Llama 3.1 8B FP8 (or Mistral 7B / Qwen 2.5 7B by language) + BGE-large + reranker + Qdrant + LiteLLM router
• Ops: DCGM + Prometheus + Grafana + structured logs + RAGAS eval harness + feature flags
• Compliance: UK / EU residency + comprehensive audit logs + per-tenant isolation
• Cost: ~£0.20/M tokens self-hosted Mistral 7B; semantic + prefix caching for 30-60% hit rate; per-feature attribution

Predictions

• Cost reduction continues: ~£0.10/M by mid-2027
• Open-weight catches frontier on harder tasks (reasoning, multimodal, long-context)
• FP4 + algorithmic improvements compound to ~2-3× throughput
• Multi-LoRA serving becomes the SaaS default
• EU AI Act drives further self-hosted adoption in EU
• Hybrid (self-hosted + frontier) remains the production default

Verdict

1,000 posts in, the picture for self-hosted AI in 2026 is clear: it's the production default for any deployment above SMB scale. The economics, model quality, operational tooling, and compliance fit have all matured. The remaining hosted-API role is fallback for hardest queries plus prototyping. For teams committing to AI as core infrastructure, self-hosted is the right architecture; build it deliberately, document it carefully, and the patterns are mature enough to be replicable.