Table of Contents
After a year of customer deployments and the broader open-weight ecosystem maturing, this is the consolidated state-of-the-industry retrospective.
2026 winners: Blackwell hardware FP8, vLLM as the default engine, multi-LoRA serving, open-weight reasoning models (DeepSeek R1), Qwen 2.5 family. Still hard: frontier-quality agents, real-time vision/multimodal at scale. Coming in 2027: FP4 mainstream, better multi-tenant isolation.
What worked
- Blackwell FP8 / FP4: real 2× throughput uplift over Ada
- vLLM matured into the default production engine
- Multi-LoRA serving made multi-tenant SaaS viable
- Qwen 2.5 family genuinely competitive with closed frontier on most tasks
- RAG patterns standardised (BGE + reranker + LLM)
- LiteLLM as the default router
What's still hard
- Frontier-quality agents (Claude / GPT-4o still lead on hardest reasoning)
- Real-time multimodal at production concurrency
- Long-context (>128K) cost-effectively
- Eval discipline — most teams still ship without it
What's coming
- FP4 mainstream (NVFP4 / MX-FP4 supported in vLLM 0.7+)
- Better tooling for vector store + RAG eval
- More open-weight reasoning models
- Cheaper multi-GPU clusters (PCIe Gen 5 helping)
Verdict
Self-hosted AI is now a real choice for most production workloads. The hard problems are no longer infrastructure — they're evaluation, data, and operational discipline.
Bottom line
2026 was the year self-hosted open-weight AI became the default for cost-anchored, residency-bound, or customisation-heavy workloads. See infrastructure patterns 2026.
