Table of Contents
Picks: 5060 Ti / 4090 / 5090 / 6000 Pro. Stack: vLLM + Llama / Qwen / Mistral / DeepSeek + BGE / Jina / Nomic embeddings. Patterns: hybrid (self-hosted + hosted-API fallback), OpenAI-compatible API everywhere, eval harness as first-class artefact. Self-hosted is the production default for steady-state, residency-sensitive, or cost-anchored deployments.
Status
- Hardware: Blackwell native FP8 / FP4 mainstream
- Software: vLLM 0.6+ continuous batching, prefix caching, FP8 KV
- Models: open-weight competitive with frontier hosted
- Cost: 15-25× cheaper than hosted at scale
- Compliance: UK / EU residency simplified
Common patterns
Hybrid stacks dominate: self-hosted Llama / Qwen for the bottom 90% of traffic, frontier hosted (Claude, GPT-4o) for the hardest 5-10%. OpenAI-compatible API as the universal contract — clients do not care what is behind the URL. Eval harnesses live in version control next to prompts; quality regressions are caught in CI.
Hard problems
The hard problems shifted up the stack. Eval design (what does "quality" mean for your domain), data engineering (clean training / RAG corpora), and ops (deploy / observability / cost monitoring) are now the time sinks. Hardware and basic deployment are well-trodden. Reasoned guidance from observed 2025-26 patterns.
Verdict
Self-hosted AI is now the production default for steady-state, residency-sensitive, or cost-anchored deployments. The hard problems are eval, data, and ops, not hardware.
Bottom line
Self-hosted is the production default. See dedicated GPU hosting.
