After a year of customer deployments on dedicated GPU hardware, certain patterns recur. This page is the retrospective.
TL;DR
Worked: vLLM + LiteLLM + Caddy, FP8 by default, multi-LoRA for SaaS, Qdrant, private dedicated for steady traffic + hosted API for overflow. Did not work: Ollama in production, consumer-PC hosting, per-tenant model copies, SageMaker for cost-anchored teams.
The patterns that worked
- vLLM + LiteLLM: solved auth, rate limiting, model routing in one layer
- FP8 by default on Blackwell: free throughput
- Multi-LoRA for SaaS: 30+ tenants per GPU
- Qdrant for RAG: simple, fast, just-works
- Hybrid private + public APIs: best cost for most teams
- systemd over Kubernetes: simpler for single-server
- Caddy for TLS: minimal config, automatic certs
- Pinned versions everywhere: prevented quiet regressions
- UK datacenter for EU customers: compliance painless
- Mistral 7B / Llama 3.1 8B as defaults: 80% of workloads
The patterns that did not
- Ollama in production: weak metrics, no tracing, queue collapse under load
- Consumer-PC GPU hosting: thermal events, power instability
- Per-tenant model copies: doesn't scale past 5 tenants
- SageMaker for cost-anchored teams: 30-40% premium not justified
- Skipping observability: outages diagnosed after the fact
- Naive round-robin load balancing: kills prefix cache hits
- Trusting vLLM defaults: tuned for benchmarks, not production
- FP16 on Blackwell "to be safe": 50% throughput left on the table
What we expect in 2027
- FP4 native paths becoming the default for inference
- Multi-LoRA hot-swap latency <10 ms
- Frontier-quality 30B-class open models making 70B less necessary
- More mature multi-tenant isolation in vLLM
- Better tooling around RAG quality eval
Verdict
The self-hosted AI stack has converged on a small set of well-tested patterns. The hard problems are no longer infrastructure — they’re data, evaluation, and operational discipline.
Bottom line
Build the boring infrastructure first. The model is the easy part. See build a production AI inference server for the canonical recipe.
