Internal AI tooling has predictable characteristics: known concurrent users (~team size, not scaled web traffic), latency-tolerant, accuracy-critical, residency-sensitive (touches internal data). The 4090 24GB is the right tier for most teams of 50-500 people.
4090 + Llama 3.1 8B FP8 + BGE-large + reranker fits a complete internal-tooling stack with comfortable headroom for a 50-500-person team. Co-host Whisper for transcription, optional Qwen 2.5 Coder 7B for code questions. £289/mo all-in. UK / EU residency compliant.
Scope
Typical internal AI tools:
- Engineering Q&A: questions over internal docs, runbooks, code
- Meeting transcription + summary: Whisper + LLM
- Standup / status summarisation: across Slack / Linear / Jira
- Code review assistance: PR comment generation
- HR Q&A: holiday policy, expense rules, etc.
- Search across internal wikis: with rerank + LLM synthesis
Stack
- Primary LLM: Llama 3.1 8B FP8 (~7 GB) — handles ~80% of queries
- Code LLM: Qwen 2.5 Coder 7B FP8 (~7 GB) — co-resident, swap on demand
- Embeddings: BGE-large or BGE-m3 multilingual (~1.5 GB)
- Reranker: BGE-reranker-v2-m3 (~2 GB)
- STT: Whisper Turbo v3 (~3 GB) co-resident on 4090
- Vector store: Qdrant on the same box (CPU + disk)
Total VRAM: ~13.5 GB on 24 GB → ~10 GB headroom for KV cache + concurrency.
Sizing
For a 200-person team with typical AI-tool usage (~5 queries/person/day = 1,000 queries/day):
- Peak concurrency: ~20 queries/min during work hours
- 4090 covers this with sub-1-second TTFT
- Daily peak ~50% utilisation; nights idle
- ~15K hours of meeting transcription/month possible alongside
Above 500 people, step up to 5090 for headroom or split LLM and embeddings across two 4090s.
Verdict
For 50-500-person internal tooling deployments, a single 4090 covers the full stack at £289/mo. Below 50 people, 5060 Ti is the right tier. Above 500, multi-GPU or 6000 Pro.
Bottom line
4090 = 200-person internal tooling sweet spot. See multi-model sizing.
