Table of Contents
Capacity Planning Overview
Building an AI SaaS product requires matching GPU capacity to user demand. Undersize your infrastructure and users experience slow responses; oversize it and you waste money. This guide provides a systematic framework for GPU capacity planning across the most common AI workloads, using benchmark data from our dedicated GPU server testing.
We cover four common product patterns: chatbots, LLM APIs, image generation services, and voice agents. Each pattern has different latency requirements, traffic shapes, and scaling characteristics. For raw benchmark data, see the tokens per second benchmark and individual GPU capacity guides linked throughout.
The Sizing Framework
GPU capacity planning follows three steps. First, define your latency SLA — the maximum acceptable response time at a given percentile (typically p99). Second, determine your peak concurrent load — how many users or requests are active simultaneously during peak traffic. Third, select the GPU that meets both requirements at the lowest cost.
Key metrics to gather before sizing:
- Peak concurrent users: not total users, but simultaneously active ones. A 10,000-user SaaS might have 200 concurrent during peak hours.
- Request pattern: interactive (chatbot, voice) or batch (document processing, content generation).
- Latency target: sub-500 ms TTFT for chatbots, sub-1 s for APIs, sub-2 s for voice agents, no constraint for batch.
- Model requirements: which model and quantisation level meets your quality bar.
Use the LLM cost calculator to translate these parameters into monthly costs, and check the cost per million tokens benchmark for token-level pricing.
Sizing for Chatbot Products
Chatbots have a favourable scaling profile because users spend most of their time reading and typing, not waiting for generation. A typical 10:1 session-to-active ratio means 100 concurrent chatbot sessions produce only 10 simultaneous generation requests.
| Peak Concurrent Sessions | Active Generating (10:1) | Recommended GPU | Approx. Monthly Cost |
|---|---|---|---|
| 50 | 5 | RTX 4060 | ~£60 |
| 150 | 15 | RTX 3090 | ~£110 |
| 200 | 20 | RTX 5080 | ~£160 |
| 340 | 34 | RTX 5090 | ~£250 |
| 600 | 60 | 2x RTX 3090 + LB | ~£220 |
These assume Mistral 7B INT4 with sub-500 ms TTFT via vLLM. For detailed numbers by GPU, see the chatbot capacity guide and the individual concurrent user benchmarks for the RTX 3090, RTX 5080, and RTX 5090.
Sizing for LLM API Services
LLM APIs (document summarisation, content generation, code completion) typically have higher throughput requirements and more relaxed latency targets than chatbots. The key metric is requests per second at your acceptable end-to-end latency.
| Target req/min | Latency SLA | Recommended GPU | Approx. Monthly Cost |
|---|---|---|---|
| 50 | ≤ 5 s | RTX 4060 | ~£60 |
| 150 | ≤ 5 s | RTX 3090 | ~£110 |
| 300 | ≤ 5 s | RTX 5080 | ~£160 |
| 500 | ≤ 5 s | RTX 5090 | ~£250 |
| 1,000 | ≤ 10 s | 2x RTX 5090 + LB | ~£500 |
For throughput-maximised batch processing with no latency constraint, a single RTX 5090 processes over 600 requests per minute with Mistral 7B INT4. See the RTX 5090 throughput benchmark and RTX 3090 throughput benchmark for detailed numbers.
Sizing for Image and Audio Workloads
Image generation and audio processing have distinct scaling characteristics. Image generation is compute-bound and benefits from batch processing. Audio (Whisper, TTS) is often latency-sensitive for real-time use cases.
Image generation (SDXL): A single RTX 3090 produces 22 SDXL images per minute at batch 8. For an image generation API handling 100 images per minute, you need 5 RTX 3090 cards or 2 RTX 5090 cards. See the concurrent image generation benchmark for complete data.
Audio transcription (Whisper): The RTX 3090 handles 5 concurrent real-time Whisper Large v3 streams. For a call centre with 20 simultaneous calls, you need 4 RTX 3090 cards or 2 RTX 5090 cards. See the Whisper concurrent streams benchmark.
Voice agents: Voice pipelines (STT + LLM + TTS) are the most latency-demanding workload. The RTX 5090 achieves 310 ms end-to-end; the RTX 3090 achieves 710 ms. See the voice agent latency benchmark for full pipeline data.
Conclusion
GPU capacity planning starts with your latency SLA and peak concurrent load, then works backward to the right hardware. For chatbots, the RTX 3090 at 150 sessions is the value sweet spot. For high-throughput APIs, the RTX 5090 delivers the most requests per pound. For media workloads, horizontal scaling across multiple cards is almost always the right approach. Use our benchmark data to size precisely, then deploy on GigaGPU dedicated servers. See the 1 GPU vs 2 GPU scaling guide when you outgrow a single card.
