Table of Contents
Production capacity planning for an RTX 3090 deployment comes down to a single question: how many simultaneous users can it serve before latency degrades unacceptably? This page is the answer with real numbers.
An RTX 3090 24 GB hosting Mistral 7B FP16 sustains ~25 concurrent active users with median TTFT under 500 ms. Llama 3.1 8B tops out around 22. Qwen 2.5 14B INT4 sustains around 12. Above those thresholds, p99 TTFT blows past 1 second.
Test setup
- vLLM 0.6.3 with continuous batching
- Locust driver, 1K-token prompt + 256-token output
- Each "active user" sends a request every ~10 seconds (typical chat pattern)
- Steady state at 10 minutes
Results by model
| Concurrent users | Mistral 7B FP16 | Llama 3.1 8B FP16 | Qwen 2.5 14B INT4 |
|---|---|---|---|
| 10 | 180 ms / 380 ms | 200 ms / 410 ms | 320 ms / 640 ms |
| 25 | 320 ms / 720 ms | 380 ms / 820 ms | 720 ms / 1.4 s |
| 50 | 620 ms / 1.4 s | 780 ms / 1.7 s | 1.5 s / 3.2 s |
| 100 | 1.4 s / 2.8 s | 1.8 s / 3.4 s | queue overflow |
Median TTFT / p99 TTFT. Bold = SLA-friendly limit.
Latency degradation
The 3090 degrades smoothly until ~30 concurrent users, then sharply. Continuous batching helps but Ampere-era memory bandwidth is the bottleneck once KV cache pressure builds.
Verdict
- Comfortable: 25 concurrent active Mistral 7B users.
- Maximum tolerable: 40 concurrent before p99 exceeds 1 s.
- Above 50 concurrent: upgrade to 5090 (handles 80+).
Bottom line
For ~25 concurrent active users on a 7B model, the RTX 3090 at £159/mo is the cheapest credible host. Above that, the 5090 doubles capacity at 2× the cost. See RTX 3090 vs 5090 throughput per pound.
