Can a 3.8-billion-parameter model really saturate a 32 GB flagship GPU? Not even close — and that is precisely the point. Pairing Phi-3 Mini with the RTX 5090 delivers triple-digit throughput while leaving enough VRAM for an entire second workload. We benchmarked it on GigaGPU dedicated infrastructure.
Performance Figures
| Metric | Value |
|---|---|
| Tokens/sec (single stream) | 100 tok/s |
| Tokens/sec (batched, bs=8) | 160.0 tok/s |
| Per-token latency | 10.0 ms |
| Precision | FP16 |
| Quantisation | FP16 |
| Max context length | 16K |
| Performance rating | Excellent |
Conditions: 512-token prompt, 256-token completion, single-stream, llama.cpp. The 5090’s massive memory bandwidth pushes Phi-3 Mini past the 100 tok/s barrier at FP16 with no quantisation needed.
VRAM: 24 GB Going Spare
| Component | VRAM |
|---|---|
| Model weights (FP16) | 8.0 GB |
| KV cache + runtime | ~1.2 GB |
| Total RTX 5090 VRAM | 32 GB |
| Free headroom | ~24.0 GB |
Twenty-four gigabytes of free headroom is remarkable. Practical uses include running Phi-3 at 16K context, co-hosting a Stable Diffusion or Whisper model on the same card, or handling dozens of concurrent inference streams in a multi-tenant setup.
Is the 5090 Overkill for Phi-3?
| Cost Metric | Value |
|---|---|
| Server cost | £1.50/hr (£299/mo) |
| Cost per 1M tokens | £4.167 |
| Tokens per £1 | 239,981 |
| Break-even vs API | ~1 req/day |
The per-token cost of £4.17/M is higher than the RTX 3090 (£3.36/M) because you are paying a premium for that 32 GB envelope. Batched at bs=8, the effective rate falls to around £2.60/M. The 5090 makes economic sense when you plan to use that spare VRAM — co-locating models, pushing longer contexts, or serving heavy concurrency. If Phi-3 Mini is your only workload, the 3090 or 5080 may be smarter buys. Compare everything in our tok/s benchmark tool.
Deployment
Quick start:
docker run --gpus all -p 8080:8080 ghcr.io/ggerganov/llama.cpp:server -m /models/phi-3-mini.Q4_K_M.gguf --host 0.0.0.0 --port 8080 -ngl 99Full setup walkthrough in our Phi-3 hosting guide. See also: best GPU for LLM inference, cheapest GPU for AI, and the complete benchmark archive.
Triple-Digit Phi-3 Speed on the RTX 5090
100 tok/s, 32 GB VRAM, flat monthly pricing. UK datacentre with full root access.
Build Your RTX 5090 Server
