Sixty-two tokens per second from a 3.8B model. That is the headline figure when you pair Microsoft’s Phi-3 Mini with the RTX 3090, and it changes the economics of small-model deployment significantly. We ran the numbers on a GigaGPU dedicated server so you do not have to guess.
Raw Throughput
| Metric | Value |
|---|---|
| Tokens/sec (single stream) | 62 tok/s |
| Tokens/sec (batched, bs=8) | 99.2 tok/s |
| Per-token latency | 16.1 ms |
| Precision | FP16 |
| Quantisation | FP16 |
| Max context length | 16K |
| Performance rating | Excellent |
Tested at 512-token prompt / 256-token completion, single-stream, llama.cpp backend. The 3090’s 936 GB/s memory bandwidth is the key enabler here — Phi-3 Mini is small enough that inference becomes almost entirely memory-bound, and the 3090’s bus keeps tokens flowing.
Memory Footprint
| Component | VRAM |
|---|---|
| Model weights (FP16) | 8.0 GB |
| KV cache + runtime | ~1.2 GB |
| Total RTX 3090 VRAM | 24 GB |
| Free headroom | ~16.0 GB |
Sixteen gigabytes of unused VRAM is unusual for an inference workload. That surplus means you can push context to 16K, batch aggressively, or even co-host a second model (for example, a Whisper instance for a voice-to-text pipeline) on the same card.
Cost Breakdown
| Cost Metric | Value |
|---|---|
| Server cost | £0.75/hr (£149/mo) |
| Cost per 1M tokens | £3.360 |
| Tokens per £1 | 297,619 |
| Break-even vs API | ~1 req/day |
Under £3.36 per million tokens single-stream, falling to about £2.10/M when batching at bs=8. At that rate, the RTX 3090 is one of the most cost-effective ways to self-host any small LLM. Our benchmark comparison tool visualises how this stacks up against every other GPU tier.
Production Suitability
With 62 tok/s single-stream and nearly 100 tok/s batched, this setup comfortably handles production chat APIs, real-time summarisation, and multi-turn assistant workloads. The 16.1 ms per-token latency keeps responses feeling instantaneous to end users. If your workload is Phi-3 Mini and you need reliability over raw peak speed, the 3090 is arguably the sweet spot in the GigaGPU lineup.
One-liner to get running:
docker run --gpus all -p 8080:8080 ghcr.io/ggerganov/llama.cpp:server -m /models/phi-3-mini.Q4_K_M.gguf --host 0.0.0.0 --port 8080 -ngl 99Read the full Phi-3 hosting guide for tuning tips. Also worth checking: best GPU for LLM inference, cheapest GPU for AI, and all benchmark data.
Phi-3 Mini at 62 tok/s — Ready to Ship
Production-grade throughput on dedicated hardware. UK datacentre, flat monthly rate.
Provision an RTX 3090
