Mixtral 8x7B and the RTX 5090 are a natural pairing. The 5090’s 32 GB VRAM comfortably fits the 4-bit MoE model with room to spare, and the Blackwell-gen bandwidth pushes throughput to 45 tok/s — the fastest single-GPU Mixtral result in our testing. Here are the full numbers from GigaGPU dedicated hardware.
Full Benchmark Results
| Metric | Value |
|---|---|
| Tokens/sec (single stream) | 45 tok/s |
| Tokens/sec (batched, bs=8) | 72.0 tok/s |
| Per-token latency | 22.2 ms |
| Precision | INT4 |
| Quantisation | 4-bit GGUF Q4_K_M |
| Max context length | 32K |
| Performance rating | Very Good |
512-token prompt, 256-token completion, single-stream, llama.cpp Q4_K_M. The 5090 achieves Mixtral’s full native 32K context window — something neither the 3090 nor 5080 can manage at this quantisation level.
Memory Picture
| Component | VRAM |
|---|---|
| Model weights (4-bit GGUF Q4_K_M) | 26 GB |
| KV cache + runtime | ~3.9 GB |
| Total RTX 5090 VRAM | 32 GB |
| Free headroom | ~6.0 GB |
Six gigabytes of headroom after loading — enough for Mixtral’s generous 32K context, and potentially room for a small secondary model or embedding layer. Unlike the 3090 setup (1 GB free) or the 5080 (which requires offloading), the 5090 keeps the entire model on-GPU with genuine breathing room.
Cost Efficiency
| Cost Metric | Value |
|---|---|
| Server cost | £1.50/hr (£299/mo) |
| Cost per 1M tokens | £9.259 |
| Tokens per £1 | 108,003 |
| Break-even vs API | ~1 req/day |
£9.26/M single-stream, approximately £5.79/M batched. While pricier per-token than running a dense 7B model, this is Mixtral territory — you are paying for multi-expert reasoning quality that smaller models simply cannot match. At moderate to high volume, self-hosting comfortably undercuts commercial Mixtral API pricing. Check precise breakpoints in the cost-per-million-tokens tool.
Our Assessment
This is the best single-GPU Mixtral 8x7B configuration we have tested. Forty-five tok/s at 32K context opens up production use cases that were impossible on lesser hardware: long-document analysis, complex multi-turn agents, and function-calling workloads that benefit from Mixtral’s routing diversity. If MoE is part of your stack, the 5090 is where Mixtral comes alive.
Get running:
docker run --gpus all -p 8080:8080 ghcr.io/ggerganov/llama.cpp:server -m /models/mixtral-8x7b.Q4_K_M.gguf --host 0.0.0.0 --port 8080 -ngl 99Setup details: Mixtral hosting guide. More reading: best GPU for LLM inference, tok/s benchmark, all results.
Mixtral 8x7B at Full 32K Context — RTX 5090
45 tok/s, no offloading, no compromises. UK datacentre, dedicated hardware.
Build Your 5090 Server
