Can the RTX 5090 Run Mixtral 8x7B?
Only at INT4. Mixtral 8x7B needs 94 GB at FP16, which doesn’t fit any single GPU we host. AWQ-INT4 (~26 GB) fits the 5090’s 32 GB with comfortable KV cache headroom.
Yes — at INT4The RTX 5090 (32 GB) does not fit Mixtral 8x7B at FP16 (94 GB needed), but runs comfortably at INT4 / AWQ (26 GB).
Detailed Breakdown
Mixtral 8x7B is a Mixture-of-Experts model — 8 experts of 7B parameters each, with 2 active per token. The full 47B parameter footprint needs 94 GB at FP16 because all experts have to be resident in VRAM. The RTX 5090 has 32 GB, so FP16 fails by a wide margin. Quantisation is the only path on a single 5090.
- Mixtral 8x7B FP16 — 94 GB needed. Doesn’t fit.
- Mixtral 8x7B FP8 — 47 GB needed. Doesn’t fit.
- Mixtral 8x7B AWQ-INT4 — 26 GB. Fits with ~6 GB KV cache room.
- Mixtral 8x7B GPTQ-INT4 — 26 GB. Also fits.
- Mixtral 8x22B — 282 GB FP16. Multi-GPU only.
For quality-critical Mixtral deployments, use a 6000 Pro 96 GB (FP16 fits) or 2× RTX 5090 in tensor parallel.
Frequently Asked Questions
The questions buyers actually ask before committing to a GPU server.
How much quality drop with INT4?
Marginal — typically <2% on reasoning benchmarks for AWQ-INT4 vs FP16.
Throughput on a 5090 INT4?
Roughly 280 tok/s aggregate, 25 tok/s single-stream. The MoE architecture means active params are only 12.9B, so it’s faster per token than dense 47B.
Why not just use Mistral 7B?
Mixtral has higher quality on hard reasoning tasks. If your workload doesn’t need that, Mistral 7B is cheaper and faster.
Related Pages
Pages our visitors typically read next.
Ready to deploy?
Same-day deployment on in-stock GPUs. Talk to a specialist who actually understands your workload.
