Qwen 2.5 32B is genuinely strong on reasoning — better than Mixtral 8x7B and approaching Llama 3 70B on some benchmarks. The deployment options are constrained.
TL;DR
Qwen 2.5 32B at AWQ-INT4 fits a single RTX 5090 32 GB. At FP8 needs RTX 6000 Pro 96 GB. INT4 is the cost-leading deployment at ~£0.30/1M tokens.
Hardware
| Config | Cost per 1M tokens | Notes |
|---|---|---|
| 1× RTX 5090 AWQ-INT4 | £0.30 | Cost leader |
| 1× RTX 6000 Pro FP8 | £1.05 | Best quality, simplest ops |
| 2× RTX 5090 FP8 (TP=2) | £0.65 | High quality, multi-card complexity |
Config
vllm serve Qwen/Qwen2.5-32B-Instruct-AWQ \
--quantization awq_marlin \
--max-model-len 32768 \
--gpu-memory-utilization 0.92 \
--enable-prefix-cachingVerdict
For Qwen 2.5 32B specifically, RTX 5090 AWQ-INT4 is the right pick — cheapest, fits cleanly, single card.
Bottom line
32B INT4 on 5090 = best per-pound. See Qwen 2.5 32B VRAM.
