Table of Contents
Qwen 2.5 32B is the awkward middle child of the Qwen lineup — meaningfully stronger than 14B on reasoning, dramatically smaller than 72B, but with a VRAM profile that doesn’t fit on a single consumer GPU at FP16. This page is the precise sizing reference we use when customers ask "will it fit?" for a Qwen 2.5 32B deployment.
Qwen 2.5 32B is ~64 GB at FP16, ~32 GB at FP8, ~18 GB at AWQ-INT4. Plus 2–8 GB of KV cache depending on context. Practical homes: RTX 6000 Pro 96 GB at FP16 or FP8, single RTX 5090 32 GB at INT4, or 2× consumer GPUs in tensor-parallel for FP8.
Headline numbers
| Component | FP16 (BF16) | FP8 | AWQ-INT4 | GGUF Q5_K_M |
|---|---|---|---|---|
| Weights | ~63.5 GB | ~32 GB | ~18 GB | ~22 GB |
| Activations (batch=1) | ~1.5 GB | ~1 GB | ~0.8 GB | ~0.8 GB |
| KV cache @ 8K context | ~2.5 GB | ~1.5 GB | ~1.5 GB | ~1.5 GB |
| KV cache @ 32K context | ~10 GB | ~5 GB | ~5 GB | ~5 GB |
| Minimum VRAM (8K context) | ~67 GB | ~34 GB | ~20 GB | ~24 GB |
32.5B parameters, 64 transformer layers. KV cache uses FP16 in vLLM regardless of weight precision.
Where the VRAM actually goes
Three buckets dominate VRAM for any LLM:
- Model weights. 32.5B parameters × bytes-per-parameter. FP16 = 2 bytes, FP8 = 1 byte, INT4 = 0.5 bytes plus a small per-group scale tensor.
- Activations. Forward-pass intermediate tensors. For inference these are small — a few hundred MB to ~1 GB at modest batch sizes.
- KV cache. The keys and values for every previous token in every attention head. This is the silent killer of long-context deployments. For a 32B model with 64 layers and 64-head attention, the KV cache is approximately
context_length × 200 KBper concurrent request.
Most VRAM-OOM crashes in the wild are KV-cache-driven. The model itself fits; the cache fills up under load.
VRAM by precision
FP16 / BF16 — "reference"
~63.5 GB of weights. Plus 8 GB of cache for a 32K context. Total ~72 GB. Fits on a single A100 80 GB or RTX 6000 Pro 96 GB. Does not fit any consumer GPU.
FP8 — production-grade
~32 GB of weights. Plus the same KV cache budget (vLLM keeps cache in FP16 for accuracy). Fits comfortably on the 6000 Pro with ~50 GB of headroom for cache and concurrent batching. On a 2× RTX 5090 cluster, FP8 needs ~16 GB per card before cache — comfortable in tensor-parallel.
AWQ-INT4 — single consumer GPU
~18 GB of weights. Fits a single RTX 5090 32 GB with ~10 GB of headroom for cache. The right pick when you want one card to host this model and have low concurrency demands.
GGUF Q5_K_M — llama.cpp deployments
~22 GB. Slightly larger than AWQ-INT4 but with better quality on edge-case prompts. Useful when you’re running with llama.cpp instead of vLLM.
KV cache by context length
Per concurrent request:
| Context length | KV cache (FP16) | KV cache (FP8) |
|---|---|---|
| 2K | 0.6 GB | 0.4 GB |
| 8K | 2.5 GB | 1.5 GB |
| 16K | 5.0 GB | 3.0 GB |
| 32K | 10.0 GB | 5.0 GB |
| 64K (YARN) | 20.0 GB | 10.0 GB |
| 128K (YARN) | 40.0 GB | 20.0 GB |
Per request. For 10 concurrent users at 32K context on FP16-cache vLLM, that’s 100 GB of KV alone — only fits on multi-GPU at scale.
vLLM 0.6.5 introduced FP8 KV cache as a beta feature. Halves the cache footprint at a small accuracy cost. For long-context heavy workloads on a single 6000 Pro it’s worth enabling: --kv-cache-dtype fp8_e4m3.
Which GPU fits which deployment
| GPU | VRAM | FP16 weights | FP8 weights | INT4 weights | Verdict |
|---|---|---|---|---|---|
| RTX 3050 6 GB | 6 GB | No | No | No | Too small |
| RTX 4060 8 GB | 8 GB | No | No | No | Too small |
| RTX 3060 12 GB | 12 GB | No | No | No | Too small even at INT4 |
| RTX 5060 Ti 16 GB | 16 GB | No | No | Tight | 14B is a better fit |
| RTX 5080 | 16 GB | No | No | Tight | 14B is a better fit |
| RTX 3090 | 24 GB | No | No | Yes | INT4 only, comfortable |
| RTX 4090 | 24 GB | No | No | Yes | INT4 only, comfortable |
| RTX 5090 | 32 GB | No | Tight | Yes | INT4 best, FP8 needs sharding |
| A100 80 GB | 80 GB | Yes | Yes | Yes | Reference card |
| RTX 6000 Pro 96 GB | 96 GB | Yes | Yes | Yes | Best single-card |
Real-world deployments we run
From our customer base, the four common deployments of Qwen 2.5 32B:
1. Single RTX 5090 + AWQ-INT4 (most common)
Cheapest deployment. ~28 tok/s single-stream, ~280 tok/s aggregate. Fine for a B2B chatbot or internal tooling.
2. Single RTX 6000 Pro + FP8
Production tier. ~38 tok/s single-stream, ~480 tok/s aggregate. The right home if you want minimal regression vs FP16 and have budget.
3. 2× RTX 5090 (TP=2) + FP8
Highest single-server throughput. ~45 tok/s single-stream, ~620 tok/s aggregate. Worth it only if you’ll genuinely use the throughput.
4. 2× A100 80 GB + FP16
Reference / regulated workloads. ~38 tok/s single-stream, ~440 tok/s aggregate. Pick this when ECC and certified drivers are required.
When 32B is too big and you should drop to 14B
Qwen 2.5 32B beats 14B on roughly the right tasks: multi-step reasoning, code generation on novel problems, and long-context coherence. It does not meaningfully beat 14B on:
- Standard chatbot Q&A with retrieval grounding
- Translation
- Classification / sentiment / extraction
- Most function-calling workflows
If your evaluation harness shows <3% improvement going from 14B to 32B, the smaller model is the right deployment. Best GPU for Qwen 2.5 walks through the 14B sizing in detail; can RTX 5090 run Qwen? covers the most common single-card deployment.
Bottom line
For 32B specifically: pick RTX 5090 32 GB at AWQ-INT4 if cost-driven, RTX 6000 Pro 96 GB at FP8 if quality-driven, and a multi-GPU configuration only when single-card throughput becomes the bottleneck.
