Table of Contents
Codestral 22B is Mistral’s coding-specialised model — strong on Python, MIT licensed, 32K context. At 22B dense parameters it’s borderline for a 24 GB GPU. Here’s the practical deployment.
TL;DR
Codestral 22B fits a 4090 24 GB only at AWQ-INT4 (~12 GB weights), and even then KV cache budget is tight. ~310 tok/s aggregate. Workable but not comfortable. For real production, the 5090 32 GB is the right home.
VRAM fit
| Precision | Weights | + KV (8K, 8 streams) | Total | Fits 24 GB? |
|---|---|---|---|---|
| FP16 | 44 GB | n/a | n/a | No |
| FP8 (no native on Ada) | 22 GB | +4 GB | 26 GB | No (no FP8 hw) |
| AWQ-INT4 | 12 GB | +4 GB | 16 GB | Yes |
| GGUF Q5_K_M | 15 GB | +4 GB | 19 GB | Yes |
| AWQ-INT4 + 16 concurrent streams | 12 GB | +8 GB | 20 GB | Tight, watch OOM |
vLLM config
vllm serve mistralai/Codestral-22B-v0.1 \
--quantization awq_marlin \
--max-model-len 16384 \
--max-num-seqs 8 \
--gpu-memory-utilization 0.93 \
--kv-cache-dtype fp8_e4m3 \
--enable-prefix-caching \
--served-model-name codestral \
--host 0.0.0.0 --port 8000Note --max-num-seqs 8 — keep concurrency conservative; 22B at INT4 with KV cache OOMs more easily than smaller models.
Performance
| Metric | RTX 4090 (Codestral 22B AWQ) | RTX 5090 (FP8) |
|---|---|---|
| Aggregate tok/s @ 8 concurrent | ~310 | ~620 |
| Single-stream tok/s | ~22 | ~52 |
| p99 TTFT | ~890 ms | ~410 ms |
| Cost per 1M (60% util) | £0.40 | £0.20 |
Upgrade signals
- p99 TTFT regularly exceeds 1.5 s — the 4090 is queueing
- You want to run more than ~10 concurrent users
- You want FP8 weights (not available on Ada)
- You want 32K+ context with comfortable concurrency
Bottom line
The 4090 24 GB hosts Codestral 22B at INT4, but not comfortably. For production at any meaningful scale, the 5090 is the better choice. For broader sizing see cost to run an AI coding assistant.
