RTX 4090 24GB for Qwen 2.5 Coder 32B: GPT-4o-Tier Code Completion on One Card

Qwen 2.5 Coder 32B-Instruct is the open-weight code model that finally crossed the line on standard benchmarks: HumanEval 92.7, MBPP 87.0, MultiPL-E 78.5 across 18 languages. Numbers in the same neighbourhood as GPT-4o on the public leaderboards. The fact that it runs on a single RTX 4090 24GB via AWQ INT4 quantisation – no multi-GPU rig, no offloading – is the headline for self-hosted teams. This guide on our UK GPU hosting covers the architecture, the exact VRAM math that makes the 18 GB AWQ fit work, throughput trade-offs, the seven gotchas you will hit in production, and the decision logic for picking Coder 32B over the 14B sibling or other open-weight coders.

Contents

• Why a 32B coder on a 24 GB card matters
• Architecture, FIM and KV math
• VRAM budget breakdown
• Throughput, latency and concurrency limits
• Quality vs CodeLlama 70B and GPT-4o
• vLLM deployment and FIM endpoint
• Fine-tuning notes
• Production gotchas (seven items)
• When to pick this over alternatives
• Verdict

Why a 32B coder on a 24 GB card matters

For two years the rule was simple: top-tier code completion meant 70B+ models or proprietary APIs. Qwen 2.5 Coder 32B breaks that. AWQ INT4 weights are 18 GB, FP8 KV cache adds about 2 GB at 16k context, leaving roughly 4 GB for activations, vLLM scheduler state and CUDA workspace on a 24 GB card. It is the tightest production-grade fit on a consumer GPU – and the first time a single 4090 can host a model that matches the headline open-source benchmark numbers of GPT-4o (HumanEval 92.7 vs 92.1).

The trade-off is concurrency. The 14B sibling handles 16 concurrent IDE clients comfortably; the 32B handles 4. So Coder 32B is the right pick for a small senior team where every completion needs to be high-quality, or for batch workloads (pull-request review, refactor generation) where latency is less critical. For mass IDE deployment across 12+ engineers, drop to Coder 14B.

Architecture, FIM and KV math

Qwen 2.5 Coder 32B is architecturally identical to Qwen 2.5 32B-Instruct: 64 transformer layers, 40 query heads, 8 KV heads (5:1 GQA), 128-dim heads, SwiGLU MLP at 27,648 intermediate dim, 152,064 vocabulary, native 32,768 context. The differentiation is the training data: 5.5T tokens of code-heavy text spanning 80+ programming languages with FIM supervision via <|fim_prefix|>, <|fim_suffix|>, <|fim_middle|> tokens. License is the Qwen Community License (Apache-2.0-equivalent under 100M MAU).

KV cache per token at FP16: 8 KV heads x 128 dim x 64 layers x 2 (K+V) x 2 bytes = 262 KB. At FP8 KV: 131 KB. So 16k context = 2.1 GB FP8 per sequence; 32k = 4.2 GB. The 16k ceiling for production is forced by the budget arithmetic, not by model quality – Coder 32B handles 32k cleanly when you can afford the KV memory.

VRAM budget breakdown

For coding-assistant workloads, 16k context is usually plenty – whole-file edits and multi-file context typically fit within that envelope. See the Qwen 2.5 32B VRAM breakdown for the full per-precision tables.

Throughput, latency and concurrency limits

vLLM 0.6.3, AWQ Marlin, FP8 KV, eager mode, prompt 1024, generate 256:

The aggregate plateau at batch 4 is the hard limit on this card – past that, KV pressure forces preemption and effective per-stream throughput collapses. For higher concurrency, drop to Coder 14B or step up to a 5090 32GB. For dedicated benchmark detail with concurrency curves see the Qwen 32B benchmark.

Quality vs CodeLlama 70B and GPT-4o

Qwen 2.5 Coder 32B AWQ matches GPT-4o on HumanEval and beats CodeLlama 70B by a wide margin – on a fraction of the hardware. The LiveCodeBench gap (31.4 vs 34.7) is real – GPT-4o still wins on novel competitive programming problems where reasoning depth matters more than pattern matching, but on day-to-day completion, refactoring and bug-fixing tasks the gap is unmeasurable. Quantisation loss measured on these benchmarks is under 0.6 points vs the FP16 reference.

vLLM deployment and FIM endpoint

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-Coder-32B-Instruct-AWQ \
  --quantization awq_marlin \
  --kv-cache-dtype fp8 \
  --max-model-len 16384 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.95 \
  --enforce-eager \
  --enable-prefix-caching

Three flags are critical. --enforce-eager avoids the 1.5 GB CUDA-graph memory premium – mandatory at 22 GB used. --gpu-memory-utilization 0.95 pushes vLLM to use the full envelope; the default 0.90 leaves 1.2 GB unused which you cannot afford. --max-num-seqs 4 caps concurrency at the safe ceiling – go higher and you hit NoFreeBlocks errors under load.

FIM mode endpoint – the same as Coder 14B but with the larger weights:

def fim_prompt(prefix: str, suffix: str) -> str:
    return f"<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>"

import openai
client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="x")
resp = client.completions.create(
    model="Qwen/Qwen2.5-Coder-32B-Instruct-AWQ",
    prompt=fim_prompt(prefix_code, suffix_code),
    max_tokens=128,
    temperature=0.2,
    stop=["<|endoftext|>", "<|fim_pad|>"]
)
print(resp.choices[0].text)

See the AWQ quantization guide for AutoAWQ vs llm-compressor build choices and the vLLM setup guide for the full toolchain.

Fine-tuning notes

QLoRA on a single 4090 is feasible at sequence length 2048, batch 1, gradient accumulation 16, rank 16. Expect 32 hours for a 50,000-sample SFT pass over diff-style patches – twice as long as Coder 14B. For longer sequences (4k+), drop the rank to 8 or move to multi-GPU. Full fine-tuning needs ~250 GB of optimiser-state memory; rent multiple H100s. See our QLoRA guide and the LoRA fine-tuning analysis. For most teams, fine-tuning Coder 14B and using Coder 32B as a stronger base is the better economic split.

Production gotchas (seven items)

• Eager mode is mandatory at 16k+: CUDA graphs add 1.5 GB overhead. With weights at 18 GB and KV at 2 GB you cannot afford them. Throughput penalty is ~6%.
• Cap max-num-seqs at 4: above 4 concurrent sequences, KV pressure forces preemption and per-stream throughput collapses. Aggregate stays roughly flat.
• Disable swap and zram on the host: OOM here is fatal not graceful. The kernel OOM killer can take down vLLM mid-request and the recovery time (45 sec cold start) is unacceptable for IDE traffic.
• Pin the model in HBM with a warmup request: AWQ weight load is 30 seconds, eager-mode warmup another 20. Pre-warm before exposing to traffic; first-after-cold-start completion is otherwise 50+ seconds.
• Set VLLM_USE_V1=1 if available: the v1 engine is more memory-frugal and gives you another 200-400 MB of headroom that translates directly to longer context or one more concurrent stream.
• YaRN extension to 131k is not production-grade: long-context retrieval drops sharply past 32k on the 32B base. Use 16k native; chunk-summarise larger files.
• Power and thermal headroom: at sustained batch 4 the 4090 will draw 440-450 W. Cap to 410 W with nvidia-smi -pl 410 for stable numbers – throughput loss is under 4%, see power draw analysis.

When to pick this over alternatives

Pick Coder 32B over Coder 14B when raw quality on long, hard tasks (whole-class refactors from natural-language specs, complex multi-file changes, novel algorithm implementation) is the headline and you can tolerate 4-stream concurrency. Pick it over Codestral 22B on every benchmark – Coder 32B wins HumanEval 92.7 vs 81.1 and the only reason to take Codestral is licensing simplicity for permissively-licensed corporate code. Pick it over DeepSeek Coder V2 Lite when raw quality matters; Lite is faster but 11 points behind on HumanEval.

Step down to Coder 14B for high-concurrency IDE deployment (12+ engineers). Step up to Llama 3.1 70B INT4 across two cards if you need MATH-heavy reasoning. Switch to Qwen 2.5 32B-Instruct (non-coder variant) if your workload is mixed code + general reasoning.

Verdict

Qwen 2.5 Coder 32B AWQ on the RTX 4090 24GB is the most capable single-card open-weight code model in 2026. HumanEval 92.7 matches GPT-4o on the public benchmark; MBPP 87.0 and MultiPL-E 78.5 are best-in-class for an open-weight model that fits a consumer GPU. The trade-offs are real: 65 t/s decode, 4-stream concurrency cap, 16k context ceiling for production, eager mode mandatory, 45-second cold start. For a small senior team where every completion must be high quality, an async PR-review pipeline, or a research lab doing serious code-AI work, this is the default 4090 deployment. For mass IDE rollouts pick Coder 14B instead. Compare with the 5090 upgrade decision if you need higher concurrency or want FP8 32B headroom.

See also: Qwen Coder 14B, Qwen 2.5 32B base, Codestral 22B, DeepSeek Coder V2 Lite, coding assistant deployment, monthly hosting cost, vs OpenAI API cost.