RTX 4090 24GB for Codestral 22B: Mistral’s Code Specialist on a Single Card

Mistral Codestral 22B is the company’s purpose-built code model: dense 22B parameters, native fill-in-the-middle support, 80+ programming languages, 32k context, released May 2024 under the Mistral Non-Production Licence (free for non-commercial use, commercial licence required for production). AWQ INT4 brings weights to roughly 13.4 GB – one of the most comfortable single-card fits for a 22B-class flagship code model on the RTX 4090 24GB. This guide on our UK dedicated GPU hosting covers architecture, the FP8/AWQ trade-offs, throughput, FIM endpoint specifics (Codestral uses an unusual inverted format), the licensing caveat that you must understand before deploying commercially, and where Codestral fits against Qwen 2.5 Coder 14B/32B.

Contents

• Codestral in brief, including the licence caveat
• Architecture, FIM and KV math
• VRAM accounting across precisions
• Throughput, latency and concurrency
• Quality benchmarks vs alternatives
• vLLM deployment and FIM endpoint
• Production gotchas (seven items)
• When to pick this over alternatives
• Verdict

Codestral in brief, including the licence caveat

Codestral 22B is Mistral’s flagship code-specialist model and the only Mistral model in this size class that is built specifically for code rather than adapted from a general-purpose base. It posts HumanEval 81.1, MBPP 78.2, MultiPL-E 69.0 across 80+ languages, and is particularly strong on JavaScript, Python, Java, C++ and SQL. Native context is 32k tokens.

The licensing matters. Codestral is released under the Mistral Non-Production Licence (MNPL), not Apache 2.0. You can self-host it for personal use, research, evaluation and non-commercial work freely, but commercial production deployment requires a paid commercial licence from Mistral. For internal company tools at a startup or enterprise, this is a real cost – typically tens of thousands of dollars per year for an unlimited internal-use licence. If your use case is commercial production and licence cost is a blocker, switch to Qwen 2.5 Coder 14B (Apache 2.0 equivalent) which beats Codestral on every benchmark anyway.

Architecture, FIM and KV math

Codestral 22B is a dense 56-layer transformer with 48 query heads, 8 key/value heads (6:1 GQA), 128-dim heads, an SwiGLU MLP at 16,384 intermediate dim, and a 32k vocabulary tokenizer (smaller than Llama 3’s 128k – a deliberate choice for code where vocabulary efficiency matters less). FIM is supported via the unusual inverted format: [SUFFIX]{after_cursor}[PREFIX]{before_cursor}, where the model fills in the middle. This is opposite to most other coder models which use [PREFIX]{before}[SUFFIX]{after}[MIDDLE]. Get the order wrong and the model produces gibberish.

KV cache per token at FP16: 8 KV heads x 128 dim x 56 layers x 2 (K+V) x 2 bytes = 229 KB. At FP8 KV: 114.7 KB. So 32k context = 3.67 GB FP8 per sequence – similar to Mistral Small 3 24B (same architecture style).

VRAM accounting across precisions

FP8 weights at 22 GB plus KV plus overhead exceed 24 GB – so on the 4090 it is AWQ only. Note that this is identical to Mistral Small 3’s situation; both 22-24B Mistral models on the 4090 require AWQ INT4. Codestral has 56 attention heads with KV cost moderate enough that 32k context with FP8 KV keeps roughly 4 GB free for activations and small batch concurrency.

Throughput, latency and concurrency

vLLM 0.6.3, AWQ Marlin, FP8 KV, prompt 1024, generate 256:

Codestral’s 80 t/s decode at AWQ is broadly comparable to Mistral Small 3 24B (85 t/s) – the 22B vs 24B param count and the slightly lighter SwiGLU intermediate dim explain the small gap. For RepoBench-style real-repo workloads, see the wider coding assistant deployment notes.

Quality benchmarks vs alternatives

Codestral’s distinctive edge is RepoBench – real-world repo-level FIM where it leads the 14B class – but it trails Qwen 2.5 Coder 14B on standalone benchmarks (HumanEval 81.1 vs 83.5) and Qwen Coder 32B on everything. Choose based on workload: real-repo completion (Codestral wins) vs benchmark-style problems (Qwen wins).

vLLM deployment and FIM endpoint

python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Codestral-22B-v0.1-AWQ \
  --quantization awq_marlin \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.94 \
  --enable-prefix-caching

Codestral FIM uses the inverted Mistral format. The completions-endpoint pattern:

# Codestral FIM is INVERTED vs most other coders
def codestral_fim(prefix: str, suffix: str) -> str:
    return f"[SUFFIX]{suffix}[PREFIX]{prefix}"

import openai
client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="x")
resp = client.completions.create(
    model="TheBloke/Codestral-22B-v0.1-AWQ",
    prompt=codestral_fim(
        prefix="def fibonacci(n):\n    if n < 2:\n        return n\n    ",
        suffix="\n\nprint(fibonacci(10))"
    ),
    max_tokens=64,
    temperature=0.2,
    stop=["</s>"]
)
print(resp.choices[0].text)

Note the inverted [SUFFIX] before [PREFIX] order – this is opposite to Qwen, DeepSeek and most other coders, and getting it wrong produces gibberish without an obvious error. For instruction (chat) mode use the standard Mistral <s>[INST]...[/INST] wrapping. See our coding assistant guide for client-template details and the vLLM setup notes for toolchain prerequisites.

Production gotchas (seven items)

• Mistral Non-Production Licence applies: free for personal/research/evaluation use, paid commercial licence required for production deployment. If commercial licence cost is a blocker, switch to Qwen 2.5 Coder 14B which is Apache-equivalent and benchmarks better.
• FIM order is inverted: [SUFFIX]{after}[PREFIX]{before} not the standard order. Many editor plugins assume the standard order and silently produce garbage with Codestral. Always test with a known-good completion before rolling out to engineers.
• FP8 will not fit on a 24 GB card: 22 GB weights plus KV exceeds 24 GB. AWQ INT4 only on the 4090.
• No chat template auto-detect for legacy clients: older OpenAI-compatible clients may not apply the Mistral [INST] template correctly. Use --chat-template to pin the template explicitly in vLLM if you see degraded outputs.
• Tokenizer is smaller than Llama 3: 32k vocabulary vs 128k means more tokens per character of code in some cases. Budget your context-length cap accordingly.
• Prefix caching value is high: editor sessions reuse imports and surrounding context. Enable prefix caching – the hit rate is high and TTFT improves 50%+ for sustained editing.
• Power discipline at sustained load: continuous code completion keeps the 4090 at high utilisation. Cap to 410 W with nvidia-smi -pl 410, see power draw analysis.

When to pick this over alternatives

Pick Codestral 22B over Qwen 2.5 Coder 14B only when (a) you specifically prefer Mistral’s writing style for code generation, (b) you have a Mistral commercial licence already, or (c) the RepoBench FIM win matters for your real-repo workload. For green-field commercial deployment, Qwen 2.5 Coder 14B is the better default – Apache-equivalent licence, 60% higher throughput, comparable or better quality.

Step up to Qwen 2.5 Coder 32B AWQ for absolute peak code quality on a single 4090. Step laterally to DeepSeek Coder V2 Lite for higher throughput at comparable quality (Apache 2.0 too). Step down to Mistral Nemo 12B for general-purpose work with code as a secondary concern.

Verdict

Codestral 22B AWQ on the RTX 4090 24GB is a solid single-card code-specialist deployment – 80 t/s decode, 32k context, RepoBench leader for the 14-22B class. The MNPL licensing means it is the right pick for non-commercial research, internal evaluation, or organisations that already hold a Mistral commercial licence. For commercial green-field deployment, Qwen 2.5 Coder 14B is faster, cheaper to license, and benchmarks better – so most teams will land there instead. The inverted FIM format is a real production gotcha that bites integrations every release. Compare with the RTX 5090 32GB if you need higher concurrency.

See also: Qwen 2.5 Coder 14B, Qwen 2.5 Coder 32B, DeepSeek Coder V2 Lite, coding assistant deployment, Mistral Nemo 12B, AWQ quantization guide, vLLM setup.