RTX 4090 24GB Mistral 7B v0.3 Benchmark: Full Quant Sweep and Llama Comparison

Mistral 7B v0.3 remains a popular base for fine-tuning and a useful throughput reference point. On the RTX 4090 24GB running on a Gigagpu UK dedicated host, FP8 weights decode at 215 t/s – slightly faster than Llama 3.1 8B because Mistral has fewer hidden parameters per layer despite a similar parameter count, and aggregate throughput hits 1,260 t/s at batch 32 with comfortable VRAM headroom. This benchmark sweeps every viable quantisation format (FP16, FP8, AWQ-Marlin, GPTQ-Marlin, EXL2, GGUF Q4/Q5), every batch size from 1 to 64, every context length from 256 to 32k, and produces both aggregate and per-user latency numbers with p50 and p99 breakdowns. We also include a head-to-head comparison against Llama 3.1 8B at equivalent context and a cross-card comparison so you can pick the right model and the right card for your workload.

Contents

• Methodology and test rig
• Memory budget by quant
• Decode throughput sweep
• Concurrency and per-user latency
• TTFT vs prompt length
• Mistral 7B vs Llama 3.1 8B
• Cross-card comparison
• Production gotchas
• Verdict and recommended config

Methodology and test rig

All measurements were captured on a single-tenant Gigagpu node: RTX 4090 24GB Founders Edition at stock 450 W TDP, Ryzen 9 7950X with 64 GB DDR5-5600, Samsung 990 Pro 2TB Gen 4 NVMe with the model files mounted on a dedicated dataset; Ubuntu 24.04 LTS, NVIDIA driver 560.x, CUDA 12.6, vLLM 0.6.4, llama.cpp build b3500 for GGUF, ExLlamaV2 0.2.4 for EXL2, PyTorch 2.5, FlashAttention 2.6. Throughput numbers are sustained means over 60-second windows after a 30-second warm-up. Concurrency was driven by an asyncio harness wrapped around vLLM’s benchmark_throughput.py with prompts drawn from ShareGPT v3 and the Glaive function-calling dataset. Tail latencies are p50 and p99 over 1,000 samples per batch size. Mistral 7B v0.3’s GQA layout (32 layers, 32 query heads, 8 KV heads, 14336 intermediate, 32k native context window) is structurally identical to Llama 3.1 8B except for fewer hidden parameters per layer, which is why FP8 attention kernels apply unchanged. See our vLLM setup guide for the full installation walkthrough.

Memory budget by quant

Decode throughput sweep

Per-quant single-stream and aggregate decode

Aggregate batch sweep, FP8 + FP8 KV

Concurrency and per-user latency

TTFT vs prompt length

Mistral 7B vs Llama 3.1 8B

Mistral wins ~10% on raw throughput; Llama wins decisively on quality across MMLU, HumanEval, GSM8K and MT-Bench. For high-volume embedding generation, classification jobs, and Mistral-tuned LoRA stacks, Mistral 7B remains compelling. For assistants, RAG, and any workload where output quality matters, Llama 3.1 8B is the better default. See the Llama 3 8B benchmark for full numbers.

Cross-card comparison

The 4090 is the strict sweet spot for self-hosted Mistral 7B in the UK. See 4090 vs 5090 and 5060 Ti benchmark for the wider patterns.

Production gotchas

1. Mistral 7B v0.3 has a v3 tokeniser. The v0.1 and v0.2 tokenisers will not work with v0.3 weights; verify you are using the right one.
2. The 32k context is real but expensive. Mistral 7B was trained with sliding-window attention up to 32k; vLLM applies full attention which is more accurate but eats more KV. Plan VRAM accordingly.
3. HumanEval is genuinely poor. If your workload involves code, do not pick Mistral 7B; the 30.5 score is not a benchmark artefact, it reflects real-world coding quality.
4. Mistral instruct templates differ from Llama. The [INST] wrapper is required; missing it gives garbage outputs that look fluent.
5. EXL2 4.0bpw is the speed champion but loses MT-Bench quality. AWQ-Marlin is the better quality/speed tradeoff for 7B.
6. Aggregate throughput plateaus around batch 32. Going from 32 to 64 only gains 30 t/s; the per-user latency cost is severe.
7. llama.cpp does not scale past batch 8. For multi-user serving use vLLM with FP8 or AWQ.

Verdict and recommended config

For batched embedding/classification workloads at maximum throughput use FP8: --quantization fp8 --kv-cache-dtype fp8_e4m3 --max-model-len 32768 --max-num-seqs 32 --enable-chunked-prefill --enable-prefix-caching. For lowest single-user latency use AWQ-Marlin: --quantization awq_marlin --kv-cache-dtype fp8_e4m3 --max-model-len 32768 --max-num-seqs 16. For any workload where output quality matters, prefer Llama 3.1 8B over Mistral 7B at the modest 8% throughput cost. Mistral’s GQA layout is identical to Llama’s, so all the FP8 attention kernels apply identically.

See also: Llama 3 8B benchmark, Mistral 7B model guide, vLLM setup, FP8 deployment, AWQ guide, tokens per watt, concurrent users, prefill/decode, 4090 vs 5090.