RTX 4090 24GB for Phi-3 Medium 14B: 140 t/s in FP8 with deep deployment notes

Microsoft’s Phi-3 Medium 14B is the most under-rated dense reasoner in its weight class. Trained on a curated, synthetic-heavy “textbooks plus filtered web” corpus rather than a raw scrape, it earns 78.0 on MMLU and 90.0 on GSM8K despite being roughly the same parameter count as Llama 3 13B. The catch is its attention pattern: 40 layers and 40 attention heads with no grouped-query attention, which makes the KV cache unusually expensive per token. The good news is that on a single RTX 4090 24GB dedicated server with native 4th-gen FP8 tensor cores, Phi-3 Medium fits comfortably at 14 GB of weights and decodes at roughly 140 tokens per second. This guide on our UK GPU hosting walks through VRAM math, throughput, the 4k versus 128k variant trade-off, deployment, gotchas, and where Phi-3 sits among realistic alternatives.

Contents

• The model: architecture and licence
• VRAM math without GQA
• Throughput and concurrency
• 4k vs 128k variants
• vLLM deployment recipes
• Quality benchmarks and named scenarios
• Production gotchas
• Alternatives and verdict

The model: architecture and licence

Phi-3 Medium has 14B parameters spread across 40 transformer blocks. The hidden dimension is 5120, intermediate dimension 17920, and crucially it has 40 query heads with 40 KV heads (head_dim 128). That means no GQA: every query head has its own dedicated key and value projections. The model is released under MIT, which is unusually permissive for a 14B-class model and meaningfully easier for compliance review than Llama’s community licence.

Why the data recipe matters

Microsoft’s Phi line is famous for the “textbooks are all you need” thesis. The medium variant reuses that synthetic-heavy curriculum, which is why it punches well above weight on MMLU and grade-school maths but trails Qwen 2.5 14B on programming benchmarks where Qwen’s enormous code-data mix dominates. If you need a reasoner with strong sciences, schoolbook maths and structured logic, Phi-3 Medium is competitive with models twice its size.

VRAM math without GQA

For a transformer the KV cache size per token is 2 * num_layers * num_kv_heads * head_dim * bytes_per_element. With 40 layers, 40 KV heads, head_dim 128 and FP8 (1 byte) you get 2 * 40 * 40 * 128 * 1 = 409,600 bytes, which is 400 KB per token. That is roughly 4x what a GQA model like Mistral Nemo or Qwen 2.5 32B costs per token. Plan KV budgets accordingly.

The headline: FP8 weights at 14 GB are easy. Long-context Phi-3 with FP8 KV is what eats VRAM, because you pay 0.4 MB per token rather than the ~0.1 MB a GQA peer would charge. The AWQ quantisation guide and FP8 deployment guide are useful companion reads here.

Throughput and concurrency

Decode on the 4090 is bandwidth-bound for 14B-class models. With 14 GB of FP8 weights and 1008 GB/s of GDDR6X you can compute the upper bound on single-stream decode as 1008 / 14 = 72 weight-traversals per second. Real-world overheads (sampling, KV reads, kernel launches) bring practical decode to roughly 140 t/s, because the 4090’s L2 caches a meaningful fraction of weight bytes and FlashAttention 3 amortises the KV reads efficiently.

Above batch 8 the KV cache becomes the gating factor on Phi-3 because of the no-GQA tax, not the compute. Most production deployments cap --max-num-seqs around 16 with 4k contexts, or 8 with 8k contexts. Cross-reference with the Llama 3 8B benchmark to see how much cheaper a GQA model is at similar batch sizes.

4k vs 128k variants

Microsoft ships two instruct variants: phi-3-medium-4k-instruct with native 4096-token context, and phi-3-medium-128k-instruct with LongRoPE-extended 128k context. Both share the same weights for the first 4k of position embeddings; the 128k variant rescales rotary positions to extend the window. Quality on long-context retrieval (Needle in a Haystack) holds well to roughly 64k for the 128k variant, then degrades.

Practical advice: ship the 128k variant with --max-model-len 16384 for chat workloads, or 32768 for retrieval over medium documents. Going to full 128k on a single 4090 is not realistic without AWQ plus aggressive batch caps, and quality at that range is questionable anyway.

vLLM deployment recipes

Two reference launches that we run in production. The first is the chat-class FP8 deploy; the second is the AWQ long-context deploy.

python -m vllm.entrypoints.openai.api_server \
  --model microsoft/Phi-3-medium-4k-instruct \
  --quantization fp8 --kv-cache-dtype fp8 \
  --max-model-len 4096 --max-num-seqs 16 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching --trust-remote-code

python -m vllm.entrypoints.openai.api_server \
  --model microsoft/Phi-3-medium-128k-instruct \
  --quantization awq_marlin --kv-cache-dtype fp8 \
  --max-model-len 32768 --max-num-seqs 4 \
  --gpu-memory-utilization 0.94 \
  --enable-prefix-caching --enable-chunked-prefill \
  --trust-remote-code

Both require --trust-remote-code because Phi-3 ships custom modelling files that vLLM has to import directly. See the vLLM setup guide for the prerequisite install steps and CUDA driver matrix.

Quality benchmarks and named scenarios

Headline benchmark numbers tell only part of the story. Below are public scores plus three named scenarios drawn from realistic production workloads we have measured.

Scenario A: maths tutor for 5,000 students

An ed-tech product running step-by-step algebra explanations issues bursts of around 3 requests/second per active session. Phi-3 Medium’s 90.0 GSM8K and good chain-of-thought adherence make it the strongest sub-30B option here. With FP8 at max-num-seqs 16 the 4090 sustains roughly 600 t/s aggregate, enough for around 200 concurrent active sessions assuming a typical 250-token reply.

Scenario B: clinical decision-support summariser

A NHS-aligned clinical pilot summarises medical literature into structured notes. Phi-3 Medium’s MIT licence, strong reasoning and modest size make it a much easier governance approval than Llama 70B. With 16k context capped, it handles full-paper inputs at 110 t/s decode.

Scenario C: internal RAG copilot for an analytics team

Long-context retrieval over a knowledge base of 5,000 PDFs. Capping context at 32k on the 128k variant gives the right window without OOM risk, and prefix caching means repeated system prompts (RAG instructions) are reused at near-zero cost. Pair with the SaaS RAG sizing guide for batch tuning.

Production gotchas

1. KV cache exhaustion at long context. Without GQA the cache is 4x denser than peers. Monitor vllm:gpu_cache_usage_perc and set hard limits on max_tokens in your API gateway, or you will see request rejections during traffic spikes.
2. Custom modelling code. Phi-3 requires --trust-remote-code. Pin a specific HF revision in your image to avoid an upstream change breaking the launcher mid-deploy.
3. Tokeniser quirks. Phi-3 uses a Llama-style sentencepiece tokeniser but with extra system tokens. Always render system prompts via the official chat template (tokenizer.apply_chat_template); raw string concatenation produces silently degraded outputs.
4. 128k quality cliff. Empirical RULER scores fall sharply past 32k for the 128k variant. Don’t sell users on “128k context” without a long-context retrieval benchmark on your own data.
5. FP8 weight-only vs W8A8. vLLM’s --quantization fp8 by default does W8A8 (weights and activations). If you only have static FP8 checkpoints, use --quantization fp8_w8 instead. Mismatching causes a runtime calibration error at boot.
6. Coding regressions. Phi-3 is weaker than Qwen 14B at code. If your product mixes Q&A and code-completion, route by intent rather than choosing one model.
7. Driver matrix. FP8 marlin kernels require CUDA 12.4+ and driver R550 series or newer. Older container bases will silently fall back to slower paths.

Alternatives and verdict

Verdict. Phi-3 Medium is the right pick when you want strong reasoning and a permissive MIT licence at moderate hardware cost, particularly for maths, sciences, and analysis-heavy workloads where Qwen’s coding lead is irrelevant. The KV cost is its only real ergonomic weakness on a 24 GB card, and capping context at 16k removes the issue. For coding-led products go Qwen 14B; for everything else, Phi-3 on a 4090 is excellent value.

See also: spec breakdown, FP8 deployment, AWQ guide, vLLM setup, Qwen 14B benchmark, tier positioning, monthly hosting cost.