RTX 4090 24GB for Llama 3.1 70B INT4: Best Single-Card 70B Setup

Running a 70B-class model on a single consumer card sounds implausible, but Llama 3.1 70B AWQ INT4 fits the RTX 4090 24GB with room for 16k context, FP8 KV cache and four concurrent sequences. It is the best 70B-on-one-card configuration shipping today, the cheapest way to host frontier-class quality on UK dedicated GPU hosting, and the headline workload that justifies the 4090 over a 3090 or smaller card. This guide goes deep: line-by-line memory math, the calculation that produces the max-model-len ceiling, the max-num-seqs trade-off, expected throughput by batch, a worked real chat session, comparison to running 70B on the 5090, RTX 6000 Pro and H100, deployment recipes, and the production gotchas that decide whether the configuration holds up at scale.

Contents

• Why 70B on one 4090 is even possible
• Memory math line by line
• KV cache and max-model-len calculation
• max-num-seqs trade-off and the throughput curve
• Decode throughput by batch and context
• Real chat session walkthrough
• Comparison: 4090 vs 5090 vs RTX 6000 Pro vs H100
• vLLM, TGI, EXL2 deployment recipes
• INT4 quality cost vs FP16
• Production gotchas
• Verdict and when to pick this stack

Why 70B on one 4090 is even possible

Llama 3.1 70B has 70.6B parameters. At FP16 that is 141 GB, far beyond a single 24 GB card. AWQ (Activation-aware Weight Quantization, Lin et al. 2023) packs weights to 4 bits with a 16-bit scale per group of 128 elements; the result occupies roughly 17 GB on disk and in VRAM. The weight-only quantisation is well-suited to LLMs because LLM weights have low intrinsic precision once you account for which weights actually matter: AWQ identifies the salient channels and protects them with higher-precision scales, then quantises the rest aggressively. The Marlin INT4 GEMM kernel (NeuralMagic, 2024) then dequantises weights into FP16 registers at runtime and runs the matmul on the 4th-generation tensor cores at FP16 rates, hitting roughly 62 percent of dense FP8 peak on prefill.

FP8 KV cache halves the attention memory cost relative to FP16 KV. The combination (AWQ INT4 weights + FP8 KV cache + Marlin kernel) is what makes 70B on a single 4090 work; remove any one of the three and the configuration collapses. AWQ alone leaves 7 GB for KV (about 8k context maximum at batch 1); FP8 KV alone leaves no room for the weights; the Marlin kernel makes the AWQ throughput acceptable rather than 3x slower. See FP8 tensor cores and AWQ guide for the underlying mechanics.

Memory math line by line

Llama 3.1 70B has 80 layers, 8 KV heads (grouped-query attention), 128 head dimension and 8192 hidden. The per-token KV cost is 2 (K and V) x 8 (KV heads) x 128 (dim) x 80 (layers) x 2 bytes (FP16) = 320 KB per token, or 160 KB per token at FP8. That is one of the smaller per-token KV costs in the 70B class because grouped-query attention shares K and V across query heads (8 KV heads serving 64 query heads).

32k context single-stream is technically achievable but leaves you sailing close to the OOM line and breaks any batching. 16k single-stream is the sustainable production target with one user; 16k context plus 2-stream batching with prefix caching is the right configuration if you need two concurrent slow conversations. For four concurrent streams you must drop to 8k context.

KV cache and max-model-len calculation

The vLLM --max-model-len flag sets the longest sequence (input + output) that the engine will accept. It does not by itself allocate KV; vLLM allocates KV in blocks (default 16 tokens) on demand. But it does dictate the maximum KV reservation per sequence, which determines how many sequences can fit in the KV pool.

# KV budget = (gpu_memory_utilization * 24 GB) - weights - activations - overhead
# = (0.95 * 24) - 17.0 - 0.5 - 0.5
# = 22.8 - 18.0
# = 4.8 GB usable KV pool

# KV per token at FP8 = 160 KB
# Tokens supported at the cap = 4.8 GB / 160 KB = 30,000 tokens

# Distribute across max-num-seqs:
#   1 stream  x 30k context = fits but no batching, no prefix cache margin
#   2 streams x 16k context = 32k tokens of KV ~ 5.1 GB, just over — push max_seqs back to 1
#   2 streams x 12k context = 24k tokens ~ 3.84 GB, comfortable
#   4 streams x  8k context = 32k tokens ~ 5.1 GB, just over
#   4 streams x  7k context = 28k tokens ~ 4.5 GB, comfortable

# Production recommendation: --max-model-len 16384 --max-num-seqs 4
#   with prefix caching enabled, so 4 streams sharing common system prompt
#   can dedupe their first N tokens of KV and fit comfortably

The recommended configuration is --max-model-len 16384 --max-num-seqs 4 with prefix caching enabled. Prefix caching is essential here: without it, four streams at 16k context cost 4 x 2.6 = 10.4 GB of KV, which exceeds the budget. With prefix caching, the system prompt (typically 1-2k tokens) is deduplicated to a single set of KV blocks shared across all streams, recovering 4-8 GB of effective KV. The realistic concurrent serving capacity is 4 streams at 16k context with shared system prompt, or 2 streams at 16k with fully unique prompts.

max-num-seqs trade-off and the throughput curve

Increasing --max-num-seqs on the 70B configuration is bound by KV memory, not by compute. Every additional concurrent sequence at 16k context costs 2.6 GB. The curve below assumes 16k context with prefix caching (1.5k tokens shared system prompt):

The aggregate throughput peaks at 88 t/s with 8 streams at 4k context, but per-stream latency degrades to ~91 ms inter-token, which is on the edge of acceptable for chat. The “right” configuration depends on what your product values: 4 streams at 16k delivers 62 t/s aggregate at responsive 65 ms inter-token; 8 streams at 4k delivers 88 t/s at slower 91 ms inter-token. Most production deployments settle on 4 streams at 16k as the balance.

Decode throughput by batch and context

Measured on a 4090 24GB with vLLM 0.6.3, AWQ Marlin kernels, FP8 KV cache, single-stream and batched, warm vLLM instance.

20-24 tokens per second per stream feels brisk for chat (faster than typical reading speed of about 5 t/s, comparable to the GPT-4 API’s perceived response rate). It is slower than 8B FP8 (~198 t/s) but you are getting 8.75x the parameter count and substantially better quality on reasoning, code, and tool-use tasks. The 1.1s TTFT on a 4k prompt is acceptable for most chat UIs; the 3.4s TTFT on 16k is on the edge and benefits significantly from --enable-chunked-prefill which interleaves prefill with decode of other streams.

Real chat session walkthrough

A typical 70B chat session on a single 4090 looks like this. The user is a developer asking the model to refactor a 600-line Python file (~4k tokens of code).

System prompt:        1500 tokens (cached, shared across users)
User message:          150 tokens
Code paste:           4000 tokens
Total input:          5650 tokens

vLLM behaviour:
- Prefix cache hit on system prompt: 1500 tokens reused (zero compute)
- Prefill on remaining 4150 tokens: 1.6 seconds
- TTFT (first generated token):     1.6 s + 50 ms decode = 1.65 s
- Generation: 800 tokens output at 22 t/s = 36.4 s
- Total session wall time:          38 s
- KV used after session:            5650 + 800 = 6450 tokens = 1.0 GB
- Card occupancy during session:    18 GB static + 1 GB KV = 19 GB / 24 GB

For a 200-MAU SaaS product where users send roughly 10 such sessions per day, a single 4090 supports approximately 40-60 active concurrent users at this latency profile (capped by the 4-stream concurrency ceiling and human think-time between messages). For a 12-engineer coding team using the same model behind Cursor or Continue.dev, the 4 concurrent streams cover the realistic peak; at 70B quality the model is strong enough on code review and complex refactors that the 22 t/s output rate is a good fit for the developer’s reading rate.

Comparison: 4090 vs 5090 vs RTX 6000 Pro vs H100

The 4090 24GB at AWQ INT4 is the cheapest 70B host by a wide margin. The 5090 32GB is the natural upgrade: roughly 1.7x the per-stream throughput at 1.5x the monthly cost, with significantly more headroom for batching (32 GB allows 4 streams at 32k context rather than 16k). The RTX 6000 Pro 96GB is the right choice if you specifically need FP8 weights for the small quality lift over INT4, but at £1,199/month it is 3.6x the 4090 for less than 3x the throughput. The H100 is overkill unless you need NVLink for tensor parallel or are running additional FP8 workloads alongside.

The 2x 4090 pipeline-parallel option is interesting: it gets you to FP8 quality for £658/month, with 1.7x the throughput of single-card AWQ. But the per-token latency suffers from the inter-card hop (PCIe Gen 4 round-trip per layer boundary), so unless you specifically need FP8 quality the single-card AWQ INT4 setup is usually the right answer. See 4090 vs 5090, vs RTX 6000 Pro, and vs H100.

vLLM, TGI, EXL2 deployment recipes

The minimal vLLM launch for production 70B AWQ INT4:

pip install "vllm>=0.6.3" autoawq

python -m vllm.entrypoints.openai.api_server \
  --model hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4 \
  --quantization awq_marlin \          # AWQ packed weights via Marlin INT4 GEMM
  --kv-cache-dtype fp8 \               # FP8 KV halves attention memory
  --max-model-len 16384 \              # 16k is the sustainable target
  --max-num-seqs 4 \                   # KV constraint, not compute
  --enable-prefix-caching \            # essential for batched serving
  --enable-chunked-prefill \           # caps p99 TTFT
  --gpu-memory-utilization 0.95 \      # pushed high; weights are static
  --port 8000

Increase --max-num-seqs only if you are running with shorter contexts. The single binding constraint is KV memory; every additional concurrent request at 16k context costs 2.6 GB of KV. Drop to 8k context if you need more concurrent users. For TGI:

# HuggingFace TGI launch (uses Marlin internally on Ada)
docker run --gpus all --shm-size 1g -p 8080:80 \
  -v $PWD/data:/data \
  ghcr.io/huggingface/text-generation-inference:2.3.0 \
  --model-id hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4 \
  --quantize awq \
  --max-batch-prefill-tokens 4096 \
  --max-input-tokens 14000 \
  --max-total-tokens 16384

For ExLlamaV2 (single-stream, slightly higher t/s but no continuous batching):

# EXL2 4.65bpw quantisation, single-stream
python -m exllamav2.server \
  --model-dir ./Meta-Llama-3.1-70B-Instruct-exl2-4.65bpw \
  --port 8001 \
  --max-seq-len 16384

Most production deployments settle on vLLM for the OpenAI-compatible API and continuous batching. TGI is a good fit if you are already in the HF ecosystem; EXL2 is the right choice for single-user maximum-throughput interactive use. See 70B INT4 deployment deep-dive for the full production setup.

INT4 quality cost vs FP16

Quality drop from FP16 to AWQ INT4 is small enough to be invisible in most product use cases. AWQ outperforms GPTQ by ~0.7 MMLU and beats round-to-nearest INT4 by ~1.5 points. If you are comparing 70B AWQ to 8B FP8, the 70B retains an 18-point MMLU lead and a 16-point HumanEval lead even after quantisation; the model size advantage dominates the quantisation cost.

Production gotchas

1. Cold-start is slow. First load of the AWQ checkpoint takes 14-20 seconds from NVMe; budget for it in your autoscaling.
2. Prefix caching is mandatory for batching. Without it, 4 streams at 16k context will OOM. Always enable.
3. Marlin requires group_size=128. Other group sizes fall back to a slower kernel (50 percent throughput drop). Stick to the hugging-quants AWQ checkpoints.
4. 32k context single-stream is risky. KV reaches 5.3 GB and any spike (a sudden batch arrival) tips into OOM. Cap at 16k unless you truly need more.
5. FP8 KV at 32k+ context has measurable quality drop on summarisation. Validate on your eval; consider FP16 KV with smaller context if quality is critical.
6. The Llama 3 Community Licence has acceptable use clauses and a 700M MAU cap. Read them before commercial deployment.
7. Two streams compete for HBM bandwidth. Per-stream throughput drops from 23.5 to 21.0 t/s at batch 2 because the 1008 GB/s bus is shared. The 5090’s 1792 GB/s is the upgrade path if this is the bottleneck.

Verdict and when to pick this stack

Pick Llama 3.1 70B AWQ INT4 on a single 4090 24GB when: you need frontier-class quality (MMLU 85, HumanEval 79) at a fixed monthly cost; your traffic is 1-4 concurrent active users at a time; 22-23 t/s per stream is fast enough for your UX (it is for chat, code review, document analysis); you can tolerate 1-3 second TTFT on long prompts. This is the cheapest commercially viable way to host a 70B-class model in the UK in 2026.

Pick the 5090 32GB instead if: you need 6+ concurrent streams, longer sustained context (32k+), or specifically need 1.7x the throughput per stream. Pick the RTX 6000 Pro 96GB if: you need full FP8 quality and have 8+ concurrent users. Pick the H100 if: you have additional FP8 workloads to share the card or need NVLink for tensor parallel. For a 12-engineer coding team running 70B AWQ on one 4090, the 4-stream concurrent ceiling matches the realistic active-typing concurrency of a small team; for a 200-MAU SaaS deployment, the same single card supports roughly 40-60 active users with chat-style think-time gaps.

See also: 70B INT4 deployment deep-dive, 70B benchmark, AWQ guide, 70B INT4 VRAM requirements, FP8 deployment, vLLM setup, 70B monthly cost, vs 5090, vs H100, vs RTX 6000 Pro, 4090 or 5090?.