RTX 4090 24GB for Long-Document Summarisation: 64k Context, FP8 KV, Map-Reduce, Production Pipeline

Long-document summarisation is the workload where the RTX 4090 24GB dedicated server repeatedly punches above its sticker. Ada AD102 with native FP8 tensor cores, 1008 GB/s of GDDR6X bandwidth and 24 GB of VRAM is enough to fit Llama 3.1 8B at 64k context with FP8 KV cache, or Qwen 2.5 14B AWQ at 32k context with measurably higher quality. That is roughly two hundred pages of A4 in a single shot, or arbitrarily long under map-reduce. This article is the senior-engineer playbook: model menu, KV maths, single-shot vs map-reduce trade-offs, a 25k-word worked latency budget, capacity tables, evaluation harness and the production gotchas that bite. Wider hardware menu is on dedicated GPU hosting.

Contents

• The named workload: 25k-word docs, 600-token summaries
• Context windows and KV maths on 24 GB
• Single-shot summarisation latency budget
• Map-reduce for 100k+ token corpora
• Capacity, concurrency and scaling triggers
• Cost vs hosted API alternatives
• Quality, evaluation and prompt patterns
• Production gotchas and verdict

The named workload: 25k-word docs, 600-token summaries

The reference workload for this article is what most teams actually ship: an internal pipeline that ingests PDFs and Office documents averaging 25,000 words (roughly 33,000 tokens after tokenisation, allowing for whitespace and punctuation), and produces a 600-token executive summary plus a 200-token bullet abstract. Latency target is sub-15 seconds end-to-end at p95, with cost capped at infrastructure flat-rate so finance can plan twelve months ahead. Documents arrive in bursts: legal teams dump fifty contracts at month-end, research teams batch a dozen literature reviews on Friday. The pipeline must absorb bursts of 50 documents inside ten minutes without queueing past the SLA.

Why the 4090 fits this brief

FP8 weights plus FP8 KV cache let an 8B model carry a 64k context inside 22 GB. That covers any single document up to about 200 pages of dense prose without truncation. For the longer literature reviews and consolidated case files that exceed 64k tokens, a 32k-chunk map-reduce pattern keeps wall-clock under a minute. Qwen 2.5 14B AWQ INT4 fits at 32k context with higher recall on technical and multilingual material; the router picks the model by source language and document class.

Context windows and KV maths on 24 GB

The hard constraint is the KV cache, which scales as 2 * num_layers * num_kv_heads * head_dim * seq_len * dtype_bytes. On Llama 3.1 8B (32 layers, 8 KV heads, 128 head dim) one token of FP16 KV is 131 KB; FP8 halves it to 65.5 KB. At 65,536 tokens that is 4.3 GB. Add 8.5 GB of FP8 weights, 3 GB activations and 2.5 GB CUDA workspace and you land near 22 GB with roughly 2 GB of headroom for spike absorption.

64k tokens is roughly 200 pages of dense prose or 700 KB of plain text. The full sweep including Qwen 32B is on the Llama 3 8B page, the Qwen 14B page and the Qwen 32B page. KV maths cross-checked on the spec breakdown.

Standard launch for 64k context

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --quantization fp8 --kv-cache-dtype fp8 \
  --max-model-len 65536 --max-num-seqs 32 \
  --enable-chunked-prefill --enable-prefix-caching \
  --gpu-memory-utilization 0.92

--enable-chunked-prefill is non-negotiable at 64k context: without it a single 32k-token prefill blocks all other sessions for 1.5-2 seconds. --enable-prefix-caching pays for itself when the system prompt and document boilerplate repeat across calls. See the full launch in the vLLM setup tutorial, FP8 specifics in the FP8 deployment guide.

Single-shot summarisation latency budget

Single-shot means the entire document is fed to the model in one prefill, then a single decode pass produces the summary. Quality is the strongest of any approach because the model sees every cross-reference, every footnote and every contradicting clause in the same attention window. The cost is prefill latency, which scales roughly linearly with sequence length on Ada at FP8 (about 16,000 tokens per second of prefill on the 4090 with chunked prefill enabled).

Llama 3 8B FP8 batch-1 decode is benchmarked at 198 t/s on the 4090, dropping to about 152 t/s at very long context due to attention overhead. See the Llama 3 8B benchmark for the full curves. Switching to Qwen 14B AWQ raises decode quality on technical and multilingual content but pulls per-doc latency to about 12 seconds at 32k context (135 t/s decode), which is still inside the 15-second SLA.

Map-reduce for 100k+ token corpora

Map-reduce splits the document into chunks at clean boundaries (heading, paragraph, sentence as fallback), summarises each chunk independently (“map”), then summarises the concatenated map outputs (“reduce”). It scales to arbitrary length, parallelises trivially across vLLM’s continuous batching and tolerates models with smaller context windows, but it loses cross-chunk cohesion unless you add overlap and a careful reduce prompt.

vLLM batches map calls on the same card, so a 100k-token document split into eight 8k chunks does not take 8x the wall-clock of one chunk; with batch 8 saturating the 4090 at ~880 t/s aggregate, it lands around 22 seconds plus a small reduce. Hierarchical reduce (map -> intermediate reduce per 10 maps -> final reduce) keeps the reduce stage itself inside the model’s context window even at million-token inputs. Pattern detail aligns with the SaaS RAG playbook.

Chunking and overlap

For prose, chunk at heading boundaries first, paragraph second, sentence third. Add 256-token overlap between chunks to preserve context for entities introduced near boundaries. For tables, never split a row; treat the whole table as an indivisible chunk even if it exceeds the target chunk size (raise context for that single call). For code, split on function or class boundaries.

Capacity, concurrency and scaling triggers

Aggregate Llama 3 8B FP8 throughput on the 4090 is 198 t/s at batch 1, 880 t/s at batch 8 and 1100 t/s at batch 32, saturating near 1140 t/s at batch 64. For a summarisation pipeline, the dominant cost is prefill, so the right capacity model is “documents per hour” rather than “tokens per second of decode”.

Concurrent SLA-compliant active sessions on Llama 3 8B FP8 land near 30 for short interactions, scaling to roughly 300 monthly active users at typical 10x multiplier; numbers cross-check the concurrent users page.

Scaling triggers

• Sustained queue depth > 5: add a second 4090 behind a round-robin load balancer; both share the same prefix cache config.
• p95 latency > 15 s for 25k-word docs: switch from Qwen 14B back to Llama 8B for that lane, or pin a card per doc class.
• Map-reduce reduce stage exceeds context: introduce hierarchical reduce or move the reduce step to Qwen 32B AWQ on a sister card.
• Bursty 50-doc month-end load missing SLA: pre-warm with prefix cache hits, or allocate one card per concurrent burst slot. See when to upgrade.

Cost vs hosted API alternatives

A 4090 dedicated UK server runs ~£550/month (~$700) flat with no per-token meter. Sustained Llama 3 8B FP8 capacity is 2.85 B tokens/month at 100% utilisation, well above any realistic summarisation pipeline. For comparison, GPT-4o blends to $5/M tokens and Claude Sonnet to $7/M.

Break-even against GPT-4o is roughly 140 M tokens/month and against Claude Sonnet 100 M tokens/month. A summarisation pipeline processing 1,000 docs/day at the 25k-word reference size hits 1 B tokens/month inside the first quarter and is comfortably inside the cost win. Full crossover maths on the break-even calculator and vs OpenAI.

Quality, evaluation and prompt patterns

Track three metrics in CI: ROUGE-L for extractive faithfulness, BERTScore-F1 for semantic overlap and entity-recall against a gold set for factual fidelity. Llama 3 8B FP8 versus FP16 baseline drops ROUGE-L by under 0.3 points on CNN/DailyMail and ArXiv summarisation; FP8 KV adds another 0.1-0.2 point cost. For the reference 25k-word workload Qwen 14B AWQ scores about 1.2 ROUGE-L points higher than Llama 8B, at roughly 1.7x the latency.

Prompt patterns that move the needle

• Two-stage prompt: first ask for an outline, then expand. Cuts hallucination by 30-40% on legal documents.
• Explicit role and audience: “Summarise for a CFO who needs to decide funding in five minutes” outperforms “Summarise this”.
• Anchor with section headers: include the document’s existing section structure in the prompt so the model uses it as a frame.
• Forbid speculation: “Only state facts present in the source. If a fact is missing, say ‘not stated’.” Improves entity-recall by 8-12 points.

Production gotchas and verdict

Production gotchas

1. FP8 KV cache numerical drift on very long sequences: at 64k context the last 5-10% of generation can lose coherence. Mitigate by capping summary length at 800 tokens or stepping KV back to FP16 above 48k input.
2. Chunked prefill off by default in older vLLM: pre-0.6.x will block the worker on a 32k prefill. Always explicitly set --enable-chunked-prefill regardless of version.
3. Prefix cache invalidation on tool registry change: rotating glossary or system prompt invalidates the prefix cache and can spike latency by 200-400 ms. Treat the system prompt as immutable in production.
4. Tokeniser truncation silently dropping doc tail: if you do not enforce truncation=False and instead pass docs that exceed max_model_len, transformers will silently drop the tail. Always pre-count and route to map-reduce above the threshold.
5. OOM on burst bursts at 64k context: a single rogue 64k call alongside 30 small ones can OOM. Cap --max-num-seqs 4 on the 64k lane and serve small interactive calls on a separate vLLM instance with smaller --max-model-len.
6. UTF-8 BOM and PDF artefacts inflating token count by 5-15%: Office docs and many PDF extractors leave hidden bytes that look like real tokens. Strip aggressively at ingest.
7. Map-reduce reduce-step length blowing context: ten 800-token map outputs is 8k tokens of reduce input, fine; one hundred is 80k, not fine. Always plan hierarchical reduce above 50 maps.

Verdict

For long-document summarisation up to 50 pages of A4, the 4090 with Llama 3 8B FP8 at 64k context is the cheapest credible production option in 2026, hitting sub-10-second wall clock and £550/month flat. For higher-quality recall on technical or multilingual material, Qwen 14B AWQ at 32k context trades ~50% latency for ~1.2 ROUGE-L points. Above 100k tokens use map-reduce; above 1 M tokens use hierarchical reduce. Above 2.5 B tokens/month or a sustained 25 concurrent summarisation users, add a second 4090 or move to a denser deployment per 4090 vs 5090.

See also: Llama 3 8B on the 4090, Qwen 14B, vLLM setup, FP8 deployment, 8B benchmark, SaaS RAG, translation.