What You’ll Build
In 30 minutes, you will have a production summarisation API that accepts documents up to 100,000 tokens and returns concise summaries in your chosen format — bullet points, executive briefs, one-paragraph abstracts, or structured JSON. Running an open-source LLM on a dedicated GPU server through vLLM, your API summarises a 50-page document in under 10 seconds at zero per-token cost.
Cloud summarisation APIs charge $0.01-$0.06 per 1,000 tokens of input. Summarising 1,000 long documents daily at 10,000 tokens each means $100-$600 per day in API fees. Self-hosted summarisation on GPU hardware delivers equivalent quality with predictable monthly costs and complete data sovereignty — essential when processing confidential business documents, legal filings, or medical reports.
Architecture Overview
The API routes requests through two summarisation paths. Short documents (under 8,000 tokens) go directly to the LLM with a single summarisation prompt. Long documents pass through a chunking pipeline that splits text into overlapping segments, summarises each segment, then generates a unified summary from the segment summaries. This hierarchical approach handles documents of any length within fixed VRAM constraints.
The API layer accepts plain text, HTML, PDF uploads, and JSON payloads. A preprocessing module strips formatting, extracts text from PDFs, and estimates token count to route through the appropriate path. Output formats include plain text, markdown, HTML, and structured JSON with section-level summaries.
GPU Requirements
| Model Size | Recommended GPU | VRAM | Throughput |
|---|---|---|---|
| 8B (fast) | RTX 5090 | 24 GB | ~200 docs/hour |
| 70B (quality) | RTX 6000 Pro | 40 GB | ~80 docs/hour |
| 70B + long context | RTX 6000 Pro 96 GB | 80 GB | ~120 docs/hour |
Models with extended context windows (32K-128K tokens) handle longer documents in a single pass, avoiding the quality loss from hierarchical chunking. The RTX 6000 Pro 96 GB card runs a 70B model with 32K context comfortably. See our self-hosted LLM guide for long-context model recommendations.
Step-by-Step Build
Deploy vLLM on your GPU server with a long-context model. Build the summarisation API with document parsing and output formatting.
from fastapi import FastAPI, UploadFile
import requests, tiktoken
app = FastAPI()
VLLM_URL = "http://localhost:8000/v1/chat/completions"
MODEL = "meta-llama/Llama-3-70b-chat-hf"
enc = tiktoken.get_encoding("cl100k_base")
SUMMARY_PROMPT = """Summarise the following document.
Format: {format}
Target length: {length}
Document:
{text}
Provide a clear, accurate summary that captures the key points,
main arguments, and critical details."""
@app.post("/v1/summarise")
async def summarise(text: str, format: str = "paragraph",
length: str = "medium"):
tokens = len(enc.encode(text))
if tokens > 8000:
# Hierarchical summarisation for long documents
chunks = split_into_chunks(text, max_tokens=6000, overlap=500)
chunk_summaries = []
for chunk in chunks:
resp = call_llm(chunk, format="bullet", length="short")
chunk_summaries.append(resp)
combined = "\n".join(chunk_summaries)
final = call_llm(combined, format=format, length=length)
else:
final = call_llm(text, format=format, length=length)
return {"summary": final, "input_tokens": tokens,
"format": format}
def call_llm(text, format, length):
prompt = SUMMARY_PROMPT.format(text=text, format=format, length=length)
resp = requests.post(VLLM_URL, json={
"model": MODEL,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 1000, "temperature": 0.3
})
return resp.json()["choices"][0]["message"]["content"]Add PDF upload parsing and HTML output formatting for direct embedding in web applications. The OpenAI-compatible endpoint means existing chat completion clients can call your summarisation API with a system prompt. See production setup for throughput tuning.
Output Quality and Tuning
Different summarisation tasks need different approaches. Legal document summaries must preserve specific clauses and defined terms. Meeting notes need action items extracted alongside discussion summaries. Financial reports require numerical accuracy above all else. Tune the system prompt per document type and validate output against manual summaries to establish quality baselines.
For production use, add a confidence scorer that flags summaries where the model may have hallucinated — checking that key figures, names, and dates in the summary actually appear in the source document. Pair with an AI chatbot so users can ask follow-up questions about summarised documents.
Deploy Your Summarisation API
A self-hosted summarisation API eliminates per-token billing while keeping confidential documents on your infrastructure. Power document workflows, research tools, or customer-facing features with unlimited summarisation capacity. Launch on GigaGPU dedicated GPU hosting and summarise at scale. Browse more API use cases and tutorials in our library.
