RTX 5060 Ti 16GB as Reranker API

A reranker is the quality multiplier on every serious RAG stack: it turns a bag of 50 dense-retrieval candidates into the three or five your LLM should actually read. Self-hosting it on the RTX 5060 Ti 16GB via UK dedicated GPU hosting delivers 3,200 query-document pairs per second on BGE-reranker-base and removes the per-query Cohere Rerank bill entirely.

Contents

• Deploying TEI
• Capacity and latency
• Client integration snippet
• Cost vs Cohere
• Pairing with an embedder

Deploying TEI

Hugging Face Text Embeddings Inference (TEI) ships a production-grade reranker binary with batching, dynamic padding and CUDA graphs. BGE-reranker-base loads in under 900 MB of VRAM; the larger BGE-reranker-v2-m3 fits in 2.3 GB. Both co-reside with a BGE-base embedder on the same 5060 Ti. See our reranker server setup.

Capacity and latency

At 3,200 pairs/second and 50% utilisation, one 5060 Ti reranks 138M pairs/day – roughly 2.8M user queries each paired against 50 candidates. See reranker throughput.

Client integration snippet

import httpx

def rerank(query: str, docs: list[str], top_k: int = 5):
    r = httpx.post(
        "https://rerank.example.com/rerank",
        json={"query": query, "texts": docs, "top_k": top_k},
        timeout=10.0,
    )
    r.raise_for_status()
    return r.json()  # [{index, score}, ...]

# Drop-in for cohere.rerank() by mapping field names.

Cost vs Cohere

Break-even vs Cohere Rerank 3 at $2 per 1,000 queries lands around 150k queries/month; above 1M/month self-hosting is decisively cheaper and also unlocks GDPR-clean UK data residency, which matters for regulated industries.

Pairing with an embedder

A proper RAG tier co-locates embeddings and rerank on one card: BGE-base at 10,200 texts/sec plus BGE-reranker-base at 3,200 pairs/sec fit comfortably in 3.2 GB combined. See embedding server and SaaS RAG for the joint deployment pattern.

See also: embedding throughput, embedding API, vLLM setup, classification.