BGE Reranker v2 M3 Deployment

A reranker reads the full query-document pair and scores relevance directly – slower than embedding but more accurate. In a typical RAG pipeline on dedicated GPU hosting, retrieving 50-100 candidates with embeddings and reranking to the top 5 improves answer quality meaningfully.

Contents

• Why rerank
• VRAM
• Deployment
• Latency cost

Why Rerank

Embeddings approximate relevance via vector distance – fast but lossy. A reranker cross-encodes query and document in one transformer pass, producing a precise relevance score. Quality lift: typically 10-30% improvement in precision@5 on hard retrieval tasks.

VRAM

BGE-reranker-v2-m3 is a 568M-parameter model. ~1.1 GB at FP16. Fits any GPU.

Deployment

Via TEI’s reranker endpoint:

docker run --gpus all -p 8081:80 \
  ghcr.io/huggingface/text-embeddings-inference:1.5 \
  --model-id BAAI/bge-reranker-v2-m3

Client call:

curl http://localhost:8081/rerank \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "query": "what is RAG",
    "texts": ["doc1...", "doc2...", "doc3..."]
  }'

Returns scores per document. Sort descending and keep the top k.

Latency Cost

Per 100 candidates on a 4060 Ti:

• Embedding retrieval of top 100: ~5 ms
• Rerank 100 candidates: ~80-120 ms
• Total pipeline: ~100 ms (dominated by rerank)

Trade-off: cheaper overall than running a larger embedding-only model, and more accurate.

See BGE-M3 embedder and splitting embedding and LLM across GPUs.