A typical RAG deployment on dedicated GPU hosting runs an embedding model for retrieval, a reranker for filtering, and an LLM for generation. Putting all three on one GPU is common and usually wrong. Splitting the embedder (and reranker) onto a separate card from the LLM is one of the highest-impact architecture moves you can make.
Topics
Why Colocation Hurts
Three problems arise when embedder and LLM share a card:
- VRAM contention. LLM KV cache fills dynamically; embedder weights are fixed. Spikes in LLM concurrency can push the embedder out.
- Scheduler contention. vLLM and a separate embedding server compete for CUDA kernels on the same device.
- Failure coupling. If the LLM serving process crashes, the embedder is affected.
The Split
Card 0: LLM serving (vLLM on a 7-70B model). Card 1: embedder and reranker. Two HTTP endpoints. The application orchestrates the pipeline. Each card is independent and isolated.
Right-Sizing
The LLM needs the big card. The embedder card can be small – a 4060 Ti 16GB happily hosts BGE-M3 (2 GB), a cross-encoder reranker (1-2 GB), and a BM25-to-embedding re-rank with room for batching. Pairing a 5090 for the LLM with a 4060 Ti for embedding covers almost any RAG stack.
| Role | Suggested Card | Capacity |
|---|---|---|
| LLM (13B class) | RTX 5080 or 3090 | 16-24 GB |
| LLM (70B class) | RTX 5090 or 6000 Pro | 32-96 GB |
| Embedder + reranker | RTX 4060 Ti 16GB | Plenty of room |
| Embedder only | RTX 4060 8GB | Tight but works |
Right Card for Each Role
Mixed GPU chassis on UK dedicated hosting with one tuned per-workload configuration.
Browse GPU ServersConfiguration
The embedder runs via Text Embeddings Inference (TEI) or a simple FastAPI wrapper around sentence-transformers. Bind it to GPU 1 with CUDA_VISIBLE_DEVICES=1. vLLM on GPU 0 via CUDA_VISIBLE_DEVICES=0. Your application layer calls one endpoint then the other.
Latency implication: the embedder typically adds 20-50 ms to a request. Not a problem unless your total budget is under 200 ms. The reduced contention usually saves more latency than the network hop costs.
See heterogeneous multi-GPU workload split and BGE-M3 self-hosted for specifics.
