Table of Contents
Embedding Throughput Overview
Embedding models power retrieval-augmented generation (RAG), semantic search, and recommendation systems. For production pipelines on a dedicated GPU server, you need to know how many embedding requests per second your GPU can process — both for real-time query embedding and for batch indexing of document collections. We benchmarked two popular embedding models across six GPUs at varying batch sizes.
All tests ran on GigaGPU bare-metal servers with input texts averaging 128 tokens. We measured sustained requests per second (where each request embeds one text passage) at batch sizes from 1 to 256. For LLM throughput comparisons, see the tokens per second benchmark.
BGE Large Throughput by GPU
BAAI/bge-large-en-v1.5 (335M parameters, ~1.3 GB VRAM). Requests per second at each batch size.
| GPU | Batch 1 | Batch 8 | Batch 32 | Batch 128 | Batch 256 |
|---|---|---|---|---|---|
| RTX 3050 (6 GB) | 28 | 145 | 380 | 620 | 720 |
| RTX 4060 (8 GB) | 48 | 260 | 680 | 1,120 | 1,300 |
| RTX 4060 Ti (16 GB) | 62 | 350 | 920 | 1,520 | 1,780 |
| RTX 3090 (24 GB) | 85 | 480 | 1,280 | 2,150 | 2,500 |
| RTX 5080 (16 GB) | 115 | 650 | 1,750 | 2,900 | 3,400 |
| RTX 5090 (32 GB) | 155 | 880 | 2,400 | 4,000 | 4,700 |
The RTX 5090 handles 4,700 BGE embeddings per second at batch 256 — over 400 million embeddings per day. The RTX 3090 at 2,500 req/s is more than sufficient for most RAG pipelines. Even the RTX 4060 at 1,300 req/s handles substantial volumes.
E5 Large Throughput by GPU
intfloat/e5-large-v2 (335M parameters, ~1.3 GB VRAM). Performance is nearly identical to BGE Large as both models share the same architecture and parameter count.
| GPU | Batch 1 | Batch 8 | Batch 32 | Batch 128 | Batch 256 |
|---|---|---|---|---|---|
| RTX 3050 (6 GB) | 26 | 140 | 365 | 600 | 695 |
| RTX 4060 (8 GB) | 46 | 250 | 655 | 1,080 | 1,260 |
| RTX 4060 Ti (16 GB) | 60 | 340 | 890 | 1,470 | 1,720 |
| RTX 3090 (24 GB) | 82 | 465 | 1,240 | 2,080 | 2,420 |
| RTX 5080 (16 GB) | 112 | 630 | 1,690 | 2,810 | 3,290 |
| RTX 5090 (32 GB) | 150 | 850 | 2,320 | 3,870 | 4,550 |
E5 and BGE Large perform within 3-5 percent of each other across all GPUs and batch sizes, as expected for architecturally identical models. Choose based on benchmark quality on your specific domain rather than speed differences.
Batch Size Scaling Curve
Embedding models scale exceptionally well with batch size because the computation is dense and uniform across inputs. From batch 1 to batch 256, throughput increases by 30-35x on most GPUs. The curve flattens around batch 128-256 as compute saturation is reached.
For real-time query embedding (user searches), batch 1 latency matters most: 6-12 ms on the RTX 3090, 4-7 ms on the RTX 5090. These are negligible in the context of a RAG pipeline where the LLM generation step takes seconds. For batch indexing of document collections, use batch 128-256 for maximum throughput. The batch size impact analysis covers similar scaling dynamics for LLM workloads.
Sizing for RAG Pipelines
A typical RAG pipeline embeds the user query (1 request, real-time), retrieves relevant chunks from a vector database, then generates a response with an LLM. The embedding step at batch 1 takes 6-12 ms on an RTX 3090 — effectively zero compared to the LLM generation step. This means the LLM is always the bottleneck in a RAG pipeline, not the embedding model.
You can comfortably run an embedding model alongside an LLM on the same GPU. BGE Large uses only 1.3 GB of VRAM, leaving the rest for LLM inference. On an RTX 3090 with 24 GB, that is 22+ GB for the LLM and KV cache. For complete LLM capacity planning, see our GPU capacity planning for AI SaaS guide and the chatbot capacity guide.
For initial document indexing (loading millions of documents into your vector store), a dedicated embedding job on the same GPU can process 2,500 passages per second on an RTX 3090. That is 216 million passages per day — sufficient to index most enterprise document collections in hours. Use open-source LLM hosting for the full RAG stack.
Conclusion
Embedding models are among the most GPU-efficient AI workloads, reaching thousands of requests per second even on budget cards. The RTX 3090 at 2,500 req/s (batch 256) handles virtually any RAG pipeline’s embedding needs with headroom to spare. For batch indexing, the RTX 5090 at 4,700 req/s processes hundreds of millions of documents per day. Browse all throughput data in the Benchmarks category at GigaGPU.
