Retrieval-augmented generation is only as fast as its slowest component. We benchmarked the full RAG loop — BGE-M3 embedding for semantic search plus LLaMA 3 8B for answer generation — running concurrently on a single RTX 3090 (24 GB VRAM) inside a GigaGPU dedicated server. The goal: determine whether one GPU can handle both the retrieval and generation stages of a production RAG system without unacceptable compromise.
Models tested: BGE-M3 Embedding + LLaMA 3 8B
RAG Performance Metrics
| Component | Metric | Value |
|---|---|---|
| BGE-M3 Embedding | Tokens/sec | 570 |
| BGE-M3 Embedding | Doc chunks/sec (256 tok) | 2.2 |
| LLaMA 3 8B (FP16) | Generation tok/sec | 52.7 |
| End-to-end RAG query | Latency (retrieve+generate) | 2.95s |
All models loaded simultaneously in GPU memory. Throughput figures reflect concurrent operation with shared VRAM and compute.
VRAM for the Full Stack
| Component | VRAM |
|---|---|
| Combined model weights | 19.2 GB |
| Total RTX 3090 VRAM | 24 GB |
| Free headroom | ~4.8 GB |
Both the embedding model and the LLM fit comfortably with 4.8 GB of headroom. That spare capacity covers the KV cache during generation, embedding batch buffers, and ChromaDB or similar vector store operations. The 3090’s 24 GB VRAM is genuinely well-matched to RAG workloads where you need an embedding model and a generation model resident simultaneously.
Cost vs. API Pricing
| Cost Metric | Value |
|---|---|
| Server cost (single GPU) | £0.75/hr (£149/mo) |
| Equivalent separate GPUs | £1.50/hr |
| Savings vs separate servers | 50% |
At £149/mo for both embedding and generation on one GPU, self-hosted RAG is dramatically cheaper than calling separate cloud APIs for each stage. A single RAG query that might cost fractions of a penny through APIs becomes effectively free once you are paying the flat monthly rate — the economics improve with every additional query. Full GPU comparison at our benchmark page.
Building RAG on the 3090
The RTX 3090 handles the core RAG pipeline well: 2.95 seconds end-to-end is responsive enough for internal knowledge bases, support chatbots, and document Q&A systems. At 2.2 doc chunks per second, real-time indexing of new documents is feasible for moderate ingest rates. For higher-throughput indexing or faster generation, move up to the RTX 5080. But for teams getting started with self-hosted RAG infrastructure using tools like LangChain, the 3090 is a proven, affordable foundation.
Quick deploy:
docker compose up -d # text-embeddings-inference + llama.cpp + chromadb containersSee our LLM hosting guide, RAG hosting guide, LangChain hosting, and all benchmark results. Related benchmarks: LLaMA 3 8B on RTX 3090.
Deploy RAG Pipeline on RTX 3090
Order this exact configuration. UK datacenter, full root access.
Order RTX 3090 Server
