RTX 5070 for RAG Pipelines: Embeddings + LLM on 12 GB GDDR7

RAG on 12 GB: What Fits

Retrieval-Augmented Generation (RAG) pipelines combine three components: an embedding model (to encode documents into vectors), a vector store (to retrieve relevant chunks), and a generative LLM (to answer questions based on retrieved context). On the RTX 5070:

• Embedding model: tiny (0.1–1.5 GB VRAM). Runs permanently on GPU for fast encode throughput.
• Vector store: runs on CPU RAM — no GPU VRAM needed for Chroma, Qdrant, or FAISS.
• LLM: 4–9 GB at Q4 for 7B–14B models — the dominant VRAM consumer.

Total GPU VRAM for a full RAG stack: ~5–10 GB, comfortably within 12 GB. The vector search is CPU-bound (fast enough for most workloads at moderate corpus sizes).

Embedding Model Setup

pip install sentence-transformers

from sentence_transformers import SentenceTransformer
import torch

# Load a high-quality embedding model on GPU
model = SentenceTransformer("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True)
model = model.to("cuda")

# Encode documents
documents = ["GPU hosting is fast.", "Intel Arc B60 has 24 GB ECC VRAM.", ...]
embeddings = model.encode(documents, batch_size=64, show_progress_bar=True)
# Shape: (N, 768) — 768-dim vectors for nomic-embed-text-v1.5

Recommended embedding models for 12 GB GPU RAG:

Vector Store Options

All run on CPU (no GPU VRAM needed):

• Chroma: easiest setup, persistent local storage, LangChain/LlamaIndex native. Best for <1M documents.
• Qdrant: production-grade, supports filtering, Docker deployment. Scales to millions of vectors.
• FAISS: Facebook’s in-memory index. Fastest for similarity search, no persistence by default.
• pgvector: PostgreSQL extension. Best if you already use Postgres for your application data.

LLM Component

For RAG, the LLM’s context window matters — retrieved chunks consume tokens. Recommended models for 12 GB RAG:

• Llama 3.1 8B Q4_K_M: 128K context window, ~5.2 GB VRAM. Best balance of quality and context for RAG.
• Qwen 2.5 14B Q4_K_M: ~8.7 GB VRAM, strong reasoning, 128K context. Best quality answer synthesis on 12 GB.
• Mistral 7B Q4_K_M: 32K context, ~4.4 GB, fastest generation for short-context RAG.

Full RAG Stack

pip install langchain langchain-community chromadb sentence-transformers

from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import Chroma
from langchain.chains import RetrievalQA
from langchain.llms import LlamaCpp

# 1. Embedding model on GPU
embeddings = HuggingFaceEmbeddings(
    model_name="nomic-ai/nomic-embed-text-v1.5",
    model_kwargs={"device": "cuda", "trust_remote_code": True},
    encode_kwargs={"normalize_embeddings": True}
)

# 2. Vector store (CPU)
vectorstore = Chroma(
    persist_directory="./chroma_db",
    embedding_function=embeddings
)

# 3. LLM on GPU (llama.cpp CUDA backend)
llm = LlamaCpp(
    model_path="/models/llama-3.1-8b-instruct-q4_k_m.gguf",
    n_gpu_layers=32,    # all layers on GPU
    n_ctx=8192,
    temperature=0.1,
    max_tokens=1024
)

# 4. RAG chain
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5})
)

# Query
result = qa_chain.invoke("What are the key specs of the RTX 5070?")
print(result["result"])

Throughput at Scale

Embedding throughput on RTX 5070 (nomic-embed-text-v1.5, batch_size=64):

• ~2,000–3,000 sentences/second for short documents (<128 tokens)
• ~500–800 sentences/second for medium documents (~512 tokens)
• A 100,000-document corpus takes ~1–3 minutes to embed initially

Retrieval + LLM answer latency (Llama 3.1 8B Q4, k=5 chunks, ~1500 context tokens):

• Vector retrieval: ~5–20ms (CPU-bound, depends on index size)
• LLM generation (200 tokens): ~2–3 seconds at ~75 tok/s
• Total end-to-end: ~2–3 seconds per query