RTX 5070 for Embedding Generation: Throughput and Setup Guide

Embedding Generation on GPU

Embedding models convert text into dense vector representations used in semantic search, RAG, clustering, and classification. They’re encoder-only transformers (BERT-family) — small, fast, and vastly cheaper to run than generative LLMs. Even the best embedding models are under 1.5 GB on the RTX 5070, leaving virtually all 12 GB for other tasks.

The RTX 5070’s 6,144 CUDA cores and high GDDR7 bandwidth make it well-suited for batch embedding generation — each batch of 64–128 sentences runs through the encoder in a single forward pass, fully utilising the GPU. For a 1M document RAG corpus, the initial embedding pass takes ~10–30 minutes depending on document length.

Embedding Model Comparison

sentence-transformers Setup

pip install sentence-transformers torch

from sentence_transformers import SentenceTransformer
import torch

# Load model on GPU
model = SentenceTransformer(
    "nomic-ai/nomic-embed-text-v1.5",
    trust_remote_code=True,
    device="cuda"
)

# Single sentence
embedding = model.encode("The RTX 5070 has 12 GB GDDR7 memory.")
print(embedding.shape)  # (768,)

# Batch encoding (most efficient)
sentences = [
    "GPU hosting in the UK.",
    "Blackwell architecture brings FP8 support.",
    "Flux.1 requires at least 12 GB for Q4 GGUF.",
    # ... thousands more
]

# Efficient batch encoding
embeddings = model.encode(
    sentences,
    batch_size=128,
    show_progress_bar=True,
    normalize_embeddings=True,  # L2 normalise for cosine similarity
    convert_to_tensor=True      # keep on GPU for downstream ops
)
print(embeddings.shape)  # (N, 768)

Throughput Benchmarks

Time to embed a corpus of 100K short documents (~50 tokens each) using nomic-embed-text at batch_size=128: approximately 20–30 seconds. A 1M document corpus: 3–5 minutes. The RTX 5070’s throughput makes the initial embedding build-phase fast enough to iterate on corpus updates frequently.

Embedding API Server

Expose an OpenAI-compatible embeddings endpoint using Ollama or a custom FastAPI server:

# Ollama exposes /v1/embeddings natively
ollama pull nomic-embed-text
curl http://localhost:11434/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model":"nomic-embed-text","input":"Hello, RTX 5070!"}'

# Or custom FastAPI server with batch support
from fastapi import FastAPI
from sentence_transformers import SentenceTransformer
from pydantic import BaseModel
from typing import List

app = FastAPI()
model = SentenceTransformer("nomic-ai/nomic-embed-text-v1.5",
                             trust_remote_code=True, device="cuda")

class EmbedRequest(BaseModel):
    input: List[str]
    model: str = "nomic-embed-text-v1.5"

@app.post("/v1/embeddings")
def embed(req: EmbedRequest):
    vecs = model.encode(req.input, normalize_embeddings=True).tolist()
    return {"data": [{"embedding": v, "index": i} for i, v in enumerate(vecs)],
            "model": req.model}

# uvicorn server:app --host 0.0.0.0 --port 8001

Embedding + LLM Combo

Combined VRAM for embedding + LLM pipelines on 12 GB:

Running the embedding model and LLM simultaneously on 12 GB is easily achievable. The embedding model handles document indexing or query encoding while the LLM handles answer generation — a clean pipeline with no model-swapping overhead.