How to Build a RAG Pipeline with LangChain on a GPU Server

What Is RAG and Why Use a GPU Server

Retrieval-augmented generation (RAG) combines a vector search engine with a large language model to ground responses in your own documents. Instead of relying solely on an LLM’s training data, RAG retrieves relevant passages from a knowledge base and passes them as context to the model, producing answers that are accurate, up-to-date, and verifiable. Hosting the entire pipeline on a dedicated GPU server keeps your data private and eliminates per-query API costs.

GigaGPU’s RAG hosting infrastructure provides the GPU power needed to run both embedding models and LLMs locally. Combined with LangChain hosting support, you get a turnkey platform for building production RAG systems. This tutorial walks through every component: document loading, embedding, vector storage, retrieval, and LLM generation.

GPU VRAM Requirements for RAG Pipelines

A RAG pipeline involves two GPU-intensive models: an embedding model and a generation LLM. The table below shows combined requirements. For LLM-specific guidance, see our best GPU for LLM inference guide.

For large-scale deployments, GigaGPU’s multi-GPU clusters provide the bandwidth needed for concurrent embedding and generation.

Setting Up Your GPU Server

Verify your NVIDIA stack and prepare the environment:

sudo apt update && sudo apt upgrade -y
sudo apt install -y python3 python3-pip python3-venv git
nvidia-smi

Create an isolated environment for the RAG pipeline:

python3 -m venv ~/rag-env
source ~/rag-env/bin/activate
pip install --upgrade pip

Install PyTorch with CUDA. Our PyTorch GPU installation guide has version-specific details:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Installing LangChain and Dependencies

Install LangChain along with the vector store and document loader packages:

pip install langchain langchain-community langchain-huggingface
pip install chromadb sentence-transformers
pip install unstructured pypdf python-docx

Install vLLM to serve the LLM backend. See our vLLM production setup guide for advanced configuration:

pip install vllm

Start the LLM server in the background:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --dtype float16 \
  --max-model-len 8192 \
  --port 8000 &

Building the RAG Pipeline

The pipeline has four stages: load documents, create embeddings, store vectors, and query with retrieval. Here is a complete working example:

from langchain_community.document_loaders import DirectoryLoader, PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_community.llms import VLLMOpenAI
from langchain.chains import RetrievalQA

# 1. Load documents
loader = DirectoryLoader("./docs", glob="**/*.pdf", loader_cls=PyPDFLoader)
documents = loader.load()

# 2. Split into chunks
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = splitter.split_documents(documents)

# 3. Create embeddings and vector store
embeddings = HuggingFaceEmbeddings(
    model_name="BAAI/bge-large-en-v1.5",
    model_kwargs={"device": "cuda"}
)
vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory="./chroma_db")

# 4. Connect to vLLM and build the chain
llm = VLLMOpenAI(
    openai_api_base="http://localhost:8000/v1",
    model_name="meta-llama/Meta-Llama-3-8B-Instruct",
    openai_api_key="not-needed",
    max_tokens=512,
    temperature=0.3
)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4})
)

# Query the pipeline
result = qa_chain.invoke({"query": "What are the GPU requirements for LLaMA 3?"})
print(result["result"])

This setup runs entirely on your GPU server with no external API calls. For an alternative retrieval framework, see our guide on how to set up LlamaIndex on a GPU server.

Exposing the RAG API

Wrap the pipeline in a FastAPI endpoint for production use:

pip install fastapi uvicorn

Create a minimal server and launch it:

uvicorn rag_api:app --host 0.0.0.0 --port 8080

Test the RAG endpoint:

curl -X POST http://localhost:8080/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What GPU do I need for a 70B model?"}'

For managed API infrastructure, explore GigaGPU’s API hosting with built-in load balancing.

Production Tips and Next Steps

Harden your RAG pipeline for production workloads:

• Persist your vector store — Use Chroma’s persist_directory or switch to pgvector/Qdrant for multi-server setups.
• Optimise chunk size — Experiment with 500-1500 character chunks; smaller chunks improve precision, larger chunks provide more context.
• Use reranking — Add a cross-encoder reranker (e.g., cross-encoder/ms-marco-MiniLM-L-12-v2) between retrieval and generation for better relevance.
• Scale the LLM — Upgrade to Qwen 72B or LLaMA 3 70B for more nuanced answers. Compare models in our cost per 1M tokens: GPU vs OpenAI analysis.
• Monitor costs — Use our LLM cost calculator to estimate self-hosted RAG expenses.

To explore other LLM options for your RAG backend, read our guides on deploying LLaMA 3, Mistral, and Qwen. Browse all available walkthroughs in our tutorials category.