How to Set Up LlamaIndex on a Dedicated GPU Server

Why Use LlamaIndex on a GPU Server

LlamaIndex (formerly GPT Index) is one of the leading frameworks for building data-augmented LLM applications. It provides powerful abstractions for document loading, indexing, retrieval, and query synthesis, making it straightforward to connect your private data to a language model. Running LlamaIndex on a dedicated GPU server lets you host both the embedding models and the LLM locally, keeping your documents private and eliminating per-query API costs.

GigaGPU’s LlamaIndex hosting provides GPU infrastructure purpose-built for retrieval-augmented generation workloads. Whether you are building a knowledge base chatbot, a document Q&A system, or an agentic research tool, dedicated hardware ensures consistent performance. For a comparison of retrieval frameworks, see how LlamaIndex compares to the LangChain approach in our LangChain RAG pipeline tutorial. GigaGPU’s RAG hosting platform supports both frameworks.

GPU VRAM Requirements

LlamaIndex itself is CPU-based, but the embedding models and LLMs it calls require GPU acceleration. The table below shows combined VRAM for typical configurations. See our best GPU for LLM inference guide for more hardware details.

For multi-GPU configurations, GigaGPU offers multi-GPU cluster hosting with high-bandwidth NVLink.

Preparing Your GPU Server

Start by verifying your NVIDIA installation:

sudo apt update && sudo apt upgrade -y
sudo apt install -y python3 python3-pip python3-venv git
nvidia-smi

Create a virtual environment:

python3 -m venv ~/llamaindex-env
source ~/llamaindex-env/bin/activate
pip install --upgrade pip

Install PyTorch with CUDA. Follow our PyTorch GPU installation guide for version details:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Start a local vLLM server to act as the LLM backend. See our vLLM production setup guide for advanced configuration:

pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --dtype float16 \
  --max-model-len 8192 \
  --port 8000 &

Installing LlamaIndex

Install LlamaIndex core along with the integrations needed for local LLM and embedding models:

pip install llama-index llama-index-llms-openai-like llama-index-embeddings-huggingface
pip install llama-index-vector-stores-chroma chromadb
pip install llama-index-readers-file pypdf python-docx

Verify the installation:

python -c "import llama_index; print(llama_index.__version__)"

Building a Document Index

Create a complete document indexing pipeline. Place your PDF or text files in a ./documents directory, then run:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai_like import OpenAILike
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.core.storage import StorageContext
import chromadb

# Configure the local LLM (vLLM backend)
Settings.llm = OpenAILike(
    api_base="http://localhost:8000/v1",
    api_key="not-needed",
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    max_tokens=512,
    temperature=0.3,
    is_chat_model=True
)

# Configure local embeddings (GPU-accelerated)
Settings.embed_model = HuggingFaceEmbedding(
    model_name="BAAI/bge-large-en-v1.5",
    device="cuda"
)

# Load documents
documents = SimpleDirectoryReader("./documents").load_data()

# Create persistent Chroma vector store
chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("documents")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# Build the index
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
    show_progress=True
)

print(f"Indexed {len(documents)} documents successfully.")

Creating a Query Engine

Once the index is built, create a query engine and test it:

# Create query engine with retrieval parameters
query_engine = index.as_query_engine(
    similarity_top_k=4,
    response_mode="compact"
)

# Query the index
response = query_engine.query("What are the system requirements?")
print(response)

Wrap the query engine in a FastAPI server for production access:

pip install fastapi uvicorn

uvicorn llamaindex_api:app --host 0.0.0.0 --port 8080

Test the API:

curl -X POST http://localhost:8080/query \
  -H "Content-Type: application/json" \
  -d '{"question": "Summarise the deployment requirements."}'

For managed API hosting, see GigaGPU’s API hosting platform with built-in SSL and load balancing.

Production Tips and Next Steps

Harden your LlamaIndex deployment for production:

• Persist your index — Use Chroma’s PersistentClient so you do not need to re-embed documents on every restart.
• Tune chunk parameters — Adjust chunk_size (default 1024) and chunk_overlap (default 20) in Settings to balance retrieval precision and context length.
• Add a reranker — Use LlamaIndex’s SentenceTransformerRerank node postprocessor to improve retrieval quality.
• Upgrade the LLM — Switch to Qwen 72B or LLaMA 3 70B for more nuanced answers. See our cost per 1M tokens: GPU vs OpenAI analysis for budget planning.
• Calculate costs — Use our LLM cost calculator to estimate your self-hosted running costs.

To explore other LLM backends, read our deployment guides for LLaMA 3, Mistral, and Gemma. You can also build a similar pipeline using LangChain — see our LangChain RAG pipeline guide. Browse all walkthroughs in our tutorials category.