From a freshly-provisioned 5060 Ti server to a working RAG stack in under an hour. This is the literal install recipe.
TL;DR
Five components, ~50 minutes total. vLLM serves the LLM; TEI serves embeddings; Qdrant is the vector store; LiteLLM routes; a curl test verifies. All in 100 lines of bash.
1. vLLM with Llama 3.1 8B
sudo apt install -y python3.10-venv
python3.10 -m venv ~/llm && source ~/llm/bin/activate
pip install vllm==0.6.3
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
--quantization fp8 \
--gpu-memory-utilization 0.55 \
--port 8000 &2. Embeddings with TEI
docker run -d --gpus all -p 8001:80 \
-v /data/embed-cache:/data \
ghcr.io/huggingface/text-embeddings-inference:latest \
--model-id BAAI/bge-large-en-v1.53. Qdrant
docker run -d -p 6333:6333 -p 6334:6334 \
-v /data/qdrant:/qdrant/storage \
qdrant/qdrant4. LiteLLM
pip install litellm[proxy]
cat > ~/litellm-config.yaml <<EOF
model_list:
- model_name: chat
litellm_params:
model: openai/Meta-Llama-3.1-8B-Instruct
api_base: http://localhost:8000/v1
EOF
litellm --config ~/litellm-config.yaml --port 4000 &5. Test
curl http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-d "{\"model\":\"chat\",\"messages\":[{\"role\":\"user\",\"content\":\"hello\"}]}"Bottom line
~50 minutes from blank server to RAG stack. Add your application code on top. See RAG architecture guide.
