How to Self-Host an LLM on a Dedicated GPU Server

Why Self-Host an LLM?

Every API call to a hosted LLM provider sends your data to a third party, adds latency, and charges you per token. Self-hosting on a dedicated GPU server eliminates all three problems. You get full control over your data, sub-50ms latency for local requests, and a fixed monthly cost regardless of how many tokens you generate.

Self-hosting is especially compelling for teams handling sensitive data under GDPR, building customer-facing chatbots that need reliable uptime, or running high-volume batch workloads where API costs spiral. With open source LLM hosting, you can serve models like LLaMA 3, Mistral, and DeepSeek from your own infrastructure without vendor lock-in.

This guide walks you through the full process: selecting hardware, installing drivers, deploying an inference engine, and exposing an OpenAI-compatible API. No prior GPU server experience needed.

Choosing the Right GPU Server

Your GPU determines which models you can run and at what speed. VRAM is the critical constraint. The table below shows common model sizes and the minimum GPU required:

For most production use cases at 7B-13B scale, the RTX 3090 offers the best value. Our best GPU for LLM inference guide benchmarks every card we offer. If you’re weighing specific cards, the RTX 3090 vs RTX 5090 comparison breaks down the trade-offs.

Server Setup & Dependencies

Once your dedicated server is provisioned, SSH in and install the core dependencies. All GigaGPU servers ship with Ubuntu 22.04 and NVIDIA drivers pre-installed, but here’s the full sequence if you’re starting from scratch:

# Update system packages
sudo apt update && sudo apt upgrade -y

# Install NVIDIA driver (if not pre-installed)
sudo apt install -y nvidia-driver-535
sudo reboot

# Verify GPU is detected
nvidia-smi

# Install Python 3.10+ and pip
sudo apt install -y python3.10 python3.10-venv python3-pip

# Create a virtual environment
python3.10 -m venv ~/llm-env
source ~/llm-env/bin/activate

Confirm your GPU shows up in nvidia-smi with the expected VRAM before proceeding. If you’re running multiple models, consider partitioning them across separate virtual environments.

Deploying with vLLM

vLLM is the production-grade choice for LLM inference. It uses PagedAttention to maximise VRAM efficiency and supports continuous batching for high-throughput serving. Install and launch an OpenAI-compatible API in three commands:

# Install vLLM
pip install vllm

# Start an OpenAI-compatible API server with Mistral 7B
python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 4096

Test the endpoint with a curl request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.3",
    "messages": [{"role": "user", "content": "Explain PagedAttention in one paragraph."}],
    "max_tokens": 256
  }'

vLLM automatically downloads the model from Hugging Face on first launch. For larger models like LLaMA 3 or DeepSeek, pre-download the weights to avoid timeout issues. Check our vLLM vs Ollama comparison to understand when vLLM is the right choice.

Deploying with Ollama

Ollama is the fastest way to get a model running if you want simplicity over maximum throughput. It bundles model management, quantisation, and serving into a single binary:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull and run Mistral 7B
ollama pull mistral
ollama run mistral

# Serve as an API (runs on port 11434 by default)
OLLAMA_HOST=0.0.0.0 ollama serve

Query the Ollama API:

curl http://localhost:11434/api/chat \
  -d '{
    "model": "mistral",
    "messages": [{"role": "user", "content": "What is self-hosted AI?"}]
  }'

Ollama is ideal for development, prototyping, and single-user workloads. For production deployments serving multiple concurrent users, vLLM’s continuous batching delivers significantly higher throughput. See the LLM hosting category for more deployment guides.

Production Hardening

A running model is not a production service. Apply these steps before exposing your API to users:

1. Run behind a reverse proxy:

# Install nginx
sudo apt install -y nginx

# Basic reverse proxy config
server {
    listen 443 ssl;
    server_name llm.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/llm.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/llm.yourdomain.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_read_timeout 300s;
    }
}

2. Add API key authentication: Use nginx auth_request or a lightweight middleware to require an API key on every request.

3. Create a systemd service:

# /etc/systemd/system/vllm.service
[Unit]
Description=vLLM Inference Server
After=network.target

[Service]
User=llm
WorkingDirectory=/home/llm
ExecStart=/home/llm/llm-env/bin/python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3 --host 127.0.0.1 --port 8000
Restart=always

[Install]
WantedBy=multi-user.target

4. Monitor GPU utilisation: Use nvidia-smi dmon or Prometheus with the NVIDIA GPU exporter to track VRAM usage, temperature, and throughput.

For cost monitoring, our cost per million tokens tool helps you track the economics of self-hosting versus API providers.

Next Steps

You now have a fully functional self-hosted LLM running on dedicated hardware. From here, consider:

• Scaling throughput: Switch to vLLM if you started with Ollama, or add tensor parallelism across a multi-GPU setup
• Serving multiple models: Run Mistral and LLaMA side by side on a high-VRAM card
• Comparing costs: Use the GPU vs API cost comparison tool to quantify your savings over OpenAI or Anthropic
• Exploring alternatives: If you’re migrating from a cloud GPU provider, see our RunPod alternatives guide

Self-hosting puts you in control. Pick a dedicated GPU server, follow this guide, and you’ll have a private AI API running within the hour.