How to Set Up Ollama on a Dedicated GPU Server

Why Use Ollama on Dedicated Hardware

Ollama makes running large language models remarkably simple. A single command downloads, quantises, and serves a model with GPU acceleration. Running Ollama on a dedicated GPU server combines that simplicity with the performance and privacy of bare-metal hardware. GigaGPU offers managed Ollama hosting for teams that want zero-setup deployment, but this guide covers the full manual setup for those who want complete control.

Unlike running Ollama on a laptop or desktop, a dedicated server provides 24/7 uptime, static IP addresses for API access, enterprise-grade GPUs with 24-80 GB of VRAM, and the bandwidth to serve multiple concurrent users. It is the foundation for turning a local experiment into a production-ready private AI service.

Installation and GPU Verification

Start with a GigaGPU server running Ubuntu 22.04 with CUDA drivers pre-installed. Ollama installation is a single command:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version

# Check GPU detection
nvidia-smi
ollama list

Ollama automatically detects NVIDIA GPUs and uses them for inference. Verify this by checking GPU utilization while running a model:

# Start a model and watch GPU usage
ollama run llama3:8b "Hello, world" &
watch -n 1 nvidia-smi

If GPU utilization spikes during inference, everything is working correctly. If the model runs on CPU instead, check that the NVIDIA drivers are properly installed. Our GPU driver setup guide covers troubleshooting steps that apply to any CUDA-dependent application.

Downloading and Managing Models

Ollama provides a curated library of pre-quantised models optimised for inference. Pull models based on your GPU’s VRAM capacity:

# Pull popular models
ollama pull llama3:8b          # 4.7 GB - fits on any modern GPU
ollama pull llama3:70b         # 40 GB - needs RTX 6000 Pro/RTX 6000 Pro or multi-GPU
ollama pull mistral:7b         # 4.1 GB - fast and capable
ollama pull deepseek-r1:14b   # 9 GB - strong reasoning model
ollama pull codellama:13b      # 7.4 GB - code generation
ollama pull phi3:14b           # 7.9 GB - compact but capable

# List downloaded models
ollama list

# Show model details
ollama show llama3:8b

# Remove a model
ollama rm codellama:13b

For help choosing the right GPU for your target model, see our GPU selection guide for LLM inference.

API Configuration and Remote Access

By default, Ollama serves its API on localhost:11434. To allow remote access from your applications, configure it to listen on all interfaces:

# Set environment variable for Ollama service
sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo tee /etc/systemd/system/ollama.service.d/override.conf > /dev/null << 'EOF'
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_KEEP_ALIVE=24h"
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_MAX_LOADED_MODELS=2"
EOF

# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart ollama

Key configuration variables:

• OLLAMA_HOST: Bind address and port for the API
• OLLAMA_KEEP_ALIVE: How long to keep models loaded in GPU memory (default 5m)
• OLLAMA_NUM_PARALLEL: Maximum concurrent requests per model
• OLLAMA_MAX_LOADED_MODELS: Maximum models loaded simultaneously

Test remote API access:

# From your local machine or application server
curl http://YOUR_SERVER_IP:11434/api/generate -d '{
  "model": "llama3:8b",
  "prompt": "Explain GPU inference in one paragraph.",
  "stream": false
}'

# OpenAI-compatible endpoint
curl http://YOUR_SERVER_IP:11434/v1/chat/completions -d '{
  "model": "llama3:8b",
  "messages": [{"role": "user", "content": "What is GPU inference?"}]
}'

Running Multiple Models

One advantage of dedicated GPU servers is running multiple models simultaneously. With sufficient VRAM, Ollama can keep several models loaded and route requests to the appropriate one:

# Configure for multi-model serving
sudo tee /etc/systemd/system/ollama.service.d/override.conf > /dev/null << 'EOF'
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_KEEP_ALIVE=24h"
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_MAX_LOADED_MODELS=3"
EOF

sudo systemctl daemon-reload
sudo systemctl restart ollama

# Pre-load models by sending a dummy request
curl http://localhost:11434/api/generate -d '{"model":"llama3:8b","prompt":"hi","stream":false}'
curl http://localhost:11434/api/generate -d '{"model":"mistral:7b","prompt":"hi","stream":false}'

# Check which models are loaded
curl http://localhost:11434/api/ps

On a 24 GB GPU, you can typically keep two 7B models loaded simultaneously. On a 48 GB RTX 6000 Pro, three or four smaller models fit comfortably. For larger models that exceed your VRAM, consider multi-GPU configurations.

Production Setup and Hardening

Harden your Ollama deployment for production use with an Nginx reverse proxy for TLS and basic authentication:

# Install Nginx and create password file
sudo apt install -y nginx apache2-utils
sudo htpasswd -c /etc/nginx/.htpasswd apiuser

# /etc/nginx/sites-available/ollama
server {
    listen 443 ssl http2;
    server_name llm.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/llm.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/llm.yourdomain.com/privkey.pem;

    auth_basic "LLM API";
    auth_basic_user_file /etc/nginx/.htpasswd;

    location / {
        proxy_pass http://127.0.0.1:11434;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 300s;
        proxy_buffering off;  # Important for streaming
    }
}

# Enable and test
sudo ln -s /etc/nginx/sites-available/ollama /etc/nginx/sites-enabled/
sudo certbot --nginx -d llm.yourdomain.com
sudo nginx -t && sudo systemctl reload nginx

Monitor your Ollama deployment with GPU utilization and request logging. Set up log rotation and basic alerting on service failures. The tokens-per-second benchmark tool helps validate that your Ollama deployment is performing at expected levels.

When to Use Ollama vs vLLM

Ollama excels at simplicity and flexibility. It is the right choice when you want quick setup, easy model switching, and multi-model serving without complex configuration. For detailed comparison, see our vLLM vs Ollama analysis.

Choose Ollama when:

• You need to switch between models frequently
• Simplicity is more important than maximum throughput
• You want a single tool for downloading, managing, and serving models
• You are building a development or staging environment

Choose vLLM when:

• Maximum throughput and minimum latency are critical
• You serve a single model at high volume
• You need advanced features like continuous batching and PagedAttention
• Your deployment is a high-traffic production API

Both frameworks run well on GigaGPU hardware. For production LLM APIs, many teams start with Ollama for development and migrate to vLLM for production. Explore the tutorials section for deployment guides covering both approaches across different model families.