Docker GPU Passthrough Not Working: Fix Guide

The Problem: Docker Containers Cannot Access GPUs

You run a GPU workload inside Docker on your dedicated GPU server and hit one of these:

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]

nvidia-smi
Failed to initialize NVML: Unknown Error

docker run --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
# Returns: no NVIDIA GPU found

The host machine shows GPUs via nvidia-smi, but Docker containers are completely blind to them. This is expected default behaviour — Docker isolates hardware resources. GPU access requires explicit configuration through the NVIDIA Container Toolkit.

Why Docker Does Not See GPUs by Default

Docker uses Linux cgroups and namespaces to isolate containers from host hardware. GPU passthrough requires three things to be in place:

1. NVIDIA driver on the host — must be installed and functional.
2. NVIDIA Container Toolkit — provides the runtime hook that exposes GPU devices to containers.
3. Docker runtime configuration — Docker must be configured to use the NVIDIA runtime.

If any of these is missing or misconfigured, GPU passthrough fails.

Step 1: Verify the Host GPU Works

nvidia-smi

If this fails on the host, fix the driver first. Our CUDA installation guide covers driver setup. Docker GPU passthrough cannot work without a functioning host driver.

Step 2: Install the NVIDIA Container Toolkit

# Add the repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Install
sudo apt update
sudo apt install -y nvidia-container-toolkit

# Configure Docker to use the NVIDIA runtime
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Step 3: Test GPU Passthrough

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

This should display the same GPU information as running nvidia-smi on the host. If it works, GPU passthrough is functional.

Troubleshooting Common Failures

Error: could not select device driver

The NVIDIA runtime is not registered with Docker. Verify the configuration:

cat /etc/docker/daemon.json

It should contain:

{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

If not, run sudo nvidia-ctk runtime configure --runtime=docker again and restart Docker.

Error: Failed to initialize NVML inside container

This often means the container’s CUDA version exceeds what the host driver supports. Use a container image whose CUDA version is less than or equal to the host driver’s CUDA ceiling.

Permission denied on /dev/nvidia*

The user inside the container may lack permissions. Add the user to the video group or run with --privileged for testing:

docker run --rm --gpus all --privileged nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

Multi-GPU and Specific GPU Assignment

On multi-GPU servers, you can assign specific GPUs to containers:

# All GPUs
docker run --gpus all ...

# Specific GPU by index
docker run --gpus '"device=0"' ...

# Multiple specific GPUs
docker run --gpus '"device=0,2"' ...

# By UUID
docker run --gpus '"device=GPU-abc123"' ...

This is essential for running multiple workloads — a vLLM container on GPU 0 and a Stable Diffusion container on GPU 1, for example.

Production Docker GPU Configuration

For production inference services on your GPU server:

docker run -d \
  --gpus '"device=0"' \
  --shm-size=16g \
  --restart=unless-stopped \
  --name inference-server \
  -p 8000:8000 \
  your-inference-image

• --shm-size=16g is required for multi-GPU NCCL communication and large batch processing.
• --restart=unless-stopped ensures the container survives host reboots.
• For PyTorch and TensorFlow workloads, the official GPU Docker images include CUDA and cuDNN.

See our complete Docker GPU workloads guide for compose files, networking, and security hardening. Monitor GPU health inside containers using our monitoring setup.