The default ML deployment guide of 2024-2026 assumes Kubernetes. For single-server self-hosted inference on a dedicated GPU, that’s overkill. systemd handles process management, restart policies, logging, and dependency ordering with zero added infrastructure.
For a single GPU server: systemd. For multi-server clusters or auto-scaling: Kubernetes. For one-server-but-want-portability: K3s or Docker Compose. Most teams over-engineer this.
When systemd wins
- Single GPU server (or 2-3 servers under one operator)
- You want to ship in a week, not a quarter
- Operational complexity is the bottleneck
- Your team has Linux experience but not Kubernetes experience
Example unit file for vLLM:
[Unit]
Description=vLLM Mistral 7B
After=network-online.target nvidia-persistenced.service
Wants=network-online.target
[Service]
Type=simple
User=ubuntu
WorkingDirectory=/home/ubuntu
Environment="HF_HOME=/data/hf-cache"
ExecStart=/home/ubuntu/venv/bin/vllm serve \
mistralai/Mistral-7B-Instruct-v0.3 \
--host 0.0.0.0 --port 8000 \
--quantization fp8 \
--enable-prefix-caching
Restart=on-failure
RestartSec=10
KillMode=mixed
TimeoutStopSec=30
[Install]
WantedBy=multi-user.targetWhen Kubernetes wins
- Multi-server (5+) deployments
- Auto-scaling on traffic
- Multi-tenant with strong isolation needs
- You already operate K8s for non-AI workloads
- You need rolling updates with zero downtime
Hybrid: K3s on a single GPU server
K3s is a lightweight Kubernetes distribution that runs comfortably on a single GPU server. Useful when:
- You want declarative deployment (Helm charts)
- You may add a second server later
- You want network policies for tenant isolation
NVIDIA’s k8s-device-plugin handles GPU scheduling. Setup is ~30 minutes.
Verdict
- Single server, ship fast: systemd
- Want some K8s benefits without full K8s: K3s
- Multi-server or auto-scale: full Kubernetes
Bottom line
Most self-hosted AI deployments are single-server and don't need Kubernetes. systemd is fine and dramatically simpler. See build a production AI inference server.
