RTX 4090 24GB First-Day Checklist: Verify, Secure, Deploy, Monitor

You have just provisioned a fresh RTX 4090 24GB dedicated server. Before you start training or serving, spend an hour on the basics: verify the hardware is what you ordered, harden SSH, install a clean CUDA stack, set up monitoring and capture a baseline benchmark you can refer back to. Skipping this hour is the single biggest cause of “my GPU is slow” tickets two weeks later, where the answer turns out to be a wrong driver, a thermal-throttled chassis, or a missing FlashAttention build. This is the same checklist we hand to GigaGPU dedicated hosting customers on day one.

Contents

• Verify the hardware
• Harden SSH and firewall
• Install the CUDA stack
• Set up monitoring
• First benchmark
• Backup and snapshots
• Production gotchas

Verify the hardware

Run these commands and keep the output for support tickets and capacity planning. Anything outside the expected ranges is a support ticket, not a thing to debug yourself.

nvidia-smi
nvidia-smi --query-gpu=name,memory.total,driver_version,vbios_version,pcie.link.gen.current,pcie.link.width.current --format=csv
nvidia-smi -q | grep -E "(Persistence|Power|PCIe|Bus Id|Compute Mode)"
lscpu | head -20
free -h
df -hT
ls /sys/class/nvme/ && sudo nvme list

For background on the spec see spec breakdown and PCIe Gen4 x16.

Harden SSH and firewall

The single highest-impact security step. Disable password auth, enable key-only login, restrict ports.

# add your key
mkdir -p ~/.ssh && chmod 700 ~/.ssh
echo "ssh-ed25519 AAAA..." >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

# disable password login
sudo sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo sed -i 's/^#*PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
sudo sed -i 's/^#*PubkeyAuthentication.*/PubkeyAuthentication yes/' /etc/ssh/sshd_config
sudo systemctl restart ssh

# UFW: SSH + your inference port only
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 8000/tcp   # vLLM
sudo ufw --force enable

# fail2ban for SSH
sudo apt install -y fail2ban
sudo systemctl enable --now fail2ban

Put inference ports behind a reverse proxy with TLS (Caddy is the easiest path) rather than exposing them directly to the public internet. If you serve traffic from multiple IPs, add a UFW allowlist by source.

Install the CUDA stack

# minimal install
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# verify
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If the final container command fails, the most common cause is a stale containerd config from a previous toolkit install. sudo nvidia-ctk runtime configure --runtime=docker --set-as-default usually fixes it.

Set up monitoring

Install nvtop for live GPU view, plus a Prometheus exporter for long-term graphs:

sudo apt install -y nvtop htop iotop
docker run -d --gpus all --name dcgm \
  --restart unless-stopped \
  -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:3.3.7-3.5.0-ubuntu22.04

Point a Prometheus + Grafana stack at :9400/metrics. The dashboards you actually want: GPU utilisation, VRAM used, power draw, hot-spot temperature, edge temperature, PCIe Rx/Tx, ECC errors. Healthy 4090 ranges:

For more on the thermal envelope see thermal performance and power draw efficiency.

First benchmark

Use a single repeatable script. Llama 3.1 8B FP8 with vLLM is the canonical baseline because it exercises the FP8 tensor cores, FlashAttention path and PCIe simultaneously.

pip install vllm==0.6.3
python -m vllm.entrypoints.openai.api_server \
  --model neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8 \
  --max-model-len 8192 --gpu-memory-utilization 0.92 &

# wait for the server to come up
sleep 60

# benchmark
pip install vllm-benchmark
python -m vllm.entrypoints.openai.bench \
  --model neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8 \
  --num-prompts 200 --request-rate 16 --output baseline.json

Save the JSON output as your baseline and re-run after every driver, vLLM or kernel update. For the full sweep see the Llama 3 8B benchmark and FP8 deployment guide. For 70B and Qwen 32B baselines see the 70B INT4 benchmark and 4090 for Qwen 32B.

Backup and snapshots

Set a nightly rsync of /etc, /home, and any LoRA checkpoints to remote storage. For model weights, prefer to re-download from Hugging Face rather than back them up; saves bandwidth on restore. Useful crontab line:

0 3 * * * rsync -aHAX --delete /home/ /backups/home-$(date +\%u)/ && \
          rsync -aHAX --delete /etc/ /backups/etc-$(date +\%u)/

Keep adapters, training data and configs in git or S3. Anything reproducible from public sources is not worth backing up.

Production gotchas

1. Wrong driver version: drivers older than 535 silently disable FP8 tensor cores. Always verify with nvidia-smi on day one.
2. PCIe down-trained to Gen3: a poorly-seated card or BIOS issue can leave you on Gen3 x16 and you only notice when training tokens/sec is half what it should be.
3. Persistence mode off: without nvidia-smi -pm 1 the driver unloads between processes, adding 2-3 seconds of cold-start latency. Enable persistence on all production boxes.
4. Default Docker non-root: forgetting usermod -aG docker $USER means every docker run needs sudo, which breaks systemd unit files.
5. Open inference port: leaving vLLM’s port 8000 open to the world is the most common security incident on dedicated GPU boxes. Always proxy and authenticate.
6. No swap, paged AdamW fails: bitsandbytes paged optimisers need either system RAM or swap. With 64 GB RAM and no swap, the 70B QLoRA from the QLoRA fine-tune guide OOMs on optimiser allocation.
7. Disk filling silently: vLLM logs, checkpoint files and Docker images can fill 2 TB faster than you think. Set up a logrotate config and a disk-usage Prometheus alert at 80%.

Verdict

An hour of day-one verification saves you days of “why is my GPU slow” debugging later. Verify the card, harden SSH, install the CUDA 12.4 stack, set up DCGM monitoring and capture a Llama 3 8B FP8 baseline. After this you can deploy the vLLM setup, run the LoRA fine-tune or stand up the ComfyUI stack with confidence. For the cost case before you order, see the monthly hosting cost writeup.

See also: vLLM setup, Llama 3 8B benchmark, thermal performance, spec breakdown, 5060 Ti checklist, monthly cost, PCIe Gen4 x16.