GPU Power Management: Persistence Mode

Your First Inference Request Takes 10x Longer Than Expected

The initial request to your AI API takes seconds when subsequent requests complete in milliseconds. NVIDIA GPUs enter low-power states when idle, spinning down clocks and unloading driver state. The first CUDA call after idle triggers a cold initialization that adds significant latency. On a dedicated GPU server running always-on inference, persistence mode and proper power management eliminate these delays while controlling energy consumption.

Enable NVIDIA Persistence Mode

Persistence mode keeps the GPU driver loaded and ready between CUDA calls:

# Check current persistence mode
nvidia-smi -q | grep "Persistence Mode"

# Enable persistence mode (temporary, resets on reboot)
sudo nvidia-smi -pm 1

# Enable for all GPUs
sudo nvidia-smi -pm 1 -i 0,1,2,3

# Verify
nvidia-smi --query-gpu=index,persistence_mode --format=csv
# 0, Enabled
# 1, Enabled

# Permanent: use nvidia-persistenced daemon
sudo systemctl enable nvidia-persistenced
sudo systemctl start nvidia-persistenced

# Verify the daemon is running
systemctl status nvidia-persistenced

# Without persistence mode:
# First CUDA call: ~500ms-2s (driver initialization)
# With persistence mode:
# First CUDA call: ~5-10ms (driver already loaded)

Set GPU Power Limits

Inference workloads rarely need full TDP. Reducing power limits saves energy and reduces heat without significant performance loss:

# Check current power limits
nvidia-smi --query-gpu=index,power.limit,power.default_limit,power.max_limit \
    --format=csv

# Example output for RTX 6000 Pro:
# 0, 300.00 W, 300.00 W, 400.00 W

# Set power limit to 250W (inference rarely needs 300W)
sudo nvidia-smi -pl 250 -i 0

# For inference-optimized power across all GPUs
for GPU_ID in 0 1 2 3; do
    sudo nvidia-smi -pl 250 -i $GPU_ID
done

# Persist across reboots with a systemd oneshot service
cat <<'EOF' | sudo tee /etc/systemd/system/gpu-power-config.service
[Unit]
Description=Configure GPU Power Limits
After=nvidia-persistenced.service

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi -pm 1
ExecStart=/usr/bin/nvidia-smi -pl 250
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl enable gpu-power-config

Lock GPU Clocks for Consistent Inference

GPU boost clocks fluctuate based on thermal headroom. Locking clocks provides predictable latency:

# Query supported clock speeds
nvidia-smi -q -d SUPPORTED_CLOCKS | head -30

# Lock GPU and memory clocks (RTX 6000 Pro example)
sudo nvidia-smi -lgc 1410,1410 -i 0  # GPU clocks: min,max
sudo nvidia-smi -lmc 1593 -i 0        # Memory clock

# Verify locked clocks
nvidia-smi --query-gpu=clocks.gr,clocks.mem --format=csv

# Reset to default (let GPU boost freely)
sudo nvidia-smi -rgc -i 0
sudo nvidia-smi -rmc -i 0

# For training (need max performance): lock at highest stable clock
sudo nvidia-smi -lgc 1980,1980 -i 0

# For inference (need consistency over peak): lock at moderate clock
sudo nvidia-smi -lgc 1410,1410 -i 0

# Monitor actual vs requested clocks
watch -n 1 'nvidia-smi --query-gpu=clocks.gr,clocks.max.gr,clocks.mem,clocks.max.mem --format=csv'

Monitor Power Draw

# Real-time power monitoring
nvidia-smi --query-gpu=index,power.draw,power.limit,temperature.gpu,clocks.gr \
    --format=csv -l 5

# Log power data for analysis
nvidia-smi --query-gpu=timestamp,index,power.draw,utilization.gpu,temperature.gpu \
    --format=csv -l 10 -f /var/log/gpu-power.csv &

# Calculate energy cost estimate (per GPU per month)
# If average draw is 220W at $0.10/kWh:
# 0.220 kW * 24 hours * 30 days * $0.10 = $15.84/month/GPU

# Compare throughput at different power limits
# Run this benchmark at each power limit:
for PL in 200 250 300; do
    sudo nvidia-smi -pl $PL -i 0
    sleep 5
    echo "=== Power limit: ${PL}W ==="
    python3 -c "
import torch, time
x = torch.randn(4096, 4096, device='cuda')
torch.cuda.synchronize()
start = time.time()
for _ in range(1000):
    y = torch.mm(x, x)
torch.cuda.synchronize()
elapsed = time.time() - start
print(f'  TFLOPS: {2 * 4096**3 * 1000 / elapsed / 1e12:.1f}')
"
done

Power Management Best Practices

# Complete GPU power configuration for inference servers
#!/bin/bash
# /opt/scripts/gpu-power-setup.sh

# Enable persistence mode
nvidia-smi -pm 1

# Set inference-optimized power limits
GPU_COUNT=$(nvidia-smi --query-gpu=count --format=csv,noheader | head -1)
for ((i=0; i