Log Management for GPU Servers

Inference Logs Filled Your Disk and Killed the Server

A busy inference API generates gigabytes of logs per day. Each request logs prompt tokens, generation parameters, and timing data. Without rotation, those logs consume the same NVMe storage your models need, eventually filling the disk and crashing the inference process. A dedicated GPU server running AI workloads needs log management that captures useful diagnostics without eating storage alive.

Configure journald for GPU Workloads

Systemd services log through journald by default. Set sane limits before logs spiral:

# /etc/systemd/journald.conf
[Journal]
SystemMaxUse=2G
SystemMaxFileSize=200M
SystemKeepFree=10G
MaxRetentionSec=2week
Compress=yes
RateLimitIntervalSec=30s
RateLimitBurst=10000

# Apply changes
sudo systemctl restart systemd-journald

# Verify current disk usage
journalctl --disk-usage

# Manually trim if already oversized
sudo journalctl --vacuum-size=2G
sudo journalctl --vacuum-time=2weeks

Log Rotation for Inference Services

vLLM, Ollama, and custom inference scripts often write their own log files:

# /etc/logrotate.d/ai-inference
/var/log/vllm/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    maxsize 500M
    postrotate
        systemctl reload vllm-inference 2>/dev/null || true
    endscript
}

/var/log/ollama/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    maxsize 500M
}

# Custom inference API logs
/opt/inference/logs/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    maxsize 200M
    create 0640 www-data www-data
}

# Force rotation test
sudo logrotate -d /etc/logrotate.d/ai-inference
sudo logrotate -f /etc/logrotate.d/ai-inference

Structured Logging for AI APIs

Plain text logs are hard to search. Structured JSON logging makes analysis possible:

# Python structured logging for inference APIs
import logging
import json
from datetime import datetime

class InferenceFormatter(logging.Formatter):
    def format(self, record):
        log_entry = {
            "timestamp": datetime.utcnow().isoformat(),
            "level": record.levelname,
            "message": record.getMessage(),
            "module": record.module,
        }
        # Add inference-specific fields if present
        if hasattr(record, "model"):
            log_entry["model"] = record.model
        if hasattr(record, "tokens_generated"):
            log_entry["tokens_generated"] = record.tokens_generated
        if hasattr(record, "latency_ms"):
            log_entry["latency_ms"] = record.latency_ms
        if hasattr(record, "gpu_mem_mb"):
            log_entry["gpu_mem_mb"] = record.gpu_mem_mb
        return json.dumps(log_entry)

# Configure
handler = logging.FileHandler("/var/log/inference/api.jsonl")
handler.setFormatter(InferenceFormatter())
logger = logging.getLogger("inference")
logger.addHandler(handler)
logger.setLevel(logging.INFO)

# Usage in request handler
logger.info("Request completed",
    extra={
        "model": "meta-llama/Llama-3-70B",
        "tokens_generated": 256,
        "latency_ms": 1430,
        "gpu_mem_mb": 38400
    }
)

NVIDIA and GPU-Specific Logs

GPU driver and CUDA logs live in specific locations:

# NVIDIA kernel driver logs
dmesg | grep -i nvidia | tail -20
journalctl -k | grep -i nvrm

# Xid errors (GPU hardware/software faults)
dmesg | grep -i "xid"
# Xid 79 = GPU fallen off bus
# Xid 31 = GPU memory page retirement
# Xid 48 = Double-bit ECC error

# nvidia-smi event log
nvidia-smi -q -d ECC | grep -A 5 "Aggregate"

# Monitor GPU events continuously
nvidia-smi daemon --loop=10 --filename=/var/log/nvidia-smi.log &

# CUDA error logs for debugging
export CUDA_LAUNCH_BLOCKING=1
export CUDA_VISIBLE_DEVICES=0
export NCCL_DEBUG=INFO
export NCCL_DEBUG_FILE=/var/log/nccl-debug.log

Prevent Log-Driven Disk Failures

# Monitor disk usage and alert before logs fill storage
cat <<'EOF' > /opt/scripts/check-log-disk.sh
#!/bin/bash
THRESHOLD=85
USAGE=$(df /var/log --output=pcent | tail -1 | tr -d ' %')

if [ "$USAGE" -gt "$THRESHOLD" ]; then
    echo "WARNING: /var/log at ${USAGE}% capacity" | \
        logger -p user.warning -t "disk-check"

    # Emergency rotation
    logrotate -f /etc/logrotate.d/ai-inference
    journalctl --vacuum-size=1G

    echo "Emergency log rotation triggered at ${USAGE}%"
fi
EOF
chmod +x /opt/scripts/check-log-disk.sh

# Run every 15 minutes
echo "*/15 * * * * root /opt/scripts/check-log-disk.sh" >> /etc/crontab

# Quick check: biggest log consumers
du -sh /var/log/* 2>/dev/null | sort -rh | head -10

Proper log management keeps your GPU server storage healthy while preserving the diagnostics you need for debugging inference issues. For vLLM logging specifics, see the production deployment guide. Monitor Ollama logs alongside GPU usage metrics. Browse our infrastructure guides, tutorials, and benchmarks for related setup.