Cron Jobs: Model Updates & Backups

Manual Maintenance Does Not Scale

You SSH in every Monday to check for model updates, manually copy checkpoints to backup storage, and clear old logs. Eventually you forget, the disk fills up, and the inference server crashes at 2 AM. On a GPU server running production AI workloads, cron jobs automate these repetitive tasks so they happen reliably whether you are awake or not.

Automated Model Updates

Pull updated model weights on a schedule without interrupting serving:

#!/bin/bash
# /opt/scripts/update-models.sh
# Downloads new model versions during off-peak hours

set -euo pipefail
LOCKFILE="/tmp/model-update.lock"
LOGFILE="/var/log/model-updates.log"
MODEL_DIR="/models/huggingface"

# Prevent concurrent runs
exec 200>"$LOCKFILE"
flock -n 200 || { echo "Update already running"; exit 0; }

log() { echo "$(date '+%Y-%m-%d %H:%M:%S') $1" >> "$LOGFILE"; }

log "Starting model update check"

# Update Hugging Face model
export HF_HOME="$MODEL_DIR"
python3 -c "
from huggingface_hub import snapshot_download
snapshot_download(
    'meta-llama/Meta-Llama-3.1-8B-Instruct',
    local_dir='$MODEL_DIR/Meta-Llama-3.1-8B-Instruct',
    local_dir_use_symlinks=False
)
" 2>&1 | tee -a "$LOGFILE"

# Verify download integrity
python3 -c "
from safetensors import safe_open
import glob
files = glob.glob('$MODEL_DIR/Meta-Llama-3.1-8B-Instruct/*.safetensors')
for f in files:
    safe_open(f, framework='pt')
print(f'Verified {len(files)} safetensors files')
" >> "$LOGFILE" 2>&1

log "Model update complete"

# Crontab entry: run at 3 AM on Sundays
# 0 3 * * 0 /opt/scripts/update-models.sh

Checkpoint Backup Automation

Back up training checkpoints and model files to remote storage:

#!/bin/bash
# /opt/scripts/backup-models.sh
# Incremental backup of model files to remote storage

set -euo pipefail
LOGFILE="/var/log/model-backups.log"
SOURCE="/models"
DEST="s3://my-bucket/gpu-server-backups/models"
RETENTION_DAYS=30

log() { echo "$(date '+%Y-%m-%d %H:%M:%S') $1" >> "$LOGFILE"; }

log "Starting model backup"

# Sync to S3 (only changed files)
aws s3 sync "$SOURCE" "$DEST" \
    --exclude "*.tmp" \
    --exclude "__pycache__/*" \
    --exclude ".cache/*" \
    --storage-class STANDARD_IA \
    2>&1 | tee -a "$LOGFILE"

# Remove old backups beyond retention
aws s3 ls "$DEST" --recursive | \
    awk -v cutoff="$(date -d "-${RETENTION_DAYS} days" +%Y-%m-%d)" \
    '$1 < cutoff {print $4}' | \
    while read -r key; do
        aws s3 rm "s3://my-bucket/$key"
        log "Removed old backup: $key"
    done

log "Backup complete: $(aws s3 ls "$DEST" --summarize | tail -1)"

# Crontab: daily at 4 AM
# 0 4 * * * /opt/scripts/backup-models.sh

Disk Cleanup and Log Rotation

Prevent disk full errors that crash inference servers:

#!/bin/bash
# /opt/scripts/cleanup-gpu-server.sh

set -euo pipefail

# Remove old PyTorch cache files (>7 days)
find /tmp -name "torch_*" -mtime +7 -delete 2>/dev/null
find /root/.cache/torch -name "*.pt" -mtime +30 -delete 2>/dev/null

# Clean Hugging Face cache (keep only current model versions)
python3 -c "
from huggingface_hub import scan_cache_dir
cache = scan_cache_dir()
for repo in cache.repos:
    revisions = sorted(repo.revisions, key=lambda r: r.last_modified)
    # Keep latest, delete older revisions
    for rev in revisions[:-1]:
        print(f'Deleting old revision: {rev.commit_hash[:8]}')
        cache.delete_revisions(rev.commit_hash).execute()
"

# Compress old logs
find /var/log -name "*.log" -mtime +7 -exec gzip {} \;

# Remove old compressed logs
find /var/log -name "*.log.gz" -mtime +90 -delete

# Report disk usage
echo "Disk usage after cleanup:"
df -h / /models

# Crontab: daily at 5 AM
# 0 5 * * * /opt/scripts/cleanup-gpu-server.sh >> /var/log/cleanup.log 2>&1

GPU-Aware Scheduling

Schedule maintenance jobs only when the GPU is idle to avoid impacting inference:

#!/bin/bash
# /opt/scripts/gpu-aware-wrapper.sh
# Only runs the wrapped command if GPU utilization is below threshold

THRESHOLD=20  # percent
COMMAND="$@"

GPU_UTIL=$(nvidia-smi --query-gpu=utilization.gpu \
    --format=csv,noheader,nounits | head -1 | tr -d ' ')

if [ "$GPU_UTIL" -gt "$THRESHOLD" ]; then
    echo "GPU busy (${GPU_UTIL}%), deferring: $COMMAND"
    exit 0
fi

echo "GPU idle (${GPU_UTIL}%), running: $COMMAND"
exec $COMMAND

# Usage in crontab:
# 0 3 * * * /opt/scripts/gpu-aware-wrapper.sh /opt/scripts/update-models.sh

Monitoring Cron Job Health

# /opt/scripts/cron-health-check.sh
#!/bin/bash
# Verify cron jobs ran successfully

JOBS=("model-updates" "model-backups" "cleanup")
MAX_AGE=90000  # 25 hours in seconds

for job in "${JOBS[@]}"; do
    logfile="/var/log/${job}.log"
    if [ ! -f "$logfile" ]; then
        echo "MISSING: $logfile"
        continue
    fi
    age=$(( $(date +%s) - $(stat -c %Y "$logfile") ))
    if [ $age -gt $MAX_AGE ]; then
        echo "STALE: ${job} last ran $(( age / 3600 )) hours ago"
    else
        echo "OK: ${job} last ran $(( age / 60 )) minutes ago"
    fi
done

# Example full crontab for an AI GPU server:
# 0 3 * * 0 /opt/scripts/update-models.sh
# 0 4 * * * /opt/scripts/backup-models.sh
# 0 5 * * * /opt/scripts/cleanup-gpu-server.sh >> /var/log/cleanup.log 2>&1
# */5 * * * * /opt/scripts/cron-health-check.sh > /var/log/cron-health.log

Automated cron jobs keep your GPU server healthy without manual intervention. For vLLM production serving, see the production guide. Monitor GPU health with our monitoring guide. Check the infrastructure section for related sysadmin topics, tutorials for setup walkthroughs, and benchmarks for performance data.