Model Versioning on GPU Servers

Switching models in production without a versioning system leads to untracked changes, broken rollbacks, and confusion about which checkpoint is actually serving traffic. This guide implements a model versioning workflow on dedicated GPU servers that tracks every model you deploy, enables instant rollback, and supports A/B testing between versions.

Versioning Strategy

AI models need versioning at two levels: the base model identity (which architecture and weights) and the deployment configuration (quantisation, context length, serving parameters). Both must be tracked together because changing either affects output quality.

Local Model Registry

Build a file-based model registry on your GPU server. Each version gets a directory containing the model files and a metadata manifest.

# Directory structure
/opt/models/
  registry.json
  llama-3.1-8b-instruct/
    v1/
      model/           # Model weights
      manifest.json    # Version metadata
    v2/
      model/
      manifest.json
  deepseek-coder-v2/
    v1/
      model/
      manifest.json

# scripts/model_registry.py
import json
import hashlib
import shutil
from pathlib import Path
from datetime import datetime

REGISTRY_PATH = Path("/opt/models")
REGISTRY_FILE = REGISTRY_PATH / "registry.json"

def load_registry():
    if REGISTRY_FILE.exists():
        return json.loads(REGISTRY_FILE.read_text())
    return {"models": {}, "active": {}}

def save_registry(registry):
    REGISTRY_FILE.write_text(json.dumps(registry, indent=2))

def register_model(name: str, version: str, source_path: str, metadata: dict):
    registry = load_registry()
    version_dir = REGISTRY_PATH / name / version / "model"
    version_dir.mkdir(parents=True, exist_ok=True)

    # Copy model files
    shutil.copytree(source_path, str(version_dir), dirs_exist_ok=True)

    # Calculate checksum of key files
    checksums = {}
    for f in version_dir.rglob("*.safetensors"):
        checksums[f.name] = hashlib.sha256(f.read_bytes()).hexdigest()[:16]

    manifest = {
        "name": name,
        "version": version,
        "registered_at": datetime.utcnow().isoformat(),
        "source": source_path,
        "checksums": checksums,
        **metadata
    }

    manifest_path = REGISTRY_PATH / name / version / "manifest.json"
    manifest_path.write_text(json.dumps(manifest, indent=2))

    if name not in registry["models"]:
        registry["models"][name] = []
    registry["models"][name].append(version)
    save_registry(registry)

    return manifest

def activate_version(name: str, version: str):
    registry = load_registry()
    version_path = REGISTRY_PATH / name / version / "model"

    if not version_path.exists():
        raise ValueError(f"Version {version} not found for {name}")

    registry["active"][name] = {
        "version": version,
        "activated_at": datetime.utcnow().isoformat(),
        "path": str(version_path)
    }
    save_registry(registry)
    return registry["active"][name]

def get_active(name: str):
    registry = load_registry()
    return registry["active"].get(name)

Zero-Downtime Version Switching

Switching between model versions on a running vLLM server requires restarting the inference process. Wrap the switch in a script that validates the new version before cutting over traffic.

#!/bin/bash
# scripts/switch_version.sh
set -e

MODEL_NAME="$1"
TARGET_VERSION="$2"
REGISTRY="/opt/models"

MODEL_PATH="$REGISTRY/$MODEL_NAME/$TARGET_VERSION/model"

if [ ! -d "$MODEL_PATH" ]; then
    echo "Error: Version $TARGET_VERSION not found"
    exit 1
fi

echo "Validating $MODEL_NAME version $TARGET_VERSION..."
python -c "
from vllm import LLM
llm = LLM(model='$MODEL_PATH', gpu_memory_utilization=0.5, max_model_len=2048)
out = llm.generate(['Test prompt for validation'])
assert len(out[0].outputs[0].text) > 0
print('Validation passed')
"

echo "Stopping current server..."
systemctl stop vllm-inference

echo "Updating symlink..."
ln -sfn "$MODEL_PATH" "$REGISTRY/$MODEL_NAME/current"

echo "Starting server with new version..."
systemctl start vllm-inference

# Wait for server ready
for i in $(seq 1 30); do
    if curl -s http://localhost:8000/health > /dev/null 2>&1; then
        echo "Server ready with $MODEL_NAME v$TARGET_VERSION"
        python scripts/model_registry.py activate "$MODEL_NAME" "$TARGET_VERSION"
        exit 0
    fi
    sleep 2
done

echo "Server failed to start, rolling back"
systemctl stop vllm-inference
ln -sfn "$REGISTRY/$MODEL_NAME/$PREVIOUS_VERSION/model" "$REGISTRY/$MODEL_NAME/current"
systemctl start vllm-inference
exit 1

Metadata and Lineage Tracking

Every model version should record its lineage so you can trace production issues back to training decisions.

# manifest.json example
{
  "name": "llama-3.1-8b-customer-support",
  "version": "v3",
  "registered_at": "2025-03-15T14:30:00Z",
  "base_model": "meta-llama/Llama-3.1-8B-Instruct",
  "fine_tune": {
    "method": "LoRA",
    "dataset": "customer-support-v2.jsonl",
    "epochs": 3,
    "learning_rate": 2e-5,
    "adapter_path": "/training/outputs/run-20250315/"
  },
  "quantisation": "none",
  "serving_config": {
    "max_model_len": 8192,
    "gpu_memory_utilization": 0.9,
    "dtype": "bfloat16"
  },
  "benchmarks": {
    "tokens_per_second": 142,
    "latency_p50_ms": 85,
    "latency_p99_ms": 340,
    "validation_score": 0.94
  },
  "checksums": {
    "model-00001-of-00004.safetensors": "a1b2c3d4e5f67890",
    "model-00002-of-00004.safetensors": "b2c3d4e5f6789012"
  }
}

The benchmarks section records inference performance at registration time. Compare against production metrics in Prometheus and Grafana to detect performance drift over time.

A/B Testing Between Versions

Run two model versions simultaneously and route a percentage of traffic to each. This requires two inference processes and a load balancer.

# docker-compose.ab-test.yml
version: "3.8"
services:
  model-v2:
    image: vllm/vllm-openai:latest
    command: ["--model", "/models/llama-3.1-8b/v2/model", "--port", "8000"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]

  model-v3:
    image: vllm/vllm-openai:latest
    command: ["--model", "/models/llama-3.1-8b/v3/model", "--port", "8000"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["1"]
              capabilities: [gpu]

  nginx:
    image: nginx:latest
    volumes:
      - ./nginx-ab.conf:/etc/nginx/conf.d/default.conf
    ports:
      - "8080:80"

This approach requires a multi-GPU server. For single-GPU setups, use time-based rotation or deploy the blue-green deployment pattern instead. Log which version served each request through the ELK stack for analysis.

Automating the Workflow

Integrate versioning with your CI/CD pipeline so every model that passes validation is automatically registered and optionally deployed. Use webhooks to notify your team when a new version is activated.

For Ollama-based deployments, model versions map to Modelfile tags. The self-hosting guide covers initial model deployment, and our tutorials section has more production infrastructure patterns for PyTorch workflows.