Table of Contents
Model checkpoint versioning is part of the broader AI versioning discipline. Specifically: weights (base + fine-tunes), LoRA adapters, tokenizer configs, structured-output schemas. Each needs reproducible identification and audit trail.
Version everything by SHA: base model commit, LoRA adapter checkpoint, tokenizer revision, schema version. Store in HF Hub (with access controls) + content-addressed S3 backup. Metadata: training data SHA, training config, eval scores, deploy date. Reference checkpoints by SHA, not tag, in production. Audit trail across full lifecycle.
What to version
- Base model weights: Llama 3.1 8B SHA, etc. From HF Hub commit
- Fine-tune adapters: per LoRA training run; tagged with input training data SHA
- Tokenizer + chat template: usually moves with base model
- Output schemas: Pydantic models / JSON schemas tied to specific deployment
- Prompt templates: separate versioning (covered elsewhere)
Storage
- HF Hub: primary; access controls, version history, easy fetch
- Content-addressed S3 mirror: backup + air-gapped deployments
- Internal artefact registry: for proprietary fine-tunes; never to public Hub
- DVC for experimental checkpoints: pre-promotion to formal storage
Metadata
Per-checkpoint metadata file (JSON or YAML):
- Base model SHA + version
- Training data SHA + version (link to DVC / dataset commit)
- Training config (PEFT settings, LoRA rank, learning rate, batch size, etc.)
- Eval scores at promotion
- Deploy date and target environment
- Author + reviewer
- Approval references for regulated environments
Verdict
Checkpoint versioning is non-negotiable for production AI — especially in regulated industries. Reference by SHA, not tag. Maintain metadata for reproducibility + audit. Use HF Hub + S3 mirror. The discipline pays off the first time a regulator asks "what model produced this output on date X?" — you can answer with confidence.
Bottom line
Version by SHA; metadata everywhere. See dataset versioning.
