Model Cards and AI Documentation

An auditor asks your team: “What model powers your customer-facing chatbot, what data was it evaluated on, and what are its known failure modes?” Three engineers give three different answers. One says LLaMA 3.1 8B, another says the 70B variant, and the third is not sure but thinks it was fine-tuned. Nobody can produce evaluation metrics or a list of known limitations. Without model cards — standardised documentation for AI models — your organisation cannot demonstrate responsible AI governance. This guide covers creating model cards for self-hosted models on dedicated GPU infrastructure.

Model Card Structure

A model card documents a specific model deployment. Follow the structure below for each production model:

Example Model Card

A practical model card for a production vLLM deployment:

# Model Card: Customer Support LLM
## Model Details
- **Name**: Meta LLaMA 3.1 70B Instruct
- **Quantisation**: GPTQ 4-bit
- **Served via**: vLLM v0.4.2
- **GPU**: NVIDIA RTX 6000 Pro 96 GB
- **Context length**: 8,192 tokens (production limit)
- **Deployed**: 2025-01-15
- **Owner**: AI Platform Team ([email protected])

## Intended Use
- **Primary**: Customer support query answering via RAG
- **Approved**: Product information, order status, FAQ responses
- **Out of scope**: Medical advice, legal guidance, financial recommendations
- **Prohibited**: Automated decisions without human review

## Evaluation Results
- **MMLU**: 79.3% (general knowledge)
- **Customer query accuracy**: 87% on internal test set (n=2,000)
- **Hallucination rate**: 4.2% on domain-specific queries
- **Bias test**: Passed demographic parity (DI ratio > 0.85 across groups)

## Known Limitations
- Occasionally fabricates product features not in the knowledge base
- Underperforms on queries mixing multiple languages
- Response quality degrades beyond 4,000 input tokens
- May produce overly verbose responses for simple queries

Documenting Upstream Model Information

For open-source models, your model card extends the upstream model’s documentation. The original model publisher provides base evaluation results, training data descriptions, and intended use. Your model card adds your specific deployment context: how you configured the model, what guardrails you added, your own evaluation results on your data, and your operational constraints. Reference the upstream model card rather than duplicating it. If you fine-tuned the model, document the fine-tuning data, process, and how evaluation results changed.

System-Level Documentation

Model cards document individual models. System cards document the complete AI system — the model plus its surrounding infrastructure. For a RAG chatbot, the system card covers: the model (documented in its model card), the retrieval pipeline and knowledge base, the prompt template and guardrails, the API gateway and authentication, monitoring and alerting, and human escalation pathways. Teams running chatbots backed by RAG systems with vector databases should document the complete pipeline, not just the LLM.

Maintaining Model Cards

Model cards are living documents. Update them when you deploy a new model version, run new evaluation tests, discover new failure modes, change the deployment configuration, or modify the intended use scope. Store model cards in version control alongside your deployment configuration. Make them accessible to governance, legal, and operations teams — not buried in an engineering wiki. Review quarterly as part of your AI governance cycle.

For compliance with the UK’s AI regulatory framework, model cards demonstrate accountability and transparency — two of the five core principles. Regulators increasingly expect this documentation. Deployments processing personal data should cross-reference model cards with GDPR impact assessments. See governance guides for framework integration, industry examples for sector-specific documentation requirements, and tutorials for deployment setup.