Self-hosted AI cost is harder to track than per-token API billing because it's a fixed monthly + variable utilisation. Without explicit monitoring, cost-per-token drifts invisibly. With the right metrics dashboard, cost stays under control.
Track six metrics: tokens served per month, £/M tokens (computed), GPU utilisation, cache hit rate (prefix + semantic), hosted-API fallback rate + cost, per-tenant attribution. Dashboard in Grafana with the same Prometheus / log-aggregation stack you already have for ops monitoring.
Metrics
- Tokens served / month: from structured logs aggregated to Postgres / ClickHouse
- £/M tokens computed: (server cost + ops cost + fallback cost) / tokens served
- GPU utilisation: from DCGM Exporter; sustained > 70% is healthy, < 30% suggests over-provisioning
- Prefix cache hit rate: from vLLM
/metrics - Semantic cache hit rate: from your application-layer cache
- Hosted-API fallback rate: % of queries routed to Claude / GPT-4o; cost per fallback query
- Per-tenant tokens: from logs; basis for attribution / billing
Dashboard
Single Grafana dashboard with panels:
- Tokens served (last 24h, 7d, 30d trend)
- £/M tokens (last 30d trend)
- GPU utilisation heatmap
- Cache hit rate (prefix + semantic, separate panels)
- Fallback rate + estimated fallback cost
- Per-tenant token consumption (top 10)
- Cost-per-tenant trend
Alerts
- £/M tokens drifted up > 20% over 7 days — investigate efficiency regression
- GPU utilisation sustained < 30% — over-provisioned, consider downgrade
- Cache hit rate dropped > 10% — investigate prompt or query distribution change
- Fallback rate > 15% — investigate self-hosted quality regression
Verdict
Self-hosted AI cost monitoring is the difference between "saved 80% on API" (year 1) and "saving has eroded to 40%" (year 2 from invisible drift). Build the dashboard early; alert on drift; review monthly with finance.
Bottom line
Track £/M tokens and drift. See ROI measurement.
