Table of Contents
LLM-as-judge has documented biases: position, length, self-preference, sycophancy. Mitigations are straightforward but require discipline. Apply consistently for stable eval over time.
Mitigations: (1) position randomisation when comparing two outputs, (2) length normalisation in rubric, (3) cross-family judging (Claude judges Llama, GPT judges Mistral), (4) structured rubric not open prose, (5) multi-judge consensus when accuracy critical. Apply 2-3 mitigations consistently; track agreement with human review on sample.
Biases
- Position bias: first response favoured in pairwise comparison
- Length bias: longer outputs favoured
- Self-preference: judge favours outputs from same model family
- Sycophancy: agrees with first claim presented
- Format bias: structured outputs favoured over prose
Mitigations
- Position randomisation: 50/50 randomise A/B order; eval both orderings; consensus or coin-flip on disagreement
- Length normalisation: explicit rubric "ignore length; rate quality"; cap output length to similar ranges
- Cross-family judging: judge with Claude when production is Llama; GPT-4o when production is Mistral
- Structured rubric: 5-point scale per dimension (accuracy, completeness, tone, etc.) not open prose
- Multi-judge consensus: 2-3 different judges; majority verdict; flags for disagreement
- Periodic human calibration: 10% sample human-reviewed; tracks judge accuracy over time
Verdict
For LLM-as-judge to produce reliable eval, mitigations must be applied consistently. Position randomisation is the single highest-ROI mitigation. Cross-family judging eliminates self-preference. Periodic human calibration on a sample tracks judge accuracy — if judge agreement with humans drops, time to investigate.
Bottom line
Position randomisation + cross-family + human calibration. See LLM-as-judge.
