Bias Testing for AI Models

Your recruitment platform uses a self-hosted LLM to score CVs. After three months in production, analysis reveals that candidates with traditionally female names receive scores 12% lower on average for engineering roles. The model was never explicitly trained to discriminate — it absorbed statistical patterns from historical hiring data that reflected existing bias. Under the Equality Act 2010, using a system that produces discriminatory outcomes is unlawful indirect discrimination, regardless of intent. Bias testing before deployment would have caught this. This guide covers bias testing for AI models on dedicated GPU servers.

Protected Characteristics and AI Risk

The Equality Act 2010 defines nine protected characteristics. AI systems can discriminate across any of them:

The most common bias vectors in LLMs are name-based (the model treats “James” differently from “Aisha”) and language-pattern-based (formal English scores higher than equally valid but culturally different phrasing).

Building Evaluation Datasets

Create paired test inputs that differ only in the protected characteristic being tested. For a CV screening model, construct identical CVs with only the name changed across demographic groups. For a customer service model, submit identical queries with different cultural phrasing. For a credit scoring model, vary only demographic indicators while keeping financial data identical. Generate these datasets systematically using open-source models to create variations, then manually review for quality.

import json

def create_bias_test_pairs(base_input, variations):
    """Create paired test inputs for bias evaluation."""
    pairs = []
    for group_name, substitutions in variations.items():
        modified = base_input
        for key, value in substitutions.items():
            modified = modified.replace(f"{{{key}}}", value)
        pairs.append({
            "group": group_name,
            "input": modified,
            "expected_output_range": "equivalent"
        })
    return pairs

# Example: CV screening bias test
name_variations = {
    "group_a": {"name": "James Smith", "pronoun": "he"},
    "group_b": {"name": "Aisha Patel", "pronoun": "she"},
    "group_c": {"name": "Wei Chen", "pronoun": "they"},
    "group_d": {"name": "Oluwaseun Adeyemi", "pronoun": "he"},
}

Statistical Analysis Methods

Run paired test inputs through your model on the GPU server and analyse the outputs statistically. Measure disparate impact ratio (the selection rate for the disadvantaged group divided by the rate for the advantaged group — below 0.8 indicates adverse impact under the four-fifths rule), mean score differences across groups, statistical significance of any observed differences (use appropriate tests with correction for multiple comparisons), and consistency of results across multiple runs (LLMs are stochastic — run each test at least 30 times).

For vLLM deployments, set temperature to 0 during bias testing to reduce randomness, then retest at production temperature to measure variance.

Bias Mitigation Strategies

When testing reveals bias, several mitigation approaches apply to self-hosted models. Prompt engineering adds explicit fairness instructions to system prompts. Input anonymisation strips demographic indicators before model processing. Output calibration applies statistical corrections to equalise outcomes. Model selection involves switching to a model with better fairness properties. Human oversight adds mandatory human review for high-stakes decisions. On private infrastructure, you have full control to implement any combination of these without API provider restrictions.

Continuous Bias Monitoring

Bias testing is not a one-time activity. Production data distributions shift, and bias can emerge in previously fair systems. Run automated bias tests weekly against your evaluation dataset. Monitor production outcomes disaggregated by demographic group where that data is available and lawfully collected. Alert when disparate impact metrics deteriorate beyond thresholds. Document all testing results for regulatory audit. Deployments serving customer interactions, processing identity documents, or analysing images of people require the most rigorous ongoing testing. Review GDPR compliance and governance documentation for regulatory context. See industry examples for sector-specific bias considerations.