LLM A/B Testing in Production

You Cannot Tell Which Model Configuration Is Better

You are considering switching from Llama 3.1 8B to Mistral 7B, or from temperature 0.5 to 0.7, or from your old system prompt to a new one. Offline evaluation says the new configuration scores higher, but production traffic behaves differently from benchmarks. A/B testing on your GPU server routes a fraction of live traffic to the candidate model and measures real-world performance before committing to a full switch.

Traffic Splitting Architecture

Route requests to different model backends based on a deterministic split:

import hashlib, random
from fastapi import FastAPI, Request
import httpx

app = FastAPI()

# Define experiment configuration
EXPERIMENTS = {
    "model_comparison": {
        "control": {
            "backend": "http://localhost:8000",  # vLLM instance A
            "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
            "weight": 0.8  # 80% of traffic
        },
        "variant": {
            "backend": "http://localhost:8001",  # vLLM instance B
            "model": "mistralai/Mistral-7B-Instruct-v0.3",
            "weight": 0.2  # 20% of traffic
        }
    }
}

def get_variant(user_id, experiment="model_comparison"):
    # Deterministic assignment: same user always gets same variant
    hash_val = int(hashlib.md5(f"{user_id}:{experiment}".encode()).hexdigest(), 16)
    threshold = EXPERIMENTS[experiment]["control"]["weight"]
    if (hash_val % 1000) / 1000 < threshold:
        return "control", EXPERIMENTS[experiment]["control"]
    return "variant", EXPERIMENTS[experiment]["variant"]

@app.post("/v1/chat/completions")
async def chat(request: Request):
    body = await request.json()
    user_id = request.headers.get("X-User-Id", "anonymous")

    variant_name, config = get_variant(user_id)

    async with httpx.AsyncClient() as client:
        body["model"] = config["model"]
        response = await client.post(
            f"{config['backend']}/v1/chat/completions", json=body)

    result = response.json()
    # Tag response with experiment metadata
    result["_experiment"] = {"variant": variant_name, "user": user_id}
    return result

Collecting Comparison Metrics

Track the metrics that matter for your use case across both variants:

import time, json
from prometheus_client import Histogram, Counter

# Latency metrics per variant
ttft = Histogram('llm_ttft_seconds', 'Time to first token',
                 ['variant'], buckets=[0.05, 0.1, 0.2, 0.5, 1, 2, 5])
total_latency = Histogram('llm_total_latency', 'Total response time',
                          ['variant'])
tokens_generated = Histogram('llm_tokens_generated', 'Tokens per response',
                             ['variant'])

# Quality signals
user_thumbs_up = Counter('llm_thumbs_up', 'Positive feedback', ['variant'])
user_thumbs_down = Counter('llm_thumbs_down', 'Negative feedback', ['variant'])
regeneration_requests = Counter('llm_regenerations', 'User asked to regenerate',
                                ['variant'])

# Log detailed results for offline analysis
def log_experiment(variant, user_id, prompt, response, latency, tokens):
    record = {
        "timestamp": time.time(),
        "variant": variant,
        "user_id": user_id,
        "prompt_tokens": len(prompt.split()),
        "response_tokens": tokens,
        "latency_ms": latency * 1000,
        "response_preview": response[:200]
    }
    with open("/var/log/llm_ab_test.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")

Determining a Winner

Do not eyeball results. Apply statistical tests to confirm differences are real:

import numpy as np
from scipy import stats

def analyze_experiment(control_latencies, variant_latencies,
                       control_thumbs_up, control_total,
                       variant_thumbs_up, variant_total):
    # Latency comparison (Mann-Whitney U test — non-parametric)
    stat, p_latency = stats.mannwhitneyu(control_latencies, variant_latencies)
    print(f"Latency: control={np.median(control_latencies):.0f}ms, "
          f"variant={np.median(variant_latencies):.0f}ms, p={p_latency:.4f}")

    # Quality comparison (proportion test)
    control_rate = control_thumbs_up / max(control_total, 1)
    variant_rate = variant_thumbs_up / max(variant_total, 1)
    z_stat, p_quality = stats.proportions_ztest(
        [control_thumbs_up, variant_thumbs_up],
        [control_total, variant_total])
    print(f"Quality: control={control_rate:.2%}, "
          f"variant={variant_rate:.2%}, p={p_quality:.4f}")

    # Decision
    if p_latency < 0.05 or p_quality < 0.05:
        print("Statistically significant difference detected")
    else:
        print("No significant difference — need more data")

# Minimum sample size for reliable results:
# ~1000 requests per variant for latency comparisons
# ~500 feedback signals per variant for quality comparisons

Running Multiple Models on One GPU

A/B tests require running both models simultaneously. Fit them on a single GPU or split across cards:

# Option A: Two vLLM instances with GPU memory splitting
# Instance 1 (control): use 60% of GPU memory
CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.55 --port 8000

# Instance 2 (variant): use 40% of GPU memory
CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --gpu-memory-utilization 0.40 --port 8001

# Option B: Use different GPUs (no memory contention)
CUDA_VISIBLE_DEVICES=0 python -m vllm... --port 8000
CUDA_VISIBLE_DEVICES=1 python -m vllm... --port 8001

Safe Rollback Strategy

Always be able to instantly revert to the control configuration:

# Feature flag for instant rollback
EXPERIMENT_ENABLED = True  # Set to False to route all traffic to control

def get_variant(user_id, experiment="model_comparison"):
    if not EXPERIMENT_ENABLED:
        return "control", EXPERIMENTS[experiment]["control"]
    # ... normal routing logic

# Automated rollback trigger
ERROR_THRESHOLD = 0.05  # 5% error rate
async def health_monitor():
    while True:
        variant_errors = get_error_rate("variant")
        if variant_errors > ERROR_THRESHOLD:
            EXPERIMENT_ENABLED = False
            alert("A/B test auto-rolled back due to high error rate")
        await asyncio.sleep(60)

A/B testing gives you confidence when changing models or configurations on your GPU server. For vLLM multi-instance setups, the production guide covers process management. See the LLM hosting section for architecture patterns, benchmarks for model comparison data, and tutorials for deployment guides.