Gen AI

Judge Bias & Calibration

LLMs used as automated judges suffer from systematic blind spots—such as preferring the first option, the longer option, or their own writing style—which invalidate their scores if left unmitigated.

An unmitigated LLM judge often suffers from position bias, selecting whichever model is presented first regardless of quality.

Why Does This Exist?

When teams first discover llm-as-a-judge, they treat it like a magic oracle. They feed it two outputs, ask "Which is better?", and record the answer. But if you run an experiment where you feed the judge Output A and Output B, and the judge selects A, you should immediately try swapping the order. If you feed the exact same judge Output B first and Output A second, it will frequently change its mind and select B.

This happens because an LLM is not a deterministic calculator of objective truth; it is a next-token prediction engine susceptible to the structure and style of its input. Judge bias is the umbrella term for the systematic errors LLMs make when evaluating text. If you do not actively control for these biases, your evaluation pipeline will not tell you which model is actually better—it will only tell you which model is better at exploiting the judge's blind spots.

Think of It Like This

Think of an LLM judge like a tired human grading hundreds of essays.

A tired grader might implicitly give higher marks to essays that are longer, assuming that more words means more effort (verbosity bias). They might grade the first essay in a stack more favorably because they haven't gotten exhausted yet (position bias). And if a student writes an essay that perfectly mirrors the grader's own political opinions or writing style, the grader will naturally prefer it over a contrarian essay (self-preference bias).

Just as a school must audit tired graders to ensure fairness, ML engineers must audit and control LLM judges.

How It Actually Works

To build a reliable automated evaluation pipeline, you must understand the three dominant failure modes of LLM judges and implement their specific algorithmic controls.

1. Position Bias

When presented with a pairwise comparison ("Which is better, Output 1 or Output 2?"), many models exhibit a strong preference for Output 1 simply because it appears first in the context window.

The Control: The industry standard control is position swapping. You never evaluate a pair once. You evaluate it twice: once as (A, B) and once as (B, A).

If the judge picks A both times, A wins.
If the judge picks B both times, B wins.
If the judge picks the first option both times (A then B) or the second option both times (B then A), you discard the result as a tie, because the judge is exhibiting position bias rather than evaluating quality.

2. Verbosity Bias

LLMs are highly susceptible to verbosity. If Model A answers a question correctly in one sentence, and Model B answers the same question correctly but adds three paragraphs of polite, repetitive fluff, an unmitigated LLM judge will almost always declare Model B the winner. It conflates length with comprehensiveness.

The Control: You must explicitly penalize verbosity in the prompt rubric. For example, adding: "Prioritize concise, direct answers. Penalize outputs that include unnecessary filler, repetitive formatting, or overly polite conversational filler." Even with strict prompting, verbosity bias is notoriously difficult to eradicate, which is why tracking the average length of a model's outputs alongside its win-rate is crucial.

3. Self-Preference Bias

Models prefer their own writing. If you use GPT-4 as your judge, it will systematically assign higher scores to outputs generated by GPT-4 than to identically correct outputs generated by Claude 3 or Llama 3. The judge recognizes its own structural tics, vocabulary preferences, and formatting habits, and implicitly scores them higher.

The Control: You cannot use a single model to judge a multi-model arena if you want absolute fairness. The control is to use a panel of judges. Instead of relying solely on GPT-4, you average the preferences of GPT-4, Claude 3.5 Sonnet, and Gemini 1.5 Pro. By aggregating diverse judges, the self-preference biases cancel each other out.

Show Me the Code

Here is how you implement a robust pairwise evaluation that automatically controls for position bias via swapping.

1from typing import Callable2
3def evaluate_pair_with_swap(4    judge_fn: Callable[[str, str, str], str], 5    question: str, 6    output_a: str, 7    output_b: str8) -> str:9    """10    Evaluates two outputs while controlling for position bias.11    judge_fn takes (question, opt1, opt2) and returns '1' or '2'.12    """13    # 1. Forward Pass14    forward_winner = judge_fn(question, output_a, output_b)15    16    # 2. Swapped Pass17    backward_winner = judge_fn(question, output_b, output_a)18    19    # 3. Resolve the True Winner20    if forward_winner == '1' and backward_winner == '2':21        return "Model A Wins"22    elif forward_winner == '2' and backward_winner == '1':23        return "Model B Wins"24    else:25        # The judge picked the first position both times, 26        # or the second position both times. It is biased.27        return "Tie (Inconsistent Judge)"28
29# If the judge always picks the first option:30# forward_winner = '1'31# backward_winner = '1'32# -> Returns "Tie (Inconsistent Judge)"

Watch Out For

The Sycophancy Trap

LLM judges are highly sycophantic. If your prompt includes an assertive statement like "Output A is written by an expert," the judge will almost always pick Output A, even if it is garbage. Never leak the identity of the models, the baseline scores, or your own expectations into the judge's prompt. The evaluation must be perfectly blind.

Tone over Substance

LLMs are easily distracted by formatting. If Model A provides a mathematically correct answer in plain text, and Model B provides a mathematically incorrect answer formatted in a beautiful Markdown table with bold headers, a weak judge will often pick Model B. You must use state-of-the-art reasoning models (like GPT-4-class or above) for judging; smaller models are too easily tricked by superficial aesthetics.

The Quick Version

llm-as-a-judge is powerful, but it is not objective. Models have systematic biases that inflate specific types of outputs.
Position bias causes judges to blindly prefer whichever output is presented first in the context window. Control this by evaluating every pair twice in swapped orders.
Verbosity bias causes judges to prefer longer, fluffier answers over concise, accurate ones. Control this with aggressive rubric penalties for filler.
Self-preference bias causes models to inflate the scores of their own generated text. Control this by using a diverse panel of different foundational models as judges.
If you do not implement these controls, your evaluation pipeline is measuring bias exploitation, not model capability.