Judge Bias & Calibration
An LLM asked to grade answers has favorite shortcuts: the first option, the longer answer, and writing that sounds like itself. Left unchecked, those habits decide the winner instead of quality.
Why Does This Exist?
Grading open-ended answers by hand does not scale. Nobody can reread thousands of replies every time a prompt changes, so teams hand the job to a strong model and treat its verdicts as truth. That setup has a name, LLM-as-a-Judge, and it is good enough to be dangerous: reliable enough that you stop checking, biased enough that what you stopped checking quietly picks your roadmap.
Here is the failure that makes this page necessary. Ask the judge once and it picks Answer A. Swap the two answers so A sits second, change nothing else, and ask again. A biased judge flips and picks B. Same content, opposite crown, and the only thing that moved was position. If you never run that second ask, your leaderboard measures which answer flatters the judge instead of which answer serves the user.
You need one prior idea before the rest lands. Judges grade two ways: pointwise scoring rates a single answer against a rubric, while pairwise judging picks a winner between two answers side by side. This page covers the pairwise case and the three habits that corrupt it. The setup itself lives on the LLM-as-a-Judge page, turning piles of pairwise votes into a ranking lives on pairwise preference and Elo, and fixing the numbers a judge reports lives on calibration.
Think of It Like This
A rushed tasting contest with two cakes
You run a cake contest. Two bakers hand you slices and you taste one first, then the other. That first bite sets your scale, so the second cake gets judged against a mouth already full of the first. Whoever you taste first keeps winning the rematches you never hold. That is position bias.
One baker now serves a towering slice while the other serves a small neat one. You are rushed, so you hand the prize to the big slice because it felt like more. You never checked whether the extra height was flavor or frosting. That is verbosity bias.
Then you notice one cake tastes exactly like the one you bake at home. You vote for it before you mean to, because its habits are your habits. That is self-preference.
So you fix the contest the way teams fix judges: you taste in both orders on different days, you hide the bakers' names, and you bring in two more tasters with different tastes. A cake wins only if it wins both tastings.
Where it stops: you get full, the judge does not. Its first-bite effect comes from training and from how strongly early context anchors its next tokens, not from a palate. And its fix is cheap in a way yours is not. Rerunning the same judge twice costs pennies, while restaging your whole contest costs a weekend. Run the cheap rerun every time.
How It Actually Works
A judge reads a prompt containing both answers and emits a preference. Nothing about that pipeline promises neutrality. Order, length, and style all leak into the verdict through the same next-token machinery that makes the model fluent, so each leak needs its own control.
The three habits
Position bias is the pull toward whichever answer appears first. Early tokens anchor everything after them, so the first answer frames the comparison and the second gets read against it. Control it with the swap test in the next section: every pair gets judged twice, once per order, and a split vote counts as a tie.
Verbosity bias is the pull toward the longer answer. Long replies share more surface with the polished, thorough responses the judge was trained to praise, so length reads as completeness even when the extra words add nothing. Control it in the rubric with an explicit brevity rule, and always plot win rate next to average answer length. If the winner is also the longest by far, you have not measured quality yet.
Self-preference bias is the pull toward writing that sounds like the judge itself: its pet phrases, its formatting tics, its sentence rhythm. Control it with blindness and variety. Strip model names, use generic labels like Response 1 and Response 2, and spread verdicts across a panel of different judges instead of letting one model grade a contest it also entered.
The swap test and the consistency score
Call one question judged in both orders a trial. Let be the number of trials, plain count of questions you ran twice. Sort every trial into exactly one bucket: trials where Answer A wins both orders, trials where Answer B wins both orders, trials where the first-shown answer wins both times, and trials where the second-shown answer wins both times. The consistency score is the share of trials with a repeat winner:
In words: consistent trials over total trials. The flip rate is whatever remains, , equal to . High means the judge has a stable preference. Low means position is doing the judging and the win rates are decoration.
This test doubles your judge bill, since each question costs two calls instead of one. That is the trade-off, stated plainly: you buy validity with latency and tokens. And the test rests on one assumption worth naming. It assumes a repeat winner is a true preference. That fails when both orders agree for the wrong reason, because verbosity and self-preference survive swapping untouched. A long, house-styled answer can win both orders and still be the worse answer. Swapping catches position. The rubric and the blind panel catch the rest.
Do not crown a release on judge votes alone. Use the swap test to throw out flips, then spot-check a sample of the consistent wins against human grades before the decision ships. Order effects are one way an eval lies to you; benchmark contamination is another, and it needs its own audit.
Worked example: a 20-pair audit
Picture a small audit you run yourself: questions, each judged twice, so 40 verdicts. Suppose the buckets come back as , , , and . Check the total first: , every trial exactly where it belongs.
Consistent trials are , so:
Flip rate is . Nearly half the verdicts moved with position.
Now watch what single-order reporting would have claimed. Forward order shows A first, so A wins the trials plus the trials: of 20, a 0.65 win rate. Swapped order shows B first, so A wins the trials plus the trials: of 20, a 0.40 win rate. Identical answers, a 25-point swing, and the only thing that changed was seating. Keep only the 11 consistent winners and report the 9 flips next to them. Any leaderboard built from the 13-to-8 margins without that note is fiction.
Code
This snippet is the swap test as code: resolve one pair judged in both orders, then score a whole audit. The inputs name the content winner in each order, and the audit below replays the 20-pair example from the previous section.
def resolve_pair(forward: str, backward: str) -> str: """Decide one pair judged in both orders.
forward and backward name the CONTENT winner ("A" or "B") in each order, not the position. Only a repeat winner counts. """ if forward == backward: return f"{forward} wins" return "tie: the order flipped the vote"
def consistency_rate(votes: list[tuple[str, str]]) -> float: """Share of pairs with the same content winner in both orders.""" agreed = sum(1 for forward, backward in votes if forward == backward) return agreed / len(votes)
audit: list[tuple[str, str]] = ( [("A", "A")] * 6 + [("B", "B")] * 5 + [("A", "B")] * 7 + [("B", "A")] * 2)print(resolve_pair("A", "B")) # -> tie: the order flipped the voteprint(resolve_pair("B", "B")) # -> B winsprint(consistency_rate(audit)) # -> 0.55print(len(audit)) # -> 20The audit list holds 6 repeat-A votes, 5 repeat-B votes, 7 first-position votes, and 2 second-position votes: 20 pairs, 11 consistent, rate 0.55. That matches the hand computation above, which is the point. When your real audit disagrees with your paste-up example, trust the run and fix the example.
Watch Out For
Counting a flipped pair as a win
The symptom is a leaderboard that wobbles between runs. A new prompt wins on Monday, loses the rematch on Tuesday, and nobody changed the models. What changed was the seating chart: flipped pairs got counted as victories, so the ranking tracks order luck instead of quality.
The fix is mechanical. Only consistent pairs award a win, flipped pairs are ties, and the flip rate ships next to the win rate in every report. If a decision is close, break it with a third blind run under new labels or a small human spot check, never with a third reading of the same biased order.
Telling the judge who wrote what
The symptom is a suspicious home-field advantage. Your house model climbs the board, answers tagged as expert-authored keep winning, and formatted replies beat plainer ones that say the same thing. The judge is grading nameplates and wrapping paper.
The fix is blindness. Label answers Response 1 and Response 2, strip model names and pet formatting before judging, and confirm important results with a panel of different judges. If blinding an answer flips its verdict, the verdict was never about the answer.
The Quick Version
- An LLM judge has three favorite shortcuts: the first answer it reads, the longer answer, and writing that sounds like itself.
- Judge every pair twice, once in each order, and count a win only when the same answer wins both times.
- On an illustrative 20-pair audit, 11 consistent winners against 9 flips give a consistency score of 0.55, so nearly half the verdicts moved with position.
- Swapping doubles the judge bill and still misses shared blind spots, so spot-check consistent wins against human grades before you ship a decision.
- If a win rate moves but the flip rate moved more, you measured the judge, not the models.