Pairwise Preference and Elo
You stop asking how good one answer is and ask which of two answers wins. Those blind wins feed a chess style rating that ranks every model.
Why Does This Exist?
Ask a judge to score one answer from 1 to 10 and you get drift. What counted as a 9 last year counts as a 7 now that models write better. Open ended work makes it worse. Code, summaries, and stories have no single right answer, so two careful graders give two different numbers and both feel right.
Pairwise preference drops the ruler and asks a simpler question: here are two answers to the same prompt, which one wins. People and models answer that far more steadily than they assign absolute numbers. But with 50 models you cannot run every pairing. That is matches, and each match costs a judge call. You need a way to take sparse random pairings and still get one ranking you can trust. That is what the Elo rating does.
This page covers how those wins become ratings. How you get a single vote in the first place lives in LLM-as-a-Judge: one prompt, two answers, one blind call. What the same votes teach a trainable scorer lives in reward modeling, which feeds RLHF. Here you learn the vote, the win chance formula, and the update, with numbers you can check.
Think of It Like This
A cafe game night with a running ladder
You run a board game night at a busy cafe. Dozens of regulars show up and you cannot have everyone play everyone. So each table plays one game, you write down who beat whom, and you keep one ladder on the wall. Beat the reigning champ and you jump ten spots. Beat a newcomer and you move one. Lose to someone far below you and you drop hard.
Each player is a model. Each table is a prompt. Each result is a blind vote. The ladder is the Elo table, and the jump size is the update. Surprise moves you a lot. Boring results barely move you.
Where it stops: your regulars practice and get better, while the models are frozen during voting, so the ladder measures this voter pool on this mix of games, not fixed smarts. It also assumes strength runs in one line. If you beat Sam at chess, Sam beats Priya at cards, and Priya beats you at dice, no single ladder number tells that story. Models do the same across code, math, and writing. Change the games and you change the champ.
How It Actually Works
One match has two ratings going in and two ratings coming out. The formula in between has two jobs: turn the current gap into a win chance, then move points based on how surprising the result was.
The vote has to be blind
Send the same prompt to model A and model B. Hide the names, shuffle left and right, and show both answers to one judge. The judge returns an actual score for A, written : for a win, for a loss, for a tie. Blinding plus shuffling matters because judges favor the first answer and the longer answer, a habit covered in judge bias. A vote you cannot trace to quality poisons every step after it.
Ties count as half a win for each side. That keeps the books balanced: what one side gains, the other loses.
Expected score turns a gap into a probability
Write for the rating of A in plain words, for the rating of B, and for the expected score of A, meaning the win chance the current ratings imply. The chess scale uses base 10 and a divisor of 400:
Read it as: take the gap, divide by 400, raise 10 to that power, add one, flip it. Equal ratings give . A 100 point lead gives about . A 400 point lead gives about . The expected score for B is the rest, .
Its cousin, the Bradley Terry model, says the same thing with natural exponentials. Write and for two hidden strengths, then the chance A is picked is , which equals a sigmoid of the gap. Elo is that idea on a base 10 scale built for leaderboards. Fitting all votes at once by maximum likelihood gives batch Bradley Terry ratings. Updating after each vote gives online Elo. Same shape, different bookkeeping.
The update charges for surprise
Write for the K factor in plain words, the cap on how many points one match can move. Small is steady and slow. Large reacts fast and jumps around. Many arenas use as a middle ground.
is the new rating. is the surprise: actual minus expected. Win when you were supposed to and the bracket is near zero, so you gain little. Win when you were supposed to lose and the bracket is near one, so you gain close to a full . The B side mirrors it with and , so the pair stays zero sum.
That is the trade off. Pairwise voting cancels drifting standards because each call only compares two answers side by side. It costs you absolutes: a rating of 1250 means nothing alone, only what it beats and by how often. Do not use it when you need a pass mark, a latency number, or a per task profile. One number hides task splits, so keep it for ranking and use sliced evals for shipping calls.
Worked example: one expected win, one upset
Keep for both matches so only surprise changes the move. These are the numbers drawn in the diagram above.
Start even. , , A wins so .
Surprise is . Move is .
The new gap is 32. Feed it back through the same formula and the leader is now about a favorite next time, since and . Narrow leads need many votes before you trust them.
Now the upset. , , A wins so .
Surprise is about . Move is .
Same , same win, but the upset pays 27 points against 16 for the expected win. That gap is the whole method: expected results confirm, upsets rewrite.
Code
This snippet is the check you run before trusting a leaderboard script. It recomputes both matches from the worked example, so you can see the 16 point move and the 27 point move come out of the same two functions.
def expected_score(rating_a: float, rating_b: float) -> float: """Win chance for A from the current gap. Equal ratings give 0.5.""" return 1.0 / (1.0 + 10.0 ** ((rating_b - rating_a) / 400.0))
def update_elo( rating_a: float, rating_b: float, score_a: float, k: float = 32.0) -> tuple[int, int]: """New ratings after one vote. score_a is 1.0, 0.0, or 0.5 for a tie.""" exp_a = expected_score(rating_a, rating_b) exp_b = 1.0 - exp_a score_b = 1.0 - score_a new_a = rating_a + k * (score_a - exp_a) new_b = rating_b + k * (score_b - exp_b) return (round(new_a), round(new_b))
print(expected_score(1200.0, 1200.0)) # -> 0.5print(update_elo(1200.0, 1200.0, 1.0)) # -> (1216, 1184)print(f"{expected_score(1000.0, 1300.0):.4f}") # -> 0.1510print(update_elo(1000.0, 1300.0, 1.0)) # -> (1027, 1273)The last line prints 1027 and 1273, the same pair the worked example derives by hand. If your copy prints anything else, your exponent or your sign is flipped, not the idea.
Watch Out For
The top model flops on your prompts
You pick the arena leader and it writes lovely prose but wrong code for your repo. No bug in the math. The rating only measures the arena prompt mix. A coding heavy mix crowns a coder. A chat heavy mix crowns a charmer.
You will spot it when category slices disagree: first overall, tenth on your task. Fix it by matching the mix to your use before you trust the order. Filter to your task slice, check how many votes that slice holds, and rerun a small blind test on your own prompts. A thin slice is a rumor, not a ranking.
You read the gap as a grade
A 100 point lead looks like 10 percent better. It is not. It means about a 64 percent win chance head to head. A 400 point lead means about 91 percent, not four times smarter. Worse, one number assumes rock beats scissors logic: if A beats B and B beats C, A must beat C. Models break that across skills when a coder beats a reasoner, the reasoner beats a writer, and the writer beats the coder.
You will spot it when the head to head matrix loops or ties pile up. Fix it by translating gaps to win chances and opening the matrix. Quote 1216 versus 1184 as about 55 percent next time, not as plus 32 smarts. When skills split, keep the global board for fun and ship on per task evals.
The Quick Version
- Pairwise judging asks which of two answers wins instead of how good one answer is, so shifting standards cancel out.
- Elo turns sparse blind wins into one ranking with and .
- Equal 1200 ratings plus one win with move to 1216 and 1184, a 16 point confirm.
- A 1000 beating a 1300 with the same moves to about 1027 and 1273, a 27 point rewrite, because surprise sets the pay.
- A 400 point gap means about a 91 percent win chance and a 100 point gap means about 64 percent, never percent better.
- Use Elo to rank on a fixed prompt mix and judge pool, not to certify quality or to pick across different skills.