Gen AI

Pairwise Preference & Elo

Instead of grading models on an absolute scale, we pit them against each other in blind head-to-head matches and use chess rating math to build a global leaderboard.

Pairwise preference converts head-to-head match results into a global Elo leaderboard representing relative, not absolute, capability.

Why Does This Exist?

If you want to know which large language model is the best in the world, you cannot rely on absolute scores like "9.4 out of 10." Absolute scores are unstable; a judge's definition of a "10" drifts over time as models become more capable. Furthermore, open-ended tasks like coding, creative writing, or summarization do not have a single correct answer, making absolute grading subjective and noisy.

Instead of asking a judge "How good is this model?", it is mathematically much more robust to ask, "Which of these two models is better?" This is pairwise preference.

However, if you have 50 different foundational models, forcing every model to play every other model requires $O(N^2)$ matches, which is prohibitively expensive. You need a system that can take sparse, randomized head-to-head match data and synthesize it into a single, mathematically sound global ranking. That system is the Elo rating system.

Think of It Like This

Think of model evaluation like a global tennis tournament.

Tennis players do not receive an absolute "skill score" based on a rubric of how well they hit the ball. Instead, they play matches against each other. If an unknown rookie beats the world champion, the rookie's ranking skyrockets, and the champion's ranking drops significantly, because the result was statistically surprising. If the champion beats the rookie, the rankings barely move, because the result was expected.

The Elo system treats AI models exactly like tennis players. It calculates their relative capability based on their win-rate against other models of known capability.

How It Actually Works

The core of this evaluation paradigm is the Elo rating system (originally designed by Arpad Elo for chess) and its mathematical cousin, the Bradley-Terry model.

1. The Head-to-Head Match

A prompt is sent to two different models (e.g., Model A and Model B) concurrently. The models generate their answers. The identities of the models are hidden. A judge—either a human on a crowdsourced platform like LMSYS Chatbot Arena, or an automated llm-as-a-judge—reads both answers side-by-side and declares a winner (or a tie).

Crucially, to prevent position bias, the judge does not know which model generated the left output and which generated the right output.

2. The Elo Update Math

Every model starts with a baseline rating, typically 1000 or 1200. When a match concludes, the system updates the ratings of both models.

The size of the rating change depends on the expected outcome of the match.

If a 1300-rated model plays a 1000-rated model, the math expects the 1300 model to win easily.
If the 1300 model wins, it gains only a tiny amount of points (e.g., +2), and the loser loses -2.
But if the 1000-rated model pulls an upset and wins, it proves it is much stronger than its current rating implies. It might gain +25 points, and the 1300 model loses -25 points.

Over thousands of randomized matches, the ratings converge. A model's final Elo score is a mathematically pure representation of its relative win-rate against the rest of the ecosystem.

3. What Elo Actually Means

An Elo rating is not a percentage. A model with an Elo of 1200 is not "20% better" than a model with an Elo of 1000.

Elo represents the probability of winning. In the standard chess Elo formula, a rating difference of 400 points means the higher-rated model has roughly a 91% chance of beating the lower-rated model in any given prompt. A difference of 100 points translates to roughly a 64% chance of winning. If you know the Elo difference between two models, you can instantly compute the probability that one will generate a better response than the other.

Show Me the Code

Here is a simplified Python implementation of the classic Elo rating update formula following a pairwise match.

1def expected_score(rating_a: float, rating_b: float) -> float:2    """Calculates the probability that Model A wins against Model B."""3    return 1 / (1 + 10 ** ((rating_b - rating_a) / 400))4
5def update_elo(6    rating_a: float, 7    rating_b: float, 8    actual_score_a: float, 9    k_factor: int = 3210) -> tuple[float, float]:11    """12    Updates ratings. actual_score_a is 1 for a win, 0 for a loss, 0.5 for a tie.13    """14    # 1. Calculate expected probability of A winning15    expected_a = expected_score(rating_a, rating_b)16    17    # 2. Score for B is the inverse18    actual_score_b = 1.0 - actual_score_a19    expected_b = 1.0 - expected_a20    21    # 3. Apply the update rule22    new_rating_a = rating_a + k_factor * (actual_score_a - expected_a)23    new_rating_b = rating_b + k_factor * (actual_score_b - expected_b)24    25    return round(new_rating_a), round(new_rating_b)26
27# Match: 1200 vs 1200. A wins.28# -> update_elo(1200, 1200, 1) -> (1216, 1184)29
30# Upset: 1000 vs 1300. A (1000) wins.31# -> update_elo(1000, 1300, 1) -> (1027, 1273)

Watch Out For

Prompt Distribution Bias

An Elo leaderboard is only valid for the distribution of prompts it was tested on. If an arena primarily feeds models Python coding questions, a model optimized for coding will achieve a massive Elo rating. If you then deploy that model to write creative fiction, it will fail miserably despite its "number one ranking." An Elo rating is not a measure of general intelligence; it is a measure of win-rate on a specific prompt distribution.

The Intransitivity Trap

Elo math assumes transitivity: if A beats B, and B beats C, then A should beat C. In language models, this is often false. Model A might be great at coding (beating B), Model B might be great at reasoning (beating C), but Model C might be great at creative writing (beating A). If your models have highly specialized strengths, a single global Elo number will obscure these nuances.

The Quick Version

Absolute scoring (e.g., 1 to 5) is highly subjective and unstable when evaluating open-ended generative AI.
Pairwise preference solves this by forcing a judge to look at two outputs side-by-side and pick a winner.
The Elo rating system aggregates thousands of these head-to-head matches into a single global leaderboard.
The system heavily rewards upsets and barely rewards expected wins, ensuring ratings converge to the model's true relative capability.
An Elo rating translates directly into a probability of winning a match against another model; it does not measure absolute intelligence.