Preference Data Collection

Asking which of two answers is better is far more reliable than asking anyone to score one. Every aligned model is built on that switch from rating to ranking.

Two raters using different parts of a five-point scale give the same answer a 4 and a 2, yet both pick answer A over answer B, which is why alignment data is collected as comparisons.

Why Does This Exist?

Ask ten people to rate a chatbot answer out of five and you get numbers that do not mean the same thing. One person 4 is another person 2. Some never touch the extremes, some never touch the middle, and every private scale drifts over an afternoon of labelling.

Ask the same ten people which of two answers is better and agreement jumps, because the question no longer depends on where anyone anchors a scale. It is a comparison, and humans are far better at comparisons than at absolute judgements.

That switch sits under every aligned model. You cannot write a loss function for "be helpful", and you cannot reliably collect a number for it either. What you can collect is a pile of comparisons, fit a model that scores answers consistently with them, and optimise against that. The operation is a data-collection problem wearing an alignment hat, which is why it lives here rather than with the training algorithms.

This page covers how comparison data is gathered and turned into scores. Writing the instructions raters follow belongs to labelling and annotation, and using a model as the judge belongs to weak supervision, since an AI judge is a labelling function with opinions.

The one idea you need first: a reward model is a scorer fitted to comparisons. It takes any response and returns one number, trained so that responses humans preferred get higher numbers than the ones they rejected. The Bradley-Terry section below is the maths that makes "preferred" trainable.

Think of It Like This

Judging cheese without a scale

Ask a hundred people to score a cheese out of ten and the answers are mush. Some are generous, some are harsh, some anchor on the last cheese they ate, and one person has decided 7 is the highest any cheese can reach.

Hand them two cheeses and ask which they prefer, and the data sharpens. Nobody needs a shared idea of what 7 means. Each person compares two things in front of them, which is what tasting is.

Then the useful part: enough pairwise verdicts across enough pairs reconstruct a full ranking, and you can put a number on each cheese afterwards, derived from the comparisons rather than asked for directly. Those numbers are comparable, because they came from one consistent fit instead of a hundred private scales.

How It Actually Works

What one record looks like

A prompt, two or more candidate responses, and a verdict: which was preferred. Optionally a strength ("much better" versus "slightly better"), a tie option, and a written rationale, which costs more and is worth it, because rationales are how you debug the guidelines.

Where the candidates come from matters more than people expect. Sample them from the model you intend to improve, at a realistic temperature, because preferences over answers your model would never produce say nothing about how to improve it. Preference data goes stale as the model changes, so collection is continuous, not a one-off dataset.

Turning comparisons into scores

The standard bridge is the Bradley-Terry model: assume each response carries a latent quality score, and the chance one is preferred is a sigmoid of the difference. With $r_A$ and $r_B$ the two latent scores and $\sigma$ the sigmoid:

P(A \succ B) = \sigma(r_A - r_B)

Read it plainly: the bigger the gap, the more lopsided the verdicts. Invert it and a preference rate becomes a score gap: if annotators prefer A seven times in ten, the gap is $\ln(0.7/0.3) = 0.85$ . A near-tie at 55 percent is a gap of 0.20; a clear 95 percent win is 2.94. That inversion is what a reward model learns to predict, and it is why comparisons are enough. You never needed the absolute numbers.

The agreement ceiling, and it is low

Human-human agreement on open-ended preference tasks lands between 60 and 75 percent. That is far below a classification task, and it is a hard ceiling: a reward model cannot be more consistent than the preferences it was fitted to, so measure agreement before drawing conclusions from small differences.

The practical consequence is uncomfortable. If your annotators agree 68 percent of the time, a reward model at 70 percent accuracy sits near the achievable limit, and chasing 80 means fixing the guidelines rather than the model.

The biases that get baked in

Preference data encodes whatever your annotators reward, including things nobody intended.

Length. Annotators prefer longer answers, substantially, well past usefulness. Uncorrected, this is the single largest artefact in preference datasets, and it is why aligned models became verbose.

Confidence and formatting. A confident, well-structured wrong answer beats a hedged correct one. Bullet points and headings score well on their own.

Position. Whichever answer shows first has an edge, so randomise the order and log which position each held.

Sycophancy. Answers that agree with the prompt premise get preferred, which teaches the model to agree.

Each becomes a target the model optimises, and that is reward hacking: the model finds the artefact instead of the intent. Control for length explicitly, ablate formatting, and check whether padding alone fools your reward model before you trust it.

AI feedback

Using a strong model as judge is now standard: orders of magnitude cheaper and more consistent than humans, and a labelling function with all that implies. Correlated errors, its own position and length biases, and a fondness for text that resembles its own output.

The workable pattern is a hybrid: AI judgements for volume, a human-labelled subset to calibrate against, and a measured judge-human agreement rate you monitor rather than assume.

Worked example

Ten annotators compare answer A against answer B. Seven prefer A.

Under Bradley-Terry that 0.7 rate is a reward gap of $\ln(0.7/0.3) = 0.8473$ . Push it back through the sigmoid and you recover 0.7 exactly, which is the round trip a reward model trains to reproduce.

Now the sensitivity. A 55/45 split is a gap of 0.2007: small, and inside sampling noise on ten comparisons. A 95/5 split is 2.9444. So the same scale spans an order of magnitude between "barely distinguishable" and "clearly better", and with ten comparisons per pair you cannot tell 0.55 from 0.60. That is the argument for volume over precision: many pairs judged a few times each beats few pairs judged many times, because the reward model fits across pairs. Note what a 50/50 split gives: $\ln(1) = 0$ , so a tie carries no information about the gap. Collect ties, then handle them deliberately instead of dropping them.

Code

1import numpy as np2
3wins_a: int = 74comparisons: int = 105
6def reward_gap(p_a_wins: float) -> float:7    """Bradley-Terry inverted: a preference rate becomes a score gap."""8    return float(np.log(p_a_wins / (1 - p_a_wins)))9
10def preference_rate(gap: float) -> float:11    """Sigmoid: a score gap becomes a preference rate."""12    return float(1 / (1 + np.exp(-gap)))13
14p: float = wins_a / comparisons15print(p, round(reward_gap(p), 4))                # -> 0.7 0.847316print(round(preference_rate(reward_gap(p)), 4))  # -> 0.7  the round trip17print(round(reward_gap(0.55), 4))               # -> 0.2007  a near-tie18print(round(reward_gap(0.95), 4))               # -> 2.9444  a clear win

Every # -> value above is real output, and the second line proves the round trip: rate to gap and back lands exactly where it started.

Watch Out For

Length going uncontrolled

You collect 50,000 comparisons, fit a reward model, optimise against it, and answers get noticeably longer. Reviewers say quality improved. Users say it waffles.

The mechanism is plain: annotators prefer longer answers, so length correlates with the label, so the reward model learns length as a feature. Optimising against it optimises length, and that is the largest single artefact in preference data.

Two things hide it. Longer answers genuinely are better sometimes, so the correlation is over-weighted rather than spurious. And held-out accuracy stays high, because the held-out pairs carry the same bias. Measure it: correlate length with the label across your dataset, and check whether your reward model prefers the same answer with filler appended. If it does, you own a length model. Match candidate lengths within a pair where you can, or add length as an explicit term you regress out.

Trusting an AI judge you never calibrated

An AI judge produces 500,000 comparisons for the price of 500 human ones, so human collection stops.

Now every bias of that judge is your reward signal. It prefers its own phrasing, so a model optimised against it converges toward the judge style rather than toward quality. Its position and length biases usually run stronger than a human. And its errors are correlated, so unlike human noise they do not average out with volume: a hundred thousand judgements from one model sit closer to one opinion repeated than to a hundred thousand opinions.

Without a human-labelled subset, no measurement exists that would tell you the judge drifted after a version upgrade. Keep a rolling human calibration set, report judge-human agreement as a tracked metric, and randomise candidate order in every judge prompt. If agreement falls below your human-human rate, the judge is the weaker annotator and volume will not fix it.

The Quick Version

Absolute ratings do not survive contact with multiple annotators. Comparisons do, because they need no shared scale.
One record is a prompt, two or more candidates, and a verdict. Sample candidates from the model you are improving, or the data will not apply to it.
Bradley-Terry turns a preference rate into a score gap: 70 percent is 0.85, 55 percent is 0.20, 95 percent is 2.94, and a tie is 0.
Human-human agreement on open-ended preferences runs 60 to 75 percent, and that ceilings any reward model.
Length is the biggest artefact in preference data, and optimising against it is why aligned models got verbose.
Confidence, formatting, position, and sycophancy all get rewarded. Randomise order and ablate formatting.
Reward hacking is the model finding the artefact instead of the intent. Test whether padding fools your reward model.
AI judges are cheap and correlated, so volume never averages out their errors. Keep a human calibration set and track agreement.
Many pairs judged a few times beats few pairs judged many times.