Skip to content
AI360XpertBeta

Reward Modeling

You cannot ask a person to grade every reply your model writes while it trains. A reward model learns what your annotators preferred from side by side comparisons, then grades each new reply alone with one fast number.

A reward model turns one prompt and two replies into two scores, and training pushes the chosen reply above the rejected one
A reward model turns one prompt and two replies into two scores, and training pushes the chosen reply above the rejected one

Why Does This Exist?

Picture yourself running reinforcement learning on a chatbot. The loop writes a reply, asks for a grade, adjusts, and repeats. For chess that grade is easy: won or lost. For an explanation of photosynthesis, no script can tell you whether the wording is clear, honest, and useful. And you cannot station a person next to the GPU to grade every draft the loop produces. That gap is the whole problem.

You have already taught the model the shape of a good answer with supervised fine-tuning: it imitates demonstrations. But two replies can both match the shape while only one is the answer a person would actually pick. Imitation cannot tell you which. You need a preference signal, something that says "this one, not that one" for any pair.

A reward model is that signal in a box. It is a language model rewired to output one number per reply, trained on pairs humans compared, and fast enough to grade everything the policy writes. This page builds the grader: how it scores, how pairs teach it, and one worked example with real numbers. The loop that climbs those scores lives in RLHF. What happens when the loop overdoes it lives in reward hacking. And when the answer is checkable by a test or a proof, skip the learned taste and use rules instead, which is RLVR.

Think of It Like This

A pie contest and its apprentice judge

Your town runs a pie contest. Each round has a theme, say apple pie. Two bakers present their pies side by side, and the head judge tastes both and points: this one. The apprentice standing behind her tastes too, takes notes, and watches dozens of these calls. Flaky crust wins over soggy. Clear apple flavor wins over extra sugar. After enough rounds, the apprentice can score new pies alone, without the head judge in the room. The bakers then tweak their recipes to please the apprentice, because her scores decide the next round.

The mapping is exact. The theme is your prompt. The two pies are the two replies. The head judge pointing is your annotator picking a winner. The apprentice is the reward model, and her solo scores are the scalars. The bakers adjusting recipes is the policy learning.

Where it stops: the apprentice copies quirks along with wisdom. If the head judge happened to favor tall pies, the apprentice learns tall means good, and soon every pie arrives piled a foot high. Pies are tasted directly, while text is judged for truth and tone the apprentice cannot bite into. And the apprentice never learned to bake, only to grade.

How It Actually Works

From text to one number

Start from your fine tuned model and perform one surgery. Remove the next token head, the layer that predicts words, and attach a small linear head that outputs a single scalar. Call the result rϕ(x,y)r_\phi(x, y), where xx is the prompt in plain words, yy is one reply, ϕ\phi means the model weights you train, and rr is the score. Higher means people would prefer it. That is the full interface, and the reward model entry states it in one paragraph.

One warning belongs here early because it bites everyone once. A score of 2.1 means nothing by itself. It only means "preferred to a 0.4 on the same prompt under the same weights." Never compare raw scores across prompts, and never threshold them like pass marks. Gaps decide. Levels do not.

The pairwise rule that teaches it

Humans are shaky at rating one reply 7 out of 10 and excellent at pointing at the better of two. So training data comes in triples (x,yw,yl)(x, y_w, y_l), where xx is the prompt, ywy_w is the winning reply the annotator chose, and yly_l is the losing reply they rejected. Thousands of triples, each one a small verdict.

The standard rule turning those verdicts into gradients is the Bradley Terry model. It says the chance the human picks the winner equals the logistic function σ\sigma applied to the score gap:

P(yw beats yl∣x)=σ(rw−rl),σ(z)=11+e−zP(y_w \text{ beats } y_l \mid x) = \sigma(r_w - r_l), \quad \sigma(z) = \frac{1}{1 + e^{-z}}

In plain words: rwr_w is short for rϕ(x,yw)r_\phi(x, y_w), the winner score. rlr_l is rϕ(x,yl)r_\phi(x, y_l), the loser score. zz is any real number fed to the logistic. ee is the exponential constant. σ\sigma squashes any gap into a probability between 0 and 1. A gap of zero gives one half, a coin flip. A large positive gap approaches one.

Training minimizes the negative log of that probability over your dataset:

L(ϕ)=−log⁡σ(rw−rl)L(\phi) = -\log \sigma(r_w - r_l)

LL is the loss, log⁡\log is the natural log. When the model already ranks the pair correctly with room to spare, σ\sigma sits near one and the loss sits near zero. When it ranks them backward, σ\sigma sits near zero and the loss grows large. Every step pushes rwr_w up relative to rlr_l. Nothing here ever scores a lone reply in isolation. Comparisons do all the teaching.

Worked example with real numbers

Say one training triple leaves your model scoring the chosen reply at rw=1.2r_w = 1.2 and the rejected reply at rl=0.4r_l = 0.4. The picture above draws exactly these numbers.

First the gap, which the diagram labels between the two bars:

δ=1.2−0.4=0.8\delta = 1.2 - 0.4 = 0.8

δ\delta is the Greek letter delta, here just the gap. Next the predicted preference:

σ(0.8)=11+e−0.8=11+0.449=11.449≈0.69\sigma(0.8) = \frac{1}{1 + e^{-0.8}} = \frac{1}{1 + 0.449} = \frac{1}{1.449} \approx 0.69

So the model assigns about a 69 percent chance that a person would pick the chosen reply. Finally the loss on this triple:

L=−log⁡(0.69)≈0.371L = -\log(0.69) \approx 0.371

A modest loss for a correct but unconfident ranking. Had the scores been flipped, with the rejected reply at 1.2, the gap would be −0.8-0.8, the probability about 0.31, and the loss about 1.171, nearly three times larger and pulling hard the other way.

The trade is now visible. You buy unlimited fast grades at the cost of grading through a proxy. The proxy inherits every bias in the triples, length, politeness, formatting, and it has no notion of truth beyond what annotators rewarded. So do not use a learned reward where a checkable rule exists. Math with an answer, code with tests, a game with a scoreboard: those want RLVR, not taste. Use a reward model where quality is genuinely a matter of human preference: tone, clarity, helpfulness, harmlessness. And when you need to choose whether one final score or many step scores fits, read process and outcome rewards before you commit.

Code

The point of this snippet is the check you will rerun whenever the math feels slippery: gap in, probability and loss out, on the same 1.2 and 0.4 from the example.

import math

def prob_preferred(r_w: float, r_l: float) -> float:    gap: float = r_w - r_l    return 1.0 / (1.0 + math.exp(-gap))

def preference_loss(r_w: float, r_l: float) -> float:    return -math.log(prob_preferred(r_w, r_l))

r_w, r_l = 1.2, 0.4print(round(prob_preferred(r_w, r_l), 3))  # -> 0.69print(round(preference_loss(r_w, r_l), 3))  # -> 0.371

Both prints match the hand computation: about 0.69 probability, about 0.371 loss. Flip the two inputs and watch the loss jump past 1.1. That asymmetry is the entire training signal.

Watch Out For

Your grader falls in love with long answers

Annotators often pick the longer reply when both look okay, so the triples quietly teach length means quality. The symptom is unmistakable: average reply length climbs week after week while head to head win rates stall, and a scatter of score against length slopes upward. You are watching verbosity get rewarded, not helpfulness.

Fix it at the data and at the eval. Rebalance pairs so the winner is sometimes the shorter reply, add an explicit length penalty or length controlled comparison to your evals, and plot score against length before every policy update. If that slope is steep, stop trusting the scores until you flatten it.

Reading one score like a school grade

A raw 2.1 feels like a B plus, so teams threshold it, average it across prompts, or compare it across model versions. The symptom: dashboards of mean reward drift for no reason, or a "quality gate" at some cutoff blocks good replies on hard prompts and passes weak ones on easy prompts. Scores were never calibrated across prompts, so the average is noise.

Fix it by using only comparisons. Report win rate of new replies against fixed references on the same prompts, judged by the same frozen reward model or by fresh human picks. Keep the grader frozen while you measure, and never ship a cutoff on a raw scalar.

The Quick Version

  • A reward model reads a prompt plus one reply and returns one scalar score saying how much annotators would prefer it.
  • It learns from triples of prompt, chosen reply, and rejected reply, never from lone scores.
  • Training follows the Bradley Terry rule: turn the score gap into a preference probability with a logistic, then minimize negative log probability.
  • Scores mean nothing alone. Only gaps between replies to the same prompt, under the same weights, carry information.
  • It buys fast unlimited grading at the price of a proxy: every annotator bias becomes model behavior, which is how reward hacking starts.
  • Where answers are checkable by tests or proofs, use verifiable rewards instead of learned taste.