RLHF

You start from a model that already follows instructions, have people pick the better of two answers, and use those picks to steer it toward replies people prefer.

Human preference pairs train a reward model that scores policy outputs, and PPO pushes the policy toward higher scores while a KL penalty holds it back near the starting model

Why Does This Exist?

You have finished instruction tuning, and your model answers instead of continuing. Good. But now read its answers closely. One is correct and three paragraphs too long. Another is correct in a tone nobody would use with a friend. The tuning loss never complained, because every answer matched the shape of its demonstration. Shape is not taste, and no stack of demonstrations covers every prompt your users will try.

RLHF (reinforcement learning from human feedback) closes that gap with a cheaper question. Instead of writing ideal answers, people compare: for one prompt, answer A or answer B. Comparing is fast, and two annotators who would score the same answer differently on a ten point scale will usually pick the same winner. Those picks become the training signal.

This page covers the full three stage loop: comparisons, then a scorer, then the policy update with its leash. The scorer on its own lives in reward modeling. The one stage shortcut that deletes the scorer is DPO. Rule checked rewards for math and code belong to RLVR, and the verdict across all three options sits in RLHF vs DPO vs GRPO.

Think of It Like This

Training a dog with treats, on a leash

You cannot explain good behavior to a dog in words, so you reward it. A treat for sitting when asked, nothing for jumping on guests. After enough repetitions the dog drifts toward whatever earns treats. The treats are the whole curriculum.

But a dog that only chases treats finds shortcuts. It learns to sit beside the treat bag, or to bark and then sit, because the sequence once paid off. So you train on a leash. The leash never teaches anything by itself. It just bounds how far exploration can wander from acceptable behavior while the treats do the teaching.

RLHF runs the same shape. The reward model hands out the treats, and a KL penalty is the leash. Where the analogy stops: a dog wants the treat, while the policy is just numbers shifting uphill. Nobody is pleased or disappointed. The leash is arithmetic, a subtraction from the score, and it works whether or not any human is watching that step.

How It Actually Works

The loop starts from your instruction tuned model and never asks annotators for scores, only for picks.

Stage 1: comparisons become a reward model

An annotator sees one prompt $x$ and two answers sampled from the starting model, then marks a winner. Call the winner $y_w$ (the chosen response) and the loser $y_l$ (the rejected response). Thousands of these triples $(x, y_w, y_l)$ form the preference dataset. Nobody writes a rubric. The only fact recorded is which answer a person preferred.

A second network, the reward model $r_\theta$ , learns from those triples. The symbol $r_\theta$ means a function with learned weights $\theta$ that reads a prompt plus one answer and returns a single number, the predicted human liking. It trains under one standard assumption about pairwise choice, the Bradley-Terry model: the chance the annotator picks $y_w$ over $y_l$ depends only on the gap between their scores, passed through the sigmoid function $\sigma$ (the S shaped squasher $\sigma(z) = 1 / (1 + e^{-z})$ that turns any gap into a probability):

P(y_w \succ y_l \mid x) = \sigma(r_\theta(x, y_w) - r_\theta(x, y_l))

In plain words: the bigger the score gap in favor of the chosen answer, the more likely the model says a human would pick it. Training minimizes the negative log of that probability, so a confident correct call costs almost nothing and a wrong call costs a lot. Once trained, the reward model scores any single answer alone. No more pairs needed.

Stage 2: PPO chases the reward under a KL leash

Now the starting model gets a new job title, the policy $\pi_\theta$ : a function with weights $\theta$ that picks answers. A frozen copy of where it started is kept aside as the reference $\pi_{\text{ref}}$ . The policy writes an answer, the reward model scores it, and PPO (proximal policy optimization, the cautious update rule that caps how far one step may move) shifts the weights toward higher scoring answers. Small steps, thousands of rounds.

The score being maximized is not the raw reward. Every step subtracts a drift charge, the KL penalty $\beta \cdot \text{KL}(\pi_\theta \parallel \pi_{\text{ref}})$ . Here KL (Kullback Leibler divergence) measures how far the policy output distribution has wandered from the frozen starting distribution on the same prompt, and $\beta$ (a small weight such as $0.1$ ) sets the leash length:

J = r - \beta \cdot \text{KL}

So $J$ is the number PPO actually climbs: reward $r$ minus drift times $\beta$ . A crisper answer that stays near the starting style keeps nearly all its reward. An answer that games the scorer by drifting far pays for every step of drift. That subtraction is the whole defense against reward hacking, the failure where the policy optimizes the proxy until the proxy stops tracking human taste.

Worked example: one pair, end to end

Prompt: why is this loop slow. Answer A gets straight to the cause in one line. Answer B buries the same fix under three sentences of throat clearing. The annotator picks A, so $y_w = A$ and $y_l = B$ .

Suppose the young reward model scores them $r_w = 2.0$ and $r_l = 0.5$ . The gap is $2.0 - 0.5 = 1.5$ . The predicted chance a human picks A is $\sigma(1.5) = 1 / (1 + e^{-1.5})$ . Since $e^{-1.5} \approx 0.223$ , that is $1 / 1.223 \approx 0.82$ . The training loss on this pair is $-\ln(0.82) \approx 0.20$ , small, because the model already ranks the pair correctly and fairly confidently.

Later, the trained reward model scores three candidate policies with $\beta = 0.1$ . Candidate one earns $r = 8.0$ at $\text{KL} = 0.5$ , so $J = 8.0 - 0.1 \times 0.5 = 7.95$ . Candidate two looks stronger at $r = 9.5$ but drifted to $\text{KL} = 40.0$ , so $J = 9.5 - 0.1 \times 40.0 = 5.50$ . Candidate three earns $r = 7.0$ at $\text{KL} = 1.0$ , so $J = 6.90$ . The hacker with the highest raw reward finishes last. That ordering is the leash doing its job, and the diagram above draws exactly these numbers.

The trade is now visible. RLHF buys generalization: preferences learned on rated prompts steer unrated ones, which demonstrations cannot do. It costs a second trained model, a rollout loop, and constant vigilance, because the reward model is a stand in for taste, not taste itself. When answers have checkable truth, as in math or code, skip the stand in and use verifiable rewards in RLVR. When a written constitution can replace the human harmlessness vote, read Constitutional AI. And when the PPO machinery itself is the burden, DPO trains on the same pairs with one loss and no scorer at all.

Code

The point of this snippet is the check, not the formula. Run both stages on the worked numbers and confirm the hacker loses.

1import math2
3
4def preference_prob(r_chosen: float, r_rejected: float) -> float:5    """Bradley-Terry chance the annotator picks the chosen answer."""6    return 1.0 / (1.0 + math.exp(-(r_chosen - r_rejected)))7
8
9def rlhf_objective(reward: float, kl: float, beta: float = 0.1) -> float:10    """The number PPO climbs: reward minus the drift charge."""11    return reward - beta * kl12
13
14p = preference_prob(2.0, 0.5)15print(f"{p:.2f}")  # -> 0.8216print(f"{-math.log(p):.2f}")  # -> 0.2017print(f"{rlhf_objective(8.0, 0.5):.2f}")  # -> 7.9518print(f"{rlhf_objective(9.5, 40.0):.2f}")  # -> 5.5019print(f"{rlhf_objective(7.0, 1.0):.2f}")  # -> 6.90

The first two lines replay stage one: a $0.82$ predicted preference and a $0.20$ loss. The last three replay stage two, and the $9.5$ raw reward still nets the worst objective once its drift is priced in.

Watch Out For

Loosening the KL leash to chase a higher reward number

The dashboard tempts you. Lower $\beta$ and the reward curve climbs faster, which reads as progress. Then someone reads actual transcripts and finds the rot: stock phrases, relentless cheer, filler that flatters the scorer. That is reward hacking in progress, and the reward number will never report it because the number is what got hacked.

Symptom: reward rising while blind human ratings of fresh outputs stall or fall. Fix: freeze a held out set of prompts, have people (or a trusted LLM judge you calibrated against people) rerank outputs every so often, and stop or tighten $\beta$ the moment real preference diverges from model score. Never tune $\beta$ against the reward model itself.

Trusting the reward model far from its training pairs

The scorer learned taste on the prompts annotators actually saw. On a new domain, a new language, or a tricky prompt unlike anything rated, its scores are guesses wearing the uniform of measurements. PPO then optimizes those guesses with full confidence, and quality falls precisely where you have no labels to notice.

Symptom: confident high scores on inputs your annotators never rated, followed by user complaints the metrics missed. Fix: track score distributions on fresh traffic against the training mix, keep collecting pairs where they diverge, and retrain the scorer before the next policy round. A reward model is a perishable proxy, so expiry is the default assumption.

The Quick Version

RLHF turns human picks between answer pairs into training signal: comparisons, then a reward model, then a PPO policy update.
A pair beats a rating because two annotators who anchor scales differently still pick the same winner.
The reward model learns $P = \sigma(r_w - r_l)$ , so our $2.0$ versus $0.5$ pair predicts the human pick at $0.82$ with loss $0.20$ .
PPO climbs $J = r - \beta \cdot \text{KL}$ , which is why the $9.5$ reward with drift $40.0$ loses to the honest $8.0$ .
The KL penalty is a leash, not a teacher: it bounds drift from the starting model while the reward does the teaching.
The scorer is a proxy with an expiry date, so real human checks on fresh prompts decide when to stop.