RL from Human Feedback

Learn a scorer from human picks between two behaviors, then optimize the policy against that scorer while a leash holds it near where it started.

Human picks train a scorer that ranks the chosen reply 1.2 above the rejected 0.4, and PPO climbs that score under a KL leash

Why Does This Exist?

Pretraining teaches what is likely. It does not teach what is wanted. A model that predicts the next word well can still be verbose, sycophantic, or confidently wrong, because all of those patterns appear in its training text. No likelihood loss penalizes them directly.

Human feedback fixes the objective. Instead of writing ideal answers, people compare two outputs and pick the better one. Comparing is fast and consistent where absolute scoring is slow and noisy. Those picks train a scorer, and the policy climbs the scorer. This page covers the general loop for any policy: pairs, then scorer, then KL-constrained update. The LLM specific three stage version lives in RLHF, the scorer alone lives in reward modeling, and the shortcut that deletes the scorer lives in DPO.

The one idea you need first is a learned reward. Write a prompt $x$ and two replies $y_w$ (winner) and $y_l$ (loser). A reward model $r_\theta(x, y)$ outputs one scalar per reply, with $\theta$ its weights. Training pushes $r_\theta(x, y_w)$ above $r_\theta(x, y_l)$ . The policy $\pi$ then learns to produce replies that score high. The true task reward from reward function is replaced here by learned human taste, which is why the leash below matters so much.

Think of It Like This

A student who finally gets graded

A student has read a thousand essays and writes fluently but has never been graded. Fluency is not taste. So the teacher stops assigning readings and starts holding votes. For each prompt the student writes two essays, the teacher points at the better one, and after dozens of votes the student infers the rubric: clear thesis, evidence, no padding.

That maps part for part. The two essays are $(y_w, y_l)$ . The inferred rubric is the reward model $r_\theta$ . Writing to maximize the rubric is the RL step with policy gradients. The starting style the student must not lose is the reference policy the KL penalty protects.

Where it stops: a student can ask what the rubric means, but a policy cannot. It only sees scores, so it will exploit any loophole in the inferred rubric, such as padding that the teacher once rewarded. That is why the loop needs a leash and why long optimization against a fixed scorer always ends in hacking.

How It Actually Works

Three stages run in order, each with its own input and output.

Stage 1: collect comparisons

Sample a prompt $x$ , generate two replies from the starting policy, and ask a person to pick. The dataset $D$ holds triples $(x, y_w, y_l)$ . No scores, only picks. Quality beats quantity here because the scorer will amplify whatever bias the picks contain.

Stage 2: train the scorer

Fit $r_\theta$ with the Bradley Terry loss, which turns a score gap into a pick probability with the sigmoid $\sigma$ :

$\mathcal{L}(\theta) = -\mathbb{E}_{(x, y_w, y_l) \sim D} \left[ \log \sigma \left( r_\theta(x, y_w) - r_\theta(x, y_l) \right) \right]$

Here $\sigma(z) = 1 / (1 + e^{-z})$ maps a gap to a probability. Worked numbers: scorer gives $r = 1.2$ to the chosen reply and $r = 0.4$ to the rejected one. Gap $0.8$ gives $\sigma(0.8) = 0.6900$ , so the model assigns 69 percent chance to the human pick. Loss is $-\log(0.69) = 0.3711$ . Training pushes the gap wider until the chosen reply reliably outscores the rejected one. The diagram draws exactly these two bars at 1.2 and 0.4 with the 0.69 label.

Stage 3: optimize under a leash

Freeze the scorer and tune the policy $\pi_\phi^{\text{RL}}$ starting from $\pi^{\text{SFT}}$ to maximize scored replies minus drift:

$\text{objective} = r_\theta(x, y) - \beta \text{KL}(\pi_\phi^{\text{RL}} \parallel \pi^{\text{SFT}})$

Here $\beta$ (beta) sets leash strength and KL measures how far the new policy drifted from the start. Worked numbers: scored reply $r = 1.2$ , measured KL $0.05$ , $\beta = 0.1$ . Adjusted objective $1.2 - 0.1 \times 0.05 = 1.195$ . Without the leash the policy drifts until the scorer praises gibberish. With it the policy stays near fluent language while shifting toward preferred replies. When drift means danger rather than taste, that is safe RL.

Code

Same 1.2 versus 0.4 comparison and same KL math, so the run matches the hand trace:

1import math2
3
4def pick_prob(chosen: float, rejected: float) -> float:5    """Bradley-Terry probability that the chosen reply wins."""6    return 1.0 / (1.0 + math.exp(-(chosen - rejected)))7
8
9def preference_loss(chosen: float, rejected: float) -> float:10    """Negative log likelihood of the human pick."""11    return -math.log(pick_prob(chosen, rejected))12
13
14def kl_adjusted_reward(score: float, kl: float, beta: float = 0.1) -> float:15    """Policy objective for one reply."""16    return score - beta * kl17
18
19if __name__ == "__main__":20    print(round(pick_prob(1.2, 0.4), 4))21    # -> 0.6922    print(round(preference_loss(1.2, 0.4), 4))23    # -> 0.371124    print(round(kl_adjusted_reward(1.2, kl=0.05, beta=0.1), 4))25    # -> 1.195

Outputs are 0.69, 0.3711, and 1.195, the three numbers from the mechanism section.

Watch Out For

Optimizing until verbosity wins

The symptom is longer answers every checkpoint while human raters grow less happy. The scorer once correlated length with quality, and the policy found the correlation. Scores climb, usefulness falls. This is Goodhart in one line: a fixed proxy plus a strong optimizer equals exploitation.

Fix it with the leash plus early stopping on human judged samples, not on scorer value. Track response length beside score, stop when length grows and judged quality stalls, and refresh the scorer with new comparisons from the current policy. Never select the final checkpoint by scorer value alone.

Collapsing to one safe phrase

The symptom is every answer opening the same way or refusing in the same words. Diversity falls off a cliff late in PPO. The cause is a reliable local optimum: one style always scores decently, so the policy puts all mass there and stops exploring phrasing.

Fix it by watching entropy and distinct n-gram rates during PPO, not just mean score. Lower the step size, raise the KL weight slightly, or mix in a small fraction of the original supervised data. If entropy collapses while scores rise, you are selecting for monotony.

The Quick Version

Pretraining predicts likely text. RLHF optimizes preferred behavior learned from comparisons.
Pairs $(x, y_w, y_l)$ train a scorer $r_\theta$ with a Bradley Terry loss on the score gap.
Scores 1.2 versus 0.4 give pick probability 0.69 and loss 0.3711.
PPO tunes the policy against the frozen scorer minus a KL leash to the starting model.
Score 1.2 with KL 0.05 at beta 0.1 gives objective 1.195.
Long optimization hacks the scorer and collapses diversity, so stop on judged quality with entropy watched.