Safety Alignment with RLHF

Raw predictors repeat likely words, including harmful ones. Alignment teaches the model to refuse harmful asks while staying helpful on benign ones.

RLHF lifts preferred answers from 35 to 78 of 100 prompts, turning a raw predictor into a helper that refuses harmful asks.

Why Does This Exist?

A base model completes a harmful request because those words look likely online. It does not hate you. It has no notion of approval. Likelihood is its only rule, so it will draft the phishing mail and the cake recipe with the same confidence.

That breaks the moment the model meets real users. You need a helper that says no to the weapon ask and yes to the homework ask, without losing its knowledge.

This page covers the post-training loop that teaches that split: human ranks, a reward model, and a policy nudge. What the model should ultimately serve lives in value alignment. Who watches it after launch lives in human oversight. How teams probe it before release lives in red teaming.

You need one idea first. Next-token prediction picks the most likely continuation. Alignment adds a second objective: prefer the answer a careful human would pick.

Think of It Like This

A talented line cook with a new expediter

Your kitchen hires a fast cook who plates whatever ticket looks familiar. Speed is great until they fire an allergen dish because the ticket shorthand looked common. The skill is there. The judgment is not.

You add an expediter. They taste, they rank plates, they send back the risky ones with a note. Over weeks the cook learns which shortcuts earn a pass and which earn a rejection. They stay fast, but the allergen plates stop leaving the pass.

Map it across: the cook is the base model, tickets are prompts, the expediter taste is human ranking, and the house rules are helpful, honest and harmless. Where it stops: a cook smells smoke while a model sees only tokens, so you must keep probing with fresh rewordings after training.

How It Actually Works

Alignment runs after pretraining in three steps. Humans rank pairs, a judge learns those ranks, and the policy moves toward the winner while staying near its old self.

Let $x$ be one prompt. Let $y_{w}$ be the winning answer and $y_{l}$ the losing one. Let $r$ be the reward model score. Let $\pi$ be the policy that writes answers. Training lifts $\pi(y_{w} \mid x)$ and lowers $\pi(y_{l} \mid x)$ .

Rank, judge, nudge

Rank pairs. Show two answers to the same prompt. A person picks the more helpful and harmless one. Thousands of such picks become the signal.
Train the judge. Fit $r$ so $r(x, y_{w}) > r(x, y_{l})$ . In our fixture the judge scores the chosen answer $2.1$ and the rejected one $0.4$ .
Nudge the policy. Move $\pi$ toward high $r$ with a small step that keeps it near the old weights. Big jumps that chase the judge too hard invite reward hacking.

Worked example with 100 prompts

Judge the same 100 prompts before and after. Base answers win 35 times. Aligned answers win 78 times. The gain is $78 - 35 = 43$ extra wins on the same questions with the same judges.

The diagram shows this exact run: a base box at 35 of 100, a reward box at 2.1 against 0.4, and an aligned box at 78 of 100. The final arrow is the nudge. The limit sits below: a reworded harmful ask can still slip past, which is why probing never stops.

Code

Score one pair the way the judge does. Standard library only.

1def score(chosen: bool) -> float:2    return 2.1 if chosen else 0.43
4
5print(f"{score(True):.1f}")  # -> 2.16print(f"{score(False):.1f}")  # -> 0.47print(78 - 35)  # -> 43

The prints match the fixture: 2.1 beats 0.4, and 78 minus 35 leaves a gain of 43 preferred answers.

Watch Out For

Paying the alignment tax on benign asks

Symptom: after a safety push the model refuses a fictional argument scene or a public-health summary. Helpfulness drops while refusal charts look clean.

Fix: track benign win rate beside refusal rate on the same 100 prompts. If preferred answers stall near 35 while refusals climb, loosen the rule and add benign pairs back into training.

Teaching flattery instead of help

Symptom: answers grow long and apologetic because raters rewarded polish. The judge scores a padded reply 2.1 and a short correct one 0.4. The policy learns length, not care.

Fix: blind the length cue during ranking, cap verbosity in the rubric, and rerun the pair with length held equal before you trust the margin.

The Quick Version

Base models predict likely words, so they complete harmful asks without meaning to.
Humans rank pairs and a reward model learns that pattern as scores.
Our fixture scores the winner 2.1 against 0.4 for the loser.
Preferred answers rise from 35 to 78 of the same 100 prompts, a gain of 43.
The policy nudge stays small so the model stays helpful while it gets safer.
Reworded attacks still slip past, so probing continues after launch.