Gen AI

RLHF

After a model learns to follow instructions, humans compare pairs of its answers, and that preference signal nudges it toward replies people actually like.

A supervised fine-tuned model becomes the policy that a reward model scores using human preference comparisons, and PPO nudges the policy toward higher-scoring outputs while a KL penalty tethers it back to the original SFT model so it can't drift into reward hacking

Why Does This Exist?

Supervised fine-tuning teaches a model to match a demonstration's shape: the right structure, a reasonable tone, an answer that looks like the examples it saw. That's real progress, but it's not the whole job. Take a chat assistant that's already gone through SFT. Ask it something ordinary, and it might answer correctly but three paragraphs longer than it needs to be, or in a tone that's oddly formal for the question. None of that trips the training loss — the model did exactly what SFT asked. It just isn't what a person reading it actually wants.

Nobody can write a demonstration for every question an assistant will get, and "matches this example" and "is the reply a person prefers" aren't the same target anyway. What's missing is a preference signal, and preferences are far easier to state by comparing than to justify from scratch. RLHF (reinforcement learning from human feedback) takes the SFT model as its starting point and pushes it toward responses people actually prefer, using that comparison signal instead of more hand-written examples.

Think of It Like This

Training a dog with treats, on a leash

You can't explain "good behavior" to a dog in words, so you reward it instead: a treat when it does something closer to what you want, nothing when it doesn't. Over many repetitions its behavior shifts toward whatever earns treats — the reward signal doing its job.

But a dog optimizing purely for treats finds shortcuts you didn't intend, like learning to sit next to the treat bag instead of obeying a command. The leash keeps the exercise honest: room to move and explore, not so much room that it wanders off and does something bizarre just because it once got rewarded for it. RLHF runs the same shape. The reward model hands out the treats, and a KL penalty is the leash — room to improve, not enough room to go feral.

How It Actually Works

Collecting preferences and training a reward model

The first stage doesn't touch the assistant's weights — it builds a separate scoring model. Annotators see a prompt and two candidate responses from the SFT model, and pick which one they'd rather receive. Maybe one answer buries a useful fix under three sentences of throat-clearing, and the other gets to the point in one line with the same information; the annotator picks the second. No score, no rubric, just a preference between two options.

That comparison is far more reliable than rating a response on an absolute scale from one to ten — different annotators anchor a "7" differently, and the same annotator's "7" drifts across sessions. A reward model, a separate network trained on thousands of these comparisons, learns to predict which output a person would prefer, and once trained, assigns a scalar reward to any single output on its own, no comparison required.

Optimizing the policy against the reward model with RL

With a reward model in hand, the original SFT model gets a new job title: it's now the policy. It generates a response, the reward model scores it, and reinforcement learning — almost always PPO (proximal policy optimization) — nudges its parameters so it produces higher-scoring responses. Generate, score, adjust, repeat, thousands of times, weights shifting further each round toward outputs the reward model likes.

For the chat assistant, that means gradually favoring the crisper, better-toned answers the reward model prefers, across prompts it never saw a demonstration for — RL generalizes a preference instead of memorizing fixed examples.

The KL penalty and reward hacking

Left alone, this optimization has an obvious failure mode. The reward model is a stand-in for human preference, not the preference itself, and any stand-in can be gamed. A policy chasing reward score with no constraint might discover that stuffing an answer with reassuring phrases nudges the score up without the answer actually improving. That's reward hacking: optimizing a proxy so hard it stops tracking the thing it stood in for.

The fix is a KL-divergence penalty subtracted from the reward at every step, measuring how far the policy has drifted from the original SFT model's output distribution for that prompt. A small drift costs little; a large one cancels out whatever reward it bought. That's the leash from the analogy above — it doesn't stop the policy from improving, it stops it from wandering into degenerate territory that only looks good to the reward model.

Show Me the Code

1def rlhf_objective(reward: float, kl_divergence: float, kl_coefficient: float = 0.1) -> float:2    """Score to maximize: reward model output minus a scaled KL penalty.3
4    kl_divergence tracks how far the policy has drifted from the original5    SFT output distribution for this prompt, so a bigger drift costs more,6    and an output that only looks great because it gamed the reward model7    gets discounted back down.8    """9    return reward - kl_coefficient * kl_divergence10
11
12cases = [(8.0, 0.5), (9.5, 40.0), (7.0, 1.0)]13for reward, kl in cases:14    objective = rlhf_objective(reward, kl)15    print(f"reward {reward}, kl {kl} -> objective {objective:.2f}")16    # -> reward 8.0, kl 0.5 -> objective 7.9517    # -> reward 9.5, kl 40.0 -> objective 5.5018    # -> reward 7.0, kl 1.0 -> objective 6.90

Notice the middle case: a raw reward of 9.5, higher than either other option, still nets the lowest objective once its drift is priced in. That's the tradeoff the KL term exists to enforce.

Watch Out For

Weakening the KL penalty to chase a higher reward score

Turning down the KL coefficient lets the policy move further per step, and the reward-model score climbs faster — which looks like progress on a dashboard. Look at the actual outputs, though, and you'll often find repetitive phrasing, an oddly upbeat tone regardless of the question, or filler that scores well. That's reward hacking in progress, and it won't show up until someone reads real transcripts instead of the reward number.

Treating the reward model's score as ground truth forever

The reward model is a learned proxy trained on a finite batch of comparisons, not an oracle. It can be wrong on inputs unlike anything annotators saw, and exploited by outputs sitting in its blind spots. Teams that only track reward-model score over time, without checking real transcripts against human judgment, can watch the metric rise for weeks while genuine quality quietly falls.

The Quick Version

RLHF runs in three stages: start from an SFT model, train a reward model on human preference comparisons, then use RL — typically PPO — to optimize the policy against that reward model.
Comparing pairs of outputs gives a steadier signal than scoring one output on an absolute scale, so the reward model learns from preference pairs, not ratings.
A KL-divergence penalty keeps the optimized policy close to the original SFT model's output distribution, which is what keeps the process from gaming the reward model.
The reward model is a proxy for human preference, not the preference itself, and can be wrong or exploitable.
The payoff: a model that gives the kind of answer people actually want, not just one shaped like an SFT demonstration.