Skip to content
AI360XpertBeta
Core ML

Visual explainer

RLHF / DPO

Trace ranked answers through reward modeling, PPO and the DPO pair loss, with checked probability examples and visible limits of preference feedback.

A likely answer need not be the answer a human prefers.
A likely answer need not be the answer a human prefers.

RLHF and DPO use preference feedback to change a language model's behavior. PPO-based RLHF learns a reward model and optimizes the policy against it. Direct Preference Optimization trains the policy on ranked pairs with a reference-relative loss. Both depend on the feedback and evaluation they receive; neither guarantees truth or safety.

Our synthetic two-answer example starts with likelihoods A = 0.4 and B = 0.6, while a human prefers A. The larger segment is the more likely answer, not the better answer. Pretraining likelihood and human preference are different objectives. A base model is not compelled to produce a harmful answer to every harmful prompt, and preference training does not make every output safe.

Record a Preference Pair

One prompt and two ranked answers form a preference training example.
One prompt and two ranked answers form a preference training example.

A shared prompt produces candidate answers A and B. An annotator selects A, yielding the record (prompt, preferred A, rejected B). The check denotes that judgement, not an objective proof of quality. Annotation instructions, prompt context and disagreement matter. Instruction-tuning pipelines often begin with supervised fine-tuning on demonstrations before collecting comparisons. The InstructGPT method describes demonstrations, ranked responses, reward modeling and policy optimization.

Fit a Reward Model

A reward model learns a score gap between preferred and rejected answers.
A reward model learns a score gap between preferred and rejected answers.

In this illustrative learned reward model, A scores 2.1 and B scores 0.4. The bars use a shared zero and scale; the bracket is their 1.7-point gap. Under the Bradley–Terry model, the probability of preferring A is sigmoid(2.1 − 0.4) ≈ 0.845535. Train the reward model by minimizing the negative log probability of the observed rankings. Reward scores are not probabilities, and adding a common constant leaves pairwise probabilities unchanged. These scores do not determine a particular PPO update or an evaluation win rate.

Compare Against a Reference

DPO compares policy likelihood ratios against a frozen reference.
DPO compares policy likelihood ratios against a frozen reference.

Now use a separate reference distribution (0.5, 0.5) and an illustrative policy distribution (0.7, 0.3), with A/B in the same order on both bars. DPO's margin compares the policy's relative likelihood changes for the two completions:

Δ=log⁡πθ(A∣x)πref(A∣x)−log⁡πθ(B∣x)πref(B∣x).\Delta = \log\frac{\pi_\theta(A\mid x)}{\pi_{\rm ref}(A\mid x)} - \log\frac{\pi_\theta(B\mid x)}{\pi_{\rm ref}(B\mid x)}.

Here Δ = log(0.7/0.5) − log(0.3/0.5) = log(7/3) ≈ 0.847298. Actual completion likelihoods are products of conditional token probabilities, computed as sums of token log probabilities. The two-answer universe makes the arithmetic readable; it is not a realistic vocabulary of complete responses. The DPO derivation replaces a separately fitted reward model with a policy-based reward parameterization under its stated preference and regularization assumptions.

Balance Reward and Drift

A KL penalty balances reward seeking against drift from the reference.
A KL penalty balances reward seeking against drift from the reference.

PPO-based RLHF commonly optimizes expected reward minus β times KL divergence from a reference policy. To isolate that trade-off, change the reward fixture to r(A) = log(7/3), r(B) = 0, keep the uniform reference, and set β = 1. If p = P(A), the objective is J(p) = p log(7/3) − KL((p,1−p) || (0.5,0.5)). The plot shows J(p) + log 2, an additive vertical shift that does not change the optimum. Seven exact samples are joined by straight segments; this is an objective slice, not a training trajectory.

The optimum is p = 0.7, rather than all probability on A. Its KL divergence from the reference is about 0.082283 nats. This follows from the chosen reward gap and β; it is independent of the earlier 2.1/0.4 reward fixture. A KL penalty moderates drift, but cannot guarantee the reward model remains reliable or the answers remain safe.

Compare the Training Loops

PPO samples from the policy while offline DPO trains on stored pairs.
PPO samples from the policy while offline DPO trains on stored pairs.

The top lane samples answers from the current policy, scores them with the reward model, and feeds an update back into the policy. A typical PPO implementation also estimates values and uses a reference penalty; those are omitted from this small dependency view. The lower lane evaluates an offline pair loss and backpropagates through policy likelihoods, without that on-policy rollout loop or a separate reward model.

DPO still uses a frozen reference or precomputed reference log probabilities. Memory depends on caching, parameter sharing and implementation; there is no universal one-model-versus-four-model allocation. Both methods need hyperparameter tuning and held-out evaluation. Offline DPO lacks on-policy sampling during its training loop, but that does not mean the trained model cannot generalize beyond literal dataset answers.

Audit the Proxy

An imperfect reward proxy can rank the worse answer higher.
An imperfect reward proxy can rank the worse answer higher.

This separate synthetic failure reverses the ordering: the proxy scores rejected B at 9.2 and preferred A at 6.1. Both bars share the same score scale, and the human column checks A and crosses B. If policy optimization exploits a feature the proxy rewards, such as an irrelevant stylistic cue, rising proxy scores can accompany worse answers. The reward overoptimization study investigates this proxy failure. Diverse evaluation and checking optimized outputs are necessary; a KL penalty or length normalization alone is no universal fix. DPO can also inherit biased preference labels without an explicit reward model.

Compute the Pair Loss

A larger DPO margin lowers the loss for the observed preferred answer.
A larger DPO margin lowers the loss for the observed preferred answer.

For the observed label A preferred to B, DPO minimizes L = −log sigmoid(βΔ). At β = 1 and policy equal to reference, Δ = 0, the predicted preference probability is 0.5 and loss is log 2 ≈ 0.693147. For the earlier margin 0.847298, that probability becomes 0.7 and loss falls to −log 0.7 ≈ 0.356675. The bars show predicted pairwise preference, not the probability of sampling answer A. They happen to match numerically in this uniform-reference two-answer fixture. Training on this pair does not establish better behavior on unseen prompts.

Keep Beta Explicit

Beta rescales the DPO preference logit for a fixed policy margin.
Beta rescales the DPO preference logit for a fixed policy margin.

Hold Δ = log(7/3) fixed. Changing β from 0.5 to 1 changes sigmoid(βΔ) from about 0.604356 to 0.7, and the pair loss from about 0.503592 to 0.356675. These bars show how the loss interprets an unchanged policy margin; the policy distribution itself has not changed. In the regularized reward objective, β controls the cost of reference drift. Comparing trained policies at different β values requires optimizing and evaluating them; the fixed-margin picture alone does not predict that outcome.

Where It Breaks

Conflicting labels for the same pair cannot both be satisfied with certainty.
Conflicting labels for the same pair cannot both be satisfied with certainty.

Two equally weighted records contain the same prompt and answers but opposite preferences. A single deterministic scoring function cannot make both labels certain. If s is its pairwise logit, the mean loss is [−log sigmoid(s) − log sigmoid(−s)] / 2, minimized at s = 0: equal predicted preference. The merged bar shows that compromise, not an average of generated text or a claim that answer sampling must be uniform. Missing context, annotation mistakes or real disagreement can cause such conflicts. Define whose preferences matter, preserve relevant context and test factual correctness and safety separately from preference fit.

The Quick Version

  • Likelihood and human preference optimize different objectives.
  • Ranked pairs record contextual judgements.
  • Reward models learn score gaps from comparisons.
  • DPO compares policy likelihood changes against a reference.
  • KL regularization trades reward against reference drift.
  • Offline DPO avoids PPO's on-policy rollout loop.
  • Reward proxies can favor worse answers.
  • DPO pair loss uses an explicit reference-relative margin.
  • Beta rescales the preference logit for a fixed margin.
  • Conflicting labels cannot all become certain predictions.