DPO

Why Does This Exist?

You've finished supervised fine-tuning. Your model follows instructions, but it still cannot tell a sharp answer from a bland one. What you do have is a pile of human votes: for the same prompt, answer A beat answer B. Hundreds or thousands of such pairs.

RLHF puts those votes to work in two moves. First you train a separate scorer to imitate the voters, which is reward modeling. Then you run PPO, a reinforcement learning loop, to push the model toward high scores while a KL penalty holds it near where it started. That works. It also means two training loops, a rollout sampler, value heads, and a whole second model to store and tune.

DPO (direct preference optimization) asks a shorter question. If the votes already say what to prefer, why build a scorer just so an RL loop can chase it? Under one standard assumption about how preferences work, you can train the policy straight from the pairs with a single classification-style loss. Same data, same goal, one stage.

This page covers that one loss: what it compares, what holds it back from drifting, and where it breaks. For the full loop it replaces, read RLHF. For the scorer it deletes, read reward modeling. For the follow-up methods that remove even more, read preference optimization variants.

You need two ideas going in. First, supervised fine-tuning gave you the starting model, and DPO keeps a frozen copy of it as an anchor. Second, a preference pair is one prompt plus two completions, one marked chosen and one marked rejected. Nothing else is assumed.

Think of It Like This

How It Actually Works

DPO starts where RLHF starts: people prefer one answer over another, and you model that choice as coming from a hidden quality gap.

The preference bet

Write $x$ for the prompt in plain words, $y_w$ for the winner (the chosen response), and $y_l$ for the loser (the rejected response). Write $r(x, y)$ for a hidden score of how good response $y$ is for prompt $x$. The Bradley-Terry assumption says the chance that a voter picks $y_w$ over $y_l$ is:

Here $\sigma$ is the sigmoid, the function $\sigma(z) = 1 / (1 + e^{-z})$ that squashes any number into a probability between 0 and 1. A big score gap means near-certain preference. A zero gap means a coin flip. This is the bet DPO inherits. If real voters do not behave like a score gap passed through a sigmoid, for example when votes are noisy, cyclic, or driven by length alone, the loss trains confidently toward the wrong thing.

RLHF learns $r$ as a neural network and then runs RL against it. DPO never learns $r$ at all.

How the reward model disappears

The RLHF objective says: earn high reward, but pay a cost for drifting from where you started. Write $\pi_\theta(y \mid x)$ for the policy you are training, meaning the probability your current model assigns to response $y$ given prompt $x$, where $\theta$ holds its weights. Write $\pi_{ref}(y \mid x)$ for the frozen reference, almost always the supervised fine-tuned model before DPO. Write $\beta$ for a small positive number that sets how hard the anchor pulls, with larger $\beta$ holding you closer to the reference.

For that objective the best policy has a closed form, and you can solve it for the reward:

Here $C(x)$ depends on the prompt only, so when you subtract two rewards for the same prompt it cancels. Plug this into the Bradley-Terry rule and the reward model is gone. What remains is the DPO loss over the policy alone:

Read it inside out. The first bracket is your policy gap: how much more likely you make the chosen answer than the rejected one, in log space. The second bracket is the same gap under the frozen reference. The loss rewards you only for beating the reference gap, scaled by $\beta$. The outer $-\log \sigma$ is the same shape as logistic classification: confident and correct gives near-zero loss, wrong gives large loss.

You get stability because there is no sampler, no value network, and no PPO clipping to tune. You lose something real too. There is no standalone scorer left to reuse for reranking, filtering, or scoring another model. And because training is offline on fixed pairs, the policy never gets to try a new answer and ask a voter about it. It can only reshuffle probability among answers shaped like the training set.

Don't reach for DPO when you need that scorer, when you need online trial and error, or when your pairs are thin and you want to inspect a reward model on held-out prompts before it touches the policy. DPO skips that checkpoint, so biased pairs land straight in the weights.

Worked example with small numbers

Fix one prompt $x$ and one pair. Suppose your current policy assigns log-probabilities $\log \pi_\theta(y_w \mid x) = -1.2$ to the chosen answer and $\log \pi_\theta(y_l \mid x) = -2.4$ to the rejected one. The frozen reference assigns $\log \pi_{ref}(y_w \mid x) = -1.3$ and $\log \pi_{ref}(y_l \mid x) = -1.8$. Take $\beta = 0.1$ for this example.

Your policy gap is $-1.2 - (-2.4) = 1.2$. The reference gap is $-1.3 - (-1.8) = 0.5$. You lead the reference by $1.2 - 0.5 = 0.7$. Scaled, the margin inside the sigmoid is $0.1 \times 0.7 = 0.07$.

Then $\sigma(0.07) = 1 / (1 + e^{-0.07}) \approx 0.5175$. The model is barely more confident in the right order than a coin flip, once the reference is accounted for. The loss is $-\log(0.5175) \approx 0.6588$. Training pushes $\log \pi_\theta(y_w \mid x)$ up and $\log \pi_\theta(y_l \mid x)$ down, but the gradient shrinks as that 0.07 grows, so easy pairs stop shouting and hard pairs keep teaching. If you delete the reference terms, the same pair reports a margin of $0.1 \times 1.2 = 0.12$ and a smaller loss, yet nothing stops the policy from drifting far away while chasing it.

Code

This snippet mirrors the worked example. It computes the DPO signal for one pair from log-probabilities you already have, so you can check the arithmetic before trusting a trainer.

1import math2
3
4def dpo_loss_signal(5    chosen_logprob: float,6    rejected_logprob: float,7    ref_chosen_logprob: float,8    ref_rejected_logprob: float,9    beta: float = 0.1,10) -> float:11    """One-pair DPO loss from log-probs. Lower means the policy ranks correctly."""12    policy_gap: float = chosen_logprob - rejected_logprob13    ref_gap: float = ref_chosen_logprob - ref_rejected_logprob14    margin: float = beta * (policy_gap - ref_gap)15    prob_correct: float = 1.0 / (1.0 + math.exp(-margin))16    return -math.log(prob_correct)17
18
19loss: float = dpo_loss_signal(-1.2, -2.4, -1.3, -1.8)20print(round(loss, 4))  # -> 0.6588

The print shows 0.6588, the same number the hand calculation gives. Change the rejected log-probability to a less likely value and watch the loss fall. That is the whole teaching signal.

Watch Out For

The Quick Version

• DPO trains the policy straight from chosen and rejected answers with one classification-style loss, so no reward model and no PPO loop are needed.
• The loss rewards only the lead over a frozen reference gap, scaled by beta, which keeps the tuned model near the supervised model it started from.
• The worked fixture gives gaps of 1.2 for the policy and 0.5 for the reference, a margin of 0.07 at beta 0.1, and a loss near 0.6588.
• DPO buys simplicity and stable training at the cost of a reusable scorer and any online correction from fresh human feedback.
• It assumes preferences follow a Bradley-Terry score gap, so noisy, biased, or length-driven pairs flow straight into the weights.
• When you need reranking, filtering with a separate scorer, or exploration beyond fixed pairs, train a reward model instead of skipping it.

What to Read Next

• RLHF
• Reward Modeling
• Preference Optimization Variants
• Supervised Fine-Tuning