Preference Optimization Variants
DPO still loads a frozen copy of your model as an anchor. Its variants cut that cost, cap its overconfidence, or learn from plain thumbs up and down instead of ranked pairs.
Why Does This Exist?
You have done the expensive part. People compared answers, you hold pairs of winners and losers, and DPO turns those pairs into training with a single loss. No reward model to train, no PPO loop to babysit. Then you start the run and count the models in memory. There are two. The policy you train, plus a frozen copy of the model you started from. That copy is the reference, and it exists to anchor you. Every score is measured against it so the policy can't drift into fluent nonsense while chasing preferences.
That anchor is priced in weights. Roughly twice the parameters sit in memory, and the pairs are priced in labeling effort. Strict pairs, one winner and one loser per prompt, cost far more than loose approval. And the DPO loss itself has no ceiling. Train long enough and the chosen minus rejected gap grows without bound while the model gets cocky and dull.
This family exists because each of those costs got its own answer. ORPO and SimPO dismiss the second judge. They score the policy alone, so one model sits in memory instead of two. KTO dismisses the ranked ballot. It learns from single thumbs up or down answers, the kind of feedback you already collect. IPO keeps the reference but caps the ambition. It tells training which gap is good enough and stops pushing past it.
This page covers those four deltas and nothing else. The base loss, with its derivation and its own worked numbers, lives in DPO. The reinforcement loop DPO replaced lives in RLHF. The scorer DPO deleted lives in reward modeling. The sibling method that saves memory inside RL instead lives in GRPO.
You need one thing going in. A preference pair is one prompt plus two answers, a winner and a loser . The policy is the model you train, with weights . The reference is the frozen starting model. And is a small positive number that sets how hard the anchor pulls. That's the whole vocabulary below.
Think of It Like This
Judging a cook-off with last year's cookbook on the counter
Your town runs a cook-off. Two new dishes per round, one voted better. Last year's winning cookbook sits on the counter.
The DPO judge cooks with the book open. Every score is how far today's better dish beats the book's version of it. The book keeps judging honest, but someone must hold it open all day. That's your second model in memory.
The ORPO judge closes the book. She teaches the recipe and judges the ranking in one shift, scoring how much likelier the winning dish is than the losing one, head to head. The SimPO judge goes further. She scores per bite, not per plate, so a bigger plate cannot win on size, and she demands the winner clear a set margin. The KTO judge stops ranking plates at all. She puts out two jars, one for thumbs up slips and one for thumbs down, and learns from each slip alone. The IPO judge keeps the book open but caps the score, so no dish can win by a hundred points and warp the whole contest.
Where it stops: judges taste food. These losses only read probabilities. The analogy hides the dials that make each variant work, beta, gamma, the odds weight, the IPO target, and what happens when the votes themselves are wrong. A biased crowd teaches a biased kitchen through any scoring rule.
How It Actually Works
DPO is the trunk and the four variants are branches. Read this section as four small edits to one loss, each buying something and paying something.
The shared starting point
Write for the prompt in plain words, for the winner and for the loser. Write for the policy you train, meaning the probability your current model assigns to answer given , with weights . Write for the frozen reference, almost always the supervised fine-tuned model from before preference training. Write for a small positive number setting how hard the anchor pulls, and for the sigmoid, . The DPO loss is:
Read it inside out. The first half is your policy gap, how much likelier you make the winner than the loser in log space. The second half is the same gap under the frozen reference. You score only for beating the reference gap. Everything below is a change to this shape. The full derivation is in DPO.
ORPO learns the recipe and the ranking in one go
ORPO asks why training takes two stages at all. Standard practice runs supervised fine-tuning first, then preference tuning. ORPO folds both into one update and drops the reference entirely.
Write the odds of an answer as its probability against itself not happening:
In plain words, odds of 3 to 1 mean the model finds that answer three times likelier than everything else combined. The preference term rewards a large log odds ratio between winner and loser:
And the full loss adds the ordinary imitation term, with meaning the negative log probability of the winner, plus a weight balancing the two:
No appears anywhere, so one model sits in memory instead of two. The price moved. Imitation and preference now share one gradient, and decides the split. Set it too high and the model ranks without learning the style. Set it too low and you relabeled SFT. Tune it on held-out pairs, not training loss.
SimPO grades per bite and demands a clear win
SimPO also drops the reference, but it fixes a second flaw while doing so. Raw sequence scores confound quality with length, so a model can win by padding. SimPO divides by length and demands a margin.
Write for the token length of answer . The SimPO reward is the length normalized log probability, scaled by :
In plain words, each answer is scored per token, not per sequence. The loss then asks the winner to beat the loser by a margin (gamma), a positive number you set:
The margin is the point. Without it, any tiny lead scores near zero loss and training coasts. With it, the policy must separate the pair by at least before the loss quiets. The price is itself. Too small and the margin means nothing. Too large and no pair ever satisfies it, so training pushes forever on noise. And note the tilt this creates. Per token scoring slightly favors concise answers when quality is equal, which is usually what you want, but if your task genuinely needs long answers, watch completion length during training.
IPO puts a ceiling on the score
IPO keeps the reference and changes the ambition. DPO pushes the lead over the reference gap toward infinity. IPO names a finite target and penalizes distance from it in either direction.
Write (tau) for a positive regularization number you set. The target gap is , and the loss is squared error against it:
In plain words, the bracket is the same lead DPO measures, policy gap minus reference gap. DPO says bigger is always better. IPO says 0.7 against a target of 1.0 still has something to learn, and 5.0 against a target of 1.0 has overshot and should come back. That squared penalty is what stops the overconfidence spiral where DPO ends up absurdly sure and quietly worse.
The price is honesty about memory. IPO still loads the reference, so it saves nothing in VRAM. And needs care in the opposite direction from most dials. A tiny sets a huge target, which rebuilds the very runaway you bought IPO to prevent. Start from the target gap you actually want, then set from it.
KTO counts thumbs instead of ranking plates
KTO drops a different requirement, the pairing. Some feedback is just a like or a dislike on one answer, with no rival attached. Throwing that away because it is not a pair wastes your cheapest data.
Each example is one prompt, one answer, and one label. Desirable means thumbs up, undesirable means thumbs down. Write the implicit reward , meaning how much likelier the current policy finds this answer than the frozen reference did. Write for the reference point, a running average of how far the policy has drifted overall. Write and for the weights on desirable and undesirable examples, with the undesirable weight usually larger because bad outputs teach more per sample.
The update pushes desirable answers above and undesirable ones below it, with strength for a liked answer and for a disliked one. Because both pass through a sigmoid, the rule has diminishing returns. An already likely good answer earns little extra, while a likely bad answer is penalized steeply. That asymmetry is borrowed from prospect theory, the finding that losses sting more than equal gains please.
The price is twofold. KTO still needs the reference to compute and , so memory is unchanged from DPO. And binary labels carry less information per sample than pairs, so KTO wants volume. A small pile of noisy likes teaches style and verbosity just as fast as pairs do.
Worked example with one pair
Fix one prompt and one pair for all four losses. Your policy assigns to the winner and to the loser. The frozen reference assigns and . Take . The winner runs 6 tokens, the loser 12, so and .
DPO first, as the baseline. Your policy gap is . The reference gap is . You lead by , scaled to . Then and the loss is . Same numbers as the DPO page, which is the point. The variants below all start here.
SimPO with and : the rewards are and . Per token both answers score , so the gap is and the pair is a tie. Inside the sigmoid sits , giving and a loss near . DPO saw a 0.7 lead and scored 0.659. SimPO sees two equally fluent answers at different lengths, refuses the length win, and keeps the loss high until the policy separates them per token. That refusal is the diagram above.
ORPO with : convert to probabilities, and . The log odds are for the winner and for the loser, a log odds ratio near . Then , so . The imitation term is , giving a total near . Two terms, one model, no reference in sight.
IPO with : the target gap is . Your lead is , so the loss is . Small, because 0.7 is close to good enough, and it can never collapse to zero by pushing to infinity. That ceiling is the whole method.
KTO on a single slip: say a liked answer has against a reference point . It sits barely above the bar, so with its push scales with , barely above a coin flip, and training spends little on it. A disliked answer at sits far below, pushes with times the larger , and moves the weights. Easy wins whisper, clear failures shout.
So the trade is plain. ORPO and SimPO buy memory by deleting the reference, and pay with new dials, and , that need held-out tuning. IPO buys stability by capping the gap, and pays by keeping both models. KTO buys cheap unary data, and pays with volume and the same two model memory as DPO.
Don't reach for any of them when the real problem sits elsewhere. If you need a reusable scorer for reranking or filtering, train one with reward modeling instead of deleting it. If answers have checkable truth, like code that passes tests, prefer verifiable rewards as in RLVR, where GRPO spends generation instead of memory. All four variants assume your labels say what you mean. If chosen answers are just longer, every loss here learns that longer is better, and no variant cleans labels for you.
Code
This snippet replays the fixture so you can check the arithmetic before trusting a trainer. It computes the SimPO rewards and loss plus the ORPO odds term from the same log probabilities and lengths.
import math
def sigmoid(z: float) -> float: """Squash any number into a probability between 0 and 1.""" return 1.0 / (1.0 + math.exp(-z))
# Fixture: one prompt, one pair (sequence log-probs plus token lengths).logp_w: float = -1.2logp_l: float = -2.4len_w: int = 6len_l: int = 12
# SimPO: length normalized rewards with a margin gamma.beta_s: float = 2.0gamma: float = 0.5r_w: float = (beta_s / len_w) * logp_wr_l: float = (beta_s / len_l) * logp_lsimpo_loss: float = -math.log(sigmoid(r_w - r_l - gamma))
# ORPO: odds ratio term plus the plain imitation term (lambda = 1).p_w: float = math.exp(logp_w)p_l: float = math.exp(logp_l)log_or: float = math.log(p_w / (1.0 - p_w)) - math.log(p_l / (1.0 - p_l))or_term: float = -math.log(sigmoid(log_or))orpo_loss: float = -logp_w + or_term
print(round(r_w, 4)) # -> -0.4print(round(r_l, 4)) # -> -0.4print(round(simpo_loss, 4)) # -> 0.9741print(round(log_or, 4)) # -> 1.4633print(round(orpo_loss, 4)) # -> 1.4083The prints match the hand calculation. Both rewards land on , the tie SimPO sees. The margin turns that tie into a loss of , the odds ratio lands near , and the ORPO total sits near . Change the loser length to 6 and watch the SimPO tie break.
Watch Out For
Deleting the reference from DPO instead of switching losses
Symptom: you drop the reference terms from the DPO loss to fit a bigger batch, training loss falls nicely, and the model gets worse. It repeats winning phrases, pads answers, and forgets supervised skills it had on day one. Lowering the learning rate only slows the rot.
The cause is a missing leash, not a slow optimizer. Without the reference gap the loss rewards any widening of chosen over rejected, including drift into regions the starting model never visited. Do not delete the reference from DPO. Switch to a loss designed to live without one. ORPO brings the imitation term that holds style, SimPO brings length normalization plus the margin, and both still need their dials tuned on held-out pairs. Track the policy gap and a held-out quality read during training. If the gap climbs while quality falls, the leash is gone.
Switching losses while the pairs teach length
Symptom: you try DPO, then ORPO, then SimPO, and every run prefers verbose answers. You check and find the chosen answer is longer than the rejected one in most pairs. Each new loss inherits the same lesson because the lesson is in the labels.
The fix is counting before tuning. Measure how often the winner is simply the longer answer. If that rate is high, rebalance or trim the pairs first, then pick the loss. SimPO softens length bias through per token scoring and the margin, but it can't overturn labels that say long equals good. Variants reshape the loss. They never clean your data.
The Quick Version
- DPO trains beside a frozen reference, so the policy and its starting copy both sit in memory.
- ORPO drops the reference by adding an odds ratio term to the plain imitation loss, teaching style and ranking in one stage.
- SimPO drops the reference with per token rewards and a margin gamma. The fixture ties at minus 0.4 each and scores loss 0.974.
- IPO keeps the reference but replaces endless pushing with a squared gap target. The fixture lead of 0.7 against a target of 1.0 scores 0.09.
- KTO drops the pairing and learns from single liked or disliked answers, weighting failures harder than equal wins.
- Reference free is not discipline free. Every variant needs its own leash, and biased pairs teach bias through any loss.