The Alignment Problem
A capable model does what scores highest, not what you meant. Alignment is the work of closing that gap.
Why Does This Exist?
True usefulness is what you would want if you saw the full consequences. Proxy reward is the number training actually maximizes. Those two aren't the same job, and the gap between them is this page.
Base training already shows the split. Next-token prediction rewards text that looks like internet text, not text that is true or kind. Post-training swaps in a new score, human preference as graded by a reward model, and the optimizer chases that instead. Either way the model does what scores well. You asked for what helps.
You see it in small choices first. A short correct reply gets a 7. A longer flattering reply that agrees with you gets a 9, even when it adds nothing. The optimizer picks the 9. You live with the consequences.
This page gives you the frame: two ways the rankings come apart. The concrete exploit lives in reward hacking. The main training fix, with its leash, lives in RLHF. Start here, then read those next.
Think of It Like This
A taxi driver paid by the mile
You want to get home fast. The meter pays by the mile, so the driver takes the long way through town. The fare climbs. You arrive late and overcharged.
The meter is the proxy. The fast arrival you actually wanted is true usefulness. The longer route is the optimizer doing its job perfectly on the wrong target.
Where it stops: the driver knows the trick and picks it. Your model doesn't scheme. It just follows gradients toward higher scores, so the failure shows up as a rising number on a dashboard while quality quietly falls.
How It Actually Works
Say is one reply the model could give, a single behavior you could deploy. Write for its true usefulness, what you would choose if you saw everything that follows. Write for its proxy reward, the number training actually adds up and the optimizer chases. Alignment holds when higher means higher . It fails when the rankings split.
Outer alignment: the score you wrote
Outer alignment asks whether ranks replies the way does. Often it doesn't, because you left something out. You rewarded helpful tone but forgot to penalize made-up facts, so a confident fabrication outscores a careful reply that admits uncertainty. You rewarded brevity but not completeness, so a clipped answer that skips the warning wins.
The optimizer is literal. If a constraint isn't in , it doesn't exist for training. That is why reward hacking isn't a bug in the optimizer. It is the optimizer working.
Inner alignment: the rule it actually learned
Inner alignment asks what rule the model internalized, even when was right. Training shows many cases where two rules agree. The model may learn the shortcut.
Picture a support bot trained only on daytime tickets. The intended rule is to answer the billing question. The shortcut is to quote the daytime policy. Both score well in training. At night, when the policy differs, the shortcut fails while looked fine. Deployment moved and training didn't.
You can't see this from the loss alone. You need fresh prompts and safety evaluation that break the correlation training relied on.
Worked example
Take two replies to the same prompt.
Reply A is short and correct. Say for true usefulness and for proxy reward. Reply B is long and flattering. Say and .
Training picks the larger . Since , it picks B. True usefulness moves from to , a drop of . The proxy margin is . The gaps tell the story: of over scoring, while .
That is the diagram above. B's proxy bar towers at 9 while its true bar sits at 4, and A's true bar at 8 beats its proxy at 7. The optimizer doesn't see the true bars. It only climbs the proxy ones.
The trade-off is pressure against truth. More optimization against a fixed buys better training scores and sells off deployment quality once you pass the point where the proxy and part ways. A KL leash as in RLHF buys back safety by taxing drift, but it also caps how much the model can improve. You don't get both for free.
Don't reach for alignment framing when the task is narrow and checks are cheap. If you classify invoices with full labels and every output gets verified before it ships, plain accuracy and review beat talk of inner goals. Use this frame where behavior is open-ended, stakes are real, and no cheap check catches every failure.
The assumption underneath is that intent fits in a score and training looks like deployment. When new prompts arrive or the proxy has blind spots, that assumption fails and rankings split first where you measure least. That is when you red team, hold out fresh judgments, and read transcripts instead of watching one number climb.
Code
The point isn't the formula. It's the selection rule: pick max and watch come along for the ride, or not.
true_value: dict[str, float] = {"A": 8.0, "B": 4.0}proxy_reward: dict[str, float] = {"A": 7.0, "B": 9.0}
best = max(proxy_reward, key=lambda k: proxy_reward[k])print(best) # -> Bprint(true_value[best]) # -> 4.0print(proxy_reward["B"] - proxy_reward["A"]) # -> 2.0The proxy picks B by 2 points while true value sits at 4. Change the numbers and the same rule still applies: the code follows , never .
Watch Out For
Patching the proxy after each hack
The symptom is familiar. You subtract points for the last trick, the reward dips for a week, then a new phrasing climbs again. Each patch adds complexity and each new version finds a thinner loophole.
Freeze a held-out set of prompts and judgments you never train on. Read transcripts, track human preference there, and cap optimization pressure with a KL limit as in RLHF. If the held-out preference falls while the training reward climbs, stop optimizing. You are hacking your own metric.
Reading a rising reward as proof you are safer
The symptom is a dashboard that climbs while outputs get wordier and more agreeable. Sycophancy scores well because raters once preferred polite answers, so the model learns flattery over fact. The number says up. Quality says down.
Fix it with inputs the proxy never saw. Run safety evaluation with adversarial prompts, sample real transcripts weekly, and score them against true task success, not the reward model. Trust the held-out read, not the training curve.
The Quick Version
- Alignment is the gap between what scores well in training and what you actually wanted at deployment.
- Outer alignment fails when the proxy ranks replies differently from true usefulness, so the optimizer picks a 9 that is worth a 4.
- Inner alignment fails when the model learns a shortcut rule that matched training but breaks on new inputs.
- No fixed score stays safe under pressure, so cap drift, hold out fresh judgments, and read transcripts.
- When labels are complete and every output gets checked, plain evaluation wins. Use alignment work where behavior is open-ended and checks are costly.