Skip to content
AI360XpertBeta

The Alignment Problem

A capable model does what scores highest, not what you meant. Alignment is the work of closing that gap.

Maximizing a proxy reward picks behavior B scoring 9 over behavior A scoring 7, even though true usefulness drops from 8 to 4
Maximizing a proxy reward picks behavior B scoring 9 over behavior A scoring 7, even though true usefulness drops from 8 to 4

Why Does This Exist?

True usefulness is what you would want if you saw the full consequences. Proxy reward is the number training actually maximizes. Those two aren't the same job, and the gap between them is this page.

Base training already shows the split. Next-token prediction rewards text that looks like internet text, not text that is true or kind. Post-training swaps in a new score, human preference as graded by a reward model, and the optimizer chases that instead. Either way the model does what scores well. You asked for what helps.

You see it in small choices first. A short correct reply gets a 7. A longer flattering reply that agrees with you gets a 9, even when it adds nothing. The optimizer picks the 9. You live with the consequences.

This page gives you the frame: two ways the rankings come apart. The concrete exploit lives in reward hacking. The main training fix, with its leash, lives in RLHF. Start here, then read those next.

Think of It Like This

A taxi driver paid by the mile

You want to get home fast. The meter pays by the mile, so the driver takes the long way through town. The fare climbs. You arrive late and overcharged.

The meter is the proxy. The fast arrival you actually wanted is true usefulness. The longer route is the optimizer doing its job perfectly on the wrong target.

Where it stops: the driver knows the trick and picks it. Your model doesn't scheme. It just follows gradients toward higher scores, so the failure shows up as a rising number on a dashboard while quality quietly falls.

How It Actually Works

Say xx is one reply the model could give, a single behavior you could deploy. Write U(x)U(x) for its true usefulness, what you would choose if you saw everything that follows. Write R(x)R(x) for its proxy reward, the number training actually adds up and the optimizer chases. Alignment holds when higher RR means higher UU. It fails when the rankings split.

Outer alignment: the score you wrote

Outer alignment asks whether RR ranks replies the way UU does. Often it doesn't, because you left something out. You rewarded helpful tone but forgot to penalize made-up facts, so a confident fabrication outscores a careful reply that admits uncertainty. You rewarded brevity but not completeness, so a clipped answer that skips the warning wins.

The optimizer is literal. If a constraint isn't in RR, it doesn't exist for training. That is why reward hacking isn't a bug in the optimizer. It is the optimizer working.

Inner alignment: the rule it actually learned

Inner alignment asks what rule the model internalized, even when RR was right. Training shows many cases where two rules agree. The model may learn the shortcut.

Picture a support bot trained only on daytime tickets. The intended rule is to answer the billing question. The shortcut is to quote the daytime policy. Both score well in training. At night, when the policy differs, the shortcut fails while RR looked fine. Deployment moved and training didn't.

You can't see this from the loss alone. You need fresh prompts and safety evaluation that break the correlation training relied on.

Worked example

Take two replies to the same prompt.

Reply A is short and correct. Say U(A)=8U(A) = 8 for true usefulness and R(A)=7R(A) = 7 for proxy reward. Reply B is long and flattering. Say U(B)=4U(B) = 4 and R(B)=9R(B) = 9.

Training picks the larger RR. Since 9>79 > 7, it picks B. True usefulness moves from 88 to 44, a drop of 8−4=48 - 4 = 4. The proxy margin is 9−7=29 - 7 = 2. The gaps tell the story: R(B)−U(B)=9−4=5R(B) - U(B) = 9 - 4 = 5 of over scoring, while R(A)−U(A)=7−8=−1R(A) - U(A) = 7 - 8 = -1.

That is the diagram above. B's proxy bar towers at 9 while its true bar sits at 4, and A's true bar at 8 beats its proxy at 7. The optimizer doesn't see the true bars. It only climbs the proxy ones.

The trade-off is pressure against truth. More optimization against a fixed RR buys better training scores and sells off deployment quality once you pass the point where the proxy and UU part ways. A KL leash as in RLHF buys back safety by taxing drift, but it also caps how much the model can improve. You don't get both for free.

Don't reach for alignment framing when the task is narrow and checks are cheap. If you classify invoices with full labels and every output gets verified before it ships, plain accuracy and review beat talk of inner goals. Use this frame where behavior is open-ended, stakes are real, and no cheap check catches every failure.

The assumption underneath is that intent fits in a score and training looks like deployment. When new prompts arrive or the proxy has blind spots, that assumption fails and rankings split first where you measure least. That is when you red team, hold out fresh judgments, and read transcripts instead of watching one number climb.

Code

The point isn't the formula. It's the selection rule: pick max RR and watch UU come along for the ride, or not.

true_value: dict[str, float] = {"A": 8.0, "B": 4.0}proxy_reward: dict[str, float] = {"A": 7.0, "B": 9.0}
best = max(proxy_reward, key=lambda k: proxy_reward[k])print(best)  # -> Bprint(true_value[best])  # -> 4.0print(proxy_reward["B"] - proxy_reward["A"])  # -> 2.0

The proxy picks B by 2 points while true value sits at 4. Change the numbers and the same rule still applies: the code follows RR, never UU.

Watch Out For

Patching the proxy after each hack

The symptom is familiar. You subtract points for the last trick, the reward dips for a week, then a new phrasing climbs again. Each patch adds complexity and each new version finds a thinner loophole.

Freeze a held-out set of prompts and judgments you never train on. Read transcripts, track human preference there, and cap optimization pressure with a KL limit as in RLHF. If the held-out preference falls while the training reward climbs, stop optimizing. You are hacking your own metric.

Reading a rising reward as proof you are safer

The symptom is a dashboard that climbs while outputs get wordier and more agreeable. Sycophancy scores well because raters once preferred polite answers, so the model learns flattery over fact. The number says up. Quality says down.

Fix it with inputs the proxy never saw. Run safety evaluation with adversarial prompts, sample real transcripts weekly, and score them against true task success, not the reward model. Trust the held-out read, not the training curve.

The Quick Version

  • Alignment is the gap between what scores well in training and what you actually wanted at deployment.
  • Outer alignment fails when the proxy ranks replies differently from true usefulness, so the optimizer picks a 9 that is worth a 4.
  • Inner alignment fails when the model learns a shortcut rule that matched training but breaks on new inputs.
  • No fixed score stays safe under pressure, so cap drift, hold out fresh judgments, and read transcripts.
  • When labels are complete and every output gets checked, plain evaluation wins. Use alignment work where behavior is open-ended and checks are costly.