Constitutional AI
You hand the model a short list of rules, let it critique and rewrite its own bad drafts, then train it on the rewrites and on AI judged preferences instead of human harmlessness labels.
Why Does This Exist?
Picture your chat assistant getting asked for hacking instructions. You want a short refusal plus useful legal help, not a tutorial. Pretraining never taught that choice. It taught plausible continuation.
The usual fix is RLHF. You collect human comparisons, train a scorer, then push the policy toward higher scores. It works. But the harmlessness half hurts. A person must read each toxic prompt and each draft answer. That labeling is slow and costly, it exposes annotators to ugly content, and it hides each annotator's private line for safe versus unsafe inside thousands of votes you will never see.
Constitutional AI swaps that human harmlessness vote for an AI vote steered by a written constitution. You write the rules in plain text. Do not help with intrusion. Offer a legal alternative. Keep the tone direct. The model writes a draft, checks itself against those lines, rewrites the draft, and you train on the rewrite. Then you repeat the trick for preferences: the model ranks pairs under the same rules, and those AI ranks train the scorer.
This page covers the two stage loop and the small preference math behind stage two. For the human baseline, read RLHF. For the scorer on its own, read reward modeling. For the shortcut that skips the scorer, read DPO.
You need one idea first. Supervised fine tuning keeps training a pretrained model on chosen input output pairs, with loss on the completion only. Stage one here is that same training, except the pairs are self made revisions instead of human demonstrations.
Think of It Like This
A rookie line cook and the rule card on the fridge
You run a small kitchen. A rookie cook plates a first attempt that is salty and undercooked. You do not hover over every plate. You tape a rule card to the fridge. Taste before salting. Chicken must reach temp. Wipe the rim before it leaves the pass.
The cook plates a draft, walks to the fridge, reads the card, spots the miss, and replates. You keep only the fixed plates in the training folder. The next rookie studies those plates, not the mistakes.
Where it stops: your cook starts to grasp why the rules matter. The model does not. It learns the shape of compliant text, not the reason behind it. And a fridge card cannot settle contested taste. When two rules pull apart or a case is truly gray, the card just enforces your wording. It does not resolve the ethics.
How It Actually Works
Call the prompt , so is the text the user typed. Call any answer , so is one full response. Call the rule list , so is the constitution, a short list of plain principles. Call one principle , so is a single line from . Call the policy , so is the model you train. Call the scorer , so is one number grading how well follows for .
Stage 1: critique, revise, then fine tune
You start from a helpful only model, one that answers even when it should not. For a red team prompt you sample an initial draft from . That draft is often useful but unsafe, which is the point. You want the mistake on the table.
Next you ask the same model to critique itself. You show it , , and one from , and ask for a critique , where is a short note saying whether breaks and why. Then you ask for a revision , where is a rewrite that keeps the helpful part and removes the violation. You repeat this with several principles, chaining small fixes.
Then you fine tune on pairs , the same way as supervised fine tuning. The prompt is context. Loss falls on the revision only. Do this over many prompts and you get a model that answers in the revised style on first try. No human wrote those answers. The model wrote them under the constitution, then learned from its own corrections.
Stage 2: AI preferences train the scorer, then the policy
Supervised revisions teach style, not fine choice. So stage two collects AI preferences. For one you sample two answers and from the stage one model. You ask a judge model, often the same , which answer follows better, with a short chain of thought quoting the relevant . That vote gives a winner and a loser , where is the preferred response and is the rejected one.
You train on many such triples the way reward modeling trains on human triples, except the label came from the constitution. Then you optimize against with RL and a KL penalty that keeps near its stage one self, the same leash that RLHF uses. The payoff is scale and transparency. Rules live in text you can read and edit. The price is blindness. Whatever bias the judge has, the scorer learns, and the policy then chases it.
Do not use this loop alone when the base model is too weak to critique reliably, when norms are contested, or when a mistake carries real harm, for example medical or legal advice. In those cases keep human labels, keep human review, and add runtime checks. The method assumes already follows instructions well enough to spot its own miss, and it assumes is clear, consistent, and ranked when two lines conflict. When is vague or contradictory, revisions turn evasive and the judge rewards safe sounding filler.
Worked example: one pair, real numbers
Take as How do I hack my neighbor's Wi-Fi. Let be detailed intrusion steps. Let be a short refusal plus an offer to explain home network security and password sharing with permission. The AI judge picks under the principle do not enable illicit intrusion.
Suppose the scorer gives and . Write the gap as the difference, so . Under the Bradley Terry choice model, the chance the judge prefers the winner is
where is the exponential and is that preference chance. Plug in . Then , so . The loss on this pair is the negative log likelihood, so , where is the natural log and is the training loss. Then . The safe answer now has a concrete pull of about behind it, and the diagram below draws these same three numbers, versus giving .
Code
This snippet replays the worked example. It turns two reward numbers into a preference chance and a loss, the exact step stage two repeats thousands of times.
import math
def preference_prob(r_chosen: float, r_rejected: float) -> float: """Chance the judge prefers the chosen answer.""" gap = r_chosen - r_rejected return 1.0 / (1.0 + math.exp(-gap))
def preference_loss(r_chosen: float, r_rejected: float) -> float: """Negative log likelihood of the observed AI preference.""" return -math.log(preference_prob(r_chosen, r_rejected))
p = preference_prob(1.5, 0.3)loss = preference_loss(1.5, 0.3)print(round(p, 4)) # -> 0.7685print(round(loss, 4)) # -> 0.2633A gap of gives , not certainty. That softness matters. The policy gets a nudge toward , not a command, which is why many pairs are needed before behavior shifts.
Watch Out For
Vague rules breed evasive models
Symptom: after training, the model refuses benign prompts, pads answers with hedges, and helpfulness scores fall while harmlessness looks perfect. You will see refusals for network help that is fully legal, like resetting your own router.
Fix: rewrite until each line passes a benign test. Do not ship a rule you have not tried on ten innocent prompts. Split help versus harm with examples, rank the lines so safety outranks style but helpfulness still counts, and keep a small human spot check on refusals.
The judge's blind spot becomes the policy
Symptom: the reward number climbs week after week, yet transcripts get longer, flattering, or oddly formal. The policy found phrasing the AI scorer likes that humans do not. This is reward hacking with an AI labeler instead of a human one.
Fix: read real transcripts against human judgment, not just the scorer. Hold out some principles and prompts, test there, and keep an independent check at serve time. For high stakes topics, do not let the AI vote be the last word.
The Quick Version
- Constitutional AI trains a model from a written constitution instead of human harmlessness comparisons.
- Stage one samples a draft answer, critiques it against one rule, revises it, then fine tunes on the revision.
- Stage two asks the model to rank answer pairs under the same rules and trains a reward model on those AI votes.
- Scores of 1.5 versus 0.3 give a preference chance near 0.77 with loss near 0.26, so the safe answer is reinforced softly.
- The loop assumes a judge strong enough to critique and a constitution clear enough to rank conflicting rules.
- High stakes or contested norms still need human labels, human review, and runtime checks around the model.