Skip to content
AI360XpertBeta

Constitutional AI

You hand the model a short list of rules, let it critique and rewrite its own bad drafts, then train it on the rewrites and on AI judged preferences instead of human harmlessness labels.

A harmful draft is critiqued against a written constitution, revised into a safe answer, and the chosen pair teaches the reward model which response to prefer
A harmful draft is critiqued against a written constitution, revised into a safe answer, and the chosen pair teaches the reward model which response to prefer

Why Does This Exist?

Picture your chat assistant getting asked for hacking instructions. You want a short refusal plus useful legal help, not a tutorial. Pretraining never taught that choice. It taught plausible continuation.

The usual fix is RLHF. You collect human comparisons, train a scorer, then push the policy toward higher scores. It works. But the harmlessness half hurts. A person must read each toxic prompt and each draft answer. That labeling is slow and costly, it exposes annotators to ugly content, and it hides each annotator's private line for safe versus unsafe inside thousands of votes you will never see.

Constitutional AI swaps that human harmlessness vote for an AI vote steered by a written constitution. You write the rules in plain text. Do not help with intrusion. Offer a legal alternative. Keep the tone direct. The model writes a draft, checks itself against those lines, rewrites the draft, and you train on the rewrite. Then you repeat the trick for preferences: the model ranks pairs under the same rules, and those AI ranks train the scorer.

This page covers the two stage loop and the small preference math behind stage two. For the human baseline, read RLHF. For the scorer on its own, read reward modeling. For the shortcut that skips the scorer, read DPO.

You need one idea first. Supervised fine tuning keeps training a pretrained model on chosen input output pairs, with loss on the completion only. Stage one here is that same training, except the pairs are self made revisions instead of human demonstrations.

Think of It Like This

A rookie line cook and the rule card on the fridge

You run a small kitchen. A rookie cook plates a first attempt that is salty and undercooked. You do not hover over every plate. You tape a rule card to the fridge. Taste before salting. Chicken must reach temp. Wipe the rim before it leaves the pass.

The cook plates a draft, walks to the fridge, reads the card, spots the miss, and replates. You keep only the fixed plates in the training folder. The next rookie studies those plates, not the mistakes.

Where it stops: your cook starts to grasp why the rules matter. The model does not. It learns the shape of compliant text, not the reason behind it. And a fridge card cannot settle contested taste. When two rules pull apart or a case is truly gray, the card just enforces your wording. It does not resolve the ethics.

How It Actually Works

Call the prompt xx, so xx is the text the user typed. Call any answer yy, so yy is one full response. Call the rule list CC, so CC is the constitution, a short list of plain principles. Call one principle pp, so pp is a single line from CC. Call the policy π\pi, so π\pi is the model you train. Call the scorer RR, so R(x,y)R(x, y) is one number grading how well yy follows CC for xx.

Stage 1: critique, revise, then fine tune

You start from a helpful only model, one that answers even when it should not. For a red team prompt xx you sample an initial draft y0y_0 from π\pi. That draft is often useful but unsafe, which is the point. You want the mistake on the table.

Next you ask the same model to critique itself. You show it xx, y0y_0, and one pp from CC, and ask for a critique cc, where cc is a short note saying whether y0y_0 breaks pp and why. Then you ask for a revision y1y_1, where y1y_1 is a rewrite that keeps the helpful part and removes the violation. You repeat this with several principles, chaining small fixes.

Then you fine tune π\pi on pairs (x,y1)(x, y_1), the same way as supervised fine tuning. The prompt is context. Loss falls on the revision only. Do this over many prompts and you get a model that answers in the revised style on first try. No human wrote those answers. The model wrote them under the constitution, then learned from its own corrections.

Stage 2: AI preferences train the scorer, then the policy

Supervised revisions teach style, not fine choice. So stage two collects AI preferences. For one xx you sample two answers yay_a and yby_b from the stage one model. You ask a judge model, often the same π\pi, which answer follows CC better, with a short chain of thought quoting the relevant pp. That vote gives a winner ywy_w and a loser yly_l, where ywy_w is the preferred response and yly_l is the rejected one.

You train RR on many such triples (x,yw,yl)(x, y_w, y_l) the way reward modeling trains on human triples, except the label came from the constitution. Then you optimize π\pi against RR with RL and a KL penalty that keeps π\pi near its stage one self, the same leash that RLHF uses. The payoff is scale and transparency. Rules live in text you can read and edit. The price is blindness. Whatever bias the judge has, the scorer learns, and the policy then chases it.

Do not use this loop alone when the base model is too weak to critique reliably, when norms are contested, or when a mistake carries real harm, for example medical or legal advice. In those cases keep human labels, keep human review, and add runtime checks. The method assumes π\pi already follows instructions well enough to spot its own miss, and it assumes CC is clear, consistent, and ranked when two lines conflict. When CC is vague or contradictory, revisions turn evasive and the judge rewards safe sounding filler.

Worked example: one pair, real numbers

Take xx as How do I hack my neighbor's Wi-Fi. Let yly_l be detailed intrusion steps. Let ywy_w be a short refusal plus an offer to explain home network security and password sharing with permission. The AI judge picks ywy_w under the principle do not enable illicit intrusion.

Suppose the scorer gives R(x,yw)=1.5R(x, y_w) = 1.5 and R(x,yl)=0.3R(x, y_l) = 0.3. Write the gap dd as the difference, so d=1.5−0.3=1.2d = 1.5 - 0.3 = 1.2. Under the Bradley Terry choice model, the chance the judge prefers the winner is

P=11+exp⁡(−d)P = \frac{1}{1 + \exp(-d)}

where exp⁡\exp is the exponential and PP is that preference chance. Plug in d=1.2d = 1.2. Then exp⁡(−1.2)≈0.3012\exp(-1.2) \approx 0.3012, so P=1/(1+0.3012)≈0.7685P = 1 / (1 + 0.3012) \approx 0.7685. The loss on this pair is the negative log likelihood, so L=−ln⁡(P)L = -\ln(P), where ln⁡\ln is the natural log and LL is the training loss. Then L=−ln⁡(0.7685)≈0.2633L = -\ln(0.7685) \approx 0.2633. The safe answer now has a concrete pull of about 0.770.77 behind it, and the diagram below draws these same three numbers, 1.51.5 versus 0.30.3 giving 0.770.77.

Code

This snippet replays the worked example. It turns two reward numbers into a preference chance and a loss, the exact step stage two repeats thousands of times.

import math

def preference_prob(r_chosen: float, r_rejected: float) -> float:    """Chance the judge prefers the chosen answer."""    gap = r_chosen - r_rejected    return 1.0 / (1.0 + math.exp(-gap))

def preference_loss(r_chosen: float, r_rejected: float) -> float:    """Negative log likelihood of the observed AI preference."""    return -math.log(preference_prob(r_chosen, r_rejected))

p = preference_prob(1.5, 0.3)loss = preference_loss(1.5, 0.3)print(round(p, 4))     # -> 0.7685print(round(loss, 4))  # -> 0.2633

A gap of 1.21.2 gives 0.76850.7685, not certainty. That softness matters. The policy gets a nudge toward ywy_w, not a command, which is why many pairs are needed before behavior shifts.

Watch Out For

Vague rules breed evasive models

Symptom: after training, the model refuses benign prompts, pads answers with hedges, and helpfulness scores fall while harmlessness looks perfect. You will see refusals for network help that is fully legal, like resetting your own router.

Fix: rewrite CC until each line passes a benign test. Do not ship a rule you have not tried on ten innocent prompts. Split help versus harm with examples, rank the lines so safety outranks style but helpfulness still counts, and keep a small human spot check on refusals.

The judge's blind spot becomes the policy

Symptom: the reward number climbs week after week, yet transcripts get longer, flattering, or oddly formal. The policy found phrasing the AI scorer likes that humans do not. This is reward hacking with an AI labeler instead of a human one.

Fix: read real transcripts against human judgment, not just the scorer. Hold out some principles and prompts, test there, and keep an independent check at serve time. For high stakes topics, do not let the AI vote be the last word.

The Quick Version

  • Constitutional AI trains a model from a written constitution instead of human harmlessness comparisons.
  • Stage one samples a draft answer, critiques it against one rule, revises it, then fine tunes on the revision.
  • Stage two asks the model to rank answer pairs under the same rules and trains a reward model on those AI votes.
  • Scores of 1.5 versus 0.3 give a preference chance near 0.77 with loss near 0.26, so the safe answer is reinforced softly.
  • The loop assumes a judge strong enough to critique and a constitution clear enough to rank conflicting rules.
  • High stakes or contested norms still need human labels, human review, and runtime checks around the model.