Activation Patching
Swap one clean layer state into a corrupted run. If the right answer comes back, that layer caused it.
Why Does This Exist?
Your model answers "Paris" when you type "The capital of France is". Nice. But which part of the network moved the France fact into that answer? A probe can tell you that some layer contains France information, yet that is only correlation. The information could sit there unused while a different layer does the real work.
That gap is the whole problem. Activation patching is the causal test: you break the behavior with a corrupted prompt, then repair one hidden state at a time with the clean value. When the answer snaps back, you've found a state the model actually uses, not one that merely looks informative.
This page covers that swap test and the restoration score it produces. It does not cover reading layers without touching them (see the logit lens), untangling features into readable directions (see sparse autoencoders), or reusing the same machinery to steer behavior (see activation steering). Those are neighbours; this page is the causality check.
The one idea you need first: a language model picks each word by scoring every vocabulary item and turning those scores into probabilities, so "Paris at 0.82" means 82 percent of the probability mass sits on Paris. The full treatment is in next-token prediction.
Think of It Like This
Two cars and one borrowed spark plug
Your car starts every morning. Your neighbour owns the same model, but theirs only coughs. You suspect one spark plug. So you pull a plug from your working engine, fit it into the dead engine, and turn the key. It roars to life. You have your answer, and you never opened a manual.
The mapping is direct. Your car is the clean run, the dead car is the corrupted run, the borrowed plug is the patched hidden state, and the engine catching is the restored answer. One part moves, everything else stays put, so the recovery names the cause.
Where it stops: car parts are separable, but network layers share the job. A borrowed plug fixes a car completely, while one patched state often restores only part of the probability. Partial recovery still counts as evidence. It just means the circuit is spread out.
How It Actually Works
A transformer turns a prompt into a stack of hidden states, one per layer per token position, then decodes the last position into next-word probabilities. Patching reaches into that stack mid-run and overwrites a single entry. Everything downstream of the overwrite runs as normal, so any change in the output must have travelled through the state you replaced.
The three runs
A clean run feeds the model the prompt that works, say for "The capital of France is", and records every hidden state plus the target probability , where means the probability of "Paris" on this run. A corrupted run feeds a minimal-pair prompt that keeps grammar and length but removes the fact, say "The capital of Italy is", and records , the probability of "Paris" there. Expect near zero, since the correct continuation is now "Rome".
The patched run repeats the corrupted prompt, but pauses the forward pass at one chosen layer (with meaning the layer index, counting from the input side) and one token position, throws away the corrupted hidden state , where means the hidden vector at layer , and writes in the saved clean vector . Then the run continues untouched. Call the resulting Paris probability . If jumps back toward , the state at carries causally useful France information.
The restoration score
Raw probabilities mislead, because baselines differ per prompt. So you score the patch by how much of the gap it closes:
In plain words: , the restoration score, is the recovered probability divided by the total probability lost to corruption. means the single patch fully restored the clean behavior. means it did nothing. Values in between mean the circuit is spread across states, which is the common case.
Worked example: France against Italy
Take these toy readings for the target word "Paris". Clean run: . Corrupted run with the Italy prompt: . Patched run, swapping only the layer 15 state at the last token: .
Step one, recovered probability:
Step two, total gap corruption opened:
Step three, restoration:
So this one state closes about 73 percent of the gap. That's strong evidence that layer 15 at the final position moves France information toward the output, and equally strong evidence that it's not the whole story, since 27 percent is still missing.
Sweeping the network to find the circuit
You repeat the patched run once per (layer, token position) cell and record each time. Plotted as a grid, the high cells are the circuit: typically an early bump over the subject words where the fact gets read, and a late bump at the final token where the model commits to emitting it. That picture is the payoff, and it is what superposition and circuits builds on.
The trade is cost for proof. A full sweep needs hundreds of patched runs, while a read-only method needs one. So don't reach for patching when you only want a quick look at what a layer holds. Reach for it when you need to say a state causes the behavior. The method assumes the pasted state still looks natural enough that later layers behave, which holds for minimal pairs and fails for wildly different prompts.
Code
The point of this snippet is the score itself: given three measured probabilities, how much of the gap did one patch close? It mirrors the worked example above.
def restoration(p_clean: float, p_corr: float, p_patch: float) -> float: """Fraction of the corrupted-to-clean gap that one patch closes.""" return (p_patch - p_corr) / (p_clean - p_corr)
p_clean, p_corr, p_patch = 0.82, 0.04, 0.61r = restoration(p_clean, p_corr, p_patch)print(f"{r:.4f}") # -> 0.7308print(f"{r:.0%}") # -> 73%The two prints agree with the hand computation: about 0.73, or 73 percent. In a real sweep you would call this function once per patched cell and keep the cells with the largest .
Watch Out For
The corrupted prompt changes more than the fact
If your pair differs in length, grammar, or topic, the pasted state lands in alien territory. Symptom: every patch scores near zero, or outputs turn to gibberish, and you conclude nothing matters. The fix is a strict minimal pair. Change France to Italy and nothing else, check that the corrupted run is fluent but wrong (it should say "Rome" confidently), and only then trust the sweep.
One strong patch is not the whole circuit
A restoration of 0.73 feels like case closed, so people stop and name that layer "the France layer". Symptom: later edits to that layer alone fail to remove the behavior, because backup states compensate. The fix is to report the full grid and test combinations: patch the top two or three cells together, and expect partial scores from single cells whenever the circuit is spread out.
The Quick Version
- Activation patching proves causality by writing one clean hidden state into a corrupted run and watching the answer.
- A clean run, a corrupted run, and a patched run form the experiment, with the corrupted prompt differing by one fact.
- The restoration score divides recovered probability by the total gap, so 0.73 means one state closed 73 percent of the gap.
- Sweeping every layer and token position turns single scores into a visible circuit map.
- Minimal prompt pairs keep patched states natural, since mismatched prompts push later layers off distribution.
- Partial restoration is normal, because facts usually travel through several states rather than one.