Core ML

Activation Patching

To prove a specific layer handles a specific task, you run a corrupted input that fails the task, then overwrite just that layer's activations with the clean ones. If the model suddenly gets the right answer, you found the circuit.

A corrupted prompt breaks the prediction, but overwriting a single hidden state with the clean state restores it—proving that specific state holds the missing concept.

Why Does This Exist?

If you want to know how a neural network represents a concept like "Paris is the capital of France", you could use a probing classifier (probing-classifiers). You train a small model to read the hidden states and see if it can predict the word "Paris".

But probing has a fatal flaw: it is purely correlational. Just because a layer knows about Paris doesn't mean the model actually uses that knowledge to generate the next word. The information might just be sitting there, inert.

Mechanistic interpretability requires causal proof. To prove a specific component is responsible for a behavior, you have to break the component and watch the behavior stop, or fix the component and watch the behavior return. Activation patching (also known as causal tracing or resample ablation) is the standard technique for doing exactly this.

Think of It Like This

Imagine trying to figure out which fuse in your house controls the kitchen lights.

Looking at a blueprint (the weights) is overwhelming. Probing is like putting a thermometer near the fuse box to see which fuses get warm when the kitchen lights are on—it's a hint, but not proof.

Activation patching is flipping the main breaker so the whole house goes dark (the corrupted run), and then manually bypassing one specific fuse with a battery (the patch). If the kitchen lights suddenly turn back on, you have absolute causal proof that you found the kitchen fuse.

How It Actually Works

An activation patching experiment requires three forward passes.

1. The Clean Run

You feed the model a clean prompt that triggers the behavior you want to study. Prompt: "The capital of France is" Output: " Paris"

During this run, you save every single hidden state and activation inside the network. You now have a complete snapshot of the model's internal memory when it is doing the task correctly.

2. The Corrupted Run

You feed the model a corrupted prompt that destroys the specific concept you are studying, while keeping the rest of the sentence structure identical. Prompt: "The capital of Rome is" Output: " Rome" (The model fails to output Paris).

3. The Patched Run

You run the corrupted prompt again, but this time you perform surgery during the forward pass.

At a specific layer, you pause the execution, throw away the corrupted activation, and replace it with the saved clean activation from Step 1. Then you let the model finish the forward pass.

If the model suddenly outputs " Paris", you have proven that the specific layer you patched is causally responsible for moving the "France" concept to the output.

Sweeping the Network

You don't just do this once. You run a loop over every single layer, and every single token position, applying the patch and recording how much it restored the probability of the correct answer.

When you plot the results as a heatmap (layers on the Y-axis, token positions on the X-axis), a "circuit" emerges. You will typically see that early layers process the raw token, middle layers recall the factual association, and late layers at the final token position decide to actually emit the word.

Watch Out For

The out-of-distribution trap. When you copy an activation from the clean run and paste it into the corrupted run, you are forcing the network into a state it would never reach naturally. The later layers in the network might react wildly to this Frankensteined hidden state, producing garbage outputs that you misinterpret as a lack of causal effect.

This is why you must design your clean and corrupted prompts to be nearly identical in length, grammar, and structure. The closer the two distributions are, the more trustworthy the patch.

The Quick Version

The Problem: Probing only shows correlation. We need to prove causality to map a neural network's internal circuits.
The Fix: Run a clean prompt and save the activations. Run a corrupted prompt, but swap in one clean activation. If the output is fixed, you found the circuit.
The Execution: By sweeping this patch across all layers and token positions, you can trace the exact flow of information through the transformer.
The Trap: If your corrupted prompt is too different from your clean prompt, the patched activations will crash the later layers.