Gen AI

Sparse Autoencoders (SAEs)

LLMs store concepts by mixing them together into dense, unreadable vectors. Sparse Autoencoders untangle this mess, expanding one dense vector into thousands of isolated, readable concepts.

Sparse Autoencoders untangle dense, unreadable neural networks by expanding their hidden states into a massive dictionary of distinct, human-readable concepts.

Why Does This Exist?

For a long time, researchers trying to understand Large Language Models faced a massive roadblock called Polysemanticity.

If you looked at a single "neuron" (one dimension of the 4,096-dimensional hidden state), you would expect it to represent one thing. For example, Neuron #42 might light up when the model sees a dog.

But when researchers actually looked at Neuron #42, they found it lit up for dogs, the Eiffel Tower, C++ syntax, and the concept of sadness. The neuron was polysemantic—it meant many different things at once. Because neural networks are desperate to compress information, they pack multiple distinct concepts into a single dimension. This packing is mathematically known as Superposition.

Because of superposition, reading the brain of an LLM directly is impossible. It is a tangled mess of overlapping concepts. Sparse Autoencoders (SAEs) were invented by Anthropic in 2023 to solve this problem. They are secondary neural networks that "untangle" the LLM's brain, transforming the dense, overlapping mess into a massive dictionary of pure, separated concepts.

Think of It Like This

Think of the LLM's hidden state like a smoothie made of 10 different fruits.

If you look at a single drop of the smoothie, it doesn't taste like strawberries or bananas; it tastes like a polysemantic, untangled mess of everything.

A Sparse Autoencoder is a magical centrifuge. You pour the smoothie into it, and it separates the mixture back out into 10 distinct, pure glasses of juice. Once the juices are separated, you can clearly label them: "This is Apple," "This is Banana."

How It Actually Works

An SAE is trained separately after the LLM has already been trained.

1. The Bottleneck (The LLM)

The LLM's hidden state is typically around 4,096 dimensions. This is too small to store the millions of facts the LLM knows, which forces the LLM to entangle them.

2. The Expansion (The Encoder)

You train a simple, 1-layer neural network (the SAE) that takes the 4,096-dimensional vector and expands it into a massive, 100,000-dimensional vector.

Because this new vector is so huge, there is finally enough room to store every concept in its own dedicated lane. Neuron #12 can only mean Dog. Neuron #4,051 can only mean Eiffel Tower.

3. The Sparsity Penalty

If you just expand a vector, the network will keep everything entangled. To force the SAE to separate the concepts, you apply an L1 Sparsity Penalty during training.

This mathematical penalty forces the SAE to keep almost every number in the 100,000-D vector exactly at zero. For any given word, maybe only 10 out of the 100,000 neurons are allowed to "light up." Because it can only use a few neurons at a time, the SAE is forced to make each neuron represent a pure, highly specific concept.

4. The Reconstruction (The Decoder)

To ensure the SAE didn't destroy the data, it must compress the 100,000-D vector back down to 4,096 dimensions and recreate the original hidden state perfectly.

Why This is a Massive Breakthrough

Once you have a trained SAE, you can pipe any LLM hidden state through it and instantly get a perfectly readable list of concepts. You no longer have to guess what the LLM is thinking.

If the LLM generates a toxic response, you can look at the SAE output and see that Neuron #89,201 ("Malicious intent") lit up. Even better, combining SAEs with activation-steering, you can surgically go into the SAE, set Neuron #89,201 to zero, and the model will instantly stop being toxic.

Show Me the Code

Training an SAE requires massive amounts of data and compute, but using a pre-trained SAE to analyze an LLM is straightforward.

1import torch2import torch.nn as nn3
4# Assume 'sae_model' is a pre-trained Sparse Autoencoder for Layer 125# Assume 'llm_hidden_state' is a [1, 4096] vector from the LLM6
7def extract_pure_concepts(llm_hidden_state, sae_model, feature_dictionary):8    """9    Pipes an entangled LLM state through an SAE to extract readable concepts.10    """11    12    # 1. Pass the dense vector through the SAE encoder13    # This expands it to [1, 100000]14    sparse_vector = sae_model.encode(llm_hidden_state)15    16    # 2. Because of the L1 penalty, 99.9% of these values are exactly 0.017    # Find the indices of the few neurons that actually lit up18    active_indices = torch.nonzero(sparse_vector).squeeze()19    20    print("The model is currently thinking about:")21    22    # 3. Look up the human-readable labels for those specific neurons23    for idx in active_indices:24        activation_strength = sparse_vector[0, idx].item()25        concept_name = feature_dictionary[idx.item()]26        27        print(f"- {concept_name} (Strength: {activation_strength:.2f})")28
29# Output might look like:30# - Concept: "Golden Retrievers" (Strength: 4.2)31# - Concept: "Running outdoors" (Strength: 1.8)32# - Concept: "Joy/Happiness" (Strength: 0.5)

Watch Out For

The Labeling Bottleneck

An SAE mathematically isolates 100,000 pure concepts, but it doesn't give them human names. It just outputs "Neuron 4,051." To figure out what Neuron 4,051 means, researchers have to feed thousands of sentences into the LLM, see what triggers Neuron 4,051, and have an AI (like GPT-4) read those sentences and guess the label (e.g., "Oh, it only lights up when Eiffel Tower is mentioned!"). Labeling 100,000 neurons requires a massive, automated secondary pipeline.

The Quick Version

LLMs store information in "superposition", compressing many concepts into single neurons, making them a polysemantic, unreadable mess.
Sparse Autoencoders (SAEs) are secondary networks that expand the LLM's dense vectors into massive, sparse vectors (e.g., 100,000 dimensions).
A sparsity penalty forces the SAE to use very few neurons at a time, forcing each neuron to represent one single, pure, readable concept.
SAEs represent the current state-of-the-art in Mechanistic Interpretability, giving researchers a nearly complete dictionary of the thoughts occurring inside an LLM's brain.

Related concepts

The Logit Lens