Toy Models of Superposition

The Problem

Neural networks frequently display "polysemantic" neurons—individual neurons that respond to multiple, seemingly unrelated concepts (e.g., a single neuron firing for both "cats" and "cars"). This makes mechanistically interpreting these networks exceedingly difficult, as we cannot simply map one neuron to one clear, understandable feature. For a long time, the prevailing assumption was that models should ideally have orthogonal, interpretable features, and polysemanticity was just an artifact of flawed training or a lack of capacity.

The Idea

The authors proposed that polysemanticity is not a bug but a feature. They hypothesized that neural networks engage in "superposition": a strategy to represent more distinct, meaningful features than they have available dimensions. Because features in the real world are often "sparse" (they don't all occur at the same time), the network can compress many features into a lower-dimensional space by representing them as almost-orthogonal vectors. The interference between these vectors is manageable as long as the features rarely co-occur.

How It Works

The authors created "toy models"—highly simplified, small neural networks (often just a linear projection followed by a ReLU)—to study this phenomenon in isolation.

Feature Sparsity: They explicitly controlled how often features occurred in the synthetic training data.

Dimensionality vs. Features: They trained the network to autoencode inputs where the number of underlying features was strictly greater than the number of dimensions in the hidden layer.

Geometry of Superposition: By analyzing the weights of these toy models, they found beautiful, discrete geometric structures. When features were dense (common), the model assigned orthogonal basis vectors to them. As features became more sparse, the model began packing them into antipodal pairs, triangles, pentagons, and other regular polytopes in the hidden space, effectively squishing more features into fewer dimensions.

Why It Mattered

This paper provided a mathematically rigorous and visually intuitive explanation for polysemanticity. It proved that models are structurally incentivized to compress sparse features into superposition, fundamentally changing how the field of mechanistic interpretability views neural network representations. Instead of looking for single interpretable neurons, researchers realized they need to look for directions in the activation space.

What Came After

This insight directly inspired the use of Sparse Autoencoders (SAEs) as a tool for mechanistic interpretability. By training a sparse autoencoder on a model's hidden activations, researchers (at Anthropic, DeepMind, and OpenAI) successfully untangled these superposed representations, extracting thousands of clean, monosemantic features from complex models like Claude and GPT-4.