Feature Learning vs Lazy Training

Wide networks can either learn new features or freeze near their start and fit data linearly, and which regime you are in decides what depth is for.

Lazy training keeps hidden features almost fixed and fits the output linearly, while feature learning moves the features themselves to fit the data

Why Does This Exist?

Train a very wide network and something odd happens: the hidden features barely move, yet the loss still falls. The network behaves like a fixed kernel machine, fitting data with its random starting features instead of learning better ones. Train the same architecture with a different scaling and the features swing substantially, discovering structure the initialization lacked.

Both behaviors are real, and confusing them causes real errors. Kernel-regime analysis predicts training well but misses the feature discovery that makes depth valuable on structured data. This page separates the lazy (kernel) regime from the rich (feature-learning) regime, names the scaling that picks between them, and gives a two-number diagnostic you can run on your own model. Kernel details live with NTK-adjacent material in optimization theory; the geometry of the surface itself is on the loss surface page.

The one idea you need first

Write the network as $f(x; \theta)$ with parameters $\theta$ starting at $\theta_0$ . Lazy means $\theta$ stays close to $\theta_0$ while the outputs still fit: the first-order Taylor expansion around $\theta_0$ stays accurate. Rich means $\theta$ travels far and that linear approximation breaks.

Think of It Like This

Tuning a guitar two ways

One guitarist never touches the tuning pegs. They play every song with the factory tuning, pressing strings harder or softer to fake the missing notes. With enough strings this works surprisingly well, and every adjustment is small and predictable.

The other guitarist retunes between songs. The pegs travel far, the instrument becomes a different guitar per song, and the factory setting is forgotten. This is riskier and more powerful: genuinely new chords become playable, but the setup no longer resembles where it started.

Lazy training is the first guitarist: fixed random features, linear fitting on top. Feature learning is the second: the hidden representation itself moves. Wide nets default to the first unless the scaling invites the second.

How It Actually Works

The switch is parameterization, meaning how weights scale with width $m$ and how large the learning rate is.

Lazy (NTK-style) scaling sets output weights of order $1/\sqrt{m}$ with an order-one learning rate. Each of the $m$ units must move only $O(1/m)$ to change the output by $O(1)$ , so relative movement shrinks as $1/\sqrt{m}$ . As $m$ grows, hidden features freeze and training follows the fixed kernel $K(x, x')$ defined by gradients at initialization. Predictions equal kernel regression with that kernel.

Rich (mean-field-style) scaling sets output weights of order $1/m$ with a larger effective step. Units must move $O(1)$ each to fit, so features travel, neurons specialize, and the kernel itself evolves. The same width that froze features under lazy scaling now learns them.

Worked example

A one-hidden-layer ReLU net with width $m = 1{,}000$ fits $50$ points. Under lazy scaling, total output change needed is $O(1)$ , shared across $1{,}000$ units, so each unit's weights move about $1/1{,}000 = 0.001$ in absolute terms, or relative movement $0.001 \times \sqrt{1{,}000} \approx 0.03$ (about $3\%$ ). The kernel at the end matches the kernel at the start within a few percent. Under rich scaling with $1/m$ output weights, each unit must contribute $1/50$ of the fit on the points it serves, moving $O(0.1)$ or more: features swing visibly, hidden activations at the end correlate weakly (say $0.4$ ) with their starting values. Same data, same width, different learned object.

A two-number diagnostic

After training, compute (1) the relative movement $\lVert \theta - \theta_0 \rVert / \lVert \theta_0 \rVert$ and (2) the kernel change $\lVert K_{\text{end}} - K_{\text{start}} \rVert / \lVert K_{\text{start}} \rVert$ on a sample batch. Both below about $0.1$ : lazy. Either above about $0.5$ : rich. Between is the honest middle where most practical nets live.

Watch Out For

Applying kernel predictions to a feature-learning net

NTK analysis says a wide net generalizes like its fixed kernel. That holds only while the kernel stays fixed. If your features move (diagnostic above $0.5$ ) and you still quote kernel bounds, the numbers describe a different machine than the one you trained. Check movement before citing the kernel.

Assuming width alone buys feature learning

Widening a lazily-scaled net freezes features harder, since movement per unit shrinks with $m$ . Teams that scale width expecting richer features and keep the same parameterization get the opposite: a better kernel machine with identical frozen features. Change the scaling and step size, not just the width.

The Quick Version

Lazy training freezes hidden features near initialization and fits outputs linearly, matching a fixed kernel.
Rich training moves features substantially, letting neurons specialize to the data.
Output scaling of $1/\sqrt{m}$ with standard steps gives lazy; $1/m$ with large steps gives rich.
At width $1{,}000$ , lazy units move about $3\%$ relative while rich units move tens of percent.
Diagnose with relative parameter movement and kernel change: both under $0.1$ is lazy, over $0.5$ is rich.
Widening a lazy net freezes features further; only rescaling switches regimes.