Neural Tangent Kernel

Infinitely wide networks stop learning features and start doing kernel regression. The neural tangent kernel names that kernel, predicting training speed and which examples get learned first.

Kernel regression through two training points predicts 1.0 at the new input, the infinite-width picture of network training

Why Does This Exist?

Training dynamics resisted theory: millions of coupled parameters moving at once, and every explanation dissolved on contact with real networks. The infinite-width limit tames the chaos. As hidden layers widen, the network's behavior converges to kernel regression with a fixed kernel — the NTK — whose spectrum predicts convergence speed per direction and generalization in closed form.

For vision specifically, the convolutional NTK (CNTK) computes that kernel for conv architectures, letting researchers rank designs and study pooling and depth effects without training anything. This page covers only that residual: the kernel picture and its vision use. The general infinite-width theory belongs to its own owners (see the donor table), and the neighboring dynamics pages are generalization theory and loss landscapes.

The one idea you need first. In the lazy regime, weights barely move and the network behaves as its first-order Taylor expansion around initialization: $f(\theta) \approx f(\theta_0) + \nabla f \cdot (\theta - \theta_0)$ . Training becomes linear regression over fixed features $\nabla f(x)$ , and the kernel $\Theta(x, x') = \nabla f(x) \cdot \nabla f(x')$ governs everything: gradient descent converges along each eigen-direction at a rate set by its eigenvalue.

Think of It Like This

A frozen river versus carving new channels

Spring melt can reach the sea two ways: float down the frozen river's fixed channels (fast, predictable, same route every year) or carve new channels through the banks (slow, channel-carving, new route each time). Lazy training floats the frozen river: features fixed at initialization, only the readout weights move. Rich training carves banks: features themselves reshape.

The NTK maps the frozen river exactly — current speeds per channel (eigenvalues), arrival times, final destination. But it says nothing about carving, which is what finite vision networks spend much of training doing. Use the map where rivers stay frozen (very wide nets, small learning rates); expect surprises where banks reshape.

How It Actually Works

At infinite width the NTK stays constant during training, so function-space dynamics are linear: errors decay exponentially per eigen-component with rate proportional to the eigenvalue. Large-eigenvalue directions (smooth, low-frequency patterns) converge first — the kernel reason behind spectral bias. The CNTK extends this to convolutions by accounting for weight sharing and pooling in the kernel recursion, giving a deterministic number per architecture per dataset that correlates with trained accuracy well enough to screen designs.

Finite-width reality deviates exactly where feature learning matters: transfer, few-shot adaptation, and grokking-style phase changes all live outside the frozen river.

Worked example

Kernel regression on two points with kernel matrix $K = [[2, 1], [1, 2]]$ and targets $y = [1, -1]$ . Solve $\alpha = K^{-1}y$ : determinant $= 4 - 1 = 3$ , so $K^{-1} = \frac{1}{3}[[2, -1], [-1, 2]]$ . Then $\alpha_1 = (2 \times 1 + (-1) \times (-1))/3 = 3/3 = 1$ and $\alpha_2 = ((-1) \times 1 + 2 \times (-1))/3 = -3/3 = -1$ .

Predict a new input with kernel similarities $k^* = [1.5,\ 0.5]$ : $f = 1.5 \times 1 + 0.5 \times (-1) = 1.5 - 0.5 = 1.0$ . That is infinite-width training in one line: invert the kernel on training points, weight by similarity. Real networks approximate this when wide, and depart from it as features move.

Code

The point of this snippet is the kernel solve: two-by-two inversion plus similarity weighting reproduces the $1.0$ prediction.

1def solve_2x2(a: float, b: float, c: float, d: float, y1: float, y2: float) -> tuple:2    det = a * d - b * c  # never zero for a valid kernel matrix3    return ((d * y1 - b * y2) / det, (-c * y1 + a * y2) / det)4
5
6alpha = solve_2x2(2.0, 1.0, 1.0, 2.0, 1.0, -1.0)7print([round(a, 3) for a in alpha])  # -> [1.0, -1.0]8print(round(1.5 * alpha[0] + 0.5 * alpha[1], 3))  # -> 1.0

Watch Out For

Finite networks learn features the kernel cannot see

The NTK is frozen at initialization; real networks move their features, sometimes dramatically (early-layer Gabor filters emerge, they are not born). Explanations of transfer learning, grokking, or lottery tickets that invoke only the kernel miss the mechanism doing the work.

Check the regime before citing NTK results: tiny learning rates and huge widths stay lazy, while standard vision training with augmentation and schedules goes rich quickly. Most practical phenomena live in the rich regime the kernel cannot describe.

CNTK rankings are screening, not verdicts

CNTK correlates with trained accuracy across architectures but imperfectly: it ignores optimization choices, augmentation, and finite-width feature learning that decide close races. Promoting a design on CNTK alone has crowned architectures that training later dethroned.

Use CNTK to discard hopeless designs cheaply, then train the survivors. A kernel ranking is a filter for the search, not a replacement for it.

The Quick Version

Infinitely wide networks train as kernel regression with the fixed neural tangent kernel.
Eigenvalues set convergence speed per direction: smooth patterns first, details later.
The worked fixture solves a two-point kernel regression and predicts $1.0$ at the new input.
CNTK extends the picture to conv architectures for cheap design screening.
Finite networks learn features outside the kernel's frozen world, so check the regime first.
Kernel rankings filter candidates; training still delivers the verdict.