Core ML

Weight Initialization

Random weights break symmetry between units, but the wrong scale shrinks every layer's output toward zero or blows it up — the right scale depends on the layer's width.

Twenty identical layers, three initial weight scales: too small and the output collapses toward zero by layer twenty, too large and it explodes, and He scaling holds it roughly level

Why Does This Exist?

Build a 20-layer network to classify satellite images by crop type, set every weight to zero for a clean start, and training goes nowhere. Every unit in a layer computes the exact same thing, because they all read the same inputs through the same weight and bias. Every unit gets the exact same gradient too, so every unit updates to the exact same new value. Twenty layers, and the whole network behaves like one unit repeated many times — no amount of training breaks that symmetry, because nothing gives one unit a reason to end up different from its neighbor.

So the fix is obvious: make the starting weights random instead. That solves symmetry completely — but it opens a second problem, easy to miss until a network is actually deep. Draw every weight from a distribution with standard deviation 1.0, feed data through the crop classifier, and layer twenty's output has magnitude around $10^{20}$ , off the scale of anything a float usefully represents. Shrink that standard deviation to 0.01 instead and layer twenty's output is around $10^{-20}$ — indistinguishable from zero. Neither number carries signal about the crop in the picture, and neither is a training bug. Both are the compounding consequence of multiplying by the same wrong-sized random matrix twenty times running.

Think of It Like This

Twenty microphones in a relay, one gain knob per stage

A twenty-stage audio relay, each stage picking up the previous stage's output on a microphone and re-broadcasting it to the next. Every stage has one gain knob, and whoever built the relay set every knob to the same setting before testing it.

Turn every knob down slightly too low, and the signal gets a little quieter at each stage. After twenty stages in a row, "a little quieter" compounds into total silence — nobody at the far end hears anything, even though each individual stage barely changed the volume. Turn every knob up slightly too high instead, and the same compounding runs the other way: a whisper becomes a wall of feedback by stage twenty, drowning out everything.

There's exactly one gain setting where a signal fed in at a normal volume comes out at roughly that same volume twenty stages later. Weight initialization is finding that one setting before you turn the relay on, because listening at the far end and adjusting stage by stage, after the fact, is a much harder way to find it.

How It Actually Works

Why the same random weight, twenty times, doesn't cancel out

A layer computes $y = Wx$ : $W$ 's entries are drawn independently with variance $\sigma^2$ , and $x$ 's entries carry some variance $\text{Var}(x)$ , so each entry of $y$ is a sum of many independent random products. Variance of a sum of independent terms adds, so for a layer with $n$ inputs:

\text{Var}(y) \approx n \, \sigma^2 \, \text{Var}(x)

$n$ is how many inputs feed each unit, $\sigma^2$ the weight variance, $\text{Var}(x)$ the incoming activations' variance. If $n\sigma^2$ isn't close to 1, this layer's output variance is scaled relative to its input, and that scaling multiplies in again at the next layer, and the next — twenty layers means the mismatch compounds twenty times over, why a deviation that looks fine for one layer wrecks a deep stack.

Two fixes, tuned to two different activations

Xavier (Glorot) initialization picks $\sigma^2 = 1/n$ , which keeps $n\sigma^2 = 1$ and holds the variance roughly constant layer to layer — the right target for a linear layer or one followed by tanh, where the activation doesn't systematically shrink the signal.

He initialization doubles that to $\sigma^2 = 2/n$ , because ReLU zeroes out roughly half its inputs — every negative value becomes exactly zero — which by itself would halve the variance passing through. Doubling the weight variance compensates for that halving, so output variance after a ReLU layer matches the input instead of shrinking by half each time.

What this buys, and what it doesn't

Getting the scale right means signal and gradient magnitudes stay workable long enough for backpropagation to actually train the early layers — a big part of why deep networks became trainable at all with plain gradient descent, before other fixes existed. It's not a complete solution on its own: initialization sets the starting variance, and nothing stops it drifting as training proceeds and weights move away from their initial values. That drift, compounding layer after layer as training runs, is the deeper, more persistent version of the same problem — vanishing gradients is what it's called once it shows up mid-training rather than at step zero, and batch normalization is the standard fix for the drift, layered on top of a sane initialization rather than replacing it.

Show Me the Code

The same 20-layer ReLU stack, three initialization scales, watching the output's standard deviation survive, vanish, or explode.

1import numpy as np2
3
4def forward_std(std: float, width: int = 256, depth: int = 20) -> float:5    rng = np.random.default_rng(0)6    x = rng.normal(0.0, 1.0, size=(1, width))7    for _ in range(depth):8        w = rng.normal(0.0, std, size=(width, width))9        x = np.maximum(x @ w, 0.0)  # ReLU10    return float(x.std())11
12
13he_std = (2.0 / 256) ** 0.514print(f"{forward_std(0.01):.2e}")   # -> 8.77e-20 — too small, vanished15print(f"{forward_std(1.0):.2e}")    # -> 8.77e+20 — too large, exploded16print(f"{forward_std(he_std):.3f}")  # -> 0.743 — He scaling, held near the input's std of 1.0

Same architecture, same random seed for the data, same twenty layers — the only thing that changes across the three runs is one number, the standard deviation the weights were drawn from.

Watch Out For

Copying a fixed standard deviation across a different width or depth

A std=0.01 that worked for a shallow, narrow prototype gets reused unchanged when the network grows to twenty layers, and training that used to converge now stalls or produces nan within the first few steps. The right scale depends on $n$ , the number of inputs to that layer — Xavier and He compute it from the layer's own shape rather than hard-coding one number, which is why they generalize across architectures a memorized constant doesn't.

Mismatching the initialization scheme to the activation function

He initialization on a tanh network, or Xavier on a ReLU network, runs without erroring and often trains adequately — which is exactly what makes the mismatch easy to miss. The factor-of-two gap between the schemes compensates for ReLU zeroing out half its inputs; apply the ReLU-tuned variance to tanh and the signal runs slightly hot at every layer, an error visible only at real depth, long after a shallow prototype looked fine.

The Quick Version

Zero or identical initial weights leave every unit computing the same thing forever — random weights are needed to break that symmetry.
The scale of the randomness matters as much as the randomness itself: variance compounds multiplicatively across layers.
Xavier initialization targets $\sigma^2 = 1/n$ for linear or tanh layers; He initialization targets $\sigma^2 = 2/n$ to compensate for ReLU zeroing out half its inputs.
Getting the scale wrong doesn't error — it silently vanishes or explodes the signal, worse the deeper the network goes.
Initialization sets the starting variance only; drift during training is a related but separate problem that normalization layers address.