Weight Initialization

Random weights break symmetry between units, but the wrong scale shrinks every layer's output toward zero or blows it up, and the right scale depends on the layer's width.

Across twenty identical layers a too-small starting scale drives the output toward zero, a too-large one drives it off the chart, and He scaling holds it level with the input

Why Does This Exist?

Build a 20-layer network to classify satellite images by crop type, set every weight to zero for a clean start, and training goes nowhere. Every unit in a layer computes the exact same thing, because all of them read the same inputs through the same weights. Every unit gets the exact same gradient too, so every unit updates to the exact same new value. Twenty layers, and the whole network behaves like one unit repeated many times. No amount of training breaks that symmetry, because nothing gives one unit a reason to end up different from its neighbour.

So the fix is obvious: start the weights random instead. That settles symmetry completely, but it opens a second problem, easy to miss until a network is actually deep. Draw every weight from a distribution with standard deviation 1.0, and layer twenty's output has magnitude around $10^{20}$ , off the scale of anything a float usefully represents. Shrink that standard deviation to 0.01 instead and layer twenty's output sits around $10^{-20}$ , indistinguishable from zero. Neither number carries signal about the crop in the picture, and neither is a training bug. Both are the compounding consequence of multiplying by the same wrong-sized random matrix twenty times running.

Think of It Like This

Twenty microphones in a relay, one gain knob per stage

A twenty-stage audio relay, each stage picking up the previous stage's output on a microphone and re-broadcasting it to the next. Every stage has one gain knob, and whoever built the relay set every knob to the same setting before testing it.

Turn every knob slightly too low, and the signal gets a little quieter at each stage. After twenty stages, a little quieter compounds into total silence. Nobody at the far end hears anything, even though each individual stage barely changed the volume. Turn every knob slightly too high instead, and the same compounding runs the other way: a whisper becomes a wall of feedback by stage twenty, drowning out everything.

There is exactly one gain setting where a signal fed in at normal volume comes out at roughly that same volume twenty stages later. Weight initialization is finding that setting before you switch the relay on, because listening at the far end and adjusting stage by stage afterwards is a much harder way to find it.

Where it stops: a relay knob is one number per stage, while a layer holds thousands of random weights whose variance is what matters. The analogy shows why small per-stage errors compound, not the $1/n$ arithmetic that sets the right scale. That arithmetic is below.

How It Actually Works

Why the same random weight, twenty times, does not cancel out

A layer computes $y = Wx$ . The entries of $W$ are drawn independently with variance $\sigma^2$ , and the entries of $x$ carry some variance $\text{Var}(x)$ , so each entry of $y$ is a sum of many independent random products. Variance of a sum of independent terms adds, so for a layer with $n$ inputs:

\text{Var}(y) \approx n \, \sigma^2 \, \text{Var}(x)

$n$ is how many inputs feed each unit, $\sigma^2$ the weight variance, $\text{Var}(x)$ the incoming activations' variance. If $n\sigma^2$ is not close to 1, this layer's output variance is scaled relative to its input, and that scaling multiplies in again at the next layer, and the next. Twenty layers means the mismatch compounds twenty times over, which is why a deviation that looks fine for one layer wrecks a deep stack.

Two fixes, tuned to two different activations

Xavier (Glorot) initialization picks $\sigma^2 = 1/n$ , which keeps $n\sigma^2 = 1$ and holds the variance roughly constant layer to layer. That is the right target for a linear layer or one followed by tanh, where the activation does not systematically shrink the signal.

He initialization doubles that to $\sigma^2 = 2/n$ , because ReLU zeroes out roughly half its inputs, every negative value becoming exactly zero, which by itself would halve the variance passing through. Doubling the weight variance compensates for that halving, so output variance after a ReLU layer matches the input instead of shrinking by half each time.

What this buys, and what it does not

Getting the scale right means signal and gradient magnitudes stay workable long enough for backpropagation to actually train the early layers. That is a large part of why deep networks became trainable at all with plain gradient descent, before other fixes existed. It is not a complete solution on its own: initialization sets the starting variance, and nothing stops it drifting as training proceeds and weights move away from their initial values. That drift, compounding layer after layer as training runs, is the deeper and more persistent version of the same problem. Once it shows up mid-training rather than at step zero it is called vanishing gradients, and batch normalization is the standard fix for the drift, layered on top of a sane initialization rather than replacing it.

Code

The same 20-layer ReLU stack, three initialization scales, watching the output's standard deviation survive, vanish, or explode.

1import numpy as np2
3
4def forward_std(std: float, width: int = 256, depth: int = 20) -> float:5    rng = np.random.default_rng(0)6    x = rng.normal(0.0, 1.0, size=(1, width))7    for _ in range(depth):8        w = rng.normal(0.0, std, size=(width, width))9        x = np.maximum(x @ w, 0.0)  # ReLU10    return float(x.std())11
12
13he_std = (2.0 / 256) ** 0.514print(f"{forward_std(0.01):.2e}")   # -> 8.77e-20, too small, vanished15print(f"{forward_std(1.0):.2e}")    # -> 8.77e+20, too large, exploded16print(f"{forward_std(he_std):.3f}")  # -> 0.743, He scaling, held near the input's std of 1.0

Same architecture, same random seed, same twenty layers. The only thing changing across the three runs is one number, the standard deviation the weights were drawn from.

Watch Out For

Copying a fixed standard deviation across a different width or depth

A std=0.01 that worked for a shallow, narrow prototype gets reused unchanged when the network grows to twenty layers, and training that used to converge now stalls or produces nan within the first few steps. The right scale depends on $n$ , the number of inputs to that layer. Xavier and He compute it from the layer's own shape rather than hard-coding one number, which is why they generalise across architectures a memorised constant does not.

Mismatching the initialization scheme to the activation function

He initialization on a tanh network, or Xavier on a ReLU network, runs without erroring and often trains adequately, which is exactly what makes the mismatch easy to miss. The factor-of-two gap between the schemes compensates for ReLU zeroing out half its inputs. Apply the ReLU-tuned variance to tanh and the signal runs slightly hot at every layer, an error visible only at real depth, long after a shallow prototype looked fine.

The Quick Version

Zero or identical initial weights leave every unit computing the same thing forever. Random weights are needed to break that symmetry.
The scale of the randomness matters as much as the randomness itself: variance compounds multiplicatively across layers.
Xavier initialization targets $\sigma^2 = 1/n$ for linear or tanh layers. He initialization targets $\sigma^2 = 2/n$ to compensate for ReLU zeroing out half its inputs.
Getting the scale wrong does not error. It silently vanishes or explodes the signal, worse the deeper the network goes.
Initialization sets the starting variance only. Drift during training is a related but separate problem that normalization layers address.