Weight Initialization
The scheme for picking a network's starting weights, chosen so activations and gradients keep a stable scale across layers instead of collapsing or exploding.
Think of It Like This
Like tuning an acoustic guitar before a concert: setting the strings to the right starting tension so notes don't immediately screech or buzz.
Every weight starts at a small random number, never zero, because identical weights would compute identical gradients forever and no unit would ever specialize. How large that random number should be is the actual question, and it depends on how many inputs feed the unit: too small and every layer's output shrinks toward zero, too large and it blows up.
Xavier initialization balances the variance for a linear or tanh unit; He initialization doubles that variance to account for ReLU zeroing out half the values. Both scale down as the layer gets wider, which is the part people forget when they hand-pick a single fixed standard deviation for every layer in a network.