Vanishing and Exploding Gradients

Backprop multiplies one derivative per layer, so values below one shrink toward zero and values above one blow up. The product decides whether early layers learn at all.

Each layer backward multiplies the gradient by one local derivative, so thirty steps near 0.05 leave the earliest layer about 9.31e-40 while steps near 1.5 explode

Why Does This Exist?

Train a 30 layer sigmoid network on digits and the last layers learn while the first ten barely move. Weights sit near their random start the whole run. It is not a code bug. Backpropagation computes exactly what the math asks. That number is genuinely tiny by the time it arrives early.

The cause is the chain rule. The gradient at one layer equals the gradient from the layer after, times that layer own local derivative. Chain thirty multiplications with typical factors below one, and the product shrinks every step. Sigmoid derivative peaks at exactly $0.25$ and falls fast away from zero. Thirty small numbers multiplied do not give thirty times smaller. They compound downward, the same way interest compounds upward.

This page covers both directions of the same multiplication. Below one it vanishes, above one it explodes. Fixes live in activation functions, weight initialization, and residual connections.

Think of It Like This

A whisper chain of thirty people each quieter than before

Thirty people stand in a line, passing a message from the far end to you. Each repeats at 80 percent of the volume they heard, not from carelessness, just how each whispers.

Person 30 hears it clearly and passes 80 percent on. Person 29 passes 80 percent of that. By person 1 the volume has been multiplied by $0.8$ twenty nine times, which is about $0.002$ , so you hear near silence although everyone did their job.

Nobody made an error. Compounding alone is the problem. Flip the number above one and the same chain shouts. A gradient through many layers behaves the same way, shrinking or growing one factor at a time.

Where it stops. People could simply speak up. Layers cannot choose their derivatives freely. The derivatives come from weights, inputs, and the bend in use.

How It Actually Works

The multiplication made explicit

For a chain of layers the gradient for an early preactivation $z_1$ is a product of every local derivative to the output:

\frac{dL}{dz_1} = \frac{dL}{dz_T} \prod_{t=2}^{T} \frac{dz_t}{dz_{t-1}}

Here $L$ is the loss, $z_T$ is the final preactivation, $T$ is depth, and each factor is one layer local derivative. For sigmoid with $s = 1 / (1 + exp(-z))$ the factor is $s(1-s)$ , which peaks at $0.25$ when $z = 0$ and nears zero far from zero. That flat region is called saturation. Once units sit there, every layer below pays for it.

Worked example

Fix each factor at $0.05$ , a realistic saturated value, start from $1.0$ , and multiply. After 5 layers the gradient is $3.13e-07$ . After 10 it is $9.77e-14$ . After 20 it is $9.54e-27$ . After 30 it is $9.31e-40$ . Ten more layers cost far more orders than the first ten did, because compounding steepens with depth.

The mirror uses $0.25$ and $1.5$ . Ten sigmoid peaks give $0.25^{10}$ , which is about $9.54e-07$ , near $0.000001$ . Ten growth steps give $1.5^{10}$ , which is about $57.67$ . One side freezes early layers, the other throws updates that overflow and turn to missing values. The diagram above draws the shrinking side with the $9.31e-40$ endpoint, and the code below prints the same four depths.

Three fixes and what each targets

Better weight initialization keeps preactivations out of saturation at the start, so derivatives begin near $0.25$ rather than near zero. Non saturating bends remove shrinkage for firing units, since a firing rectified unit has derivative exactly $1$ . Residual connections change the structure itself. The added skip path carries gradient past a layer derivative, so one near zero factor no longer blocks everything below it. Clipping caps the exploding side by rescaling an oversized gradient norm back to a ceiling such as $1.0$ .

Code

Thirty fixed factors and the magnitude at four depths, straight from the product above.

1def shrink(depth: int, factor: float) -> float:2    g = 1.0  # start at the output end3    for _ in range(depth):4        g *= factor  # one local derivative per layer back5    return g6
7
8for d in [5, 10, 20, 30]:9    print(d, f"{shrink(d, 0.05):.2e}")10# -> 5 3.13e-0711# -> 10 9.77e-1412# -> 20 9.54e-2713# -> 30 9.31e-40

Ten layers already cost seven orders. By thirty the earliest gradient is smaller than a 32 bit float can hold apart from zero, so its update does effectively nothing.

Watch Out For

Blaming data or learning rate for early layers that never move

Symptom: loss falls, late layers clearly learn, and the fix tried is a higher learning rate or more data. Nothing moves early. Any single global rate sane for the last layer is far too small for the first when magnitudes span tens of orders.

Fix: check per layer gradient magnitudes directly rather than only the loss. A healthy stack stays within a couple orders across depth. A vanishing stack spans tens of orders, worse the earlier the layer. Fix the multiplication, not the data.

Assuming rectified units alone end the problem

Symptom: every bend switches to rectified linear, and the team treats depth as free. Then some units go permanently negative, their derivative is exactly $0$ , and whole paths block again as dead units.

Fix: keep rectified units as the default plus clipping, then add initialization care and skip paths once depth passes about ten layers. Treat dead share as a metric to watch, not a solved past issue.

The Quick Version

Backprop multiplies one local derivative per layer across the backward path.
Sigmoid peaks at $0.25$ and nears zero when saturated, so deep stacks shrink exponentially.
Fixed $0.05$ gives $3.13e-07$ at 5 layers, $9.77e-14$ at 10, $9.54e-27$ at 20, and $9.31e-40$ at 30.
Ten $0.25$ steps land near $0.000001$ , while ten $1.5$ steps land near $57.67$ and explode.
Better start values, rectified bends, skip paths, and clipping each attack a different piece of the same product.