Exploding Gradients

The same layer-by-layer multiplication behind vanishing gradients runs the other way when local derivatives stay above one, and the gradient grows exponentially with depth instead of shrinking toward zero.

A local factor of 1.5 at every layer compounds from 7.59 at layer 5 to 191751.06 at layer 30, showing growth that mirrors vanishing gradients in reverse

Why Does This Exist?

Training a recurrent network on long transcripts can run smoothly for hours, then one step jumps from a normal loss of 2.1 to nan, and every step after stays nan. Nothing crashed. The gradient at that step was correctly computed, but it was so large that the update threw the weights where the loss cannot be evaluated.

This is the mirror image of vanishing and exploding gradients. Backpropagation multiplies local derivatives across every layer on the way back. When those factors sit below one, the product shrinks toward zero. When they sit above one, the same multiplication compounds upward, just as exponentially.

This page covers the growth case: why the product explodes, where it hits hardest, and the direct fix. The fix that caps the damage lives in gradient clipping. The recurrent structure that turns long sequences into deep stacks lives in vanilla RNNs.

Think of It Like This

A rumor that grows at every retelling

One person hears that a shipment is a little late and repeats it as somewhat late. The next person hears somewhat and repeats quite late. The next says severely late. After thirty retellings a mild delay has become a catastrophe, not because anyone lied outright, but because each small amplification multiplied across the chain.

That is the mechanism: no single layer looks unreasonable at 1.5, but thirty of them multiplied turn a normal signal into a number the optimizer cannot safely use.

Where it stops: people can pause and check the original message. Layers cannot. A network has no common sense brake, so you must add the brake yourself with clipping or better initialization.

How It Actually Works

Call the loss $L$ and call $z_t$ the pre-activation at layer $t$ . Backpropagation carries the gradient backward through the chain rule product:

\frac{\partial L}{\partial z_1} = \frac{\partial L}{\partial z_T} \prod_{t=2}^{T} \frac{\partial z_t}{\partial z_{t-1}}

Here $\partial L / \partial z_T$ is the gradient at the output end, and each $\partial z_t / \partial z_{t-1}$ is one local factor, meaning how much layer $t$ amplifies a change from the layer before. When every local factor sits near 1.5, typical of weights started with too large a variance and no dampening structure, the product is a pure power of 1.5.

Worked example: one factor of 1.5 across thirty layers

Start from a gradient of 1.0 at the output and multiply by 1.5 at each step back:

1.5^{5} = 7.59375 \approx 7.59

1.5^{10} = 57.6650390625 \approx 57.67

1.5^{20} = 3325.256730079651 \approx 3325.26

1.5^{30} = 191751.0592328841 \approx 191751.06

By layer 30 the gradient is about 191,751 times its starting size. The shape matches vanishing gradients exactly, only inverted: one curve collapses toward zero, this one climbs toward overflow. That is the diagram headline, with four bars at 7.59, 57.67, 3325.26 and 191751.06 on a log scale.

Where it hits hardest

Vanilla RNNs reuse the same weight matrix at every time step of backpropagation through time, so a 30 step sequence is a 30 layer stack. If that matrix stretches vectors, every step stretches again. Poor weight initialization with spectral norm above one causes the same growth in any deep stack, independent of recurrence.

Why the failure is loud

Vanishing gradients fail quietly: loss falls slowly and early layers stall. Exploding gradients fail loudly: one oversized update overflows to inf or nan within a step or two. Log the gradient norm before the update and you usually see it climbing steeply for several steps, not spiking from nowhere.

Code

The point is the check: rebuild the four depths from the worked example with plain powers and confirm the same four numbers.

1base: float = 1.52g5: float = base**53g10: float = base**104g20: float = base**205g30: float = base**306
7print(f"{g5:.2f} {g10:.2f} {g20:.2f} {g30:.2f}")  # -> 7.59 57.67 3325.26 191751.06

The printout matches the hand computation: 7.59 at depth 5, 57.67 at depth 10, 3325.26 at depth 20 and 191751.06 at depth 30. Change the base from 1.5 to 0.5 and the same code shows the vanishing mirror, collapsing toward zero instead.

Watch Out For

Reading a nan loss as a code bug first

A nan halfway through healthy training reads like an indexing slip or a bad custom loss. Logging the gradient norm for the steps before the nan tells the two apart.

Symptom: loss 2.1 then nan, with gradient norm climbing steeply over several steps before the break. Fix: treat it as magnitude first. Add gradient clipping or lower the learning rate, and only review code if the norm was flat right up to the nan.

Clipping every step without asking why

Clipping after one nan and never checking how often the clip fires hides whether the cause was one outlier batch or a permanently unstable setup.

Symptom: training runs only because the clip fires on most steps, so the effective step size is far below the configured rate. Fix: count clip triggers per epoch. Rare triggers mean keep clipping. Routine triggers mean fix the learning rate or weight initialization underneath.

The Quick Version

Exploding gradients are the mirror of vanishing gradients: the same chain rule product, with local factors above one instead of below.
A steady factor of 1.5 grows from 7.59 at layer 5 to 57.67 at layer 10, 3325.26 at layer 20 and about 191751.06 at layer 30.
Recurrent networks are the textbook case because one reused matrix compounds across every time step.
The failure is loud: overflow to nan within a step or two, usually after a visibly climbing gradient norm.
Gradient clipping bounds the damage per step without explaining why the growth happened.