Core ML

Exploding Gradients

The same layer-by-layer multiplication behind vanishing gradients runs the other way when local derivatives are consistently above one, and the gradient grows exponentially with depth instead of shrinking toward zero.

A local derivative of 1.5 at every layer compounds to a gradient nearly 145,000 times larger by layer thirty — the mirror image of vanishing gradients, growing instead of shrinking

Why Does This Exist?

Training a recurrent network on long customer-service transcripts runs smoothly for hours, and then, without any change to the code or the data, one step's loss jumps from a normal-looking 2.1 to nan, and every step after it is nan too. Nothing crashed. The gradient computed at that step was a real, correctly-computed number — it just happened to be enormous, large enough that the resulting parameter update threw the weights somewhere the loss function can't evaluate.

This is vanishing gradients' mirror image, built from the exact same mechanism running in the opposite direction. Backpropagation multiplies local derivatives together across every layer on the way back; vanishing gradients is what happens when those local derivatives are consistently below one and the product shrinks toward zero. Exploding gradients is what happens when they're consistently above one instead — the same repeated multiplication, compounding upward rather than down, and just as exponential either way.

Think of It Like This

A rumor that gets exaggerated at every retelling instead of softened

Picture the whispered-message chain from vanishing gradients, except this time each person doesn't quiet the message down — they exaggerate it. Person 30 hears "the shipment is a little late" and passes it on as "the shipment is somewhat late." Person 29 hears that and passes on "the shipment is quite late." By the time it reaches person 1, a mild delay has become an all-out catastrophe, not because anyone lied outright, but because each person's small, individually reasonable amplification compounded across thirty retellings into something wildly disproportionate to the original message.

That's exploding gradients: no single layer's amplification looks unreasonable on its own, but thirty or more of them multiplied together turns a normal gradient into a number the optimizer can't safely act on.

How It Actually Works

The same product, running the other way

Vanishing gradients showed the chain-rule product for a deep stack:

\frac{\partial L}{\partial z_1} = \frac{\partial L}{\partial z_T} \prod_{t=2}^{T} \frac{\partial z_t}{\partial z_{t-1}}

Exploding gradients is exactly this product, except the local derivatives $\partial z_t / \partial z_{t-1}$ are consistently greater than 1 instead of less than 1. A local derivative around 1.5, typical of weights initialized with too large a variance or an architecture without any dampening structure, compounds exponentially: thirty layers with that factor multiply out to roughly $1.5^{30}$ , on the order of $10^5$ — a gradient a hundred thousand times larger than a healthy one, computed correctly, and disastrous the moment it reaches the optimizer.

Where it's most likely to appear

Vanilla RNNs are the textbook case, for the identical structural reason they're the textbook case for vanishing gradients: the same weight matrix multiplies in at every time step of backpropagation through time, so a long sequence turns into a very deep effective stack, and whichever direction that weight matrix pushes — shrinking or growing — compounds across every step of the sequence. Poor weight initialization that leaves weight matrices with a spectral norm consistently above 1 is the other common cause, independent of architecture.

Why this failure is louder than vanishing gradients

Vanishing gradients is quiet: training runs, loss falls slowly, early layers just don't learn, and nothing obviously errors. Exploding gradients is loud: a parameter update large enough to overflow typically produces nan or inf within one or two steps, which is easy to notice but easy to misdiagnose as a bug in the code rather than a property of the gradient's magnitude. Checking the gradient norm directly, right before the step that produces nan, usually shows it climbing steeply over the preceding several steps rather than appearing out of nowhere.

The direct fix

Gradient clipping caps the gradient's magnitude before it reaches the optimizer, which addresses the immediate symptom without requiring a structural change to the architecture. It's a bound on the damage, not a fix for why the gradient grew large in the first place — the same way clipping doesn't explain why a batch happened to produce an unusually large gradient, only limits how far that gradient can push the weights in one step.

Show Me the Code

Thirty layers with a local derivative consistently above one, and the gradient's magnitude at four depths — the exact mirror of the vanishing-gradients calculation, with the multiplier flipped above 1.

1import numpy as np2
3rng = np.random.default_rng(0)4local = rng.normal(1.5, 0.1, size=30)  # local derivative consistently greater than 15print(f"mean local derivative: {local.mean():.4f}")  # -> 1.48796
7g = 1.08for depth, factor in enumerate(local, start=1):9    g *= factor10    if depth in (5, 10, 20, 30):11        print(f"gradient after {depth:2d} layers: {g:.2e}")12# -> gradient after  5 layers: 7.68e+0013# -> gradient after 10 layers: 6.03e+0114# -> gradient after 20 layers: 2.51e+0315# -> gradient after 30 layers: 1.44e+05

By layer thirty the gradient is nearly 145,000 times its starting value — the identical exponential shape as vanishing gradients' collapse, just growing instead of shrinking. Neither curve is linear; both are the same repeated multiplication, running in opposite directions.

Watch Out For

Treating a NaN loss as a code bug before checking the gradient norm

A nan appearing partway through an otherwise-healthy training run reads like a bug — an indexing error, a division by zero somewhere in a custom loss function. Before debugging the code, log the gradient norm for the steps immediately before the nan appeared; a norm climbing steeply over several steps, rather than a single freak spike, is the signature of exploding gradients, not a logic error. The fix in that case is clipping or a lower learning rate, not a code review.

Clipping the gradient without checking whether it's a rare event or a routine one

Adding gradient clipping after seeing a nan and never checking how often the clip actually triggers leaves an important question unanswered: is the model hitting one rare outlier batch, or is every single step's gradient over the threshold? The second case means the underlying instability — usually the learning rate or initialization — is the real problem, and clipping on every step is silently running the model at a lower effective learning rate than configured, not fixing the cause.

The Quick Version

Exploding gradients is vanishing gradients' mirror image: the same chain-rule product, but with local derivatives consistently above one instead of below.
The growth is exponential in depth, exactly as the shrinkage is — thirty layers at a factor of 1.5 compounds to roughly a hundred-thousand-fold increase.
Recurrent networks are especially prone to it, for the same structural reason they're prone to vanishing gradients: one weight matrix reused at every time step.
It's loud rather than quiet — usually producing nan within a step or two, versus vanishing gradients' silent failure to learn.
Gradient clipping is the direct fix, bounding the damage without addressing why the gradient grew large in the first place.