Visual explainer
Vanishing and Exploding Gradients
Trace how products of local derivatives shrink or amplify gradients, then separate what identity skips and norm clipping can actually fix.
Backward sensitivity can shrink or grow as local derivatives multiply. The picture follows a unit gradient through three complete scalar factors of .25: 1, .25, .0625 and .015625. All bar heights use the same linear scale; the ring selects the final tiny bar without enlarging it. The horizontal axis counts backward steps, opposite to forward computation through the network.
Small gradients can slow earlier parameter updates. Large gradients can make updates unstable at a given learning rate. Neither follows from depth alone: the actual factors, directions and paths matter.
Count The Whole Layer
The plotted function is , with zero bias. Its output axis runs from zero toward one; the dashed tangent selects . Sigmoid's activation derivative there is .25, but the full input derivative is . In general, for , the factor is . The curve uses sampled exact sigmoid values with line interpolation; the tangent uses the analytic derivative. Glorot and Bengio examine how activations, weights and layer Jacobians affect propagation.
Follow Ten Small Factors
For the declared toy chain, . At five backward steps the magnitude is .0009765625; at ten it is .0000009536743. The vertical axis is logarithmic: equal vertical distances mean equal ratios, not equal differences. Horizontal ticks count backward factors, with white points at zero and five. This value is small and finite, rather than automatically zero in machine arithmetic. The assumption is about complete local factors, not every network that uses sigmoid.
Follow Ten Large Factors
Change every complete factor to 1.5. Then , giving 7.59375 at five backward steps and 57.665039 at ten. This plot deliberately uses a linear vertical scale from zero to 60, unlike the previous logarithmic view. Integer-step values are connected by straight segments. A gradient of 57.7 may make an update too large, but does not itself imply overflow or NaN. Pascanu et al. analyze gradient growth and decay, including recurrent networks unrolled through time.
Add An Identity Route
Read this reverse-only graph from right to left. For with incoming sensitivity 1, the identity route contributes 1 and a branch derivative contributes .25. They sum to 1.25 at the shared input. The left plus represents accumulated gradients, while the right plus is the forward addition viewed backward. A direct route helps transport a signal, but its contribution is only one term in the total. Identity Mappings in Deep Residual Networks explains the architecture conditions behind this route.
Clip The Norm
The axes are gradient components. The dashed vector has Euclidean norm 5. A cap of 1 rescales it to , exactly on the unit circle. Both axes share the same units, and the solid arrow stays on the original ray. Norm clipping uses for nonzero gradients and leaves zero unchanged. Vectors already within the cap remain unchanged. Clipping can bound a spike; it cannot revive a tiny signal or repair an invalid numerical value already produced.
A Local Gate Can Close
At preactivation , ReLU outputs zero and has local slope zero. The highlighted horizontal segment and stop bar show the blocked local route. Positive preactivations have slope one, but weights still affect the complete layer derivative. A unit inactive across all relevant samples may stop receiving a data-loss update through this route; a single inactive sample does not establish that the whole network has stopped learning.
Check The Whole Product
This variant starts at 1, multiplies by .25, then by 4. The shared-scale bars return to height 1 because . Individual factors can look alarming while their product is moderate. Conversely, repeated modest changes can compound. This scalar example does not guarantee stability for changing nonlinear states or matrix products.
Track Directions Too
Replace scalar factors with a two-dimensional Jacobian . Backward multiplication uses , which equals here. Starting with gradient , one step produces and two produce . The horizontal component shrinks while the vertical component grows. Dashed/circle, solid/square and mint/diamond endpoints distinguish the three vectors on equal-unit axes. Their norms are approximately 1.414, 2.016 and 4.0005: a growing total norm can hide a disappearing component. Real Jacobians can also rotate directions, so one average scalar factor is not a complete diagnosis.
Where It Breaks
For a residual block with incoming gradient 1 and , the signed contributions are +1 and −1. Equal opposing arrows cancel at the marked zero: . Even an identity skip does not guarantee a nonzero total gradient. Clipping this zero leaves it zero. Inspect layer gradients, activations and data before choosing initialization, normalization, activation or architecture changes; no one fix guarantees healthy training.
The Quick Version
- Complete layer derivatives include weights and activation slopes.
- Products can shrink or amplify backward sensitivity.
- Logarithmic and linear plots answer different magnitude questions.
- Norm clipping caps spikes without reviving tiny gradients.
- Different vector directions can shrink and grow simultaneously.
- Identity contributions can cancel against residual branches.