Skip to content
AI360XpertBeta
Core ML

Visual explainer

Vanishing and Exploding Gradients

Trace how products of local derivatives shrink or amplify gradients, then separate what identity skips and norm clipping can actually fix.

Quarter factors rapidly weaken backward sensitivity.
Quarter factors rapidly weaken backward sensitivity.

Backward sensitivity can shrink or grow as local derivatives multiply. The picture follows a unit gradient through three complete scalar factors of .25: 1, .25, .0625 and .015625. All bar heights use the same linear scale; the ring selects the final tiny bar without enlarging it. The horizontal axis counts backward steps, opposite to forward computation through the network.

Small gradients can slow earlier parameter updates. Large gradients can make updates unstable at a given learning rate. Neither follows from depth alone: the actual factors, directions and paths matter.

Count The Whole Layer

A weighted sigmoid's full slope differs from its activation slope.
A weighted sigmoid's full slope differs from its activation slope.

The plotted function is h=σ(2x)h=\sigma(2x), with zero bias. Its output axis runs from zero toward one; the dashed tangent selects x=0,h=.5x=0,h=.5. Sigmoid's activation derivative there is .25, but the full input derivative is 2(.25)=.52(.25)=.5. In general, for h=σ(wx+b)h=\sigma(wx+b), the factor is wσ′(wx+b)w\sigma'(wx+b). The curve uses sampled exact sigmoid values with line interpolation; the tangent uses the analytic derivative. Glorot and Bengio examine how activations, weights and layer Jacobians affect propagation.

Follow Ten Small Factors

Ten quarter factors reduce a unit gradient to about one millionth.
Ten quarter factors reduce a unit gradient to about one millionth.

For the declared toy chain, gn=.25ng_n=.25^n. At five backward steps the magnitude is .0009765625; at ten it is .0000009536743. The vertical axis is logarithmic: equal vertical distances mean equal ratios, not equal differences. Horizontal ticks count backward factors, with white points at zero and five. This value is small and finite, rather than automatically zero in machine arithmetic. The assumption is about complete local factors, not every network that uses sigmoid.

Follow Ten Large Factors

Ten factors of 1.5 amplify a unit gradient to 57.7.
Ten factors of 1.5 amplify a unit gradient to 57.7.

Change every complete factor to 1.5. Then gn=1.5ng_n=1.5^n, giving 7.59375 at five backward steps and 57.665039 at ten. This plot deliberately uses a linear vertical scale from zero to 60, unlike the previous logarithmic view. Integer-step values are connected by straight segments. A gradient of 57.7 may make an update too large, but does not itself imply overflow or NaN. Pascanu et al. analyze gradient growth and decay, including recurrent networks unrolled through time.

Add An Identity Route

An identity skip contributes one alongside the branch derivative.
An identity skip contributes one alongside the branch derivative.

Read this reverse-only graph from right to left. For y=x+F(x)y=x+F(x) with incoming sensitivity 1, the identity route contributes 1 and a branch derivative F′=.25F'=.25 contributes .25. They sum to 1.25 at the shared input. The left plus represents accumulated gradients, while the right plus is the forward addition viewed backward. A direct route helps transport a signal, but its contribution is only one term in the total. Identity Mappings in Deep Residual Networks explains the architecture conditions behind this route.

Clip The Norm

Norm clipping keeps the ray while shortening a large gradient.
Norm clipping keeps the ray while shortening a large gradient.

The axes are gradient components. The dashed vector (3,4)(3,4) has Euclidean norm 5. A cap of 1 rescales it to (.6,.8)(.6,.8), exactly on the unit circle. Both axes share the same units, and the solid arrow stays on the original ray. Norm clipping uses gclip=gmin⁡(1,c/∥g∥)g_{clip}=g\min(1,c/\lVert g\rVert) for nonzero gradients and leaves zero unchanged. Vectors already within the cap remain unchanged. Clipping can bound a spike; it cannot revive a tiny signal or repair an invalid numerical value already produced.

A Local Gate Can Close

A negative ReLU input has zero local derivative.
A negative ReLU input has zero local derivative.

At preactivation z=−1.5z=-1.5, ReLU outputs zero and has local slope zero. The highlighted horizontal segment and stop bar show the blocked local route. Positive preactivations have slope one, but weights still affect the complete layer derivative. A unit inactive across all relevant samples may stop receiving a data-loss update through this route; a single inactive sample does not establish that the whole network has stopped learning.

Check The Whole Product

A later large factor can restore a shrunken scalar signal.
A later large factor can restore a shrunken scalar signal.

This variant starts at 1, multiplies by .25, then by 4. The shared-scale bars return to height 1 because .25×4=1.25\times4=1. Individual factors can look alarming while their product is moderate. Conversely, repeated modest changes can compound. This scalar example does not guarantee stability for changing nonlinear states or matrix products.

Track Directions Too

The same Jacobian can shrink one direction and amplify another.
The same Jacobian can shrink one direction and amplify another.

Replace scalar factors with a two-dimensional Jacobian J=diag⁡(.25,2)J=\operatorname{diag}(.25,2). Backward multiplication uses JTJ^T, which equals JJ here. Starting with gradient (1,1)(1,1), one step produces (.25,2)(.25,2) and two produce (.0625,4)(.0625,4). The horizontal component shrinks while the vertical component grows. Dashed/circle, solid/square and mint/diamond endpoints distinguish the three vectors on equal-unit axes. Their norms are approximately 1.414, 2.016 and 4.0005: a growing total norm can hide a disappearing component. Real Jacobians can also rotate directions, so one average scalar factor is not a complete diagnosis.

Where It Breaks

A residual branch can cancel its identity gradient.
A residual branch can cancel its identity gradient.

For a residual block with incoming gradient 1 and F′(x)=−1F'(x)=-1, the signed contributions are +1 and −1. Equal opposing arrows cancel at the marked zero: dy/dx=1+F′=0dy/dx=1+F'=0. Even an identity skip does not guarantee a nonzero total gradient. Clipping this zero leaves it zero. Inspect layer gradients, activations and data before choosing initialization, normalization, activation or architecture changes; no one fix guarantees healthy training.

The Quick Version

  • Complete layer derivatives include weights and activation slopes.
  • Products can shrink or amplify backward sensitivity.
  • Logarithmic and linear plots answer different magnitude questions.
  • Norm clipping caps spikes without reviving tiny gradients.
  • Different vector directions can shrink and grow simultaneously.
  • Identity contributions can cancel against residual branches.