Visual explainer

Backpropagation

How a neural network measures its mistakes and assigns blame. See how the chain rule passes errors backward to update every weight.

To learn, we must measure how much each individual weight contributed to the final loss.

A neural network contains thousands or billions of weights. When the network makes a mistake, we need a mathematically exact way to answer: if we change this specific weight by a tiny amount, how much does the final loss change?

The Forward Cache

During the forward pass, every activation is cached in the computation graph for later use.

To calculate those changes later, the network must remember what it just did. During the forward pass, it caches every intermediate activation in a massive computation graph.

The Chain Rule

The chain rule passes the error signal backward, layer by layer.

Calculus gives us the chain rule: to find the derivative of a composite function, multiply the derivatives of its parts. Backpropagation applies this by taking the final error and passing it backward, multiplying by the local derivative at each step.

One Pass, All Gradients

One backward pass computes one exact gradient for every parameter simultaneously.

Because the graph was cached, a single backward pass sweeping from the output to the input computes the exact gradient for every parameter at the exact same time.

Depth Interaction

Because gradients multiply backward, they shrink or explode through many layers.

Since backpropagation relies on multiplying numbers together, a deep network risks exponential scaling. If the numbers are slightly less than 1, the gradient vanishes. If they are slightly more than 1, it explodes.

Where It Breaks

Exploding gradients cause massive updates that crash training, requiring gradient clipping as a patch.

If gradients explode, a single update can throw the weights into entirely useless ranges, causing the loss to become NaN (Not a Number) and crashing training. The standard engineering patch is gradient clipping: manually capping the size of the update.

The Quick Version

Backpropagation calculates how every weight affects the loss.
The forward pass caches all intermediate activations.
The chain rule pushes the error backward, multiplying local derivatives.
A single backward sweep calculates gradients for every weight simultaneously.
Multiplying derivatives through many layers causes gradients to vanish or explode.
Exploding gradients crash training, requiring clipping as an engineering fix.