Skip to content
AI360Xpert
Core ML
Visual explainer

Residual Connections

How skip connections solve the vanishing gradient problem by creating a superhighway for information and error signals.

In a standard deep network, passing the gradient backwards through many layers causes it to vanish.
In a standard deep network, passing the gradient backwards through many layers causes it to vanish.

In traditional deep networks, every layer transforms the data. During training, the error signal must flow backward through all these transformations. Because of the chain rule, the gradient is multiplied at each step. In very deep networks, this causes the gradient to shrink exponentially until it vanishes, freezing the early layers.

The Skip Connection

A residual block adds the original input directly to the layer's output, bypassing the transformation.
A residual block adds the original input directly to the layer's output, bypassing the transformation.

Instead of forcing a layer to learn a complete transformation, a residual connection passes the input xx around the layer and adds it to the output F(x)F(x). The layer now only has to learn the "residual" change—the difference between the input and the desired output.

Gradient Superhighways

Addition distributes gradients equally, allowing the error signal to flow backward unimpeded.
Addition distributes gradients equally, allowing the error signal to flow backward unimpeded.

In calculus, the derivative of an addition operation is 1. This means that during backpropagation, the gradient flows directly through the skip connection without being altered. It creates a "superhighway" that delivers a strong, uncorrupted error signal directly to the earlier layers.

Unlocking Extreme Depth

Networks with residual connections can scale to hundreds of layers without suffering from vanishing gradients.
Networks with residual connections can scale to hundreds of layers without suffering from vanishing gradients.

Because the early layers receive a strong training signal, networks can be built much deeper. If a layer isn't needed, its weights can easily be pushed to zero, leaving the skip connection to pass the data through as an identity mapping. This allowed models to jump from 20 layers to over 100.

Where It Breaks

You cannot add two tensors of different shapes; dimension changes require an extra projection step.
You cannot add two tensors of different shapes; dimension changes require an extra projection step.

A skip connection is an element-wise addition. If the main path changes the shape of the tensor—such as reducing spatial dimensions with a stride or changing the number of channels—the direct skip connection fails. You must insert a projection (like a 1x1 convolution) on the skip path to match the dimensions, which adds computational cost.

The Quick Version

  • Standard deep networks suffer from vanishing gradients.
  • Residual connections add the input directly to the output.
  • Layers only learn the residual difference, not the whole mapping.
  • Gradients flow backward perfectly through the addition operation.
  • Dimension mismatches break the direct addition.

What to Read Next