Neural Networks
Skip / Residual Connections
How residual blocks provide a "gradient highway", completely solving the vanishing gradient problem in very deep neural networks.
Stage 1 of 4: The Shallow Network
Gradient at first layer: 0.2500
- Layer
- Strong Gradient
- Weak/Vanishing Gradient
In a shallow network, the error signal (gradient) easily reaches the first layer during the backward pass.
Check your understanding
1 questions in the bank. Each attempt draws a fresh set in a fresh order, so a second go is a real second go.
See why deep networks fail to train, and how adding skip connections lets gradients flow cleanly to early layers.
For a long time, researchers couldn't train deep neural networks. The problem was the Vanishing Gradient.
During training, the error signal (gradient) travels backward from the output to the input layer. But at every layer, it gets multiplied by weights and derivatives. If these numbers are smaller than 1, the gradient shrinks. In a 20-layer network, the gradient becomes microscopically small by the time it reaches the first layer. The early layers simply stop learning.
In 2015, ResNet solved this with Skip Connections (or Residual Connections). A skip connection literally bypasses a layer, adding the input directly to the output. During the backward pass, gradients flow freely down this shortcut, acting as a "gradient highway". This simple trick allows us to train networks with hundreds or thousands of layers!
Reference
- Vanishing Gradient
- When gradients become too small to effectively update early layers.
- Skip Connection
- An architectural shortcut that adds an earlier layer's output to a later layer.
- ResNet
- Residual Network. The architecture that popularized skip connections.
Break it on purpose
Without skip connections, the gradient at Layer 1 in a 10-layer network approaches zero. The early layers learn nothing, and the network fails to train.