Residual Connections

Why Does This Exist?

You have a 19 layer network reading scanned receipts and faded characters still misread. The natural move is more depth. A deeper network should manage at least what a shallower one manages, because the extra layers could simply pass their input through unchanged and leave accuracy where it was.

That is not what happened. Past a certain depth, stacked plain layers made training accuracy itself worse. Not held out accuracy, which would point at overfitting. Training accuracy, on data the network had already seen. Researchers building very deep plain networks hit this wall directly, and it has a name: the degradation problem.

The puzzle is sharp. If extra layers can do nothing, the deeper net matches the shallower one. So the extra layers are failing to learn something that sounds trivial, which is to output their input unchanged. This page covers that failure and the one line fix. The surrounding architecture story lives in cnn-architectures, and the stacked operation itself is in convolution-operation. You need only one prerequisite idea here: a layer computes some function of its input, and training adjusts its weights by gradient descent.

Think of It Like This

How It Actually Works

A plain block computes some function $F(x)$ of its input $x$. Here $x$ is the incoming activation vector and $F$ is whatever the block layers do, such as a weight multiply followed by a nonlinearity, then another weight multiply. A residual block changes the output to $y = F(x) + x$. The input splits. One copy travels the untouched skip path. The other passes through the block layers. The two rejoin by plain addition.

Identity becomes the free default. When every weight inside the block equals zero, $F(x) = 0$ and the output is exactly $x$. Small weights give small $F(x)$, so the block starts near do nothing and moves away only as far as training pushes it. The layers no longer reconstruct the whole mapping. They learn only the residual, the correction on top of identity.

Why identity is hard without the skip

A randomly set $F$ lands nowhere near the identity function. Reproducing $x$ through a matrix multiply and a nonlinearity is one narrow target in a huge space of possible outputs. Gradient descent must find that target separately in every stacked layer, with no help from the structure. Empirically it does not reliably get there, which is why the deeper plain net trains worse.

Gradient keeps a clean path

The forward sum changes the backward flow. Because $y = F(x) + x$, the gradient arriving at the block splits into a term that travels back through $F$ plus a term of $1$ that rides the skip straight to earlier layers. Even where the path through $F$ shrinks toward zero, the skip term survives untouched. That is why residuals also soften the vanishing gradient trouble described in vanishing-exploding-gradients. Keep the two benefits apart: degradation is the optimizer failing to find identity, while vanishing gradients are chained small derivatives compounding toward zero. One line of wiring helps both, and batch-normalization still has its own separate job keeping activations well scaled.

Worked example

Take a scalar input $x = 2.0$ and suppose the block layers currently compute $F(x) = 0.5$. Then $y = 0.5 + 2.0 = 2.5$. If training has pushed the block weights to zero, $F(x) = 0.0$ and $y = 0.0 + 2.0 = 2.0$, which is the input reproduced exactly. With a tiny learned correction $F(x) = 0.02$, the output is $2.02$. Backward, if the path through $F$ carries a gradient of $0.3$, the total reaching earlier layers is $1.0 + 0.3 = 1.3$, where the $1.0$ rode the skip.

Code

The check below runs the scalar case from the worked example with only the standard library. It confirms the identity default and the small correction behavior directly.

1def residual_block(x: float, f: float) -> float:2    return f + x3
4
5x = 2.06print(residual_block(x, 0.5))  # -> 2.57print(residual_block(x, 0.0))  # -> 2.08print(residual_block(x, 0.02))  # -> 2.02

Zero inside gives the input back exactly. A small inside value gives a small nudge away from the input, which is the whole point: the block starts at do nothing and learns a correction from there.

Watch Out For

The Quick Version

• Stacking plain layers past a certain depth makes training accuracy itself worse, which is the degradation problem rather than overfitting.
• A residual block outputs $y = F(x) + x$, so zero block weights give the input back exactly.
• The layers inside learn only the correction on top of identity, a far smaller target than the full mapping.
• The skip gives gradients a path that bypasses $F$, which also softens vanishing gradients without replacing normalization.
• Shape changes need a projection shortcut on the skip path at that stage only.

What to Read Next

• CNN Architectures
• Convolution Operation