Residual Connections

In traditional feedforward architectures, stacking additional layers causes the degradation problem: beyond a certain depth, training error actually increases rather than plateauing. For example, a 56-layer plain convolutional network plateaus at $14.8\%$ training error on CIFAR-10, whereas a shallower 20-layer network converges to $6.2\%$. This is fundamentally an optimization failure rather than overfitting: theoretically, the deeper network could match the shallower model simply by learning identity mappings ($f(\mathbf{x}) = \mathbf{x}$) across its extra 36 layers, yet unconstrained nonlinear transformations fail to discover identity functions.

Learn the Nudge

To bypass the identity paradox, He et al. reformulate the objective. Instead of requiring stacked layers to approximate the underlying mapping $\mathcal{H}(\mathbf{x})$, the network explicitly learns a residual function:

$\mathcal{F}(\mathbf{x}) := \mathcal{H}(\mathbf{x}) - \mathbf{x} \implies \mathbf{y} = \mathcal{F}(\mathbf{x}) + \mathbf{x}$

The input $\mathbf{x}$ travels unaltered across an identity shortcut wire, while the residual branch $\mathcal{F}(\mathbf{x})$ computes only the additive correction. If the identity mapping is optimal, standard weight decay naturally drives the parameter weights toward zero ($\mathcal{F}(\mathbf{x}) \to \mathbf{0}$), ensuring the block defaults safely to $\mathbf{y} = \mathbf{x}$ rather than degrading performance.

Numbers That Add

In the forward pass, scalar input $x = 2.0$ combines with residual branch output $\mathcal{F}(x) = 0.5$ at the summation node to produce $y = 2.5$. In the backward pass, applying the chain rule to the additive node yields the gradient highway:

$\frac{\partial \mathcal{E}}{\partial \mathbf{x}} = \frac{\partial \mathcal{E}}{\partial \mathbf{y}} \left( \mathbf{I} + \frac{\partial \mathcal{F}}{\partial \mathbf{x}} \right) = \frac{\partial \mathcal{E}}{\partial \mathbf{y}} \mathbf{I} + \frac{\partial \mathcal{E}}{\partial \mathbf{y}} \frac{\partial \mathcal{F}}{\partial \mathbf{x}}$

With an incoming gradient $\frac{\partial \mathcal{E}}{\partial y} = 1.0$ and residual derivative $\mathcal{F}' = 0.3$, the downstream gradient splits into a direct $+1.0$ through the skip highway and $+0.3$ through the residual branch, totaling $1.3$. Even if the residual Jacobian $\frac{\partial \mathcal{F}}{\partial \mathbf{x}}$ completely vanishes to zero, the additive identity term $\mathbf{I}$ ensures an unobstructed $1.0$ gradient flows back to earlier layers.

Reshape the Skip

When convolutional stages downsample spatial dimensions (e.g., stride 2 reducing $56 \times 56$ to $28 \times 28$) while doubling feature channels ($64 \to 128$), raw identity addition $\mathcal{F}(\mathbf{x}) + \mathbf{x}$ fails due to dimensional incompatibility. To reconcile shapes, a projection shortcut introduces a linear $1 \times 1$ convolution $W_s$ with stride 2 along the skip path:

$\mathbf{y} = \mathcal{F}(\mathbf{x}) + W_s \mathbf{x}$

The projection matrix $W_s$ matches spatial grid dimensions and channel depth without adding non-linearities, allowing seamless element-wise summation across hierarchical scale boundaries.

Signal Survives Depth

Over a sequence of 10 plain layers, backpropagation multiplies successive weight Jacobians ($\prod_{l=1}^{10} J_l$). If average layer amplification is $0.25$, gradient magnitude attenuates exponentially ($0.25^{10} \approx 10^{-6}$), completely freezing early layer learning. In a ResNet, the gradient across $L$ layers decomposes additively:

$\frac{\partial \mathcal{E}}{\partial \mathbf{x}_0} = \frac{\partial \mathcal{E}}{\partial \mathbf{x}_L} \left( \mathbf{I} + \sum_{l=0}^{L-1} \frac{\partial \mathcal{F}_l}{\partial \mathbf{x}_0} \right)$

The direct highway term $\mathbf{I}$ provides a 1,000,000x gradient survival advantage, allowing networks with 100 to 1,000+ layers to train reliably from random initialization.

Keep Skips Bare

In original ResNet-v1 (post-activation), the non-linear activation followed addition ($\mathbf{y} = \text{ReLU}(\mathcal{F}(\mathbf{x}) + \mathbf{x})$). Because ReLU zeroes out negative activations, it truncates signals travelling along the identity highway. ResNet-v2 (pre-activation) fixes this by moving Batch Normalization and ReLU entirely inside the residual branch $\mathcal{F}(\mathbf{x})$. Keeping the skip path a completely clean, bare identity wire guarantees unrestricted, bidirectional forward signal and backward gradient flow.

Where It Breaks

While residual shortcuts resolve optimization degradation, they introduce specific failure boundaries:

1. Noise and Corruption Conduction: Because identity shortcuts are transparent conduits, mislabeled training inputs or adversarial perturbations travel directly to output layers without filtration. Skips facilitate information flow, but they cannot clean corrupted data.
2. Variance Accumulation: Unbounded additive accumulation ($\mathbf{x}_L = \mathbf{x}_0 + \sum \mathcal{F}_l$) causes feature variance to grow linearly with depth $L$. Without normalization layers (BatchNorm or LayerNorm) or residual scaling factors ($\alpha \approx 0.1$), activation magnitudes explode ($10 \to 1,000+$), destabilizing deep architectures.

The Quick Version

• Degradation paradox: Deeper plain networks suffer worse training error because optimization fails to learn identity mappings.
• Residual mapping: Shifting the target to $\mathcal{F}(\mathbf{x}) = \mathcal{H}(\mathbf{x}) - \mathbf{x}$ allows layers to default to identity via zero weights.
• Additive gradient highway: Addition distributes upstream gradients directly: $\nabla_{\mathbf{x}} = \nabla_{\mathbf{y}}(\mathbf{I} + \mathcal{F}')$, guaranteeing that $+1.0$ survives.
• 1x1 projection shortcuts: Stride-2 $1 \times 1$ convolutions match spatial resolutions and channel counts across downsampling stages.
• Signal preservation: Multiplicative plain layers decay to $10^{-6}$ over 10 layers, while the residual highway preserves $\approx 1.0$ gradient magnitude.
• Clean skip design: Pre-activation (moving BN and ReLU inside $\mathcal{F}$) ensures the skip wire remains an unobstructed identity channel.
• Failure boundaries: Skips pass corrupted labels straight to the output and require normalization to prevent cumulative activation variance explosion.