CNN Architectures

Why Does This Exist?

Say you must sort a million phone photos into a thousand object classes. A plain dense net on 224 by 224 colour images needs tens of millions of weights just to look once, relearns the same edge at every position, and stops improving once you stack past twenty layers. You get huge, hungry, and shallow all at once.

Convolution plus pooling broke that wall with a reusable shape: share small kernels across space, shrink the map while growing channels, then vote with a classifier. That base shape lives in full at convolutional neural networks, with field growth at receptive fields. This page doesn't re-teach it.

It tracks the five edits that made the shape practical. LeNet proved it on digits in 1998. AlexNet proved depth at scale in 2012 with 8 layers, 60M weights, and 15.3 percent top-5 error on ImageNet. VGG forced uniformity with only 3 by 3 kernels at 16 layers and 138M weights. GoogLeNet cut the bill with 1 by 1 bottlenecks. ResNet added one skip to reach 152 layers. When patches replace convolution entirely, that sequel lives at vision transformers.

Think of It Like This

How It Actually Works

The shared skeleton and how to count it

Call the input edge $n$, meaning how wide the map is. Call the kernel edge $k$, meaning how wide one kernel is. Call the zero border $p$, meaning how many zeros pad each side. Call the step $s$, meaning how far the window jumps. One conv layer maps edge to edge by:

A 5 by 5 input with a 3 by 3 kernel, no padding, stride 1 visits $(5 - 3) + 1 = 3$ positions per side, so a 3 by 3 output, nine dot products with the same nine weights. Pad by 1 and it stays 5 by 5. That stay-even trick is what lets deep stacks keep their maps large until a pooling step halves them.

Channels set the price. Call incoming channels $C_{in}$, meaning how many feature maps enter, and outgoing detectors $C_{out}$, meaning how many kernels run. Weights only, no bias:

A 3 by 3 layer reading 128 maps and writing 256 costs $9 \times 128 \times 256 = 294,912$ weights. Every later section reuses this exact arithmetic, so check it here before trusting the bigger totals below.

LeNet in 1998 and AlexNet in 2012 prove shape then scale

LeNet-5 read handwritten digits with a handful of conv-pool stages feeding a small classifier, seven layers in all. Its job in this lineage is proof the shape works at all, not scale.

AlexNet kept the shape and proved scale once graphics cards made it affordable: 5 conv layers plus 3 dense ones, 8 weight layers, 60M weights, 15.3 percent top-5 error on ImageNet, more than ten points ahead of the next entry that year. No new mechanism carried that jump. Depth, data, graphics compute, and rectified units did. The field believed depth was worth paying for after that scoreboard, not before.

VGG in 2014 makes depth uniform with only 3 by 3

VGG replaced judgment calls about kernel sizes with one rule: every conv is 3 by 3, everywhere, and depth grows by stacking more of that same block. Two stride-1 3 by 3 layers cover the same 5 by 5 field as one 5 by 5 kernel, since $\text{RF} = 1 + 2 \times (3 - 1) = 5$, the same growth receptive fields traces by hand.

Uniformity costs less than it looks per layer yet more than it looks in total. Two 3 by 3 layers with $C$ channels cost $2 \times 9C^2 = 18C^2$ against $25C^2$ for one 5 by 5, about 28 percent fewer for the same footprint. Stack sixteen weight layers of that block without ever economizing elsewhere and VGG-16 still lands at 138M weights, more than twice AlexNet, most of it in the dense head. Discipline bought accuracy and a clean design, not efficiency.

GoogLeNet in 2014 cuts the bill with 1 by 1 bottlenecks

GoogLeNet answered VGG the same year with the opposite bet: run several kernel shapes in parallel on the same input, a 1 by 1 alongside a 3 by 3 and a 5 by 5, then concatenate the outputs instead of picking one shape per layer. The 1 by 1 path is the trick. It drops channel count before the expensive kernel runs, so the 3 by 3 and 5 by 5 only ever see a thin input.

Call the entering width $C$, the bottleneck width $B$, with $B$ much smaller than $C$. A direct 5 by 5 from 256 to 256 costs $25 \times 256 \times 256$. Routed through $B = 64$ it costs one 1 by 1 down, one thin 5 by 5, one 1 by 1 back up. Same in-out width, far less work. The worked example below runs those exact numbers to 12.1 times fewer. Whole-net accuracy stayed competitive with VGG at roughly 5M weights against 138M, proof more weights and more depth are different levers.

ResNet in 2015 removes depth with one addition

Push a plain stack past about twenty layers and training accuracy itself gets worse, not validation accuracy, which would just be overfitting. Training accuracy. Extra layers fail to learn even identity, the do-nothing map a deeper net should manage at worst. That is the degradation problem, and residual connections covers its full treatment.

A residual block rewrites the ask. Call the input $x$, meaning the maps entering the block. Call the block layers $F(x)$, meaning the correction those layers compute, two convs with a nonlinearity. The block outputs:

The $+x$ is the skip path, the untouched input added back. When $F(x) = 0$ the block returns $x$ exactly, so identity is the free default and training only hunts a small correction. That one-line change made 50, 101, and 152 layer nets routinely trainable, with 152-layer ResNet winning its ImageNet year outright. Where channels change between stages the bare skip no longer fits, and a trained 1 by 1 projection reshapes $x$ on the skip alone.

Worked example: one 5 by 5 layer from 1,638,400 to 135,168

Fix $C = 256$ in and out, bottleneck $B = 64$, kernel $k = 5$ for the spatial step. Direct:

Bottlenecked, three terms in order:

Sum to $16,384 + 102,400 + 16,384 = 135,168$. The ratio is $1,638,400 / 135,168 \approx 12.1$, so about 12.1 times fewer weights for the same 256 to 256 job. The diagram draws these two totals side by side, and the code below recomputes them from the same $k$, $C$, and $B$.

Code

The point is the check: recompute the diagram totals from kernel and channel counts, not by copying them.

1def conv_params(kernel: int, in_ch: int, out_ch: int) -> int:2    return kernel * kernel * in_ch * out_ch  # weights only, no bias3
4
5direct = conv_params(5, 256, 256)6bottleneck = conv_params(1, 256, 64) + conv_params(5, 64, 64) + conv_params(1, 64, 256)7
8print(f"direct 5x5: {direct:,}")  # -> direct 5x5: 1,638,4009print(f"1x1 cut: {bottleneck:,}")  # -> 1x1 cut: 135,16810print(f"saves: {direct / bottleneck:.1f}x")  # -> saves: 12.1x

The printout matches the hand arithmetic above and the left half of the diagram: 1,638,400 against 135,168, about 12.1 times fewer.

Watch Out For

The Quick Version

• CNN architectures share one skeleton: stacked convolution and pooling shrink space while growing channels, then a dense head votes one label.
• LeNet proved that skeleton on digits in 1998 with 7 layers, and AlexNet proved depth at scale in 2012 with 8 layers, 60M weights, and 15.3 percent top-5 error on ImageNet.
• VGG-16 stacks only 3 by 3 kernels to 16 layers at 138M weights, and two 3 by 3 layers cover the same 5 by 5 field as one 5 by 5 with about 28 percent fewer weights.
• GoogLeNet runs parallel kernels with 1 by 1 channel cuts, so one 256 channel 5 by 5 layer drops from 1,638,400 weights to 135,168, about 12.1 times fewer.
• A residual block outputs F(x) plus x, so zero inner weights return the input exactly and 50, 101, and 152 layer nets stay trainable.
• Small sets and tight latency want older small backbones reused by freezing and fine-tuning, not the newest giant trained from scratch.

What to Read Next

• Convolutional Neural Networks
• Residual Connections
• Transfer Learning