Convolution Operation

One small grid of weights slides over the image and takes one dot product at each stop, reusing the same nine numbers everywhere.

A 3 by 3 kernel slides over a 5 by 5 input and writes one dot product per stop, so the centre stop reads 2.

Why Does This Exist?

Feed a 200 by 200 grey photo into a plain dense layer and you hand every one of its 40,000 pixels its own weight for every output unit. Give that layer 10,000 units and you pass 400 million weights before the network has spotted a single edge.

The count hurts, but the blindness hurts more. A weight under pixel (10, 10) shares nothing with the weight under (10, 11). Learn a vertical edge on the left and you must relearn it from scratch on the right, then again everywhere else an edge could land.

Convolution fixes both with one move. Slide a tiny 3 by 3 grid over the whole image and reuse the same nine numbers at every stop. If the multiply and add step feels fuzzy, the dot product is the two minute version.

This page covers that slide, from one dot product to stride and padding. Convolutional neural networks stack these slides into a full classifier, pooling layers shrink what they produce, and receptive fields explain how far a deep unit can see.

Think of It Like This

The rubber stamp

You own one rubber stamp with a fixed pattern cut into it. You press it top left, lift, shift one inch right, press again. The mark changes because the paper underneath changed, never because the stamp changed.

The kernel is the stamp. Its nine numbers stay fixed after training. Each press writes one number that says how strongly the patch underneath matched, and all presses together build the feature map.

Where it stops: a stamp cannot learn. It presses what it was cut to press. A kernel starts random and earns its pattern from data, then freezes into a stamp you reuse.

How It Actually Works

Call the input $x$ , meaning the grid of pixel values. Call the small grid $w$ , meaning the kernel weights. Call one cell of the result $y$ , meaning a single match score. At each stop you lay $w$ over a patch of $x$ of the same size, multiply each overlapping pair, and add everything up.

One stop is one dot product

The highlighted stop in the diagram sits at row 1, column 1. The patch underneath reads [[4, 2, 0], [2, 5, 1], [1, 3, 4]] and the kernel reads [[1, 0, -1], [1, 0, -1], [1, 0, -1]], a vertical edge hunter that keeps the left column and subtracts the right:

y = 4(1) + 2(0) + 0(-1) + 2(1) + 5(0) + 1(-1) + 1(1) + 3(0) + 4(-1) = 2

Left column sums to 6, right column sums to 4, so 6 minus 4 leaves 2. That 2 lands in the centre cell of the output, exactly where the diagram circles it. Slide one column right, repeat the same nine multiplies, write the next cell.

Stride, padding and dilation set the size

Call the input width $n$ , meaning cells per side. Call the kernel width $k$ , the ring of zeros $p$ for padding, and the step $s$ for stride. Then:

\text{output size} = \left\lfloor \frac{n + 2p - k}{s} \right\rfloor + 1

Our picture uses $n = 5$ , $k = 3$ , $p = 0$ , $s = 1$ , so $(5 - 3 + 1) = 3$ stops per side and a 3 by 3 result. Stride 2 would skip every other stop and shrink the map. Padding 1 would ring the input with zeros first and keep the map at 5 by 5. Dilation spreads the kernel cells apart so a 3 by 3 kernel with dilation 2 reads from a 5 by 5 footprint while still holding nine weights.

The same nine numbers press at all nine stops. That is a bet: a pattern worth spotting in one place is worth spotting with the identical detector everywhere else. For photos the bet pays, since an edge is an edge wherever it falls.

Count it. A dense map from a 200 by 200 image to another 200 by 200 map needs one weight per input and output pair, which is 1.6 billion. A 3 by 3 convolution producing the same size map needs nine. Every position also trains those nine, so each training edge teaches the detector no matter where it landed.

Worked example: nine stops fill the map

Run the same kernel over all nine windows of this input:

x = \begin{bmatrix} 3 & 1 & 0 & 2 & 1 \\ 1 & 4 & 2 & 0 & 1 \\ 0 & 2 & 5 & 1 & 0 \\ 2 & 1 & 3 & 4 & 2 \\ 1 & 0 & 2 & 1 & 3 \end{bmatrix}

The nine answers, row by row, are:

\begin{bmatrix} -3 & 4 & 5 \\ -7 & 2 & 7 \\ -7 & -3 & 5 \end{bmatrix}

Top left reads -3 because its right column outweighs its left. The centre reads 2, as computed above. A real layer then repeats this with dozens of kernels at once, and each kernel writes its own channel, so 64 kernels turn one input into 64 stacked maps.

Code

The point is the check: run the same slide in plain Python and confirm the hand numbers, centre 2 and corners -3 and 5.

1x: list[list[float]] = [2    [3, 1, 0, 2, 1],3    [1, 4, 2, 0, 1],4    [0, 2, 5, 1, 0],5    [2, 1, 3, 4, 2],6    [1, 0, 2, 1, 3],7]8w: list[list[float]] = [9    [1, 0, -1],10    [1, 0, -1],11    [1, 0, -1],12]13
14def conv2d(a: list[list[float]], k: list[list[float]]) -> list[list[float]]:15    n: int = len(a)16    m: int = len(k)17    size: int = n - m + 118    out: list[list[float]] = [[0.0] * size for _ in range(size)]19    for i in range(size):20        for j in range(size):21            total: float = 0.022            for u in range(m):23                for v in range(m):24                    total += a[i + u][j + v] * k[u][v]25            out[i][j] = total26    return out27
28result: list[list[float]] = conv2d(x, w)29print(result)  # -> [[-3.0, 4.0, 5.0], [-7.0, 2.0, 7.0], [-7.0, -3.0, 5.0]]30print(result[1][1])  # -> 2.0

The printout matches the diagram: full map [[-3.0, 4.0, 5.0], [-7.0, 2.0, 7.0], [-7.0, -3.0, 5.0]] with centre 2.0.

Watch Out For

Calling cross-correlation convolution

True convolution flips the kernel before sliding it. Every deep learning library skips that flip and still says convolution, so what you run is cross-correlation.

Symptom: you copy a textbook example by hand and your signs come out mirrored, or a kernel you drew to catch a left edge catches a right edge instead. Fix: flip the kernel once with w[::-1, ::-1] when matching textbook math, then forget it, since a learned kernel simply learns the mirrored pattern and works the same.

Zero padding inventing an edge

Zero padding says the world outside the image is exactly zero. Near a bright border that lie creates a sharp fake step the kernel answers to.

Symptom: a row of strong activations hugs the border of the feature map even on blank inputs, and accuracy shifts when you change input size. Fix: switch the padding mode to reflect or replicate for that layer so the border continues the image instead of dropping to zero.

The Quick Version

One convolution stop lays a small kernel over same-size patch and adds up the nine pairwise products.
Our 3 by 3 vertical edge kernel turns the centre patch [[4, 2, 0], [2, 5, 1], [1, 3, 4]] into 6 minus 4, which is 2.
Nine stops over a 5 by 5 input fill a 3 by 3 feature map reading [[-3, 4, 5], [-7, 2, 7], [-7, -3, 5]].
The same nine weights serve every stop, so a 3 by 3 layer needs nine weights where a dense 200 by 200 map would need 1.6 billion.
Stride sets the step between stops, padding rings the input with zeros to preserve size, and dilation spreads kernel cells to widen the view.