Autoencoders

Force data through a layer too narrow to hold it, then ask for the original back. Whatever survives the squeeze is the structure the data actually has.

A 4-number input squeezes through a 2-number code and rebuilds with mean squared error 0.0625, so the input is its own label.

Why Does This Exist?

A camera above a kiln photographs every ceramic floor tile coming off the line. Each shot is 64 pixels square in greyscale, so 4096 numbers describe one tile, and 200,000 photos pile up every week with none of them labelled. An inspector tagged 400 tiles as cracked or fine months ago, and that is the entire labelled set.

Those 4096 numbers oversell the tile. Same glaze, same grout spacing, same camera height. What actually varies is brightness, a little rotation, speckle density, and whether a crack runs through. About a dozen quantities, with the other 4084 along for the ride.

Two familiar tools do not finish the job. A multi-layer perceptron maps numbers to numbers but needs labels to learn from, and you have 400. PCA drops dimensions with no labels at all, yet it only finds a flat slice through pixel space, while a two-degree rotation bends every pixel along a curve.

This page sticks to the plain autoencoder: squeeze through a narrow code, rebuild, score the round trip. The sequel that repairs sampling is the variational autoencoder. The wide-code variant that trades width for sparsity lives with sparse autoencoders. And when most inputs are normal and you flag whatever rebuilds badly, you have crossed into anomaly detection.

Think of It Like This

The card through the hatch

Two draughtsmen sit with a wall between them and one narrow slot. The first looks at a tile and writes 2 numbers on a card. The second never sees the tile and redraws it from the card alone. Score how close the redrawing comes, and repeat on this factory's tiles for a month.

They will invent a shorthand. The first number becomes shade, the second becomes crack, because there is room for nothing else. Nobody handed them that vocabulary. The scoring produced it.

Where it stops: widen the slot to a 4-number card and the first draughtsman copies the readings across, the score goes perfect, and the shorthand is gone. And if you invent 2 numbers yourself and post the card through, the second draughtsman draws something. It just is not a tile.

How It Actually Works

Two halves with a pinch in the middle. The encoder $f$ maps an input $x$ down to a short code $z$ . The decoder $g$ maps $z$ back out to a rebuild $\hat{x}$ , read "x-hat". Training minimises the mean squared gap between each input and its own rebuild:

\mathcal{L} = \frac{1}{n}\sum_{i=1}^{n} \big\lVert x_i - g(f(x_i)) \big\rVert^2

$x_i$ is one input, $g(f(x_i))$ is its rebuild, and the bars mean squared difference summed over every number in the input, averaged over all $n$ inputs. No label appears anywhere. The target is the input, which makes this self-supervised learning, and the gradients travel by backpropagation like any other network.

The bottleneck does the work

The narrow width is the constraint that makes learning happen, not a knob you tune for accuracy. Give the code as many slots as the input and the cheapest solution is a copy: perfect score, zero shorthand. Squeeze 4096 tile readings through 32 numbers and every slot has to earn its place. Shade earns it. Pixel row 17, column 40 does not.

Worked example

Take $x = [1.0, 0.5, 0.5, 0.0]$ . The encoder averages each pair:

z_1 = 0.5 \cdot 1.0 + 0.5 \cdot 0.5 = 0.75, \quad z_2 = 0.5 \cdot 0.5 + 0.5 \cdot 0.0 = 0.25

So $z = [0.75, 0.25]$ : four numbers through two. The decoder repeats each code number:

\hat{x} = [0.75, 0.75, 0.25, 0.25]

The gaps are $x - \hat{x} = [0.25, -0.25, 0.25, -0.25]$ . Square each to $0.0625$ , four times over, sum to $0.25$ , divide by 4:

\text{MSE} = \frac{0.0625 + 0.0625 + 0.0625 + 0.0625}{4} = 0.0625

That $0.0625$ is the whole training signal. Each input pays its own rebuild gap, and gradient descent pushes the average gap down. The diagram above draws this exact round trip, numbers and all.

Where PCA ends and this starts

The relationship is exact, not a family resemblance. Make both halves single matrices with no activation, keep squared-error loss, and the optimum spans the same subspace as the top principal components. Same subspace, not the same axes: nothing in that loss asks for orthogonal or ordered directions.

So a linear autoencoder buys nothing over PCA except a slower way to compute it. The payoff arrives with the nonlinearities. The decoder traces a curved surface through pixel space instead of a flat slice, so two degrees of rotation become a short move along that surface. That surface is the latent manifold, and following it is the only reason to reach for a network.

Two variants close the lazy paths. A denoising autoencoder knocks out a share of the input pixels and asks for the clean tile back, so copying turns actively wrong. A sparse autoencoder keeps the code wide but penalises how many units switch on, so activity becomes the limit instead of width. The sparse variant has its own page under sparse autoencoders.

Code

The point is the check, not the formula: average each pair, repeat each code, and watch the gap land exactly where the hand computation says.

1x: list[float] = [1.0, 0.5, 0.5, 0.0]2
3z: list[float] = [0.5 * x[0] + 0.5 * x[1], 0.5 * x[2] + 0.5 * x[3]]4x_hat: list[float] = [z[0], z[0], z[1], z[1]]5errs: list[float] = [a - b for a, b in zip(x, x_hat)]6mse: float = sum(e * e for e in errs) / len(errs)7
8print(z)  # -> [0.75, 0.25]9print(x_hat)  # -> [0.75, 0.75, 0.25, 0.25]10print(mse)  # -> 0.0625

The printout matches the worked example: code $[0.75, 0.25]$ , rebuild $[0.75, 0.75, 0.25, 0.25]$ , error $0.0625$ .

Watch Out For

A bottleneck wide enough to cheat through

Reconstruction error drops to almost nothing, so you ship it. Then the defect classifier trained on those code numbers performs like it is reading noise. The bottleneck had room to pass the input through nearly unchanged, so the code is a lightly scrambled copy of the readings.

Symptom: a tiny rebuild gap paired with a useless downstream model. Fix: narrow the code on purpose. If downstream accuracy goes up when you cut 64 numbers to 16, the wide version was passing readings through. A denoising objective helps for the same reason: corruption makes copying wrong.

Treating the latent space as something you can sample

Someone interpolates between two tile codes to make training examples, or draws fresh code numbers from a bell curve and decodes them. Out come smears and grey fog. The instinct says the model undertrained.

It did not. Nothing in the loss constrained the region between two encoded points, so a decoder asked about it returns mush with full confidence. Symptom: midpoints between two real codes decode to nothing plausible. Fix: test that interpolation first, and when you need sampling that works, move to a variational autoencoder, which pins that space down.

The Quick Version

An autoencoder squeezes input to a short code and rebuilds it, scoring the gap against the input itself, so training needs no labels.
The bottleneck is the mechanism. Too wide and the network learns a pass-through that rebuilds perfectly and means nothing.
Four numbers through 2 codes rebuild at mean squared error 0.0625, and that gap is the entire training signal.
Linear layers with squared error land on the same subspace as PCA. Curves are what buy the network.
Nothing pins down the space between codes, so invented codes decode to mush and real sampling needs a variational autoencoder.