Autoencoders

A 28 by 28 grayscale image contains 784 pixel intensities, but raw pixel count is misleading. Most background pixels remain zero, while neighboring stroke pixels exhibit intense spatial correlation. The meaningful information describing the digit does not fill 784 orthogonal dimensions; it resides on a compact, curved low-dimensional manifold embedded within high-dimensional ambient space.

Squeeze, Then Rebuild

An autoencoder pairs two coordinated neural sub-networks. The encoder $f_\theta$ maps high-dimensional input $x \in \mathbb{R}^{784}$ into a low-dimensional latent vector $z = f_\theta(x) \in \mathbb{R}^{32}$. The decoder $g_\phi$ then expands that compact code back into a reconstruction $\hat{x} = g_\phi(z) \in \mathbb{R}^{784}$. Because the network optimizes mean squared reconstruction error $L(x, \hat{x}) = \frac{1}{D} \sum_{i=1}^D (x_i - \hat{x}_i)^2$, the narrow intermediate layer prevents trivial copying, forcing the network to discover informative latent coordinates.

24.5 Times Smaller

Compressing 784 inputs into 32 bottleneck activations enforces a $784 / 32 = 24.5\times$ reduction in representational volume. The network cannot memorize individual pixels; it must discard high-frequency variance and retain only the underlying structural features required to regenerate the full pattern.

One Code Rebuilt

In latent space, an input pattern maps to specific coordinates such as $(z_1, z_2) = (0.6, -0.4)$. Passing this 32-dimensional latent vector through the decoder reconstructs the original shape with a mean squared error of 0.02. Nearby vectors in latent space correspond to semantically similar shapes, forming an organized representation of visual concepts.

Noise In, Digit Out

Denoising autoencoders deliberately corrupt the training input with random noise ($\tilde{x} = x + \epsilon$) while requiring the decoder to reproduce the original clean input $x$. Because the bottleneck lacks capacity to encode arbitrary random noise, the network learns an orthogonal projection operator: it pulls off-manifold noise vectors back onto the learned data manifold, effectively stripping corruption.

Size the Bottleneck

Tuning the latent dimension balances fidelity against generalization. A bottleneck of width 8 is overly restrictive, producing blurry reconstructions with a high reconstruction error of 0.09. A width of 32 captures crisp structural variance with a minimal loss of 0.02. Conversely, an overcomplete bottleneck of width 256 possesses enough unconstrained capacity to memorize noise directly, failing to learn any compressed structural representation.

Where It Breaks

Deterministic autoencoders only learn to reconstruct points near the training distribution. The resulting latent space often develops unpopulated voids and disjoint clusters between classes. Sampling a vector from these unregularized gaps decodes into a distorted, meaningless smear. Generating realistic novel instances requires continuous latent regularization, leading directly to Variational Autoencoders.

The Quick Version

• High-dimensional images hide low-dimensional structural manifolds.
• An encoder compresses to a bottleneck; a decoder reconstructs.
• 784 inputs squeezed into 32 units yields a 24.5x compression ratio.
• Latent coordinate vectors decode with low reconstruction error.
• Denoising autoencoders strip corruption by manifold projection.
• Capacity tuning balances sharp reconstruction against memorization.
• Sampling unregularized latent voids produces smeared artifacts.