Autoencoders: Vanilla, Denoising, and Sparse

An autoencoder compresses input data into an information bottleneck and reconstructs it, forcing the latent code to capture essential underlying structure.

Autoencoder variants prevent trivial identity mapping through undercomplete bottlenecks, denoising corruption, or overcomplete sparsity penalties.

Why Does This Exist?

In unsupervised representation learning, the fundamental objective is to discover compact, meaningful coordinates describing high-dimensional data without human annotations. If a neural network is trained simply to reproduce its input $x$ with an encoder $f_\theta$ and decoder $g_\phi$ such that $g_\phi(f_\theta(x)) \approx x$ , an unconstrained network with equal or greater hidden capacity can trivially memorize the identity function $f(x) = x$ . In doing so, the latent neurons simply copy input coordinates without extracting semantic patterns, statistical dependencies, or underlying manifold geometry.

To force the network to discard noise and discover the intrinsic low-dimensional manifold where real data concentrates, practitioners impose deliberate architectural or regularizing constraints:

Vanilla (Undercomplete) Autoencoders enforce a strict dimensional bottleneck where the latent dimension $k$ is dramatically smaller than the input dimension $d$ ( $k \ll d$ ).
Denoising Autoencoders (DAE) deliberately corrupt the input with stochastic noise $\tilde{x} \sim q(\tilde{x}|x)$ and force the network to recover the clean original $x$ , learning a vector field that projects off-manifold points back onto the true data distribution.
Sparse Autoencoders (SAE) allow an overcomplete latent space ( $m \gg d$ ) to capture rich, non-linear combinations of features, but impose extreme sparsity penalties so that only a tiny fraction (often 1% to 5%) of neurons fire for any single input—a technique foundational to modern mechanistic interpretability in large language models.

Understanding these architectures requires familiarity with loss-functions that balance reconstruction fidelity with regularization penalties.

Think of It Like This

A courtroom sketch artist and a damaged fax machine

Imagine trying to transmit the description of a complex face over a constrained channel.

A Vanilla Autoencoder is like a courtroom sketch artist who is allowed to write down exactly three words on an index card (for example: "oval, beard, glasses"). When handed the card, a second artist reconstructs the portrait. Because the card holds only three words, the artists cannot memorize pixel-by-pixel skin blemishes; they are forced to capture the primary geometric landmarks of the face.

A Denoising Autoencoder is like receiving a portrait through a fax machine smeared with heavy coffee stains and static tears. The recipient is tasked not with copying the coffee stains, but with painting the pristine original photograph. To do so, the recipient must understand what clean, plausible human faces actually look like, effectively wiping away the noise by projecting the image back toward realistic facial anatomy.

A Sparse Autoencoder is like a massive dictionary containing 50,000 distinct visual descriptors, but the writer is strictly penalized for using more than five words per sentence. Even though the dictionary is enormous (overcomplete), any single description must choose five precise, monosemantic concepts ("freckled", "smiling", "tilted-left") rather than blending 50 generic words together.

How It Actually Works

Architectural Constraints and Loss Formulations

An autoencoder consists of an encoder $f_\theta: \mathcal{X} \to \mathcal{Z}$ and a decoder $g_\phi: \mathcal{Z} \to \mathcal{X}$ . The training mechanisms diverge based on the constraint applied:

1. Vanilla (Undercomplete) Autoencoder

With input $x \in \mathbb{R}^d$ and latent code $z = f_\theta(x) \in \mathbb{R}^k$ where $k \ll d$ , the network minimizes the mean squared reconstruction error:

$\mathcal{L}_{\text{vanilla}}(\theta, \phi) = \frac{1}{N} \sum_{i=1}^N \|x_i - g_\phi(f_\theta(x_i))\|_2^2$

When $f_\theta$ and $g_\phi$ are linear, the learned latent space spans the exact same principal subspace as Principal Component Analysis (PCA). Non-linear activations allow the autoencoder to learn non-linear manifold coordinate charts.

2. Denoising Autoencoder (DAE)

The clean input $x$ is corrupted by a noise distribution $q(\tilde{x}|x)$ (such as additive Gaussian noise $\tilde{x} = x + \epsilon, \epsilon \sim \mathcal{N}(0, \sigma^2 I)$ or binary dropout / masking). The encoder processes the corrupted $\tilde{x}$ , but the reconstruction loss is computed against the clean ground-truth $x$ :

$\mathcal{L}_{\text{DAE}}(\theta, \phi) = \mathbb{E}_{x \sim p_{\text{data}},\, \tilde{x} \sim q(\tilde{x}|x)} \left[ \|x - g_\phi(f_\theta(\tilde{x}))\|_2^2 \right]$

Alain and Bengio (2014) proved that as noise variance $\sigma^2 \to 0$ , the optimal reconstruction vector minus the corrupted input estimates the score function (the gradient of the data log-density):

$g_\phi(f_\theta(\tilde{x})) - \tilde{x} \propto \sigma^2 \nabla_{\tilde{x}} \log p(\tilde{x})$

The reconstruction vector points directly toward the nearest region of high data density on the data manifold, establishing the mathematical bridge connecting autoencoders to modern score-based diffusion models.

3. Sparse Autoencoder (SAE)

In a sparse autoencoder, the latent dimension is expanded ( $m \ge d$ or $m \gg d$ ). Sparsity is enforced either through an $L_1$ penalty on activations or via Kullback-Leibler (KL) divergence against a tiny target firing rate $\rho \approx 0.05$ :

$\mathcal{L}_{\text{SAE}}(\theta, \phi) = \frac{1}{N} \sum_{i=1}^N \|x_i - \hat{x}_i\|_2^2 + \beta \sum_{j=1}^m \text{KL}(\rho \parallel \hat{\rho}_j)$

where $\hat{\rho}_j = \frac{1}{N} \sum_{i=1}^N a_j(x_i)$ is the average activation of hidden unit $j$ over the training batch, and the KL divergence for Bernoulli distributions is:

$\text{KL}(\rho \parallel \hat{\rho}_j) = \rho \log \frac{\rho}{\hat{\rho}_j} + (1 - \rho) \log \frac{1 - \rho}{1 - \hat{\rho}_j}$

In modern interpretability architectures (such as Anthropic's dictionary learning on LLM residual streams), the activation function is a JumpReLU or TopK selection, or regularized with $\lambda \|z\|_1$ .

Worked Example

Let us trace a concrete numeric step for a Denoising Autoencoder on a 4-dimensional vector $x = [2.0, -1.0, 3.0, 0.5]$ :

Noise injection: Add Gaussian perturbation $\epsilon = [0.2, -0.4, 0.3, -0.1]$ : $\tilde{x} = x + \epsilon = [2.2, -1.4, 3.3, 0.4]$
Encoder mapping to $k=2$ dimensions: Given encoder weights $W_e$ of shape $(2, 4)$ and bias $b_e = [0.0, 0.0]$ : $W_e = \begin{bmatrix} 0.5 & 0.0 & 0.2 & 0.1 \\ -0.1 & 0.4 & 0.1 & 0.0 \end{bmatrix}$ Calculate pre-activation: $z_1 = 0.5 \cdot 2.2 + 0.0 \cdot (-1.4) + 0.2 \cdot 3.3 + 0.1 \cdot 0.4 = 1.10 + 0.00 + 0.66 + 0.04 = 1.80$ $z_2 = -0.1 \cdot 2.2 + 0.4 \cdot (-1.4) + 0.1 \cdot 3.3 + 0.0 \cdot 0.4 = -0.22 - 0.56 + 0.33 + 0.00 = -0.45$ Applying ReLU: $z = [\max(0, 1.80), \max(0, -0.45)] = [1.80, 0.00]$ .
Decoder reconstruction to 4 dimensions: Given decoder weights $W_d = W_e^T$ of shape $(4, 2)$ and bias $b_d = [0.0, 0.0, 0.0, 0.0]$ : $\hat{x}_1 = 0.5 \cdot 1.80 + (-0.1) \cdot 0.0 = 0.90$ $\hat{x}_2 = 0.0 \cdot 1.80 + 0.4 \cdot 0.0 = 0.00$ $\hat{x}_3 = 0.2 \cdot 1.80 + 0.1 \cdot 0.0 = 0.36$ $\hat{x}_4 = 0.1 \cdot 1.80 + 0.0 \cdot 0.0 = 0.18$ Reconstructed vector: $\hat{x} = [0.90, 0.00, 0.36, 0.18]$ .
Loss evaluation against CLEAN $x$ : Compute squared error against clean $x = [2.0, -1.0, 3.0, 0.5]$ (NOT against noisy $\tilde{x}$ ): $\|x - \hat{x}\|_2^2 = (2.0 - 0.90)^2 + (-1.0 - 0.0)^2 + (3.0 - 0.36)^2 + (0.5 - 0.18)^2$ $= 1.10^2 + (-1.0)^2 + 2.64^2 + 0.32^2 = 1.21 + 1.00 + 6.9696 + 0.1024 = 9.282$ The reconstruction gradient pulls the network parameters toward producing clean coordinates from noisy states.

Code

1import torch2import torch.nn as nn3import torch.nn.functional as F4
5class DenoisingSparseAutoencoder(nn.Module):6    """Autoencoder supporting noise corruption and L1 activation sparsity."""7    def __init__(self, in_dim: int = 64, latent_dim: int = 128, noise_std: float = 0.2):8        super().__init__()9        self.noise_std = noise_std10        # Overcomplete latent space (128 > 64) with sparsity11        self.encoder = nn.Linear(in_dim, latent_dim)12        self.decoder = nn.Linear(latent_dim, in_dim)13
14    def forward(self, x: torch.Tensor, add_noise: bool = True) -> tuple[torch.Tensor, torch.Tensor]:15        clean_x = x16        if add_noise and self.training:17            noise = torch.randn_like(x) * self.noise_std18            corrupted_x = x + noise19        else:20            corrupted_x = x21            22        latent = F.relu(self.encoder(corrupted_x))  # Latent representation z23        recon = self.decoder(latent)                # Reconstructed output x_hat24        return recon, latent25
26def compute_loss(clean_x: torch.Tensor, recon_x: torch.Tensor, latent: torch.Tensor, l1_lambda: float = 1e-3):27    # Reconstruction loss evaluated strictly against clean_x28    recon_loss = F.mse_loss(recon_x, clean_x)29    # L1 sparsity penalty encouraging latent neurons to remain zero30    sparsity_loss = l1_lambda * torch.mean(torch.sum(torch.abs(latent), dim=1))31    total_loss = recon_loss + sparsity_loss32    return total_loss, recon_loss.item(), sparsity_loss.item()33
34# Verify execution35torch.manual_seed(42)36model = DenoisingSparseAutoencoder(in_dim=16, latent_dim=32, noise_std=0.1)37x = torch.randn(8, 16)  # Batch of 8 clean vectors38
39reconstructed, z = model(x, add_noise=True)40total_loss, mse, l1 = compute_loss(clean_x=x, recon_x=reconstructed, latent=z)41
42print(f"Total loss: {total_loss.item():.4f} (MSE: {mse:.4f}, Sparsity: {l1:.4f})")43# -> Total loss: 1.0583 (MSE: 1.0545, Sparsity: 0.0038)44print("Sparsity (% zeros in latent):", f"{(z == 0).float().mean().item() * 100:.1f}%")45# -> Sparsity (% zeros in latent): 51.6%

Watch Out For

Computing denoising loss against the corrupted input instead of the clean ground truth

A disastrous implementation bug in Denoising Autoencoders occurs when passing the corrupted tensor $\tilde{x}$ to the loss function instead of the original clean tensor $x$ : loss = F.mse_loss(recon_x, corrupted_x).

When this bug is present, the training loop appears to succeed: the loss decreases smoothly, and the gradients remain stable. However, instead of learning to project corrupted data back onto the clean data manifold, the network simply learns to reproduce the noise! It acts as a lossy identity filter for corrupted signals, completely failing to remove artifacts at test time. Always pass the untouched input clean_x as the target for recon_x.

The Quick Version

Vanilla Autoencoders restrict latent dimension ( $k \ll d$ ) to force the network to discard noise and isolate primary modes of data variation.
Denoising Autoencoders corrupt input data with noise and reconstruct the clean ground-truth, learning vector fields that project off-manifold points back toward high-density regions.
Sparse Autoencoders employ overcomplete latent spaces regularized with $L_1$ or KL penalties so only a tiny fraction of neurons fire, disentangling monosemantic features.