ESRGAN

ESRGAN trades pixel-perfect averages for believable texture by judging relative realism and matching deep features instead of raw pixels.

Dense residual blocks feed a relativistic critic and a feature loss that together reward believable texture

Why Does This Exist?

SRCNN and its MSE descendants hit a perceptual wall: higher PSNR, waxier images. Users preferred visibly textured outputs that scored worse, and the field needed a recipe that optimised what eyes want instead of what squared error measures.

ESRGAN supplied it. Residual-in-residual dense blocks (RRDB) without batch normalisation carry rich features, a relativistic discriminator judges real-versus-fake relatively rather than absolutely, and perceptual loss matches VGG features before activation. Bricks regain mortar lines and hair regains strands. The task framing is super-resolution, and the real-world-degradation successor is Real-ESRGAN.

Think of It Like This

A court painter versus a portrait photographer

An MSE painter renders the mathematically average sitter: smooth, inoffensive, slightly waxy. A perceptual photographer captures pores and flyaway hairs, technically noisier per pixel, unmistakably alive. Sitters always pick the photograph.

The relativistic critic is the photographer's eye for aliveness, and the feature loss is the insistence on pores over smoothness. The sitting fails when the photographer invents beauty the sitter lacks, hallucinated texture that reads as detail but matches nothing real.

How It Actually Works

The generator stacks RRDB blocks: dense connections inside residual groups inside a global residual, with all batch normalisation removed because it washed out texture variance and introduced artefacts. The relativistic discriminator $D_{Ra}$ predicts how much more realistic a real image is than a fake one on average, rather than scoring each alone:

D_{Ra}(x_r, x_f) = \sigma(C(x_r) - \mathbb{E}[C(x_f)])

$C$ is the raw critic output and $\sigma$ the sigmoid. Loss combines three terms: L1 pixel loss for grounding, perceptual loss on pre-activation VGG features for texture, and the relativistic adversarial term for realism. Pre-activation features matter because post-ReLU sparsity washes out the brightness consistency the loss should enforce.

Worked example

A brick-wall patch: the MSE model outputs mortar lines at contrast $0.05$ against bricks, nearly invisible. ESRGAN's perceptual term compares VGG edge-filter responses where the real wall spikes at $2.4$ and the blurry output sits at $0.3$ ; the $2.1$ feature gap dwarfs the $0.05$ pixel gap in the total loss, so training restores mortar contrast to $0.35$ even as PSNR drops $1.5$ dB. The numbers move opposite because they measure different things.

Code

Relativistic edge over a batch, the comparison that replaces absolute scoring.

1import numpy as np2
3
4def relativistic_edge(c_real: np.ndarray, c_fake: np.ndarray) -> np.ndarray:5    """How much more realistic each real looks than the average fake."""6    return c_real - c_fake.mean()  # sigmoid applied downstream7
8
9print(np.round(relativistic_edge(np.array([2.0, 1.0]), np.array([0.5, 1.5])), 2))  # -> [1. 0.]

Watch Out For

Hallucinated texture mistaken for recovered detail

GAN upscalers invent plausible texture that was never in the scene: readable text from blur, regular brick from rubble. The outputs look detailed and are partly fiction, dangerous in medical, forensic, or archival use.

Label GAN outputs as enhanced, never recovered. Where truth matters, ship fidelity models and disclose the tradeoff.

Batch normalisation snuck back in

Standard blocks with batchnorm reintroduce the very artefacts ESRGAN removed: blotchy flat regions and shifting statistics between training and inference batches.

Audit every block for normalisation layers when adapting the architecture. RRDB uses none by design, not by oversight.

The Quick Version

ESRGAN replaces MSE fidelity with perceptual, adversarial, and L1 losses combined.
RRDB blocks drop batch normalisation to protect texture variance.
The relativistic critic scores real-versus-fake relatively, stabilising texture learning.
Perceptual loss reads pre-activation VGG features for brightness-consistent detail.
Outputs look real but partly invent texture; never present them as recovered truth.