Real-ESRGAN

Real-ESRGAN upscales messy real photos by training on synthetic degradations stacked twice, so blur, noise, and compression all appear together as in life.

Two stacked rounds of blur, noise, and compression synthesize training pairs that look like real damaged photos

Why Does This Exist?

ESRGAN dazzled on benchmarks and stumbled on phones. Its training pairs used clean bicubic downsampling, but real photos arrive pre-damaged: lens blur, sensor noise, aggressive JPEG, sharpening halos. The benchmark-to-street gap made the model a lab curiosity.

Real-ESRGAN keeps the generator and rebuilds the data. A high-order pipeline degrades clean images twice in a row with randomised blur, noise, resize, and JPEG at each stage, plus sinc filters that reproduce ringing and overshoot artefacts. Train on that mess and real mess becomes familiar. The task framing is super-resolution.

Think of It Like This

Training lifeguards in storm pools

Calm-pool lifeguards fail their first ocean rescue: currents, cold, and panic were never in training. Storm-pool training runs recruits through waves, undertow drills, and night swims, twice over, so the ocean feels like a familiar bad day.

Bicubic pairs are the calm pool. Second-order degradation is the storm schedule. It fails when storms exceed the script: damage types outside the pipeline, exotic codecs or motion blur, still surprise the swimmer.

How It Actually Works

Each training pair starts from a clean high-resolution image $y$ . Stage one applies blur (isotropic, anisotropic, or sinc kernel), downsampling, Gaussian or Poisson noise, then JPEG compression, all with randomised strengths. Stage two repeats the full sequence on the result, compounding damage the way phone pipelines do:

x = D_2(D_1(y)), \qquad D_i = \mathrm{JPEG}(\mathrm{noise}(\mathrm{resize}(\mathrm{blur}(\cdot))))

Sinc filters deserve special mention: their oscillating tails reproduce the ringing halos around text and edges that cheap sharpening leaves, artefacts other kernels cannot synthesise. The generator, discriminator, and perceptual losses carry over from ESRGAN nearly unchanged; the data is the contribution.

Worked example

One synthesis run: stage-one Gaussian blur $\sigma = 1.5$ , downsample $4\times$ , noise $\sigma = 8/255$ , JPEG quality $70$ ; stage-two blur $\sigma = 1.0$ , JPEG quality $60$ , sinc cutoff ringing near text edges. The pair now shows soft detail, grain, block boundaries, and halos together, matching a two-year-old phone photo of a menu board. A bicubic-trained model reads the blocks as content and sharpens them; the Real-ESRGAN model reads them as damage and removes them.

Code

Two-stage degradation sketch, showing damage compounding.

1import numpy as np2
3
4def degrade(x: np.ndarray, blur: float, noise: float) -> np.ndarray:5    """One cartoon degradation stage: soften, then grain."""6    soft = (1 - blur) * x + blur * x.mean()  # blur pulls toward local mean7    return soft + noise * np.ones_like(x)  # grain lifts everything8
9
10x = np.array([0.9, 0.2])11once = degrade(x, 0.3, 0.02)12twice = degrade(once, 0.2, 0.03)13print(np.round(twice, 3))  # -> [0.686 0.316] damage visibly compounds

Watch Out For

Over-smoothing damage the pipeline never modelled

Exotic damage, motion blur streaks, chromatic aberration, unseen codecs, falls outside the script. The model treats it as content and either preserves it lovingly or melts it into plastic.

Scope claims to the modelled damage family. For new damage, extend the pipeline first; no network outlearns its data script.

Faces that lose identity to generic priors

At $4\times$ on tiny faces the model substitutes plausible generic features: wrong eyes, smoothed moles. The output looks like a clean photo of a slightly different person.

Route faces to dedicated face-restore models past a size threshold, or flag identity risk explicitly. Generic priors and identity are natural enemies.

The Quick Version

Real-ESRGAN changes the data, not the network: two stacked degradation rounds.
Each round randomises blur, resize, noise, and JPEG; sinc filters add ringing.
Our toy pixel falls $0.9 \to 0.686$ across two stages of compounding damage.
Bicubic-trained models sharpen blocks as content; storm-trained models remove them.
Unmodelled damage and tiny-face identity loss are the standing limits.