Masked Autoencoder Vision

Hide three quarters of an image and force the net to paint the missing pixels. Guessing so much from so little teaches shape and context.

MAE hides three quarters of patches and rebuilds pixels from the rest

Why Does This Exist?

Language models learn by filling blanks, but vision blanks are easy to copy from neighbors. MAE raises the rate to $75$ percent masked, so copying fails and real structure must be learned. The encoder sees only $25$ percent, which also trains three times faster. Later object detection and semantic segmentation fine-tunes start from that encoder.

This page covers pixel rebuilding only. Token-prediction variants are separate pages in this wave.

The one idea you need first.

A patch is a $16 \times 16$ square turned into one token. Mask ratio $r = 0.75$ means $147$ of $196$ tokens hide for a $224$ photo. Loss is mean squared error on masked pixels only.

Think of It Like This

A restorer with most of a fresco missing

A restorer gets a wall with three quarters chipped away. Small holes could be dabbed over, but huge gaps force study of poses, folds, and perspective across the whole wall. The rebuilt wall proves the study.

Visible patches are the chips that remain, the decoder is the restorer, and pixel error is the inspection. Where it stops: fresco gaps stay fixed, while MAE draws a fresh random mask each pass.

How It Actually Works

Split $x$ into $N = 196$ patches. Sample mask $M$ with $|M| = 0.75N$ . Encode only visible $x_{vis}$ with ViT $f$ . Pad with learned mask tokens and decode with light transformer $g$ to $\hat{x}$ . Pay $L = \sum_{i \in M} \lVert \hat{x}_i - x_i \rVert_2^2 / |M|$ on normalized pixels.

In plain words: $f$ is the big reader, $g$ is the small painter, and $M$ is the hidden set scored each step.

Worked example

Take one masked patch with true normalized pixels $[0.5, -0.3, 0.8]$ and prediction $[0.4, -0.1, 0.6]$ . Errors are $0.1, -0.2, 0.2$ . Squares are $0.01 + 0.04 + 0.04 = 0.09$ , mean $0.03$ . Over $147$ patches the mean stays near $0.03$ early and falls below $0.01$ when context clicks. Only global shape lowers every patch at once.

Watch Out For

Linear probing MAE and calling it weak

MAE features are built for fine-tuning, not frozen probes. Linear tests trail contrastive runs by many points. The symptom is poor frozen accuracy with top fine-tune scores. Fine-tune the backbone for pretrained models use instead of judging frozen numbers.

The Quick Version

MAE masks 75 percent of patches and encodes the rest.
A light decoder rebuilds normalized pixels with mean squared error.
High masking stops copying from neighbors.
Encoding only visible patches speeds training.
Features shine after fine-tuning, not frozen probing.