SimMIM Pixel Rebuilding

Mask large squares of a Swin input and ask one linear layer to repaint raw pixels. Simple targets plus window attention still teach layout.

SimMIM masks patches and predicts raw pixels with a light head

Why Does This Exist?

Heavy decoders add tuning and cost. SimMIM shows raw pixels plus one linear head suffice when the mask is large and the backbone is a Swin pyramid. Object detection gains follow without tokenizer or discrete codes.

This page covers SimMIM only. Heavier decoder variants are the MAE page in this wave, and backbones are in pretrained models.

The one idea you need first.

Mask ratio $r = 0.6$ over $32 \times 32$ patches hides more than half the photo. Loss is $L_1 = |\hat{x} - x|$ averaged on masked pixels. A linear head is one matrix with no hidden layers.

Think of It Like This

Patching drywall with one trowel pass

A plasterer covers taped squares with one flat pass and no sculpting tools. Big holes still force care about edges and planes, while the single tool keeps the job honest about what the wall taught.

Masked squares are the holes, the linear head is the trowel, and $L_1$ is the straightedge. Where it stops: drywall stays put, while SimMIM remasks randomly each iteration.

How It Actually Works

Tokenize with $32$ pixel patches, mask $60$ percent, encode all tokens with Swin $f$ using shifted windows, and predict masked pixels with linear $h$ . Pay mean $L_1$ on masked spots. Masked positions keep a learnable filler during encoding.

In plain words: $f$ reads around holes with local windows, $h$ repaints, and $r$ sets hole share.

Worked example

One masked pixel true $0.70$ , predicted $0.55$ . $L_1$ is $|0.15| = 0.15$ . A $32 \times 32$ patch holds $1024$ pixels; if mean $|e| = 0.12$ , patch loss is $0.12$ . Falling below $0.06$ means edges and shading now come from context, since copying is blocked by the $0.6$ mask.

Watch Out For

Small masks that invite copy-paste

A $0.15$ mask lets the net interpolate neighbors and loss falls with no transfer. Fine-tune scores on semantic segmentation then stall. Keep the $0.6$ ratio and $32$ patch size together.

The Quick Version

SimMIM masks 60 percent of 32 pixel patches in a Swin backbone.
One linear head rebuilds raw pixels with L1 loss.
No tokenizer or heavy decoder is used.
Large masks force context use.
Fine-tuned detection and segmentation rise after pretraining.