BEiT Pretrains by Masked Visual Tokens

BEiT masks image patches and predicts discrete visual tokens, so transformers learn semantics before labels appear.

Masked patches predict discrete tokens from a codebook before fine-tuning

Why Does This Exist?

Labeling millions of photos costs months. Masked pretraining uses raw pixels instead, but pixels regress to blurry averages.

BEiT discretizes first. It feeds pretrained models, lifts image classification and semantic segmentation, and complements dinov2 distillation.

Think of It Like This

Fill-in blanks with a word bank

You blank out 40 percent of a paragraph and pick each missing word from a fixed bank of 8192 words. Guessing exact letters is vague, but picking bank words forces meaning.

Patches are words, the dVAE bank is vocab, and prediction is the quiz. Where it stops: a weak bank teaches weak words, just as a weak tokenizer caps BEiT.

How It Actually Works

Tokenize each $16 \times 16$ patch with a discrete VAE into $z \in [0, 8191]$ . Mask $40$ percent of $196$ tokens (about $78$ ). Train ViT to predict $z$ with cross entropy on masked spots only, then fine-tune with labels.

Worked example

$196 \times 0.40 = 78.4$ , so $78$ masked. Suppose model gives true token logit $3.0$ versus two distractors $1.0, 0.0$ . Softmax is $e^3 = 20.09$ , $e^1 = 2.72$ , $e^0 = 1$ , sum $23.81$ , prob $20.09/23.81 = 0.844$ . Loss $-\log 0.844 = 0.170$ . Averaged over $78$ spots, semantics emerge without labels.

Watch Out For

Reusing a mismatched tokenizer for new domains

A natural-photo tokenizer on medical scans maps distinct lesions to one code. Symptom: fine-tune stalls below supervised baseline. Fix: retrain or adapt the tokenizer on target imagery before BEiT pretraining.

The Quick Version

BEiT predicts discrete visual tokens for masked patches, not raw pixels.
Masks cover 40 percent, about 78 of 196 tokens per image.
Fixture true-token prob 0.844 gives loss 0.170 per masked patch.
Tokenizer quality bounds the whole run.
Fine-tunes then transfer to classification and segmentation.