DINOv2 Self Distillation

Train a student to copy a slow teacher on big and small crops, at both image and patch level. Scale plus curation makes frozen features excel.

DINOv2 distills patch features from a slow teacher at large scale

Why Does This Exist?

Small self-supervised runs give good image tags but weak dense maps. DINOv2 scales data to $142$ million curated photos and adds a patch-level match, so frozen features serve depth, segmentation, and retrieval with no tuning. A pretrained backbone becomes a general tool, and semantic segmentation probes show the dense gain.

This page covers student-teacher plus patch loss only. Masked rebuilding variants are separate pages in this wave.

The one idea you need first.

Self-distillation means the teacher is a slow copy of the student, updated by momentum. Class token $c$ names the image; patch tokens $p_1$ through $p_N$ name small regions. Both get matched.

Think of It Like This

An apprentice copying a master on big and small tiles

An apprentice copies a master mosaic on the full wall and on single tiles. The master works slowly and never explains, yet the apprentice learns both scene sense and tile craft by matching at two scales.

Student and teacher are the pair, class loss is the wall, and patch loss is the tile. Where it stops: mosaics stay still, while crops and blur keep changing the views.

How It Actually Works

From each photo cut global and local crops. Student predicts teacher distributions for class tokens with cross entropy and for masked patch tokens with a second head. Teacher weights follow by momentum, centered and sharpened. Data curation to LVD-142M removes near duplicates and balances scenes.

In plain words: global crops teach layout, local crops teach detail, and curation keeps the $142$ million useful.

Worked example

Say class logits student $[2.0, 1.0, 0.1]$ versus teacher soft target $[0.70, 0.25, 0.05]$ . Student softmax is $[0.659, 0.242, 0.099]$ . Cross entropy is $-(0.70 \log 0.659 + 0.25 \log 0.242 + 0.05 \log 0.099) = 0.292 + 0.355 + 0.116 = 0.763$ . Patch terms add a similar $0.7$ per masked patch, so dense detail carries real weight in the total.

Watch Out For

Using class features for dense tasks

Class tokens pool the scene and smear boundaries. Depth maps then look blobby. The symptom is fine retrieval with weak segmentation. Read patch tokens for dense heads and keep class tokens for global search in image retrieval.

The Quick Version

DINOv2 distills class plus patch tokens from a momentum teacher.
Training spans 142 million curated photos with global and local crops.
Patch loss is what lifts dense tasks like depth and segmentation.
Frozen backs work without fine-tuning.
Curation matters as much as scale.