DeiT Trains Vision Transformers Fast

DeiT adds a distillation token that copies a CNN teacher, so vision transformers learn ImageNet in days instead of months.

A distillation token learns teacher soft targets beside class token over 196 patches to lift accuracy

Why Does This Exist?

Plain ViT needs 300M JFT photos to beat CNNs. Most labs have 1.2M ImageNet photos and a week of GPUs, so ViT underfits there.

DeiT keeps ViT shape and changes training. It pairs image classification with a CNN teacher, reuses convolutional layers bias via distillation, and ships pretrained models others fine-tune.

Think of It Like This

A student with answer key plus tutor hints

You study past exams alone and score 79. Then a tutor shows not just answers but how confident each option is. You learn which distractors are close and jump to 81.

Patches are questions, class token is your answer, and distill token is tutor hints. Where it stops: a wrong tutor teaches wrong confidence, just as a weak teacher caps the student.

How It Actually Works

Split $224 \times 224$ into $16 \times 16$ patches for $N = 196$ tokens. Add class token $c$ and distillation token $d$ , plus position codes. Both tokens train classifiers. Loss mixes true labels and teacher:

L = 0.5\, CE(s_c, y) + 0.5\, KL(s_d / T, t / T)

$s_c$ and $s_d$ are student heads, $t$ teacher soft targets, $T$ temperature, $CE$ cross entropy, $KL$ divergence. Strong augmentation RandAugment plus Mixup plus CutMix supplies data variety.

Worked example

Student logits $[2.0, 1.0, 0.1]$ softmax to $[0.659, 0.242, 0.099]$ since $e^2 = 7.39$ , $e^1 = 2.72$ , $e^0 = 1$ , sum $11.11$ . Teacher gives $[0.70, 0.25, 0.05]$ . Cross term is $-(0.70 \log 0.659 + 0.25 \log 0.242 + 0.05 \log 0.099) = 0.292 + 0.355 + 0.116 = 0.763$ . The distill token chases 0.763 while class token chases the hard label, and the pair beats either alone.

Watch Out For

Distilling from a weaker teacher than the student

A small CNN teacher below student capacity drags accuracy down. Symptom: distilled run underperforms plain training. Fix: use RegNetY level teacher or turn distillation weight down when teacher top-1 trails student by points.

The Quick Version

DeiT adds a distillation token beside class token over 196 patch tokens.
Loss mixes hard labels and teacher soft targets with temperature scaling.
Fixture KL term 0.763 teaches which distractors are close.
Strong augmentation replaces JFT scale for 1.2M ImageNet training.
Teacher quality caps gains, so match teacher above student target.