TinyViT Distills Giants Into Phones

TinyViT saves teacher logits plus augments offline, so tiny transformers learn giant accuracy without giant training.

Saved logits plus offline augmentation gain 4 points at phone size

Why Does This Exist?

Training tiny ViTs with live teachers costs teacher forward passes each step. Large teachers plus Mixup each batch strain clusters.

TinyViT saves logits once. It follows deit distillation, fits efficient architectures, and ships pretrained models tiny variants.

Think of It Like This

Canning summer fruit for winter meals

You cook harvest once into jars, then eat fast all winter without farming daily. Upfront work pays daily speed.

Teacher logits are jars, saved augments are labels. Where it stops: jars fix menus, just as saved logits fix teachers.

How It Actually Works

Run giant teacher over augmented ImageNet once, save top logits plus aug params. Train TinyViT-5M/11M/21M with CE to saved logits plus hard labels. No live teacher passes.

Worked example

Teacher forward $20$ ms per batch live adds $20$ hours over $3.6$ M steps? Saved cuts it to $0$ . Tiny-21M moves $80$ to $84$ percent, a $4$ point gain. Storage $100$ GB logits trades for $40$ percent faster training in paper setups.

Watch Out For

Reusing stale logits after changing augmentation

Saved params mismatch new pipelines and labels misalign. Symptom: sudden 2 point drops after pipeline edits. Fix: regenerate logits when augmentation changes.

The Quick Version

TinyViT pre-saves teacher logits plus augmentation to skip live teachers.
Tiny 5M to 21M variants gain about 4 points to 84 percent.
Fixture $80$ to $84$ percent shows distillation value.
Storage trades for 40 percent faster training.
Regenerate jars when pipelines change.