Distilling Foundation Teachers

Ask a huge vision teacher for soft answers plus features, then train a small net to copy them. Small models keep most skill.

Distillation teaches a small net from a foundation teacher outputs

Why Does This Exist?

Giant backs miss phones and cameras. Distillation moves their skill into small nets with soft targets that carry dark knowledge like dog versus fox closeness. Image classification students then near teachers at tenth cost.

This page covers teacher-to-student only. From-scratch training is sibling material.

The one idea you need first.

Soft targets $p = \mathrm{softmax}(z/T)$ with $T = 2$ spread labels. Loss mixes hard cross entropy with KL to teacher soft outputs plus feature $L_2$ . Weight $\alpha$ near $0.7$ favors teacher signals.

Think of It Like This

Apprenticing under a master with explained answers

A master gives not just answers but odds across choices plus sketches of reasoning. An apprentice copying odds plus sketches learns faster than one grading right or wrong alone.

Logits are odds, features are sketches, and $T$ softens certainty. Where it stops: masters err too, so student data still filters mistakes.

How It Actually Works

Freeze teacher $f_t$ . Train student $f_s$ on $L = (1-\alpha) CE(y, s) + \alpha T^2 KL(p_t, p_s) + \beta \lVert \phi_t - \phi_s \rVert^2$ . Match resolutions and augment strongly. Distill on broad data, then tune to tasks.

In plain words: $T$ spreads answers, $\alpha$ trusts teacher, and $\phi$ aligns intermediate eyes.

Worked example

Teacher $86.2$ percent, student alone $78.4$ . Distilled student $83.9$ , keeping $(83.9-78.4)/(86.2-78.4) = 5.5/7.8 = 71$ percent of the gap closed. Cost falls $10$ x while skill holds $97$ percent of teacher. The ratio justifies the extra teacher pass.

Watch Out For

Teacher errors copied as truth

Biased teachers pass blind spots and students inherit them with higher confidence. Fairness gaps widen. Audit teacher on slices and mix real labels with weight $(1-\alpha)$ before shipping.

The Quick Version

Distillation copies soft outputs plus features to small nets.
Temperature 2 spreads dark knowledge across classes.
Alpha near 0.7 balances teacher plus true labels.
Students near teachers at tenth cost.
Audit teachers or biases transfer too.