Knowledge Distillation

Train a tiny model to copy a big trained model's full probability spread across every class, not just its top pick. The runner-up scores teach distinctions the hard labels never recorded.

Teacher logits 4.2 and 3.9 soften from 0.554 and 0.411 at T of 1 to 0.329 and 0.305 at T of 4, and the small student trains on that spread

Why Does This Exist?

A photo app must name plants on the phone with no network round trip. The best model for the job scores 96 percent and weighs 400 megabytes, trained on millions of photos across thousands of species. The phone budget sits near 10 megabytes.

Training a small network from scratch on the same labelled photos is the obvious fallback, and it lands near 89 percent. The large model knows more than its top answer shows. Shown a maple leaf, it also rates the photo somewhat like oak, less like ivy, nothing like cactus. That ranking of wrong answers is genuine information, and a hard maple label throws all of it away.

Distillation recovers it. Train the small model to match the large model full output, including the confidence on every wrong answer, and the student learns distinctions no annotator ever wrote down.

This page covers that transfer. Sibling ways to shrink models are model pruning, which deletes weights, and quantization-aware training, which shrinks number precision. Reusing learned structure for a new task rather than compressing for the same one is transfer learning.

Think of It Like This

Ranked tasting notes versus one-word verdicts

A veteran food critic tastes a dish and files the verdict good. One word, no texture. That is the hard label.

Ask the critic to talk through the tasting instead. Mostly like the classic version of this dish, a little like a fusion variant tried once, nothing like the bland one from last week. That ranked comparison carries what the expert actually noticed: which distinctions the judgment turns on.

A trainee judge learns faster from a stack of ranked notes than from a stack of verdicts, because the notes reveal near misses the verdicts hide. The veteran is the teacher, the trainee is the student, the verdicts are hard labels, and the ranked notes are soft targets.

Where it stops: the critic can explain a ranking in words. The teacher only outputs numbers, so the student inherits the teacher blind spots too, with no commentary attached.

How It Actually Works

Call the teacher raw scores $z$ , meaning one logit per class before the final softmax. Call the temperature $T$ , meaning the divisor that softens the output. Ordinary softmax is $T = 1$ . The softened teacher distribution is:

p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}

Here $z_i$ is the logit for class $i$ and $p_i$ is the teacher probability for that class. Raising $T$ spreads mass across runner-up classes without changing their order. Values of 4 or 5 are common starting points, chosen by sweep rather than guessed from the class count.

The student trains on two terms at once. The hard term is ordinary cross-entropy against the true label. The soft term is cross-entropy against the teacher temperature-raised distribution, scaled by $T^2$ because higher temperatures shrink gradients:

\mathcal{L} = \alpha \, \mathcal{L}_{\text{hard}}(y, p_{\text{student}}) + (1-\alpha) \, T^2 \, \mathcal{L}_{\text{soft}}(p_{\text{teacher}}^{T}, p_{\text{student}}^{T})

Here $y$ is the true label, $\alpha$ weights trust in labels versus teacher, and both $T$ and $\alpha$ are tuned per problem. The softmax turning logits into probabilities is the one prerequisite; the rest is built on top of it.

Worked example: one teacher, five classes

One teacher produces logits for five species:

z = [4.2, 3.9, 0.8, 0.3, -0.5]

At $T = 1$ , exponentiate and normalise:

p = [0.554, 0.411, 0.019, 0.011, 0.005]

The top two classes hold 96.5 percent of the mass between them and the last three are nearly zero. At $T = 4$ , divide first, then exponentiate:

p^{4} = [0.329, 0.305, 0.141, 0.124, 0.102]

Every class now carries trainable mass while the ranking never moves. The student sees that class 3 outranks class 5 by a real margin, a fact the hard label never contained. The diagram draws these five bars at their $T = 4$ heights, with the peaked $T = 1$ pair quoted beneath for contrast.

Why runner-up classes are worth training on

A hard label says this is maple and nothing else. The softened output adds that it resembles oak more than cactus. No annotator ranked wrong answers, so that similarity never appears in the dataset. It is a byproduct of the teacher having seen far more examples, sometimes called dark knowledge. Distillation exists to move exactly that byproduct into a model too small to have learned it alone, and the visual sequence in knowledge distillation visualized walks the same transfer beat by beat.

Code

The point is the softening: one logit vector, two temperatures, and the runner-up classes gaining mass.

1import math2
3logits: list[float] = [4.2, 3.9, 0.8, 0.3, -0.5]4
5def softened(logits: list[float], temp: float) -> list[float]:6    scaled: list[float] = [z / temp for z in logits]7    top: float = max(scaled)8    exps: list[float] = [math.exp(s - top) for s in scaled]9    total: float = sum(exps)10    return [round(e / total, 3) for e in exps]11
12print(softened(logits, 1.0))  # -> [0.554, 0.411, 0.019, 0.011, 0.005]13print(softened(logits, 4.0))  # -> [0.329, 0.305, 0.141, 0.124, 0.102]

The printout matches the hand computation. At $T = 1$ the tail is dead; at $T = 4$ the smallest class holds 0.102. Nudge the temperature to 8 and watch the five values flatten further while their order stays put.

Watch Out For

Dropping the hard-label term

Training only on soft targets caps the student at the teacher accuracy. Wherever the teacher is wrong, the student is pulled wrong with no counter-signal, and the final model inherits every teacher blind spot at a smaller size.

Symptom: student accuracy tracks teacher errors case for case, including confident teacher mistakes. Fix: keep weight on the hard-label loss so true labels can overrule the teacher where they disagree.

Distilling from a mismatched teacher

A teacher trained on different class proportions teaches its own skew as dark knowledge. The student then inherits a sense of which mistakes are reasonable that belongs to another distribution entirely.

Symptom: distilled student underperforms a from-scratch small model on the deployment mix despite winning on the teacher validation mix. Fix: confirm teacher and student target the same label distribution, and evaluate the student on deployment data before trusting the soft-target gains.

The Quick Version

Distillation trains a small student on the teacher full softened distribution, not just its top pick.
Our logits 4.2 and 3.9 give 0.554 and 0.411 at T of 1 but 0.329 and 0.305 at T of 4, with all five classes trainable.
Temperature spreads mass without reordering classes, and the T squared factor keeps the soft-term gradients comparable.
The extra signal is dark knowledge about runner-up similarity that hard labels never recorded.
Keeping hard-label weight lets the student overrule teacher mistakes instead of inheriting them.
Distillation transfers capability across architectures, so teacher and student need not share a structure.