Core ML

Knowledge Distillation

A large trained model's full probability spread over every class, not just its top pick, becomes the training target a much smaller model learns to match.

A large teacher model's softened probability distribution over all classes, not just its top answer, becomes the training target for a much smaller student model

Why Does This Exist?

A photo app needs to classify plants on-device, no network round trip. The best model anyone's trained for this gets 96% accuracy — and it's 400 megabytes, because it's a large network trained on millions of labeled photos across thousands of species. A phone camera feature has a budget closer to 10 megabytes.

Training a small model from scratch on the same labeled photos is the obvious fallback, and it underperforms — a small network trained on hard labels alone tops out around 89%. The large model knows more than its single top answer reveals: when it sees a photo of a maple leaf, it doesn't just say "maple" — internally it also rates the photo somewhat similar to oak, less similar to ivy, and nothing at all like cactus. That relative similarity is real information the training labels never contained, and a hard "maple" label throws every bit of it away.

Knowledge distillation extracts that information. Instead of training the small model on the same hard labels, train it to match the large model's full output — including how confident it was about every wrong answer, not just the right one.

Think of It Like This

A seasoned reviewer's ranked notes, not just their final verdict

A veteran food critic tastes a dish and delivers a verdict: "good." That's the hard label — one word, no texture.

But ask the critic to talk through the tasting and you get something richer: mostly reminds me of a classic version of this dish, a little like a fusion variant I had once, nothing at all like the bland version I had last week. That ranked comparison carries far more about what the critic actually noticed than the one-word verdict does.

A trainee learning to judge the same dishes learns faster from the ranked notes than from a stack of one-word verdicts, because the ranking reveals which distinctions the expert's judgment actually turns on. Knowledge distillation gives a small model the ranked notes instead of the verdict.

How It Actually Works

Softening the teacher's output with temperature

A trained classifier's raw output, before the final softmax, is a vector of scores called logits — one per class. Ordinary softmax turns those into a probability distribution, but for a confident, well-trained model that distribution is usually so peaked that the runner-up classes round to nearly zero, hiding exactly the relative-similarity information distillation wants to use.

Temperature fixes this by dividing the logits before the softmax:

p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}

$z_i$ is the logit for class $i$ , and $T$ is the temperature — 1 recovers ordinary softmax, and raising $T$ above 1 spreads probability mass out across the runner-up classes without changing which class ranks highest. A temperature of 4 or 5 is a common starting point; the right value is something you sweep, not something you can guess from the class count.

The distillation loss

Train the student on a combination of two terms: a soft-target loss against the temperature-raised teacher distribution, and the ordinary hard-label loss against the true class. The soft-target term is what carries the extra signal — it's cross-entropy between the student's and teacher's temperature-raised distributions, so the student is pushed to reproduce the whole shape of the teacher's output, not just its peak.

\mathcal{L} = \alpha \, \mathcal{L}_{\text{hard}}(y, p_{\text{student}}) + (1-\alpha) \, T^2 \, \mathcal{L}_{\text{soft}}(p_{\text{teacher}}^{T}, p_{\text{student}}^{T})

$\alpha$ weights how much the student trusts the true labels versus the teacher's softened output, and the $T^2$ factor rescales the soft-target gradient back to a comparable size, since raising $T$ shrinks the gradients from that term. Both hyperparameters are tuned per problem — there's no default that transfers.

Why the runner-up classes are worth training on at all

A hard label says "this is a maple leaf" and nothing else. A softened teacher output says "this is a maple leaf, and by the way it's more similar to oak than to cactus." That second fact never appears anywhere in the original labeled dataset — no annotator was asked to rank similarity between wrong answers. It's a byproduct of the teacher having seen far more examples and built a finer-grained sense of which mistakes are reasonable and which aren't. Distillation is the mechanism for transferring specifically that byproduct, sometimes called dark knowledge, into a model too small to have learned it on its own.

Show Me the Code

Raising the temperature on one teacher's logits, and watching the runner-up classes gain probability mass.

1import numpy as np2
3
4def softmax_temp(logits: np.ndarray, temperature: float) -> np.ndarray:5    """Softmax with temperature scaling: T=1 is ordinary softmax."""6    scaled = logits / temperature7    scaled = scaled - scaled.max()  # subtract max for numerical stability8    exp = np.exp(scaled)9    return exp / exp.sum()10
11
12logits = np.array([4.2, 3.9, 0.8, 0.3, -0.5])  # one teacher's logits, 5 classes13for t in (1.0, 4.0):14    p = softmax_temp(logits, t)15    print(f"T={t}: top-1={p.max():.3f}  probs={np.round(p, 3)}")16# -> T=1.0: top-1=0.554  probs=[0.554 0.411 0.019 0.011 0.005]17# -> T=4.0: top-1=0.329  probs=[0.329 0.305 0.141 0.124 0.102]

At T=1 the top two classes already hold 96.5% of the mass between them, and the remaining three classes are barely distinguishable from zero. At T=4 all five classes carry meaningful probability, and the relative order — class 1 above class 2 above class 3, and so on — is unchanged. That preserved ranking, spread out enough to actually train on, is the entire point of the temperature.

Watch Out For

Skipping the hard-label term entirely

Training only against the teacher's soft targets, with no weight on the true labels, ties the student's ceiling to the teacher's accuracy — the student can at best match a possibly-flawed teacher and has no signal correcting its mistakes. Keep some weight on the hard-label loss so the student can outperform teacher errors it happens to disagree with correctly.

Distilling from a teacher trained on a different label distribution

If the teacher was trained on a dataset with different class proportions than the student's actual deployment target, its "dark knowledge" about which mistakes are reasonable reflects that mismatch too — and the student inherits a skew that has nothing to do with the task at hand. Check that teacher and student are trained toward the same label distribution, or at least evaluate the student against the true deployment distribution before trusting the soft-target gains.

The Quick Version

A teacher's raw softmax output is usually too peaked to carry useful information about runner-up classes; temperature scaling spreads it out without changing the ranking.
The distillation loss combines a soft-target term against the teacher's temperature-raised distribution with the ordinary hard-label loss.
The extra signal is "dark knowledge" — relative similarity between wrong answers, something the original hard labels never captured.
Keeping some weight on the hard-label term lets the student exceed the teacher on cases where the teacher itself is wrong.
Distillation compresses capability, not architecture — the student can have a completely different structure from the teacher.