Skip to content
AI360Xpert
Core ML
Visual explainer

Knowledge Distillation

Training a small model to mimic a larger one by matching its probability distribution.

A massive teacher model is highly accurate but too large and slow to run on edge devices.
A massive teacher model is highly accurate but too large and slow to run on edge devices.

Large language models and massive vision networks achieve incredible accuracy, but their size makes them impossible to deploy on mobile phones, edge devices, or cheap servers. The latency is too high, and the memory footprint simply exceeds the hardware's limits.

Dark Knowledge

Soft targets reveal the teacher's dark knowledge about how similar incorrect classes are.
Soft targets reveal the teacher's dark knowledge about how similar incorrect classes are.

When a model predicts a class, it doesn't just output a single '1' for the right answer. It outputs a probability distribution. The tiny probabilities assigned to incorrect answers—like knowing a dog is more similar to a cat than a car—contain rich structural information called dark knowledge.

Temperature Scaling

Dividing logits by a Temperature greater than 1 softens the probabilities for the student.
Dividing logits by a Temperature greater than 1 softens the probabilities for the student.

To transfer this knowledge, we divide the teacher's pre-softmax outputs (logits) by a Temperature parameter. This mathematically softens the distribution, amplifying the tiny probabilities. The student model is then trained to match this exact soft distribution, forcing it to learn the teacher's internal logic rather than just the final hard labels.

Edge Deployment

The student model achieves near-teacher accuracy while running directly on edge devices with minimal latency.
The student model achieves near-teacher accuracy while running directly on edge devices with minimal latency.

Because the student learns the nuanced landscape of the problem from the teacher, it converges to a much higher accuracy than it would if trained from scratch on the original data. The result is a highly compressed model that fits in memory and runs instantly on small hardware.

Where It Breaks

If the student model lacks the raw parameter capacity, it cannot absorb the teacher's knowledge.
If the student model lacks the raw parameter capacity, it cannot absorb the teacher's knowledge.

Distillation can only go so far. If the student model is fundamentally too small, it lacks the representational capacity to model the teacher's complex decision boundaries. The soft targets become too nuanced for the student's limited parameters, resulting in a strict ceiling on performance.

The Quick Version

  • The Problem: Best models are too heavy for deployment.
  • The Primitive: Soft probabilities hold rich relationship data.
  • The Mechanism: Temperature scaling amplifies hidden knowledge.
  • The Payoff: Teacher accuracy inside a tiny footprint.
  • The Limit: The student still needs sufficient baseline capacity.

What to Read Next