Knowledge Distillation
Training a small student model to match a larger teacher's full softened output distribution, not just its top prediction, transferring more signal than hard labels.
A large, already-trained teacher model's output is softened with a temperature parameter before the softmax, spreading probability mass across runner-up classes instead of collapsing almost entirely onto the top prediction. A smaller student model then trains against that softened distribution alongside the ordinary hard labels, picking up the teacher's relative confidence across wrong answers, sometimes called dark knowledge.
The technique compresses capability rather than architecture, so the student can use a completely different network structure from the teacher it's learning from.