Visual explainer

Knowledge Distillation

Big teachers cannot fit on phones. See how softened odds carry hidden structure into a small student that ships.

Your best model is too big to fit on a phone.

Your giant teacher nails answers but needs a server rack. Phones and cheap boxes cannot hold it. You need its brain inside a body that actually ships.

Soft Labels Carry Structure

Soft labels say a dog is closer to a cat than to a car.

Hard labels call a dog 1 and all else 0. Soft labels spread small odds: dog 0.7, cat 0.2, car 0.1. That spread hides dark knowledge about which wrong answers look alike.

Soften With Real Numbers

$Logits 2, 1, 0 divided by 4 soften to 0.42, 0.33 and 0.25.$

Logits 2, 1, 0 divided by 4 soften to 0.42, 0.33 and 0.25.

Here is the focal math. Logits 2, 1, 0 at temperature 1 give sharp odds near 0.67, 0.24, 0.09. Divide by 4 first and they soften to 0.42, 0.33, 0.25. Hidden structure now shows.

Student Copies Soft Odds

Your small student trains on soft targets plus a few hard labels.

Freeze your teacher and let your small student study its softened odds. Mix in a few hard labels so truth anchors the copy. The student copies reasoning instead of final picks alone.

Same Accuracy Smaller Body

$The shrunken student keeps most teacher accuracy at a fraction of size.$

The shrunken student keeps most teacher accuracy at a fraction of size.

Same test, smaller body. Teacher calls 10 of 10 right at full size while your student calls 9 of 10 at one tenth the weights. That kept accuracy inside a shippable body is the whole payoff.

Dial Temperature With Care

Raise temperature to reveal structure, lower it when the student must commit.

Raise temperature near 4 when wrong-answer structure matters most. Lower it toward 1 when your student must commit sharply. Flip high only while soft odds still guide better than hard ones.

Where It Breaks

A tiny student still caps out below its teacher on hard tasks.

Small bodies cap learning. Below some size your student cannot hold the teacher logic, and gaps stay stuck. When scores plateau far under the teacher, your student needs more capacity itself.

The Quick Version

Giant teachers cannot ship on phones.
Soft labels reveal wrong-answer structure.
Temperature 4 softens odds to study.
Students copy soft plus hard targets.
Small bodies keep most accuracy.
High temperature reveals, low commits.
Tiny students still cap below teachers.