Skip to content
AI360Xpert
Beta
Core ML

Visual explainer

Loss Functions

How a network scores its own mistakes so gradients know which way to push.

No loss number means no direction to learn from.
No loss number means no direction to learn from.

You get a prediction and a truth but no grade. Without one number that says how bad you were, backprop has no slope to follow. You pick the ruler before you train.

Square Big Misses

Errors of 2 and 0.5 become losses of 4 and 0.25.
Errors of 2 and 0.5 become losses of 4 and 0.25.

You subtract truth from prediction, then square it. A miss of 2 becomes 4 while a miss of 0.5 becomes 0.25. The big miss now shouts sixteen times louder than the small one.

Punish False Confidence

Confident and wrong at 0.05 costs 3.00, not 0.11.
Confident and wrong at 0.05 costs 3.00, not 0.11.

Truth is 1 but you said 0.05, so loss is minus log 0.05, which is 3.00. Had you said 0.9, loss would be 0.11. Cocky and wrong pays twenty seven times more.

Flat Absolute View

Absolute error treats both misses the same at 2.0.
Absolute error treats both misses the same at 2.0.

You drop the square and take distance only. Misses of 2 and minus 2 both cost 2.0. Outliers stop yanking you, but you lose the loud alarm squared error gives.

Right Ruler Learns

The right loss pulls hard where it matters most.
The right loss pulls hard where it matters most.

Train one head with squares on prices and one with cross-entropy on odds. Price errors above 2 collapse fast while timid odds sharpen quickly. Each loss fixes its own kind fastest.

Match Loss To Task

Use squared error for numbers and cross-entropy for odds.
Use squared error for numbers and cross-entropy for odds.

You reach for squared error when you predict amounts. You switch to cross-entropy when you predict odds. Flip when your classes unbalance, then weight the rare class higher.

Where It Breaks

Outliers yank squared error and confident misses explode cross-entropy.
Outliers yank squared error and confident misses explode cross-entropy.

One bad label with error 10 costs 100 and drags your line. One confident 0.01 on truth 1 costs 4.61 and shakes training. You clip, clean, or reweight before you trust either.

The Quick Version

  • No grade means no gradient to follow.
  • Miss 2 becomes loss 4.
  • Miss 0.5 becomes loss 0.25.
  • Odds 0.05 on truth cost 3.00.
  • Odds 0.9 on truth cost 0.11.
  • Squares for amounts, logs for odds.
  • Outliers and cocky misses distort training.