Skip to content
AI360Xpert
Beta
Core ML

Visual explainer

Learning Rate Scheduling, Visually

Start bold and finish gentle: warm the rate from 0 to 0.1, glide it down a cosine, and never decay before the loss settles.

A small fixed step crawls while a large one bounces around the minimum.
A small fixed step crawls while a large one bounces around the minimum.

One fixed rate forces a bad bargain. At 0.001 you crawl across flat ground for epochs, at 0.5 you ricochet around the minimum forever. Neither extreme finishes the job you started.

Shrink Over Time

Let the step start at 0.1 and glide all the way down to 0.
Let the step start at 0.1 and glide all the way down to 0.

The fix is a timetable, not a number. Open at 0.1 for bold early progress, pass 0.05 mid-race, and land at 0.0 to lock the weights in. Big moves early, tiny ones late, each phase doing its share.

Warm Up First

Ramp the rate from 0 to 0.1 across the first 5 epochs.
Ramp the rate from 0 to 0.1 across the first 5 epochs.

Fresh weights plus full steps spell disaster. Random gradients spike wildly at the start, so epoch 1 runs at 0.02 and you reach 0.1 only by epoch 5. Small steps until the noise settles keep the run alive.

Glide Down Smoothly

Cosine decay glides 0.1 to 0 by epoch 100 with no sudden drops.
Cosine decay glides 0.1 to 0 by epoch 100 with no sudden drops.

Step drops kick the optimiser mid-stride. A half cosine instead eases 0.1 down to 0 across epochs 5 to 100, steep early and feather soft at the end. Continuous curves never startle the weights.

Proof It Worked

The schedule settles at 0.16 while the fixed step rattles at 0.38.
The schedule settles at 0.16 while the fixed step rattles at 0.38.

Same 100 epochs, same start, different endings. The fixed 0.05 rattles around 0.38 indefinitely while the schedule glides to 0.16 and rests. Planning the step size more than halves the leftover loss.

Pick Your Shape

Exploding starts add warmup, steady budgets take the cosine, simple nets keep steps.
Exploding starts add warmup, steady budgets take the cosine, simple nets keep steps.

Match the curve to the patient. Big fresh networks need warmup plus cosine, simple sturdy nets run fine on step drops, short fine tunes take one late drop. If your starts explode, add the ramp before anything else.

Where It Breaks

Decaying at epoch 10 strands the run at 0.33 with no way back up.
Decaying at epoch 10 strands the run at 0.33 with no way back up.

Decay never reverses. Crash the rate to 0 by epoch 10 and the run strands at 0.33, with steps too tiny to climb out. The 0.16 you wanted stays visible yet unreachable. Schedule for your full budget, not your nerves.

The Quick Version

  • Fixed rates crawl or bounce.
  • Bold early, gentle late.
  • Warm 0 to 0.1 by epoch 5.
  • Cosine glides to 0 by 100.
  • Schedules reach 0.16 over 0.38.
  • Exploding starts need the ramp.
  • Early decay strands you at 0.33.