Learning Rate Warmup

Ramp the learning rate up from near zero over the first few hundred steps, because the earliest updates run on the least trustworthy gradient estimates.

A 10-step warmup climbs 0.0001 per step to the 0.001 peak, spending 0.0055 of rate budget instead of 0.0100, before decay takes over

Why Does This Exist?

Train a 12-layer transformer from scratch at its intended peak rate from step zero, and the run often dies in the first few hundred steps: loss spikes, then not-a-number, on a model that would train cleanly for the next hundred thousand steps at that exact same rate. The rate was never wrong for the run. It was wrong for step one.

At initialization, weight initialization sets every layer's variance to a sane starting point, but the network has not taken a single training step yet. The first gradients are a rough first impression of the loss surface, not a settled read. Adam makes this worse in one specific way: its second moment starts at zero and needs several steps of bias correction before it measures a parameter's gradient scale reliably, so the earliest updates get divided by an estimate that has not stabilized. A full-strength step, scaled by an unstable estimate, on never-trained weights, is the combination that blows up.

Think of It Like This

Idling a cold engine before flooring it

An engine left out overnight has cold, thick oil that has not reached the parts it protects. Floor the accelerator immediately and the engine runs hard with no protection in place, which is how metal gets scored.

Idle for a minute first while the oil warms, thins, and circulates, and the same hard acceleration afterward is completely fine. Nothing about the car changed except reaching a stable operating state first.

Cold oil is the uncorrected moment estimate, idling is the ramp from near zero, and full throttle is the peak rate. Where it stops: an engine warms itself by idling, but a model does not fix its estimates by waiting. Warmup works because each small step feeds the estimates another real gradient, so the ramp buys data, not just time.

How It Actually Works

Call the current step $t$ , meaning how many updates have run so far. Call the warmup length $W$ , meaning the number of ramp steps, commonly a few hundred to a few thousand. Call the schedule peak $\eta_{\max}$ .

The ramp itself

Warmup is almost always linear. Over the first $W$ steps the rate climbs in equal increments from near zero to the peak:

\eta_t = \eta_{\max} \cdot \frac{t}{W}, \qquad t \le W

With $W = 10$ and $\eta_{\max} = 0.001$ , the first five rates are 0.0001, 0.0002, 0.0003, 0.0004, and 0.0005. Past step $W$ the rate hands off to whatever decay runs the rest of training, which is why warmup always pairs with a schedule: ramp-up here, ramp-down there.

Why the ramp fixes the Adam instability

Trace one early Adam update: raw step size $\eta_t \cdot \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)$ , where $\hat{m}_t$ and $\hat{v}_t$ are the bias-corrected moment estimates. At step 1, $\eta_1 = \eta_{\max} / W$ is tiny by construction, so the unreliable early estimates get multiplied by a rate near zero and their damage is bounded exactly during the window where they are least trustworthy. By the time $\eta_t$ reaches full strength, $W$ steps of real gradients have corrected the estimates. Summed over the first 10 steps, warmup spends 0.0055 of rate budget against 0.0100 at full rate the whole time, roughly half, concentrated exactly where the estimates are worst.

Where it matters most

Warmup is close to mandatory for transformers trained from scratch and standard in nearly every large language model recipe since the original architecture. It matters less for small convolutional networks on plain SGD, where the instability it prevents is milder. The bigger the model and the more aggressive the peak rate, the more the ramp earns its place.

Code

The same first 10 steps with and without a 10-step linear warmup to a 0.001 peak.

1def warmup_lr(step: int, warmup_steps: int, lr_max: float) -> float:2    return lr_max * min(1.0, (step + 1) / warmup_steps)3
4
5rates = [warmup_lr(s, 10, 1e-3) for s in range(10)]6print([f"{r:.4f}" for r in rates[:5]])7print(f"{sum(rates):.4f} vs {10 * 1e-3:.4f}")8# -> ['0.0001', '0.0002', '0.0003', '0.0004', '0.0005']9# -> 0.0055 vs 0.0100

Warmup spends 0.0055 against 0.0100, and the saving sits entirely in the earliest steps where the moment estimates are least reliable.

Watch Out For

Copying a warmup length from a different model size

A warmup of 4000 steps tuned for a giant model, pasted onto a small one with a short run, spends a large fraction of the entire training budget still ramping up.

Symptom: the loss is still climbing its ramp when a third of training is already gone, and the peak rate barely gets used. Fix: scale $W$ to the instability you actually see. Watch the loss across the warmup window itself before treating the number as universal.

Adding warmup but skipping the decay half

Warmup only controls the ramp-up and says nothing about what follows. A run with warmup but a flat peak rate afterward reintroduces the exact one-rate-for-the-whole-run problem under a different name.

Symptom: clean start, then jittery late loss that never settles. Fix: pair the ramp with a decay from learning rate scheduling. Both halves are usually needed together.

The Quick Version

Early training instability comes from moment estimates that have not settled yet, not from a peak rate that is wrong for the rest of the run.
Warmup climbs the rate linearly from near zero to its peak over a fixed step count, so the first five rates to a 0.001 peak over 10 steps are 0.0001 through 0.0005.
Ten warmup steps spend 0.0055 of rate budget against 0.0100 at full rate, halving early update size.
Warmup length should scale with model size and observed instability, never copied as a fixed constant.
Warmup is close to mandatory for transformers from scratch and less critical for small plain-SGD models.
The ramp handles the start while a separate decay schedule handles the rest, and most runs need both.