Learning Rate Scheduling

The learning rate determines the size of the steps the optimizer takes in the loss landscape. But using the same learning rate for the entire training process is rarely optimal.

If the learning rate is too high, the model will initially learn quickly but eventually bounce around the minimum without ever settling down. If the learning rate is too low, training will be painfully slow and might get stuck.

Learning rate scheduling solves this by adjusting the learning rate dynamically as training progresses.

Constant Learning Rate

The simplest approach is no schedule at all—keeping the learning rate constant. While easy to set up, it often requires a compromise between fast initial progress and stable final convergence.

Step Decay

Step decay drops the learning rate by a certain factor after a fixed number of epochs. This allows the model to make large strides early on, and then take finer, more careful steps as it approaches the minimum.

Cosine Annealing

Cosine annealing smoothly reduces the learning rate following a cosine curve. It starts decreasing slowly, accelerates the decay in the middle, and then slows down again towards the end. This often yields better and faster convergence than a jagged step decay.

Linear Warmup

Starting with a high learning rate can sometimes cause the model to diverge immediately. Linear warmup gradually increases the learning rate from zero to the base value over the first few epochs, stabilizing the early training phase before decaying.