Mixed Precision Training
Running the forward and backward pass in a lower-precision format for speed and memory savings, while keeping a full-precision master copy of the weights.
fp16 arithmetic runs faster and uses half the memory of fp32, but its narrower range lets small gradients round straight down to zero. Loss scaling multiplies the loss before the backward pass so the resulting gradients land inside fp16's range, then divides them back down afterward — lossless in principle, since it recovers the original magnitude exactly.
The optimizer's actual update is applied to a separate fp32 master copy of the weights, kept in full precision the entire run, so a small update never gets lost to fp16 rounding. bfloat16 keeps fp32's exponent range at lower fraction precision, which is why it often skips loss scaling entirely.