Skip to content
AI360Xpert
Glossary
Definition

Gradient Clipping

Capping the size of a gradient before the optimizer step, so one unusually large batch can't throw the parameters somewhere training can't recover from.

Two versions exist. Clip by value caps every individual gradient entry at a fixed threshold, which is cheap but changes the direction of the update. Clip by global norm rescales the whole gradient vector when its length exceeds a threshold, preserving direction and only shrinking magnitude — the version almost every modern training recipe means by default.

It's a safety net, not a cure: a model that needs clipping on every single step still has an underlying instability, usually the learning rate or the initialization, that clipping is only masking rather than fixing.