Gradient Clipping

Cap the gradient's size before the optimizer step, either by shrinking each entry to a fixed range or by rescaling the whole vector, so one bad batch can't throw the weights somewhere training never recovers from.

Gradient 0.5, minus 8.0, 3.0 and 40.0 has norm 40.91 and rescales by 0.122 to 0.06, minus 0.98, 0.37 and 4.89 at norm 5.0, keeping direction exactly

Why Does This Exist?

A character model trains smoothly for 40,000 steps, then one rare batch with long nested quotes gives an enormous gradient. The next update throws every weight somewhere unseen, loss reports nan at step 40,002, and stays nan. Nothing was wrong for 40,000 steps. One correctly computed but huge gradient ended the run.

That is exploding gradients: a valid gradient far larger than recent steps, whose large step lands where the loss returns inf or nan. Removing the odd batch is impractical, since the model must eventually handle nested quotes too. The fix caps how large any single gradient may be before the optimizer, so an odd batch gives a large but bounded step. Gradient descent covers the step being capped, and vanishing and exploding gradients covers the multiply chain that grows it.

Think of It Like This

A steering wheel with a mechanical stop

A go-kart wheel could be yanked to a full 90 degrees, but a mechanical stop limits how far it physically turns no matter how hard the driver pulls. A jolt still steers the correct way, just never past the stop, so the kart corrects by a bounded amount instead of spinning out.

Pull strength is the gradient norm, steering direction is the gradient direction, and the stop is the threshold $c$ . Clip by norm keeps direction and caps distance, the way the stop keeps heading and caps angle.

Where it stops: a stop protects every yank equally and never loosens for a genuinely sharp corner. Clipping likewise cannot tell a freak batch from a steep but honest slope, so a threshold that bites every step slows real progress rather than just blocking accidents.

How It Actually Works

Two ways to impose the cap

Clip by value caps each entry independently at $\pm c$ , cheap but rotating: shrinking one huge entry while leaving a moderate one untouched changes direction, not just length.

Clip by norm, what recipes mean by gradient clipping unqualified, treats the whole vector $g$ as one object:

g' = \begin{cases} g & \text{if } \lVert g \rVert \le c \\ g \cdot \dfrac{c}{\lVert g \rVert} & \text{if } \lVert g \rVert > c \end{cases}

Here $\lVert g \rVert$ is the Euclidean length, the square root of the summed squares, and $c$ the chosen maximum. When the norm exceeds $c$ , every entry multiplies by the same scalar $c / \lVert g \rVert$ , so length lands at exactly $c$ with direction unchanged.

Worked example: 40.91 rescales to 5.0 with direction kept

Take $g = [0.5, -8.0, 3.0, 40.0]$ , where one entry dwarfs the rest, and $c = 5.0$ . Squares are $0.25$ , $64.0$ , $9.0$ , and $1600.0$ . Sum $1673.25$ and root:

\lVert g \rVert = \sqrt{1673.25} \approx 40.91

Scale factor $5.0 / 40.91 \approx 0.1222$ . Multiply through:

g' \approx [0.06, -0.98, 0.37, 4.89]

Its norm is $5.0$ and its unit vector matches the original entry for entry, which the snippet checks. Clip by value at $\pm 5.0$ instead gives $[0.5, -5.0, 3.0, 5.0]$ , which trims only the two large entries and rotates direction. The diagram draws the long raw arrow and the short clipped arrow on the same ray, with the dashed arc marking every point at norm 5.0.

Where it is load-bearing

Vanilla RNNs multiply by the same matrix every step, so a long backward pass compounds upward, the mirror of vanishing. Clip-by-global-norm at 1.0 or 5.0 across all parameters at once is close to default there, not a safety net. A run that clips nearly every step still has a deeper instability, usually a rate too high for the curvature or a hot initialization, which belongs to weight initialization and the schedule rather than the cap.

Code

The point is the check: rebuild both clips on the worked vector and confirm norm rescaling preserves direction while value clipping does not.

1import math2
3
4def clip_by_norm(g: list[float], max_norm: float) -> list[float]:5    norm: float = math.sqrt(sum(x * x for x in g))6    if norm <= max_norm:7        return g8    scale: float = max_norm / norm9    return [x * scale for x in g]10
11
12g: list[float] = [0.5, -8.0, 3.0, 40.0]13by_value: list[float] = [max(-5.0, min(5.0, x)) for x in g]14by_norm: list[float] = clip_by_norm(g, 5.0)15
16print(f"{math.sqrt(sum(x * x for x in g)):.2f}")  # -> 40.9117print(f"{by_value}")  # -> [0.5, -5.0, 3.0, 5.0]18print(f"{[round(x, 2) for x in by_norm]}")  # -> [0.06, -0.98, 0.37, 4.89]19print(f"{math.sqrt(sum(x * x for x in by_norm)):.1f}")  # -> 5.0

The printout matches the hand computation: norm 40.91, value clip trims only two entries, norm clip lands at 5.0 with entries 0.06, minus 0.98, 0.37, and 4.89. Divide the unit vectors and they agree, which is the direction claim in one check.

Watch Out For

Clipping every step and calling it fixed

Symptom: nan disappears so clipping looks sufficient, but the norm exceeds the threshold on nearly every step and training runs at a lower effective rate than configured, slow or flat without an error.

Fix: log the fraction of clipped steps. Healthy runs clip rarely on outliers. If most steps clip, lower the learning rate or fix initialization instead of tightening the cap.

Clipping each tensor separately instead of globally

Symptom: per-layer clips to the same threshold distort relative update sizes, so a small-gradient layer and a huge-gradient layer get equal treatment and training skews across depth.

Fix: compute one norm across every parameter together and apply one scale. Most framework calls default to global; verify rather than assume when hand-rolling.

The Quick Version

Gradient clipping caps size before the optimizer step, so one huge but valid gradient gives a bounded step rather than a run-ending jump.
Vector 0.5, minus 8.0, 3.0, and 40.0 has norm 40.91 and rescales by 0.122 to 0.06, minus 0.98, 0.37, and 4.89 at norm 5.0.
Clip by value caps entries independently and rotates direction, while clip by global norm rescales the whole vector and preserves it.
Global norm across all parameters at once is the version most recipes default to.
Recurrent nets need it structurally because repeated multiplication through time compounds gradients upward.
Clipping on nearly every step signals a rate or initialization problem rather than a solved one.