Model Pruning
Most trained weights sit near zero and contribute almost nothing. Pruning zeroes exactly those, leaving a sparse network that acts the same but fits smaller chips.
Why Does This Exist?
A wake-word detector must listen all day on a tiny speaker chip with a strict memory budget. The best model the team trained has far more parameters than the chip can hold. Training a small network from scratch is the obvious fix, and it underperforms: a network born small never had spare capacity to explore with, so it settles for worse accuracy than the big one found easily.
But the trained big network hides a gift. A large share of its weights ended near zero. Each contributes almost nothing to any output, and keeping them was free during training on a server farm. On the chip, every parameter costs real memory. This page covers the cut: rank weights by size, zero the smallest, and retrain briefly so survivors adjust. Sibling ways to shrink a model are knowledge distillation, which teaches a small student from a big teacher, and model quantization, which rounds weights instead of deleting them. Rounding during training itself lives in quantization-aware training.
Think of It Like This
Thinning seedlings in a garden row
A row of carrots sprouts too thick. Every sprout is alive, but the tiny crowded ones steal water while adding no real roots. The gardener does not replant the row. They pull the smallest sprouts and leave the strong ones with room, and the harvest barely changes.
Weights are the sprouts, absolute size is height, the cut line is the gardener's thumb width, and fine-tuning is the week of watering after thinning while survivors spread into the gaps.
Where it stops: a gardener sees each sprout directly, while magnitude is only a rough proxy for a weight's worth. A small weight can still matter inside its circuit, which is why the thinned network gets tested rather than trusted.
How It Actually Works
Call a weight's absolute value its magnitude. Magnitude pruning sorts every weight by magnitude and zeroes the smallest fraction, often half or more. The reasoning is direct: a near-zero weight barely moves its layer's output whatever the input, so deleting it changes behavior least. It is a cheap proxy for importance, not a perfect one, and it works well enough to be the default first try.
Two cut shapes exist. Unstructured pruning zeroes individual weights wherever they fall, giving the best accuracy at any sparsity. But ordinary chips still multiply by zero at full cost, so nothing speeds up without sparse-matrix support, and the sparse matrix format decides whether memory shrinks either. Structured pruning removes whole neurons or channels at once. Matrices truly shrink and any hardware benefits immediately, but each cut is coarser and costs more accuracy at equal sparsity.
The cut alone is not the method. Survivors were trained assuming deleted neighbors still contributed, so the network always loses more than it must right after zeroing. Fine-tuning continues training briefly with pruned weights held at zero so survivors adjust. Skipping it understates how sparse the network could have gone.
Worked example: eight weights, one cut line
Take weights 0.05, minus 0.12, 0.9, minus 1.4, 0.03, 2.1, minus 0.4, 0.7. Absolute values sorted: 0.03, 0.05, 0.12, 0.4, 0.7, 0.9, 1.4, 2.1. Prune the smallest half, so the cut falls between 0.12 and 0.4 and four weights become zero.
Squared magnitudes: 0.0025 plus 0.0144 plus 0.81 plus 1.96 plus 0.0009 plus 4.41 plus 0.16 plus 0.49, total 7.8478. The four kept weights, 0.7, 0.9, 1.4 and 2.1, hold 0.49 plus 0.81 plus 1.96 plus 4.41, which is 7.67 of 7.8478, about 97.7 percent. Half the weights gone, nearly all the signal kept. Push to 75 percent sparsity, keeping only 1.4 and 2.1, and the kept share falls to about 81.2 percent. That cliff is the diagram headline: four bars below the line, four above, and the kept energy printed beside them.
Code
The point is the check: rebuild the 8-weight cut at two sparsity levels and confirm 97.7 percent kept at half sparsity against 81.2 at three-quarters.
def prune_by_magnitude(w: list[float], sparsity: float) -> list[float]: """Zero out the smallest `sparsity` fraction of weights by absolute value.""" k = int(round(sparsity * len(w))) if k == 0: return list(w) cutoff = sorted(abs(x) for x in w)[k - 1] return [x if abs(x) > cutoff else 0.0 for x in w]
w: list[float] = [0.05, -0.12, 0.9, -1.4, 0.03, 2.1, -0.4, 0.7]total: float = sum(x * x for x in w)for sparsity in (0.5, 0.75): pruned = prune_by_magnitude(w, sparsity) kept = sum(x * x for x in pruned) nonzero = sum(1 for x in pruned if x != 0.0) print(f"{sparsity:.0%} kept {nonzero}/{len(w)} holds {kept / total:.1%}")# -> 50% kept 4/8 holds 97.7%# -> 75% kept 2/8 holds 81.2%The printout matches the hand computation at both levels: 50 percent sparsity keeps 4 of 8 and holds 97.7 percent, 75 percent keeps 2 of 8 and holds 81.2 percent. Move the 0.4 weight to 0.05 and rerun to watch kept energy barely move, which is the whole argument in one edit.
Watch Out For
Judging the cut before fine-tuning
Zeroing weights and scoring immediately, with no retraining, measures a wounded network rather than a pruned one. The survivors have not adjusted to their emptier neighborhood, so every sparsity level looks worse than it really is and teams stop cutting far too early.
Symptom: accuracy falls off a cliff the moment sparsity passes 30 percent, yet papers report 90 percent on the same architecture. Fix: always fine-tune with pruned weights frozen at zero before scoring, and compare methods only after that step. The training habit that prevents a related mistake, stopping before memorization, is early stopping.
Shipping unstructured sparsity to dense-only hardware
Unstructured pruning wins every accuracy-per-sparsity table, so a team ships a 90 percent sparse model to a phone chip with no sparse kernels. The file is still dense in memory, every zero still gets multiplied, and none of the promised savings appear. The benchmark number was real; the deployment target just could not cash it.
Symptom: pruned model identical in size and latency to the dense one on device, despite impressive sparsity reports. Fix: check the target runtime for sparse support before choosing, and pick structured channel cuts when it has none. The rounding route to small models, which needs no sparse support at all, is model quantization, and the side-by-side on small versus large models is settled in small vs large models.
The Quick Version
- Magnitude pruning zeroes the smallest weights of a trained network, since near-zero weights move outputs least.
- Our 8 weights total 7.8478 in squared magnitude, and keeping the top 4 holds 7.67 of it, about 97.7 percent.
- Pushing to 75 percent sparsity keeps only 2 weights and 81.2 percent of signal, which is where cutting turns into damage.
- Unstructured cuts keep accuracy best but need sparse-matrix support, while structured channel cuts pay off on any hardware.
- Fine-tuning after the cut is part of the method, since survivors must adjust to neighbors that are gone.