Differential Privacy
Differential privacy adds a calculated amount of mathematical noise during training, so the finished model looks essentially the same whether any one person's data was included or not.
Why Does This Exist?
A hospital trains a model on patient records, then releases it. An attacker keeps asking questions: was one specific person in the training set? With membership inference, slightly-too-confident answers on one record say yes. Stripping names does not stop it, because zip code plus age plus admission dates re-identify people anyway, which is exactly where older tricks like k-anonymity break.
The fix is not another scrubbing rule. It is a mathematical promise: the finished model must look essentially identical whether any single row was included or not. If one person's data cannot visibly move the output, no questionnaire can reveal they participated.
This page covers that promise and the training loop that keeps it, DP-SGD. The optimizer it modifies is in gradient descent. The clipping step it depends on is in gradient clipping. The governance around it sits in data privacy and governance.
Think of It Like This
The survey with a secret coin flip
A researcher asks students whether they cheated on an exam. Nobody answers honestly with their name on the sheet, so each student flips a coin in secret first. Heads means answer truthfully. Tails means flip again and answer yes on heads, no on tails, regardless of the truth.
Any single yes is now deniable: that student may have just flipped tails then heads. Yet the researcher knows the coin odds, so across a thousand sheets the expected noise is a known number that subtracts out, leaving a solid estimate of the true cheating rate. Privacy for the individual, signal for the population.
The mapping: the secret coin is the Gaussian noise, the deniable yes is a clipped gradient plus noise, and subtracting the expected noise is averaging over a batch. Where it stops: the survey noise is fixed coins, while DP-SGD noise is sized by the clipping bound and a privacy budget, and the budget runs out as training continues.
How It Actually Works
The promise, written down
Call two datasets neighbors if they differ in exactly one row. A training procedure is differentially private with budget (epsilon) when every possible output is almost as likely with or without that row:
and are the neighboring datasets, is any set of outcomes, and caps how much one row can tilt the odds. At the cap is . At it is , a far stricter promise. Smaller means more noise and a stronger guarantee; larger means less noise and a weaker one. Every epoch spends some of the budget, so the guarantee degrades as training continues.
DP-SGD: clip, then add noise
Standard training lets one unusual row produce a huge gradient that drags the weights toward memorizing it. DP-SGD adds two steps to each update, and both matter:
- Clip every per-example gradient. If is one example's gradient and is the clipping bound, replace with . No single row can now pull harder than , which bounds how much the answer could change if that row vanished. That bound is what sizes the noise.
- Add Gaussian noise to the sum. Draw noise with standard deviation proportional to times a noise multiplier, add it to the summed clipped gradients, then average and update. The batch trend survives because noise averages toward zero; the individual contribution drowns because it was capped first.
Worked example: four gradients, clipped and noised
Four per-example gradients from one batch, clipping bound :
Clip each by , giving scales :
Sum them:
Add one Gaussian draw of at noise scale to the sum, then average over 4:
With learning rate and current weight , the update is . The row that wanted to pull with got capped at first, and the of noise means no observer can tell whether any one of the four rows was present. That is the diagram: four arrows squeezed through a clipping gate, one noise cloud, one update.
Code
The point is the check: replay the worked example and confirm the clipped sum 1.5 and the noisy update 1.76.
grads = [0.5, 2.0, -4.0, 1.5]bound = 1.0
clipped = [g * min(1.0, bound / abs(g)) for g in grads]print(clipped) # -> [0.5, 1.0, -1.0, 1.0]
total = sum(clipped)print(f"clipped sum: {total}") # -> clipped sum: 1.5
noise_draw = 0.42 # one Gaussian draw at noise scale 1.0noisy_mean = (total + noise_draw) / len(grads)print(f"noisy mean: {noisy_mean:.2f}") # -> noisy mean: 0.48
weight = 2.0 - 0.5 * noisy_meanprint(f"updated weight: {weight:.2f}") # -> updated weight: 1.76Change the to and rerun: the clipped value stays and the update barely moves. That flat response to outliers is the guarantee made visible. In practice teams use libraries such as Opacus for PyTorch, which handle per-example gradients and noise accounting automatically.
Watch Out For
Spending the whole budget before training finishes
Each epoch queries the data again, and each query spends budget. The symptom is a model that looked private at epoch 5 and meaningless-or-leaky by epoch 50: either the noise was scaled up to cover all epochs and nothing learned, or the budget ran dry halfway and the late epochs trained unprotected. Fix: decide total first, then set epochs, batch size, and noise multiplier from it using the library's accountant, instead of training to convergence and hoping.
Expecting privacy to hide population truths
Differential privacy hides individuals, not groups. If the model learns that smoking predicts a condition, a smoker's risk is exposed by that general trend, not by any leak of their row. The symptom is a team that passes a membership-inference test yet still surprises users with what the model reveals about them. Fix: treat group-level disclosure as a product decision owned with data privacy and governance, not as something the noise was supposed to prevent.
The Quick Version
- Differential privacy promises the model looks essentially the same whether any single row trained it or not, capped by .
- DP-SGD keeps that promise in two steps: clip each per-example gradient to a bound , then add Gaussian noise sized to before updating.
- Four gradients clip to at bound 1.0, summing to 1.5.
- One noise draw of 0.42 turns the mean update from 0.375 into 0.48, moving weight 2.0 to 1.76 at learning rate 0.5.
- Epsilon 1.0 caps the odds shift at about 2.72 times; smaller epsilon means stronger privacy and worse accuracy.
- The guarantee covers individuals only, and every epoch spends budget, so total epsilon comes first and training plans follow it.