Privacy in ML

Differential privacy and data protection techniques ensure machine learning models learn general patterns without memorizing and leaking individual training records.

A count of 87 with sensitivity 1 and epsilon 0.5 takes Laplace noise at scale 2.0, so including one row shifts the odds by at most 1.65 times.

Why Does This Exist?

Large models memorize. Asked the right way, they recite training text verbatim: names, addresses, medical notes that were never meant to leave the dataset. Deleting names first does not save you, because the model links the leftovers. A zip code plus a birth date plus a diagnosis re-identifies most Americans, and the model holds all three at once.

Attackers industrialized this with membership inference: ask whether a specific record trained the model, and the model's own confidence answers. If it is suspiciously sure about your row, your row was probably inside. For health, finance, and messaging data that question alone is a breach.

Privacy in ML exists to bound that leakage mathematically instead of hoping for it. The flagship guarantee is differential privacy: learn the population's patterns while making any single person's presence nearly undetectable in the result.

This page covers the threat, the guarantee, and the budget that prices it. For the mechanism in full depth, read differential privacy. For the rights side of which data may train at all, see consent and data rights. For running the whole privacy program, AI governance is next.

Think of It Like This

A crowd photo with a deliberate blur

A stadium photographer publishes a crowd shot. At full resolution you can zoom into section 12, row 4, and read a fan's face. That is standard training: every individual recoverable from the artefact. Now the photographer applies a careful blur before publishing. You can still count the crowd, see which sections overflow, and read the team colours. The patterns survive. But zooming for one face gives you smear, and no enhancement brings the face back because the detail is gone, not hidden.

The mapping runs clean. The crowd is the dataset, each face is a training record, counting sections is learning population patterns, and the blur is calibrated noise. Zooming is membership inference, and it fails because the pixels it needs no longer exist.

Where it stops: a blurred photo annoys everyone equally, while privacy noise can hurt small groups most. Rare patterns drown first, so the blur that protects a minority can also erase them from what the model learns. Privacy and inclusion genuinely compete, and the budget has to be set with both in view.

How It Actually Works

Call two datasets neighbours when they differ in exactly one person's row, and call the randomized training procedure $M$ . Differential privacy says the odds of any output barely change between neighbours:

P(M(D) = y) \le e^{\epsilon} \cdot P(M(D') = y)

In plain words: $D$ and $D'$ are the two neighbour datasets, $y$ is any output you might observe, $P$ is its probability, and $\epsilon$ (epsilon) is the privacy budget you chose. Smaller $\epsilon$ means tighter indistinguishability and stronger privacy. The guarantee holds against attackers with outside information, because it constrains the output distribution itself rather than any particular secret.

Sensitivity sets the noise

Sensitivity, written $\Delta$ , is the most one row can move the true answer. For a count, one person changes the total by at most 1, so $\Delta = 1$ . The Laplace mechanism then adds noise with scale $b = \Delta / \epsilon$ . Lower $\epsilon$ means a wider blur, exactly like turning up the photo filter.

Worked example: a count of 87 at epsilon 0.5

Ask how many of 200 records carry some trait. The true answer is 87. Sensitivity is $\Delta = 1$ , because adding or removing one person moves a count by exactly 1. Choose $\epsilon = 0.5$ . The noise scale is $b = 1 / 0.5 = 2.0$ , so released answers scatter a few counts around 87 rather than landing on it.

The protection reads straight off epsilon. Since $e^{0.5} \approx 1.65$ (because $e \approx 2.71828$ and $\sqrt{2.71828} \approx 1.6487$ , rounded to 1.65), observing any released count is at most 1.65 times more likely with your row present than without it. An attacker watching the output cannot push their odds past that factor no matter what else they know. Halve the budget to $\epsilon = 0.25$ and the factor tightens to $e^{0.25} \approx 1.28$ , at the price of roughly double the scatter. That exchange, tighter factor for wider blur, is the entire privacy-utility tradeoff in one number. The diagram above draws it: two output curves for the neighbouring datasets, nearly on top of each other inside a 1.65 bracket.

Clip and count the spend

Gradients need the same treatment during training, with one extra step. Each example's gradient is clipped to a maximum norm $C$ before averaging, so no single outlier can swing the update further than $C$ allows, and then noise scaled to $C$ is added. Every query and every training step spends budget, and the spends add up. Run 100 steps at $\epsilon = 0.05$ each and the composed guarantee is far weaker than any single step's. Track the total, publish it, and stop or refresh data when it runs out.

Code

Two helpers in plain Python with no randomness and no extra packages. One returns the Laplace noise scale for a query, the other returns the odds-shift factor a budget allows.

1import math2
3
4def laplace_scale(sensitivity: float, epsilon: float) -> float:5    """Noise scale b for one query: sensitivity divided by epsilon."""6    return sensitivity / epsilon7
8
9def odds_shift_factor(epsilon: float) -> float:10    """Worst odds ratio between neighbouring datasets: e to the epsilon."""11    return math.exp(epsilon)12
13
14print(f"{laplace_scale(1.0, 0.5):.1f}")  # -> 2.015print(f"{odds_shift_factor(0.5):.2f}")  # -> 1.65

The printouts match the hand arithmetic: scale 2.0 for the count, and a 1.65 cap on the odds shift. Use the second function in reviews whenever someone proposes an epsilon, so the budget discussion stays in concrete odds instead of vibes.

Watch Out For

Stripped names mistaken for protection

The team removes names and IDs, calls the data anonymous, and trains. Then a linkage attack combines zip code, birth date, and diagnosis to re-identify rows the model happily memorized.

Symptom: a dataset labelled safe while quasi-identifiers sit in plain text and the model overfits them. Fix: treat removal of identifiers as hygiene, not privacy. Bound the leakage with noise and a tracked budget, and test with membership inference probes before release. Pseudonymization hides labels; only the guarantee hides presence.

An epsilon that protects nothing or teaches nothing

Set $\epsilon = 20$ and the odds factor is $e^{20}$ , hundreds of millions to one, which is no protection wearing a privacy costume. Set $\epsilon = 0.01$ on a small dataset and the noise buries every pattern, so the team quietly trains a second unprotected model for actual use.

Symptom: either complaints that the private model leaks under probing, or a private model nobody uses beside its shadow twin. Fix: pick epsilon from the odds factor you can defend, commonly in the 0.5 to 2 range for statistics like counts, validate utility per subgroup since rare slices degrade first, and publish the budget with the release.

The Quick Version

Big models memorize rows, and membership inference turns that memory into a breach.
Differential privacy bounds leakage: neighbours differing in one row produce nearly identical output distributions.
A count of 87 with sensitivity 1 at epsilon 0.5 takes Laplace noise at scale 2.0.
Epsilon 0.5 caps the attacker's odds shift at 1.65 times, no matter their outside knowledge.
Clipping bounds each example's influence, and every query spends budget that composes.
Stripped names are not privacy, and extreme epsilons either leak or teach nothing.