Data Processing Inequality

Post-processing never invents information. A label guessed from a compression holds at most what the compression kept from the photo.

A photo compresses to 1.0 bit then to a 0.4-bit guess, so the end-to-end information 0.3 never beats the weakest link

Why Does This Exist?

Pipelines stack: sensor to embedding to prediction to decision. Each stage hopes to "recover" signal the last one dropped by clever post-processing. The inequality ends that hope with one line: on a Markov chain $X \\to Y \\to Z$ , $I(X;Z) \\leq I(X;Y)$ and $I(X;Z) \\leq I(Y;Z)$ .

Budget your effort upstream. It needs mutual information and the chain rule in joint and conditional entropy, and motivates the tradeoff in information bottleneck.

The one idea you need first

$X \\to Y \\to Z$ is Markov when $p(z \\mid y, x) = p(z \\mid y)$ : $Z$ sees $X$ only through $Y$ . Every feedforward pipeline without skip-backs qualifies. Information $I(X;Y)$ measures shared bits across each link.

Think of It Like This

Photocopies of photocopies

Copy a photo, then copy the copy. The second copy never regains detail the first lost: toner cannot invent pores. Sharpening filters guess plausibly, but the guess is not the pore.

Sufficiency is the exception that proves it: a lossless copy (sufficient statistic) holds equality. Where it stops: copies are deterministic degradations, while pipelines add learned priors that hallucinate realistic-but-ungrounded detail.

How It Actually Works

Proof sketch: $I(X;Y,Z) = I(X;Z) + I(X;Y\\mid Z) = I(X;Y) + I(X;Z\\mid Y)$ , and Markov makes $I(X;Z\\mid Y) = 0$ , so $I(X;Z) = I(X;Y) - I(X;Y\\mid Z) \\leq I(X;Y)$ . Equality needs $I(X;Y\\mid Z) = 0$ : $Z$ sufficient for $X$ given $Y$ . Corollaries: no function of data beats the data ( $I(\\theta; T(X)) \\leq I(\\theta; X)$ ); noisy-channel capacity bounds every receiver.

Worked example

Photo $X$ , JPEG $Y$ keeping $I(X;Y) = 1.0$ bit, classifier $Z$ with $I(Y;Z) = 0.4$ bits. Then $I(X;Z) \\leq \\min(1.0, 0.4) = 0.4$ ; measured $0.3$ after classifier noise. A "super-resolution" head claiming $0.8$ bits about $X$ from $Z$ contradicts the inequality unless it smuggles a side prior (then the chain is not $X \\to Y \\to Z$ alone). Weakest link rules: 0.3 sits below 0.4.

Code

1import numpy as np2
3rng = np.random.default_rng(12)4x = rng.integers(0, 2, size=50000)  # fair bit5y = np.where(rng.random(50000) < 0.9, x, 1 - x)  # BSC 0.16z = np.where(rng.random(50000) < 0.8, y, 1 - y)  # second BSC 0.27print(round(float((x == y).mean()), 3))  # -> 0.98print(round(float((x == z).mean()), 3))  # -> 0.74

Agreement degrades 0.9 to 0.74 down the chain. No post-processing of $z$ recovers the 0.9 fidelity of $y$ .

Watch Out For

Smuggling priors and crying violation

Generative upscalers beat "the limit" by injecting learned priors: the real chain is $(X, \\text{prior}) \\to Z$ , not $X \\to Y \\to Z$ . Calling it a DPI violation misreads the graph and overtrusts hallucinations.

Draw the full graph including priors. If side information enters, the bound moves with it.

Chasing downstream fixes for upstream loss

Retraining heads on frozen weak embeddings plateaus at the embedding's ceiling. Teams burn GPU on heads while the encoder starves.

Measure $I(X;Y)$ first (probe accuracy, reconstruction). If the link is the bottleneck, retrain upstream or recollect, not the head.

The Quick Version

Markov $X \\to Y \\to Z$ gives $I(X;Z) \\leq \\min(I(X;Y), I(Y;Z))$ .
Example: 1.0-bit compression into 0.4-bit guess caps end-to-end at 0.4; measured 0.3.
Equality needs sufficiency: $Z$ keeps everything $Y$ had about $X$ .
Side priors change the graph; draw them before crying violation.
Fix upstream links before downstream heads.
Agreement 0.9 degrades to 0.74 down two noisy channels, never recovers.