Independence and Conditional Independence

Independence means learning one thing teaches you nothing about another. Conditional independence means that stays true once the shared cause is known.

$Four equally likely coin outcomes with their XOR flag show X and Z meeting at chance 0.25, exactly the product of halves, while the triple meeting has chance zero$

Four equally likely coin outcomes with their XOR flag show X and Z meeting at chance 0.25, exactly the product of halves, while the triple meeting has chance zero

Why Does This Exist?

Multiplying probabilities is the move everyone wants to make and few earn. The chance of two events together equals the product of their chances only when they are independent, and assuming independence without checking is how naive Bayes earns the word naive, how risk models underprice joint failures, and how test accuracy gets misread as deployment certainty.

Conditional independence is the subtler sibling that runs modern ML. Features are rarely independent outright, but they are often independent given the label: symptoms correlate with each other until you know the disease, at which point each adds its own evidence. That structure is what makes Bayes' theorem computations factor and what graphical models encode. The combining rules underneath live in probability foundations; this page teaches the two independence notions plus the trap between pairwise and mutual.

The one idea you need first

The joint chance $P(A \text{ and } B)$ is the fraction of outcomes where both happen. Independence is the claim that restricting to $B$ leaves $A$ 's fraction unchanged: $P(A \mid B) = P(A)$ , equivalently $P(A \text{ and } B) = P(A)P(B)$ . Everything here is that factorization, with and without a given third event.

Think of It Like This

Two newspapers, one wire service

Two local papers print the same national story. Reading one tells you exactly what the other says: dependent, through a shared cause. Now suppose you already read the wire copy both drew from. The leftover differences, local typos and headlines, are independent given the wire. Knowing the common cause screened off the connection.

That screening is conditional independence: $A$ and $B$ correlate until $C$ is known, then factor. Naive Bayes bets each word is independent given the topic. It is wrong in the way the analogy suggests, shared causes beyond the label always leak, but useful because factoring turns an impossible joint table into one probability per word.

The analogy stops at direction. Screening works when the known variable is the common cause. Condition on a common effect instead, like knowing both papers sold out, and you couple them rather than separate them: explaining-away creates dependence instead of removing it.

How It Actually Works

Two events are independent when their joint chance factors:

P(A \text{ and } B) = P(A)\, P(B)

In plain words: the fraction where both happen is the product of the separate fractions. Equivalently, learning $B$ never moves $A$ 's chance. Three events are mutually independent when every sub-collection factors, all three pairs and the triple. Pairwise alone is weaker, as the worked example proves.

Events are conditionally independent given $C$ when they factor inside every slice of $C$ :

P(A \text{ and } B \mid C) = P(A \mid C)\, P(B \mid C)

In plain words: once $C$ is known, learning $B$ teaches nothing further about $A$ . Note the double dissociation people trip over. Independence does not imply conditional independence: two independent coin flips become coupled given their sum, since a sum of 1 forces them apart. And conditional independence does not imply independence: two symptoms correlate overall yet factor given the disease.

Worked example

Flip two fair coins $X$ and $Y$ , and set $Z = 1$ when they differ (the XOR).

Each pair is independent. $P(X = 1) = 0.5$ and $P(Z = 1) = 0.5$ , since two of four outcomes differ. Their meeting $X = 1, Z = 1$ is exactly the outcome $(1, 0)$ , chance $0.25 = 0.5 \times 0.5$ . Same for the other pairs. Pairwise, everything factors.

But the triple does not: $P(X = 1, Y = 1, Z = 1) = 0$ , because equal coins never differ, while the product $0.5^3 = 0.125$ . Pairwise independent, mutually dependent. Any two determine the third, so the trio carries less information than independence would claim, and any model multiplying all three double-counts.

Watch Out For

Multiplying because the product looks available

Joint tables factor only under independence, yet pipelines multiply classifier scores, sensor readings, and annotator votes by default. Correlated inputs make the product overconfident: two tests sharing a failure mode are barely more evidence than one. Before multiplying, name the shared causes and check whether conditioning on them restores the factorization.

Confusing uncorrelated with independent

Zero correlation means no straight-line relationship, while independence means no relationship of any kind. A symmetric U-shape has zero correlation with loud dependence, and distance correlation or mutual information catch what Pearson misses. Passing a correlation check never licenses the product rule.

The Quick Version

Independence means the joint chance factors, so learning one event moves nothing.
Conditional independence restores factoring once the shared cause is known, and naive Bayes is exactly this bet.
Conditioning on a shared cause screens off; conditioning on a shared effect couples.
Pairwise independence does not imply mutual independence: two coins and their XOR prove it.
Uncorrelated is far weaker than independent, so correlation checks never license multiplying.