Probability Foundations

Why Does This Exist?

Supervised learning has a one-line definition that only makes sense once this page does: a classifier models the conditional distribution $P(y \mid x)$. It does not output a label. It outputs a belief about labels, given an input — and the vertical bar in that expression is conditional probability, which is the single most important idea here.

Get this wrong and a whole class of mistakes follows. Multiplying probabilities that are not independent is how naive Bayes earns the word "naive". Reading a 99%-accurate test as 99% certainty is the base-rate error that misprices risk in production. Treating a model's softmax output as a calibrated probability when it was trained to be confident is how teams ship systems that are wrong and sure about it.

None of this needs measure theory. It needs three rules and one habit: always ask conditional on what.

Think of It Like This

How It Actually Works

Three objects, and you need all three.

The sample space $\Omega$ is every outcome that could occur — six faces for a die, every real number for a temperature reading. An event is any subset of it: "the die shows an even number" is the subset $\{2, 4, 6\}$. A probability assigns each event a number in $[0, 1]$ such that $P(\Omega) = 1$ and the probabilities of disjoint events add.

That is the whole foundation. Everything below is consequence.

The two combining rules

For "or", add and subtract the overlap:

In plain words: add the two chances, then remove the double-count where both happen. Skipping that subtraction is the most common arithmetic slip in the whole subject.

For "and", multiply — but the second factor must be conditional:

In plain words: the chance both happen is the chance $B$ happens, times the chance $A$ happens given that it did. Notice this is exact and always true. It is the definition of conditional probability rearranged, not an approximation.

Conditional probability

Rearranging the "and" rule gives the definition:

In plain words: of all the times $B$ happens, what fraction also has $A$? The denominator is the re-measuring — you have restricted the world to $B$, so you divide by how big $B$ is. It requires $P(B) > 0$: conditioning on something impossible is not a question with an answer.

Two things worth fixing in place now, because both cause real errors:

• $P(A \mid B)$ and $P(B \mid A)$ are different numbers. The probability of a positive test given illness is not the probability of illness given a positive test. Swapping them is so common it has a name, the prosecutor's fallacy, and Bayes' theorem exists to convert between them.
• Conditioning can move a probability either way. It is not "narrowing down" toward certainty. $P(A \mid B)$ can be larger than, smaller than, or equal to $P(A)$.

Independence is a conclusion, not an assumption

$A$ and $B$ are independent exactly when learning one tells you nothing about the other:

The multiply-the-probabilities shortcut everyone remembers is only valid under independence. That is the caveat that gets dropped, and dropping it is how models become confidently wrong — see the second pitfall.

Joint, marginal, conditional

Three views of the same table of probabilities, and the vocabulary matters because papers use it constantly:

• Joint $P(x, y)$ — both together.
• Marginal $P(x)$ — one of them, ignoring the other, obtained by summing the joint over every value of the other. The name comes from literally totalling a table in its margin.
• Conditional $P(y \mid x)$ — one of them once the other is known.

Machine learning lives in the third. A generative model learns the joint $P(x, y)$; a discriminative model learns only the conditional $P(y \mid x)$ and never models how inputs are distributed. That distinction is the whole difference between the two families.

Worked example

Roll a fair six-sided die. Let $A$ be "the roll is even" and $B$ be "the roll is 4 or higher".

$A = \{2, 4, 6\}$, so $P(A) = 3/6 = 0.5$. $B = \{4, 5, 6\}$, so $P(B) = 3/6 = 0.5$. The overlap $A \cap B = \{4, 6\}$, so $P(A \cap B) = 2/6 \approx 0.333$.

Now condition:

In plain words: among rolls of 4 or more, two of the three are even. Learning $B$ raised the chance of $A$ from 0.5 to 0.667 — so these events are not independent. Check it against the product rule: $P(A)P(B) = 0.25$, but $P(A \cap B) = 0.333$. Not equal, so not independent, and multiplying 0.5 by 0.5 here would have been wrong.

Contrast that with $C$ = "the roll is 3 or higher", $P(C) = 4/6$. Then $A \cap C = \{4, 6\}$, so $P(A \cap C) = 2/6 = 0.333$, and $P(A)P(C) = 0.5 \times 0.667 = 0.333$. Equal — so $A$ and $C$ are independent, which is not obvious from looking at them. Independence is arithmetic, not intuition.

Show Me the Code

1from itertools import product2
3
4def prob(event, space: list[int]) -> float:5    return sum(1 for s in space if event(s)) / len(space)6
7
8die = [1, 2, 3, 4, 5, 6]9is_even, at_least_4 = lambda r: r % 2 == 0, lambda r: r >= 410
11p_a = prob(is_even, die)                                   # 0.512p_b = prob(at_least_4, die)                                # 0.513p_ab = prob(lambda r: is_even(r) and at_least_4(r), die)   # 2/614
15print(round(p_a, 3), round(p_b, 3), round(p_ab, 3))   # -> 0.5 0.5 0.33316print(round(p_ab / p_b, 3))                            # -> 0.667  P(A|B)17print(round(p_a * p_b, 3))                             # -> 0.25   != 0.333, so dependent18
19# Two dice are genuinely independent, so here the product rule DOES hold.20pairs = list(product(die, die))21both = sum(1 for a, b in pairs if a % 2 == 0 and b >= 4) / len(pairs)22print(round(both, 3), round(p_a * p_b, 3))             # -> 0.25 0.25

Every number matches the hand calculation. The last two lines are the contrast worth keeping: the product rule failed for two events on one die and held for two events on separate dice.

Watch Out For

The Quick Version

• The sample space is everything that could happen; an event is a subset; probabilities of disjoint events add.
• For "or": add, then subtract the overlap. Forgetting the subtraction double-counts.
• For "and": $P(A \cap B) = P(A \mid B)P(B)$. Always true, and the definition of conditioning rearranged.
• $P(A \mid B) = P(A \cap B) / P(B)$ — restrict the world to $B$, then re-measure.
• $P(A \mid B) \neq P(B \mid A)$. Confusing them is the base-rate error.
• Independence means $P(A \mid B) = P(A)$. Only then may you multiply probabilities.
• Joint is both, marginal is one summed over the other, conditional is one given the other.
• Supervised learning models the conditional $P(y \mid x)$; choosing a label is a later decision.

What to Read Next

• Bayes' Theorem is how you flip a conditional around, and the fix for the first pitfall.
• Probability Distributions puts these rules on whole distributions rather than single events.
• Expectation, Variance and Covariance turns a distribution into the numbers that summarise it.
• Definitions worth a look: Random Variable, Bayes' Theorem, and Expected Value.
• Related interview question: What is maximum likelihood estimation?