Probability Theory

Why Does This Exist?

Machine learning runs on uncertainty. Data is a sample, not the world. Labels are noisy. Models output beliefs, not certainties. Every one of those statements needs a language precise enough to compute with, and probability theory is that language: it assigns numbers to happenings, rules for combining them, and a principled way to update them when news arrives.

This page is the map, not the territory. It introduces the landscape once, names each region, and points to the dedicated page that teaches it. The rules of combining chances live in probability foundations. Counting the outcomes lives in counting and basic probability. Distributions, expectations, and limit laws each have their own pages below. Read this for orientation, then walk the streets.

The one idea you need first

A sample space is the set of everything that could happen, and an event is a subset you care about. Probability assigns each event a number between 0 and 1, with the whole space getting 1. Everything else is consequences of that assignment plus three rules: complements subtract from one, disjoint events add, and conditioning divides by what you learned.

Think of It Like This

How It Actually Works

The theory builds in layers, each resting on the last:

• Outcomes and events. The sample space lists everything possible; events carve out the interesting subsets. Finite spaces use counting, continuous ones use lengths, areas, and volumes.
• The rules. Complements, unions, and intersections combine events, and conditional probability re-normalizes onto what was learned. These three rules plus independence are taught in probability foundations.
• Random variables. Numbers attached to outcomes: the die shows 4, the latency reads 230 milliseconds. Variables come with distributions describing where their values land, covered in probability distributions.
• Expectations. Averages over the distribution: means, variances, and covariances that summarize behavior, taught in expectation, variance and covariance.
• Updating. Bayes' rule turns prior beliefs plus evidence into posterior beliefs, the engine of Bayes' theorem.
• Large samples. Averages settle near expectations and rescaled sums turn bell-shaped, the limit laws of the central limit theorem.

In plain words: name what can happen, weigh the happenings, attach numbers to them, average the numbers, update on news, and trust large samples. Every probabilistic argument in ML is some composition of those six moves.

Worked example

A bag holds 3 red and 7 blue balls. One draw: the sample space has 10 equally likely balls and the red event has 3, so $P(\text{red}) = 3/10 = 0.3$.

Two draws without replacement: after one red leaves, 2 red remain among 9 balls, so the second draw is red with chance $2/9$. Chaining along the tree in the diagram: $P(\text{both red}) = (3/10) \times (2/9) = 6/90 \approx 0.067$. The second fraction differs from the first because the draws are dependent: the first outcome reshapes the second sample space, a preview of conditioning doing real work.

Watch Out For

The Quick Version

• Probability theory assigns numbers between 0 and 1 to events carved from a sample space.
• Conditioning re-normalizes onto what was learned; the sample space moves with every observation.
• Random variables attach numbers to outcomes, distributions weigh them, expectations average them.
• Bayes updates beliefs on news; limit laws describe large samples settling down.
• Drawing twice from 3 red and 7 blue gives both red with chance 6 in 90, because the bag changes.

What to Read Next

• Probability Foundations
• Counting and Basic Probability
• Descriptive Statistics