Advanced Probability

Finite counting rules break on infinite data and exact measurements. Advanced probability rebuilds them on sets and limits so learning theory stays legal.

Finite counting handles three coin flips, but infinite sequences need a sigma-algebra and a probability triple so limits and conditioning stay well defined

Why Does This Exist?

Three coin flips are easy: 8 outcomes, count the good ones, divide. Nothing in that recipe survives contact with real ML. A training stream never ends, a sensor reports an exact value like 0.37 with probability zero under any continuous model, and every generalization claim is a limit as sample size grows.

The naive fix is to wave your hands. That fails the moment you condition on a zero-probability event or swap a limit with an expectation. Advanced probability is the rebuild: outcomes live in a set $\\Omega$ , allowed questions live in a $\\sigma$ -algebra $\\mathcal{F}$ , and probability $P$ is a set-size function with three axioms. Random variables become functions $X: \\Omega \\to \\mathbb{R}$ , and convergence gets four precise meanings instead of one vague one.

This page maps the territory. The machinery itself lives in probability foundations, probability distributions, and expectation, variance and covariance. Here you learn which tool answers which question.

The one idea you need first

A probability is a number between 0 and 1 attached to an event, a set of outcomes. For three fair flips, $P(\\text{at least one head}) = 7/8$ because 7 of 8 outcomes qualify. Everything advanced keeps that meaning and extends it to infinite outcome sets where counting breaks.

Think of It Like This

Surveying land instead of counting bricks

Counting bricks works for a small wall: count them, divide good by total. Try that on a landscape and you fail, because there are no bricks. A surveyor instead draws legal plot boundaries, then measures areas.

$\\Omega$ is the landscape. The $\\sigma$ -algebra $\\mathcal{F}$ is the registry of legal plots: closed under complements and unions, so any sensible combination of plots is also a plot. $P$ is the area survey. A random variable is a function painted over the land, like elevation. Convergence modes are four different promises about what happens as you survey further out.

The analogy stops where all good ones do: land area is finite and visible, while probability mass can concentrate on points, curves, or dust-like sets that no surveyor would recognize.

How It Actually Works

The triple $(\\Omega, \\mathcal{F}, P)$ carries three axioms: $P(A) \\geq 0$ , $P(\\Omega) = 1$ , and countable additivity, meaning the probability of disjoint pieces adds up even for infinitely many of them. Additivity for infinitely many pieces is the whole game; without it, limits escape the theory.

The four convergences

For a sequence $X_n$ approaching $X$ :

Almost sure: paths converge with probability 1. Strongest path claim.
In $L^p$ : $\\mathbb{E}[|X_n - X|^p] \\to 0$ . Moments converge.
In probability: $P(|X_n - X| > \\epsilon) \\to 0$ for every $\\epsilon > 0$ . Outliers allowed if rare.
In distribution: CDFs meet at continuity points. Only laws converge, used by the central limit theorem.

Almost sure and $L^p$ each imply convergence in probability, which implies convergence in distribution. No other arrow holds in general.

Worked example

Flip a fair coin 3 times. $\\Omega$ has $2^3 = 8$ outcomes. The event $A = \\{\\text{at least one head}\\}$ holds 7 of them, so $P(A) = 7/8 = 0.875$ . Now extend to an infinite stream and ask for $B = \\{\\text{heads appear infinitely often}\\}$ . Counting cannot touch $B$ , but the triple assigns $P(B) = 1$ by symmetry and countable additivity. That is the upgrade: same meaning of probability, now defined on events counting could never list.

Code

1import numpy as np2
3rng = np.random.default_rng(0)4flips = rng.integers(0, 2, size=(20000, 3))  # 1 = head5prob = (flips.sum(axis=1) >= 1).mean()  # fraction with at least one head6print(round(float(prob), 3))  # -> 0.874

Twenty thousand trials land at 0.874 against the exact 0.875. The gap is Monte Carlo noise with standard error about 0.002, not a theory problem.

Watch Out For

Conditioning on an exact continuous value

Asking for $P(Y \\mid X = 0.37)$ with continuous $X$ conditions on a probability-zero event, and elementary $P(A\\mid B) = P(A \\cap B)/P(B)$ divides by zero. Practitioners hit this when they filter a dataframe to an exact float and wonder why the conditional looks like noise.

Use densities or a limiting interval: $P(Y \\mid X \\in [0.365, 0.375])$ . If you need the exact point, you need conditional expectation given a $\\sigma$ -algebra, not the elementary formula.

Reading convergence in distribution as path convergence

The central limit theorem says standardized means converge in distribution to a normal. It does not say any single training run looks normal, or that moments converge. A sequence can converge in distribution while every path oscillates forever.

If your argument needs $\\mathbb{E}[X_n] \\to \\mathbb{E}[X]$ , you need uniform integrability or $L^1$ convergence, not the CLT. Check which arrow your proof actually uses.

The Quick Version

Advanced probability keeps the meaning of probability and extends it to infinite outcomes via the triple $(\\Omega, \\mathcal{F}, P)$ .
The $\\sigma$ -algebra lists which questions are legal; closure under complements and countable unions keeps limits inside the theory.
Random variables are functions on $\\Omega$ , so expectation is an integral, not a weighted list.
Four convergence modes answer four questions: paths, moments, rare deviations, or laws.
Almost sure and $L^p$ imply convergence in probability, which implies convergence in distribution.
Zero-probability conditioning needs densities or conditional expectation, never naive division.