Convergence of Random Variables

There are four ways for randomness to settle down: paths lock on, moments lock on, big deviations get rare, or only the distribution matches.

$Almost sure and Lp convergence each imply convergence in probability, which implies convergence in distribution, and no arrow reverses in general$

Almost sure and Lp convergence each imply convergence in probability, which implies convergence in distribution, and no arrow reverses in general

Why Does This Exist?

"More data helps" is too vague to prove anything. Does the estimate itself settle, do its moments settle, or does only its histogram settle? Each answer licenses different conclusions: deploying a model needs paths to settle, reporting an error bar needs the distribution to settle.

The four modes separate these claims. Estimation consistency is usually convergence in probability. The strong law is almost sure. Mean-squared-error analysis is $L^2$ . Confidence intervals from the central limit theorem are convergence in distribution. Mixing them up is how people claim path guarantees from a CLT.

The one idea you need first

A random variable is a function on outcomes, so a sequence $X_n$ is a sequence of functions. Convergence can mean the values settle for each outcome, or only some average of them settles. The four modes pick which.

Think of It Like This

Darts players converging differently

Four players throw at a bullseye. Ana's darts physically cluster tighter each round: paths converge. Ben's average distance shrinks though stray darts still fly: moments converge. Carla still throws wild ones, but wild throws get rarer: convergence in probability. Dana's throws look like Ana's in aggregate photos, though no single dart behaves: convergence in distribution.

Betting on one specific dart favors Ana. Betting on the photo favors Dana too. Where it stops: darts are independent efforts, while estimators share data, so their joint behavior needs more than the picture.

How It Actually Works

For $X_n \\to X$ :

Almost surely (a.s.): $P(\\{\\omega: X_n(\\omega) \\to X(\\omega)\\}) = 1$ . Every path settles except a null set.
In $L^p$ : $\\mathbb{E}[|X_n - X|^p] \\to 0$ . The $p$ -th moment of the error vanishes; $p = 2$ is mean square.
In probability: for every $\\epsilon > 0$ , $P(|X_n - X| > \\epsilon) \\to 0$ . Deviations get rare.
In distribution: $F_n(x) \\to F(x)$ at continuity points of $F$ . Only laws meet.

The hierarchy

Almost sure $\\Rightarrow$ in probability. $L^p$ $\\Rightarrow$ in probability. In probability $\\Rightarrow$ in distribution. Nothing else holds without extra assumptions: a.s. does not give $L^p$ (mass can escape to infinity), and in distribution does not give in probability (variables may live on different spaces).

Worked example

Let $X_n = 1$ with probability $1/n$ and $0$ otherwise. Fix $\\epsilon = 0.5$ : $P(|X_n| > \\epsilon) = 1/n \\to 0$ , so $X_n \\to 0$ in probability. Also $\\mathbb{E}[|X_n|^2] = 1/n \\to 0$ , so $L^2$ holds too. At $n = 100$ the spike probability is exactly $0.01$ . Yet with independent copies, $\\sum_n 1/n$ diverges, so spikes recur forever and almost-sure convergence to 0 fails. One number, $1/n$ , separates three modes from the fourth.

Code

1import numpy as np2
3rng = np.random.default_rng(2)4for n in [10, 100, 1000, 10000]:5    draws = rng.random(n) < 1 / n  # spike with prob 1/n6    print(n, round(float(draws.mean()), 5))  # -> ~0.1, 0.01, 0.001, 0.0001

Empirical spike rates track $1/n$ : about 0.1, 0.01, 0.001, 0.0001. Rare does not mean eventually absent.

Watch Out For

Upgrading the CLT to a path claim

The CLT gives convergence in distribution of standardized means. It says nothing about any single dataset's mean path, and moments need not converge. Teams sometimes ship "the estimate is now normal" and act as if paths settled.

Use the CLT for intervals and p-values only. For path claims, prove consistency in probability or a strong law instead.

Assuming a.s. gives moment convergence

A sequence can converge on every path while its moments explode: picture spikes of height $n$ with probability $1/n^2$ . Paths still go to 0, but $\\mathbb{E}[X_n^2] = 1$ forever. Bounded or uniformly integrable families close this gap; bare a.s. does not.

Check tails before swapping limits and expectations. If extremes grow as probability shrinks, compute the moment directly.

The Quick Version

Four modes: almost sure controls paths, $L^p$ controls moments, in probability controls rare deviations, in distribution controls laws.
The hierarchy runs a.s. and $L^p$ into in-probability into in-distribution, one way only.
$X_n = 1$ with probability $1/n$ converges in probability and $L^2$ to 0 but spikes forever almost surely.
Consistency of estimators is usually convergence in probability; intervals come from convergence in distribution.
The CLT never promises path or moment convergence by itself.
Almost sure convergence needs extra integrability before moments follow.