Kolmogorov Extension Theorem

If your finite snapshots of a random process agree with each other when you zoom out, exactly one infinite process extends them all. Consistency builds the whole film.

Three nested finite laws project onto each other by summing out coordinates, and one infinite law extends them all

Why Does This Exist?

Stochastic processes run forever: infinite coin tosses, lifelong sensor streams, endless text. Nobody can write down an infinite-dimensional distribution directly, so practitioners specify what they can: the law of the first toss, of the first two, of the first ten. But arbitrary finite snapshots may contradict each other, and then no infinite process matches them all.

Kolmogorov's theorem names the single check that prevents this. If snapshots agree under marginalization, exactly one infinite law extends them. Gaussian processes, Brownian motion and every autoregressive text model stand on this guarantee: finite pieces you can write down, infinite behavior you actually want. The pieces live in probability spaces; assembling independent ones uses product measures.

The one idea you need first

Marginalization sums (or integrates) out coordinates you stop tracking. If $P_2$ governs two tosses, its first-toss marginal is $P_2(X_1 = a) = \sum_b P_2(a, b)$ . Consistency demands this marginal equal the directly specified $P_1(a)$ for every $a$ .

Think of It Like This

A film from matching stills

An animator draws frame 1, then frames 1 to 2, then frames 1 to 3. Each longer strip must start exactly like the shorter one it extends: same poses, same timing. If strip 3 opens differently than strip 2 did, no single film contains both.

Consistency is that matching-edges rule. Kolmogorov says matching strips splice into exactly one infinite reel. Where it stops: the theorem hands you existence on abstract sequence space, not a simulator. Sampling the infinite object efficiently is a separate engineering problem.

How It Actually Works

For each $n$ , specify a law $P_n$ on the first $n$ coordinates. The family is consistent when projecting $P_{n+1}$ down recovers $P_n$ :

P_n(A) = P_{n+1}(A \times \mathcal{X}) \quad \text{for every event } A \text{ about the first } n \text{ coordinates}

In plain words: $P_{n+1}$ is a law about $n+1$ variables, $A \times \mathcal{X}$ ignores the last one, and consistency says the long snapshot agrees with the short one wherever they overlap. The theorem: on standard spaces (finite alphabets, Euclidean chunks), a consistent family extends to exactly one law $P$ on infinite sequences whose every $n$ -marginal is $P_n$ . Uniqueness comes free because finite-cylinder events pin down the whole law.

Worked example

Fair coin tosses. $P_1$ gives heads and tails $1/2$ each. $P_2$ gives each pair $1/4$ . Check consistency: $P_2(X_1 = H) = P_2(H,H) + P_2(H,T) = 1/4 + 1/4 = 1/2 = P_1(H)$ . Same for tails.

$P_3$ gives each triple $1/8$ ; projecting onto two coordinates gives $1/8 + 1/8 = 1/4 = P_2$ on every pair. So the family is consistent, and Kolmogorov delivers the unique infinite fair-coin law $P$ . Break it instead: declare $P_2(H,H) = 1/2$ and $P_2 = 1/6$ on the other three pairs. Then $P_2(X_1 = H) = 1/2 + 1/6 = 2/3 \ne 1/2$ : inconsistent, no infinite process extends both.

Code

Check marginal consistency numerically: project the two-toss law down and compare with the one-toss law.

1import numpy as np2
3
4def is_consistent(p1: np.ndarray, p2: np.ndarray) -> tuple[bool, np.ndarray]:5    marginal = p2.sum(axis=1)  # sum out the second toss6    return bool(np.allclose(marginal, p1)), marginal7
8
9p1 = np.array([0.5, 0.5])10p2 = np.full((2, 2), 0.25)11print(is_consistent(p1, p2))  # -> (True, array([0.5, 0.5]))

Marginal $[0.5, 0.5]$ matches $P_1$ : consistent, so the extension exists.

Watch Out For

Assuming any finite specs extend to a process

Symptom: separately fitted n-gram tables for different $n$ contradict each other, and sampling drifts between contexts. Fix: build longer laws by extending shorter ones (conditionals times marginals) rather than fitting each order independently.

Forgetting the space must be standard

Symptom: consistency holds yet pathologies appear (no regular conditional, no continuous version) on exotic function spaces. Fix: stay on Polish spaces (complete separable metric) where the theorem's conclusion is watertight; that covers every ML use.

The Quick Version

Finite snapshots $P_n$ must marginalize onto each other; that agreement is consistency.
Consistent families on standard spaces extend to exactly one infinite law.
Fair coins pass the check: $1/4 + 1/4 = 1/2$ at every projection step.
Inconsistent snapshots admit no common process, which independent per-order fitting risks.
Gaussian processes and autoregressive models inherit their legitimacy from this theorem.