Radon-Nikodym Theorem

When one distribution has a density against another, and the exact condition that guarantees the density exists.

Radon Nikodym gives the density as a slope: one measure divided by another where absolute continuity holds

Why Does This Exist?

Densities are quotients: $p(x) = dP/d\mu$ behaves like $P$ divided by a base measure $\mu$ . Likelihood ratios, importance weights and ELBO densities all assume such a quotient exists. Sometimes it does not, as when a point mass meets Lebesgue measure.

Radon-Nikodym states the exact license: absolute continuity plus $\sigma$ -finiteness yields a density, unique up to null sets. Singular parts are the obstruction.

This page states the condition and the ML reading. Base structure is in measure spaces and working densities in probability distributions.

The one idea you need first

$\nu \ll \mu$ (absolute continuity) means $\mu(A) = 0$ implies $\nu(A) = 0$ . No event null under $\mu$ carries mass under $\nu$ . This is strictly weaker than equality of measures.

Think of It Like This

Exchange rates between currencies

Two currencies price the same goods. Absolute continuity means no good free in euros costs dollars: zero in one is zero in the other. The density is the exchange rate varying by good.

A point mass is a good free in Lebesgue terms but priced under counting: no finite rate converts, so no density exists. Where the analogy stops: rates are single numbers, while Radon-Nikodym densities vary measurably across the space.

How It Actually Works

Theorem: if $\mu$ is $\sigma$ -finite and $\nu \ll \mu$ is $\sigma$ -finite, there is measurable $f \ge 0$ with $\nu(A) = \int_A f \, d\mu$ for all measurable $A$ . Write $f = d\nu/d\mu$ . Lebesgue decomposition splits general $\nu = \nu_{ac} + \nu_s$ into an absolutely continuous plus a singular part.

ML reading: importance weight $w = dP/dQ$ exists when $P \ll Q$ ; KL $D(P\|Q) = \int \log(dP/dQ) \, dP$ needs the same; Bayes updates need the likelihood density against the dominating product.

Worked example

On $\{1, 2, 3\}$ , $\mu = (0.45, 0.32, 0.23)$ and $\nu = (0.225, 0.32, 0.46)$ . Ratios: $0.225/0.45 = 0.5$ , $0.32/0.32 = 1.0$ , $0.46/0.23 = 2.0$ . So $f = (0.5, 1.0, 2.0)$ and $\nu(\{1,3\}) = 0.5 \times 0.45 + 2.0 \times 0.23 = 0.685$ , matching $0.225 + 0.46$ . Add a point mass at $0$ to $\nu$ with $\mu(\{0\}) = 0$ : absolute continuity fails, no density exists.

Watch Out For

Writing densities across singular pairs

Symptom: a Dirac spike given a "density value" under Lebesgue, then KL computed as finite when it is infinite. Fix: test $\nu \ll \mu$ first; mixtures with atoms need a mixed base of Lebesgue plus counting.

Ignoring the null-set non-uniqueness

Symptom: two density versions differ at one point and are treated as different models, breaking likelihood comparisons. Fix: densities are unique only almost everywhere; pointwise differences on null sets carry no probabilistic meaning.

The Quick Version

Densities exist exactly under absolute continuity with a $\sigma$ -finite base.
The density integrates back to the target: $\nu(A) = \int_A f \, d\mu$ .
General measures split into density plus singular parts.
Ratios like $(0.5, 1.0, 2.0)$ are the discrete Radon-Nikodym derivative.
Importance weights, likelihoods and KL all assume this license.