Cramer-Rao Bound

No unbiased estimator beats the information in the data. Cramer-Rao turns that slogan into a floor: variance sits at or above one over information.

Unbiased estimators sit on or above the one-over-information floor, with the sample mean touching it and noisier rules floating higher

Why Does This Exist?

Estimator shootouts need a referee. Is your fancy shrinkage rule actually better, or just louder? Without a floor, every comparison is relative: A beats B today, C arrives tomorrow. The Cramer-Rao lower bound (CRLB) ends the regress for unbiased rules: $\\mathrm{Var}(\\hat\\theta) \\geq 1/I(\\theta)$ .

It tells you when to stop tuning variance and start questioning unbiasedness or the model. It builds on Fisher information and certifies the MLEs in maximum likelihood estimation.

The one idea you need first

An estimator is unbiased when $\\mathbb{E}[\\hat\\theta] = \\theta$ : right on average. Variance measures its dataset-to-dataset wobble. The bound floors that wobble using information alone.

Think of It Like This

A speed limit for precision

Information is fuel, variance is travel time. The bound is the speed limit: no unbiased car arrives faster than fuel allows. An efficient estimator drives exactly at the limit; a noisy one dawdles above it.

Biased estimators are electric bikes on a separate path: they can beat the car by cutting corners (accepting bias), but they pay elsewhere. Where it stops: the limit assumes a smooth road (regularity). Potholes like boundaries void the ticket.

How It Actually Works

Under regularity, Cauchy-Schwarz on the score gives $\\mathrm{Var}(\\hat\\theta)\\,I(\\theta) \\geq (\\frac{d}{d\\theta}\\mathbb{E}[\\hat\\theta])^2$ , which is $1$ for unbiased rules. Efficiency is $\\frac{1/I}{\\mathrm{Var}}$ : 1 means optimal. Asymptotically the MLE hits efficiency 1. The multiparameter form floors covariance matrices: $\\mathrm{Cov}(\\hat\\theta) \\succeq I(\\theta)^{-1}$ .

Worked example

Hundred Bernoulli flips at $p = 0.5$ : $I = 400$ , floor $1/400 = 0.0025$ , standard error $0.05$ . The sample mean has variance $p(1-p)/n = 0.0025$ : efficient, touching the floor. A noisier rule adding $N(0, 0.01)$ noise sits at $0.0125$ , five times worse, efficiency $0.2$ . A biased shrunk rule $0.9\\bar X + 0.05$ can dip below $0.0025$ in variance but pays bias up to $0.05$ : below the floor, off the leaderboard the bound referees.

Code

1import numpy as np2
3rng = np.random.default_rng(9)4draws = rng.binomial(100, 0.5, size=20000) / 1005print(round(float(draws.var()), 5))  # -> 0.0025

Twenty thousand repeats of 100 flips show variance 0.0025, exactly the floor. The bound predicts the simulation.

Watch Out For

Applying it to biased rules

Shrinkage, early stopping, and regularization bias on purpose and can beat $1/I$ in variance while losing on bias. Declaring them "impossible" misreads the theorem and blocks useful estimators.

Compare biased rules by mean-squared error $=$ variance $+$ bias $^2$ , not by the unbiased floor. Use biased CRLB forms if you need a bound.

Forgetting regularity at boundaries

At $p = 0$ or constrained edges the score mean is not zero and differentiation under the integral fails. The "floor" number is invalid and attainable-looking estimators are artifacts.

Check interior, smooth support, and finite information first. On edges, switch to constrained or Bayesian analysis.

The Quick Version

Unbiased variance floors at $1/I(\\theta)$ ; efficiency is floor over actual.
Hundred flips at 0.5: floor 0.0025, standard error 0.05, sample mean attains it.
Noisy rules sit above; biased rules play a different game judged by MSE.
The MLE is asymptotically efficient under regularity.
Multiparameter form floors the whole covariance matrix.
Boundaries void the bound; verify regularity before quoting it.