Sufficient Statistics

A sufficient statistic keeps everything the data says about the parameter and discards the rest. Ten flips compress to one head count with zero loss.

Ten ordered flips carrying 1024 possibilities compress into the head count k equals 6, which holds all information about the bias

Why Does This Exist?

Datasets are huge and parameters are few. Storing every flip to estimate one bias $p$ wastes memory and invites overfitting to noise. Sufficiency says when compression is free: $T(X)$ is sufficient for $\\theta$ if the conditional law of $X$ given $T(X)$ no longer depends on $\\theta$ .

Then the leftover detail is pure noise for your purpose. The Fisher-Neyman theorem makes it checkable: $T$ is sufficient exactly when the likelihood splits as $g(T(x), \\theta)\\,h(x)$ . It connects exponential family distributions to maximum likelihood estimation.

The one idea you need first

Likelihood is $L(\\theta; x) = p(x \\mid \\theta)$ viewed as a function of $\\theta$ . If two datasets give likelihoods proportional in $\\theta$ , they say the same thing about $\\theta$ . Sufficiency keeps exactly that shape.

Think of It Like This

Election totals versus ballot order

Ten thousand ballots arrive in some order. To estimate vote share you need only the totals, not the sequence: every ordering with 6,000 for A says the same thing about A's support.

The count is sufficient; the order is noise given the count. Where it stops: order matters for fraud detection or time trends. Sufficiency is always for a parameter and a model; change the question and the "noise" becomes signal.

How It Actually Works

Fisher-Neyman: find $g$ and $h$ with $p(x \\mid \\theta) = g(T(x), \\theta)\\,h(x)$ . Only $g$ mentions $\\theta$ , through $T$ . Minimal sufficient statistics compress furthest without loss; complete ones additionally admit unique unbiased estimation. Rao-Blackwell improves any estimator by conditioning on a sufficient $T$ : $\\hat\\theta_{RB} = \\mathbb{E}[\\hat\\theta \\mid T]$ never has larger mean-squared error.

Worked example

Ten i.i.d. Bernoulli( $p$ ) flips with $k = 6$ heads. Likelihood: $p^6(1-p)^4$ , depending on data only through $k$ . Number of sequences mapping to $k = 6$ is $\\binom{10}{6} = 210$ out of $2^{10} = 1024$ , yet all 210 give the same likelihood shape in $p$ . MLE: $\\hat p = k/n = 0.6$ uses only $k$ . Given $k = 6$ , every one of the 210 orders is equally likely regardless of $p$ : the order is ancillary noise.

Code

1import math2
3n, k = 10, 64print(math.comb(n, k))  # -> 2105print(round(k / n, 2))  # -> 0.6

210 orders share the count 6, and the estimate 0.6 needs nothing else. The combinatorial factor absorbs the discarded order.

Watch Out For

Calling an estimator sufficient

$\\hat p = 0.6$ is a number; sufficiency is a property of a statistic's $\\sigma$ -algebra. An estimator can be a function of a sufficient statistic without being sufficient itself, and conditioning on a bare estimate can lose information the full statistic kept.

Keep the statistic, derive the estimator from it. Compress to $T$ , then compute $\\hat\\theta(T)$ .

Reusing sufficiency under a new model

Sufficiency is model-relative. The head count is sufficient for $p$ under i.i.d. Bernoulli, but not under a Markov chain where transition counts matter, nor for detecting a mid-stream change point. New likelihood, new factorization, possibly new $T$ .

Re-verify Fisher-Neyman after any model change. If the likelihood's data dependence changes shape, so does $T$ .

The Quick Version

$T(X)$ is sufficient when $X \\mid T(X)$ forgets $\\theta$ ; the rest is noise for that question.
Fisher-Neyman: likelihood splits into $g(T, \\theta)$ times $h(x)$ .
Ten flips with 6 heads: 210 orders share one likelihood shape; $\\hat p = 0.6$ needs only $k$ .
Rao-Blackwell conditions any estimator on $T$ and never hurts mean-squared error.
Sufficiency belongs to a statistic under a model, not to a bare estimate.
Change the model and re-derive $T$ ; yesterday's compression may now lose signal.