Empirical Bayes

Small groups give noisy estimates. Empirical Bayes learns one shared prior from all groups, then lets each group borrow exactly as much as it needs.

Five hospital rates scatter widely, but the 1-for-5 small sample shrinks from 0.2 toward the learned 0.3 prior mean

Why Does This Exist?

A hospital with 1 success in 5 trials reports 0.2; the national rate is 0.3. Is the hospital worse or just small? Raw rates say worse; pooled-everything says average. Both are wrong: small groups need partial pooling, borrowing strength in proportion to their noise.

Empirical Bayes does it by estimating the prior from the ensemble: fit hyperparameters $\\hat\\alpha$ by marginal likelihood across groups, then post $p(\\theta_j \\mid y_j, \\hat\\alpha)$ per group. Big groups barely move; tiny ones shrink hard. It extends Bayesian inference toward hierarchical modeling without a full hyperprior.

The one idea you need first

A Beta $(\\alpha, \\beta)$ prior plus Binomial data gives posterior mean $(k + \\alpha)/(n + \\alpha + \\beta)$ : data plus pseudo-counts. Empirical Bayes learns those pseudo-counts from all groups' margins.

Think of It Like This

Grading on a curve set by all classes

One class of 5 students averages 60%; the school averages 75%. The teacher reports neither 60 nor 75 but a blend weighted by class size: tiny classes lean on the school, huge classes stand alone.

The curve (prior) is estimated from every class, then applied back to each. Where it stops: if classes differ systematically, one curve undershrinks the odd ones. Grouped or hierarchical priors fix that.

How It Actually Works

Model $y_j \\mid \\theta_j$ with $\\theta_j \\sim p(\\cdot \\mid \\alpha)$ . Marginal $m(y \\mid \\alpha) = \\prod_j \\int p(y_j \\mid \\theta_j)p(\\theta_j \\mid \\alpha)d\\theta_j$ ; maximize over $\\alpha$ for $\\hat\\alpha$ . Then per-group posterior with $\\hat\\alpha$ plugged in. James-Stein is the Gaussian flagship: shrinking toward the grand mean beats raw means in total squared error for $3+$ groups. Report shrinkage factors $B_j$ so readers see how much pooling each group got.

Worked example

Global prior learned as Beta $(3, 7)$ : mean $3/10 = 0.3$ , strength 10 pseudo-trials. Small hospital 1 of 5: posterior mean $(1+3)/(5+10) = 4/15 \\approx 0.2667$ , pulled from 0.2 toward 0.3. Large hospital 300 of 1000: $(300+3)/(1000+10) = 303/1010 \\approx 0.3000$ , raw 0.3 unchanged to three decimals. Same prior, proportional borrowing.

Code

1alpha, beta = 3.0, 7.02for k, n in [(1, 5), (300, 1000)]:3    post = (k + alpha) / (n + alpha + beta)4    print(k, n, round(post, 4))  # -> 0.2667 and 0.3

Small $n$ moves 0.067 toward the prior; large $n$ moves 0.000. Shrinkage is automatic sample-size weighting.

Watch Out For

Double-dipping uncertainty away

Plugging $\\hat\\alpha$ in as known understates uncertainty: the prior was estimated, not given. Naive intervals run narrow, especially with few groups.

Use fully Bayesian hyperpriors or bootstrap the $\\alpha$ step when groups are few. If group count is under 10, distrust plug-in widths.

Shrinking across unlike groups

One prior for urban and rural hospitals, or all product categories, overshrinks outliers toward an irrelevant center. Real differences get smoothed into the average and the flagged groups look fine.

Stratify or model group features before pooling. If shrinkage factors cluster by known type, the single prior was wrong.

The Quick Version

Empirical Bayes learns the prior's hyperparameters from the ensemble margin, then posts per group.
Beta $(3,7)$ prior: 1-of-5 shrinks 0.2 to 0.267; 300-of-1000 stays 0.300.
Small samples borrow heavily, large ones stand alone; shrinkage factors show how much.
James-Stein proves shrinkage beats raw means for multiple groups under squared error.
Plug-in priors understate uncertainty with few groups; go hierarchical or bootstrap.
Pool like with like; stratify before shrinking.