Multiple Hypothesis Testing

One test at 5% fails 5% of the time. A thousand tests at 5% each fail about fifty times. Family-wise and FDR rules reset the bar for crowds.

A thousand null p-values produce fifty false wins under single-test rules, cut by the flat Bonferroni line and the sloping BH line

Why Does This Exist?

Feature screens, prompt sweeps, and leaderboard climbs test hundreds of ideas and report the winner. At 5% each, 1,000 pure-noise ideas produce about 50 "significant" winners. Publishing the best without the denominator is selection bias with a p-value costume.

Two fixes cover two goals. Bonferroni controls the family-wise chance of any false win at $\\alpha$ by testing each at $\\alpha/m$ . Benjamini-Hochberg (BH) controls the fraction of false wins (FDR) at $q$ by ranking p-values against $(k/m)q$ . It extends hypothesis testing to the bulk regime where question count dominates.

The one idea you need first

A p-value is uniform under the null: 5% fall below 0.05 by construction. Multiply by $m$ tests and the expected false count is $m \\times 0.05$ . The fix moves the bar, not the data.

Think of It Like This

Security lines for a stadium

One gate with a 5% false-alarm rate inconveniences a few fans. A thousand gates at the same rate stop hundreds of innocent fans while the announcer celebrates each catch.

Bonferroni searches every fan thoroughly: slow, almost no false stops. FDR accepts that 10% of stops are wrong but keeps the flow moving. Where it stops: stadiums control physical queues, while tests correlate through shared data, which bends both guarantees.

How It Actually Works

Bonferroni: reject $p_i \\leq \\alpha/m$ . With $m = 1000$ , $\\alpha = 0.05$ , the bar is $0.00005$ : expected false wins $1000 \\times 0.00005 = 0.05$ . BH at level $q$ : sort $p_{(1)} \\leq \\cdots \\leq p_{(m)}$ , find largest $k$ with $p_{(k)} \\leq (k/m)q$ , reject the first $k$ . At $q = 0.1$ with 40 rejections, expect at most 4 false. Holm steps Bonferroni down for extra power; Storey estimates the null proportion $\\pi_0$ for more.

Worked example

Thousand nulls, $\\alpha = 0.05$ : expect $50$ false wins, $P(\\text{at least one}) = 1 - 0.95^{1000} \\approx 1 - 10^{-22} \\approx 1$ . Certain embarrassment. Bonferroni at $0.00005$ cuts expectation to $0.05$ . BH example with sorted p-values $[0.001, 0.004, 0.03, 0.04]$ at $q = 0.1$ , $m = 100$ : cutoffs $[0.001, 0.002, 0.003, 0.004]$ . Only $k = 1$ passes ( $0.001 \\leq 0.001$ ), so reject one. The $0.004$ runner-up fails $0.004 \\leq 0.002$ : close is not enough.

Code

1import numpy as np2
3rng = np.random.default_rng(10)4pvals = rng.random(1000)  # all null, uniform5print(int((pvals < 0.05).sum()))  # -> ~506print(int((pvals < 0.05 / 1000).sum()))  # -> 0

Null uniforms trip the single bar about 50 times and the Bonferroni bar zero. The denominator decides the headline.

Watch Out For

Reporting the winner without the sweep size

"Best of 500 prompts gains 3 points, p = 0.01" hides 499 losers. Unadjusted, that p-value is one of 500 draws, and a 0.01 best is expected under pure noise. Leaderboards and ab-test peeking run on this fuel.

Log and report $m$ : every prompt, seed, and checkpoint tested. Adjust or hold out fresh data for the winner. No denominator, no claim.

Reading FDR as per-discovery certainty

FDR $q = 0.1$ means about 10% of rejections are false on average, not that each rejection is 90% true. With 5 rejections, one false is normal, and which one is unknown.

Follow up rejections with validation. If each discovery triggers spend, tighten $q$ or confirm independently.

The Quick Version

Single-test 5% on 1,000 nulls yields about 50 false wins and near-certain embarrassment.
Bonferroni tests each at $\\alpha/m$ ; at 1000 the bar is 0.00005.
BH ranks p-values against $(k/m)q$ and caps the false share of wins.
Example rejects one of four candidates at $q = 0.1$ ; near-misses still fail.
Always report the sweep size $m$ ; unadjusted best-of-sweep p-values are fiction.
FDR controls the batch fraction, not each discovery's truth.