\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

S02.9 · Multiple testing

A free, illustrated module of 4 lessons and one interactive in S02 Statistical Inference: Why many tests mislead; Controlling the family-wise error: Bonferroni and Holm; The false discovery rate: Benjamini-Hochberg; Multiplicity in practice.

S02.9 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of S02 Statistical Inference in the Statistics and Data Analysis track.

One test at level 0.05 is wrong 5 percent of the time when its null is true. Twenty such tests are wrong somewhere about 64 percent of the time. These four lessons count the cost of testing many hypotheses at once, control it with Bonferroni’s and Holm’s corrections for the family-wise error rate and with the Benjamini-Hochberg procedure for the false discovery rate, and carry the ideas into practice: Tukey’s comparisons after an ANOVA, many metrics in one experiment, feature screening and reused leaderboards.

What you need first. The p-value, the level alpha and the Type I error (S02.5), power (S02.6) and the union bound (M06.1.4). The last lesson uses one-way ANOVA (S02.7.3).

Lessons

1. Why many tests mislead S02.9.1 25 min

   Test twenty true null hypotheses at level 0.05 and the chance of at least one false positive is 0.64. Derive that number, define the family-wise error rate, and see the same trap in the garden of forking paths and in picking the best of many models or features.

2. Controlling the family-wise error: Bonferroni and Holm S02.9.2 25 min

   Split the level alpha among m tests and the union bound keeps the chance of any false positive at most alpha, with no independence needed. Prove it for Bonferroni and for Holm’s sharper step-down procedure, run both by hand on ten p-values, and compute adjusted p-values with NumPy.

3. The false discovery rate: Benjamini-Hochberg S02.9.3 25 min

   When many hypotheses are tested and some real effects are expected, control the average share of false discoveries instead of the chance of any. Derive the Benjamini-Hochberg sloped line, run it by hand on ten p-values, prove its guarantee and check it with scipy.stats.false_discovery_control.

4. Multiplicity in practice S02.9.4 25 min

   Compare every pair of groups after an ANOVA with Tukey’s honestly significant difference, worked on rent by district, then meet multiplicity where data scientists meet it: many metrics in one experiment, feature screens over thousands of columns, and leaderboards reused until they overfit.

The interactive

S02.9-X1 · Many-tests simulator. Choose the number of tests, the share with a real effect, its size and the level, then generate p-values. They appear sorted against their rank under Bonferroni’s flat line, Holm’s steps and the Benjamini-Hochberg sloped line, with true and false nulls told apart by shape; the read-out counts rejections, false rejections and the false discovery proportion of each method, and can average them over 500 runs. It lives in The false discovery rate: Benjamini-Hochberg.