Skip to main content

Controlling the family-wise error: Bonferroni and Holm

Split the level alpha among m tests and the union bound keeps the chance of any false positive at most alpha, with no independence needed. Prove it for Bonferroni and for Holm’s sharper step-down procedure, run both by hand on ten p-values, and compute adjusted p-values with NumPy.
S02.9.2L2 Intermediate25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: S02.9.1 Why many tests mislead M06.1 Sample spaces, events and axioms

What you will be able to do
  • Apply Bonferroni’s correction, rejecting when a p-value is at most α/m, and prove from the union bound that it keeps the family-wise error rate at most α for any dependence.
  • Carry out Holm’s step-down procedure and prove that it controls the same rate while rejecting at least as much as Bonferroni.
  • Compute Bonferroni and Holm adjusted p-values from scratch and read decisions off them.
  • State Šidák’s cut-off 1 − (1 − α)^(1/m) and when it applies.

The question

Ten features were each tested for a link with a label, giving the course’s ten p-values (invented): 0.001, 0.008, 0.012, 0.030, 0.041, 0.09, 0.20, 0.38, 0.55 and 0.81. Five are below 0.05. If you want at most a 5 percent chance that any feature you report is a false positive, which of them may you report?

From the last lesson

1. What is the chance of at least one false positive among 20 independent tests of true nulls at level 0.05?

Show answer

\({1 - 0.95^{20}} = 0.64\).

2. What is the expected number of false positives among \(m_0\) tests of true nulls at level \(\alpha\), and does it need independence?

Show answer

\({m_0 \alpha}\), with or without independence: it is a sum of indicator expectations.

Intuition

Think of \(\alpha = 0.05\) as a budget of false-alarm risk for the whole family. One test may spend all of it. Ten tests must share it, and the simplest share is equal: 0.005 each. That is Bonferroni’s correction.

Holm noticed that the budget is spent only on true nulls. Once the smallest p-value has been rejected, there are at most nine true nulls left to protect, so the next test may have 0.05/9.

The ten p-values sorted by rank from 1 to 10, plotted as blue dots on a logarithmic vertical axis from 0.0005 to 1. A grey dotted line at 0.05 is labelled no correction: 0.05, 5 pass. A black dashed line at 0.005 is labelled Bonferroni: 0.05 over 10 equals 0.005. An orange staircase rises from 0.005 at rank 1 through 0.0056, 0.0063 and so on to 0.05 at rank 10, labelled Holm: 0.05 over (m minus k plus 1). Only the dot at rank 1, 0.001, lies under both, in an orange band marked rejected. A pointer at rank 2 reads: 0.008 misses its step 0.0056: Holm stops.
S02.9.2-F1

The ten p-values, smallest first, on a log scale. Five pass the uncorrected 0.05. Bonferroni’s flat line at 0.005 and Holm’s rising steps each let only the first through.

On the log scale the two rules look alike at the bottom, where both start at 0.005, and differ at the top, where Holm’s last step reaches the full 0.05. Holm never rejects less.

Build it up

Step 1. Split the budget. Give each of the \(m\) tests the level \(\alpha/m\). With ten tests at 0.05, each p-value must be at most 0.005.

Step 2. Add up the risks. A test of a true null at level \({\alpha/m}\) rejects with chance at most \({\alpha/m}\). The chance that at least one of the true nulls is rejected is the probability of a union, and the union bound of M06.1.4 says it is at most the sum of the chances. Overlap between the events only makes the union smaller than the sum.

Two panels. Left, titled Split the budget: an orange bar labelled the budget alpha equals 0.05, cut into ten equal cells, with the text ten equal shares of 0.005, one per test, and test i rejects if p sub i is at most alpha over m. Right, titled Overlap only helps: a large square labelled all outcomes containing four purple rectangles, two of them overlapping, with the text each box: one true null rejected, chance at most 0.005.
S02.9.2-F2

Bonferroni in two pictures. The budget is split equally. The union of the rejection events can cover no more area than the boxes would side by side, and less when they overlap.

Step 3. Spend what is left. Sort the p-values. If the smallest passes \(\alpha/m\), call it a discovery and treat that hypothesis as false. The rest of the family has \({m - 1}\) hypotheses, so the next smallest is held to \({\alpha/(m - 1)}\), the one after to \({\alpha/(m - 2)}\), and so on up to \(\alpha\) itself.

Step 4. Stop at the first failure. The first p-value that misses its step ends the procedure; it and every larger one are kept. This is Holm’s step-down procedure.

The formal statement

Write the p-value of test \(i\) as \(\text{p}_i\), with an upright p to keep it apart from a proportion \(p\), and the sorted values as \({\text{p}_{(1)} \leq \text{p}_{(2)} \leq \dots \leq \text{p}_{(m)}}\), with \(H_{(k)}\) the hypothesis of the \(k\)-th smallest. A valid p-value of a true null satisfies \({\Prob(\text{p}_i \leq u) \leq u}\) for every \(u\) in \([0, 1]\).

NoteBonferroni and Holm

Bonferroni: reject \(H_i\) when \({\text{p}_i \leq \alpha/m}\).

Holm: find the first \(k\) with \({\text{p}_{(k)} > \alpha/(m - k + 1)}\) and reject \({H_{(1)}, \dots, H_{(k-1)}}\); if there is no such \(k\), reject all \(m\).

Both keep \({\text{FWER} \leq \alpha}\), whatever the dependence between the tests.

Bonferroni, proof. Let \(I_0\) be the set of true nulls, with \(m_0\) members. A false positive happens when some \(i\) in \(I_0\) has \({\text{p}_i \leq \alpha/m}\). By the union bound,

\[ \begin{aligned} \text{FWER} &= \Prob\big(\textstyle\bigcup_{i \in I_0} \{\text{p}_i \leq \alpha/m\}\big) \\ &\leq \textstyle\sum_{i \in I_0} \Prob(\text{p}_i \leq \alpha/m) \\ &\leq m_0 \cdot \alpha/m \;\leq\; \alpha . \end{aligned} \]

No step used independence. The bound is close to tight when the tests are independent and \(\alpha\) is small: for ten independent true nulls the FWER is \({1 - 0.995^{10}} = 0.0489\).

Holm, proof. Look at the true null with the smallest p-value; call its rank \(k^*\). All the hypotheses ranked before it are false nulls, and there are at most \({m - m_0}\) of those, so \({k^* \leq m - m_0 + 1}\). The denominator of its step is therefore

\[ \begin{aligned} &m - k^* + 1 \\ &\quad\geq m - (m - m_0 + 1) + 1 \\ &\quad= m_0 , \end{aligned} \]

so its step \({\alpha/(m - k^* + 1)}\) is at most \({\alpha/m_0}\).

Holm rejects in order and stops at the first failure, so if it rejects any true null it must reject this first one, which needs \({\text{p}_{(k^*)} \leq \alpha/m_0}\). Hence

\[ \begin{aligned} \text{FWER} &\leq \Prob\big(\textstyle\min_{i \in I_0} \text{p}_i \leq \alpha/m_0\big) \\ &\leq m_0 \cdot \alpha/m_0 = \alpha , \end{aligned} \]

by the union bound again.

Ten sorted p-values drawn as a row of markers with ranks 1 to 10 beneath and Holm's steps alpha over 10, alpha over 9, down to alpha over 1 above. Filled blue diamonds are false nulls at ranks 1 to 4, 6 and 8; hollow blue circles are true nulls at ranks 5, 7, 9 and 10. The circle at rank 5 is ringed in purple and labelled first true null: rank 5, step alpha over 6. A box beneath explains that at most m minus m0 equals 6 false nulls come before it, so its rank is at most 7 and its step at most alpha over 4 equals alpha over m0, and that the union bound then gives a chance of at most alpha.
S02.9.2-F4

The heart of Holm’s proof, for \({m = 10}\) and \({m_0 = 4}\). However the false nulls fall, the first true null meets a step no larger than \({\alpha/m_0}\).

Holm rejects at least as much. Holm’s first step is \({\alpha/m}\) and the later ones are larger. If \({\text{p}_{(k)} \leq \alpha/m}\), then every smaller p-value also passes its own step, so Holm reaches rank \(k\) and rejects it. Every Bonferroni rejection is a Holm rejection.

Šidák’s cut-off. If the tests are independent, a level \(c\) for each gives \({\text{FWER} = 1 - (1 - c)^{m_0}}\) by the previous lesson. Setting \({1 - (1 - c)^m = \alpha}\) and solving gives \({c = 1 - (1 - \alpha)^{1/m}}\): 0.005116 for ten tests, a hair above 0.005. It needs independence; Bonferroni does not.

Adjusted p-values. Rather than move the cut-off, move the p-values: the adjusted p-value of a hypothesis is the smallest family-wise level at which the procedure would reject it. For Bonferroni, \({\text{p}_i \leq \alpha/m}\) exactly when \({m\,\text{p}_i \leq \alpha}\), so the adjusted value is \({\min(1, m\,\text{p}_i)}\). For Holm, \(H_{(k)}\) is rejected exactly when every earlier step passes, that is when \({(m - j + 1)\,\text{p}_{(j)} \leq \alpha}\) for all \({j \leq k}\). Writing \({a_j = (m - j + 1)\,\text{p}_{(j)}}\),

\[ \tilde{\text{p}}_{(k)} = \min\big(1,\ \max_{j \leq k} a_j\big). \]

Worked example

Bonferroni and Holm on the ten p-values, at \(\alpha = 0.05\). Here \(m = 10\).

A table with ten rows and five columns: rank k, p-value, step 0.05 over (11 minus k), verdict, and Holm adjusted. Row 1: 0.001, step 0.005, a green tick, adjusted 0.01 in green. Row 2: 0.008, step 0.0056, an orange cross with the word stop, adjusted 0.072. Rows 3 to 10 are grey, marked not tested, with steps 0.0063, 0.0071, 0.0083, 0.01, 0.0125, 0.0167, 0.025 and 0.05 and adjusted values 0.096, 0.21, 0.246, 0.45, 0.8, 1, 1 and 1. A box beneath reads: Holm rejects 1; Bonferroni compares all ten with 0.005 and also rejects 1.
S02.9.2-F3

Holm’s procedure, one row at a time, with the Holm adjusted p-values. The second row fails its step, so the procedure stops there.

  1. Bonferroni: the cut-off is \({0.05/10} = 0.005\). Only 0.001 is below it: one rejection.
  2. Holm, rank 1: is \({0.001 \leq 0.05/10} = 0.005\)? Yes, reject.
  3. Holm, rank 2: is \({0.008 \leq 0.05/9} = 0.0056\)? No. Stop. One rejection.
  4. Holm adjusted values: \({10 \times 0.001} = 0.01\), then \({9 \times 0.008} = 0.072\), then \({8 \times 0.012} = 0.096\), and so on. At rank 8, \({3 \times 0.38} = 1.14\) is capped at 1, and the running maximum keeps ranks 9 and 10 at 1 although \({1 \times 0.81} = 0.81\).
  5. Šidák’s cut-off 0.005116 also passes only 0.001.

Five p-values were under 0.05, but only one survives the family-wise guarantee. Holm’s extra power shows only when several p-values are small together, as in exercise 1.

Code

The three corrections from scratch, each returning the adjusted p-values in the original order.

import numpy as np

pvals = np.array([0.001, 0.008, 0.012, 0.030, 0.041, 0.09, 0.20, 0.38, 0.55, 0.81])  # invented
alpha, m = 0.05, len(pvals)

def bonferroni(p):
    return np.minimum(1, len(p) * np.asarray(p))

def holm(p):
    p = np.asarray(p)
    order = np.argsort(p)
    steps = (len(p) - np.arange(len(p))) * p[order]          # (m - k + 1) p_(k)
    adjusted = np.empty_like(p)
    adjusted[order] = np.minimum(1, np.maximum.accumulate(steps))
    return adjusted

def sidak(p):
    return 1 - (1 - np.asarray(p)) ** len(p)

print("Bonferroni:", bonferroni(pvals).round(3), "rejects", int(np.sum(bonferroni(pvals) <= alpha)))
print("Holm:      ", holm(pvals).round(3), "rejects", int(np.sum(holm(pvals) <= alpha)))
print("Sidak cut-off:", round(1 - (1 - alpha) ** (1 / m), 6), "rejects", int(np.sum(sidak(pvals) <= alpha)))
Bonferroni: [0.01 0.08 0.12 0.3  0.41 0.9  1.   1.   1.   1.  ] rejects 1
Holm:       [0.01  0.072 0.096 0.21  0.246 0.45  0.8   1.    1.    1.   ] rejects 1
Sidak cut-off: 0.005116 rejects 1

Now a check by simulation, 20,000 families of ten one-sided z-tests (seed 4). In the first, every null is true; in the second, the last five hypotheses have a real effect of 3 standard errors.

from scipy import stats

def holm_rows(p, alpha=0.05):
    """Holm's rejections for every row of a matrix of p-values at once."""
    return np.apply_along_axis(holm, 1, p) <= alpha

rng = np.random.default_rng(4)
z = rng.normal(size=(20_000, 10))
p_null = stats.norm.sf(z)                                  # all ten nulls true
z[:, 5:] += 3                                             # five real effects
p_mix = stats.norm.sf(z)

print("all null: FWER none", np.mean((p_null <= alpha).any(1)), " Bonferroni",
      np.mean((p_null <= alpha / m).any(1)))
bon, hol = p_mix <= alpha / m, holm_rows(p_mix)
print("five real: FWER Bonferroni", bon[:, :5].any(1).mean(), " Holm", hol[:, :5].any(1).mean())
print("real effects found: Bonferroni", bon[:, 5:].sum(1).mean(), " Holm", hol[:, 5:].sum(1).mean())
all null: FWER none 0.40505  Bonferroni 0.04995
five real: FWER Bonferroni 0.0247  Holm 0.0371
real effects found: Bonferroni 3.3198  Holm 3.5321

With every null true, no correction errs in about 40 percent of families and Bonferroni in about 5 percent (0.0489 in theory). With five real effects, both stay under 0.05 (Bonferroni about 0.025, since only five nulls are true, and Holm about 0.037), and Holm finds about 3.5 of the five real effects against Bonferroni’s 3.3. SciPy has no Holm function; the next lesson’s scipy.stats.false_discovery_control covers the false discovery rate instead.

Explore

The many-tests simulator is in lesson S02.9.3. Load the ten p-values there and compare Bonferroni’s flat line with Holm’s steps; then set 50 tests with 40 percent real effects of size 3 and watch Holm’s count pull ahead of Bonferroni’s.

Pitfalls

Two panels with logarithmic p-value axes. Left, marked with a cross and titled Divide by 3: only the three smallest p-values, 0.001, 0.008 and 0.012, as purple dots, all under a dashed line at 0.05 over 3 equals 0.0167, with the note 3 of 3 pass. Right, marked with a tick and titled Divide by the 10 you ran: all ten p-values as blue dots against a dashed line at 0.05 over 10 equals 0.005; only the first is under it, noted 1 of 10 passes.
S02.9.2-F5

Wrong beside right. Reporting the three smallest p-values and correcting for three makes all of them “significant”. The family is the ten tests that were run.

“I report three features, so I divide by three.”

The selection already used the other seven p-values: the three were chosen because they were small. The correction must count every test that was run, and with ten that leaves one discovery, not three.

“Bonferroni assumes the tests are independent.”

It assumes nothing about dependence; its proof uses only the union bound. Šidák’s slightly larger cut-off is the one that needs independence. Under strong positive dependence Bonferroni is conservative: its FWER falls well below \(\alpha\).

“Holm’s adjusted p-value is just \({(m - k + 1)\,\text{p}_{(k)}}\).”

Without the running maximum the adjusted values can go down as the raw ones go up: the ten p-values would give 1.14, 1.1 and 0.81 at ranks 8 to 10, so the largest p-value would look the strongest. The maximum over \({j \leq k}\) encodes “stop at the first failure”.

Check yourself

Recall

1. What is Bonferroni’s cut-off for \(m\) tests at family-wise level \(\alpha\), and what is it for the ten p-values?

Show answer

\({\alpha/m}\); \({0.05/10} = 0.005\), which only 0.001 passes.

2. Which inequality proves that Bonferroni controls the FWER, and does the proof need independence?

Show answer

The union bound: the chance of a union is at most the sum of the chances. No independence is needed.

3. In Holm’s procedure, what is the \(k\)-th smallest p-value compared with, and when does the procedure stop?

Show answer

With \({\alpha/(m - k + 1)}\). It stops at the first p-value that is larger than its step.

4. Why does Holm never reject less than Bonferroni?

Show answer

Its steps are all at least \({\alpha/m}\), so any p-value that passes \({\alpha/m}\) is reached and passes its own step, as do all smaller ones.

5. What is the Bonferroni adjusted p-value of 0.008 among ten tests?

Show answer

\({\min(1, 10 \times 0.008)} = 0.08\).

Exercises

1. ★ Four tests give p-values 0.004, 0.011, 0.019 and 0.30. At family-wise level 0.05, which hypotheses do Bonferroni and Holm reject?

Show solution

Bonferroni’s cut-off is \({0.05/4} = 0.0125\): it rejects 0.004 and 0.011, two hypotheses.

Holm: \({0.004 \leq 0.05/4} = 0.0125\), reject; \({0.011 \leq 0.05/3} = 0.0167\), reject; \({0.019 \leq 0.05/2} = 0.025\), reject; \({0.30 > 0.05}\), stop. Holm rejects three: the third p-value is too big for Bonferroni but small enough once two hypotheses have been set aside.

2. ★★ Compute the Holm adjusted p-values for exercise 1 and check that they give the same decisions.

Show solution

The products \({(m - k + 1)\,\text{p}_{(k)}}\) are \({4 \times 0.004} = 0.016\), \({3 \times 0.011} = 0.033\), \({2 \times 0.019} = 0.038\) and \({1 \times 0.30} = 0.30\). They already increase, so the running maximum leaves them alone: 0.016, 0.033, 0.038 and 0.30. The first three are at most 0.05, the same three rejections.

3. ★★★ Show that Šidák’s cut-off is never smaller than Bonferroni’s: \({1 - (1 - \alpha)^{1/m}} \geq {\alpha/m}\) for \(\alpha\) in \([0, 1]\) and \(m \geq 1\).

Show solution

Bernoulli’s inequality (exercise 3 of S02.9.1) with \(\alpha/m\) in place of \(\alpha\) gives

\[ \begin{aligned} \Big(1 - \frac{\alpha}{m}\Big)^m &\geq 1 - m \cdot \frac{\alpha}{m} \\ &= 1 - \alpha . \end{aligned} \]

Both sides are at least 0, and taking the \(m\)-th root keeps the order: \({1 - \alpha/m} \geq {(1 - \alpha)^{1/m}}\). Rearranging, \({1 - (1 - \alpha)^{1/m}} \geq {\alpha/m}\). For ten tests at 0.05 the two are 0.005116 and 0.005: the price of making no assumption about dependence is tiny.

Summary card

Share the budget \(\alpha\) among the tests; Holm re-spends what the rejected ones release.

The anchor figure again: the ten sorted p-values on a log scale under Bonferroni's flat line at 0.005 and Holm's staircase from 0.005 to 0.05; only the first, 0.001, is rejected.
S02.9.2-F1

One rejection each.

  • Bonferroni. Reject when \({\text{p}_i \leq \alpha/m}\); adjusted \({\min(1, m\,\text{p}_i)}\).
  • Holm. Compare \({\text{p}_{(k)}}\) with \({\alpha/(m - k + 1)}\), stop at the first failure.
  • Guarantee. \({\text{FWER} \leq \alpha}\) for both, by the union bound, for any dependence.
  • Šidák. \({1 - (1 - \alpha)^{1/m}}\), for independent tests only.

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • Siva Balakrishnan, Lecture 23, 36-705 Intermediate Statistics, Carnegie Mellon University, Fall 2019. Section 23.2.2 derives Šidák’s cut-off \({1 - (1 - \alpha)^{1/d}}\) under independence; Section 23.2.3 proves Bonferroni’s control of the FWER from the union bound; Section 23.3 states Holm’s procedure, proves its control by the first-true-null argument used here, and notes that it needs no independence and dominates Bonferroni.
  • Philip B. Stark, Ninth set of notes: Multiplicity, Statistics 210B, University of California, Berkeley. Section 1 defines the family-wise error rate and strong control; Section 2.1 states Holm’s sequentially rejective method, proves that it controls the FWER (Theorem 1) and describes it as a step-down procedure.

Next. S02.9.3 The false discovery rate: Benjamini-Hochberg asks for a weaker promise that buys far more discoveries when many hypotheses are false.

This lesson unlocks

S02.9.3 The false discovery rate: Benjamini-Hochberg

Your progress is saved in this browser only. No account, no tracking.
Back to top