\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

S02.6 · Power, effect size and sample size

A free, illustrated module of 4 lessons and one interactive in S02 Statistical Inference: Power; Effect size; Choosing the sample size; Low power and simulated power.

S02.6 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of S02 Statistical Inference in the Statistics and Data Analysis track.

A test can only find an effect that its sample is large enough to see. These four lessons derive the power of a test from the overlap diagram, separate the size of an effect from the p-value that mixes it with the sample size, turn the power formula round to choose n before any data are collected, and simulate what low power does to the results that do reach significance: they exaggerate the effect and can even get its sign wrong, which is the winner’s curse of benchmark leaderboards.

What you need first. The logic of a test, the p-value and the two kinds of error (S02.5), the standard error and the t distribution (S02.1), intervals for a difference of two means (S02.4), and the normal approximation from the central limit theorem (M06.11).

Lessons

1. Power S02.6.1 25 min

   Power is the chance that a test notices an effect that is really there. Derive it for the one-sided z-test from the overlap diagram, read off what raises it, draw the power curve, and check the formula by simulation on the city’s rents.

2. Effect size S02.6.2 25 min

   A p-value mixes the size of an effect with the size of the sample. Report the effect in its own right: the raw difference with its interval, the standardised effect, the difference, relative risk and odds ratio for proportions, and the AUC as an effect size for ranks.

3. Choosing the sample size S02.6.3 25 min

   Turn the power formula round to find the sample size before collecting any data: one mean, two means and two proportions, with the exact t-test check by the noncentral t, and why detecting a one-point accuracy gain needs about 13,500 test examples per model.

4. Low power and simulated power S02.6.4 25 min

   Find the power of any test on any population by simulation, then see what low power does to the results that do reach significance: they exaggerate the effect and can get its sign wrong, the winner’s curse of leaderboards. And why power computed after the fact tells nothing new.

The interactive

S02.6-X1 · Power explorer. The overlap diagram made live. Set the standardised effect, the sample size per group, the level and one- or two-sided, and watch the alpha, beta and power areas change, with a power curve against n that marks the sample size needed for 80 percent power. It lives in Power.