\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

S02.8 · Bootstrap and permutation methods

A free, illustrated module of 4 lessons and one interactive in S02 Statistical Inference: The bootstrap idea; Bootstrap confidence intervals; Permutation tests; Resampling for machine learning.

S02.8 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of S02 Statistical Inference in the Statistics and Data Analysis track.

When no formula gives a statistic’s standard error or null distribution, let a computer make one. The bootstrap resamples the sample with replacement to show how much an estimate wobbles, and turns that into confidence intervals; the permutation test shuffles labels to show what chance alone would produce. These four lessons work both by hand on the nine rents and the two page designs, check them by simulation, find where they fail, and put them to work on a model’s test-set scores.

What you need first. Sampling distributions and the standard error (S02.1), bias and variance of an estimator (S02.2), confidence intervals (S02.4), p-values (S02.5) and the common tests (S02.7), with counting and expectation from M06. The last lesson uses the AUC of ML01.7.

Lessons

1. The bootstrap idea S02.8.1 25 min

   When no formula gives a statistic’s standard error, let the sample stand in for the population and resample it with replacement. Worked on the nine rents: the bootstrap standard error of the median and of the mean, and why a resample holds about 63 percent of the rows.

2. Bootstrap confidence intervals S02.8.2 25 min

   Turn a bootstrap distribution into a confidence interval: the percentile interval, the basic interval derived from a pivot, and BCa, scipy’s default. Worked on the nine rents and on the diabetes correlation, with coverage checked on the city and the cases where the bootstrap fails.

3. Permutation tests S02.8.3 25 min

   If the labels do not matter, every shuffle of them was equally likely, so rank the observed statistic among the shuffled ones. The two page designs by complete enumeration (38 of 252 splits, p-value 0.151), Monte Carlo shuffles with the (b + 1) / (K + 1) rule derived, and a correlation on the diabetes table.

4. Resampling for machine learning S02.8.4 25 min

   Uncertainty for a model’s test-set score without a formula: a bootstrap interval for AUC, the paired bootstrap that compares two models on the same resampled test sets, and permutation_test_score to ask whether a model has learned anything at all.

The interactive

S02.8-X1 · Resampling machine. The nine rents as nine balls: draw nine with replacement and watch which light up, how often, and where the resample’s median or mean lands on a growing histogram with its percentile interval. A second mode shuffles the A and B labels of the two page designs and builds the null distribution of the difference in means, with the observed difference marked. It lives in The bootstrap idea.