\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML01.2 · Data splits and the generalization gap

A free, illustrated module of 4 lessons and one interactive in ML01 Machine Learning Foundations: Why testing on the training data lies; Train, validation and test sets; The generalization gap; How splits go wrong.

ML01.2 L1 · Core 4 lessons · about 100 min Live · in review

Part of ML01 Machine Learning Foundations in the Machine Learning track.

A model that stores its training examples scores perfectly on them and fails on anything new, so a score on the data a model was built from proves nothing. These four lessons build the routine that every honest result rests on: hold examples back, fit on one set, choose on a second and report on a third. They measure how much better a model looks on its own data, show that a test score is itself a measurement with a give-or-take, and work through the four leaks that make a test score too good to be true.

What you need first. What a model, its parameters and a loss are, and fitting as a search for the lowest loss (ML01.1). The mean and the standard deviation of a list of numbers. A little NumPy helps for the code.

Lessons

1. Why testing on the training data lies ML01.2.1 25 min

   A model that simply stores its training examples scores perfectly on them and fails on anything new. See memorising and generalising side by side, a lookup table against a straight line, and learn why a score on seen data proves nothing.

2. Train, validation and test sets ML01.2.2 25 min

   One dataset has three jobs: fit a model, choose between models, and report how good the winner is. Learn why each job needs its own examples, how to shuffle and cut a dataset with a seed, and how scikit-learn’s train_test_split does the same.

3. The generalization gap ML01.2.3 25 min

   Training error flatters a model, and a test error is only an estimate. Measure both on real data, watch the gap between them open as a model gets more flexible, and learn why a test score moves when the split changes and how more test examples steady it.

4. How splits go wrong ML01.2.4 25 min

   A test set only works if nothing about it reaches the model. Meet the four common leaks: scaling before splitting, copies of a row on both sides, shuffling data that is ordered in time, and peeking at the test set. Each is shown with numbers, wrong beside right.

The interactive

ML01.2-X1 · Split explorer. Thirty points are split into a training set and a test set, and a table with an adjustable number of bins is fitted to the training points. Slide the number of bins, reshuffle the split, and watch the training error, the test error and the gap between them. It lives in The generalization gap.