\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML01.9 · Regularization and validation

A free, illustrated module of 4 lessons and one interactive in ML01 Machine Learning Foundations: Ridge: a price on large weights; Lasso: weights that reach zero; Cross-validation; Tuning without cheating.

ML01.9 L1 · Core 4 lessons · about 120 min Live · in review

Part of ML01 Machine Learning Foundations in the Machine Learning track.

A flexible model overfits because nothing stops its weights from growing to fit the noise. These four lessons put a price on large weights and learn to set that price honestly. Ridge shrinks the weights and lasso drives some to exactly zero, each worked by hand for one weight before nine are fitted in NumPy. Cross-validation then lets every example help both to fit and to judge, and a last lesson shows how to tune without cheating: choose on validation data, report on a test set that is opened once, and treat the number of training steps as one more dial.

What you need first. Overfitting and the diagnosis of high variance (ML01.8), training, validation and test sets (ML01.2), the best line as covariance over variance (ML01.4.2), and gradient descent with its learning rate (ML01.5). Standardising a feature (ML01.3.3) is used throughout.

Lessons

1. Ridge: a price on large weights ML01.9.1 30 min

   Add the sum of squared weights to the loss and a flexible model calms down. Derive the one-feature ridge solution and its shrinkage factor, see the penalty turn into weight decay inside gradient descent, and learn why features are standardised first and the bias is left alone.

2. Lasso: weights that reach zero ML01.9.2 30 min

   Swap the squared weight in the penalty for its absolute value and weights do not just shrink: some become exactly zero. Derive the one-feature lasso solution, soft thresholding, case by case, and see in the picture of a diamond and a circle why ridge does not do the same.

3. Cross-validation ML01.9.3 30 min

   One validation set is a small jury, and the examples in it never help to fit. k-fold cross-validation lets every example do both jobs in turn. Work it fold by fold by hand, build it from scratch in NumPy, match scikit-learn, and count what it costs.

4. Tuning without cheating ML01.9.4 30 min

   Choose a penalty strength on validation data and report on a test set that is opened once. See by formula and by simulation how choosing on the test set makes the reported error too good, and treat the number of training steps as one more dial: early stopping.

The interactive

ML01.9-X1 · Penalty explorer. Slide the strength of an L2 or an L1 penalty on a degree-9 polynomial fitted to twelve points. The curve smooths out, the nine weights shrink or snap to exactly zero, and the validation error traces a U whose lowest point you can find by hand. It lives in Lasso: weights that reach zero.