\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML02.4 · Gradient boosting

A free, illustrated module of 4 lessons and one interactive in ML02 Supervised Learning: Trees, Ensembles and Kernels: Fitting the residuals; Gradient descent in function space; Gradient boosting for classification; Shrinkage, subsampling and early stopping.

ML02.4 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of ML02 Supervised Learning: Trees, Ensembles and Kernels in the Machine Learning track.

Gradient boosting builds a strong model out of many small trees, each one trained to fix what the trees before it still get wrong. These four lessons start with the simplest version, fitting a regression stump to the residuals of eight points by hand, show that a residual is a negative gradient so that any differentiable loss can be boosted, carry the idea to classification on the log-odds scale with a Newton step per leaf, and end with the three settings that decide whether a boosted model generalises: the learning rate, the tree depth and the number of stages.

What you need first. Regression trees grown by squared-error splits (ML02.1), gradient descent (ML01.5), the log loss of logistic regression (ML01.6) and validation by held-out folds (ML01.9). AdaBoost (ML02.3) helps as a contrast but is not required.

Lessons

1. Fitting the residuals ML02.4.1 25 min

   Gradient boosting starts from the mean, fits a small regression tree to what is left over, adds it, and repeats. Three stages worked by hand on eight points take the training error from 7.4375 to 0.6071, and the model turns out to be a sum of step functions.

2. Gradient descent in function space ML02.4.2 25 min

   A residual is the negative gradient of the squared loss with respect to a prediction. That one fact turns boosting into gradient descent on the predictions, and lets the same loop boost the absolute and Huber losses, which shrug off an outlier that captures squared-loss boosting.

3. Gradient boosting for classification ML02.4.3 25 min

   Boost the log-odds, not the probability. Start from the base-rate log-odds, fit each tree to the pseudo-residuals y minus p of the log loss, and set each leaf by one Newton step, the sum of residuals over the sum of p(1 - p). Two stages by hand on the twelve points.

4. Shrinkage, subsampling and early stopping ML02.4.4 25 min

   Boosting fits its training data ever more closely, so it can overfit. Three settings control it: a learning rate that takes small steps, a tree depth that limits interactions, and a number of stages chosen by early stopping. Measured on load_diabetes with the folds of the ML01.10 project.

The interactive

ML02.4-X1 · Residual stepper. Press step to fit a stump or a depth-2 tree to the current residuals and add it. Watch the residual bars shrink, the staircase prediction build up, and the training and validation errors move as the stages pile up, with a slider for the learning rate. It lives in Fitting the residuals.