\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML01.5 · Gradient descent from scratch

A free, illustrated module of 4 lessons and one interactive in ML01 Machine Learning Foundations: Loss as a landscape; The gradient and the update rule; Choosing the learning rate; Batch, stochastic and mini-batch.

ML01.5 L1 · Core 4 lessons · about 110 min Live · in review

Part of ML01 Machine Learning Foundations in the Machine Learning track.

Almost every model in this course is trained the same way: measure how wrong it is, find which way is downhill, and take a small step. These four lessons build that loop by hand for the simplest model there is, a straight line through four points. You will see the loss as a landscape, derive the gradient that points uphill on it, work out exactly which step sizes converge and which blow up, and then trade exact steps for cheap noisy ones.

What you need first. Fitting a line to data and the mean squared error (ML01.4), and the derivative with the chain rule (M04.3). A little NumPy helps for the code.

Lessons

1. Loss as a landscape ML01.5.1 25 min

   A loss is a function of the model’s parameters, not of the data. See the mean squared error of a line as a parabola in one parameter and a bowl in two, and learn why training a model means walking downhill on that surface.

2. The gradient and the update rule ML01.5.2 25 min

   Derive the two slopes of the mean squared error with the chain rule, pack them into the gradient, and step against it. One update is worked by hand, then gradient descent is built in NumPy and checked against the exact least-squares answer.

3. Choosing the learning rate ML01.5.3 30 min

   On a quadratic loss, every gradient descent step multiplies the distance to the minimum by one fixed number. From that, find exactly which learning rates crawl, converge, oscillate or diverge, tie the limit to the data, and see why standardising the input helps.

4. Batch, stochastic and mini-batch ML01.5.4 30 min

   The full gradient is an average over the examples, so a step can be taken from one example or a handful. Derive why that cheap step is right on average, see the noise it adds, and build stochastic and mini-batch gradient descent with epochs and shuffling.

The interactive

ML01.5-X1 · Gradient descent explorer. Choose a starting point and a learning rate, then watch gradient descent walk the loss landscape of a line through four points. Push the learning rate past its limit to see the run diverge, and standardise the input to see the long valley become a round bowl. It lives in Choosing the learning rate.