\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

M05.2 · Partial derivatives and the gradient

A free, illustrated module of 4 lessons and one interactive in M05 Multivariable and Matrix Calculus: Partial derivatives and the gradient in n variables; Differentiability and the tangent plane; Gradients of losses with many weights; Numerical gradients and gradient checking.

M05.2 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of M05 Multivariable and Matrix Calculus in the Mathematics track.

Training changes every weight at once, guided by one column of slopes. These four lessons build that column for a function of any number of inputs, show when it really predicts the change of the function through the tangent plane and the definition of differentiability, derive it for the mean squared error and the log loss as X-transpose times an error vector, and test any hand-derived gradient against finite differences, catching the bugs.

What you need first. Partial derivatives and the gradient in two variables (M04.10), the rules of differentiation with the chain rule (M04.3), finite differences (M04.2.4) and Taylor polynomials (M04.8.3), and functions of several variables with limits (M05.1). Matrix products and the transpose (M02) are used from the third lesson on. The code uses NumPy and a little SciPy, explaining each call.

Lessons

1. Partial derivatives and the gradient in n variables M05.2.1 25 min

   Freeze every input but one and a function of many inputs becomes an ordinary curve with an ordinary slope. Collect those slopes, one per input, into a column: the gradient. Worked on a function of three inputs, the norm, a linear score and the logistic model.

2. Differentiability and the tangent plane M05.2.2 25 min

   The gradient was built from slopes along the axes, yet it predicts the change of a function for a small step in any direction. That prediction is the tangent plane, and a function is differentiable exactly when its error shrinks faster than the step.

3. Gradients of losses with many weights M05.2.3 25 min

   Differentiate the mean squared error of a linear model one weight at a time with the chain rule, then see the whole gradient appear as one matrix product, (2/n) X-transpose r. The mean log loss of logistic regression gives the same shape: (1/n) X-transpose (p - y).

4. Numerical gradients and gradient checking M05.2.4 25 min

   A hand-derived gradient can be wrong and still look plausible. Probe the function twice along every axis, compare the two gradients by their relative error, choose the step that balances truncation against rounding, and catch the bugs, at a cost of 2n evaluations.

The interactive

M05.2-X1 · Linear-model checker. Drag a point on the contour map of the running bowl, the running loss or xy/(x^2 + y^2), and drag a small step from it. The gradient’s prediction is set beside the true change, and as a slider shrinks the step the remainder over the step length falls to 0, except at the origin of the third function, where it grows. It lives in Differentiability and the tangent plane.