\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

M05.5 · Hessians, curvature and second-order Taylor

A free, illustrated module of 4 lessons and one interactive in M05 Multivariable and Matrix Calculus: Second partial derivatives and the Hessian; The second-order Taylor model; Minimum, maximum or saddle: the second-derivative test; Curvature of a loss.

M05.5 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of M05 Multivariable and Matrix Calculus in the Mathematics track.

The gradient says which way a function slopes; the Hessian says how that slope bends. These four lessons collect the second partial derivatives into the Hessian matrix and show why it is symmetric, build the second-order Taylor model that approximates any smooth function near a point, use it to tell a minimum from a maximum or a saddle, and read the curvature of the losses that models are trained on: why least squares is a bowl, why one Newton step solves it, and why the largest eigenvalue of the Hessian limits the learning rate.

What you need first. Partial derivatives, the gradient and the Jacobian (M05.2 and M05.4), Taylor polynomials in one variable (M04.8.3), and symmetric and positive definite matrices with their eigenvalues (M03.3 and M03.4). The power method (M03.2.4) is used in the last lesson. The code uses NumPy and SciPy, explaining each call.

Lessons

1. Second partial derivatives and the Hessian M05.5.1 25 min

   Differentiate every partial derivative again and you get n squared second partial derivatives. Stack them into the Hessian, the Jacobian of the gradient, see why it is symmetric, meet the function where it is not, and compute it by hand, by finite differences and with SciPy.

2. The second-order Taylor model M05.5.2 25 min

   Value, gradient and Hessian together give the best quadratic approximation of a smooth function near a point. Derive it from the one-variable Taylor polynomial of a slice, see its error shrink like the cube of the step, and read the curvature in every direction off the Hessian’s eigenvalues.

3. Minimum, maximum or saddle: the second-derivative test M05.5.3 25 min

   Where the gradient is zero the Taylor model is pure curvature, so the signs of the Hessian’s eigenvalues decide: a minimum, a maximum or a saddle. Prove the test, apply it to the running loss and a cubic, and see the monkey saddle where it says nothing.

4. Curvature of a loss M05.5.4 25 min

   Derive the Hessians of the mean squared error and of the logistic log loss, see why both are never negative, take the Newton step that lands on the least-squares answer in one move, and find the largest eigenvalue from Hessian-vector products: the curvature that caps the learning rate.

The interactive

M05.5-X1 · Curvature viewer. Drag a point over the running bowl, the saddle, the monkey saddle or Rosenbrock’s valley. Inside a circle round the point the function’s contour lines and those of its second-order Taylor model are drawn together, with the Hessian’s eigen-directions and eigenvalues, and a wireframe you can turn shows the model as a bowl, a dome, a saddle or a trough. At a critical point the read-out applies the second-derivative test, or says that it is silent. It lives in The second-order Taylor model.