\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

M05.3 · Directional derivatives and steepest ascent

A free, illustrated module of 4 lessons and one interactive in M05 Multivariable and Matrix Calculus: The directional derivative; Steepest ascent and the gradient’s geometry; Steepest depends on how you measure a step; Normals, tangent planes and the distance to a boundary.

M05.3 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of M05 Multivariable and Matrix Calculus in the Mathematics track.

The gradient knows the slope of a function in every direction at once. These four lessons derive the directional derivative as one dot product, prove with the Cauchy-Schwarz inequality that the gradient is the direction of steepest ascent and meets every level set at a right angle, show that “steepest” changes when you change the ruler for a step (the sign of the gradient in the max-norm, one coordinate in the L1 norm, a preconditioned gradient in a quadratic norm), and use the gradient as a normal to find tangent planes, distances to a decision boundary and the fast gradient sign attack on a classifier.

What you need first. Partial derivatives, the gradient and differentiability with the tangent plane (M05.2), slices of a function along a line (M05.1.3), the dot product, norms and the Cauchy-Schwarz inequality (M02.3), and positive definite matrices with their Cholesky factor (M03.4) for the third lesson. The code uses NumPy and scipy.optimize.minimize, explaining each call.

Lessons

1. The directional derivative M05.3.1 25 min

   Walk from a point in any direction and the heights you pass make an ordinary curve. Its slope is the directional derivative. For a differentiable function it is one dot product, the gradient times the direction, in any number of dimensions.

2. Steepest ascent and the gradient’s geometry M05.3.2 25 min

   Of all directions, the gradient climbs fastest and minus the gradient falls fastest, in any number of dimensions: a two-line proof with the Cauchy-Schwarz inequality. The same arrow crosses every level set at a right angle and grows where contours crowd.

3. Steepest depends on how you measure a step M05.3.3 25 min

   Steepest descent compares steps of length 1, so the answer depends on the ruler. In L2 it is minus the gradient, in the max-norm minus its sign, in L1 a single coordinate, and in a quadratic norm the preconditioned step minus A inverse times the gradient.

4. Normals, tangent planes and the distance to a boundary M05.3.4 25 min

   The gradient sticks straight out of a level surface, so it gives the tangent plane at once. Dividing by its length gives the distance to a level set to first order, exactly for a linear classifier’s boundary, and the smallest input change that moves a score, the idea behind the fast gradient sign attack.

The interactive

M05.3-X1 · Direction dial. Drag a point on the contour map of the running bowl, the running loss or a saddle and turn a unit direction around it. A side panel plots the slope against the angle, a cosine with its peak along the gradient and zeros along the contour line. A norm switch draws the unit ball of the L2, L1 or max-norm at the point and the steepest direction that ruler picks. It lives in Steepest depends on how you measure a step.