\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

M05.9 · Backpropagation as reverse-mode differentiation

A free, illustrated module of 4 lessons and one interactive in M05 Multivariable and Matrix Calculus: Backpropagation is reverse mode on a network; Backpropagation in matrix form for a batch; Checking and debugging backpropagation; Cost, memory and gradient flow through depth.

M05.9 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of M05 Multivariable and Matrix Calculus in the Mathematics track.

Backpropagation is not a separate algorithm to memorise: it is the reverse mode of M05.8 run on the graph of a neural network. These four lessons run it by hand on a 2-2-1 network that learns XOR, write it in matrix form for a whole batch with vector-Jacobian products, check it against central differences and catch the classic bugs, and then count what it costs in arithmetic and memory and why the gradient can shrink or grow geometrically on its way through a deep network.

What you need first. Reverse mode with adjoints and a tape (M05.8.3), the cost of the two modes (M05.8.4), the gradients of a dense layer and of an elementwise activation (M05.7.3), products of layer Jacobians (M05.4.3), and gradient checking with central differences (M05.2.4).

Lessons

1. Backpropagation is reverse mode on a network M05.9.1 25 min

   Draw a 2-2-1 network for XOR as a computational graph, run the forward pass by hand, then the backward pass of reverse mode: the output adjoint p - y, through the weights, through tanh’, into every weight. Backpropagation is nothing more.

2. Backpropagation in matrix form for a batch M05.9.2 25 min

   Stack the examples as rows and the backward pass becomes three matrix products per layer: vector-Jacobian products that never form a Jacobian. Derive the batch rules, check every shape, then train the XOR network with a few hundred steps of gradient descent.

3. Checking and debugging backpropagation M05.9.3 25 min

   A backward pass with a bug still returns numbers that look plausible. Check every weight of the XOR network against central differences in float64, away from kinks, and watch the check catch five classic bugs, from a missing transpose to overwritten adjoints.

4. Cost, memory and gradient flow through depth M05.9.4 25 min

   Count what backpropagation costs: two matrix products backward for every one forward, so a gradient costs about three forward passes whatever the number of weights. Then the price in memory, checkpointing, and why the gradient shrinks or grows geometrically through twenty layers.

The interactive

M05.9-X1 · Backprop stepper. The 2-2-1 XOR network as a graph. Step through the forward pass and the backward pass for one example or the whole batch, watch the weight gradients fill a table beside their central- difference check, then train with an adjustable learning rate and watch the loss fall and the decision map over the unit square change. It lives in Backpropagation in matrix form for a batch.