\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

DL01.5 · Backpropagation by hand

A free, illustrated module of 4 lessons and one interactive in DL01 Neural Network Foundations: One example, every number; Layers as modules with forward and backward; Branches, sums and shared weights; Training the hand-built network.

DL01.5 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of DL01 Neural Network Foundations in the Deep Learning track.

Backpropagation is the chain rule run backwards through a network, one small operation at a time. These four lessons do it by hand on the course’s running 2-2-2 network with ReLU hidden units and a softmax output, until every one of its twelve gradients is a number you computed yourself; then turn each layer into a module with a forward and a backward method, extend the rules to graphs that branch, add and share weights, and finally train a 2-16-16-2 network on the two moons with nothing but that hand-built library.

What you need first. Backpropagation on the 2-2-1 XOR network and gradient checking (M05.9), the gradients of a dense layer and of softmax with cross-entropy (M05.7.3), the multivariable chain rule summed over paths (M05.4.2), the forward pass of a multilayer perceptron (DL01.2.2), softmax and cross-entropy (DL01.4), and mini-batch gradient descent (ML01.5.4).

Lessons

1. One example, every number DL01.5.1 25 min

   Fix the course’s running 2-2-2 network with ReLU hidden units and a softmax output, run its forward pass by hand to the loss, then send the gradient back through the graph, node by node, until all twelve parameters have theirs.

2. Layers as modules with forward and backward DL01.5.2 25 min

   Turn each layer into an object with a forward method that remembers what it needs and a backward method that returns the gradient of its input: Linear, ReLU, Tanh and a fused softmax with cross-entropy, chained by a Sequential container.

3. Branches, sums and shared weights DL01.5.3 25 min

   When a value feeds two places, the gradients coming back add. Work it on a network with two output heads, a skip connection and a weight used twice, then see concatenation split a gradient and a sum broadcast it back, and catch the overwrite bug.

4. Training the hand-built network DL01.5.4 25 min

   Put the hand-built module library to work: forward, loss, backward and a gradient step for every parameter, in mini-batches over the two moons, until a 2-16-16-2 network bends its decision boundary around the data, with a validation loss beside the training loss.

The interactive

DL01.5-X1 · Backprop stepper. The running 2-2-2 network as a computational graph. Step through the forward pass to fill the nodes with values, then the backward pass to fill them with gradients, with the local rule of each step in the read-out. Change the input, the label or any weight and every number follows; switch a hidden unit off and watch its gradient stop at the ReLU. It lives in One example, every number.