\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

DL01.6 · Build an autodiff engine from scratch

A free, illustrated module of 4 lessons and one interactive in DL01 Neural Network Foundations: From scalars to tensors; Matrix products, reductions and broadcasting; Parameters, modules and an optimizer; Train with your engine and check it.

DL01.6 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of DL01 Neural Network Foundations in the Deep Learning track.

PyTorch’s autograd is not magic, and these four lessons prove it by building a small version of it. The scalar Var class of M05.8 becomes a Tensor class around NumPy arrays, with a backward rule for every elementwise operation, for matrix products, sums and broadcasting, and a fused cross-entropy. A PyTorch-shaped library of parameters, modules and an optimizer goes on top, and the engine then has to agree with the hand-derived gradients of DL01.5 before it trains a network on the moons.

What you need first. Reverse-mode automatic differentiation on a scalar graph (M05.8), backpropagation on a small network (M05.9) and the backward pass of a network by hand, with its layers as modules (DL01.5). Comfortable NumPy: arrays, shapes and the @ operator.

Lessons

1. From scalars to tensors DL01.6.1 25 min

   Rebuild the scalar Var class of M05.8 as a Tensor class around a NumPy array: data, grad, parents and a backward closure recorded as each operation runs, with a vector-Jacobian product for every elementwise operation.

2. Matrix products, reductions and broadcasting DL01.6.2 25 min

   The backward rules that make a network possible: matrix products read off from shapes, sums that spread the gradient back, reshapes and transposes undone, broadcasting summed away by an unbroadcast helper, and a fused cross-entropy.

3. Parameters, modules and an optimizer DL01.6.3 25 min

   A deliberately PyTorch-shaped library on top of the engine: Parameter, Module with a recursive parameters(), Linear, ReLU, Sequential and an SGD optimizer, with why gradients must be zeroed and why updates happen outside the graph.

4. Train with your engine and check it DL01.6.4 25 min

   Hold the engine to the hand: it must reproduce every gradient of the running 2-2-2 network of DL01.5 and of its module library, then train the 2-16-16-2 network on the moons step for step with the hand-built loop, while the memory its graph holds is measured and released.

The interactive

DL01.6-X1 · Broadcast and backward. Choose the shapes of two arrays and an operation, see the result as a grid with its broadcast copies dashed, then send back a gradient of ones and watch it summed back to each array’s shape. It lives in Matrix products, reductions and broadcasting.