Skip to main content

Autograd

PyTorch records a graph as each operation runs, exactly as the engine of DL01.6 did, and backward() fills the .grad of every leaf. Reproduce the hand gradients of the running 2-2-2 network, watch gradients accumulate, cut the graph with detach and no_grad, and check a fused loss with gradcheck.
DL01.7.2L2 Intermediate25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: DL01.7.1 Tensors

What you will be able to do
  • Explain what requires_grad, grad_fn and next_functions record, and which tensors are leaves.
  • Run backward() on a scalar loss and read the gradients from the .grad of the leaf tensors, matching them to the hand gradients of the 2-2-2 network.
  • Predict how .grad changes when backward runs twice, and use torch.autograd.grad to get a gradient without touching .grad.
  • Stop recording with detach() and torch.no_grad(), and say when each is needed.
  • Check a hand-written backward with torch.autograd.gradcheck in float64.

The question

In DL01.5 you worked every gradient of a small network by hand, and in DL01.6 you built an engine that found them by running a recorded graph backwards. PyTorch claims to do the same in one call, loss.backward(). Does it give exactly your numbers, where does it put them, and what does it remember between calls?

From the last lesson

1. A contiguous tensor has shape \((2, 3, 4)\). What are its strides?

Show answer

\((12, 4, 1)\): the last axis has stride 1, and each earlier stride is the next one times the next length.

2. Which of torch.from_numpy(a) and torch.tensor(a) shares memory with the array a?

Show answer

torch.from_numpy(a) shares it; torch.tensor(a) copies.

Intuition

Every tensor made by an operation remembers which operation made it. That memory is its grad_fn: a node of a graph that knows the backward rule of the operation and points, through next_functions, at the nodes of its inputs. The tensors you create yourself and mark with requires_grad=True are the leaves of the graph. They have no grad_fn; instead a node called AccumulateGrad waits at each of them to receive its gradient.

A computational graph of the running 2-2-2 network in two rows. Top row: a blue box x, marked no grad, then vermillion operation boxes W1 @ x, plus b1 and relu joined by thick vermillion forward arrows, with their grad_fn names underneath: MvBackward0, AddBackward0, ReluBackward0. A vermillion line leads down to the second row: W2 @ a1, plus b2 and cross_entropy, with grad_fn names MvBackward0, AddBackward0, and LogSoftmaxBackward0 with NllLossBackward0, ending in a reddish-purple box L. The orange leaves W1 and b1 sit above their operations and W2 and b2 below theirs, each with a thin orange arrow into its operation and a green dashed arrow from the operation into a green box beside it, W1.grad, b1.grad, W2.grad and b2.grad. A green dashed arrow labelled L.backward() runs from L back into cross_entropy, and a note says the backward walk follows next_functions and ends in an AccumulateGrad at each leaf.
DL01.7.2-F1

What PyTorch records for the running network. The forward pass (vermillion) builds the graph as it runs; L.backward() walks it the other way (green) and deposits a gradient in the .grad of each leaf that asked for one.

This is the engine of DL01.6 again. There, each operation stored its parents and a backward closure, and backward() sorted the graph and called each closure once, adding into grad. In PyTorch the closure is the grad_fn node, the parents are next_functions, and the adding happens in AccumulateGrad. The graph is built afresh by each forward pass, so whatever Python runs, with any loop or if, is what gets differentiated.

Build it up

Step 1. Ask for gradients. W = torch.tensor(..., requires_grad=True) makes a leaf that will receive a gradient. An operation is recorded only if at least one of its inputs requires a gradient; the input x of a network usually does not.

Step 2. Run forward. Each result gets a grad_fn, such as MvBackward0 for a matrix times a vector or ReluBackward0. Nothing is differentiated yet. The nodes keep the tensors their backward rule will need, such as the input of the ReLU for its mask.

Step 3. Run backward. L.backward() seeds the loss with 1 and walks the graph from the root to the leaves, applying each node’s rule. Each leaf’s gradient is added into its .grad, a tensor of the leaf’s own shape. Intermediate results keep no .grad, unless you call retain_grad() on them first. The saved tensors are then freed, so a second backward() through the same graph fails unless the first was called with retain_graph=True.

Three step frames of the same small graph, a leaf W, an input x, a product z and a loss L. Frame 1, after creating W with requires_grad: W.grad is None, and there is no graph. Frame 2, after the forward pass: vermillion arrows from x and W to z and on to L, z.grad_fn is MvBackward0, L.grad_fn is a loss node, and W.grad is still None. Frame 3, after L.backward(): green arrows run back from L to z to W, W.grad holds a tensor with the shape of W, z.grad is None because z is not a leaf, and the saved tensors of the graph are marked freed.
DL01.7.2-F2

Three moments in the life of a gradient. Creating a leaf records nothing; the forward pass builds the graph; backward() fills the leaf’s .grad and frees what the graph had saved.

Step 4. Gradients accumulate. backward() adds into .grad; it never overwrites it. That is what a weight used twice needs, as DL01.5.3 showed: the gradients from both uses must add. It also means that between training steps someone must clear .grad, which is the optimizer’s zero_grad() in DL01.7.4.

Step 5. Leave the graph. t.detach() returns a tensor with the same numbers, sharing memory, that is cut off from the graph. Inside with torch.no_grad(): nothing is recorded at all, which is how weights are updated and how a network is evaluated without paying for a graph.

The formal statement

For a scalar \(L\) computed from leaves \(\theta_1, \theta_2, \dots\), one call L.backward() performs, for every leaf with requires_grad=True,

\[ \theta.\mathrm{grad} \;\leftarrow\; \theta.\mathrm{grad} + \pd{L}{\theta}, \]

where a missing .grad counts as zero and \(\pd{L}{\theta}\) has the shape of \(\theta\). Each node computes a vector-Jacobian product: given the gradient \(\vect{g}\) of \(L\) with respect to its output \(\vect{u} = f(\vect{v})\), it returns \(\vect{g}\T \mat{J}\), where \(\mat{J}\) is the Jacobian of \(f\) (M05.4), the gradient with respect to its input \(\vect{v}\). Like the engine of DL01.6.1, it never forms \(\mat{J}\). For a tensor output \(\vect{u}\) that is not a scalar, u.backward(g) starts the same walk with the given \(\vect{g}\).

torch.autograd.grad(L, [theta]) runs the same walk and returns the gradients as a tuple instead of adding them to .grad.

Three framed panels with the same small graph, an orange W and a blue x feeding a vermillion z. Left, titled recorded: a green dashed arrow runs back from z to W; z has requires_grad True and grad_fn MvBackward0, and gradients can flow back. Middle, titled u = z.detach(): a box u hangs below z, joined by a dotted line marked same memory; u has requires_grad False and no history. Right, titled with torch.no_grad(): there is no backward arrow; z has requires_grad False and grad_fn None, the mode for updates and evaluation.
DL01.7.2-F3

Two ways out of the graph. detach() cuts one tensor loose and keeps its memory; torch.no_grad() stops recording for a whole block.

The running network, by hand. The 2-2-2 network of DL01.5.1 has one input \(\vect{x}\) in \(\R^2\), ReLU hidden units, a softmax output and the cross-entropy loss with true class \(k\):

\[ \begin{aligned} \vect{z}^{(1)} &= \mat{W}^{(1)} \vect{x} + \vect{b}^{(1)}, \\ \vect{a}^{(1)} &= \operatorname{relu}(\vect{z}^{(1)}), \\ \vect{z} &= \mat{W}^{(2)} \vect{a}^{(1)} + \vect{b}^{(2)}, \\ L &= -z_k + \ln \textstyle\sum_j e^{z_j}. \end{aligned} \]

Its gradients were derived in DL01.5.1 by the chain rule, one node at a time from the loss back:

\[ \begin{aligned} \vect{\delta}^{(2)} &= \vect{p} - \vect{y}, \\ \pd{L}{\mat{W}^{(2)}} &= \vect{\delta}^{(2)} \vect{a}^{(1)\top}, \\ \pd{L}{\vect{b}^{(2)}} &= \vect{\delta}^{(2)}, \\ \vect{\delta}^{(1)} &= \big(\mat{W}^{(2)\top} \vect{\delta}^{(2)}\big) \odot \vect{m}, \\ \pd{L}{\mat{W}^{(1)}} &= \vect{\delta}^{(1)} \vect{x}\T, \\ \pd{L}{\vect{b}^{(1)}} &= \vect{\delta}^{(1)}. \end{aligned} \]

Here \(\vect{p}\) is the softmax of \(\vect{z}\), \(\vect{y}\) the one-hot target, \(\vect{m}\) the ReLU mask (\(m_i = 1\) where \(z^{(1)}_i > 0\) and 0 elsewhere) and \(\odot\) the elementwise product. At \(z^{(1)}_i = 0\) exactly, PyTorch’s rule for a function that is not differentiable there picks the smallest subgradient, so the ReLU passes back 0.

Worked example

These are the numbers fixed for the running network in DL01.5.1. The input is \(\vect{x} = (1, 2)\), the true class is the second one (\(k = 2\) in maths, label 1 in code), and

\[ \begin{aligned} \mat{W}^{(1)} &= \begin{bmatrix} 1 & -0.5 \\ -0.5 & 1 \end{bmatrix}, \\ \vect{b}^{(1)} &= (0.5,\; -0.5), \\ \mat{W}^{(2)} &= \begin{bmatrix} 1 & 1 \\ -1 & 0.5 \end{bmatrix}, \\ \vect{b}^{(2)} &= (-0.5,\; 0). \end{aligned} \]

Forward. \(z^{(1)}_1 = 1 - 1 + 0.5 = 0.5\) and \(z^{(1)}_2 = -0.5 + 2 - 0.5 = 1\), so \(\vect{a}^{(1)} = (0.5, 1)\): both hidden units are on. Then \(z_1 = 0.5 + 1 - 0.5 = 1\) and \(z_2 = -0.5 + 0.5 + 0 = 0\). Dividing top and bottom by \(e^1\),

\[ \begin{aligned} p_1 &= \frac{e^1}{e^1 + e^0} = \frac{1}{1 + e^{-1}} \approx 0.7311, \\ p_2 &= 1 - p_1 \approx 0.2689, \\ L &= -z_2 + \ln(e^1 + e^0) \\ &= -0 + \ln(e + 1) \approx 1.3133. \end{aligned} \]

Backward. \(\vect{\delta}^{(2)} = {(0.7311, -0.7311)}\). The output weights get \(\vect{\delta}^{(2)} \vect{a}^{(1)\top}\), each row \(\delta^{(2)}_i\) times \({(0.5, 1)}\):

\[ \pd{L}{\mat{W}^{(2)}} = \begin{bmatrix} 0.3655 & 0.7311 \\ -0.3655 & -0.7311 \end{bmatrix}. \]

Back through \(\mat{W}^{(2)\top}\), one entry per hidden unit:

\[ \begin{aligned} &1 \cdot 0.7311 - 1 \cdot (-0.7311) \\ &\quad = 1.4621, \\ &1 \cdot 0.7311 + 0.5 \cdot (-0.7311) \\ &\quad = 0.3655. \end{aligned} \]

The ReLU mask is \((1, 1)\), so \(\vect{\delta}^{(1)} = (1.4621, 0.3655)\), and

\[ \begin{aligned} \pd{L}{\mat{W}^{(1)}} &= \vect{\delta}^{(1)} \vect{x}\T \\ &= \begin{bmatrix} 1.4621 & 2.9242 \\ 0.3655 & 0.7311 \end{bmatrix}. \end{aligned} \]

The running 2-2-2 network as a chain of boxes with every number written in. Forward values in vermillion along the top: z1 equals (0.5, 1), a1 equals (0.5, 1) with both units on, z equals (1, 0), p equals (0.731, 0.269), L equals 1.313. Gradients in green along the bottom, from right to left: p minus y equals (0.731, minus 0.731), W2 transpose times delta equals (1.462, 0.366), the mask (1, 1), and delta1 equals (1.462, 0.366). Underneath, the four .grad boxes: W1.grad with rows (1.462, 2.924) and (0.366, 0.731), b1.grad (1.462, 0.366), W2.grad with rows (0.366, 0.731) and (minus 0.366, minus 0.731), b2.grad (0.731, minus 0.731). A note says both units are on, so the mask passes the whole gradient back to W1 and b1.
DL01.7.2-F4

Every number of the running network, forward on top and backward below, the same as in DL01.5.1. Both hidden units are on, so the ReLU mask lets the whole gradient through to the first layer.

These are the twelve gradients of DL01.5.1, here to four decimals. Now PyTorch must find the same numbers.

Code

The network in float64, so that the comparison is exact to rounding. Only the weights and biases ask for gradients.

import numpy as np
import torch
import torch.nn.functional as F

torch.manual_seed(0)
torch.set_num_threads(1)
device, dtype = torch.device("cpu"), torch.float64

W1_np = np.array([[1.0, -0.5], [-0.5, 1.0]]); b1_np = np.array([0.5, -0.5])   # fixed in DL01.5.1
W2_np = np.array([[1.0, 1.0], [-1.0, 0.5]]);  b2_np = np.array([-0.5, 0.0])
x_np, label = np.array([1.0, 2.0]), 1

leaf = lambda a: torch.tensor(a, dtype=dtype, device=device, requires_grad=True)
W1, b1, W2, b2 = leaf(W1_np), leaf(b1_np), leaf(W2_np), leaf(b2_np)
x = torch.tensor(x_np, dtype=dtype, device=device)          # an input: no gradient wanted

z1 = W1 @ x + b1
a1 = torch.relu(z1)
z = W2 @ a1 + b2
L = F.cross_entropy(z, torch.tensor(label, device=device))
print("z1", z1.tolist(), " z", z.tolist(), f" L {L.item():.4f}")
print("leaves:", W1.is_leaf, x.is_leaf, z.is_leaf, " grad_fn of z:", type(z.grad_fn).__name__)
z1 [0.5, 1.0]  z [1.0, 0.0]  L 1.3133
leaves: True True False  grad_fn of z: AddBackward0

The graph, walked from the loss along next_functions, as in the anchor figure.

def walk(node, depth=0):
    if node is None:                     # an input that needs no gradient, such as x
        return
    shape = tuple(node.variable.shape) if hasattr(node, "variable") else ""
    print("  " * depth + type(node).__name__, shape)
    for child, _ in node.next_functions:
        walk(child, depth + 1)

walk(L.grad_fn)
NllLossBackward0 
  LogSoftmaxBackward0 
    AddBackward0 
      MvBackward0 
        AccumulateGrad (2, 2)
        ReluBackward0 
          AddBackward0 
            MvBackward0 
              AccumulateGrad (2, 2)
            AccumulateGrad (2,)
      AccumulateGrad (2,)

Now backward, and the hand gradients of the worked example written as NumPy lines beside it.

L.backward()

p = np.exp(z.detach().numpy()); p /= p.sum()
y = np.eye(2)[label]
d2 = p - y
hand = {"W2": np.outer(d2, a1.detach().numpy()), "b2": d2}
d1 = (W2_np.T @ d2) * (z1.detach().numpy() > 0)
hand.update({"W1": np.outer(d1, x_np), "b1": d1})

for name, tensor in [("W1", W1), ("b1", b1), ("W2", W2), ("b2", b2)]:
    torch.testing.assert_close(tensor.grad, torch.from_numpy(hand[name]), rtol=0, atol=1e-12)
    print(name, np.round(tensor.grad.numpy(), 4).tolist())
W1 [[1.4621, 2.9242], [0.3655, 0.7311]]
b1 [1.4621, 0.3655]
W2 [[0.3655, 0.7311], [-0.3655, -0.7311]]
b2 [0.7311, -0.7311]

Every gradient matches the hand to \(10^{-12}\). DL01.6.4 checks that your engine reproduces the same hand gradients, so PyTorch and the engine agree with each other too.

Accumulation, a freed graph, and torch.autograd.grad, which leaves .grad alone.

def loss_of(W1, b1, W2, b2):
    z = W2 @ torch.relu(W1 @ x + b1) + b2
    return F.cross_entropy(z, torch.tensor(label, device=device))

first = b2.grad.clone()
loss_of(W1, b1, W2, b2).backward()                 # a second backward, no zeroing
print("b2.grad after two backward calls:", np.round(b2.grad.numpy(), 4).tolist())

L2 = loss_of(W1, b1, W2, b2)
L2.backward()
try:
    L2.backward()                                  # the saved tensors are gone
except RuntimeError as error:
    print("second backward on one graph:", str(error)[:40])

(g_W2,) = torch.autograd.grad(loss_of(W1, b1, W2, b2), [W2])
print("autograd.grad:", np.round(g_W2.numpy(), 4).tolist(), " b2.grad unchanged:", torch.equal(b2.grad, 3 * first))
b2.grad after two backward calls: [1.4621, -1.4621]
second backward on one graph: Trying to backward through the graph a s
autograd.grad: [[0.3655, 0.7311], [-0.3655, -0.7311]]  b2.grad unchanged: True

detach() and torch.no_grad(): the same numbers, no graph.

with torch.no_grad():
    z_eval = W2 @ torch.relu(W1 @ x + b1) + b2
u = z.detach()
print(z_eval.requires_grad, z_eval.grad_fn, "|", u.requires_grad,
      u.untyped_storage().data_ptr() == z.untyped_storage().data_ptr())
try:
    z.numpy()
except RuntimeError as error:
    print("z.numpy():", str(error)[:42])
False None | False True
z.numpy(): Can't call numpy() on Tensor that requires

Last, a check for code you write yourself. DL01.6.2 fused the softmax and the cross-entropy into one operation with the backward rule \({(\mat{P} - \mat{Y})/n}\) for a batch of \(n\). Written as a PyTorch Function, its backward is ours, not autograd’s, so gradcheck compares it with central differences. It needs float64: in float32 the differences themselves are too noisy.

class SoftmaxCrossEntropy(torch.autograd.Function):
    @staticmethod
    def forward(ctx, Z, labels):
        logp = Z - torch.logsumexp(Z, dim=1, keepdim=True)     # stable log-softmax
        ctx.save_for_backward(logp.exp(), labels)
        return -logp[torch.arange(len(labels)), labels].mean()

    @staticmethod
    def backward(ctx, grad_out):
        P, labels = ctx.saved_tensors
        Y = F.one_hot(labels, P.shape[1]).to(P.dtype)
        return grad_out * (P - Y) / len(labels), None        # no gradient for the labels

class Forgot1OverN(SoftmaxCrossEntropy):
    @staticmethod
    def backward(ctx, grad_out):
        P, labels = ctx.saved_tensors
        return grad_out * (P - F.one_hot(labels, P.shape[1]).to(P.dtype)), None

gen = torch.Generator().manual_seed(0)
Z = torch.randn(8, 3, generator=gen, dtype=torch.float64, requires_grad=True)
labels = torch.randint(0, 3, (8,), generator=gen)
print(f"{SoftmaxCrossEntropy.apply(Z, labels).item():.6f}  {F.cross_entropy(Z, labels).item():.6f}")
print(torch.autograd.gradcheck(lambda Z: SoftmaxCrossEntropy.apply(Z, labels), (Z,)),
      torch.autograd.gradcheck(lambda Z: Forgot1OverN.apply(Z, labels), (Z,), raise_exception=False))
1.874917  1.874917
True False

The correct rule passes and the one that forgot the \(1/n\) of the mean fails: its gradients are eight times too large.

Explore

This module’s interactive, Shapes and strides, lives in DL01.7.1. For autograd, the best explorer is the code above. In the running network \({\mat{W}^{(1)}\vect{x}} = {(0, 1.5)}\), so \(z^{(1)}_1\) equals the bias \(b^{(1)}_1\). Set that bias to 0, which puts the first hidden unit exactly on the kink of its ReLU, and predict from the rule for non-differentiable points which gradients become zero before you run it.

Pitfalls

Two bar charts side by side showing the first entry of b2.grad after each of three backward calls. Left, marked with a cross and titled no zeroing: the bars rise to 0.731, 1.462 and 2.193, labelled the gradient piles up. Right, marked with a tick and titled grad cleared before each backward: three equal bars of 0.731, labelled the true gradient each time. A dashed line at 0.731 marks the true gradient in both panels.
DL01.7.2-F5

Wrong beside right. backward() adds into .grad, so three calls without clearing hold three times the gradient. Clearing .grad before each call keeps the true gradient.

“backward() gives me this step’s gradient.”

It gives this step’s gradient added to whatever was already in .grad. In a training loop the previous step’s gradient is still there, so the update uses the sum, as in the figure. Clear .grad before every backward pass: optimizer.zero_grad() does it for every parameter.

“Every tensor in the network has a .grad after backward.”

Only leaves with requires_grad=True do. z.grad of an intermediate result is None (PyTorch warns when you read it), because keeping every intermediate gradient would double the memory. Call z.retain_grad() before backward when you need one, or use torch.autograd.grad(L, [z]).

“I can update the weights with W -= lr * W.grad.”

Not on a leaf that requires a gradient: PyTorch refuses an in-place change to it while it is being recorded, and even if it allowed it, the update itself would become part of the graph. Make the update inside with torch.no_grad():, which is what the optimizer’s step() does.

Check yourself

Recall

1. What is a leaf tensor, and which leaves receive a .grad?

Show answer

A tensor not made by a recorded operation (it has no grad_fn), usually one you created. Only leaves with requires_grad=True receive a .grad.

2. What do grad_fn and next_functions hold?

Show answer

grad_fn is the graph node of the operation that made a tensor, with its backward rule; next_functions points to the nodes of that operation’s inputs, so the graph can be walked back.

3. .grad of a weight holds \(\vect{g}\). You run the same forward and backward() once more without clearing. What does it hold?

Show answer

\(2\vect{g}\): backward adds into .grad.

4. What does torch.no_grad() change, and when do you use it?

Show answer

Inside it no operation is recorded and results have requires_grad=False. It is used for weight updates and for evaluating a network.

5. Why must gradcheck be run in float64?

Show answer

It compares the backward rule with finite differences over a step of \(10^{-6}\); in float32, with about 7 digits, those differences are mostly rounding error, and the check fails.

Exercises

1. ★ With \(\vect{w} = (2, -1)\) as a leaf, \(\vect{x} = (3, 4)\) and \(L = (\vect{w} \cdot \vect{x})^2\), find w.grad after one L.backward(), and after a second forward and backward without clearing.

Show solution

\(\vect{w} \cdot \vect{x} = 6 - 4 = 2\) and \({\partial L / \partial \vect{w}} = {2(\vect{w} \cdot \vect{x})\,\vect{x}} = {4\,(3, 4)} = (12, 16)\). After a second call without clearing, w.grad is \((24, 32)\).

2. ★★ In the running network, change \(b^{(1)}_1\) from \(0.5\) to \(-1\). Find the new \(\vect{z}^{(1)}\), which hidden units are on, and both rows of \(\partial L / \partial \mat{W}^{(1)}\).

Show solution

\(z^{(1)}_1 = 1 - 1 - 1 = -1\) and \(z^{(1)}_2 = 1\), so the first unit is off and \(\vect{a}^{(1)} = (0, 1)\). Then \(z_1 = 0 + 1 - 0.5 = 0.5\) and \(z_2 = 0 + 0.5 + 0 = 0.5\), so \(\vect{p} = (0.5, 0.5)\), \(L = \ln 2\) and \(\vect{\delta}^{(2)} = {(0.5, -0.5)}\). Then

\[ \begin{aligned} &\mat{W}^{(2)\top} \vect{\delta}^{(2)} \\ &\quad = (0.5 + 0.5, \\ &\qquad\;\; 0.5 - 0.25) \\ &\quad = (1, 0.25), \end{aligned} \]

and the mask \((0, 1)\) gives \(\vect{\delta}^{(1)} = (0, 0.25)\). So the first row of \(\vect{\delta}^{(1)} \vect{x}\T\) is \({(0, 0)}\) and the second is \({0.25\,(1, 2)} = {(0.25, 0.5)}\). A unit that is off passes back nothing: its weights get no gradient from this example.

3. ★★★ A colleague’s custom Function for \(f(u) = u^3\) returns 3 * u * grad_out from its backward (they forgot one factor \(u\)). Say what gradcheck compares at \(u = 2\), the two numbers it would see, and why the bug could go unnoticed at \(u = 1\).

Show solution

With grad_out \(= 1\), gradcheck compares the backward’s \({3u} = 6\) with the central difference

\[ \begin{aligned} &\frac{f(2 + \epsilon) - f(2 - \epsilon)}{2\epsilon} \\ &\quad = \frac{24\epsilon + 2\epsilon^3}{2\epsilon} \\ &\quad = 12 + \epsilon^2 = 12 + 10^{-12}, \end{aligned} \]

so it fails. At \(u = 1\) the wrong rule gives \(3 \cdot 1 = 3\) and the true derivative is \(3 \cdot 1^2 = 3\): they agree, which is why checks use random inputs, not ones and zeros.

Summary card

The forward pass records a graph; backward() walks it and adds into the leaves’ .grad.

The anchor figure again: the 2-2-2 network as a chain of operation boxes with their grad_fn names, forward arrows in vermillion, backward arrows in green ending in the dot-grad slots of the orange leaves W1, b1, W2 and b2.
DL01.7.2-F1

The graph PyTorch records.

  • Rule. L.backward() does \(\theta.\mathrm{grad} \mathrel{+}= \partial L / \partial \theta\) for each leaf.
  • Hand check. \(\vect{\delta}^{(2)} = \vect{p} - \vect{y}\), \(\partial L / \partial \mat{W}^{(2)} = \vect{\delta}^{(2)} \vect{a}^{(1)\top}\).
  • No .grad touched. torch.autograd.grad(L, [theta]).
  • Out of the graph. t.detach() for one tensor, with torch.no_grad(): for a block.
  • Own backward. torch.autograd.gradcheck in float64.

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • PyTorch 2.15 documentation, Autograd mechanics: the graph is recorded as operations run, with inputs as leaves and outputs as roots, and each tensor’s grad_fn as its entry point; the graph is rebuilt on every iteration; an operation is recorded only if one of its inputs requires grad; computations in no-grad mode behave as if no input requires grad; gradients are accumulated into the .grad of leaf tensors; and the rules for non-differentiable points, where a locally convex function such as ReLU at 0 takes the minimum-norm subgradient.
  • PyTorch 2.15 documentation, torch.autograd.gradcheck: it checks analytical gradients against small finite differences, with defaults eps \(10^{-6}\), atol \(10^{-5}\) and rtol \(10^{-3}\) designed for double precision, and it will likely fail in lower precision.
  • A. Zhang, Z. C. Lipton, M. Li and A. J. Smola, Dive into Deep Learning, section 2.5: requires_grad_, backward and .grad (2.5.1), the gradient buffer not being reset automatically, backward on non-scalars (2.5.2), and detach (2.5.3).

Next. DL01.7.3 nn.Module and layers stops making leaves by hand: a module creates its parameters, registers them, and hands them to autograd.

This lesson unlocks

DL01.7.3 nn.Module and layers

Your progress is saved in this browser only. No account, no tracking.
Back to top