Skip to main content

Checking and debugging backpropagation

A backward pass with a bug still returns numbers that look plausible. Check every weight of the XOR network against central differences in float64, away from kinks, and watch the check catch five classic bugs, from a missing transpose to overwritten adjoints.
M05.9.3L2 Intermediate25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: M05.9.2 Backpropagation in matrix form for a batch

What you will be able to do
  • Gradient-check a network by central differences on every weight, and read the relative error against the thresholds used in practice.
  • Explain why the check is run in float64, at a point away from kinks, on a few examples.
  • Recognise five classic backpropagation bugs and the signature each leaves in a per-weight check.
  • Write a check that reports the worst weight, and use scipy.optimize.check_grad on the flattened weights.

The question

You have written the backward pass of M05.9.2. It runs, the shapes are right, and the loss even goes down when you train. Is the gradient right? Several real bugs pass all three of those tests. What test do they fail, and how do you tell from its result which bug you have?

From the last lesson

1. What is the gradient of \(\mat{W}^{(l)}\) for a batch?

Show answer

\({\mat{\Delta}^{(l)\top}\mat{A}^{(l-1)}}\), the sum over examples of adjoint times input.

2. Where does the factor \(1/n\) of the mean loss enter?

Show answer

Once, in the output adjoint \({(\vect{p} - \vect{y})/n}\).

Intuition

Backpropagation claims to know the slope of the loss along each weight. There is a slow, simple way to measure that slope without any calculus: move the weight a little up, run the network, move it a little down, run it again, and divide the change in loss by the change in weight.

Two panels. Left, titled Two runs per weight: an orange box w1 22 equals 0.5 with arrows to two boxes, 0.5 plus h giving L plus and 0.5 minus h giving L minus; below, in vermillion, L plus minus L minus over 2h equals 0.1395, in green, backprop: 0.1395, and the note they agree to ten places. Right, titled h drawn large here: the purple loss of the XOR batch as a function of w1 22 from minus 1.5 to 2.5, an orange point at 0.5, a vermillion chord through two points on either side labelled chord over plus or minus h, and a dashed green line through the point labelled backprop slope, nearly parallel to the chord.
M05.9.3-F1

A gradient check for one weight. Two extra runs of the network give a chord whose slope is the central difference; backpropagation claims the slope of the tangent. With a small \(h\) the two must agree. (The step is drawn far larger than the \({h = 10^{-5}}\) used in practice, so that the chord can be seen.)

Repeat this for every weight and you have a second gradient, computed in a completely different way. If backpropagation is right, the two agree to many digits. If it is wrong, they disagree, and where they disagree points at the bug. This is gradient checking, the numerical gradient of M05.2.4 applied to a network.

Build it up

Step 1. Flatten the weights. Put all nine parameters of the running network into one vector \(\theta\), in a fixed order: the four entries of \(\mat{W}^{(1)}\), the two of \(\vect{b}^{(1)}\), the two of \(\mat{W}^{(2)}\), and \(b^{(2)}\). The loss is a function \({L(\theta)}\) of nine numbers.

Step 2. Two runs per weight. For each \(k\), the central difference \({(L(\theta + h\vect{e}_k) - L(\theta - h\vect{e}_k))/(2h)}\) approximates \({\partial L/\partial\theta_k}\) with an error of order \(h^2\). For nine weights that is 18 forward passes, against one forward and one backward pass for backpropagation.

Step 3. One number for the agreement. Compare the two gradients by the relative error of M05.2.4, \({\norm{\vect{g} - \vect{g}_{\text{num}}} / (\norm{\vect{g}} + \norm{\vect{g}_{\text{num}}})}\), which does not care how large the gradient is. For the bad cases, also look weight by weight.

Step 4. Choose the conditions. Use float64, where rounding is about \({10^{-16}}\) of a value. float32 rounds at about \({10^{-7}}\) of a value, and dividing a rounding error of that size by \({2h = 2 \times 10^{-5}}\) leaves an error of about half a per cent of the loss in each entry of the numerical gradient itself: here a few thousandths, on entries of about 0.1. Check at a point where no ReLU or other kink sits within \(h\) of a unit’s input. And use a few examples: the check needs no more, and runs faster.

Step 5. Know the bugs. Each classic bug changes the gradient in its own pattern. Two of them change only some weights:

A plot of two curves against z from minus 3 to 3. A solid green bell, one minus tanh squared of z, peaks at 1 at z equals 0 and falls to near 0 at both ends; a green point at z equals minus 1 is labelled 0.420. A dashed vermillion curve, one minus tanh squared of tanh z, also peaks at 1 but levels off near 0.42 at both ends; a vermillion point at z equals minus 1 is labelled 0.588. Notes at the right: right: one minus tanh squared z, equal to one minus a squared; bug: one minus tanh squared a, too flat: it stays above 0.41.
M05.9.3-F6

The slope of tanh, evaluated where it should be (at the unit’s input \(z\), written \(1 - a^2\) with the recorded output) and where a common bug evaluates it (at the output \(a\)). At \({z = -1}\) the bug gives 0.588 instead of 0.420.

Two bar charts of the four examples' contributions to the gradient of w1 12: 0 for (0, 0), minus 0.185 for (0, 1), 0 for (1, 0) and 0.159 for (1, 1), then a result bar. Left, marked with a cross and titled grad equals each example: the result is 0.159, the last contribution, in vermillion. Right, marked with a tick and titled grad plus-equals each example: the result is minus 0.025, the sum, in green.
M05.9.3-F4

A weight is shared by every example of the batch, so its gradient must add their contributions (M05.8.3). A loop that assigns instead of adding keeps only the last example: here 0.159 instead of \({-0.025}\), even the wrong sign.

The formal statement

NoteGradient check

For a loss \({L : \R^n \to \R}\), a claimed gradient \(\vect{g}\) at \(\theta\) and a step \(h\), the numerical gradient has entries

\[ g_{\text{num}, k} = \frac{L(\theta + h\vect{e}_k) - L(\theta - h\vect{e}_k)}{2h}, \]

and the check reports the relative error

\[ r = \frac{\norm{\vect{g} - \vect{g}_{\text{num}}}}{\norm{\vect{g}} + \norm{\vect{g}_{\text{num}}}} , \]

with the weight \({k^\star}\) where \({\abs{g_k - g_{\text{num}, k}}}\) is largest. In float64 with \({h \approx 10^{-5}}\), for a smooth loss: \(r\) near \({10^{-7}}\) or below is good; up to \({10^{-4}}\) is acceptable only if the loss has kinks; up to \({10^{-2}}\) is suspicious; above that is probably wrong.

Why it works. If \(L\) has continuous third derivatives along each axis, the Taylor polynomials of \(L(\theta \pm h\vect{e}_k)\) (M04.8.3) give

\[ \begin{aligned} L(\theta \pm h\vect{e}_k) &= L(\theta) \pm h\,\partial_k L \\ &\quad + \tfrac{h^2}{2}\,\partial_k^2 L + R_\pm , \end{aligned} \]

where, by the remainder bound of M04.8.4, \({\abs{R_\pm} \le M h^3/6}\) and \(M\) bounds \({\abs{\partial_k^3 L}}\) between \({\theta - h\vect{e}_k}\) and \({\theta + h\vect{e}_k}\). Subtracting, the even terms cancel, and dividing by \(2h\) leaves \({\partial_k L}\) plus an error of at most \({M h^2/6}\), of order \(h^2\). Rounding adds an error of about \({\varepsilon\,\abs{L}/h}\), where \(\varepsilon\) is the machine epsilon, so \({h \approx 10^{-5}}\) balances the two in float64 (M05.2.4). A correct backward pass therefore differs from \(\vect{g}_{\text{num}}\) only by these small errors, while a bug that changes any entry of \(\vect{g}\) by more than them shows up in \(r\).

What it cannot do. A check is evidence at one point \(\theta\). A bug that vanishes there passes: a term multiplied by a bias that happens to be 0, or a ReLU unit that is never active on the chosen examples. Check at a generic point, after a few training steps if the starting weights are special, as CS231n recommends.

A horizontal logarithmic scale of relative error from 10 to the minus 13 to 1, with coloured bands along the top: good up to 10 to the minus 7, kinks only up to 10 to the minus 4, suspicious up to 10 to the minus 2, probably wrong above. Seven rows. Two green dots in the good band, both of order 10 to the minus 11: correct code, and ReLU away from kinks. Five vermillion crosses in the probably-wrong band: missing transpose, 0.10; tanh prime at the output, 0.22; ReLU at a kink, 0.57; sum, not mean, 0.60; overwritten adjoints, 0.62.
M05.9.3-F2

The relative error of seven versions of backpropagation on the XOR network, against central differences in float64. Correct code lands near \({10^{-10}}\); every bug lands above \({10^{-2}}\), and so does correct ReLU code checked exactly at a kink.

Worked example

The running network at its starting weights, the whole batch of four, \({h = 10^{-5}}\).

Correct code. \(r\) is a few times \({10^{-11}}\) (about \({5 \times 10^{-11}}\) here; the last digits depend on the machine’s rounding): about ten agreeing digits, the most a central difference can deliver. Even the largest gap, on whichever weight it falls, is about \({10^{-11}}\): noise, not a bug.

Missing transpose. Writing \({\mat{X}\T\mat{\Delta}^{(1)}}\) for \({\mat{\Delta}^{(1)\top}\mat{X}}\) gives a \({2 \times 2}\) matrix either way, so no shape check complains. It is the transpose of the right answer, so the diagonal is still right and only \(w^{(1)}_{12}\) and \(w^{(1)}_{21}\) are swapped: \(-0.025\) and 0.017 change places. The gap on each is \({0.017 + 0.025 \approx 0.043}\), and \({r \approx 0.10}\).

Two bar charts on a logarithmic scale from 10 to the minus 15 to 1, one bar per parameter: w1 11, w1 12, w1 21, w1 22, b1 1, b1 2, w2 1, w2 2 and b2. Top, marked with a tick and titled Correct: gap per weight: nine short green bars near 10 to the minus 12 to 10 to the minus 11, relative error of order 10 to the minus 11. Bottom, marked with a cross and titled X transpose Delta instead of Delta transpose X: the same short green bars except two tall vermillion bars for w1 12 and w1 21 reaching about 0.04, relative error 1.0e minus 01.
M05.9.3-F3

The gap \({\abs{g_k - g_{\text{num},k}}}\) weight by weight. The missing transpose leaves its fingerprint on exactly the two off-diagonal weights of \(\mat{W}^{(1)}\), the entries a transpose moves.

Sum instead of mean. Forgetting the \(1/n\) multiplies every entry by \({n = 4}\), so \({r} = {\norm{4\vect{g} - \vect{g}}/(4\norm{\vect{g}} + \norm{\vect{g}})} = {3/5} = {0.60}\), exactly. Every weight is off, in proportion to its size.

tanh’ at the output. Using \({1 - \tanh^2 a}\) instead of \({1 - a^2}\) inflates every slope of the hidden layer where \({a \neq 0}\) (the figure in Step 5), so all six parameters of the first layer are off and the second layer’s three are exact: \({r \approx 0.22}\).

Overwritten adjoints. The loop that assigns instead of adding keeps the last example, \({(1, 1)}\), for every parameter: \({r \approx 0.62}\), and the worst weight is \(w^{(1)}_{12}\), whose gradient even changes sign.

A false alarm. Replace tanh by ReLU, whose derivative at exactly 0 code takes as 0. With zero biases the example \({(0, 0)}\) puts both hidden units at \({z = 0}\), and \({(1, 1)}\) puts the first there too. A central difference straddles the kink and measures the average slope \({1/2}\), so \({r \approx 0.57}\) although the code is right. Move the biases to 0.1 and nothing sits on a kink: \(r\) is again of order \({10^{-11}}\).

Code

From scratch: the network’s loss and its backward pass as functions of the flat vector \(\theta\), with a switch that plants each bug, and a check that reports the relative error and the worst weight.

import numpy as np
from scipy.optimize import check_grad

X = np.array([[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]])
y = np.array([0.0, 1.0, 1.0, 0.0])
NAMES = ["W1_11", "W1_12", "W1_21", "W1_22", "b1_1", "b1_2", "W2_1", "W2_2", "b2"]

def unpack(theta):
    return theta[:4].reshape(2, 2), theta[4:6], theta[6:8], theta[8]

def run(theta, act):
    W1, b1, W2, b2 = unpack(theta)
    Z1 = X @ W1.T + b1
    A1 = np.tanh(Z1) if act == "tanh" else np.maximum(Z1, 0.0)
    return Z1, A1, 1 / (1 + np.exp(-(A1 @ W2 + b2)))

def loss(theta, act="tanh"):
    p = run(theta, act)[2]
    return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))

def backprop(theta, bug=None, act="tanh"):
    W1, b1, W2, b2 = unpack(theta)
    Z1, A1, p = run(theta, act)
    d2 = (p - y) / (1 if bug == "sum" else len(y))
    if act == "relu":
        slope = (Z1 > 0).astype(float)                 # ReLU'(0) taken as 0
    elif bug == "output":
        slope = 1 - np.tanh(A1) ** 2                   # tanh' evaluated at the output
    else:
        slope = 1 - A1 ** 2                            # tanh' at the input z, written with a = tanh z
    D1 = np.outer(d2, W2) * slope
    if bug == "overwrite":
        for i in range(len(y)):                        # "=" where "+=" was meant
            gW1, gb1, gW2, gb2 = np.outer(D1[i], X[i]), D1[i], d2[i] * A1[i], d2[i]
    else:
        gW1 = X.T @ D1 if bug == "transpose" else D1.T @ X
        gb1, gW2, gb2 = D1.sum(axis=0), d2 @ A1, d2.sum()
    return np.concatenate([gW1.ravel(), gb1, gW2, [gb2]])

def gradient_check(theta, bug=None, act="tanh", h=1e-5):
    g = backprop(theta, bug, act)
    num = np.array([(loss(theta + h * e, act) - loss(theta - h * e, act)) / (2 * h) for e in np.eye(len(theta))])
    r = np.linalg.norm(g - num) / (np.linalg.norm(g) + np.linalg.norm(num))
    return r, NAMES[int(np.argmax(np.abs(g - num)))]

theta0 = np.array([1.0, -1.0, 0.5, 0.5, 0.0, 0.0, 2.0, -1.0, 0.0])   # the starting weights of M05.9.1
for bug in [None, "transpose", "sum", "output", "overwrite"]:
    r, worst = gradient_check(theta0, bug)
    print(f"{str(bug):10s} relative error {r:.1e}   worst weight {worst}")
None       relative error 4.6e-11   worst weight b1_1
transpose  relative error 1.0e-01   worst weight W1_12
sum        relative error 6.0e-01   worst weight b1_1
output     relative error 2.2e-01   worst weight b1_1
overwrite  relative error 6.2e-01   worst weight W1_12

Every planted bug lands far above \({10^{-2}}\) and the correct code near \({10^{-10}}\). Now the two conditions of Step 4. First precision: the same check with the loss computed in float32.

def loss32(theta):
    t = np.asarray(theta, dtype=np.float32)
    W1, b1, W2, b2 = t[:4].reshape(2, 2), t[4:6], t[6:8], t[8]
    A1 = np.tanh(X.astype(np.float32) @ W1.T + b1)
    p = 1 / (1 + np.exp(-(A1 @ W2 + b2)))
    yy = y.astype(np.float32)
    return float(-np.mean(yy * np.log(p) + (1 - yy) * np.log(1 - p)))

h = 1e-5
num32 = np.array([(loss32(theta0 + h * e) - loss32(theta0 - h * e)) / (2 * h) for e in np.eye(9)])
r32 = np.linalg.norm(g0 - num32) / (np.linalg.norm(g0) + np.linalg.norm(num32))
print(f"correct code, numerical gradient in float32: relative error {r32:.1e}")
correct code, numerical gradient in float32: relative error 1.2e-02

The correct gradient now looks suspicious or worse; the exact figure depends on the machine’s float32 arithmetic, but it is many orders of magnitude above the float64 result. Then the kink:

r_kink, worst_kink = gradient_check(theta0, act="relu")
away = theta0.copy(); away[4:6] = 0.1
r_away, _ = gradient_check(away, act="relu")
print(f"ReLU at the starting weights: relative error {r_kink:.2f}, worst weight {worst_kink}")
print(f"ReLU with biases 0.1:         relative error {r_away:.1e}")
ReLU at the starting weights: relative error 0.57, worst weight b1_1
ReLU with biases 0.1:         relative error 1.9e-11

Finally the library call. scipy.optimize.check_grad takes the loss, the gradient and the point, and returns the 2-norm of the difference between the gradient and a forward-difference estimate (step about \({1.49 \times 10^{-8}}\)). It does not divide by the sizes, so read it against the size of the gradient, about 0.3 here.

print(f"norm of the gradient:   {np.linalg.norm(g0):.3f}")
print(f"check_grad, correct:    {check_grad(loss, backprop, theta0):.1e}")
print(f"check_grad, transposed: {check_grad(loss, lambda th: backprop(th, 'transpose'), theta0):.3f}")
norm of the gradient:   0.300
check_grad, correct:    2.4e-08
check_grad, transposed: 0.060

The transposed version is off by \({\sqrt{2} \times 0.043} \approx {0.060}\), the two swapped entries.

Explore

The module’s interactive, the Backprop stepper in M05.9.2, runs this check live: when the backward pass is complete, its table shows each weight’s gradient beside its central difference and the read-out gives the relative error. Train a few steps and run it again: the agreement holds at every point along the way.

Pitfalls

Two panels, each the blue ReLU graph, flat at 0 for negative z and rising with slope 1 for positive z. Left, marked with a cross and titled Checked at z equals 0: an orange point at the kink and a vermillion chord from z equals minus 0.45 to 0.45, with the notes chord slope 0.5 and backprop slope 0. Right, marked with a tick and titled Checked at z equals 0.6: the point on the rising part and a chord lying along the line, with the notes chord slope 1 and backprop slope 1.
M05.9.3-F5

Wrong beside right. At a kink the two probes sit on different pieces of the function and the chord has slope \({1/2}\), which no derivative has. Away from the kink the chord and the backward pass agree.

“The check failed, so my ReLU code is wrong.”

Not if a unit’s input is within \(h\) of 0. There the loss has no derivative, and the check compares backprop’s convention (slope 0) with an average of the two sides. Check at a point where every input is clear of the kink, or count how many inputs are within \(h\) of one before blaming the code.

“The loss goes down when I train, so the gradient is right.”

Forgetting the \(1/n\) gives four times the true gradient, which trains like a four times larger learning rate, and the tanh’ bug still points roughly downhill. Training curves hide such bugs; the check exposes all of them at once.

“A check in float32 with a tiny step is more accurate.”

Smaller \(h\) means less truncation error but more rounding error, and float32 rounds about half a billion times more coarsely than float64. Run checks in float64 with \({h \approx 10^{-5}}\), even if training runs in lower precision.

“One passing check proves the code.”

It proves agreement at one point. A bug multiplied by a zero bias, or hidden in a unit that is never active on the chosen examples, passes there. Check at a generic point, on a few examples that exercise every unit.

Check yourself

Recall

1. How many forward passes does a central-difference check of \(n\) weights need?

Show answer

\(2n\): two per weight.

2. What relative error do you expect from correct code in float64 with \({h = 10^{-5}}\), and what counts as probably wrong?

Show answer

About \({10^{-7}}\) or less (here near \({10^{-10}}\)) for correct code; above \({10^{-2}}\) probably wrong.

3. Why is a gradient check unreliable at a ReLU kink?

Show answer

The loss has no derivative there; the central difference straddles the kink and measures an average of the two one-sided slopes, while the code uses one of them.

4. Which weights does a missing transpose in the gradient of a square weight matrix get wrong?

Show answer

Only the off-diagonal entries; the diagonal of a matrix and of its transpose agree.

5. What does scipy.optimize.check_grad return?

Show answer

The 2-norm of the difference between the given gradient and a forward-difference estimate, not a relative error.

Exercises

1. ★ Show that a gradient that is \(c\) times the true one, with \({c > 0}\), has relative error \({\abs{c - 1}/(c + 1)}\) against an exact numerical gradient, and evaluate it for the sum bug of the XOR batch.

Show solution

With \({\vect{g}} = {c\,\vect{g}_{\text{num}}}\): \({\norm{c\vect{g}_{\text{num}} - \vect{g}_{\text{num}}}} = {\abs{c - 1}\,\norm{\vect{g}_{\text{num}}}}\) and \({\norm{c\vect{g}_{\text{num}}}} + {\norm{\vect{g}_{\text{num}}}} = {(c + 1)\norm{\vect{g}_{\text{num}}}}\); the norms cancel, leaving \({\abs{c - 1}/(c + 1)}\). With \({c = n = 4}\), \({3/5 = 0.6}\), as measured.

2. ★★ For the missing transpose, show that \({r} = {\sqrt{2}\,\abs{g_{12} - g_{21}} / (2\norm{\vect{g}})}\), where \(g_{12}\) and \(g_{21}\) are the true gradients of \(w^{(1)}_{12}\) and \(w^{(1)}_{21}\), and check it against 0.10 using \({\norm{\vect{g}} \approx 0.300}\).

Show solution

The buggy gradient equals the true one except that the entries for \(w^{(1)}_{12}\) and \(w^{(1)}_{21}\) are swapped. The difference has two non-zero entries, \({g_{21} - g_{12}}\) and \({g_{12} - g_{21}}\), so its norm is \({\sqrt{2}\,\abs{g_{12} - g_{21}}}\). Swapping entries does not change a norm, so the denominator is \({2\norm{\vect{g}}}\) (taking the numerical gradient as exact). With \({g_{12} \approx -0.0255}\) and \({g_{21} \approx 0.0171}\): \({\sqrt{2} \times 0.0426 / 0.600} \approx {0.100}\).

3. ★★★ A network has a million weights, so \(2n\) forward passes are too many. Instead, pick a random unit vector \(\vect{u}\) and compare \({(L(\theta + h\vect{u}) - L(\theta - h\vect{u}))/(2h)}\) with \({\vect{g}\T\vect{u}}\). What does this cost, why does it detect a wrong \(\vect{g}\) with probability 1 (ignoring rounding), and why are a few directions still better than one?

Show solution

It costs two forward passes and one dot product. The central difference along \(\vect{u}\) approximates the directional derivative \({\nabla L\T\vect{u}}\) (M05.3.1). If \({\vect{g} \neq \nabla L}\), the error vector \({\vect{e} = \vect{g} - \nabla L}\) is non-zero, and the check misses it only when \({\vect{e}\T\vect{u} = 0}\), that is when \(\vect{u}\) lies in the hyperplane perpendicular to \(\vect{e}\). A random direction, drawn from a continuous distribution, lands there with probability 0. But \({\vect{e}\T\vect{u}}\) can still be small: if the bug affects one weight out of a million, a random unit vector has a component of size about \({1/\sqrt{n} = 10^{-3}}\) along it, so the discrepancy may hide in the noise. Several directions, or a full check on a few weights per layer, make that unlikely; check_grad offers direction="random" for this.

Summary card

Check backprop against two runs per weight, and read the relative error.

The anchor figure again: one weight nudged up and down gives a chord whose slope, 0.1395, matches the slope claimed by backpropagation.
M05.9.3-F1

Two runs per weight.

  • Two runs per weight: \({(L_+ - L_-)/2h}\), with \({h \approx 10^{-5}}\).
  • Relative error: good below \({10^{-7}}\), wrong above \({10^{-2}}\).
  • float64, away from kinks, a few examples, a generic point.
  • Read the worst weight: each bug leaves its own fingerprint.

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • Stanford CS231n, Neural Networks Part 3: Learning and Evaluation, section Gradient checks: use the centred difference with \(h\) about \({10^{-5}}\); compare by a relative error, read as probably wrong above \({10^{-2}}\), acceptable below \({10^{-4}}\) only for objectives with kinks, and good at \({10^{-7}}\) or less; use double precision; beware kinks such as ReLU’s, which a step can cross; use only a few data points; and run the check after a short burn-in rather than at a possibly pathological starting point. (CS231n divides by the larger of the two sizes, or by their sum; this course divides by the sum of the two norms, as in M05.2.4.)
  • SciPy, scipy.optimize.check_grad: returns the 2-norm of the difference between the gradient and a forward finite-difference approximation, with default step \({\sqrt{\text{machine epsilon}}} \approx {1.49 \times 10^{-8}}\), and offers direction="random" to check along a random vector instead of every axis.

Next. M05.9.4 Cost, memory and gradient flow through depth counts what backpropagation costs and follows the gradient through twenty layers.

This lesson unlocks

M05.9.4 Cost, memory and gradient flow through depth

Your progress is saved in this browser only. No account, no tracking.
Back to top