Skip to main content

Training the hand-built network

Put the hand-built module library to work: forward, loss, backward and a gradient step for every parameter, in mini-batches over the two moons, until a 2-16-16-2 network bends its decision boundary around the data, with a validation loss beside the training loss.
DL01.5.4L2 Intermediate25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: DL01.5.2 Layers as modules with forward and backward ML01.5 Gradient descent from scratch

What you will be able to do
  • Write a mini-batch gradient-descent loop around the module library: shuffle, forward, loss, backward, update.
  • Carry out one gradient step on the running network by hand and predict the fall in the loss to first order.
  • Train a 2-16-16-2 network on the two moons and read its training and validation curves.
  • Compare the hand-built loop with scikit-learn’s MLPClassifier and name what the library adds by default.

The question

Three lessons of gradients, and not one step of learning yet. The module library can now compute the gradient of any loss with respect to every weight of a stack of layers. Is that enough to train a real network? Take the two moons, which no straight line can separate, and a network with 354 parameters: can a page of hand-written NumPy bend its decision boundary around them in a few seconds?

From the last lesson

1. A value feeds two heads. What is its gradient?

Show answer

The sum of the gradients that the two heads send back.

2. What goes wrong if a backward pass assigns a branch’s gradient instead of adding it?

Show answer

Whichever branch runs last overwrites the others, so the gradient misses their contributions; a central-difference check shows the error.

Intuition

Training is a loop around what we already have. Take a small batch of examples, run them forward to the loss, run backward to get every gradient, and move each parameter a little against its gradient. Repeat, with fresh batches, until the loss stops falling.

Three square panels of the two moons, 150 training points: class 1 as blue circles, class 2 as sky-blue squares. The background is shaded by the class the network predicts, with the decision boundary drawn as a vermillion curve. Epoch 1: the boundary is made of a few straight pieces that cut through both moons, and training accuracy is 81 percent. Epoch 10: the boundary has started to bend, 86 percent. Epoch 200: the boundary snakes between the two moons, 99 percent.
DL01.5.4-F1

The 2-16-16-2 network on the two moons after 1, 10 and 200 epochs. Every bend in the vermillion curve was put there by gradients that the module library computed.

Each pass over the training set is an epoch. Before the first one, the random starting weights put every training point in the same class. After one epoch the boundary is a few straight pieces that still cut through both moons. Gradient steps then fold it, a little at a time, until it follows the gap between the moons.

Build it up

Step 1. The data. The course’s two-class set, \({\texttt{make\_moons}(200, \text{noise} = 0.2)}\) with random_state=0, split with stratification into 150 training and 50 validation points: 75 and 25 of each class.

Step 2. The network. Linear(2, 16), ReLU, Linear(16, 16), ReLU, Linear(16, 2), and the fused softmax cross-entropy. Its parameters, weights plus biases layer by layer, number

\[ \begin{aligned} &(16 \times 2 + 16) + (16 \times 16 + 16) \\ &\quad + (2 \times 16 + 2) \\ &= 48 + 272 + 34 = 354 . \end{aligned} \]

Step 3. The start. Weights are drawn from a normal distribution with standard deviation \({\sqrt{2/n_{\text{in}}}}\), where \(n_{\text{in}}\) is the number of inputs of the layer, and biases start at zero. Why this scale, and why the weights must not all be equal, is the subject of DL02.3; the pitfalls below show the second point.

Step 4. One iteration.

A cycle of four boxes joined by arrows. Forward: the batch of 32 rows through the modules, each filling its cache. Loss: the mean cross-entropy over the batch. Backward: from (P minus Y) over 32 down through the modules, each storing its parameter gradients. Update: every W and b moves by minus eta times its gradient, here eta equals 0.2. An arrow from update back to forward is labelled next batch, and beside the cycle a note says shuffle the 150 training points at the start of each epoch, 5 batches per epoch, 4 of 32 and one of 22.
DL01.5.4-F2

One iteration of the loop. The update is the only new line: everything else is the forward and backward of DL01.5.2.

Step 5. One epoch. Shuffle the 150 training indices, cut them into batches of 32, and run one iteration per batch. \({150} = {4 \times 32 + 22}\), so an epoch is five updates, the last on a short batch. After the epoch, measure the loss and the accuracy on the whole training set and on the validation set.

The formal statement

NoteMini-batch gradient descent (ML01.5.4)

Let \(\theta\) stand for all the weights and biases. For a batch \(B\) of example indices, with \({L_B(\theta)} = {\frac{1}{\abs{B}}\sum_{i \in B} \ell_i(\theta)}\), one update is

\[ \theta \leftarrow \theta - \eta\, \cgrad{\nabla_\theta L_B(\theta)} , \]

applied to every parameter at once, with the gradient computed by one forward and one backward pass at the current \(\theta\).

Backpropagation supplies \({\nabla_\theta L_B}\), already divided by \({\abs{B}}\) because the loss module returns \({(\mat{P} - \mat{Y})/\abs{B}}\). A short last batch therefore gives a gradient of the same size as a full one, not a smaller one.

How much should one step lower the loss? Write \(\vect{g}\) for the gradient, flattened into one vector. To first order (the linear approximation of M05.2),

\[ \begin{aligned} L(\theta - \eta\vect{g}) &\approx L(\theta) + \vect{g}\T(-\eta\vect{g}) \\ &= L(\theta) - \eta\,\norm{\vect{g}}^2 . \end{aligned} \]

The step lowers the loss by about \(\eta\) times the squared length of the gradient, as long as \(\eta\) is small enough for the linear approximation to hold. The worked example tests this.

Two panels against the epoch from 0 to 200. Left, loss: a solid reddish-purple training curve falling from 0.87 to about 0.04, and a dashed validation curve that falls with it to about 0.1 by epoch 50, then levels off and wanders between about 0.07 and 0.22. Right, accuracy: training accuracy rising from 50 percent to 99 percent, and the dashed validation accuracy rising from 50 percent to 96 percent.
DL01.5.4-F3

Training (solid) and validation (dashed) curves of the run. Both losses fall together at first. Later the training loss keeps falling while the validation loss levels off near 0.1 and wanders: the network is starting to fit the noise of its 150 points (ML01.8).

Worked example

One step of gradient descent on the running network of DL01.5.1, by hand, with \({\eta} = {0.02}\).

A before-and-after trace for one gradient step on the running 2-2-2 network. Left column, before: W1 with rows (1, minus 0.5) and (minus 0.5, 1); the gradient of W1 with rows (1.462, 2.924) and (0.366, 0.731); loss 1.313, p of class 2 0.269. An arrow labelled minus 0.02 times the gradient leads to the right column, after: W1 with rows (0.971, minus 0.558) and (minus 0.507, 0.985), b2 equals (minus 0.515, 0.015), loss 1.015, p of class 2 0.362. A note compares the actual fall, 0.298, with the first-order prediction, 0.02 times 16.03, which is 0.321.
DL01.5.4-F4

One update, before and after. All twelve parameters move at once; the weight with the largest gradient, \({w^{(1)}_{12}}\), moves most.

The update. Every parameter moves by \({-0.02}\) times its gradient. For two of them: \({w^{(1)}_{12}} \leftarrow {-0.5 - 0.02 \times 2.924} = {-0.558}\) and \({b^{(2)}_2} \leftarrow {0 - 0.02 \times (-0.731)} = {0.015}\). The rest follow the same rule.

The prediction. The squared length of the gradient is the sum of the squares of all twelve entries:

\[ \begin{aligned} \norm{\vect{g}}^2 &\approx 11.357 + 2.271 \\ &\quad + 1.336 + 1.069 \\ &= 16.03 , \end{aligned} \]

for \(\mat{W}^{(1)}\), \(\vect{b}^{(1)}\), \(\mat{W}^{(2)}\) and \(\vect{b}^{(2)}\) in turn. The first-order estimate of the fall is \({0.02 \times 16.03} \approx {0.321}\).

The check. Run the forward pass again with the new parameters: the probability of class 2 rises from 0.269 to 0.362 and the loss falls from 1.313 to 1.015, a fall of 0.298. The estimate was close; the small difference is the curvature that the linear approximation ignores. With \({\eta} = {0.1}\) the same step would be large enough to switch hidden unit 1 off, a reminder that the step size matters.

Code

The library of DL01.5.2, made general enough for any layer sizes, is in the folded cell.

The modules of DL01.5.2 
import numpy as np

class Linear:
    def __init__(self, W, b):
        self.W, self.b = np.array(W, float), np.array(b, float)
    def forward(self, A):
        self.A = A
        return A @ self.W.T + self.b
    def backward(self, G):
        self.dW, self.db = G.T @ self.A, G.sum(axis=0)
        return G @ self.W
    def params(self):
        return [(self.W, "dW"), (self.b, "db")]

class ReLU:
    def forward(self, Z):
        self.mask = Z > 0
        return Z * self.mask
    def backward(self, G):
        return G * self.mask
    def params(self):
        return []

class SoftmaxCrossEntropy:
    def forward(self, Z, labels):
        Z = Z - Z.max(axis=1, keepdims=True)
        self.P = np.exp(Z) / np.exp(Z).sum(axis=1, keepdims=True)
        self.Y = np.eye(Z.shape[1])[labels]
        return -np.mean(np.log(self.P[np.arange(len(labels)), labels]))
    def backward(self):
        return (self.P - self.Y) / len(self.P)

class Sequential:
    def __init__(self, *layers):
        self.layers = layers
    def forward(self, X):
        for layer in self.layers:
            X = layer.forward(X)
        return X
    def backward(self, G):
        for layer in reversed(self.layers):
            G = layer.backward(G)
        return G

The worked step first, on the running network.

net = Sequential(Linear([[1, -0.5], [-0.5, 1]], [0.5, -0.5]), ReLU(), Linear([[1, 1], [-1, 0.5]], [-0.5, 0]))
x1, k1 = np.array([[1.0, 2.0]]), np.array([1])
ce = SoftmaxCrossEntropy()
before = ce.forward(net.forward(x1), k1)
net.backward(ce.backward())
grads = [getattr(m, name) for m in net.layers for _, name in m.params()]
norm_sq = sum((g ** 2).sum() for g in grads)
for m in net.layers:
    for P, name in m.params():
        P -= 0.02 * getattr(m, name)                  # the update, in place
after = ce.forward(net.forward(x1), k1)
print(f"loss {before:.3f} -> {after:.3f}; fall {before - after:.3f}, first-order estimate {0.02 * norm_sq:.3f}")
loss 1.313 -> 1.015; fall 0.298, first-order estimate 0.321

Now the loop. One generator draws the starting weights and then shuffles every epoch, so the run is the same every time; DL01.6.4 will repeat it step for step with its own engine.

import time
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

X, y = make_moons(n_samples=200, noise=0.2, random_state=0)
X_tr, X_va, y_tr, y_va = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)

rng = np.random.default_rng(0)
sizes = [2, 16, 16, 2]
layers = []
for n_in, n_out in zip(sizes[:-1], sizes[1:]):
    layers += [Linear(rng.normal(0, np.sqrt(2 / n_in), (n_out, n_in)), np.zeros(n_out)), ReLU()]
net = Sequential(*layers[:-1])                         # no ReLU after the last layer
n_params = sum(P.size for m in net.layers for P, _ in m.params())

def evaluate(net, X, y):
    loss = SoftmaxCrossEntropy().forward(net.forward(X), y)
    return loss, np.mean(net.forward(X).argmax(axis=1) == y)

lr, batch, epochs = 0.2, 32, 200
history = [evaluate(net, X_tr, y_tr) + evaluate(net, X_va, y_va)]
start = time.perf_counter()
for epoch in range(epochs):
    order = rng.permutation(len(y_tr))
    for s in range(0, len(y_tr), batch):
        idx = order[s:s + batch]
        loss_fn = SoftmaxCrossEntropy()
        loss_fn.forward(net.forward(X_tr[idx]), y_tr[idx])   # forward and loss
        net.backward(loss_fn.backward())                      # backward
        for m in net.layers:                                  # update
            for P, name in m.params():
                P -= lr * getattr(m, name)
    history.append(evaluate(net, X_tr, y_tr) + evaluate(net, X_va, y_va))
seconds = time.perf_counter() - start
history = np.array(history)                # columns: train loss, train acc, val loss, val acc
print(f"{n_params} parameters, {epochs} epochs in {seconds:.2f} s")
print("epoch 0:   train loss %.3f acc %.2f | val loss %.3f acc %.2f" % tuple(history[0]))
print("epoch 1:   train loss %.3f acc %.2f | val loss %.3f acc %.2f" % tuple(history[1]))
print("epoch 200: train loss %.3f acc %.2f | val loss %.3f acc %.2f" % tuple(history[-1]))
print("lowest validation loss %.3f, at epoch %d" % (history[:, 2].min(), history[:, 2].argmin()))
354 parameters, 200 epochs in 0.07 s
epoch 0:   train loss 0.872 acc 0.50 | val loss 0.854 acc 0.50
epoch 1:   train loss 0.481 acc 0.81 | val loss 0.449 acc 0.82
epoch 200: train loss 0.042 acc 0.99 | val loss 0.128 acc 0.96
lowest validation loss 0.071, at epoch 172

The library version: scikit-learn’s MLPClassifier with the same hidden layers, solver, batch size and learning rate, on the same split.

from sklearn.neural_network import MLPClassifier

clf = MLPClassifier(hidden_layer_sizes=(16, 16), solver="sgd", learning_rate_init=0.2,
                    batch_size=32, max_iter=200, random_state=0).fit(X_tr, y_tr)
print("hand-built: validation accuracy %.2f" % history[-1, 3])
print("MLPClassifier: validation accuracy %.2f after %d epochs" % (clf.score(X_va, y_va), clf.n_iter_))
print("what it adds: alpha =", clf.alpha, " momentum =", clf.momentum, " nesterovs_momentum =",
      clf.nesterovs_momentum, " output:", clf.out_activation_)
hand-built: validation accuracy 0.96
MLPClassifier: validation accuracy 0.92 after 62 epochs
what it adds: alpha = 0.0001  momentum = 0.9  nesterovs_momentum = True  output: logistic

Both classifiers get most of the 50 validation points right; with 50 points, one point is 2 percent, so a difference of a few points is within the noise. The library is not plain SGD, though. Its defaults add an L2 penalty on the weights (alpha=0.0001), momentum with Nesterov’s correction (momentum=0.9), and a stopping rule that ends training when the loss has not improved by tol for n_iter_no_change=10 epochs, which is why it stopped early. For two classes it also uses one logistic output unit instead of two softmax units, an equivalent model (Exercise 2 of DL01.5.1). Momentum and penalties are DL02.1 and DL02.6.

Explore

Do one step of gradient descent yourself in the Backprop stepper of DL01.5.1. Run both passes, note the gradient of \({w^{(1)}_{12}}\), 2.924, and set that weight to \({-0.5 - 0.1 \times 2.924} \approx {-0.8}\). The loss falls, but hidden unit 1 is now off: a step of 0.1 on one weight already overshoots for that unit.

Pitfalls

Two panels of the two moons after 200 epochs. Left, marked with a cross and titled All weights equal at the start: the decision boundary is a single straight line across the moons, with training accuracy 84 percent, because every hidden unit computes the same thing. Right, marked with a tick and titled Random weights at the start: the boundary curves between the moons, 99 percent.
DL01.5.4-F5

Wrong beside right. Start every weight at the same value and every hidden unit receives the same gradient, so the units stay identical forever and the network can only draw a straight boundary.

“The starting weights do not matter much; set them all to the same small number.”

If every weight of a layer is equal, every hidden unit computes the same value, receives the same gradient and takes the same step, so they stay equal for ever. Here all units of the first layer share one weight vector \(\vect{w}\) throughout, so the network sees only the single number \({\vect{w}\cdot\vect{x}}\). The units of the second layer stay identical too, so the difference of the two logits only ever rises, or only ever falls, as that number grows. The boundary is therefore a single straight line at right angles to \(\vect{w}\), however long the training runs: 84 percent on the training set, against 99 percent from random weights. Random starting weights break the symmetry; DL02.3 chooses their scale.

“A module can update its weights inside backward, as soon as it has the gradient.”

Linear’s backward returns \({\mat{G}\mat{W}}\), and that product must use the weights of the forward pass. Update \(\mat{W}\) first and the gradient handed to the layer below belongs to a different network. Finish the backward pass for every module, then update every parameter.

“The training loss is falling, so the network is getting better.”

It is getting better at the 150 training points. After about epoch 50 the training loss keeps falling while the validation loss levels off near 0.1 and wanders. The validation curve is the one that measures what the network has learned about moons in general, and it decides when to stop (ML01.8).

Check yourself

Recall

1. List the four steps of one training iteration.

Show answer

Forward the batch, compute the loss, backward to get every gradient, update every parameter by \({-\eta}\) times its gradient.

2. How many parameters does a 2-16-16-2 network have, and how?

Show answer

354: \({(32 + 16) + (256 + 16) + (32 + 2)}\), weights plus biases of the three layers.

3. How many updates make one epoch with 150 training points and batches of 32?

Show answer

Five: four batches of 32 and a last one of 22.

4. To first order, by how much does one gradient step lower the loss?

Show answer

By about \({\eta\norm{\vect{g}}^2}\), the learning rate times the squared length of the gradient.

5. Name two things that MLPClassifier(solver="sgd") adds by default that the hand-built loop does not have.

Show answer

Any two of: an L2 penalty (alpha=0.0001), momentum 0.9 with Nesterov’s correction, and a stopping rule that ends the run when the training loss has not improved by tol for 10 epochs.

Exercises

1. ★ With 1,077 training images and batches of 64, how many updates are in one epoch, and how big is the last batch? How many updates do 30 epochs make?

Show solution

\({1077} = {16 \times 64 + 53}\), so an epoch has 17 updates, the last on 53 images. Thirty epochs make \({30 \times 17} = {510}\) updates.

2. ★★ In the worked example, use the first-order estimate to find the step size that would, if the estimate held, lower the loss from 1.313 to 0.693. Then explain why the true loss after that step need not be 0.693.

Show solution

Solve \({0.693} = {1.313 - \eta \times 16.03}\): \({\eta} = {0.620/16.03} \approx {0.039}\). The estimate uses only the slope at the start. The loss curves, and a step this long also moves \({z^{(1)}_1}\) by about \({-0.039 \times 8.77} \approx {-0.34}\) (the change in \({\vect{w}^{(1)}_1 \cdot \vect{x} + b^{(1)}_1}\)), close to the ReLU’s kink at 0 where the linear picture of each unit changes. The true value has to be computed by a forward pass: it gives about 0.776, well above 0.693.

3. ★★★ Show that if all the weights of the first hidden layer are equal and all its biases are equal, then every unit of that layer gets the same gradient for its weights, so after any number of gradient steps they are still equal. Assume the next layer’s weights start all equal too.

Show solution

With equal rows of \(\mat{W}^{(1)}\) and equal biases, every unit has the same pre-activation \({z_j} = {z}\) and the same activation. Suppose also that each row of \(\mat{W}^{(2)}\) has equal entries, which holds when all its weights are equal. Then the gradient arriving at each unit, \({(\mat{W}^{(2)\top}\vect{\delta}^{(2)})_j} = {\sum_i w^{(2)}_{ij}\delta^{(2)}_i}\), is the same for every \(j\), and so is the mask, so \({\delta^{(1)}_j}\) is the same for all \(j\). The gradient of row \(j\) of \(\mat{W}^{(1)}\) is \({\delta^{(1)}_j\vect{x}\T}\), the same row for every \(j\), and its bias gradient \({\delta^{(1)}_j}\) is the same too. Column \(j\) of \(\mat{W}^{(2)}\) gets \({\vect{\delta}^{(2)}a_j}\), the same for every \(j\), so each row of \(\mat{W}^{(2)}\) keeps equal entries. One update keeps both properties, so by induction they hold after every step. (Different rows of \(\mat{W}^{(2)}\) may drift apart; only the hidden units are tied.)

Summary card

Forward, loss, backward, update: repeat over shuffled batches.

The anchor figure again: the decision boundary on the two moons after 1, 10 and 200 epochs, from a few straight pieces to a curve snaking between the moons.
DL01.5.4-F1

Gradients bend the boundary.

  • Update: \({\theta} \leftarrow {\theta - \eta\,\nabla_\theta L_B}\), after the whole backward pass.
  • One step lowers the loss by about \({\eta\norm{\vect{g}}^2}\).
  • 2-16-16-2: 354 parameters; 150 points in batches of 32 is 5 updates per epoch.
  • Watch the validation curve, not only the training curve.

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • S. J. D. Prince, Understanding Deep Learning (MIT Press, 2023), section 6.2.1: each iteration computes the gradient on a random batch, drawn without replacement, and updates the parameters by equation 6.10 (written with a sum over the batch; our mean only rescales the learning rate); one pass through the training set is an epoch. Section 7.4.1: the derivatives of every example in the batch are combined for the SGD update.
  • scikit-learn 1.9, MLPClassifier: alpha=0.0001 is the strength of the L2 term; momentum=0.9 and nesterovs_momentum=True are used when solver="sgd"; max_iter counts epochs for the stochastic solvers; training stops when the loss does not improve by at least tol for n_iter_no_change=10 consecutive epochs.
  • A. Zhang, Z. C. Lipton, M. Li and A. J. Smola, Dive into Deep Learning, section 5.3.4: backpropagation reuses the intermediate values stored by forward propagation, so they must be kept until backpropagation is complete.

Next. DL01.6 builds an autodiff engine that writes these backward passes for you and repeats this training run step for step; DL01.8 runs the same loop in PyTorch.

This lesson unlocks

DL01.6 Build an autodiff engine from scratch DL01.8 Anatomy of a training loop

Your progress is saved in this browser only. No account, no tracking.
Back to top