\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

DL01.2 · Multilayer perceptrons and universal approximation

A free, illustrated module of 4 lessons and one interactive in DL01 Neural Network Foundations: A hidden layer solves XOR; The multilayer perceptron and its forward pass; Universal approximation; Depth folds the input space.

DL01.2 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of DL01 Neural Network Foundations in the Deep Learning track.

A single neuron draws one straight line; a hidden layer lets a network draw its own features, and that is enough to solve XOR. These four lessons work the hand-built XOR network on every point, stack layers into a multilayer perceptron and run its forward pass by hand and for a batch, build one hidden layer of ReLUs into any curve hinge by hinge, and show why depth, which folds the input space, makes exponentially many pieces from a few units.

What you need first. The neuron, the layer and the failure of one neuron on XOR (DL01.1), the 2-2-1 tanh network trained on XOR in M05.9, and matrix products with examples as rows (M02.5). Some comfort with slopes of straight lines and with the second derivative (M04) helps in the third lesson.

Lessons

1. A hidden layer solves XOR DL01.2.1 25 min

   No straight line separates the four XOR points, yet a network with two hidden ReLU units classifies them exactly. Work the hand-built network on every point, see the hidden layer move the points until one line works, and compare the features a trained tanh network finds.

2. The multilayer perceptron and its forward pass DL01.2.2 25 min

   Stack layers into a multilayer perceptron: each one a matrix product, a bias and an activation. Count its parameters, run its forward pass by hand and for a whole batch, then reproduce scikit-learn’s MLPClassifier from its fitted weights.

3. Universal approximation DL01.2.3 25 min

   One hidden layer of ReLUs draws a curve out of hinges, one bend per unit, so it can follow any continuous curve as closely as you like. Build the network for sin(2 pi x) weight by weight, measure its error, and learn what the universal approximation theorem does not promise.

4. Depth folds the input space DL01.2.4 25 min

   Two ReLUs fold a line in half, and a stack of such folds makes a zigzag whose pieces double with every layer, while one hidden layer gains only one piece per unit. Count linear pieces and regions, and see why the maximum grows with depth while a random network uses little of it.

The interactive

DL01.2-X1 · Hinges and folds. On the left, drag up to a dozen knots to build a one-hidden-layer ReLU network as a sum of hinges and watch it approach sin(2 pi x), with the largest error printed. On the right, a depth slider composes the tent map and counts the linear pieces, 2^k from 2k units, against a one-layer network with the same units. It lives in Depth folds the input space.