\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

DL01.1 · From linear models to neurons

A free, illustrated module of 4 lessons and one interactive in DL01 Neural Network Foundations: A neuron is a linear model with a squashing step; The perceptron and its learning rule; A layer of neurons; What one layer cannot do.

DL01.1 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of DL01 Neural Network Foundations in the Deep Learning track.

Every neural network is built from one small unit: a weighted sum of its inputs plus a bias, passed through an activation. These four lessons show that you have met this neuron before, as linear and logistic regression, train it with Rosenblatt’s perceptron rule and bound the number of mistakes that rule can make, put many neurons side by side as a layer written with one matrix, and prove what a single layer cannot do: classify XOR.

What you need first. Linear regression, gradient descent and logistic regression with the log loss (ML01.4 to ML01.6), and matrix multiplication (M02.5). The chain rule is used once, in the first lesson.

Lessons

1. A neuron is a linear model with a squashing step DL01.1.1 25 min

   An artificial neuron takes a weighted sum of its inputs, adds a bias and passes the result through an activation. With the identity it is linear regression; with the sigmoid it is logistic regression. One training loop fits both.

2. The perceptron and its learning rule DL01.1.2 25 min

   Rosenblatt’s perceptron is a neuron with a step activation that learns only from its mistakes. Work the rule by hand on the AND points, see why each update helps, bound the number of mistakes with Novikoff’s theorem, and watch the rule fail on XOR.

3. A layer of neurons DL01.1.3 25 min

   Put several neurons side by side on the same input and their weight vectors stack into the rows of a matrix. A layer computes phi(Wx + b) for one example and phi(XW^T + 1b^T) for a batch. Learn the shapes, count the parameters and read the outputs as new features.

4. What one layer cannot do DL01.1.4 25 min

   A single neuron’s decision boundary is a straight line, so no neuron can classify XOR: four inequalities prove it. Stacking linear layers does not help, because they collapse into one. A hand-made feature fixes XOR and raises the question of learned features.

The interactive

DL01.1-X1 · Neuron playground. Set the two weights and the bias of a neuron, choose a step, sigmoid or ReLU activation, and see the plane coloured by its output over the AND, OR or XOR points. Apply the perceptron rule one mistake at a time, or train whole epochs, and the read-out says when no line can succeed. It lives in The perceptron and its learning rule.