\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

DL01.4 · Loss functions and maximum likelihood

A free, illustrated module of 4 lessons and one interactive in DL01 Neural Network Foundations: Regression losses as noise models; Binary cross-entropy from the Bernoulli; Softmax and categorical cross-entropy; The loss surface of a network.

DL01.4 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of DL01 Neural Network Foundations in the Deep Learning track.

A loss function is not a matter of taste. Let the network output the parameters of a probability distribution over the target and ask how probable it made the data: minus the log of that probability is the loss. These four lessons derive the mean squared error, the mean absolute error, binary cross-entropy and categorical cross-entropy from one recipe, show how libraries compute them without overflow, explain why squared error after a sigmoid stops learning when the model is confidently wrong, and then look at the loss of a whole network as a surface with many equal valleys.

What you need first. The neuron and its activations (DL01.1, DL01.3), linear and logistic regression with the log loss (ML01.4, ML01.6), gradient descent (ML01.5), the softmax gradient p - y (M05.7.3) and the 2-2-1 XOR network trained by backpropagation (M05.9).

Lessons

1. Regression losses as noise models DL01.4.1 25 min

   A regression network does not predict a number: it predicts the centre of a bell of possible targets. Minus the log of that bell is the loss. A Gaussian bell gives the mean squared error, a Laplace bell the mean absolute error, and Huber’s loss sits between them.

2. Binary cross-entropy from the Bernoulli DL01.4.2 25 min

   A sigmoid output is the bias of a coin. Its negative log-likelihood is the log loss, which in terms of the logit is softplus(-z) or softplus(z): the form libraries compute, because it never takes the log of zero. Its gradient is p - y, and it does not vanish when the model is confidently wrong.

3. Softmax and categorical cross-entropy DL01.4.3 25 min

   For K classes the network outputs K logits and the softmax turns them into a categorical distribution. The negative log-likelihood of the true class is minus its logit plus log-sum-exp, computed safely by subtracting the largest logit first. Its gradient is p - y.

4. The loss surface of a network DL01.4.4 25 min

   Swap two hidden units, or flip the signs of a tanh unit’s weights, and the network computes the same function from different weights. So every minimum has copies, and the loss of a network cannot be convex. Slices along a line and over a plane make the surface visible.

The interactive

DL01.4-X1 · One-example loss explorer. Set the logit of one example, or three logits in softmax mode, and choose the true class. See the predicted probabilities, the cross-entropy and its gradient p - y beside the squared error of the same probabilities, whose gradient vanishes when the model is confidently wrong, and take one gradient step with each loss to see how far it moves. It lives in Binary cross-entropy from the Bernoulli.