\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

DL01.3 · Activation functions

A free, illustrated module of 4 lessons and one interactive in DL01 Neural Network Foundations: Sigmoid and tanh; ReLU and its relatives; Smooth gates: softplus, GELU and SiLU; Choosing an activation.

DL01.3 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of DL01 Neural Network Foundations in the Deep Learning track.

The activation is the one nonlinear step in a network, and its slope is the one place it enters the backward pass. These four lessons derive the S-curves and their slopes and show how they saturate, follow ReLU’s gate that is either open or shut and the dead units it can leave behind, round the corner off with softplus, SiLU and GELU, and then race them all in one network on make_moons, with the rule that the output activation is fixed by the target.

What you need first. A neuron and a layer (DL01.1), the multilayer perceptron and its forward pass (DL01.2), logistic regression and the sigmoid (ML01.6), and backpropagation on a small network (M05.9), whose batch backward pass lesson DL01.3.4 reuses.

Lessons

1. Sigmoid and tanh DL01.3.1 25 min

   The two S-shaped activations, their derivatives derived line by line, and the identity that makes tanh a rescaled sigmoid. Then saturation: why a unit whose input lands in a flat tail passes almost no gradient back.

2. ReLU and its relatives DL01.3.2 25 min

   The rectified linear unit max(0, z): a slope of exactly 1 or 0, the convention at the kink, and why it does not saturate for positive inputs. Then dead units, which never learn again, and leaky ReLU and ELU, which keep a gradient below zero.

3. Smooth gates: softplus, GELU and SiLU DL01.3.3 25 min

   Softplus is a ReLU with the kink rounded off, and its derivative is the sigmoid. SiLU and GELU multiply z by a soft gate instead of a hard one, dip just below zero, and are the activations of modern transformers. Each derived, minimised and coded.

4. Choosing an activation DL01.3.4 25 min

   The output activation is fixed by the target; the hidden one is a choice. Race sigmoid, tanh, ReLU and GELU in the same network on make_moons, and derive why the backward signal through a stack of sigmoid layers shrinks by at least a factor of 4 per layer.

The interactive

DL01.3-X1 · Activation explorer. Choose one of seven activations to see it and its slope, then set the mean and spread of thirty pre-activations: the read-out counts how many units are saturated or dead and gives the mean slope the layer passes back. It lives in Choosing an activation.