Skip to main content

ReLU and its relatives

The rectified linear unit max(0, z): a slope of exactly 1 or 0, the convention at the kink, and why it does not saturate for positive inputs. Then dead units, which never learn again, and leaky ReLU and ELU, which keep a gradient below zero.
DL01.3.2L2 Intermediate25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: DL01.3.1 Sigmoid and tanh

What you will be able to do
  • State the ReLU and its derivative, and the value that libraries use for the derivative at exactly zero.
  • Explain why ReLU does not saturate for positive inputs and why it makes a network piecewise linear.
  • Prove that a dead unit receives no gradient for any of its incoming weights, and recognise the causes.
  • Define leaky ReLU and ELU with their derivatives, and count dead units in a NumPy network as the biases shift.

The question

The hidden activation that Goodfellow et al. call “an excellent default choice” is the plainest imaginable: keep a positive number, replace a negative one by zero. It has a kink where calculus says the derivative does not exist, and it outputs exactly zero for half of all inputs. Why does it train better than the smooth S-curves, and how can one of its units stop learning for good?

From the last lesson

1. What is the largest value of the sigmoid’s derivative, and where is it reached?

Show answer

\(\tfrac14\), at \({z = 0}\), because \({\sigma' = \sigma(1 - \sigma)}\) and \({p(1 - p) \le \tfrac14}\).

2. Why does a saturated unit learn slowly?

Show answer

The gradient for its weights is multiplied by the slope \({\phi'(z)}\), which is close to zero in the flat tails of an S-curve.

Intuition

The rectified linear unit, or ReLU, is \({\max(0, z)}\). Its graph is a hinge: flat at zero on the left, the line \({a = z}\) on the right. Its slope can only be 0 or 1.

Two panels. Left, titled ReLU: a vermillion hinge, flat at zero for z below zero, labelled zero for every z less than 0, then rising as the line a equals z, labelled equals z for z greater than 0, with max of 0 and z written beside it. Right, titled Its slope: a green step, at height 0 for z below 0, labelled gate shut, slope 0, and at height 1 for z above 0, labelled gate open, slope 1. At z equals 0 there is a solid green dot at 0 and a hollow one at 1, with the note at z equals 0 libraries use 0.
DL01.3.2-F1

The hinge and its slope. Backward, the slope acts as a gate: it passes the gradient unchanged or blocks it completely.

Read the slope as a gate on the backward pass. Where \({z > 0}\) the gate is open and the gradient arriving from above passes through untouched: no shrinking, however large \(z\) is. That is the difference from the sigmoid, whose slope is never more than \(\tfrac14\) and fades to nothing as \(z\) grows. Where \({z < 0}\) the gate is shut and nothing passes.

Build it up

Step 1. No saturation on the right. For \({z > 0}\) the slope is exactly 1, so a positive pre-activation never weakens the signal, the property the S-curves lack (DL01.3.1).

Step 2. The kink. At \({z = 0}\) the slope from the left is 0 and from the right is 1, so ReLU has no derivative there. Software does not stop to complain: it returns one of the one-sided values. PyTorch’s rules for non-differentiable functions say that where a function is convex it uses the subgradient of smallest size. Every number between 0 and 1 is a subgradient of ReLU at 0, and the smallest is 0. With real-valued data an exact zero is rare, but with integer inputs it happens, as the worked example shows.

Step 3. Piecewise linear networks. On each side of its kink a ReLU is linear. A network of ReLUs is therefore linear on every region of input space where no unit changes side: the piecewise linear functions and linear regions of DL01.2.4 come from here.

Step 4. The weakness. A shut gate is shut for every weight behind it. If a unit’s pre-activation is negative on every training input, no example ever opens the gate.

Step 5. Keeping a slope. Two fixes give the left side a small slope instead of none.

Two panels. Left, titled The functions: a dashed grey ReLU, a solid vermillion leaky ReLU that continues below zero with a gentle slope, and a dashed vermillion ELU that curves down towards minus 1 for negative z; all three agree with the line a equals z for positive z. Right, titled Their slopes: for z above 0 a green line at 1, labelled slope 1; for z below 0 a solid green line at the small value alpha, labelled leaky slope alpha, and a dashed green curve e to the z rising from near 0 to 1 at z equals 0, labelled ELU slope e to the z. A note says leaky ReLU is drawn with alpha equals 0.1 to make it visible and the usual value is 0.01.
DL01.3.2-F2

Leaky ReLU and ELU. Below zero both keep a slope, so the gate never shuts completely.

The formal statement

NoteReLU, leaky ReLU and ELU

\[ \begin{aligned} \operatorname{relu}(z) &= \max(0,\, z), \\ \operatorname{leaky}(z) &= \max(0,\, z) + \alpha \min(0,\, z), \\ \operatorname{elu}(z) &= \begin{cases} z & z > 0 \\ \alpha\,(e^{z} - 1) & z \le 0 \end{cases} \end{aligned} \]

For \({z \neq 0}\) the slopes are 1 on the right, and 0, \(\alpha\) and \({\alpha e^{z}}\) on the left. Leaky ReLU usually takes \({\alpha = 0.01}\); ELU usually takes \({\alpha = 1}\). At \({z = 0}\) the ReLU’s slope is taken to be 0.

For ELU with \({z \le 0}\) the slope can also be written \({\alpha e^{z} = \operatorname{elu}(z) + \alpha}\), the form in Clevert, Unterthiner and Hochreiter’s paper, so the backward pass can reuse the forward output. With \({\alpha = 1}\) the slope tends to 1 from both sides at 0, so ELU’s slope has no jump. A slope of \({\alpha = 0.01}\) is also the default of PyTorch’s leaky ReLU.

Dead units. Call unit \(j\) of the first hidden layer dead on a training set if \({z_j(\vect{x}_i) \le 0}\) for every training input \({\vect{x}_i}\).

NoteA dead unit gets no gradient

If unit \(j\) is dead, the gradient of the training loss with respect to its weights \({\vect{w}_j}\) and its bias \({b_j}\) is exactly zero, so gradient descent never changes them.

Proof. The training loss is a mean over examples, \({L = \frac1n \sum_i L_i}\). For one example, the unit computes \({z_j = \vect{w}_j \cdot \vect{x}_i + b_j}\) and \({a_j = \operatorname{relu}(z_j)}\), and the chain rule gives

\[ \begin{aligned} \pd{L_i}{\vect{w}_j} &= \pd{L_i}{a_j}\,\operatorname{relu}'(z_j)\,\vect{x}_i , \\ \pd{L_i}{b_j} &= \pd{L_i}{a_j}\,\operatorname{relu}'(z_j) . \end{aligned} \]

Since \({z_j \le 0}\), the slope is 0 (with the library’s value at 0), so both are zero for every example, and so is their mean. The update \({\vect{w}_j \leftarrow \vect{w}_j - \eta\,\vect{0}}\) leaves \({\vect{w}_j}\) unchanged, the inputs \({\vect{x}_i}\) are fixed data, so every \({z_j(\vect{x}_i)}\) stays the same and the unit is dead at the next step too. By induction it is dead for ever. \(\square\)

In a deeper layer the inputs are the previous layer’s activations, which keep changing, so a dead unit there can come back if the layers below move its inputs. It cannot help itself, though: the argument above still shows that its own weights get no gradient while it is dead. Units die from a bias or weights that start badly, or later from one large update, which is why CS231n warns that a learning rate set too high can leave much of a network dead. Leaky ReLU and ELU have no dead state: their slope below zero is never zero.

Worked example

One unit of the XOR network. The hand-built network of DL01.2.1 has a second hidden unit \({a = \max(0,\, x_1 + x_2 - 1)}\). Suppose the layer above sends back \({\partial L/\partial a = 1}\) for each of the four XOR points, and add up their gradients.

Two tables side by side. Left, marked with a tick and titled Bias minus 1: alive. Rows for the inputs (0, 0), (0, 1), (1, 0), (1, 1) give z equals minus 1, 0, 0, 1, a equals 0, 0, 0, 1 and slope 0, 0, 0, 1; the last row is highlighted yellow. Below it, in green: the gradient for w is (1, 1) and for b is 1. Right, marked with a cross and titled Bias minus 3: dead. The same inputs give z equals minus 3, minus 2, minus 2, minus 1, a and slope all 0, and the gradients for w and b are (0, 0) and 0.
DL01.3.2-F3

The same unit with two biases. Only rows with \({z > 0}\) (yellow) pass a gradient. With bias \(-3\) none do.

Bias \(-1\). The pre-activations are \({z = -1, 0, 0, 1}\). Two of them are exactly 0, so the convention matters: their slope is taken as 0. Only \({(1, 1)}\) passes a gradient, and

\[ \begin{aligned} \pd{L}{\vect{w}} &= \textstyle\sum_i \operatorname{relu}'(z_i)\,\vect{x}_i = (1,\ 1), \\ \pd{L}{b} &= \textstyle\sum_i \operatorname{relu}'(z_i) = 1 . \end{aligned} \]

Had the slope at 0 been taken as 1, the points \({(0, 1)}\) and \({(1, 0)}\) would add \({(1, 1)}\) more, giving \({(2, 2)}\) and 3. Exact ties are rare with measured data, but integer data like this make them common, so it is worth knowing which value your library uses.

Bias \(-3\). Now \({z = -3, -2, -2, -1}\), all negative. Every slope is 0, the gradients are \({(0, 0)}\) and 0, and the update leaves the unit exactly where it was. It is dead on these four points, and the proof above says it stays dead.

Code

From scratch: the three functions and their slopes, each slope checked by central differences away from the kink.

import numpy as np

def relu(z):        return np.maximum(z, 0.0)
def drelu(z):       return (z > 0).astype(float)            # 0 at z = 0, as libraries do
def leaky(z, a=0.01):   return np.where(z > 0, z, a * z)
def dleaky(z, a=0.01):  return np.where(z > 0, 1.0, a)
def elu(z, a=1.0):  return np.where(z > 0, z, a * np.expm1(np.minimum(z, 0)))
def delu(z, a=1.0): return np.where(z > 0, 1.0, elu(z, a) + a)  # the paper's form

z = np.linspace(-4, 4, 80)                # an even count: no grid point at the kink
h = 1e-6
for f, df in ((relu, drelu), (leaky, dleaky), (elu, delu)):
    err = np.max(np.abs((f(z + h) - f(z - h)) / (2 * h) - df(z)))
    print(f"{f.__name__:6s} largest slope error {err:.1e}")
print("slopes at 0:", drelu(np.array(0.0)), dleaky(np.array(0.0)), delu(np.array(0.0)))
relu   largest slope error 1.4e-10
leaky  largest slope error 1.4e-10
elu    largest slope error 1.4e-10
slopes at 0: 0.0 0.01 1.0

The worked example in three lines:

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
for b in (-1.0, -3.0):
    s = drelu(X @ np.array([1.0, 1.0]) + b)       # dL/da = 1 for every example
    print(f"bias {b}: dL/dw = {s @ X}, dL/db = {s.sum()}")
bias -1.0: dL/dw = [1. 1.], dL/db = 1.0
bias -3.0: dL/dw = [0. 0.], dL/db = 0.0

Now a layer of 100 ReLU units with random weights, fed the 200 points of the course’s make_moons set. Give every unit the same bias and count the units that are dead on all 200 points.

from sklearn.datasets import make_moons

Xm, ym = make_moons(n_samples=200, noise=0.2, random_state=0)
W = np.random.default_rng(0).normal(size=(100, 2))
for shift in (-3, -2, -1, -0.5, 0):
    dead = np.mean(np.all(Xm @ W.T + shift <= 0, axis=0))
    print(f"bias {shift:5}: {dead:.0%} of units dead")
bias    -3: 85% of units dead
bias    -2: 60% of units dead
bias    -1: 28% of units dead
bias  -0.5: 3% of units dead
bias     0: 0% of units dead
A step-shaped vermillion curve of the share of dead units, from 0 to 100 percent, against the bias given to every unit, from minus 4 to 1. It is 95 percent at minus 4 and falls steadily, through dots marked 85 percent at minus 3, 60 percent at minus 2 and 28 percent at minus 1, reaching 0 percent shortly before a bias of 0 and staying there.
DL01.3.2-F5

The share of dead units in the random layer against the bias. With zero biases every unit’s hinge line passes through the origin, inside the data, and none is dead.

Last, the claim of the proof, seen in training. A 2-8-1 network (ReLU hidden layer, sigmoid output, mean log loss) is trained by gradient descent on make_moons with three of its eight units started with bias \(-4\). Their weights are compared before and after, with ReLU and with leaky ReLU.

from scipy.special import expit

def train(act, dact, steps=2000, lr=0.5):
    rng = np.random.default_rng(1)
    W1, b1 = rng.normal(0, 1 / np.sqrt(2), size=(8, 2)), np.zeros(8)
    W2, b2 = rng.normal(0, 1 / np.sqrt(8), size=(1, 8)), np.zeros(1)
    b1[:3] = -4.0                                       # three units start dead
    W1_start, dead_start = W1.copy(), np.all(Xm @ W1[:3].T + b1[:3] <= 0)
    for _ in range(steps):
        Z1 = Xm @ W1.T + b1; A1 = act(Z1)
        p = expit(A1 @ W2.T + b2)
        D2 = (p - ym[:, None]) / len(ym)                # dL/dz at the output (M05.9.2)
        D1 = (D2 @ W2) * dact(Z1)                       # through the activation's slope
        W2 -= lr * D2.T @ A1; b2 -= lr * D2.sum(0)
        W1 -= lr * D1.T @ Xm; b1 -= lr * D1.sum(0)
    return dead_start, np.abs(W1 - W1_start).sum(axis=1)

dead_start, moved = train(relu, drelu)
print("ReLU : dead at start", dead_start, "| weight change of units 0-2:", moved[:3])
_, moved_leaky = train(leaky, dleaky)
print("leaky: weight change of units 0-2:", np.round(moved_leaky[:3], 4))
ReLU : dead at start True | weight change of units 0-2: [0. 0. 0.]
leaky: weight change of units 0-2: [0.0001 0.006  0.0166]

With ReLU the three units have not moved by a single bit in 2,000 steps. With leaky ReLU they move, slowly, because their slope below zero is 0.01: alive, but with a weak gradient.

Explore

The module’s interactive, the activation explorer, lives in lesson DL01.3.4. Choose ReLU, set the spread to 1 and slide the mean of the pre-activations down to \(-2\): 29 of the 30 units are dead. Switch to leaky ReLU and none is, though their slope is only 0.01.

Pitfalls

Two panels of the make_moons points, dark blue for one class and sky blue for the other. Each shows the vermillion hinge line where a ReLU unit with weights 0.8 and 0.6 has z equal to 0, and tints the side where z is positive. Left, marked with a cross and titled Bias minus 4: dead from the start: the line cuts only the top right corner, far from every point, and the label reads z greater than 0 for 0 of 200 points. Right, marked with a tick and titled Bias 0.1: alive: the line runs diagonally through the left end of the data, most of the plane is tinted, and the label reads z greater than 0 for 171 of 200 points.
DL01.3.2-F4

Wrong beside right. The bias moves the hinge line. At \(-4\) it lies outside the data and the unit is dead before training starts; at \(0.1\) it cuts through the data.

“A unit that outputs zero now will wake up later.”

Not in the first layer. A unit that is negative on every training input gets exactly zero gradient, so nothing ever changes it. Start biases at zero or a small positive value (Goodfellow et al. suggest 0.1), keep the learning rate moderate, and check the share of dead units when training stalls.

“ReLU is linear on each side, so a ReLU network is a linear model.”

Each unit is linear on each side of its own kink, but different inputs put different units on different sides. The network is piecewise linear, with a different linear map on each region, and those regions are what let it bend a decision boundary round the moons (DL01.2.4).

“The kink breaks gradient descent.”

Gradient descent never needs the exact derivative at one point. Away from \({z = 0}\) the slope is defined, and at 0 libraries use a one-sided value, 0 for ReLU. Goodfellow et al. note that training does not usually reach a point where the gradient is exactly zero anyway.

Check yourself

Recall

1. What is the derivative of ReLU, and what value do libraries use at \({z = 0}\)?

Show answer

0 for \({z < 0}\) and 1 for \({z > 0}\). At \({z = 0}\) libraries use 0, the subgradient of smallest size.

2. Why does ReLU not saturate for positive inputs?

Show answer

Its slope is exactly 1 for every \({z > 0}\), so the gradient passes through unchanged however large \(z\) is.

3. When is a first-layer ReLU unit dead, and why does it stay dead?

Show answer

When its pre-activation is at most 0 on every training input. Then the gradients for its weights and bias are all zero, so gradient descent never changes them, and the inputs are fixed data.

4. Write leaky ReLU and its slope.

Show answer

\({\max(0, z) + \alpha\min(0, z)}\), usually with \({\alpha = 0.01}\). Its slope is 1 for \({z > 0}\) and \(\alpha\) for \({z < 0}\).

5. For ELU with \({\alpha = 1}\), what is the slope for \({z \le 0}\), and in terms of the output?

Show answer

\({e^{z}}\), which equals \({\operatorname{elu}(z) + 1}\).

Exercises

1. ★ The first XOR hidden unit is \({a = \max(0,\, x_1 + x_2)}\). With \({\partial L/\partial a = 1}\) for each of the four XOR points, find \({\partial L/\partial \vect{w}}\) and \({\partial L/\partial b}\) summed over the points.

Show solution

The pre-activations are \({0, 1, 1, 2}\). The first is exactly 0, so its slope is 0. The other three pass: \(\partial L/\partial \vect{w} = {(0, 1) + (1, 0) + (1, 1)} = (2, 2)\) and \({\partial L/\partial b = 3}\).

2. ★★ Show that \({z = \operatorname{relu}(z) - \operatorname{relu}(-z)}\) and \({\abs{z} = \operatorname{relu}(z) + \operatorname{relu}(-z)}\), and write leaky ReLU as a combination of \(z\) and \({\operatorname{relu}(z)}\).

Show solution

For \({z \ge 0}\): \({\operatorname{relu}(z) = z}\) and \({\operatorname{relu}(-z) = 0}\), so the difference is \(z\) and the sum is \({z = \abs{z}}\). For \({z < 0}\): \({\operatorname{relu}(z) = 0}\) and \({\operatorname{relu}(-z) = -z}\), so the difference is \(z\) and the sum is \({-z = \abs{z}}\). For leaky ReLU, \({\min(0, z) = z - \max(0, z)}\), so

\[ \begin{aligned} \operatorname{leaky}(z) &= \max(0, z) \\ &\quad + \alpha\big(z - \max(0, z)\big) \\ &= \alpha z + (1 - \alpha)\operatorname{relu}(z) . \end{aligned} \]

3. ★★★ A ReLU unit in the first layer has weights \({\vect{w} = (0.8, 0.6)}\), a unit vector. Every training input satisfies \({\norm{\vect{x}} \le R}\). Show that the unit is dead if \({b \le -R}\). For the make_moons points \({R \approx 2.26}\); why is the unit of the pitfall figure with \({b = -4}\) certainly dead?

Show solution

By the Cauchy-Schwarz inequality, \({\vect{w} \cdot \vect{x} \le \norm{\vect{w}}\,\norm{\vect{x}} = \norm{\vect{x}} \le R}\). So \({z = \vect{w} \cdot \vect{x} + b \le R + b \le 0}\) for every input, and the unit is dead. With \({R \approx 2.26}\) the bound holds for any \({b \le -2.26}\), and \({-4}\) is well below it. A weaker bias can still kill a unit: the bound is only sufficient, and the code above finds 28% of random units dead already at \({b = -1}\).

Summary card

ReLU’s slope is a gate: open with slope 1, or shut.

The anchor figure again: the vermillion ReLU hinge on the left, and on the right its green slope, 0 for negative z and 1 for positive z, with 0 used at z equals 0.
DL01.3.2-F1

Open for \({z > 0}\), shut for \({z < 0}\).

  • \({\operatorname{relu}(z) = \max(0, z)}\), slope 1 or 0; libraries use 0 at the kink.
  • No saturation for \({z > 0}\); the network is piecewise linear.
  • Dead unit: \({z \le 0}\) on every input, so every incoming gradient is 0.
  • Leaky ReLU (slope \({\alpha = 0.01}\)) and ELU (slope \({e^{z}}\)) never shut the gate.

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, chapter 6, MIT Press, 2016. Section 6.3 calls rectified linear units an excellent default choice of hidden unit, notes that ReLU has no derivative at 0, that software returns a one-sided derivative, and that training does not usually reach a zero gradient; section 6.3.1 gives the leaky ReLU slope of 0.01 (after Maas et al., 2013), the general form \({\max(0, z) + \alpha\min(0, z)}\) and the advice to start biases at a small positive value such as 0.1.
  • Simon J. D. Prince, Understanding Deep Learning, MIT Press, 2023. The notes to chapter 3 describe the dying ReLU problem, where every training example gives a unit a negative input and its incoming parameters cannot improve; figure 3.13 draws the variants (with a leaky slope of 0.1).
  • PyTorch, Autograd mechanics, section “Gradients for non-differentiable functions”: ReLU at 0 is the example, and a convex function gets the subgradient of minimum norm.
  • Djork-Arné Clevert, Thomas Unterthiner and Sepp Hochreiter, Fast and Accurate Deep Network Learning by Exponential Linear Units, ICLR 2016. Section 3, equation 15, defines the ELU and its derivative, \({f(x) + \alpha}\) for \({x \le 0}\).
  • Stanford CS231n, Neural Networks Part 1: ReLU units can die irreversibly, as much as 40% of a network if the learning rate is too high, and leaky ReLU uses a slope of about 0.01.

Next. DL01.3.3 Smooth gates rounds off the kink and replaces the hard gate by a soft one.

This lesson unlocks

DL01.3.3 Smooth gates: softplus, GELU and SiLU

Your progress is saved in this browser only. No account, no tracking.
Back to top