Take the slope at every input and you get a new function, the derivative. Derive it for x squared and 1/x, read its sign and size off a graph and sketch it, and see where it fails: at the corners of the absolute value and of ReLU, which are continuous but have no slope there.
Derive the derivative functions of x squared and 1/x from the definition.
Read the sign and the size of f’ off the graph of f, and sketch the graph of f’ from it.
Show from one-sided difference quotients that the absolute value and ReLU have no derivative at 0.
Prove that a function differentiable at a point is continuous there, and give a continuous function that is not differentiable.
In review. The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.
The question
ReLU, \({\max(0, z)}\), sits inside a great many neural networks, and networks learn by following slopes. Yet at \({z = 0}\) its graph has a sharp corner. What is its slope there? And for any function, how can we tell where a slope exists at all?
From the last lesson
1. Write the tangent line to the graph of \(f\) at \({x = a}\).
Show answer
\({y = f(a) + f'(a)\,(x - a)}\).
2. Estimate \({\sqrt{4.1}}\) with the tangent to \({\sqrt{x}}\) at 4.
Show answer
\({2 + \tfrac{1}{4} \times 0.1 = 2.025}\); the truth is 2.02485.
Intuition
Lessons 1 and 2 found the slope of a curve at one chosen point. Nothing stops us repeating that at every point. The picture does it for the running curve \({y = x^2}\): at \({x = -1.5}\) the tangent has slope \({-3}\), at 0 it is flat, at 1.5 it has slope 3.
M04.2.3-F1
The slope of \({x^2}\) at each \(x\) (top), drawn as a height above the same \(x\) (bottom). The dots fall on one straight line, \({y = 2x}\): the graph of the derivative.
Plot each slope as a height above its own \(x\) and the dots line up on the straight line \({y = 2x}\). That line is a new function, made out of \(f\): its derivative, written \({f'}\). The two graphs are read one from the other. The height of the lower graph at \(x\) is the slope of the upper graph at \(x\).
Build it up
Step 1. The slope at any point. For \({x^2}\) at a point \(a\), the secant slope is
which tends to \({2a}\). So the slope at \(a\) is \({2a}\), whatever \(a\) is.
Step 2. A new function. Since \(a\) can be any number, call it \(x\): the slope of \({x^2}\) at \(x\) is \({2x}\). We write \({f'(x) = 2x}\), read “f prime of x”. Another common name is \({df/dx}\), which keeps the memory of a rise \({df}\) over a run \({dx}\); both mean the same function.
Step 3. Reading signs. At each point, look at the tilt of the tangent. If it tilts up, going left to right, then \({f'(x) > 0}\); if it tilts down, \({f'(x) < 0}\); if it is flat, \({f'(x) = 0}\). A steeper tangent means a larger size \({\abs{f'(x)}}\). So where a graph climbs, its derivative is positive, and where it falls, negative. (It can still be 0 at isolated points: \({x^3}\) climbs everywhere, yet its tangent at 0 is flat, as lesson 2 showed.)
Step 4. A sketch from a picture. The running cubic \({g(x) = x^3 - 3x}\) climbs to a hill at \({x = -1}\), falls to a valley at \({x = 1}\), then climbs again. So \({g'}\) is positive to the left of \({-1}\), zero at \({-1}\), negative between \({-1}\) and 1, zero at 1 and positive after. The fall is steepest midway between the hill and the valley, at \({x = 0}\), so \({g'}\) is lowest there.
M04.2.3-F2
Reading \({g'}\) off \({g}\). Rising, flat, falling, flat, rising becomes positive, zero, negative, zero, positive. Exercise 2 derives the formula \({g'(x) = 3x^2 - 3}\) and confirms the sketch.
The formal statement
NoteDefinition: the derivative function
The derivative of a function \(f\) is the function \({f'}\) whose value at \(x\) is
at every \(x\) where this limit exists. It is also written \({df/dx}\) or \({\tfrac{d}{dx} f(x)}\). A function is differentiable on an open interval if it is differentiable at every point of it.
The definition is the derivative at a point from lesson 1, asked at every point at once. Two kinds of point drop out of the domain of \({f'}\): points where \(f\) itself is not defined, such as 0 for \({1/x}\), and points where \(f\) is defined but the limit does not exist.
Differentiable means continuous. If \(f\) is differentiable at \(a\), then \(f\) is continuous at \(a\). Proof. For \({h \neq 0}\) write the change in \(f\) as the difference quotient times \(h\):
As \(h\) tends to 0 the first factor tends to \({f'(a)}\), because \(f\) is differentiable at \(a\), and the second tends to 0. The limit of a product is the product of the limits (M04.1.2), so \({f(a+h) - f(a)}\) tends to \({f'(a) \cdot 0 = 0}\). That says \({f(a+h)}\) tends to \({f(a)}\), which is what continuity at \(a\) means (M04.1.4).
But continuous does not mean differentiable. The absolute value \({\abs{x}}\) is continuous at 0: \({\abs{h}}\) tends to \({0 = \abs{0}}\). Its difference quotient at 0 is \({\abs{h}/h}\), which depends only on the sign of \(h\):
The limit from the right is 1 and from the left is \({-1}\). They disagree, so the limit does not exist (M04.1.3) and \({\abs{x}}\) has no derivative at 0. Everywhere else it does: slope \({-1}\) for \({x < 0}\) and 1 for \({x > 0}\). ReLU fails at 0 the same way. For \({h > 0}\) its quotient is \({\max(0, h)/h = 1}\), and for \({h < 0}\) it is \({0/h = 0}\): one-sided slopes 1 and 0.
M04.2.3-F4
ReLU and its derivative. The slope is 0 on the left and 1 on the right, so the derivative is a step that jumps at 0, where it has no value.
So the derivative of ReLU is a step: 0 for \({z < 0}\), 1 for \({z > 0}\), and undefined at 0. Training code cannot stop at an undefined slope. The usual practice, which Deep Learning describes in its Section 6.3, is to return one of the one-sided slopes at 0 instead of an error, since an input that comes out as exactly 0 was most likely a tiny number rounded to 0. Compare the step function of M04.1.4: its slope is 0 wherever it has one, so it gives a learner nothing to follow.
Go deeper: a third way to fail: a vertical tangent
The cube root \({x^{1/3}}\) is continuous and has no corner at 0, yet it has no derivative there. Its quotient at 0 is \({h^{1/3}/h = 1/h^{2/3}}\), which grows without bound as \(h\) shrinks from either side. The secants stand up ever more steeply, and the tangent at 0 is the vertical line \({x = 0}\), whose slope is not a number. So a derivative can fail at a corner, at a vertical tangent, or wherever \(f\) is not continuous.
Worked example
Find the derivative of \({f(x) = 1/x}\). Take \({x \neq 0}\), and a step \(h\) small enough that \({x + h}\) is not 0 either. Put the two fractions over a common denominator, then divide by \(h\):
As \(h\) tends to 0 the bottom \({x\,(x+h)}\) tends to \({x^2}\), which is not 0, so
\[
f'(x) = -\frac{1}{x^2} \qquad (x \neq 0) .
\]
Check with numbers at \({x = 2}\). The last line of the derivation gives \({-1/(2 \times 2.1) \approx -0.2381}\) for \({h = 0.1}\) and \({-1/(2 \times 2.01) \approx -0.2488}\) for \({h = 0.01}\), closing in on \({f'(2) = -1/4}\). At \({x = 1/2}\) the slope is \({-4}\), and at \({x = -1}\) it is \({-1}\).
M04.2.3-F3
The worked example as a picture. Each tangent slope of \({1/x}\) (top) is a height on \({y = -1/x^2}\) (bottom), which is negative everywhere and has no value at 0.
Read the result against the graph. On each side of 0, \({1/x}\) falls as \(x\) grows, and \({-1/x^2}\) is negative for every \({x \neq 0}\): the signs agree. Near 0 the graph is steep, and \({f'(1/2) = -4}\) is large; far out it is nearly flat, and \({f'(2) = -1/4}\) is small. The left branch is the right one turned half a turn about the origin, a turn that keeps every slope, so \({f'(-x) = f'(x)}\). At 0 neither \(f\) nor \({f'}\) exists.
Code
From scratch, the derivative as a function: a function that takes \(f\) and returns its slope function, using a short secant with a fixed small step.
import numpy as npdef derivative(f, h=1e-6):"""The slope function of f, estimated by a short secant at every x."""returnlambda x: (f(x + h) - f(x)) / hsquare =lambda x: x * xreciprocal =lambda x: 1/ xxs = np.array([-2.0, -1.0, -0.5, 0.5, 1.0, 2.0])print(" x (x^2)' 2x (1/x)' -1/x^2")for x in xs:print(f"{x:4.1f}{derivative(square)(x):7.4f}{2* x:5.1f}"f" {derivative(reciprocal)(x):7.4f}{-1/ x**2:6.2f}")
The estimates match \({2x}\) and \({-1/x^2}\) to the four decimals shown; lesson 4 explains the small difference that remains further out. Now the corners. NumPy’s abs and maximum give the absolute value and ReLU, and we take one-sided quotients at 0 from each side.
relu =lambda z: np.maximum(0.0, z)print(" |x| ReLU")print(" h right left right left")for h in [0.1, 1e-3, 1e-6]: right_abs, left_abs =abs(h) / h, abs(-h) / (-h) right_relu, left_relu = relu(h) / h, relu(-h) / (-h) +0.0print(f" {h:<7}{right_abs:5.0f}{left_abs:5.0f}{right_relu:5.0f}{left_relu:5.0f}")
|x| ReLU
h right left right left
0.1 1 -1 1 0
0.001 1 -1 1 0
1e-06 1 -1 1 0
However small the step, the right and left slopes stay 1 and \({-1}\) for \({\abs{x}}\), and 1 and 0 for ReLU. No shrinking brings them together, which is what “no derivative at 0” means.
Explore
This module’s interactive, the secant slider, lives in lesson M04.2.1. Choose \({\abs{x}}\) at 0, set any \(h\), and press Flip side: the secant slope jumps between 1 and \({-1}\) and never settles. Press Zoom in a few times: unlike \({x^2}\), the corner looks the same at every zoom, so there is no single line for it to become.
Pitfalls
M04.2.3-F5
Wrong beside right. At the corner of \({\abs{x}}\) the secants from each side keep their own slopes, 1 and \({-1}\). For \({x^2}\) the secants from both sides flatten towards one tangent, slope 0.
“A continuous function has a slope everywhere.”
Continuity only rules out breaks in the graph. \({\abs{x}}\) and ReLU are continuous at 0, yet each has two different one-sided slopes there, so neither has a derivative at 0. The implication runs one way only: differentiable at a point implies continuous there.
“Where \(f\) is high, \({f'}\) is high.”
\({f'}\) measures tilt, not height. At the running cubic’s hill, \({x = -1}\), the value \({g(-1) = 2}\) is as high as the graph gets nearby, yet \({g'(-1) = 0}\), because the tangent there is flat. Read a slope off a graph from the tangent, never from the height.
“ReLU has no slope at 0, so networks built from it cannot be trained by following slopes.”
The missing slope is at one point only. Training code usually returns one of the one-sided slopes, 0 or 1, when an input is exactly 0, and such inputs are rare. Everywhere else ReLU’s slope is a clean 0 or 1. The function that truly cannot be trained by slopes is the step, whose slope is 0 wherever it exists.
Check yourself
Recall
1. What is the derivative function of \({f(x) = x^2}\)?
Show answer
\({f'(x) = 2x}\).
2. What is the derivative of \({1/x}\), and where does it exist?
Show answer
\({-1/x^2}\), for every \({x \neq 0}\).
3. At a point where the tangent to the graph of \(f\) tilts down, what is the sign of \({f'(x)}\)?
Show answer
Negative. A tangent that tilts up gives a positive derivative, and a flat one gives 0.
4. What are the one-sided difference quotients of \({\abs{x}}\) at 0?
Show answer
1 from the right and \({-1}\) from the left, for every step. They never agree, so \({\abs{x}}\) has no derivative at 0.
5. Does differentiable at a point imply continuous there? Does continuous imply differentiable?
Show answer
Yes to the first, proved by writing \({f(a+h) - f(a)}\) as the quotient times \(h\). No to the second: \({\abs{x}}\) is continuous at 0 but not differentiable there.
Exercises
1.★ Using \({f'(x) = 2x}\) for \({f(x) = x^2}\): find the slope at \({x = -3}\), the point where the slope is 5, and the tangent line at \({x = -3}\).
Show solution
\({f'(-3) = -6}\). The slope is 5 where \({2x = 5}\), at \({x = 2.5}\). The tangent at \({-3}\) passes through \({(-3, 9)}\) with slope \({-6}\): \({y = 9 - 6(x + 3) = -6x - 9}\).
2.★★ From the definition, find \({g'(x)}\) for the running cubic \({g(x) = x^3 - 3x}\), and check the sketch: zeros at \({\pm 1}\) and the value \({-3}\) at 0.
As \(h\) tends to 0 this tends to \({g'(x) = 3x^2 - 3}\). It is 0 where \({x^2 = 1}\), at \({x = \pm 1}\), the hill and the valley, and \({g'(0) = -3}\), its lowest value, where the graph falls most steeply.
3.★★★ Let \({f(x) = x\,\abs{x}}\). Show that \(f\) is differentiable at 0 with \({f'(0) = 0}\), although \({\abs{x}}\) is not, and that \({f'(x) = 2\abs{x}}\) for every \(x\).
Show solution
At 0 the quotient is \({(h\,\abs{h} - 0)/h = \abs{h}}\), which tends to 0 from both sides, so \({f'(0) = 0}\). For \({x > 0}\), near \(x\) the function is \({x^2}\), with slope \({2x = 2\abs{x}}\). For \({x < 0}\) it is \({-x^2}\), whose quotient \({-((x+h)^2 - x^2)/h} = {-(2x + h)}\) tends to \({-2x = 2\abs{x}}\). So \({f'(x) = 2\abs{x}}\) everywhere. Multiplying by \(x\) smoothed the corner away, but the derivative \({2\abs{x}}\) now has a corner of its own at 0.
Summary card
The derivative is a function: its height at \(x\) is the slope of \(f\) at \(x\).
M04.2.3-F1
Slopes become heights.
Definition.\({f'(x)}\) is the limit of \({(f(x+h) - f(x))/h}\) as \({h \to 0}\), wherever it exists.
Two derivatives.\({x^2}\) has derivative \({2x}\); \({1/x}\) has derivative \({-1/x^2}\) for \({x \neq 0}\).
Corners.\({\abs{x}}\) (slopes \({-1}\) and 1) and ReLU (slopes 0 and 1) have no derivative at 0.
Continuity. Differentiable implies continuous, but not the other way round.
Sources and next steps
Sources. Each was opened and checked against this lesson on 12 November 2026.
Gilbert Strang and Edwin Herman, Calculus Volume 1, Section 3.2 The Derivative as a Function, OpenStax. The derivative function (equation 3.9); Theorem 3.1, differentiable implies continuous, with the same proof; \({\abs{x}}\), continuous everywhere with \({f'(0)}\) undefined; the vertical tangent of \({x^{1/3}}\) at 0; and the signs of \({f'}\) read from where \(f\) rises and falls (Figure 3.13).
Matthew Boelkins, Active Calculus, Section 1.4 The derivative function, Definition 1.4.2, where “heights on the derivative’s graph correspond to slopes on the original function’s graph”; and Section 1.7, where a function with a corner “will always look like a ‘V’, and never like a single line” however far one zooms in, and a differentiable function must be continuous.
Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, Section 6.3 Hidden Units: for \({g(z) = \max\{0, z\}}\) the left derivative at 0 is 0 and the right derivative is 1, and software “usually return[s] one of the one-sided derivatives rather than reporting that the derivative is undefined”.
