Before this lesson: M06.1.2 Events as sets M01.7 Sums, products and sequences
A weather app says 30% chance of rain. An image classifier says “cat 0.92”. A spam filter says 0.7. Each is a number between 0 and 1, but that alone does not make it a probability. What does? The answer is three short rules, and every result in the rest of this course follows from them.
From the last lesson
1. On two dice, which set is “neither die is a six”, and how many outcomes does it hold?
\({S_1^c \cap S_2^c}\), where \({S_1}\) and \({S_2}\) are “a six on the first” and “a six on the second”. Each die shows 1 to 5, so it holds \({5 \times 5 = 25}\) outcomes.
2. When are two events disjoint?
When they have no outcome in common, \({A \cap B = \emptyset}\), so they cannot both happen.
Draw the sample space \(\Omega\) as a square of area 1. Draw each event as a region inside it. Now read the probability of an event as the area of its region. Everything the rules of probability say is something you already believe about area.
Probability as area. The sample space is a square of area 1, and the probability of an event is the area of its region. The three numbered rules are the three axioms.
These three statements are the axioms of probability, proposed by Andrey Kolmogorov as the foundation of the subject. An axiom is a rule taken as the starting point and not proved. The next lesson derives every other rule you will use from these three, each with its own area picture.
Step 1. A probability measures events, not just outcomes. We write \({\Prob(A)}\), read “the probability of \(A\)”, for the number given to the event \(A\). A single outcome \(\omega\) gets a probability through the event \({\{\omega\}}\) that holds only it.
Step 2. Three rules, nothing else. The three axioms say nothing about where the numbers come from: a fair coin, a loaded die, a model’s output or a hunch. Any assignment that obeys them is a probability, and every result that follows from the axioms holds for all of them at once. That is why one theory serves gambling, physics and machine learning.
Step 3. Two readings of the same number. People mean two things by “the probability of \(A\)”.
Two dice rolled 20,000 times in a simulation. The share of rolls with a sum of 7 swings widely at first and settles near the exact probability 1/6 as the rolls pile up.
Both readings obey the same three axioms, so the mathematics is the same whichever you prefer. The frequency reading is how we will check every exact answer in this course: compute it, then simulate. The belief reading is how a classifier’s output is used: there is only one email in front of you, and the model’s 0.7 is its degree of belief that this one is spam. Lesson M06.11.2 proves that relative frequencies settle down, as the picture suggests.
Axiom 3 is stated for an endless sequence of events, because we need it for spaces like flips until the first head. It covers finitely many events too: for two disjoint events \(A\) and \(B\), use the sequence \({A, B, \emptyset, \emptyset, \dots}\). Its union is \({A \cup B}\), so \({\Prob(A \cup B)} = {\Prob(A)} + {\Prob(B)} + {\Prob(\emptyset)} + \cdots\). The next lesson proves from the axioms that \({\Prob(\emptyset) = 0}\), so the padding adds nothing and
\[ \begin{aligned} &\Prob(A \cup B) = \Prob(A) + \Prob(B) \\ &\text{when } A \cap B = \emptyset . \end{aligned} \]
Finite sample spaces: add up the outcomes. Write \({p(\omega) = \Prob(\{\omega\})}\) for the probability of a single outcome. An event \(A\) is the union of the single-outcome events of its members, and these are disjoint, because a single-outcome event shares nothing with another. So Axiom 3 gives
\[ \Prob(A) = \sum_{\omega \in A} p(\omega) . \]
The probabilities of the outcomes decide everything. Taking \({A = \Omega}\) and using Axiom 2, they must add up to 1. And the reverse holds: choose any numbers \({p(\omega) \geq 0}\) that add up to 1 over \(\Omega\), define \({\Prob(A)}\) by the sum above, and all three axioms hold. Axiom 1 holds because a sum of numbers that are not negative is not negative. Axiom 2 holds because the sum over all of \(\Omega\) is 1. Axiom 3 holds because each outcome of a union of disjoint events belongs to exactly one of them, so the sum over the union splits into one sum per event.
For a fair die, \({p(\omega) = \frac{1}{6}}\) for each face, so \({\Prob(\text{even}) = \frac{3}{6}}\). For two fair dice, each of the 36 outcomes gets \({\frac{1}{36}}\), so \({\Prob(\text{sum is } 7) = \frac{6}{36} = \frac{1}{6}}\).
A countably infinite space: flips until the first head. Number each outcome by the flip that brings the first head: outcome \(n\) is \({n - 1}\) tails followed by a head. A fair coin makes each of the \({2^n}\) ways that \(n\) flips can fall equally likely, and exactly one of them is outcome \(n\), so we give it \({p(n) = (\tfrac{1}{2})^n}\). Lessons M06.2.1 and M06.3.4 justify this step.
Is this a probability? Each \({p(n)}\) is positive, so Axiom 1 holds. For Axiom 2, add them all with the geometric sum of M01.7.3, \({a + ar + ar^2 + \cdots = \frac{a}{1 - r}}\) for \({-1 < r < 1}\), here with first term \({a = \frac{1}{2}}\) and ratio \({r = \frac{1}{2}}\):
\[ \begin{aligned} \Prob(\Omega) &= \frac{1}{2} + \frac{1}{4} + \frac{1}{8} + \cdots \\ &= \frac{1/2}{1 - 1/2} = 1 . \end{aligned} \]
An endless run of tails is not on the list. Had we added it as one more outcome, the listed outcomes would already use up the whole of the probability 1, so it could only get probability 0.
Flips until the first head as areas. Halving what is left again and again fills the square, so the probabilities add up to 1. The yellow pieces, a first head on an even flip, fill one third of it.
Now Axiom 3 does real work. “The first head comes on an even flip” is the union of the disjoint events \({\{2\}, \{4\}, \{6\}, \dots}\), so its probability is a geometric sum with first term \({\frac{1}{4}}\) and ratio \({\frac{1}{4}}\):
\[ \begin{aligned} \Prob(\text{even}) &= \frac{1}{4} + \frac{1}{16} + \frac{1}{64} + \cdots \\ &= \frac{1/4}{1 - 1/4} = \frac{1}{3} . \end{aligned} \]
No finite rule could give this answer: the event holds infinitely many outcomes.
Axiom 3 lists the events in some order, but nothing says which order. For terms that are never negative, an endless sum has the same value in every order and in every grouping, a standard fact about series. That is what lets \({\Prob(A) = \sum_{\omega \in A} p(\omega)}\) make sense on a countably infinite space, and what lets us split the sum over a union into one sum per event. With negative terms it could fail; probabilities are never negative, so it never does.
A die is loaded so that a six comes up half the time, and each other face a tenth of the time. Is this a probability? Find P(even) and P(at least 5).
Check the axioms. The outcome probabilities are \({p(1) = \cdots = p(5) = \frac{1}{10}}\) and \({p(6) = \frac{1}{2}}\). None is negative. They add to
\[ 5 \times \frac{1}{10} + \frac{1}{2} = \frac{1}{2} + \frac{1}{2} = 1, \]
so, by the reverse direction proved above, they define a probability.
Add the outcomes in each event. Think of the square cut into six strips, one per face, each as wide as that face’s probability.
The loaded die as six strips. The probability of an event is the total width of its strips: add the probabilities of its outcomes.
\[ \begin{aligned} \Prob(\{2, 4, 6\}) &= 0.1 + 0.1 + 0.5 = 0.7, \\ \Prob(\{5, 6\}) &= 0.1 + 0.5 = 0.6 . \end{aligned} \]
For a fair die the same events have probabilities 0.5 and about 0.33. Same sample space, same events, different probabilities: the axioms fix the rules, not the numbers.
A proposal that fails. Suppose someone suggests \({\frac{1}{5}}\) for each of the faces 1 to 5 and \({\frac{1}{2}}\) for the six. Every number is positive, but they add to \({1 + \frac{1}{2} = 1.5}\), so \({\Prob(\Omega)}\) would be 1.5 and Axiom 2 fails. This is not a probability.
Exact work first, with fractions.Fraction, so that no rounding hides a mistake.
from fractions import Fraction as F
p = {1: F(1, 10), 2: F(1, 10), 3: F(1, 10), 4: F(1, 10), 5: F(1, 10), 6: F(1, 2)}
def prob(event):
"""P(event) on a finite sample space: add up the outcomes."""
return sum(p[w] for w in event)
print("each p >= 0:", all(v >= 0 for v in p.values()), " P(Omega) =", prob(p))
print("P(even) =", prob({2, 4, 6}), " P(at least 5) =", prob({5, 6}))each p >= 0: True P(Omega) = 1
P(even) = 7/10 P(at least 5) = 3/5The endless sums for flips until the first head cannot be added term by term, but their partial sums show the gap closing exactly as the geometric sum predicts.
total = sum(F(1, 2**n) for n in range(1, 31)) # flips 1 to 30
even = sum(F(1, 2**n) for n in range(2, 31, 2)) # flips 2, 4, ..., 30
print("all outcomes up to 30:", float(total), " missing:", 1 - total)
print("even flips up to 30:", float(even), " target 1/3")
print("first head within 3 flips:", sum(F(1, 2**n) for n in (1, 2, 3)))all outcomes up to 30: 0.9999999990686774 missing: 1/1073741824
even flips up to 30: 0.3333333330228925 target 1/3
first head within 3 flips: 7/8Next the frequency reading: roll two dice many times in a simulation and watch the share of sums equal to 7. The shares printed below wander at first and end close to \({\frac{1}{6} \approx 0.167}\).
import numpy as np
rng = np.random.default_rng(7)
rolls = rng.integers(1, 7, size=(20_000, 2)) # high end excluded: faces 1 to 6
is_seven = rolls.sum(axis=1) == 7
share = np.cumsum(is_seven) / np.arange(1, len(is_seven) + 1)
for n in (10, 100, 1_000, 20_000):
print(f"after {n:>6} rolls: {share[n - 1]:.3f}")after 10 rolls: 0.100
after 100 rolls: 0.150
after 1000 rolls: 0.147
after 20000 rolls: 0.165Last, a classifier. A softmax layer turns three scores into three numbers that are positive and add to 1 (ML01.6.4). The classes are the outcomes, so these numbers satisfy the axioms, and the probability of “not class 3” is a sum.
scores = np.array([2.0, 1.0, 0.0])
probs = np.exp(scores) / np.exp(scores).sum()
print("softmax:", probs.round(3), " total:", probs.sum().round(6))
print("P(class 1 or class 2) =", (probs[0] + probs[1]).round(3))softmax: [0.665 0.245 0.09 ] total: 1.0
P(class 1 or class 2) = 0.91The event explorer in lesson M06.1.4 gives every cell of the dice grid the probability \({\frac{1}{36}}\) and adds up the cells of any event you build, so you can test Axiom 3 on disjoint events and see what goes wrong on overlapping ones.
Wrong beside right. Three scores between 0 and 1 for three classes that exclude each other add to 1.5, which breaks Axiom 2. Divided by their total they become a probability.
“Any numbers between 0 and 1 are probabilities.”
Three separate yes-or-no detectors for cat, dog and bird might score one picture 0.7, 0.6 and 0.2. If every picture holds exactly one of the three, the classes are disjoint and cover \(\Omega\), so by Axioms 2 and 3 their probabilities must add to 1. These add to 1.5. Dividing by the total, or using a softmax, repairs it. Being between 0 and 1 is a consequence of the axioms, not a substitute.
“I rolled the dice ten times and got one 7, so the probability of a 7 is 0.1.”
That is a relative frequency from a short run, and short runs swing widely, as the left end of the frequency chart shows. The probability is the value the share settles towards over a long run (or the value given by the axioms and a model of fair dice, \({\frac{1}{6}}\)). Ten rolls are evidence about it, not a measurement of it.
“Axiom 3 lets me add the probabilities of any two events.”
Only of disjoint ones. “The sum is 7” has probability \({\frac{6}{36}}\) and “the first die is even” has \({\frac{18}{36}}\), but the chance of one or the other is not \({\frac{24}{36}}\): three outcomes lie in both and would be counted twice. The next lesson finds the right answer, \({\frac{21}{36}}\).
1. State the three axioms of probability.
\({\Prob(A) \geq 0}\) for every event; \({\Prob(\Omega) = 1}\); and for disjoint events \({A_1, A_2, \dots}\) the probability of their union is the sum of their probabilities.
2. On a finite sample space, how do you find the probability of an event?
Add the probabilities of its outcomes: \({\Prob(A) = \sum_{\omega \in A} p(\omega)}\).
3. What must be true of numbers \({p(\omega)}\) for them to define a probability on a finite \(\Omega\)?
None may be negative, and together they must add up to 1.
4. What are the two readings of a probability?
Long-run frequency: the share of many repeated runs in which the event occurs settles towards it. Degree of belief: how strongly someone, or a model, expects a one-off event.
5. A coin is flipped until the first head. What is the probability that the first head comes on an even flip?
\(\frac{1}{4} + \frac{1}{16} + \cdots = \frac{1/4}{1 - 1/4} = \frac{1}{3}\), by Axiom 3 and the geometric sum.
1. ★ On \({\Omega = \{a, b, c\}}\), which of these assignments \({(p(a), p(b), p(c))}\) are probabilities? (i) \({(0.5, 0.3, 0.2)}\) (ii) \({(0.6, 0.6, -0.2)}\) (iii) \({(0.3, 0.3, 0.3)}\).
(i) Yes: all are at least 0 and they add to 1. (ii) No: they add to 1, but \({-0.2}\) breaks Axiom 1. (iii) No: they add to 0.9, so \({\Prob(\Omega) = 0.9}\) breaks Axiom 2.
2. ★★ A die is loaded so that each face comes up with probability proportional to its number: \({p(k) = ck}\) for \({k = 1, \dots, 6}\). Find \(c\), then P(even).
Axiom 2 needs \({c(1 + 2 + \cdots + 6) = 21c = 1}\), so \({c = \frac{1}{21}}\). Then
\[ \begin{aligned} \Prob(\{2, 4, 6\}) &= \frac{2 + 4 + 6}{21} \\ &= \frac{12}{21} = \frac{4}{7} . \end{aligned} \]
3. ★★★ A coin shows heads with probability \(q\), where \({0 < q \leq 1}\), and is flipped until the first head. Outcome \(n\) is given \({p(n) = (1 - q)^{n - 1} q}\). Show that these add up to 1, and find the probability that at least three flips are needed when \({q = \frac{1}{3}}\).
The sum is geometric with first term \({a = q}\) and ratio \({r = 1 - q}\), and \({0 \leq r < 1}\), so
\[ \begin{aligned} \sum_{n=1}^{\infty} (1 - q)^{n-1} q &= \frac{q}{1 - (1 - q)} \\ &= \frac{q}{q} = 1 . \end{aligned} \]
“At least three flips” is the union of the disjoint outcomes \({3, 4, 5, \dots}\), a geometric sum with first term \({(1 - q)^2 q}\) and the same ratio:
\[ \begin{aligned} \sum_{n=3}^{\infty} (1 - q)^{n-1} q &= \frac{(1 - q)^2 q}{q} \\ &= (1 - q)^2 . \end{aligned} \]
With \({q = \frac{1}{3}}\) this is \({(\frac{2}{3})^2 = \frac{4}{9}}\): the first two flips must both be tails.
A probability is an area on a square of area 1.
Three rules about area.
Sources. Each was opened and checked against this lesson on 14 November 2026.
Next. M06.1.4 Rules that follow from the axioms derives the complement rule, the addition rule for overlapping events and the union bound.
This lesson unlocks
M06.1.4 Rules that follow from the axioms