\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

A map of the families

Bernoulli, binomial, geometric, Poisson, uniform, exponential, normal and categorical on one map, with arrows for how one becomes another. Choose a family for data by its story, see what goes wrong when the story does not fit, and meet the categorical distribution behind a softmax.

M06.8.4L1 Core25 min

In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: M06.8.3 Poisson: counting rare events M06.6.3 The CDF, quantiles and sampling M06.6.4 Shift, scale and the normal density

What you will be able to do

Give the story, parameters, mean and variance of each family of the course.
Explain each arrow of the map: adding Bernoullis, the binomial-to-Poisson and geometric-to-exponential limits, the normal for many trials, and the inverse CDF.
Choose a family for a variable from its story, and recognise when the story does not fit.
Describe the categorical distribution, draw from it, and connect it to the softmax output of a classifier.
Build each family in scipy.stats with the right parameters.

The question

A data set has five columns: correct answers out of 50 on a quiz, support tickets in each hour, minutes between tickets, a sensor’s measurement error, and the class a network gives an image. Which distribution should model each one? The answer comes from how each number is made, its story, and not from the shape of a histogram.

From the last lesson

1. Which binomials does \({\mathrm{Pois}(\lambda)}\) approximate?

Show answer

\({\mathrm{Bin}(n, p)}\) with \(n\) large, \(p\) small and \({np = \lambda}\).

2. What are the mean and variance of \({\mathrm{Pois}(\lambda)}\)?

Show answer

Both are \(\lambda\).

Intuition

The families of this course are not a list to memorise. They are a few stories, and the stories are joined: one family turns into another when you add copies, shrink a step or pass a number through a function.

M06.8.4-F1

The map of the families. Each box is a story; each arrow is a way one story becomes another.

At the centre is the Bernoulli trial, one yes or no. Add \(n\) of them and you get a binomial. Wait for the first yes and you get a geometric. Give the trial more than two outcomes and you get a categorical. The other arrows are limits, and one is a recipe.

Build it up

Step 1. Add Bernoullis: binomial. A sum of \(n\) independent \({\mathrm{Bern}(p)}\) indicators is \({\mathrm{Bin}(n, p)}\) (M06.8.1).

Step 2. Many unlikely trials: Poisson. \({\mathrm{Bin}(n, \lambda/n) \to \mathrm{Pois}(\lambda)}\) as \({n \to \infty}\) (M06.8.3).

Step 3. Shorter and shorter trials: exponential. A geometric wait in slots of length \({1/n}\), each with chance \({\lambda/n}\), tends to \({\mathrm{Exp}(\lambda)}\) (M06.8.2).

Step 4. Many trials: the normal. Keep \(p\) fixed and let \(n\) grow. The binomial bars take on the bell shape of the normal density with the same mean \({np}\) and variance \({np(1 - p)}\).

M06.8.4-F2

Binomial bars with the normal curve of the same mean and variance in vermillion. Already at \({n = 10}\) the fit is close; at \({n = 100}\) the curve runs through the tops of the bars.

This is the central limit theorem at work, shown here and stated in M06.11: a sum of many small independent pieces is close to normal, whatever the pieces are.

Step 5. A uniform number through the inverse CDF: anything. M06.6.3 showed that if \({U \sim \mathrm{Unif}(0, 1)}\) and \(F\) is a CDF, then \({F^{-1}(U)}\) has CDF \(F\). For the exponential, solve \({u = 1 - e^{-\lambda t}}\) for \(t\):

\[ t = F^{-1}(u) = -\frac{\ln(1 - u)}{\lambda} . \]

Check it directly. For \({t \ge 0}\), since \({\ln}\) is increasing,

\[ \begin{aligned} &\Prob\big(-\ln(1 - U)/\lambda \le t\big) \\ &\quad= \Prob\big(1 - U \ge e^{-\lambda t}\big) \\ &\quad= \Prob\big(U \le 1 - e^{-\lambda t}\big) \\ &\quad= 1 - e^{-\lambda t} . \end{aligned} \]

M06.8.4-F3

The inverse CDF as a machine. Evenly spaced uniform numbers, read across to the CDF and down, crowd where the density is high: the waiting times come out exponential.

This is how computers sample every family: start from uniform numbers and transform them.

The formal statement

The families of the course, with their stories

Bernoulli \({\mathrm{Bern}(p)}\): one trial; 1 for success, 0 for failure.
Binomial \({\mathrm{Bin}(n, p)}\): successes in \(n\) independent trials; values \({0, \ldots, n}\).
Geometric \({\mathrm{Geom}(p)}\): trials up to and including the first success; values \({1, 2, \ldots}\).
Poisson \({\mathrm{Pois}(\lambda)}\): events in a window, from many unlikely chances; values \({0, 1, 2, \ldots}\).
Uniform \({\mathrm{Unif}(a, b)}\): a point with no preference anywhere in \({[a, b]}\).
Exponential \({\mathrm{Exp}(\lambda)}\): the wait for the next event at rate \(\lambda\); values \({t \ge 0}\).
Normal \({N(\mu, \sigma^2)}\): a sum of many small independent effects; any real value.
Categorical \({\mathrm{Cat}(p_1, \ldots, p_K)}\): one label out of \(K\), label \(k\) with chance \(p_k\).

The means and variances of the discrete families:

Family	Mean	Variance
\({\mathrm{Bern}(p)}\)	\(p\)	\({p(1-p)}\)
\({\mathrm{Bin}(n, p)}\)	\({np}\)	\({np(1-p)}\)
\({\mathrm{Geom}(p)}\)	\({1/p}\)	\({(1-p)/p^2}\)
\({\mathrm{Pois}(\lambda)}\)	\(\lambda\)	\(\lambda\)

And of the continuous ones:

Family	Mean	Variance
\({\mathrm{Unif}(a, b)}\)	\({(a+b)/2}\)	\({(b-a)^2/12}\)
\({\mathrm{Exp}(\lambda)}\)	\({1/\lambda}\)	\({1/\lambda^2}\)
\({N(\mu, \sigma^2)}\)	\(\mu\)	\({\sigma^2}\)

The categorical distribution. A categorical variable takes one of \(K\) labels, with chances \({p_1, \ldots, p_K}\) that are never negative and add to 1. With \({K = 2}\) it is a Bernoulli, with the labels written 1 and 0. Its labels need not be numbers, so it has no mean of its own. Write it instead as a one-hot vector \(\vect{y}\), with a 1 in the place of the label drawn and 0 elsewhere. Each entry \(y_k\) is the indicator of label \(k\), so \({\E[y_k] = p_k}\), and the mean of the one-hot vector is the probability vector \({(p_1, \ldots, p_K)}\).

A classifier with \(K\) classes outputs exactly this. Its last layer gives scores \({z_1, \ldots, z_K}\) (logits), and the softmax turns them into a categorical distribution:

\[ p_k = \frac{e^{z_k}}{e^{z_1} + \cdots + e^{z_K}} . \]

Every \(p_k\) is positive and they add to 1, whatever the scores (ML01.6.4).

M06.8.4-F6

The softmax turns the scores \({(2, 1, 0)}\) into the categorical distribution \({(0.665, 0.245, 0.090)}\). The blue diamonds are the shares of each class in 1,000 draws from it.

The families in SciPy. Each is an object in scipy.stats, and the parameter names differ from the textbook ones in four places (Poisson, uniform, exponential and normal):

Family	SciPy name	Parameters
\({\mathrm{Bern}(p)}\)	`bernoulli`	`p`
\({\mathrm{Bin}(n, p)}\)	`binom`	`n`, `p`
\({\mathrm{Geom}(p)}\)	`geom`	`p`
\({\mathrm{Pois}(\lambda)}\)	`poisson`	`mu` \({= \lambda}\)
\({\mathrm{Unif}(a, b)}\)	`uniform`	`loc` \({= a}\), `scale` \({= b - a}\)
\({\mathrm{Exp}(\lambda)}\)	`expon`	`scale` \({= 1/\lambda}\)
\({N(\mu, \sigma^2)}\)	`norm`	`loc` \({= \mu}\), `scale` \({= \sigma}\)

SciPy has no separate categorical object: draw one with rng.choice(K, p=probs) in NumPy, or use scipy.stats.multinomial(1, probs), which gives the one-hot vector.

Worked example

Choose a family for each column of the opening data set, and compute one number for each.

M06.8.4-F4

Choosing by the story. Ask how the number is made, follow the row to the family, then fill in the parameters from what you know.

Correct answers out of 50. A fixed number of questions, each right with about the same chance, say 0.9: \({\mathrm{Bin}(50, 0.9)}\). Mean \({50 \times 0.9 = 45}\), standard deviation \({\sqrt{50 \times 0.9 \times 0.1} = \sqrt{4.5} = 2.12}\).

Tickets in an hour, 4 on average. Many customers, each unlikely to write in a given hour: \({\mathrm{Pois}(4)}\). An hour with no ticket has chance \({e^{-4} = 0.0183}\).

Minutes to the next ticket. The same stream seen as a wait, at rate \({4/60 = 1/15}\) per minute: \({\mathrm{Exp}(1/15)}\), mean 15 minutes, and a gap of more than half an hour has chance \({e^{-30/15} = e^{-2} = 0.1353}\).

Retries until a call to a flaky service works, each call working with chance 0.8: \({\mathrm{Geom}(0.8)}\), mean \({1/0.8 = 1.25}\) calls, and needing more than 2 calls has chance \({0.2^2 = 0.04}\).

A sensor’s error, the sum of many small disturbances: \({N(0, \sigma^2)}\), with \(\sigma\) found by calibration. The class of an image: categorical, with the probabilities from a softmax.

Code

From scratch: an inverse-CDF sampler for the exponential, and a check of the mean and variance of every family in the tables by simulation.

import math

import numpy as np

rng = np.random.default_rng(seed=3)
N = 400_000

def exponential_from_uniform(lam, size):
    u = rng.random(size)                    # Unif(0, 1)
    return -np.log(1 - u) / lam             # the inverse CDF of Exp(lam)

draws = {
    "Bern(0.3)": (rng.random(N) < 0.3).astype(float), "Bin(50, 0.9)": rng.binomial(50, 0.9, N),
    "Geom(0.8)": rng.geometric(0.8, N), "Pois(4)": rng.poisson(4, N), "Unif(2, 5)": rng.uniform(2, 5, N),
    "Exp(1/15)": exponential_from_uniform(1 / 15, N), "N(1, 4)": rng.normal(1, 2, N),
}
table = {"Bern(0.3)": (0.3, 0.21), "Bin(50, 0.9)": (45, 4.5), "Geom(0.8)": (1.25, 0.3125), "Pois(4)": (4, 4),
         "Unif(2, 5)": (3.5, 0.75), "Exp(1/15)": (15, 225), "N(1, 4)": (1, 4)}
for name, x in draws.items():
    print(f"{name:13} mean {x.mean():8.3f} (table {table[name][0]:g})   variance {x.var():8.3f} (table {table[name][1]:g})")
print("P(T > 30 min):", round((draws["Exp(1/15)"] > 30).mean(), 4), " exact", round(math.exp(-2), 4))

Bern(0.3)     mean    0.299 (table 0.3)   variance    0.210 (table 0.21)
Bin(50, 0.9)  mean   44.998 (table 45)   variance    4.499 (table 4.5)
Geom(0.8)     mean    1.249 (table 1.25)   variance    0.311 (table 0.3125)
Pois(4)       mean    4.009 (table 4)   variance    4.015 (table 4)
Unif(2, 5)    mean    3.501 (table 3.5)   variance    0.750 (table 0.75)
Exp(1/15)     mean   14.987 (table 15)   variance  223.899 (table 225)
N(1, 4)       mean    0.999 (table 1)   variance    3.992 (table 4)
P(T > 30 min): 0.1351  exact 0.1353

Every simulated mean and variance lands within about 1% of the table. Next, a categorical draw from a softmax, from scratch and with rng.choice:

z = np.array([2.0, 1.0, 0.0])                       # scores for cat, dog, fox
p = np.exp(z) / np.exp(z).sum()                     # softmax
labels = rng.choice(3, size=100_000, p=p)           # categorical draws
one_hot = np.eye(3)[labels]                         # each draw as a one-hot vector
print("softmax:", p.round(3), "  mean one-hot vector:", one_hot.mean(axis=0).round(3))

softmax: [0.665 0.245 0.09 ]   mean one-hot vector: [0.667 0.243 0.09 ]

Last, the SciPy objects with the parameters of the third table, each asked for its mean and variance:

from scipy import stats

families = {
    "bernoulli": stats.bernoulli(0.3), "binom": stats.binom(50, 0.9), "geom": stats.geom(0.8),
    "poisson": stats.poisson(mu=4), "uniform": stats.uniform(loc=2, scale=5 - 2),
    "expon": stats.expon(scale=15), "norm": stats.norm(loc=1, scale=2),
}
for name, dist in families.items():
    print(f"{name:9} mean {dist.mean():7.4f}  variance {dist.var():8.4f}")
print(stats.multinomial(1, p).rvs(random_state=0))

bernoulli mean  0.3000  variance   0.2100
binom     mean 45.0000  variance   4.5000
geom      mean  1.2500  variance   0.3125
poisson   mean  4.0000  variance   4.0000
uniform   mean  3.5000  variance   0.7500
expon     mean 15.0000  variance 225.0000
norm      mean  1.0000  variance   4.0000
[[1 0 0]]

The last line prints one draw from multinomial(1, p): a one-hot vector with a single 1.

Explore

M06.8-X1

Distribution map

Choose a family to see its PMF or density, with sliders for its parameters and the mean and one standard deviation either side marked in orange. Then choose an arrow and step \(n\) up: watch a binomial with \({np}\) fixed settle onto a Poisson, a geometric staircase onto an exponential, or binomial bars onto a normal curve.

A map of eight boxes, categorical, Bernoulli, geometric, binomial and Poisson on the discrete side and normal, uniform and exponential on the continuous side, joined by green arrows labelled add n, wait for the first success, more than 2 outcomes, n to infinity with np equals lambda, n large, inverse CDF, and shorter and shorter trials.

Pitfalls

M06.8.4-F5

Wrong beside right. A normal curve with the right mean and variance still puts 5.7% of its probability on scores above 10.5, which cannot happen. The binomial story fits the score exactly.

“The histogram looks like a bell, so the data are normal.”

A score out of 10 can look bell-shaped, but it is a whole number between 0 and 10, and a normal model puts probability above 10, below 0 and between the whole numbers. Choose the family by the story of the number; use the normal as an approximation, knowing where it fails.

“Waits and counts are interchangeable.”

The minutes between tickets and the tickets per hour come from the same stream, but they are different numbers with different families: an exponential wait and a Poisson count. Mixing them up gives answers in the wrong units, such as a mean of 4 minutes where 15 was right.

“A softmax output is \(K\) separate yes or no probabilities.”

A categorical distribution picks exactly one label, so its probabilities add to 1: raising one lowers the others. When an image may hold a cat and a dog at once, the task needs \(K\) separate Bernoulli variables, one sigmoid per label, whose chances need not add to 1.

Check yourself

Recall

1. Which arrow turns a geometric into an exponential?

Show answer

Shorter and shorter trials: slots of length \({1/n}\) with chance \({\lambda/n}\) each, as \({n \to \infty}\).

2. Give the mean and variance of \({\mathrm{Unif}(a, b)}\) and of \({\mathrm{Exp}(\lambda)}\).

Show answer

\({(a + b)/2}\) and \({(b - a)^2/12}\); \({1/\lambda}\) and \({1/\lambda^2}\).

3. How do you turn \({U \sim \mathrm{Unif}(0, 1)}\) into an \({\mathrm{Exp}(\lambda)}\) wait?

Show answer

\({T = -\ln(1 - U)/\lambda}\), the inverse of the CDF \({1 - e^{-\lambda t}}\).

4. What is the categorical distribution, and where does it appear in machine learning?

Show answer

One label out of \(K\), label \(k\) with chance \(p_k\); the softmax output of a \(K\)-class classifier is a categorical distribution.

5. How do you build \({N(\mu, \sigma^2)}\) and \({\mathrm{Unif}(a, b)}\) in scipy.stats?

Show answer

norm(loc=mu, scale=sigma), with the standard deviation as scale; uniform(loc=a, scale=b - a).

Exercises

1. ★ Show that \({\mathrm{Var}(U) = 1/12}\) for \({U \sim \mathrm{Unif}(0, 1)}\), and deduce the variance of \({\mathrm{Unif}(a, b)}\).

Show solution

\({\E[U] = \int_0^1 u\,du = \tfrac12}\) and \({\E[U^2] = \int_0^1 u^2\,du = \tfrac13}\), so \({\mathrm{Var}(U) = \tfrac13 - \tfrac14 = \tfrac{1}{12}}\). The variable \({X = a + (b - a)U}\) is \({\mathrm{Unif}(a, b)}\), and \({\mathrm{Var}(cU + a) = c^2\,\mathrm{Var}(U)}\) (M06.7.3), so \({\mathrm{Var}(X) = (b - a)^2/12}\).

2. ★★ Emails reach an inbox at 0.2 per minute. Name the family and parameter of (a) the number of emails in an hour, (b) the minutes until the next email, (c) the number of sixes in 30 rolls of a die. Give each mean.

Show solution

(a) A count in a window: \({\mathrm{Pois}(0.2 \times 60) = \mathrm{Pois}(12)}\), mean 12.

(b) A wait in continuous time: \({\mathrm{Exp}(0.2)}\), mean \({1/0.2 = 5}\) minutes, which SciPy writes expon(scale=5).

3. ★★★ Two servers receive requests independently, the first after a wait \({S \sim \mathrm{Exp}(a)}\) and the second after \({T \sim \mathrm{Exp}(b)}\). Show that the wait until the first of them receives one, \({\min(S, T)}\), is \({\mathrm{Exp}(a + b)}\).

Show solution

The minimum is above \(t\) exactly when both waits are above \(t\). By independence,

\[ \begin{aligned} &\Prob(\min(S, T) > t) \\ &\quad= \Prob(S > t)\,\Prob(T > t) \\ &\quad= e^{-at} e^{-bt} = e^{-(a + b)t} , \end{aligned} \]

which is the tail of \({\mathrm{Exp}(a + b)}\). Rates add: two streams at 0.5 per minute each give a first request after 1 minute on average.

Summary card

Choose a family by its story; the arrows say how the stories are joined.

The anchor figure again: the map of eight families, discrete on the left and continuous on the right, joined by green arrows labelled with how one becomes another. — M06.8.4-F1
Eight families, one map.

Count out of \(n\): binomial. Count in a window: Poisson.
Wait in trials: geometric. Wait in time: exponential.
Sum of many small effects: normal. One label of \(K\): categorical.
Any family from a uniform number through the inverse CDF.

Sources and next steps

Sources. Each was opened and checked against this lesson on 14 November 2026.

Hossein Pishro-Nik, Introduction to Probability, Statistics, and Random Processes, Section 3.1.5 (the Bernoulli, geometric, binomial and Poisson distributions, the binomial as a sum of Bernoullis and the Poisson as a limit of binomials) and Section 4.2.2 (the exponential as a limit of geometric waits).
Chris Piech, Probability for Computer Scientists, Categorical Distributions: a variable whose values are categories, with a probability for each, described as a multinomial with one outcome.
SciPy developers, Statistical functions (scipy.stats) (SciPy 1.18 reference), which lists bernoulli, binom, geom and poisson among the discrete distributions, uniform, expon and norm among the continuous ones, and multinomial among the multivariate ones; norm takes the mean as loc and the standard deviation as scale, and uniform is the uniform distribution on \({[\text{loc}, \text{loc} + \text{scale}]}\).
NumPy developers, Generator.choice (NumPy 2.5 reference), whose p gives the probabilities associated with each entry.

Next. M06.9 puts two random variables on one table and asks how they move together; M06.11 states the arrow to the normal, the central limit theorem, and shows it at work; and ML01.6.4 uses the categorical distribution for classifiers with more than two classes.

This lesson unlocks

M06.9 Joint, marginal and conditional distributions M06.11 Law of large numbers and central limit theorem ML01.6.4 Decision boundaries and thresholds

Your progress is saved in this browser only. No account, no tracking.