\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

Conditional distributions and independence

Learning the value of X picks out one row of the joint table; rescaled to add to 1, that row is the conditional PMF of Y, with its own mean. The product rule, Bayes’ rule for PMFs, and independence as every row rescaling to the same PMF.

M06.9.3L1 Core25 min

In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: M06.9.2 Marginals: summing out a variable M06.4.1 Turning a conditional around

What you will be able to do

Find a conditional PMF p(y | x) by rescaling one row of a joint table, and its conditional mean E[Y | X = x].
Rebuild a joint table from a marginal and conditionals with the product rule, and turn conditionals around with Bayes’ rule for PMFs.
Prove that X and Y are independent exactly when every row rescales to the same PMF, and test a table for independence.
Explain why the conditional mean is what a regression model estimates.
Compute conditionals with NumPy broadcasting and with pandas.crosstab and normalize.

The question

Three fair flips again: \(X\) counts heads in flips 1 and 2, \(Y\) in flips 2 and 3. Someone tells you that \({X = 2}\). What should you now believe about \(Y\), and what is your best single guess for it? A regression model asks exactly this for every input: given the features, what is the distribution of the target, and what is its mean?

From the last lesson

1. How do you get the marginal PMF of \(X\) from a joint table whose rows are the values of \(X\)?

Show answer

Add each row: \({p_X(x) = \sum_y p(x, y)}\). For the course’s table this gives \(1/4\), \(1/2\), \(1/4\).

2. Do the two marginals determine the joint table?

Show answer

No. The course’s table and the product table \({p_X(x)\,p_Y(y)}\) have the same marginals but differ in all four corners.

Intuition

Learning that \({X = x}\) rules out every outcome outside row \(x\). The row that is left holds the right proportions for \(Y\), but its cells add to \({p_X(x)}\), not to 1. Divide the row by its total and it becomes a PMF: the distribution of \(Y\) once you know \({X = x}\).

M06.9.3-F1

Each row, divided by its total, is a PMF of \(Y\). The orange triangles mark the conditional means: \(1/2\), \(1\) and \(3/2\). They climb with \(x\).

This is conditioning as in M06.3.1, zoomed in on a row: keep what is still possible and rescale it to total 1. Each row is its own PMF, and its mean moves as \(x\) moves.

Build it up

Step 1. Keep one row. If \({X = 0}\), then flips 1 and 2 were both tails, and only TTT and TTH remain. In the table that is row 0: \({1/8, 1/8, 0}\).

Step 2. Rescale it. Those cells add to \({p_X(0) = 1/4}\). Dividing by \(1/4\) (multiplying by 4) keeps their proportions and makes them add to 1.

M06.9.3-F2

Rescaling a row stretches every bar by the same factor: the shape stays and the total becomes 1.

Step 3. Read it. Given \({X = 0}\), \(Y\) is 0 or 1 with chance \(1/2\) each: \(Y\) is just flip 3. Given \({X = 2}\), flip 2 was heads, so \(Y\) is 1 or 2 with chance \(1/2\) each.

Step 4. Take its mean. The best single guess under squared error is the mean (M06.7.4), so for each row we want \({\sum_y y\,p(y \mid x)}\). For row 0 that is \({0 \cdot \tfrac12 + 1 \cdot \tfrac12 = \tfrac12}\).

The formal statement

Definition: conditional PMF and conditional mean

For each \(x\) with \({p_X(x) > 0}\), the conditional PMF of \(Y\) given \({X = x}\) is

\[ p(y \mid x) = \frac{p(x, y)}{p_X(x)}, \]

and the conditional mean of \(Y\) given \({X = x}\) is

\[ \E[Y \mid X = x] = \sum_y y\, p(y \mid x). \]

It is \({\Prob(Y = y \mid X = x)}\) from M06.3.1, with the events \({Y = y}\) and \({X = x}\). It is a PMF: each value is a cell divided by a positive number, so it is never negative, and

\[ \begin{aligned} \sum_y p(y \mid x) &= \frac{\sum_y p(x, y)}{p_X(x)} \\ &= \frac{p_X(x)}{p_X(x)} = 1. \end{aligned} \]

The top of the middle step is the row sum of M06.9.2. On the course’s table the value \({y = 0}\) adds nothing to a mean, so \({\E[Y \mid X = x]} = {p(1 \mid x) + 2\,p(2 \mid x)}\), and the three conditional means are

\[ \begin{aligned} \E[Y \mid X = 0] &= \tfrac12 + 2 \cdot 0 = \tfrac12, \\ \E[Y \mid X = 1] &= \tfrac12 + 2 \cdot \tfrac14 = 1, \\ \E[Y \mid X = 2] &= \tfrac12 + 2 \cdot \tfrac12 = \tfrac32. \end{aligned} \]

They lie on the line \({\tfrac12 + \tfrac{x}{2}}\). The function \({x \mapsto \E[Y \mid X = x]}\) is the regression function: the curve that a regression model, trained with squared error, tries to estimate from data (ML01.4). Lesson M06.10.3 meets the slope \(1/2\) again, as the correlation of \(X\) and \(Y\).

M06.9.3-F6

Each column of circles is one conditional PMF, and the orange diamonds are their means. A regression line aims at these diamonds, not at the individual points.

The product rule. Multiply the definition through by \({p_X(x)}\):

\[ p(x, y) = p_X(x)\, p(y \mid x). \]

A table can be built row by row: choose how much probability each row gets, then how it is shared out along the row. That is how the mixture of M06.9.2 was built.

Bayes’ rule for PMFs. The product rule works with the roles swapped too, \({p(x, y)} = {p_Y(y)\,p(x \mid y)}\). Set the two forms equal, divide by \({p_Y(y)}\), and write \({p_Y(y)}\) as a column sum:

\[ p(x \mid y) = \frac{p_X(x)\, p(y \mid x)}{\sum_{x'} p_X(x')\, p(y \mid x')}. \]

This is M06.4.1 for a whole PMF at once: the bottom is the column total, and dividing by it rescales a column. A classifier that outputs a probability for each class is reporting such a conditional PMF.

Independence, two equivalent tests

\(X\) and \(Y\) are independent when \({p(x, y) = p_X(x)\, p_Y(y)}\) for every cell (M06.5.4). This holds exactly when every row with \({p_X(x) > 0}\) rescales to the same PMF, and that PMF is then \(p_Y\).

Why, one way. If every cell is a product, then for each row \({p(y \mid x)} = {p_X(x)\,p_Y(y) / p_X(x)} = {p_Y(y)}\): every row rescales to \(p_Y\).

The other way. Suppose every row rescales to the same PMF \(q\), so \({p(x, y) = p_X(x)\,q(y)}\) by the product rule. (Rows with \({p_X(x) = 0}\) hold only zeros, so they fit this too.) Add down column \(y\):

\[ \begin{aligned} p_Y(y) &= \sum_x p_X(x)\, q(y) \\ &= q(y) \sum_x p_X(x) = q(y). \end{aligned} \]

So \({q = p_Y}\), and every cell is \({p_X(x)\,p_Y(y)}\).

M06.9.3-F5

Top: the course’s rows rescale to three different PMFs, so \(X\) tells you something about \(Y\). Bottom: the product table’s rows all rescale to \(p_Y\); knowing \(X\) changes nothing.

The course’s table fails at the corner \({(0, 2)}\), where \({p(0, 2) = 0}\) but \({p_X(0)\,p_Y(2) = 1/16}\). One failing cell is enough.

Go deeper: the conditional means average back to E[Y]

Weight the three conditional means by how likely each row is: \({\tfrac14 \cdot \tfrac12 + \tfrac12 \cdot 1 + \tfrac14 \cdot \tfrac32} = {\tfrac18 + \tfrac12 + \tfrac38} = 1\), which is \({\E[Y]}\), the mean of the marginal \({(1/4, 1/2, 1/4)}\). This always happens: the average of the conditional means, weighted by \(p_X\), is the overall mean. It is the tower rule, proved in M07.4 with conditional expectation as a random variable in its own right.

Worked example

Given \({Y = 2}\), find the distribution of \(X\) and its mean, two ways.

By the table. Column \({y = 2}\) holds \({0, 1/8, 1/8}\), which adds to \({p_Y(2) = 1/4}\). Divide by \(1/4\): \({p(x \mid y = 2)} = {(0, 1/2, 1/2)}\) for \({x = 0, 1, 2}\).

By Bayes’ rule. Suppose we knew only \({p_X = (1/4, 1/2, 1/4)}\) and the rows \({p(2 \mid x) = 0, 1/4, 1/2}\) from the anchor figure. Multiply:

\[ \begin{aligned} p_X(0)\, p(2 \mid 0) &= \tfrac14 \times 0 = 0, \\ p_X(1)\, p(2 \mid 1) &= \tfrac12 \times \tfrac14 = \tfrac18, \\ p_X(2)\, p(2 \mid 2) &= \tfrac14 \times \tfrac12 = \tfrac18. \end{aligned} \]

These are the cells of column 2 again, and their total \(1/4\) is \({p_Y(2)}\). Dividing gives \({(0, 1/2, 1/2)}\), as before.

M06.9.3-F3

Two routes to the same conditional PMF. The products of Bayes’ rule are the cells of the column, and the bottom of Bayes’ rule is the column total.

The mean. \({\E[X \mid Y = 2]} = {0 \cdot 0 + 1 \cdot \tfrac12 + 2 \cdot \tfrac12} = {\tfrac32}\). It matches \({\E[Y \mid X = 2]}\), as it must: swapping flips 1 and 3 swaps \(X\) and \(Y\).

Code

From scratch: divide each cell by its row total, then take each row’s mean.

import itertools
from collections import Counter, defaultdict
from fractions import Fraction

import numpy as np

def pair(word):
    h = [1 if ch == "H" else 0 for ch in word]
    return h[0] + h[1], h[1] + h[2]

counts = Counter(pair("".join(w)) for w in itertools.product("HT", repeat=3))
joint = {(x, y): Fraction(counts[(x, y)], 8) for x in range(3) for y in range(3)}
p_X = defaultdict(Fraction)
for (x, y), p in joint.items():
    p_X[x] += p

cond = {(x, y): joint[(x, y)] / p_X[x] for (x, y) in joint}          # p(y | x)
cond_mean = {x: sum(y * cond[(x, y)] for y in range(3)) for x in range(3)}
for x in range(3):
    print(f"p(y | x={x}):", [str(cond[(x, y)]) for y in range(3)], "  mean:", cond_mean[x])

p(y | x=0): ['1/2', '1/2', '0']   mean: 1/2
p(y | x=1): ['1/4', '1/2', '1/4']   mean: 1
p(y | x=2): ['0', '1/2', '1/2']   mean: 3/2

In NumPy, keepdims=True keeps the row totals as a column of shape \({(3, 1)}\), and broadcasting (P02.2) divides every cell by the total of its own row in one step.

table = np.array([[1, 1, 0], [1, 2, 1], [0, 1, 1]]) / 8
rows = table / table.sum(axis=1, keepdims=True)          # p(y | x): each row adds to 1
cols = table / table.sum(axis=0, keepdims=True)          # p(x | y): each column adds to 1
y_values = np.array([0, 1, 2])
print("p(y | x):\n", rows)
print("E[Y | X = x]:", rows @ y_values)
print("p(x | y = 2):", cols[:, 2])

product = np.outer(table.sum(axis=1), table.sum(axis=0))
print("independent?", np.allclose(table, product), "  rows all equal?", np.allclose(rows, rows[0]))

p(y | x):
 [[0.5  0.5  0.  ]
 [0.25 0.5  0.25]
 [0.   0.5  0.5 ]]
E[Y | X = x]: [0.5 1.  1.5]
p(x | y = 2): [0.  0.5 0.5]
independent? False   rows all equal? False

On data, pandas.crosstab with normalize="index" divides each row by its own total, which is conditioning on the row variable.

import pandas as pd

rng = np.random.default_rng(2)
h = rng.integers(0, 2, size=(20_000, 3))
X = pd.Series(h[:, 0] + h[:, 1], name="X")
Y = pd.Series(h[:, 1] + h[:, 2], name="Y")
by_row = pd.crosstab(X, Y, normalize="index")
print(by_row.round(3))
print("estimated E[Y | X = x]:", Y.groupby(X).mean().round(3).tolist())

Y      0      1      2
X                     
0  0.501  0.499  0.000
1  0.256  0.495  0.249
2  0.000  0.492  0.508
estimated E[Y | X = x]: [0.499, 0.993, 1.508]

The estimated conditional means land within a few hundredths of \(1/2\), \(1\) and \(3/2\). Grouping by \(X\) and averaging \(Y\) is the simplest regression there is.

Explore

M06.9-X1

Joint table explorer

Choose a row or a column to condition on. The slice is lit in the table and redrawn beside it, rescaled to add to 1, with its mean marked. Then switch to the independent table, which has the same marginal bars, and check that every row and every column now rescales to the same PMF.

The course's joint table in eighths, rows x equals 0, 1, 2: 1/8, 1/8, 0; 1/8, 2/8, 1/8; 0, 1/8, 1/8. Each row is divided by its total to give a conditional PMF of Y: 1/2, 1/2, 0 with mean 1/2; 1/4, 1/2, 1/4 with mean 1; and 0, 1/2, 1/2 with mean 3/2.

Pitfalls

M06.9.3-F4

Wrong beside right. Independence needs every cell to be a product. The middle cell happens to fit; the corner does not, and that settles it.

“One cell equals the product of its margins, so \(X\) and \(Y\) are independent.”

Independence is a claim about every cell. In the course’s table \({p(1, 1) = 2/8}\) equals \({p_X(1)\,p_Y(1) = 1/4}\), but \({p(0, 2) = 0}\) does not equal \(1/16\). A single matching cell proves nothing; a single failing cell proves dependence.

“The conditional PMF is the row itself.”

The row adds to \({p_X(x)}\), not to 1, until it is divided by that total. Leaving it undivided gives bars that add to \(1/4\) for \({x = 0}\), and a “mean” of \({1 \cdot 1/8 = 1/8}\) instead of \(1/2\).

“\({p(y \mid x)}\) and \({p(x \mid y)}\) are the same thing.”

One rescales a row, the other a column, and they differ in general (the inbox of M06.3.1: \(24/30\) against \(24/40\)). The course’s table is symmetric, so its rows and columns happen to give the same PMFs; do not let that hide the difference. Divide by the total of the variable you are told.

Check yourself

Recall

1. How do you get \({p(y \mid x)}\) from a joint table?

Show answer

Take row \(x\) and divide it by its total: \({p(y \mid x) = p(x, y) / p_X(x)}\).

2. What are \({\E[Y \mid X = x]}\) for \({x = 0, 1, 2}\) on the course’s table?

Show answer

\(1/2\), \(1\) and \(3/2\), on the line \({1/2 + x/2}\).

3. State the product rule for PMFs.

Show answer

\({p(x, y) = p_X(x)\, p(y \mid x)}\).

4. What does the denominator of Bayes’ rule for PMFs equal, in the table?

Show answer

The column total \({p_Y(y) = \sum_{x'} p_X(x')\, p(y \mid x')}\).

5. Give the two equivalent tests for independence.

Show answer

Every cell equals \({p_X(x)\,p_Y(y)}\); or every row with positive total rescales to the same PMF (which is then \(p_Y\)).

Exercises

1. ★ A joint table has row \({x = 0}\) reading \({0.10, 0.20, 0.10}\) and row \({x = 1}\) reading \({0.30, 0.10, 0.20}\) over \({y = 0, 1, 2}\). Find \({p(y \mid x = 1)}\) and \({\E[Y \mid X = 1]}\), and decide whether \(X\) and \(Y\) are independent.

Show solution

Row 1 adds to \(0.60\). Dividing gives \({p(y \mid 1)} = {(1/2, 1/6, 1/3)}\). Then \({\E[Y \mid X = 1]} = {0 \cdot \tfrac12 + 1 \cdot \tfrac16 + 2 \cdot \tfrac13} = {5/6}\). Row 0 rescales to \({(1/4, 1/2, 1/4)}\), a different PMF, so they are dependent. A cell check agrees: \({p(0, 0) = 0.10}\) but \({p_X(0)\,p_Y(0)} = {0.40 \times 0.40} = {0.16}\).

2. ★★ Let \(X\) be 1 for spam and 0 otherwise, with \({p_X = (0.6, 0.4)}\), and let \(Y\) be 1 when the email contains “free”. The rows are \({p(Y = 1 \mid X = 1) = 0.6}\) and \({p(Y = 1 \mid X = 0) = 0.1}\). Build the joint table with the product rule and find \({p(x \mid Y = 1)}\).

Show solution

Product rule: \({p(1, 1)} = {0.4 \times 0.6} = {0.24}\), \({p(1, 0)} = {0.4 \times 0.4} = {0.16}\), \({p(0, 1)} = {0.6 \times 0.1} = {0.06}\), \({p(0, 0)} = {0.6 \times 0.9} = {0.54}\). The four cells add to 1. Column \({y = 1}\) holds \(0.06\) and \(0.24\), with total \({p_Y(1) = 0.30}\). Dividing, \({p(X = 1 \mid Y = 1)} = {0.24 / 0.30} = {0.8}\) and \({p(X = 0 \mid Y = 1)} = {0.2}\): the inbox of M06.3.

3. ★★★ Show that if \(X\) and \(Y\) are independent then \({\E[Y \mid X = x]}\) is the same for every \(x\). Then show that the converse fails: let \(X\) be 0 or 1 with chance \(1/2\) each; if \({X = 0}\) let \({Y = 0}\), and if \({X = 1}\) let \(Y\) be \(-1\) or \(1\) with chance \(1/2\) each.

Show solution

If they are independent, every row rescales to \(p_Y\), so \({\E[Y \mid X = x]} = {\sum_y y\,p_Y(y)} = {\E[Y]}\) for every \(x\). For the example, \({\E[Y \mid X = 0] = 0}\) and \({\E[Y \mid X = 1]} = {-1 \cdot \tfrac12 + 1 \cdot \tfrac12} = 0\), so the conditional mean is the same for both rows. Yet the rows differ: row 0 rescales to all its probability on \({y = 0}\), row 1 to half on \(-1\) and half on \(1\). A cell check: \({p(0, 0) = 1/2}\) but \({p_X(0)\,p_Y(0) = \tfrac12 \times \tfrac12 = 1/4}\). So \(X\) and \(Y\) are dependent: knowing \(X\) tells you how spread out \(Y\) is, though not where its centre is.

Summary card

Condition on a row: keep it, divide by its total.

The anchor figure again: the course's joint table with each row divided by its total into a conditional PMF of Y, with means 1/2, 1 and 3/2 marked by orange triangles. — M06.9.3-F1
Three rows, three PMFs, three means.

Conditional PMF. \({p(y \mid x) = p(x, y) / p_X(x)}\).
Conditional mean. \({\E[Y \mid X = x] = \sum_y y\,p(y \mid x)}\): \(1/2\), \(1\), \(3/2\) here.
Product rule. \({p(x, y) = p_X(x)\,p(y \mid x)}\); Bayes divides by a column total.
Independent exactly when every row rescales to the same PMF, \(p_Y\).

Sources and next steps

Sources. Each was opened and checked against this lesson on 14 November 2026.

Hossein Pishro-Nik, Introduction to Probability, Statistics, and Random Processes, Section 5.1.3. The conditional PMF \({P_{X|Y}(x_i \mid y_j)} = {P_{XY}(x_i, y_j) / P_Y(y_j)}\); independence as \({P_{XY}(x, y)} = {P_X(x)P_Y(y)}\) for all \(x\), \(y\), from which the conditional PMF equals the marginal; and conditional expectation as the ordinary expectation with the conditional PMF.
MIT OpenCourseWare 18.05, Reading 7a, Joint Distributions, Independence, Spring 2022. Section 4 defines independence by the joint pmf being the product of the marginal pmfs, and Example 12 tests a table cell by cell: a 0 cell with non-zero margins already fails.
NumPy reference, numpy.sum: with keepdims=True the reduced axis is kept with size one, “so the result will broadcast correctly against the input array”.

Next. M06.9.4 Joint densities does all of this for two continuous variables. M06.10.1 multiplies the cells of this table by \({x\,y}\) to measure how \(X\) and \(Y\) move together.

This lesson unlocks

M06.9.4 Joint densities M06.10.1 Covariance: do they move together?

Your progress is saved in this browser only. No account, no tracking.