\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

S02.3 · Maximum likelihood

A free, illustrated module of 4 lessons and one interactive in S02 Statistical Inference: The likelihood function; Maximum likelihood for the standard families; Two parameters: the normal likelihood surface; Numerical maximum likelihood and losses as likelihoods.

S02.3 L2 · Intermediate 4 lessons · about 110 min Live · in review

Part of S02 Statistical Inference in the Statistics and Data Analysis track.

Turn the question round. Instead of asking how probable the data are for a given parameter, fix the data and ask which parameter makes them most probable. These four lessons draw the likelihood of a coin as a curve, derive the maximum likelihood estimates of the Poisson, exponential and uniform families, climb the two-parameter likelihood surface of a normal sample to find that least squares is maximum likelihood in disguise, and finally find a peak that has no formula, with Newton’s method and SciPy, before showing that the log loss and cross-entropy that train classifiers are negative log-likelihoods.

What you need first. Estimators, bias and the method of moments (S02.2), the common families of distributions (M06.8), derivatives and Newton’s method in one variable (M04.4), partial derivatives and the Hessian (M05.2, M05.5) for the third lesson, and the log loss of logistic regression (ML01.6) for the last.

Lessons

1. The likelihood function S02.3.1 25 min

   Fix the data and let the parameter vary: the probability of what you saw becomes the likelihood, a curve over the parameter. Draw it for a coin with 7 heads in 10 tosses, see why logarithms keep its peak, derive the estimate k / n, and compare two values by a ratio.

2. Maximum likelihood for the standard families S02.3.2 25 min

   One recipe, write the log-likelihood, differentiate, solve and check, finds the maximum likelihood estimate for Poisson counts and exponential waiting times. The uniform needs a look at the whole curve instead. Then the invariance property, and why an MLE can be biased.

3. Two parameters: the normal likelihood surface S02.3.3 30 min

   With a mean and a spread to estimate, the likelihood becomes a hill over a plane. Climb it for five visitors’ times: both partial derivatives vanish at the sample mean and at the variance with divisor n, the Hessian confirms a peak, and the same steps turn a line with normal noise into least squares.

4. Numerical maximum likelihood and losses as likelihoods S02.3.4 30 min

   When the score equation has no closed form, search for the peak: fit a gamma to 40 city rents with Newton’s method from a method-of-moments start, then with SciPy. Then see that minimising a mean negative log-likelihood is maximum likelihood, so log loss and cross-entropy training are MLE.

The interactive

S02.3-X1 · Likelihood surface explorer. Drag up to eight data points along a line, starting from design A’s five times, and watch the peak of the normal log-likelihood move over the plane of mu and sigma. A slider fixes mu to show the slice through the surface, and the read-out compares the maximum likelihood variance (divisor n) with s squared (divisor n - 1). It lives in Two parameters: the normal likelihood surface.