\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

S02.10 · Likelihood theory: Fisher information and asymptotics

A free, illustrated module of 4 lessons and one interactive in S02 Statistical Inference: The score and Fisher information; The Cramér-Rao lower bound; Asymptotic normality of the MLE; Wald, score and likelihood-ratio tests.

S02.10 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of S02 Statistical Inference in the Statistics and Data Analysis track.

A maximum likelihood estimate is only half an answer until it comes with a measure of how sure it is. These four lessons find that measure in the shape of the log-likelihood: the score and its variance, the Fisher information; the Cramér-Rao bound, a floor no unbiased estimator can go below; the large-sample normality of the MLE, which gives every fitted model its standard errors; and the Wald, score and likelihood-ratio tests and intervals that all three read off one curve.

What you need first. Maximum likelihood (S02.3), estimators and bias (S02.2), expectation and variance (M06.7), correlation (M06.10), the law of large numbers and the central limit theorem (M06.11), Taylor polynomials (M04.8) and the Hessian (M05.5), and for the last lesson, tests, p-values and confidence intervals (S02.4, S02.5).

Lessons

1. The score and Fisher information S02.10.1 25 min

   Two coins can give the same estimate with very different precision, and the difference is written in how sharply the log-likelihood bends. Meet the score, prove that it averages zero at the truth, and measure information as its variance or as the expected curvature.

2. The Cramér-Rao lower bound S02.10.2 25 min

   No unbiased estimator can have a variance below 1 / (n I(theta)). Prove the bound from one covariance and the limit on a correlation, see the coin’s share of heads and the sample mean reach it, and see why the uniform’s corrected maximum is allowed to beat it.

3. Asymptotic normality of the MLE S02.10.3 25 min

   For large samples the maximum likelihood estimate is approximately normal, centred on the truth, with variance one over the information. Derive it from one Taylor step on the score, turn the curvature into standard errors, and carry them to other quantities with the delta method.

4. Wald, score and likelihood-ratio tests S02.10.4 25 min

   Three large-sample tests read off one log-likelihood curve: the distance to the null in standard errors, the slope at the null, and the drop from the peak. Run all three on the coin against a fair one, invert them into intervals, and see where they disagree.

The interactive

S02.10-X1 · Curvature and intervals. The log-likelihood of a coin or of exponential waiting times, with its quadratic approximation at the peak. Sliders set the number of trials and the observed proportion or sample mean; the Wald, score and likelihood-ratio intervals are drawn as bars beneath the curve, and the read-out shows them closing in on each other as n grows and the curve sharpens. It lives in Wald, score and likelihood-ratio tests.