\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

S02.2 · Point estimation: bias, variance and mean squared error

A free, illustrated module of 4 lessons and one interactive in S02 Statistical Inference: Estimators and unbiasedness; Variance and efficiency; Mean squared error and consistency; Building estimators: the method of moments.

S02.2 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of S02 Statistical Inference in the Statistics and Data Analysis track.

An estimator is a rule that turns a sample into a guess, and because samples vary, so do its guesses. These four lessons judge such rules by where their guesses centre (bias), how widely they scatter (variance) and how far they land from the truth on average (mean squared error). They prove that the divisor n - 1 makes the sample variance unbiased, compare the mean and the median by efficiency, show that a little bias can buy a lower error (the idea behind ridge), define consistency, and build estimators by the method of moments, ending with the German tank problem.

What you need first. Sampling distributions and the standard error (S02.1), expectation and variance rules, the variance of the sample mean and Chebyshev’s inequality (M06.7, M06.11), the variance with divisor n - 1 (S01.3.3), and minimising a function of one variable (M04.4).

Lessons

1. Estimators and unbiasedness S02.2.1 25 min

   An estimator is a rule that turns a sample into a guess, and its guesses vary from sample to sample. Define bias as the gap between their average and the truth, prove that x-bar, p-hat and s² are unbiased, and see why s is not.

2. Variance and efficiency S02.2.2 25 min

   Two unbiased estimators of the same centre can differ in how widely they scatter. Compare the mean and the median by their variances, read relative efficiency as the share of the data an estimator wastes, and see the median win when tails are heavy.

3. Mean squared error and consistency S02.2.3 25 min

   Mean squared error scores an estimator by how far it lands from the truth on average, and splits into bias squared plus variance. See a deliberately biased estimator win, the idea behind ridge, and prove that an MSE that falls to zero makes an estimator consistent.

4. Building estimators: the method of moments S02.2.4 25 min

   Set the population’s moments equal to the sample’s and solve for the parameters. Build estimators for the exponential, normal and uniform, then work the German tank problem and compare three answers by bias, variance and mean squared error.

The interactive

S02.2-X1 · Estimator target. Repeated estimates c times x-bar of the city’s mean rent fall as dots around the truth. Set the sample size and the shrinkage factor c, and watch bias squared and variance trade off in three bars and on the curve of the mean squared error, with the best c marked. It lives in Mean squared error and consistency.