\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML01.7 · Evaluation metrics

A free, illustrated module of 4 lessons and one interactive in ML01 Machine Learning Foundations: Accuracy and the confusion matrix; Precision, recall and F1; ROC and precision-recall curves; Metrics for regression.

ML01.7 L1 · Core 4 lessons · about 110 min Live · in review

Part of ML01 Machine Learning Foundations in the Machine Learning track.

A model is only as good as the number you judge it by, and the obvious number is often the wrong one. These four lessons sort a classifier’s answers into the four cells of the confusion matrix, show how a model that finds nothing can be 99 percent accurate, and replace accuracy with precision, recall and F1. They then sweep the threshold to draw the ROC and precision-recall curves point by point, and finish with the metrics for models that predict a number: MAE, RMSE and R squared, each read beside a baseline.

What you need first. Logistic regression, its probabilities and the decision threshold, with false alarms and misses (ML01.6), and training, validation and test sets (ML01.2). The last lesson uses the mean squared error, the mean absolute error and R squared of a fitted line (ML01.4).

Lessons

1. Accuracy and the confusion matrix ML01.7.1 25 min

   A yes-or-no prediction can be right in two ways and wrong in two ways. The confusion matrix counts all four. Learn to build it, read accuracy off its diagonal, and see why a model that never catches a single fraud can still be 99 percent accurate.

2. Precision, recall and F1 ML01.7.2 30 min

   Precision asks how many of the model’s alarms were right. Recall asks how many of the real cases it found. Compute both from the confusion matrix, see which mistakes each one ignores, combine them into F1 with a harmonic mean, and choose a threshold by what each mistake costs.

3. ROC and precision-recall curves ML01.7.3 30 min

   Every threshold gives a different confusion matrix. Build the ROC curve and the precision-recall curve point by point from ten scores, read the area under the ROC curve as the chance that a positive outranks a negative, and learn when the second curve tells you more.

4. Metrics for regression ML01.7.4 25 min

   When a model predicts a number, each prediction misses by some amount. Compute the mean absolute error, the mean squared error, its root and R squared on one small test set, see which units each carries, how a single outlier moves them, and why each needs a baseline beside it.

The interactive

ML01.7-X1 · Threshold slider. Two histograms show the scores a model gives to positive and to negative examples. Slide the threshold and watch the confusion matrix, precision, recall and the marked point on the ROC and precision-recall curves move together, then make the positives rare and see which curve notices. It lives in ROC and precision-recall curves.