\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML03.9 · Anomaly detection

A free, illustrated module of 4 lessons and one interactive in ML03 Unsupervised Learning: What makes a point anomalous; Local outlier factor; Isolation forests; Reconstruction, boundaries and judging detectors.

ML03.9 L2 · Intermediate 4 lessons · about 120 min Live · in review

Part of ML03 Unsupervised Learning in the Machine Learning track.

Fraud, faults and intrusions are rare and seldom labelled, so an anomaly detector must rank points by strangeness from the data alone. These four lessons build the scores in turn: the Mahalanobis distance with its chi-square cut-off and a robust fit against masking, the local outlier factor, which judges each point against its own neighbours, isolation forests, which count random cuts, and the PCA reconstruction error and one-class SVM; then all of them are judged on planted anomalies by precision at k and average precision, and ROC AUC is shown to flatter.

What you need first. Density estimates (ML03.8) and Gaussian mixtures (ML03.4), nearest neighbours (ML01.3), PCA as compression (ML03.5), and precision, recall and ROC curves (ML01.7). NumPy for the code.

Lessons

1. What makes a point anomalous ML03.9.1 30 min

   An anomaly detector gives every point a score of strangeness and flags the highest. Meet the kinds of anomaly, outlier against novelty detection, z-scores, the Mahalanobis distance with its chi-square cut-off, masking and the robust fix.

2. Local outlier factor ML03.9.2 30 min

   Judge each point against its own neighbourhood: the local outlier factor compares a point’s density with its neighbours’ densities. Build it from k-distance, reachability distance and local reachability density, compute it by hand on the twelve points, and use it in scikit-learn.

3. Isolation forests ML03.9.3 30 min

   Anomalies are few and different, so random cuts fence them off quickly. Grow isolation trees, measure path lengths, derive the normaliser c(n) as the average path of an unsuccessful search in a binary search tree, and score points with Liu, Ting and Zhou’s s = 2^(-E[h]/c(n)).

4. Reconstruction, boundaries and judging detectors ML03.9.4 30 min

   Two more anomaly scores, the PCA reconstruction error and the one-class SVM, then a fair contest: every detector of the module on load_breast_cancer with ten planted malignant rows, judged by precision at k and average precision, and why ROC AUC flatters a detector when anomalies are rare.

The interactive

ML03.9-X1 · Isolation forest explorer. About thirty points in the plane with one chosen point you can drag. Split the box around it with random cuts until it stands alone and count the path length, or grow fifty trees to shade the anomaly score over the plane, with a contamination slider that sets the threshold and flags the top points. It lives in Isolation forests.