\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML02.6 · Naive Bayes and generative classifiers

A free, illustrated module of 4 lessons and one interactive in ML02 Supervised Learning: Trees, Ensembles and Kernels: Generative and discriminative classifiers; The naive assumption and Bernoulli naive Bayes; Multinomial naive Bayes for text; Gaussian naive Bayes and when the assumption hurts.

ML02.6 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of ML02 Supervised Learning: Trees, Ensembles and Kernels in the Machine Learning track.

There are two ways to build a classifier: learn the border between the classes, or learn what each class looks like and let Bayes’ rule decide. These four lessons take the second road. They prove that the largest posterior gives the lowest possible error rate, make the naive assumption that turns an impossible density into one small model per feature, build Bernoulli, multinomial and Gaussian naive Bayes from counts and means, show that naive Bayes is a linear classifier on text, and measure what the assumption costs when features carry the same evidence twice.

What you need first. Logistic regression and the log loss (ML01.6), Bayes’ rule (M06.4), conditional independence (M06.3), densities and the normal curve (M06.6), and maximum likelihood (S02.3).

Lessons

1. Generative and discriminative classifiers ML02.6.1 25 min

   A classifier can learn the border between classes directly, or learn what each class looks like and let Bayes’ rule do the rest. See both routes on one feature, prove that the largest posterior gives the lowest error rate, and find the best threshold by hand.

2. The naive assumption and Bernoulli naive Bayes ML02.6.2 25 min

   A full model of which of 500 words an email contains needs more numbers than there are atoms in the universe. Assume the words are independent within each class and it needs one number per word. Build Bernoulli naive Bayes on the inbox, in logs, with add-one smoothing.

3. Multinomial naive Bayes for text ML02.6.3 25 min

   Treat a message as a bag of word counts and each class as a die with one face per word. Estimate the faces by smoothed counting, work a tiny corpus by hand, and discover that naive Bayes is a linear classifier whose weights are log ratios of word probabilities.

4. Gaussian naive Bayes and when the assumption hurts ML02.6.4 25 min

   For measurements, give every feature of every class its own bell curve. Fit Gaussian naive Bayes to the twelve points and to iris, see why its boundary curves, then watch correlated features count the same evidence twice and push its probabilities to 0 and 1.

The interactive

ML02.6-X1 · Two-class Gaussian explorer. Drag two bell curves, change their spreads and slide the prior. The weighted curves, the shaded Bayes error, the posterior and the thresholds update together, and a button duplicates the feature to show naive Bayes counting the same evidence twice. It lives in Gaussian naive Bayes and when the assumption hurts.