\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML03.7 · Matrix factorization and topic models

A free, illustrated module of 4 lessons and one interactive in ML03 Unsupervised Learning: Data as a product of two thin matrices; Non-negative matrix factorisation; Topics as mixtures: probabilistic LSA; Latent Dirichlet allocation.

ML03.7 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of ML03 Unsupervised Learning in the Machine Learning track.

A table of word counts has more structure than its size suggests: a few prototypes, mixed in different amounts, rebuild most of it. These four lessons approximate a data matrix by a thin matrix times a wide one, take the best such product from the SVD (latent semantic analysis), forbid negative entries to get readable parts (non-negative matrix factorisation), read the factors as probabilities fitted by EM (probabilistic LSA), and finally give every document’s topic shares a Dirichlet prior (latent Dirichlet allocation), all on one tiny corpus of sports and baking documents that can be checked by hand.

What you need first. The SVD and the best rank-k approximation (M03.5, M03.6), cosine similarity and distances (ML01.3), gradient descent (ML01.5), the EM algorithm for Gaussian mixtures (ML03.4), and common distributions (M06.8).

Lessons

1. Data as a product of two thin matrices ML03.7.1 25 min

   Approximate a data matrix by a thin matrix times a wide one. The truncated SVD gives the best such product, and on a tiny corpus it becomes latent semantic analysis: words and documents in one plane, compared by angle.

2. Non-negative matrix factorisation ML03.7.2 25 min

   Forbid negative entries in both factors and a factorisation turns into parts that only add up. Derive Lee and Seung’s multiplicative updates from gradient descent, run one by hand, and find sports and baking topics and pen strokes.

3. Topics as mixtures: probabilistic LSA ML03.7.3 25 min

   Read a factorisation as probabilities: a document picks a topic, the topic picks a word. Fit both tables by EM exactly as for a Gaussian mixture, work two iterations by hand, and see why the model is NMF with the Kullback-Leibler loss.

4. Latent Dirichlet allocation ML03.7.4 25 min

   Give every document’s topic shares a Dirichlet prior and the topic model can generate, and score, any document. Meet the Dirichlet on a triangle, derive the collapsed Gibbs update as a product of two counts, and fit topics from scratch and with scikit-learn.

The interactive

ML03.7-X1 · Factor explorer. The corpus’s 8 by 7 count matrix beside its two factors. Choose the rank, switch between the truncated SVD and NMF, step the multiplicative updates, and watch the reconstruction, the error in every cell and the top words of each factor, with the SVD’s negative entries flagged. It lives in Non-negative matrix factorisation.