\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

M03.5 · Singular value decomposition

A free, illustrated module of 4 lessons and one interactive in M03 Linear Algebra II: Decompositions: Every matrix turns a circle into an ellipse; Finding the SVD through A-transpose A; Rectangular matrices and the four subspaces; A sum of rank-one layers.

M03.5 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of M03 Linear Algebra II: Decompositions in the Mathematics track.

Every matrix, square or not, symmetric or not, turns the unit circle into an ellipse, and two perpendicular inputs land exactly on its axes. These four lessons find those inputs, prove that they always exist by way of the symmetric matrix A-transpose A, extend the picture to rectangular matrices and the four fundamental subspaces, and take any matrix apart into a sum of rank-one layers, largest first: the factorisation behind image compression, PCA, least squares and low-rank fine-tuning.

What you need first. Matrices as maps and orthogonal matrices (M02.4, M02.10), the four subspaces (M02.7) and determinants (M02.8); eigenvalues (M03.1), the spectral theorem (M03.3) and semidefinite matrices (M03.4). The first lesson uses the intermediate value theorem (M04.1.4).

Lessons

1. Every matrix turns a circle into an ellipse M03.5.1 25 min

   Any matrix sends the unit circle to an ellipse, and two perpendicular inputs land exactly on its axes. Find them for a 2 by 2 matrix, meet the singular values, and read the matrix as a turn, a stretch along the axes and another turn.

2. Finding the SVD through A-transpose A M03.5.2 25 min

   The right singular vectors of any matrix are the eigenvectors of the symmetric matrix A-transpose A, and the singular values are the square roots of its eigenvalues. Use this to compute an SVD by hand, prove that every matrix has one, and see why a library never forms A-transpose A.

3. Rectangular matrices and the four subspaces M03.5.3 25 min

   The SVD of a tall or wide matrix: the shapes of the full and the thin factorisation, the rank as the number of non-zero singular values, and orthonormal bases of all four fundamental subspaces read straight off the columns of U and V.

4. A sum of rank-one layers M03.5.4 25 min

   Multiply the SVD out and any matrix becomes a sum of rank-one layers, each a column times a row, largest first. Derive it, rebuild two matrices layer by layer, read off the largest stretch and the sum of squared entries, and compare the SVD of a symmetric matrix with its eigenvalues.

The interactive

M03.5-X1 · Rotate, stretch, rotate. Set a 2 by 2 matrix with four sliders or by dragging the images of i-hat and j-hat, then apply V-transpose, Sigma and U one step at a time and watch the unit circle turn, stretch and turn again onto the ellipse of the matrix, with the singular values compared with the eigenvalues. It lives in Every matrix turns a circle into an ellipse.