\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

M03.3 · Symmetric matrices and the spectral theorem

A free, illustrated module of 4 lessons and one interactive in M03 Linear Algebra II: Decompositions: Real eigenvalues, perpendicular eigenvectors; The spectral theorem; A sum of rank-one pieces; The Rayleigh quotient and the extreme directions.

M03.3 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of M03 Linear Algebra II: Decompositions in the Mathematics track.

The matrices that machine learning hands to an eigenvalue routine, covariance matrices, Gram matrices, matrices of second derivatives and graphs, are nearly always symmetric, and symmetry buys a great deal. These four lessons prove that a symmetric matrix has real eigenvalues and perpendicular eigenvectors, that it always factors as Q Lambda Q-transpose (turn, stretch along the axes, turn back), that it is a sum of scaled projections onto perpendicular lines, and that its largest and smallest eigenvalues are the extremes of the Rayleigh quotient: the fact behind the first direction of principal component analysis.

What you need first. Eigenvalues, eigenvectors and the characteristic polynomial (M03.1), diagonalisation as A = P D P inverse (M03.2), dot products and projection onto a line (M02.3), orthogonal matrices and Gram-Schmidt (M02.10), and complex conjugates (M01.9). Covariance (M06.10) and partial derivatives (M04.10) supply examples.

Lessons

1. Real eigenvalues, perpendicular eigenvectors M03.3.1 25 min

   A symmetric matrix equals its own mirror image across the diagonal. Prove that its eigenvalues are always real and that eigenvectors for different eigenvalues are always perpendicular, and meet the covariance, Gram, second-derivative and graph matrices where this matters.

2. The spectral theorem M03.3.2 25 min

   Every real symmetric matrix factors as S = Q Lambda Q-transpose with Q orthogonal: turn into the eigenbasis, stretch along the axes, turn back. Prove it for 2 by 2 in full, see the induction for any size, handle repeated eigenvalues with Gram-Schmidt, and check it with numpy.linalg.eigh.

3. A sum of rank-one pieces M03.3.3 25 min

   Multiply out S = Q Lambda Q-transpose and a symmetric matrix falls apart into a weighted sum of projections onto perpendicular lines. Use the pieces to act on a vector, to build S inverse, S to a power and the square root of S, and to glimpse low-rank approximation.

4. The Rayleigh quotient and the extreme directions M03.3.4 25 min

   The Rayleigh quotient x-transpose S x over x-transpose x is a weighted average of the eigenvalues, so it always lies between the smallest and the largest. For a covariance matrix it is the variance along a direction, which makes the top eigenvector the direction of greatest spread: the first step of PCA.

The interactive

M03.3-X1 · Spectral ellipse. Three sliders set a symmetric matrix [[a, b], [b, c]]. The unit circle and its image are drawn with the two perpendicular eigenvectors on the axes of the image ellipse, and a toggle splits the matrix into its two rank-one pieces to show what each does to a chosen vector. It lives in The spectral theorem.