\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

M03.4 · Positive definite matrices and quadratic forms

A free, illustrated module of 4 lessons and one interactive in M03 Linear Algebra II: Decompositions: Quadratic forms: bowls and saddles; Tests for positive definiteness; Cholesky: a square root by triangles; Minima, curvature and ellipses.

M03.4 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of M03 Linear Algebra II: Decompositions in the Mathematics track.

A symmetric matrix is also a function: the quadratic form x-transpose S x, whose graph is a bowl, a saddle or a trough. These four lessons read the shape off the eigenvalues, give the quick tests that tell a positive definite matrix without computing them, factor such a matrix as L times L-transpose by Cholesky’s method and use it to solve systems and draw correlated samples, and find the lowest point of a quadratic loss, where the gradient S x - b is zero and the level sets are ellipses along the eigenvectors.

What you need first. Symmetric matrices and the spectral theorem (M03.3), elimination and pivots (M02.6), partial derivatives and the gradient (M04.10) and covariance (M06.10). The code uses NumPy and a little SciPy, explaining each call.

Lessons

1. Quadratic forms: bowls and saddles M03.4.1 25 min

   A symmetric matrix defines a quadratic form, a function whose graph is a bowl, a saddle or a trough. Expand it, symmetrise any square matrix, and use the spectral theorem to remove the cross term so that the eigenvalues read off the shape.

2. Tests for positive definiteness M03.4.2 25 min

   A positive definite matrix is one whose quadratic form is a bowl. Learn the definitions, prove the quick tests by eigenvalues, by a > 0 and ac - b^2 > 0, and by pivots, and see why X-transpose X is never negative and why ridge regression can always be solved.

3. Cholesky: a square root by triangles M03.4.3 25 min

   Every positive definite matrix factors as L times L-transpose with L lower triangular. Derive the entries one by one, see that the method works exactly when the matrix is positive definite, and use it to solve systems and to draw correlated Gaussian samples.

4. Minima, curvature and ellipses M03.4.4 25 min

   A quadratic loss (1/2) x-transpose S x - b-transpose x has its lowest point where S x = b, exactly when S is positive definite. Derive the gradient, test flat points by their second partials, read level sets as ellipses along the eigenvectors, and measure distance in units of the data’s own ellipse.

The interactive

M03.4-X1 · Bowl or saddle. Set a symmetric 2 by 2 matrix with three sliders and watch its contour lines change between ellipses, parallel lines and hyperbolas, with the two eigen-axes drawn. The read-out classifies the matrix and shows its eigenvalues beside the quick test a and ac - b^2. Tap a point to start gradient steps downhill: they settle in a bowl and run away along a saddle. It lives in Tests for positive definiteness.