Skip to main content

Finding the SVD through A-transpose A

The right singular vectors of any matrix are the eigenvectors of the symmetric matrix A-transpose A, and the singular values are the square roots of its eigenvalues. Use this to compute an SVD by hand, prove that every matrix has one, and see why a library never forms A-transpose A.
M03.5.2L2 Intermediate25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: M03.5.1 Every matrix turns a circle into an ellipse M03.3 Symmetric matrices and the spectral theorem M03.4.2 Tests for positive definiteness

What you will be able to do
  • Derive A-transpose A = V Sigma-squared V-transpose from A = U Sigma V-transpose, and explain why A-transpose A is symmetric and positive semidefinite.
  • Compute the SVD of a 2 by 2 matrix by hand: eigenvalues and eigenvectors of A-transpose A, then sigma_i = sqrt(lambda_i) and u_i = A v_i / sigma_i.
  • Prove that the vectors u_i built this way are orthonormal, and so that every matrix has a singular value decomposition.
  • Build an SVD from numpy.linalg.eigh, compare it with numpy.linalg.svd, and explain the differences in sign and order.

The question

The last lesson found the singular vectors of a 2 by 2 matrix by turning a pair of arrows until their images met at a right angle. A matrix of a thousand columns cannot be searched like that. Is there a direct way to find the singular vectors, and can we be sure that every matrix, of any size, has them?

From the last lesson

1. State the equation that links \(\vect{v}_i\), \(\sigma_i\) and \(\vect{u}_i\).

Show answer

\(\mat{A}\vect{v}_i = \sigma_i\vect{u}_i\): the unit input \(\vect{v}_i\) lands on the axis \(\vect{u}_i\) of the ellipse, at distance \(\sigma_i\) from the centre.

2. What are the singular values of \(\begin{bmatrix} 3 & 0 \\ 4 & 5 \end{bmatrix}\), and how does their product compare with the determinant?

Show answer

\(\sigma_1 = 3\sqrt{5} \approx 6.708\) and \(\sigma_2 = \sqrt{5} \approx 2.236\). Their product is \(15 = \det\mat{A}\).

Intuition

Follow a right singular vector there and back. The matrix \(\mat{A}\) sends \(\vect{v}_1\) to \(\sigma_1\vect{u}_1\). The transpose \(\mat{A}\T\) runs the other way, from outputs back to inputs, and it sends \(\vect{u}_1\) back to \(\sigma_1\vect{v}_1\) (we prove this below). So the round trip \(\mat{A}\T\mat{A}\) brings \(\vect{v}_1\) home to its own line, stretched by \(\sigma_1^2\).

That is an eigenvector statement. The right singular vectors of \(\mat{A}\) are the eigenvectors of \(\mat{A}\T\mat{A}\), and the squared singular values are its eigenvalues. Since \(\mat{A}\T\mat{A}\) is symmetric, the spectral theorem of M03.3 hands us its eigenvectors, perpendicular and ready made.

Two panels joined by a green arrow labelled A transpose A. Left, titled Inputs: unit length: the blue unit circle with orange unit arrows v1 at 45 degrees and v2 at 135 degrees, at right angles, on pale orange lines. Right, titled After: same lines, longer, at a scale where one grid square is 10 units: a long thin vermillion ellipse along the 45 degree line, with a vermillion arrow 45 v1, labelled sigma1 squared v1, along the same pale line as v1, and a short arrow 5 v2, labelled sigma2 squared v2, along the line of v2. A sentence underneath says the right singular vectors are the eigenvectors of A transpose A, and its eigenvalues are sigma1 squared equal to 45 and sigma2 squared equal to 5.
M03.5.2-F1

The round trip \(\mat{A}\T\mat{A}\) for \(\mat{A} = \begin{bmatrix} 3 & 0 \\ 4 & 5 \end{bmatrix}\). It keeps \(\vect{v}_1\) and \(\vect{v}_2\) on their own lines and stretches them by \(45\) and \(5\).

Build it up

Step 1. Assume an SVD and multiply. Suppose \(\mat{A} = \mat{U}\mat{\Sigma}\mat{V}\T\). The transpose of a product is the product of the transposes in reverse order, so \(\mat{A}\T = \mat{V}\mat{\Sigma}\T\mat{U}\T\). For a square matrix \(\mat{\Sigma}\) is diagonal, so \(\mat{\Sigma}\T = \mat{\Sigma}\). Then

\[ \begin{aligned} \mat{A}\T\mat{A} &= \mat{V}\mat{\Sigma}\,\mat{U}\T\mat{U}\,\mat{\Sigma}\mat{V}\T \\ &= \mat{V}\mat{\Sigma}^2\mat{V}\T , \end{aligned} \]

because \(\mat{U}\T\mat{U} = \mat{I}\). This is exactly the form \({\mat{Q}\mat{\Lambda}\mat{Q}\T}\) of the spectral theorem (M03.3.2): orthonormal eigenvectors \(\vect{v}_i\), eigenvalues \(\sigma_i^2\). Multiplying \(\mat{A}\T = \mat{V}\mat{\Sigma}\mat{U}\T\) by \(\mat{U}\) also proves the claim of the intuition, column by column: \(\mat{A}\T\vect{u}_i = \sigma_i\vect{v}_i\).

Step 2. Turn it round. Step 1 assumed the SVD. To find it, start from \(\mat{A}\T\mat{A}\), which we can always form, and check that it has what the spectral theorem needs.

  • It is symmetric: \({(\mat{A}\T\mat{A})\T} = \mat{A}\T(\mat{A}\T)\T = \mat{A}\T\mat{A}\).
  • It is positive semidefinite (M03.4.2): for every \(\vect{x}\),

\[ \begin{aligned} \vect{x}\T\mat{A}\T\mat{A}\vect{x} &= (\mat{A}\vect{x}) \cdot (\mat{A}\vect{x}) \\ &= \norm{\mat{A}\vect{x}}^2 \ge 0 . \end{aligned} \]

So it has orthonormal eigenvectors \(\vect{v}_i\) and eigenvalues \(\lambda_i \ge 0\).

Step 3. The lengths come for free. For a unit eigenvector,

\[ \begin{aligned} \norm{\mat{A}\vect{v}_i}^2 &= \vect{v}_i\T\mat{A}\T\mat{A}\vect{v}_i \\ &= \lambda_i\,\vect{v}_i\T\vect{v}_i = \lambda_i , \end{aligned} \]

so the stretch is \(\sigma_i = \sqrt{\lambda_i}\), and \(\vect{u}_i = \mat{A}\vect{v}_i/\sigma_i\) has length one. The four steps of the recipe are below.

A column of four numbered orange boxes joined by green downward arrows, each with its reason written to the right. Box 1: form A transpose A; reason: symmetric, and x transpose A transpose A x equals the squared length of A x, at least 0. Box 2: eigenvectors v i, eigenvalues lambda i; reason: spectral theorem, orthonormal v i, all lambda i at least 0. Box 3: sigma i equals the square root of lambda i; reason: the squared length of A v i equals v i transpose A transpose A v i, which is lambda i. Box 4: u i equals A v i divided by sigma i; reason: unit length, and perpendicular to each other.
M03.5.2-F2

The recipe for an SVD, with the reason each step works.

The formal statement

NoteTheorem: every matrix has a singular value decomposition

Let \(\mat{A}\) be any real \(m\) by \(n\) matrix. Let \(\vect{v}_1, \dots, \vect{v}_n\) be orthonormal eigenvectors of \(\mat{A}\T\mat{A}\) with eigenvalues \({\lambda_1 \ge \dots \ge \lambda_n \ge 0}\), and put \(\sigma_i = \sqrt{\lambda_i}\). Then for each \(\sigma_i > 0\) the vectors \(\vect{u}_i = \mat{A}\vect{v}_i/\sigma_i\) are orthonormal, \(\mat{A}\vect{v}_i = \sigma_i\vect{u}_i\), and \(\mat{A}\vect{v}_i = \vect{0}\) whenever \(\sigma_i = 0\).

Proof. The eigenvectors and the non-negative eigenvalues exist by Step 2. Take two indices \(i\) and \(j\) with \(\sigma_i, \sigma_j > 0\). Move \(\mat{A}\) across the dot product as a transpose:

\[ \begin{aligned} \vect{u}_i \cdot \vect{u}_j &= \frac{(\mat{A}\vect{v}_i) \cdot (\mat{A}\vect{v}_j)}{\sigma_i\sigma_j} \\ &= \frac{\vect{v}_i\T\mat{A}\T\mat{A}\vect{v}_j}{\sigma_i\sigma_j} \\ &= \frac{\lambda_j\,\vect{v}_i \cdot \vect{v}_j}{\sigma_i\sigma_j} . \end{aligned} \]

If \(i \ne j\), then \({\vect{v}_i \cdot \vect{v}_j} = 0\) and the \(\vect{u}\)’s are perpendicular. If \(i = j\), the fraction is \(\lambda_i / \sigma_i^2 = 1\), so each has length one. Finally, if \(\sigma_i = 0\), Step 3 gives \(\norm{\mat{A}\vect{v}_i}^2 = 0\), so \(\mat{A}\vect{v}_i = \vect{0}\). \(\blacksquare\)

For a square matrix with no zero singular value this is the whole SVD: the \(\vect{u}_i\) fill the columns of \(\mat{U}\), and \(\mat{A}\mat{V} = \mat{U}\mat{\Sigma}\) column by column, so \(\mat{A} = \mat{U}\mat{\Sigma}\mat{V}\T\) as in lesson M03.5.1. The proof also answers the question of lesson 1 for every size at once: the perpendicular inputs that stay perpendicular are the eigenvectors of \(\mat{A}\T\mat{A}\), because the dot product of two images is \(\vect{v}_i\T\mat{A}\T\mat{A}\vect{v}_j\).

Go deeper: the general case: zero singular values and rectangular matrices

When some \(\sigma_i\) are zero, or \(m \ne n\), there are fewer \(\vect{u}_i\) than the \(m\) columns of \(\mat{U}\) need. Say \(r\) singular values are positive. The vectors \(\vect{u}_1, \dots, \vect{u}_r\) are orthonormal in \(\R^m\), so Gram-Schmidt (M02.10.3), applied to them followed by the standard basis, extends them to an orthonormal basis \(\vect{u}_1, \dots, \vect{u}_m\) of \(\R^m\). Let \(\mat{U}\) hold all \(m\) of them, \(\mat{V}\) all \(n\) of the \(\vect{v}_i\), and let \(\mat{\Sigma}\) be the \(m\) by \(n\) matrix with \(\sigma_1, \dots, \sigma_r\) down its diagonal and zeros elsewhere. Check \(\mat{A}\mat{V} = \mat{U}\mat{\Sigma}\) column by column. For \(i \le r\), column \(i\) is \(\mat{A}\vect{v}_i = \sigma_i\vect{u}_i\) on both sides. For \(i > r\), the left side is \(\mat{A}\vect{v}_i = \vect{0}\), and column \(i\) of \(\mat{\Sigma}\) is zero, so the right side is \(\vect{0}\) too. Multiplying on the right by \(\mat{V}\T\) gives \(\mat{A} = \mat{U}\mat{\Sigma}\mat{V}\T\). The extra \(\vect{u}\)’s are never used by \(\mat{A}\); lesson M03.5.3 shows they span the left null space. Axler proves the same theorem for linear maps between any two inner product spaces (results 7.70 and 7.80), by this very route.

Worked example

Find the SVD of \(\mat{A} = \begin{bmatrix} 3 & 0 \\ 4 & 5 \end{bmatrix}\) through \(\mat{A}\T\mat{A}\).

Form \(\mat{A}\T\mat{A}\). Its entries are dot products of the columns \((3, 4)\) and \((0, 5)\): \(9 + 16 = 25\), \(0 + 20 = 20\) and \(0 + 25 = 25\).

\[ \mat{A}\T\mat{A} = \begin{bmatrix} 25 & 20 \\ 20 & 25 \end{bmatrix}. \]

Eigenvalues. The trace is \(50\) and the determinant is \(625 - 400 = 225\), so the characteristic polynomial (M03.1.2) is

\[ \lambda^2 - 50\lambda + 225 = (\lambda - 45)(\lambda - 5), \]

because \(45 + 5 = 50\) and \({45 \cdot 5} = 225\). So \(\lambda_1 = 45\) and \(\lambda_2 = 5\).

Eigenvectors. For \(\lambda_1 = 45\), the matrix \(\mat{A}\T\mat{A} - 45\mat{I}\) has rows \((-20, 20)\) and \((20, -20)\), so \(x = y\) and \(\vect{v}_1 = (1, 1)/\sqrt{2}\). For \(\lambda_2 = 5\), the rows are \((20, 20)\) twice, so \(y = -x\) and \(\vect{v}_2 = (-1, 1)/\sqrt{2}\).

Singular values and left vectors. \(\sigma_1 = \sqrt{45} = 3\sqrt{5} \approx 6.708\) and \(\sigma_2 = \sqrt{5} \approx 2.236\). Dividing the images by them, as in lesson 1:

\[ \begin{aligned} \vect{u}_1 &= \frac{1}{3\sqrt{5}} \cdot \frac{(3,\, 9)}{\sqrt{2}} = \frac{(1,\, 3)}{\sqrt{10}}, \\ \vect{u}_2 &= \frac{1}{\sqrt{5}} \cdot \frac{(-3,\, 1)}{\sqrt{2}} = \frac{(-3,\, 1)}{\sqrt{10}} . \end{aligned} \]

No search was needed, and the answer matches lesson M03.5.1.

A trace of five numbered boxes joined by green arrows. Box 1, blue: A transpose A equals the matrix with rows 25, 20 and 20, 25. Box 2, blue: lambda squared minus 50 lambda plus 225 equals (lambda minus 45)(lambda minus 5). Box 3, orange: lambda1 equals 45 with v1 equal to (1, 1) over root 2, and lambda2 equals 5 with v2 equal to (minus 1, 1) over root 2. Box 4, orange: sigma1 equals root 45 equals 3 root 5, about 6.708, and sigma2 equals root 5, about 2.236. Box 5, orange: u1 equals A v1 over sigma1 equals (1, 3) over root 10, and u2 equals A v2 over sigma2 equals (minus 3, 1) over root 10. A note checks that the two images have dot product (minus 3 plus 3) over 10, which is 0.
M03.5.2-F3

The worked example as a trace: one symmetric matrix, its eigenvalues, its eigenvectors, then square roots and one division each.

Code

From scratch: the recipe in five lines on top of numpy.linalg.eigh, which returns the eigenvalues of a symmetric matrix in ascending order, so they must be reversed.

import numpy as np

def svd_via_gram(A):
    """SVD of a matrix with independent columns, through the eigenvectors of A^T A."""
    lam, Q = np.linalg.eigh(A.T @ A)          # ascending eigenvalues, orthonormal columns
    order = np.argsort(lam)[::-1]             # largest first
    lam, V = lam[order], Q[:, order]
    s = np.sqrt(np.clip(lam, 0, None))        # clip a tiny negative rounding error to zero
    U = (A @ V) / s                           # column i is A v_i / sigma_i
    return U, s, V

A = np.array([[3.0, 0.0],
              [4.0, 5.0]])
U, s, V = svd_via_gram(A)
print("A^T A =\n", A.T @ A)
print("sigma:", s, "  sigma squared:", np.round(s ** 2, 10))
print("U^T U =\n", np.round(U.T @ U, 12))
print("U Sigma V^T =\n", np.round(U @ np.diag(s) @ V.T, 12))
A^T A =
 [[25. 20.]
 [20. 25.]]
sigma: [6.70820393 2.23606798]   sigma squared: [45.  5.]
U^T U =
 [[1. 0.]
 [0. 1.]]
U Sigma V^T =
 [[3. 0.]
 [4. 5.]]

The same code works for a tall matrix with independent columns, where \(\mat{U}\) comes out with orthonormal columns but is not square (lesson M03.5.3). Now the library, which returns \(\mat{V}\T\) as Vh and the singular values in descending order:

rng = np.random.default_rng(0)
M = rng.normal(size=(5, 3))
U_m, s_m, V_m = svd_via_gram(M)
U_np, s_np, Vh = np.linalg.svd(M, full_matrices=False)
print("from scratch:", np.round(s_m, 6))
print("numpy:       ", np.round(s_np, 6))
print("column-by-column agreement of v (up to sign):", np.round(np.abs(np.sum(V_m * Vh.T, axis=0)), 6))
from scratch: [3.150902 1.70933  0.334855]
numpy:        [3.150902 1.70933  0.334855]
column-by-column agreement of v (up to sign): [1. 1. 1.]

The singular values agree to every printed digit. The vectors agree up to sign: each column of \(\mat{V}\) may be flipped, as long as the matching column of \(\mat{U}\) flips with it, because the product \(\sigma_i\vect{u}_i\vect{v}_i\T\) does not change. For the running \(\mat{A}\), NumPy may well return \(-\vect{v}_1\) and \(-\vect{u}_1\). Which sign comes out depends on the LAPACK library underneath, so code should never rely on it.

Two outer products side by side with an equals sign between them. Left, titled One choice: a column u1 with entries 0.316 and 0.949, a row v1 transpose with entries 0.707 and 0.707, and where they meet the product sigma1 u1 v1 transpose, a 2 by 2 grid with rows 1.5, 1.5 and 4.5, 4.5, shaded vermillion. Right, titled Both signs flipped: a column minus u1 with entries minus 0.316 and minus 0.949 and a row minus v1 transpose with entries minus 0.707, both shaded sky blue for negative, and the same product grid 1.5, 1.5, 4.5, 4.5.
M03.5.2-F4

Why the signs may differ: flipping both \(\vect{u}_1\) and \(\vect{v}_1\) leaves the product \(\sigma_1\vect{u}_1\vect{v}_1\T\), and so the matrix, unchanged.

Explore

The module’s interactive, Rotate, stretch, rotate, is in lesson M03.5.1. Pick a matrix there, note \(\sigma_1\) and \(\sigma_2\), and check by hand that \(\sigma_1^2\) and \(\sigma_2^2\) are the eigenvalues of \(\mat{A}\T\mat{A}\): their sum is the sum of the squares of the four entries, and their product is \({(\det\mat{A})^2}\).

Pitfalls

Two log-log graphs side by side of the relative error in the computed sigma2 against epsilon, the true sigma2, for the 3 by 2 matrix with rows (1, 1), (epsilon, 0) and (0, epsilon), as epsilon falls from 0.1 to 10 to the minus 12. Left, marked with a cross and titled Through A transpose A: the reddish-purple error curve climbs steadily from about 10 to the minus 16 and reaches 100 percent wrong, a dashed line at 1, once epsilon is below about 10 to the minus 8, with the note 1 plus epsilon squared rounds to 1 here. Right, marked with a tick and titled SVD of A itself: the green error curve stays flat near 10 to the minus 16 for every epsilon, with the note error stays near machine precision.
M03.5.2-F5

Wrong beside right. Through \(\mat{A}\T\mat{A}\) the small singular value loses accuracy fast and is lost entirely below \(\varepsilon \approx 10^{-8}\); the SVD of \(\mat{A}\) itself keeps it.

“To compute an SVD, form \(\mat{A}\T\mat{A}\) and take its eigenvectors.”

That is how we prove the SVD exists, not how to compute it. Forming \(\mat{A}\T\mat{A}\) squares every singular value, and a small one can sink below rounding error. Take the matrix with rows \((1, 1)\), \((\varepsilon, 0)\) and \((0, \varepsilon)\), with \(\varepsilon = 10^{-9}\). Its exact \(\mat{A}\T\mat{A}\) has diagonal entries \(1 + \varepsilon^2 = 1 + 10^{-18}\), which double precision stores as exactly \(1\). The stored matrix has rows \((1, 1)\) twice, whose smaller eigenvalue is exactly \(0\), so whatever tiny number the computer returns, \(\sigma_2 = 10^{-9}\) cannot be recovered from it. numpy.linalg.svd works on \(\mat{A}\) directly and finds it to about fifteen digits. Lesson M03.8.2 explains this as the condition number being squared.

“Take \(\mat{V}\) from \(\mat{A}\T\mat{A}\) and \(\mat{U}\) from \(\mat{A}\mat{A}\T\), separately.”

The \(\vect{u}_i\) are indeed eigenvectors of \(\mat{A}\mat{A}\T\), but an eigenvector is only fixed up to sign. Found independently, \(\vect{u}_1\) may come out as \(-(1, 3)/\sqrt{10}\) while \(\vect{v}_1 = (1, 1)/\sqrt{2}\), and then \(\mat{U}\mat{\Sigma}\mat{V}\T\) is not \(\mat{A}\). Compute \(\vect{u}_i = \mat{A}\vect{v}_i/\sigma_i\), which ties each sign to its partner.

“eigh lists the largest eigenvalue first.”

numpy.linalg.eigh returns eigenvalues in ascending order, while numpy.linalg.svd returns singular values in descending order. Forgetting to reverse gives \(\sigma_1 = \sqrt{5}\) and swaps the long and short axes of the ellipse.

Check yourself

Recall

1. How are the singular values of \(\mat{A}\) related to the eigenvalues of \(\mat{A}\T\mat{A}\)?

Show answer

They are their non-negative square roots: \(\sigma_i = \sqrt{\lambda_i}\).

2. Why are the eigenvalues of \(\mat{A}\T\mat{A}\) never negative?

Show answer

For a unit eigenvector, \(\lambda_i = \vect{v}_i\T\mat{A}\T\mat{A}\vect{v}_i = \norm{\mat{A}\vect{v}_i}^2 \ge 0\).

3. Once \(\vect{v}_i\) and \(\sigma_i > 0\) are known, how do you find \(\vect{u}_i\)?

Show answer

\(\vect{u}_i = \mat{A}\vect{v}_i / \sigma_i\).

4. What does \(\mat{A}\T\) do to \(\vect{u}_i\)?

Show answer

It sends it to \(\sigma_i\vect{v}_i\), because \(\mat{A}\T = \mat{V}\mat{\Sigma}\mat{U}\T\).

5. Why do library routines not compute an SVD by forming \(\mat{A}\T\mat{A}\)?

Show answer

Forming it squares the singular values, so small ones are lost to rounding; for example \(1 + 10^{-18}\) is stored as \(1\).

Exercises

1. ★ Find the SVD of \(\mat{A} = \begin{bmatrix} 2 & 2 \\ 1 & -1 \end{bmatrix}\) through \(\mat{A}\T\mat{A}\).

Show solution

The columns are \((2, 1)\) and \((2, -1)\), so \(\mat{A}\T\mat{A}\) has entries \(4 + 1 = 5\), \(4 - 1 = 3\) and \(5\): rows \((5, 3)\) and \((3, 5)\). Trace \(10\), determinant \(16\), so \(\lambda^2 - 10\lambda + 16 = {(\lambda - 8)(\lambda - 2)}\). For \(\lambda = 8\), \(x = y\) and \(\vect{v}_1 = (1, 1)/\sqrt{2}\); for \(\lambda = 2\), \(y = -x\) and \(\vect{v}_2 = (1, -1)/\sqrt{2}\). So \(\sigma_1 = 2\sqrt{2}\) and \(\sigma_2 = \sqrt{2}\). Then \(\mat{A}\vect{v}_1 = (4, 0)/\sqrt{2}\), so \(\vect{u}_1 = (1, 0)\), and \(\mat{A}\vect{v}_2 = (0, 2)/\sqrt{2}\), so \(\vect{u}_2 = (0, 1)\). Check: \(\mat{U} = \mat{I}\), and \(\mat{\Sigma}\mat{V}\T\) has rows \({2\sqrt{2}\,(1, 1)/\sqrt{2}} = (2, 2)\) and \({\sqrt{2}\,(1, -1)/\sqrt{2}} = (1, -1)\), which is \(\mat{A}\).

2. ★★ Show that \(\mat{A}\T\mat{A}\) and \(\mat{A}\mat{A}\T\) have the same non-zero eigenvalues, even when \(\mat{A}\) is not square.

Show solution

Let \(\mat{A}\T\mat{A}\vect{v} = \lambda\vect{v}\) with \(\vect{v} \ne \vect{0}\) and \(\lambda \ne 0\). Multiply on the left by \(\mat{A}\): \(\mat{A}\mat{A}\T(\mat{A}\vect{v}) = \lambda(\mat{A}\vect{v})\). The vector \(\mat{A}\vect{v}\) is not zero, because \(\norm{\mat{A}\vect{v}}^2 = \vect{v}\T\mat{A}\T\mat{A}\vect{v} = \lambda\norm{\vect{v}}^2 \ne 0\). So \(\lambda\) is an eigenvalue of \(\mat{A}\mat{A}\T\), with eigenvector \(\mat{A}\vect{v}\). The same argument with \(\mat{A}\T\) in place of \(\mat{A}\) gives the other direction. For the running \(\mat{A}\), \(\mat{A}\mat{A}\T\) has rows \((9, 12)\) and \((12, 41)\): trace \(50\), determinant \(225\), eigenvalues \(45\) and \(5\) again.

3. ★★★ Let \(\mat{Q}\) be orthogonal. Prove that \(\mat{Q}\mat{A}\) and \(\mat{A}\mat{Q}\) have the same singular values as \(\mat{A}\), and say what that means for the ellipse.

Show solution

For \(\mat{Q}\mat{A}\): \({(\mat{Q}\mat{A})\T(\mat{Q}\mat{A})} = \mat{A}\T\mat{Q}\T\mat{Q}\mat{A} = \mat{A}\T\mat{A}\), the same matrix, so the same eigenvalues and singular values. For \(\mat{A}\mat{Q}\): \({(\mat{A}\mat{Q})\T(\mat{A}\mat{Q})} = \mat{Q}\T(\mat{A}\T\mat{A})\mat{Q}\). If \(\mat{A}\T\mat{A}\vect{v} = \lambda\vect{v}\), then \(\mat{Q}\T\mat{A}\T\mat{A}\mat{Q}(\mat{Q}\T\vect{v}) = \mat{Q}\T\mat{A}\T\mat{A}\vect{v} = \lambda\mat{Q}\T\vect{v}\), using \(\mat{Q}\mat{Q}\T = \mat{I}\), so \(\lambda\) is again an eigenvalue, with eigenvector \(\mat{Q}\T\vect{v}\). In pictures: turning the plane after \(\mat{A}\) turns the ellipse without changing its axes’ lengths, and turning before \(\mat{A}\) only relabels which inputs reach them.

Summary card

The right singular vectors are the eigenvectors of \(\mat{A}\T\mat{A}\), and \(\sigma_i^2\) are its eigenvalues.

The anchor figure again: unit arrows v1 and v2 on the left, and on the right their images under A transpose A, 45 v1 and 5 v2, on the same pale lines.
M03.5.2-F1

The round trip keeps each \(\vect{v}_i\) on its line.

  • Key identity. \(\mat{A}\T\mat{A} = \mat{V}\mat{\Sigma}^2\mat{V}\T\), symmetric and semidefinite.
  • Recipe. Eigenvectors \(\vect{v}_i\), then \(\sigma_i = \sqrt{\lambda_i}\), then \(\vect{u}_i = \mat{A}\vect{v}_i/\sigma_i\).
  • Existence. The \(\vect{u}_i\) are orthonormal because \(\vect{v}_i\T\mat{A}\T\mat{A}\vect{v}_j = \lambda_j\,\vect{v}_i \cdot \vect{v}_j\).
  • Running example. \(\mat{A}\T\mat{A}\) has rows \((25, 20)\), \((20, 25)\); eigenvalues \(45\) and \(5\).
  • In practice. Call numpy.linalg.svd; never form \(\mat{A}\T\mat{A}\) for accuracy.

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • Sheldon Axler, Linear Algebra Done Right, 4th edition, Section 7E: 7.64 (\(T^*T\) is a positive operator), 7.65 (singular values are the non-negative square roots of the eigenvalues of \(T^*T\)), 7.70 (the SVD, proved through the spectral theorem for \(T^*T\) and \(f_k = Te_k / s_k\), exactly the proof above) and 7.80 (the matrix version).
  • Marc Peter Deisenroth, A. Aldo Faisal and Cheng Soon Ong, Mathematics for Machine Learning, Section 4.5, equations (4.71) to (4.79): \(\mat{A}\T\mat{A} = \mat{V}\mat{\Sigma}\T\mat{\Sigma}\mat{V}\T\), \(\sigma_i^2 = \lambda_i\), the orthogonality of the \(\mat{A}\vect{v}_i\), and \(\vect{u}_i = \mat{A}\vect{v}_i/\sigma_i\).
  • David Austin, Understanding Linear Algebra, section “Singular value decompositions”: singular values as the square roots of the eigenvalues of the Gram matrix \(\mat{A}\T\mat{A}\), whose eigenvectors are the right singular vectors.
  • NumPy reference, numpy.linalg.eigh (eigenvalues in ascending order) and numpy.linalg.svd (singular values in descending order, Vh returned, full_matrices option).

Next. M03.5.3 Rectangular matrices and the four subspaces applies the recipe to a 3 by 2 matrix and reads the four fundamental subspaces off \(\mat{U}\) and \(\mat{V}\).

This lesson unlocks

M03.5.3 Rectangular matrices and the four subspaces

Your progress is saved in this browser only. No account, no tracking.
Back to top