Format. Strang reviews by working through problems from old final exams. One of them is 26 years old, and he jokes that he hasn't learned anything since. Each problem deliberately connects several parts of the course. Where Strang leaves an answer as a formula, I've finished the arithmetic and marked it as mine.
Problem 1: What does solvability tell you about \(m\), \(n\), \(r\)?
Given. \(A\) is \(3\times n\). The equation \(Ax = (1,0,0)\) has no solution, and \(Ax = (0,1,0)\) has exactly one solution.
Deductions
| Fact | Conclusion |
|---|---|
| \(A\) has 3 rows | \(m = 3\) |
| Some \(b\) has no solution | not every row has a pivot, so \(r < m\) |
| A solution, when it exists, is unique | \(N(A) = \{0\}\), so the columns are independent and \(r = n\) |
Together: \(r = n < m = 3\).
An example
Simplest. \(A = (0,1,0)^\top\), a single column. Then \(r = n = 1\).
With two columns. Add a column such as \((0,0,1)\):
True or false
| Statement | Answer | Reason |
|---|---|---|
| \(\det(A^\top A) = \det(AA^\top)\) | False | \(A^\top A\) is invertible but \(AA^\top\) is singular. \(\det(AB) = \det(BA)\) only holds for square matrices. |
| \(A^\top A\) is invertible | True | \(r = n\), so the columns are independent. In the example, \(A^\top A = I_2\). |
| \(AA^\top\) is positive definite | False | It is \(3\times3\) with rank \(n < 3\). In the example, \(AA^\top = \text{diag}(0,1,1)\). It is always positive semidefinite. |
\(A^\top y = c\) has infinitely many solutions for every \(c\)
At least one solution. \(A^\top\) has \(n\) rows and rank \(n\), so it has full row rank. That means it is always solvable.
Infinitely many. \(\dim N(A^\top) = m - r = 3 - n > 0\). Any null vector of \(A^\top\) can be added to a solution.
Problem 2: Thinking in columns
\(A\) has columns \(v_1, v_2, v_3\).
(a) Solve \(Ax = v_1 - v_2 + v_3\). \(x = (1, -1, 1)\). This is matrix multiplication done a column at a time, the very first idea of the course.
(b) If \(v_1 - v_2 + v_3 = 0\), are solutions unique? No. \((1, -1, 1)\) is a nonzero vector in \(N(A)\), and it can be added to any solution.
(c) If \(v_1, v_2, v_3\) are orthonormal, which combination of \(v_1, v_2\) is closest to \(v_3\)? \(0v_1 + 0v_2\).
- \(v_3\) is perpendicular to the plane of \(v_1\) and \(v_2\), so its projection onto that plane is the origin.
- Picture the \(x\), \(y\), \(z\) axes: the point of the \(xy\)-plane closest to the \(z\)-axis vector is \(0\).
Problem 3: A Markov matrix
Key observation. Column 1 + column 2 = \(2\times\) column 3, so \(A\) is singular.
Eigenvalues, without a determinant
- \(\lambda = 0\), because \(A\) is singular.
- \(\lambda = 1\), because \(A\) is a Markov matrix: its columns sum to 1.
- \(\lambda = -0.2\), from the trace: \(0.8 - 0 - 1\).
Steady state from \(u_0 = (0, 10, 0)\)
General form after \(k\) steps:
The eigenvector for \(\lambda = 1\). Strang finds it partly by guessing: \(x_2 = (3, 3, 4)\). He celebrates when it works.
Check against \(A - I\):
- Rows 1 and 2: \(-2.4 + 1.2 + 1.2 = 0\).
- Row 3: \(1.2 + 1.2 - 2.4 = 0\).
The coefficient \(c_2\). A Markov process conserves the total. \(u_0\) sums to 10, and \(x_2\) already sums to 10, so \(c_2 = 1\).
Problem 4: Four quick \(2\times2\) constructions
(1) Projection onto the line through \(a = (4, -3)\)
(2) The matrix with \(\lambda = 0\) on \((1,2)\) and \(\lambda = 3\) on \((2,1)\)
Formula: \(A = S\Lambda S^{-1}\), with \(S = \begin{bmatrix} 1 & 2 \\ 2 & 1\end{bmatrix}\) and \(\Lambda = \text{diag}(0, 3)\).
Computed out (mine):
Check.
- \(A(1,2) = (0, 0)\), which is \(0 \cdot (1,2)\).
- \(A(2,1) = (6, 3)\), which is \(3 \cdot (2,1)\).
(3) A real matrix that can never equal \(B^\top B\)
Strang's answer. \(B^\top B\) is always symmetric, so any non-symmetric matrix works. Example: \(\begin{bmatrix} 0 & 1 \\ 0 & 0 \end{bmatrix}\).
A sharper version (mine). \(B^\top B\) is also always positive semidefinite. So even a symmetric matrix like \(\text{diag}(-1, 0)\) can't be written as \(B^\top B\).
(4) Orthogonal eigenvectors, but not symmetric
Use either:
- a skew-symmetric matrix, such as \(\begin{bmatrix} 0 & -1 \\ 1 & 0\end{bmatrix}\), or
- an orthogonal (rotation) matrix, such as \(\begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta\end{bmatrix}\).
Their eigenvectors are complex but orthogonal. This is the "normal matrix" fact from the Quiz 3 review.
Problem 5: Least squares, answer given ("like Jeopardy")
Verifying \(\hat{x}\) (mine). \(A^\top A = \begin{bmatrix} 3 & 3 \\ 3 & 5 \end{bmatrix}\) and \(A^\top b = (8, 6)\). Solving gives \(\hat{d} = -1\), \(\hat{c} = \tfrac{11}{3}\).
(1) The projection \(p\) of \(b\) onto \(C(A)\)
The point. Least squares finds exactly the combination of the columns closest to \(b\). So \(p = A\hat{x}\).
(2) The line-fitting picture
Data points: \((t, b) = (0, 3),\ (1, 4),\ (2, 1)\).
Best line: \(b = \tfrac{11}{3} - t\). It has intercept \(\tfrac{11}{3}\) and slope \(-1\).
(3) A nonzero \(b\) whose least-squares solution is \(\hat{x} = 0\)
What's needed. The best combination of the columns must be zero. That happens when \(b\) is perpendicular to the column space, meaning \(b \in N(A^\top)\).
Example: \(b = (1, -2, 1)\).
Check.
- Against column 1, \((1,1,1)\): \(1 - 2 + 1 = 0\).
- Against column 2, \((0,1,2)\): \(0 - 2 + 2 = 0\).
This completes the course. The final question ties least squares back to the four subspaces.
The whole course in one table
| Theme | Core result |
|---|---|
| \(Ax = b\) | elimination, \(A = LU\); solvable when \(b \in C(A)\) |
| Four subspaces | dimensions \(r\), \(n - r\), \(r\), \(m - r\); row space ⊥ nullspace, column space ⊥ left nullspace |
| Orthogonality | projections \(P = A(A^\top A)^{-1}A^\top\), least squares, \(A = QR\) |
| Determinants | three defining properties; \(\det = \pm\) product of pivots = product of eigenvalues |
| Eigenvalues | \(A = S\Lambda S^{-1}\), \(A^k\), \(e^{At}\), Markov matrices |
| Symmetric | \(A = Q\Lambda Q^\top\); positive definite has four equivalent tests |
| Every matrix | SVD \(A = U\Sigma V^\top\); pseudoinverse \(A^+ = V\Sigma^+U^\top\) |
Strang closes by thanking the class for its patience during the videotaping and for taking the course.
That's all of 18.06 (Spring 2010), Lectures 1–34. We started from your MLT lecture on PCA. Every piece of it has now appeared:
- projection onto a subspace,
- the eigenvectors of the symmetric matrix \(C\),
- the spectral theorem,
- the SVD as the practical way to compute it.