Purpose. This is the first of two review lectures before the final. The organizing picture is the four fundamental subspaces of an \(m\times n\) matrix of rank \(r\). The question is: what kind of inverse does \(A\) have in each case?
Exam scope. Strang notes the pseudoinverse (Section 7.4) won't appear by name on the final. Everything that leads to it will.
1. Four cases, organized by rank
| Case | Rank | Nullspaces | Inverse |
|---|---|---|---|
| Square, full rank | \(r = m = n\) | both \(\{0\}\) | two-sided \(A^{-1}\) |
| Full column rank | \(r = n < m\) | \(N(A) = \{0\}\) | left inverse |
| Full row rank | \(r = m < n\) | \(N(A^\top) = \{0\}\) | right inverse |
| General | \(r < m,\ r < n\) | both nontrivial | pseudoinverse \(A^+\) |
Why a rectangular matrix can't have a two-sided inverse. If \(m \neq n\), then either \(A\) or \(A^\top\) has free variables, so one of them has a nontrivial nullspace. A vector sent to zero can't be brought back.
2. Full column rank: left inverse
Setting. \(r = n\), with more rows than columns (\(m > n\)).
Properties.
- The columns are independent, so \(N(A) = \{0\}\).
- \(Ax = b\) has 0 or 1 solutions. The rows are probably dependent, so some \(b\) give no solution.
- \(A^\top A\) (\(n\times n\)) is invertible. This is the least-squares case.
The left inverse:
Other left inverses exist. This one is the natural choice.
Multiplying in the wrong order gives a projection, not \(I\):
3. Full row rank: right inverse
Setting. \(r = m\), with more columns than rows (\(n > m\)). This is the mirror image of the previous case.
Properties.
- The rows are independent, so \(N(A^\top) = \{0\}\).
- \(Ax = b\) is always solvable, because elimination never produces a zero row.
- There are infinitely many solutions, since there are \(n - m\) free variables.
- \(AA^\top\) (\(m\times m\)) is invertible.
The right inverse:
Wrong order:
4. The general case: \(A\) is invertible between the row space and the column space
The key observation.
- Every \(Ax\) lands in the column space.
- The row space and the column space both have dimension \(r\).
- Restricted to the row space, \(A\) is a perfect one-to-one map onto the column space.
Claim: if \(x \neq y\) are both in the row space, then \(Ax \neq Ay\).
Proof.
- Suppose \(Ax = Ay\). Then \(A(x - y) = 0\), so \(x - y \in N(A)\).
- But \(x\) and \(y\) are both in the row space, which is a subspace. So \(x - y\) is in the row space too.
- The row space and nullspace are orthogonal complements, so they share only the zero vector.
- Therefore \(x - y = 0\).
Onto as well as one-to-one. Both spaces have dimension \(r\), so every vector in the column space is reached.
How \(A\) treats a general vector. Most vectors have a row-space part and a nullspace part. \(A\) kills the nullspace part and maps the row-space part faithfully.
The pseudoinverse \(A^+\)
Definition. \(A^+\) is the inverse of that map, going back from the column space to the row space:
What it does to the rest. \(A^+\) kills \(N(A^\top)\), just as \(A\) kills \(N(A)\).
Why statisticians need it.
- Least squares uses \((A^\top A)^{-1}A^\top\), which requires full column rank.
- With repeated or dependent measurements, the columns of \(A\) are dependent and \(A^\top A\) is singular.
- The pseudoinverse is the right replacement in that situation.
5. Computing \(A^+\) from the SVD
Start from the SVD:
How each factor inverts.
- \(U\) and \(V\) are orthogonal, so their inverses are their transposes.
- All the difficulty sits in \(\Sigma\), and there it is easy to handle.
The pseudoinverse of \(\Sigma\)
\(\Sigma\) (\(m \times n\)) has \(\sigma_1, \dots, \sigma_r\) on the diagonal and zeros elsewhere.
\(\Sigma^+\) (\(n \times m\)) has \(1/\sigma_1, \dots, 1/\sigma_r\) on the diagonal and zeros elsewhere:
Products in both orders.
- \(\Sigma\Sigma^+\) is \(m\times m\), with \(r\) ones on the diagonal and then zeros. It projects onto the column space.
- \(\Sigma^+\Sigma\) is \(n\times n\), with \(r\) ones on the diagonal and then zeros. It projects onto the row space.
For \(A\) itself:
Interpretation. The pseudoinverse is never a true inverse on either side. It does the best possible thing: it carries vectors into the two "good" spaces and wipes out the nullspaces.
Why this \(\Sigma^+\) and not another. Extra nonzero entries in the zero block of \(\Sigma^+\) would multiply zeros and change nothing in \(\Sigma\Sigma^+\). The chosen \(\Sigma^+\) is the minimal one.
6. A small example (my addition)
Its SVD. Rank 1, with \(\sigma_1 = 2\) and \(u_1 = v_1 = \tfrac{1}{\sqrt2}(1, 1)\).
The pseudoinverse:
Check.
Big ideas from Lecture 33
- Rank decides which inverse exists:
- \(r = m = n\): two-sided.
- \(r = n\): left inverse \((A^\top A)^{-1}A^\top\).
- \(r = m\): right inverse \(A^\top(AA^\top)^{-1}\).
- otherwise: pseudoinverse.
- One-sided inverses in the wrong order give projections: onto \(C(A)\) for the left inverse, onto \(C(A^\top)\) for the right inverse.
- Restricted to the row space, \(A\) is an invertible map onto the column space. The proof is that \(N(A) \cap C(A^\top) = \{0\}\).
- \(A^+\) inverts that map and kills \(N(A^\top)\).
- Computation: \(A^+ = V\Sigma^+U^\top\), with \(\Sigma^+\) inverting only the nonzero singular values.
- Projections: \(AA^+\) and \(A^+A\) project onto the column space and the row space.
Connections (not in the lecture).
- Least squares. \(x^+ = A^+b\) is the minimum-norm least-squares solution. Among all minimizers of \(\|Ax - b\|\), it is the one with no nullspace component. In the full-rank cases it reduces to the left- or right-inverse formula.
- Ridge regression. As the regularization \(\lambda \to 0\), \((A^\top A + \lambda I)^{-1}A^\top \to A^+\).
- Overparameterized linear models. In overparameterized linear regression, gradient descent started from zero converges to exactly \(A^+b\). This is the standard "implicit bias toward minimum norm" result.