Lecture 15: Projections onto Subspaces

Goal: Project a vector \(b\) onto a subspace. That means finding the point in the subspace closest to \(b\).

Why: when \(Ax = b\) has no solution, replace \(b\) by its projection onto \(C(A)\) and solve that instead.

1. Projection onto a line (1-D case)

Setup. Project \(b\) onto the line through a vector \(a\).

The projection \(p\) lies on the line, so \(p = \hat{x}\,a\) for some number \(\hat{x}\).
On the board Strang says "closest to \(a\)"; he means the point on the line closest to \(b\).

The key fact is a right angle. The error \(e = b - p\) must be perpendicular to \(a\):

a^\top(b - \hat{x}\,a) = 0 \quad\Longrightarrow\quad \hat{x}\,a^\top a = a^\top b

Three formulas to remember:

\hat{x} = \frac{a^\top b}{a^\top a}, \qquad p = a\,\frac{a^\top b}{a^\top a}, \qquad P = \frac{a\,a^\top}{a^\top a}

No angles needed. The cosine is built into \(\hat{x}\). Strang notes that trigonometry with \(\sin\theta\) and \(\cos\theta\) gives much messier formulas than linear algebra.

Sanity checks

Doubling \(b\) doubles \(p\). The formula is linear in \(b\).

Doubling \(a\), or replacing it with \(-a\), leaves \(p\) unchanged. The line is the same. In the formula, the factor 4 above cancels the factor 4 below.

The projection matrix \(P = \dfrac{aa^\top}{a^\top a}\)

This is an \(n \times n\) matrix: a column times a row, divided by a number. It satisfies \(p = Pb\).

Column space. \(C(P)\) is the line through \(a\). Multiplying anything by \(P\) lands on that line.

Rank 1. \(P\) is a column times a row.

Symmetric. \(P^\top = P\).

Idempotent. \(P^2 = P\). Projecting a second time leaves the point where it is, because it is already on the line.

A symmetric matrix with \(P^2 = P\) is the signature of a projection matrix.

2. Why project? The unsolvable \(Ax = b\)

The problem. \(Ax\) always lies in \(C(A)\), but \(b\) usually doesn't, especially with more equations than unknowns.

The fix. Replace \(b\) by \(p\), its projection onto \(C(A)\), which is the closest vector in the column space. Then solve

A\hat{x} = p

This equation is solvable, because \(p\) lies in \(C(A)\). The hat marks the best possible \(x\), not the \(x\) that doesn't exist.

3. Projection onto a subspace (general case)

Setup. The subspace is spanned by independent vectors \(a_1, \dots, a_n\). These need not be perpendicular, only independent.

Put them as the columns of \(A\). The subspace is then \(C(A)\).

The projection is a combination of the columns:

p = \hat{x}_1 a_1 + \cdots + \hat{x}_n a_n = A\hat{x}

The key equation

The error is perpendicular to the whole subspace. It suffices to check that \(e = b - A\hat{x}\) is perpendicular to each basis vector:

a_1^\top(b - A\hat{x}) = 0, \;\; \dots, \;\; a_n^\top(b - A\hat{x}) = 0

In matrix form. The rows of \(A^\top\) are \(a_1^\top, \dots, a_n^\top\), so these equations combine into

A^\top(b - A\hat{x}) = 0

Connection to the four subspaces

Which subspace is \(e\) in? The equation \(A^\top e = 0\) says \(e \in N(A^\top)\).

That matches the geometry. From Lecture 14, \(N(A^\top) \perp C(A)\). So "\(e\) is in the left nullspace" says the same thing as "\(e\) is perpendicular to the column space", which is exactly what we required.

The three formulas

Normal equations:

A^\top A\,\hat{x} = A^\top b

The answers:

\hat{x} = (A^\top A)^{-1}A^\top b, \qquad p = A(A^\top A)^{-1}A^\top b, \qquad P = A(A^\top A)^{-1}A^\top

Comparison with the 1-D case:

1-D (line through \(a\))	General (column space of \(A\))
\(\hat{x} = \dfrac{a^\top b}{a^\top a}\)	\(\hat{x} = (A^\top A)^{-1}A^\top b\)
\(P = \dfrac{aa^\top}{a^\top a}\)	\(P = A(A^\top A)^{-1}A^\top\)

In higher dimensions you can't divide by a matrix, so division becomes multiplication by \((A^\top A)^{-1}\). This inverse exists because the columns of \(A\) are independent (Lecture 14).

A tempting mistake

The wrong move. It might seem natural to split the inverse:

(A^\top A)^{-1} \overset{?}{=} A^{-1}(A^\top)^{-1}

That would give \(P = AA^{-1}(A^\top)^{-1}A^\top = I\).

Why it's wrong. \(A\) is rectangular, so \(A^{-1}\) does not exist. Only the square matrix \(A^\top A\) has an inverse.

The one case where the split works. If \(A\) is square and invertible, then \(C(A) = \mathbb{R}^n\). Projecting onto the whole space changes nothing, so \(P = I\) is correct there.

Properties still hold

Symmetric: \(P^\top = P\). Transposing \(A(A^\top A)^{-1}A^\top\) gives the same matrix back, because \((A^\top A)^{-1}\) is symmetric.

Idempotent: \(P^2 = P\). In the product of two copies of \(P\), the middle factors \(A^\top A\) and \((A^\top A)^{-1}\) cancel:

P^2 = A(A^\top A)^{-1}\underbrace{A^\top A\,(A^\top A)^{-1}}_{I}A^\top = P

4. Application: fitting a line to data (least squares)

The data: three points \((t, b) = (1,1),\ (2,2),\ (3,2)\). No straight line passes through all three.

The model: \(b = C + Dt\). We want the best \(C\) and \(D\).

The equations we'd like to solve but can't:

C + D = 1, \qquad C + 2D = 2, \qquad C + 3D = 2

In matrix form:

\underbrace{\begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}}_{A}\begin{bmatrix} C \\ D \end{bmatrix} = \underbrace{\begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix}}_{b}

This is 3 equations in 2 unknowns with no solution. The best solution solves \(A^\top A\,\hat{x} = A^\top b\).

Strang stops here. He finishes the numbers in Lecture 16.

Preview of the answer (computed here, not in this lecture).

\(A^\top A = \begin{bmatrix} 3 & 6 \\ 6 & 14 \end{bmatrix}\) and \(A^\top b = \begin{bmatrix} 5 \\ 11 \end{bmatrix}\).
Solving gives \(\hat{C} = \tfrac23\) and \(\hat{D} = \tfrac12\).
The best line is \(b = \tfrac23 + \tfrac12 t\).

Connection to the PCA lecture you started with

Orthonormal columns simplify everything. If the columns of \(A\) are orthonormal, so \(A^\top A = I\), the projection matrix collapses to

P = AA^\top

This is the PCA projection. With \(A = W\), this is exactly the \(WW^\top\) used in the PCA lecture. It is the same projection onto a subspace, in the case where the basis is orthonormal.

Big ideas from Lecture 15

The projection is the closest point in a subspace. The error \(e = b - p\) is perpendicular to the subspace.
The normal equations \(A^\top A\hat{x} = A^\top b\) come directly from \(A^\top e = 0\), which says \(e \in N(A^\top)\).
The three formulas: \(\hat{x} = (A^\top A)^{-1}A^\top b\), \(p = A\hat{x}\), and \(P = A(A^\top A)^{-1}A^\top\).
Projection matrices satisfy \(P^\top = P\) and \(P^2 = P\).
Don't split \((A^\top A)^{-1}\) when \(A\) is rectangular.
Least squares: for an unsolvable \(Ax = b\), solve \(A^\top A\hat{x} = A^\top b\) instead. Line fitting is the standard example.