Lecture 16: Projection Matrices and Least Squares

Topics:

What $P = A(A^\top A)^{-1}A^\top$ does, checked on two extreme cases.
A complete least-squares line fit, viewed in two pictures.
Deriving the normal equations from calculus.
A proof that $A^\top A$ is invertible when $A$ has independent columns.
A preview of orthonormal columns.

1. Two extreme cases for $P = A(A^\top A)^{-1}A^\top$

Case 1: $b$ is perpendicular to $C(A)$.

Then $b \in N(A^\top)$, so $A^\top b = 0$.
The last factor of $P$ kills it: $Pb = 0$.

Case 2: $b$ is in $C(A)$.

Then $b = Ax$ for some $x$. Strang calls this fact essential and promises it will appear on exams.
The middle factors cancel: $Pb = A\underbrace{(A^\top A)^{-1}A^\top A}_{I}x = Ax = b$

The general case. Any $b$ splits into two perpendicular pieces:

b = \underbrace{Pb}_{p \,\in\, C(A)} + \underbrace{(I - P)b}_{e \,\in\, N(A^\top)}

$P$ keeps the column-space part and removes the perpendicular part.

$I - P$ is also a projection

$I - P$ projects onto the perpendicular subspace, $N(A^\top)$. It has the same properties as $P$:

Symmetric: $(I-P)^\top = I - P$.
Idempotent: $(I - P)^2 = I - 2P + P^2 = I - P$.

2. Least squares: fitting a line

Data: $(t, b) = (1,1),\ (2,2),\ (3,2)$.

Model: $b = C + Dt$.

The equations we'd like to solve:

\underbrace{\begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}}_{A}\begin{bmatrix} C \\ D \end{bmatrix} = \underbrace{\begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix}}_{b}

$A$ has independent columns, so they form a basis for $C(A)$. But $b \notin C(A)$, so there is no exact solution.

On the board Strang briefly writes $b$ as $(1,2,3)$. The data, and all his later computations, use $b = (1,2,2)$.

What "best" means

Minimize the sum of squared errors:

\|Ax - b\|^2 = e_1^2 + e_2^2 + e_3^2

Each $e_i$ is the vertical distance from the data point to the line. This is linear regression.

Caveat: outliers

Squaring makes large errors dominate the total.

A single wildly wrong measurement, an outlier, can tilt the whole fitted line.
Statisticians therefore watch for outliers.
Least squares is still the most widely used method, but it overweights outliers.

3. Two pictures of the same problem

Picture 1: the data plane ($t$ vs. $b$)	Picture 2: $\mathbb{R}^3$
3 data points and the best line	the vector $b = (1,2,2)$ and the plane $C(A)$
errors $e_i$ are vertical gaps to the line	$e$ is the perpendicular from $b$ to the plane
$p_i$ are the heights of the line at $t = 1, 2, 3$	$p = (p_1,p_2,p_3)$ is the projection of $b$
$C$ and $D$ define the line	$C$ and $D$ are the combination of columns that gives $p$

Key observation. If the $b_i$ were replaced by the $p_i$, the three equations would be solvable. A line goes through the $p$'s. That is exactly what it means for $p$ to lie in $C(A)$.

4. Solving it

Normal equations:

A^\top A\,\hat{x} = A^\top b

Strang calls this "the most important equation in statistics" and in estimation: it's the first thing to use whenever you fit a few parameters to noisy data.

Compute $A^\top A$ and $A^\top b$:

A^\top A = \begin{bmatrix} 3 & 6 \\ 6 & 14 \end{bmatrix}, \qquad A^\top b = \begin{bmatrix} 5 \\ 11 \end{bmatrix}

$A^\top A$ is symmetric and invertible. Later in the course it will also turn out to be positive definite.

The two equations:

3C + 6D = 5, \qquad 6C + 14D = 11

Elimination. Subtract 2 × the first equation from the second: $2D = 1$, so $D = \tfrac12$.

Back substitution. $3C + 3 = 5$, so $C = \tfrac23$.

\boxed{\text{best line: } b = \tfrac23 + \tfrac12 t}

The projection and the error

Projection. Evaluate the line at $t = 1, 2, 3$:

p = \left(\tfrac76,\; \tfrac53,\; \tfrac{13}{6}\right)

Error:

e = b - p = \left(-\tfrac16,\; \tfrac13,\; -\tfrac16\right)

Check: $p + e = b = (1, 2, 2)$.

Verifying the geometry

$e \perp p$:

-\tfrac{7}{36} + \tfrac{20}{36} - \tfrac{13}{36} = 0

$e$ is perpendicular to all of $C(A)$, not just to $p$.

Against the first column $(1,1,1)$: $-\tfrac16 + \tfrac26 - \tfrac16 = 0$.
Against the second column $(1,2,3)$: $-\tfrac16 + \tfrac46 - \tfrac36 = 0$.

5. The same equations from calculus

The function to minimize:

\[E(C, D) = (C + D - 1)^2 + (C + 2D - 2)^2 + (C + 3D - 2)^2\]

Set both partial derivatives to zero. Since $E$ is quadratic, the derivatives are linear:

$\dfrac{\partial E}{\partial C} = 0$ gives $3C + 6D = 5$.
$\dfrac{\partial E}{\partial D} = 0$ gives $6C + 14D = 11$.

These are exactly the normal equations. Linear algebra and calculus reach the same answer.

6. Proof: independent columns imply $A^\top A$ is invertible

Claim. If $A$ has independent columns, then $A^\top A$ is invertible.

Proof.

Suppose $A^\top A x = 0$. The goal is to show $x = 0$, which means $N(A^\top A) = \{0\}$.
Idea: take the dot product with $x$, that is, multiply on the left by $x^\top$: $x^\top A^\top A x = (Ax)^\top(Ax) = \|Ax\|^2 = 0$
A vector of length zero is the zero vector, so $Ax = 0$.
$A$ has independent columns, so $N(A) = \{0\}$. Therefore $x = 0$.

Strang notes that this proof uses everything known about nullspaces.

7. Preview: orthonormal columns

Perpendicular columns are automatically independent, provided none of them is zero. To rule out zero, make each column a unit vector.

Orthonormal vectors are perpendicular (ortho) and have length 1 (normal).

Examples:

The standard basis $(1,0,0)$, $(0,1,0)$, $(0,0,1)$.
Strang's favorite pair: $\begin{bmatrix} \cos\theta \\ \sin\theta \end{bmatrix}, \quad \begin{bmatrix} -\sin\theta \\ \cos\theta \end{bmatrix}$

Why they're the best possible columns. For a matrix $A$ with orthonormal columns, $A^\top A = I$. Then every formula simplifies; for instance, $P$ becomes $AA^\top$.

Next lecture:

Why orthonormal bases are so convenient.
How to construct one from any basis (Gram–Schmidt).

Big ideas from Lecture 16

Two extremes. $P$ fixes vectors in $C(A)$ and kills vectors in $N(A^\top)$. Every $b$ splits as $p + e$ with $p \perp e$.
The complementary projection. $I - P$ projects onto $N(A^\top)$.
Least squares minimizes $\|Ax - b\|^2$. For line fitting, it minimizes the sum of squared vertical errors.
Two pictures. The best line through the data equals the projection of $b$ onto $C(A)$ in $\mathbb{R}^m$.
Calculus agrees. Setting $\partial E/\partial C = \partial E/\partial D = 0$ gives exactly $A^\top A\hat{x} = A^\top b$.
$A^\top A$ is invertible exactly when the columns of $A$ are independent. Proof: $x^\top A^\top A x = \|Ax\|^2$.
Orthonormal columns give $A^\top A = I$. This is the subject of Lecture 17.

Picture 1: the data plane (\(t\) vs. \(b\))	Picture 2: \(\mathbb{R}^3\)
3 data points and the best line	the vector \(b = (1,2,2)\) and the plane \(C(A)\)
errors \(e_i\) are vertical gaps to the line	\(e\) is the perpendicular from \(b\) to the plane
\(p_i\) are the heights of the line at \(t = 1, 2, 3\)	\(p = (p_1,p_2,p_3)\) is the projection of \(b\)
\(C\) and \(D\) define the line	\(C\) and \(D\) are the combination of columns that gives \(p\)