Lecture 14: Orthogonal Vectors and Subspaces

Chapter 4 begins: orthogonality. Strang calls it a "ninety-degree chapter".

Goal 1: show that the four subspaces meet at right angles, in pairs.
Goal 2: set up the main problem of the chapter, which is solving $Ax = b$ "as well as possible" when no exact solution exists.

1. Orthogonal vectors

Test: $x$ and $y$ are orthogonal exactly when

x^\top y = x_1y_1 + x_2y_2 + \cdots + x_ny_n = 0

Length squared: $\|x\|^2 = x^\top x = x_1^2 + \cdots + x_n^2$. This is positive unless $x = 0$.

Why the dot-product test is correct (via Pythagoras)

For a right triangle with legs $x$ and $y$, the hypotenuse is $x + y$, and Pythagoras says

\|x\|^2 + \|y\|^2 = \|x + y\|^2

This holds only for right triangles.

Expand the right side:

(x+y)^\top(x+y) = x^\top x + y^\top y + x^\top y + y^\top x

The $x^\top x$ and $y^\top y$ terms cancel against the left side. For real vectors $x^\top y = y^\top x$, so what remains is

2\,x^\top y = 0 \quad\Longleftrightarrow\quad x^\top y = 0

Numerical check

$x = (1, 2, 3)$, so $\|x\|^2 = 14$.
$y = (2, -1, 0)$, so $\|y\|^2 = 5$, and $x^\top y = 2 - 2 + 0 = 0$.
$x + y = (3, 1, 3)$, so $\|x+y\|^2 = 19$.
Pythagoras checks: $14 + 5 = 19$.

On the board Strang says $(1,2,3)$ "has length fourteen". Strictly, $14$ is the length squared; the length is $\sqrt{14}$.

The zero vector is orthogonal to every vector, since $0^\top y = 0$.

2. Orthogonal subspaces

Definition: Subspace $S$ is orthogonal to subspace $T$ if every vector in $S$ is orthogonal to every vector in $T$.

Example: the blackboard and the floor

The setup. In $\mathbb{R}^3$, take the plane of the wall and the plane of the floor, both through the origin.

Not orthogonal. They share a nonzero vector along the line where they meet, and that vector is not orthogonal to itself. You can also find many non-perpendicular pairs, such as a 45° vector in the wall and a vector in the floor.

General consequence. Orthogonal subspaces can intersect only in $\{0\}$. But meeting only at $\{0\}$ is not enough by itself to make subspaces orthogonal.

Subspaces of $\mathbb{R}^2$

Pair	Orthogonal?
A line and the whole plane	Never
A line and $\{0\}$	Always
Two lines through 0	Only if they meet at 90°

3. Row space ⊥ nullspace

The row space is orthogonal to the nullspace. The same holds for $A^\top$, so the column space is orthogonal to the left nullspace.

Proof

Write out $Ax = 0$ row by row:

\begin{bmatrix} \text{row}_1 \\ \text{row}_2 \\ \vdots \\ \text{row}_m \end{bmatrix} x = \begin{bmatrix} 0 \\ 0 \\ \vdots \\ 0 \end{bmatrix}

Each entry says $(\text{row}_i)^\top x = 0$. So $x$ is orthogonal to every row.

The row space also contains all combinations of the rows. Multiply each equation by $c_i$ and add:

(c_1\,\text{row}_1 + \cdots + c_m\,\text{row}_m)^\top x = 0

So $x$ is orthogonal to the entire row space.

Applying the same argument to $A^\top$ shows that $C(A)$ is orthogonal to $N(A^\top)$ in $\mathbb{R}^m$.

4. Orthogonal complements

Orthogonality alone is not the whole story. Two perpendicular lines in $\mathbb{R}^3$ are orthogonal, but they cannot be a row space and a nullspace. The dimensions must add to $n$: $r + (n - r) = 3$, and $1 + 1 \neq 3$.

Example

A = \begin{bmatrix} 1 & 2 & 5 \\ 2 & 4 & 10 \end{bmatrix}

The rank is 1, so the row space is the line through $(1,2,5)$.
The nullspace has dimension $3 - 1 = 2$. It is the plane $x_1 + 2x_2 + 5x_3 = 0$, the plane perpendicular to $(1,2,5)$.
In calculus language, $(1,2,5)$ is the plane's normal vector.
Row 2 adds nothing to either subspace, since it is twice row 1.

Definition

The orthogonal complement $V^\perp$ of a subspace $V$ contains all vectors orthogonal to $V$, not just some of them.

Fundamental Theorem of Linear Algebra, Part 2

N(A) = C(A^\top)^\perp \;\text{ in } \mathbb{R}^n, \qquad N(A^\top) = C(A)^\perp \;\text{ in } \mathbb{R}^m

What this means. The nullspace contains every vector perpendicular to the row space. Each pair of subspaces carves its whole space into two perpendicular pieces whose dimensions add up correctly.

The three parts of the theorem:

Part	Content
Part 1	The dimensions: $r$, $n-r$, $r$, $m-r$
Part 2	The orthogonality, as orthogonal complements
Part 3	Orthogonal bases for the subspaces (still to come)

5. Preview: solving $Ax = b$ when there is no solution

The problem. In practice you often have more equations than unknowns ($m > n$), and $b$ contains measurement noise. Then $b \notin C(A)$, so no exact solution exists. Yet $b$ still contains a lot of information about $x$.

Examples:

Tracking a satellite: thousands of position measurements, but only about 6 unknown parameters.
Measuring a pulse rate repeatedly: one unknown, many noisy readings.

A bad approach. Throwing away equations until the system is square and solvable. No measurement deserves to be singled out as perfect while the others are discarded. You want to use all the data.

The equation of the chapter:

\boxed{A^\top A\,\hat{x} = A^\top b}

Notation. The hat on $\hat{x}$ marks the "best" solution. What "best" means is explained next lecture.

6. The matrix $A^\top A$

For $A$ of size $m \times n$:

$A^\top A$ is square, of size $n \times n$.
$A^\top A$ is symmetric, since $(A^\top A)^\top = A^\top A$.

Examples

Independent columns:

A = \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 5 \end{bmatrix} \;\Rightarrow\; A^\top A = \begin{bmatrix} 3 & 8 \\ 8 & 30 \end{bmatrix}

This is invertible: $(3, 8)$ is not a multiple of $(8, 30)$.

Here $Ax = b$ is 3 equations in 2 unknowns. It is solvable only when $b$ lies in the plane $C(A)$, and most $b$ do not.

Dependent columns (rank 1):

A = \begin{bmatrix} 1 & 3 \\ 1 & 3 \\ 1 & 3 \end{bmatrix} \;\Rightarrow\; A^\top A = \begin{bmatrix} 3 & 9 \\ 9 & 27 \end{bmatrix}

This is singular, with rank 1.

Key facts

N(A^\top A) = N(A), \qquad \text{rank}(A^\top A) = \text{rank}(A)

$A^\top A$ is invertible exactly when $A$ has independent columns.

Proof (standard; not proved in this lecture — Strang defers it).

If $Ax = 0$, then $A^\top A x = 0$.
Conversely, if $A^\top A x = 0$, multiply on the left by $x^\top$: $x^\top A^\top A x = \|Ax\|^2 = 0$ so $Ax = 0$.

Big ideas from Lecture 14

Orthogonal vectors: $x^\top y = 0$, which is equivalent to Pythagoras holding for $x$, $y$, and $x + y$.
Orthogonal subspaces: every vector in one is perpendicular to every vector in the other. Such subspaces can share only $0$.
Row space ⊥ nullspace in $\mathbb{R}^n$, and column space ⊥ left nullspace in $\mathbb{R}^m$. This can be read directly from $Ax = 0$.
Orthogonal complements: the dimensions add to $n$ (or $m$), so each subspace contains all vectors perpendicular to its partner.
Unsolvable $Ax = b$: the plan is to solve $A^\top A \hat{x} = A^\top b$ instead.
$A^\top A$ is square and symmetric, has the same nullspace and rank as $A$, and is invertible exactly when the columns of $A$ are independent.