Lecture 17: Orthogonal Matrices and Gram–Schmidt

Topics:

Orthonormal vectors, and matrices $Q$ with orthonormal columns.
Why $Q$ makes every formula in this chapter easy.
Gram–Schmidt: turning any independent set of vectors into an orthonormal one.
$A = QR$, the matrix form of Gram–Schmidt.

1. Orthonormal vectors and $Q^\top Q = I$

Definition. Vectors $q_1, \dots, q_n$ are orthonormal if

q_i^\top q_j = \begin{cases} 0 & i \neq j \quad \text{(orthogonal)} \\ 1 & i = j \quad \text{(normalized: unit length)} \end{cases}

Put them in the columns of a matrix $Q$. Every dot product $q_i^\top q_j$ appears as an entry of $Q^\top Q$:

Q^\top Q = \begin{bmatrix} q_1^\top \\ \vdots \\ q_n^\top \end{bmatrix}\begin{bmatrix} q_1 & \cdots & q_n \end{bmatrix} = I

$Q$ does not have to be square. For example, it could have 2 columns with many components each.

Orthonormal vectors are automatically independent. No combination of mutually perpendicular nonzero vectors can give zero.

Why numerical linear algebra relies on them. Computations with orthonormal vectors never blow up, never overflow, and never underflow.

Naming convention

"Orthogonal matrix" is reserved for square $Q$ with orthonormal columns.
Strang notes that "orthonormal matrix" would be the logical name, but it is not the convention.
For square $Q$, the equation $Q^\top Q = I$ means: $\boxed{Q^\top = Q^{-1}}$

2. Examples of orthogonal matrices

Permutation matrices. For example:

Q = \begin{bmatrix} 0 & 0 & 1 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix}

Its transpose is another permutation matrix, and $Q^\top Q = I$.

Rotation. Two unit vectors at 90°:

Q = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}

Rescaled $\pm 1$ matrix. The columns of $\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}$ are orthogonal, but each has length $\sqrt2$. Dividing by the length fixes this:

Q = \tfrac{1}{\sqrt2}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}

This is a rotation, by $\theta = \pi/4$ up to a sign or reflection.

Hadamard matrices

Construction. Build larger examples by repeating the $2\times2$ pattern in blocks:

Q = \tfrac12\begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & -1 & 1 & -1 \\ 1 & 1 & -1 & -1 \\ 1 & -1 & -1 & 1 \end{bmatrix}

Each column $(\pm1, \pm1, \pm1, \pm1)$ has length 2, so the matrix is divided by 2.

Which sizes are possible. Orthogonal matrices with entries $\pm1$ are called Hadamard matrices.

The block construction gives sizes 2, 4, 8, 16, …
Strang believes no $5\times5$ one exists. He is right: beyond size 2, the size must be a multiple of 4.
Whether every multiple of 4 works is still an open question.

A rectangular example (no square roots)

Q = \tfrac13\begin{bmatrix} 1 & -2 \\ 2 & -1 \\ 2 & 2 \end{bmatrix}

Each column has length $\sqrt{1 + 4 + 4} = 3$.
The columns are orthogonal: $-2 - 2 + 4 = 0$.
So the columns form an orthonormal basis for the plane they span.

Completing to a square $Q$. Strang makes the matrix square by guessing a third column from the same pattern of $1$s and $2$s with signs adjusted. The board isn't clear on which sign pattern he used. One choice that works is $\tfrac13(2, -2, 1)$, which is orthogonal to both columns.

A warning about Gram–Schmidt. It usually produces square roots, because normalizing means dividing by lengths. These examples were chosen to avoid that.

3. Why $Q$ makes everything easy

The projection matrix

General formula. Projection onto $C(Q)$ uses the usual formula, but the middle factor is the identity:

P = Q\underbrace{(Q^\top Q)^{-1}}_{I}Q^\top = QQ^\top

No inverse to compute. Projection reduces to a single product $QQ^\top$.

Square case. If $Q$ is square, its column space is the whole space, so $P = QQ^\top = I$.

Both projection properties hold:

Symmetric: $(QQ^\top)^\top = QQ^\top$.
Idempotent: the middle factor $Q^\top Q = I$ cancels: $(QQ^\top)(QQ^\top) = Q\underbrace{(Q^\top Q)}_{I}Q^\top = QQ^\top$

The normal equations

General case: $A^\top A\,\hat{x} = A^\top b$.

With $A = Q$: $Q^\top Q\,\hat{x} = Q^\top b$, and the left side is just $\hat{x}$:

\hat{x} = Q^\top b, \qquad \text{that is,} \qquad \hat{x}_i = q_i^\top b

What this means.

No system has to be solved.
Each component is just a dot product with one basis vector.
Strang calls this probably the most important formula in large parts of mathematics.

4. Gram–Schmidt

Goal. Start with independent vectors $a, b, c, \dots$ and produce orthonormal vectors $q_1, q_2, q_3, \dots$ that span the same space.

Two stages:

Gram: make the vectors orthogonal. Call the results $A, B, C, \dots$
Schmidt: normalize them: $q_1 = A/\|A\|$, $q_2 = B/\|B\|$, and so on.

Strang jokes that Schmidt's contribution was dividing by the length.

Step 1: the first vector

Keep it as is: $A = a$. Its direction is fine.

Step 2: the second vector

Subtract from $b$ its projection onto $A$. What remains is the error part $e$ from the projection lecture, and it is perpendicular to $A$:

B = b - \frac{A^\top b}{A^\top A}A

Check.

A^\top B = A^\top b - \frac{A^\top b}{A^\top A}\,A^\top A = 0

Why $B$ is not zero. $a$ and $b$ are independent, so $b$ is not a multiple of $A$.

Step 3: the third vector

Subtract from $c$ its projections onto both $A$ and $B$:

C = c - \frac{A^\top c}{A^\top A}A - \frac{B^\top c}{B^\top B}B

Use the orthogonalized vectors. The subtracted directions are $A$ and $B$, not the original $a$ and $b$.

Result. $C$ is perpendicular to both $A$ and $B$. Then normalize all three.

Numerical example

Start with: $a = (1,1,1)$ and $b = (1,0,2)$. They are independent but not orthogonal.

First vector: $A = (1,1,1)$.

Second vector:

$A^\top b = 3$ and $A^\top A = 3$, so the ratio is 1.
$B = (1,0,2) - 1\cdot(1,1,1) = (0, -1, 1)$.

Check: $A^\top B = 0 - 1 + 1 = 0$.

Normalize:

Q = \begin{bmatrix} 1/\sqrt3 & 0 \\ 1/\sqrt3 & -1/\sqrt2 \\ 1/\sqrt3 & 1/\sqrt2 \end{bmatrix}

Same column space. $C(Q) = C(A)$, the same plane in $\mathbb{R}^3$. Each new vector is a combination of the original ones; only the basis changed, and the new basis is orthonormal.

5. $A = QR$: Gram–Schmidt in matrix form

The parallel with elimination.

Process	Matrix form
Elimination	$A = LU$
Gram–Schmidt	$A = QR$

The two factors:

$Q$ has orthonormal columns.
$R$ is upper triangular.

What $R$ contains. Since $Q^\top Q = I$, multiplying $A = QR$ on the left by $Q^\top$ gives $R = Q^\top A$. So

R_{ij} = q_i^\top a_j

For two columns:

\begin{bmatrix} a_1 & a_2 \end{bmatrix} = \begin{bmatrix} q_1 & q_2 \end{bmatrix}\begin{bmatrix} q_1^\top a_1 & q_1^\top a_2 \\ q_2^\top a_1 & q_2^\top a_2 \end{bmatrix}

Why $R$ is upper triangular

The lower-left entry vanishes. $q_2^\top a_1 = 0$, because Gram–Schmidt builds each later $q$ to be perpendicular to all earlier vectors.

In general. $q_i^\top a_j = 0$ whenever $i > j$.

$R$ for the example

The entries:

$r_{11} = q_1^\top a_1 = 3/\sqrt3 = \sqrt3$
$r_{12} = q_1^\top a_2 = (1 + 0 + 2)/\sqrt3 = \sqrt3$
$r_{21} = q_2^\top a_1 = 0$
$r_{22} = q_2^\top a_2 = 2/\sqrt2 = \sqrt2$

R = \begin{bmatrix} \sqrt3 & \sqrt3 \\ 0 & \sqrt2 \end{bmatrix}

Check column 2. $\sqrt3\,q_1 + \sqrt2\,q_2 = (1,1,1) + (0,-1,1) = (1,0,2) = a_2$.

Big ideas from Lecture 17

Orthonormal columns: $Q^\top Q = I$. For square $Q$ (an "orthogonal matrix"), $Q^\top = Q^{-1}$.
Examples: permutations, rotations, and Hadamard matrices.
Everything simplifies with $Q$: $P = QQ^\top$ and $\hat{x} = Q^\top b$, so each component is a dot product $q_i^\top b$.
Gram–Schmidt: subtract from each new vector its projections onto all earlier orthogonal vectors, then normalize.
$A = QR$: $R = Q^\top A$ is upper triangular because each $q_i$ is perpendicular to all earlier columns.