Lecture 25: Symmetric Matrices and Positive Definiteness

Why this lecture matters. Symmetric matrices ($A = A^\top$) are the most important class of matrices. This lecture covers:

What symmetry guarantees about eigenvalues and eigenvectors.
The spectral theorem.
Pivots and eigenvalues have the same signs.
A first look at positive definite matrices.

1. The two main facts

For a real symmetric matrix $A$:

All eigenvalues are real. The complex eigenvalues seen for rotation matrices cannot occur.
The eigenvectors are perpendicular. More precisely, they can be chosen perpendicular.

Why "can be chosen".

If all eigenvalues are distinct, each has one line of eigenvectors, and these lines are automatically perpendicular.
If an eigenvalue repeats, it has a whole plane (or larger space) of eigenvectors. You are free to choose perpendicular ones inside it.
Example: for $A = I$, every vector is an eigenvector, so "the eigenvectors are perpendicular" only makes sense as "you can pick perpendicular ones".

A full set always exists. A symmetric matrix always has a complete set of orthogonal eigenvectors. It is never deficient the way $\begin{bmatrix} 3 & 1 \\ 0 & 3\end{bmatrix}$ was.

2. The spectral theorem

The usual case. A diagonalizable matrix factors as $A = S\Lambda S^{-1}$.

The symmetric case.

Normalize the perpendicular eigenvectors to length 1. They are now orthonormal.
So $S$ becomes an orthogonal matrix $Q$.
For an orthogonal matrix, $Q^{-1} = Q^\top$.

\boxed{A = Q\Lambda Q^\top}

The converse is immediate. Any matrix of the form $Q\Lambda Q^\top$ is symmetric:

(Q\Lambda Q^\top)^\top = Q\Lambda^\top Q^\top = Q\Lambda Q^\top

Names.

In mathematics this is the spectral theorem. The "spectrum" is the set of eigenvalues, by analogy with light splitting into pure colors.
In mechanics it is the principal axis theorem. In the right coordinate axes, the matrix becomes diagonal and the directions don't couple.

As a sum of projections

Multiply out $Q\Lambda Q^\top$ column times row:

A = \lambda_1 q_1q_1^\top + \lambda_2 q_2q_2^\top + \cdots + \lambda_n q_nq_n^\top

Each piece $q_iq_i^\top$ is a projection matrix. It projects onto the line through the unit vector $q_i$.

It is symmetric.
It satisfies $(q_iq_i^\top)^2 = q_i(q_i^\top q_i)q_i^\top = q_iq_i^\top$, because $q_i^\top q_i = 1$.

Every symmetric matrix is a combination of mutually perpendicular projection matrices, weighted by its eigenvalues.

3. Proof that the eigenvalues are real

Start from what we know. $Ax = \lambda x$. For now, $\lambda$ and $x$ might be complex.

Step 1. Take complex conjugates. Since $A$ is real, $\bar{A} = A$:

A\bar{x} = \bar{\lambda}\bar{x}

Side fact. This already shows that for any real matrix, complex eigenvalues come in conjugate pairs $\lambda, \bar\lambda$, with eigenvectors $x, \bar{x}$.

Step 2. Transpose, then use symmetry. Transposing gives $\bar{x}^\top A^\top = \bar\lambda\bar{x}^\top$. Since $A^\top = A$:

\bar{x}^\top A = \bar\lambda\,\bar{x}^\top

This is the only place symmetry is used.

Step 3. Form the same scalar two ways.

Multiply $Ax = \lambda x$ on the left by $\bar{x}^\top$: $\bar{x}^\top Ax = \lambda\,\bar{x}^\top x$
Multiply $\bar{x}^\top A = \bar\lambda\bar{x}^\top$ on the right by $x$: $\bar{x}^\top Ax = \bar\lambda\,\bar{x}^\top x$

Step 4. Compare. The left sides are identical, so

\lambda\,\bar{x}^\top x = \bar\lambda\,\bar{x}^\top x

Step 5. Cancel $\bar{x}^\top x$. This is allowed because it is positive:

\bar{x}^\top x = |x_1|^2 + \cdots + |x_n|^2 > 0 \quad\text{for } x \neq 0

Each term is $(a - ib)(a + ib) = a^2 + b^2$, so the imaginary parts cancel.

Conclusion. $\lambda = \bar\lambda$, so $\lambda$ is real.

Length of a complex vector

The right definition. For a complex vector, the squared length is $\bar{x}^\top x$, not $x^\top x$. You must conjugate one factor to get a real, positive length. The next lecture develops this.

Complex matrices

Where the proof breaks. If $A$ is complex, Step 2 produces $\bar{A}^\top$ instead of $A$.

The fix. The proof goes through when

\bar{A}^\top = A

These are Hermitian matrices. They are the complex analogue of symmetric matrices, with real eigenvalues and perpendicular eigenvectors.

For real matrices, conjugation does nothing, so the condition is just $A^\top = A$.

Perpendicular eigenvectors (Strang leaves this to the book)

Here is the standard short proof. Suppose $Ax_1 = \lambda_1x_1$ and $Ax_2 = \lambda_2x_2$ with $\lambda_1 \neq \lambda_2$. Compute $x_2^\top Ax_1$ two ways:

\lambda_1\,x_2^\top x_1 = x_2^\top Ax_1 = (Ax_2)^\top x_1 = \lambda_2\,x_2^\top x_1

The middle step uses $A = A^\top$. So $(\lambda_1 - \lambda_2)\,x_2^\top x_1 = 0$, and since $\lambda_1 \neq \lambda_2$, we get $x_2^\top x_1 = 0$.

4. Signs of pivots = signs of eigenvalues

Computing eigenvalues is hard.

For a $50 \times 50$ matrix, finding the roots of the degree-50 polynomial $\det(A - \lambda I)$ is numerically unstable. MATLAB can do it, but the results are unreliable.
Strang admits this is the method he taught. It is fine for $2\times2$ and $3\times3$ only.
Real eigenvalue algorithms are a separate subject in numerical linear algebra.

Pivots are easy. Elimination finds them quickly and stably.

For a symmetric matrix: the number of positive pivots equals the number of positive eigenvalues, and the number of negative pivots equals the number of negative eigenvalues.

This assumes elimination runs without row exchanges. Example: if 28 pivots are positive and 22 negative, then exactly 28 eigenvalues are positive and 22 negative.

Counting eigenvalues above a threshold

Shift the matrix: $A - 7I$. This lowers every eigenvalue by 7.
Count the positive pivots of $A - 7I$.
That count is the number of eigenvalues of $A$ above 7.

Repeating this narrows down where the eigenvalues lie. It is the start of a real algorithm.

Pivots are not eigenvalues

What they share. Only the product: both equal $\det A$, provided there are no row exchanges.

What they don't share. The individual values are different. Only their signs match, one for one in count.

5. Positive definite matrices

Definition. A positive definite matrix is a symmetric matrix whose eigenvalues are all positive.

Strang's ranking. If symmetric matrices are good, positive definite matrices are "excellent".

Equivalent tests (for symmetric $A$)

All eigenvalues $> 0$.
All pivots $> 0$. This follows from Section 4.
All upper-left determinants $> 0$. These are the determinants of the $1\times1$, $2\times2$, …, $n\times n$ submatrices in the top-left corner.

Example

A = \begin{bmatrix} 5 & 2 \\ 2 & 3 \end{bmatrix}

Pivots.

The first pivot is $5$.
The determinant is $15 - 4 = 11$. Since the product of the pivots is the determinant, the second pivot is $11/5$.
Both are positive, so $A$ is positive definite.

Eigenvalues, as confirmation.

\lambda^2 - 8\lambda + 11 = 0 \;\Rightarrow\; \lambda = 4 \pm \sqrt5 \approx 6.24,\; 1.76

Both are positive, as predicted.

Which test is easier. The pivots are clean numbers. The eigenvalues involve $\sqrt5$.

Why the whole determinant is not enough

B = \begin{bmatrix} -1 & 0 \\ 0 & -3 \end{bmatrix}

$\det B = 3 > 0$, so the full determinant passes.
But the pivots and eigenvalues are $-1$ and $-3$. Both are negative, so $B$ is not positive definite.
The $1\times1$ upper-left determinant is $-1 < 0$, so test 3 correctly rejects $B$.

The lesson. You must check all $n$ upper-left determinants, not just the last one.

Coming next lecture. The energy test $x^\top Ax > 0$.

The course comes together

For symmetric matrices, the three main ideas of the course connect:

pivots, from the first weeks,
determinants, from the middle of the course,
eigenvalues, from the most recent lectures.

All three tell the same story about positive definiteness. The final lectures connect them for general $n\times n$ matrices.

Big ideas from Lecture 25

Symmetric ($A = A^\top$) implies real eigenvalues and orthogonal eigenvectors, with a full set always available.
Spectral theorem: $A = Q\Lambda Q^\top = \sum_i \lambda_i q_iq_i^\top$, a weighted sum of perpendicular projections.
Proof of real eigenvalues: compare $\bar{x}^\top Ax$ computed two ways, and use $\bar{x}^\top x > 0$.
Complex vectors and matrices: length squared is $\bar{x}^\top x$. The complex analogue of symmetric is $\bar{A}^\top = A$ (Hermitian).
Signs of pivots match signs of eigenvalues for symmetric matrices. This gives a cheap way to count positive and negative eigenvalues.
Positive definite means symmetric with all $\lambda > 0$. Equivalently, all pivots $> 0$, or all upper-left determinants $> 0$.

Connection to your first question. This is exactly the theorem behind PCA. The covariance matrix $C = \frac1n X^\top X$ is symmetric, so $C = Q\Lambda Q^\top$ with orthonormal principal directions and real variances $\lambda_i \geq 0$. The "sum of projections" form $\sum\lambda_iq_iq_i^\top$ is what gets truncated when you keep only the top $k$ components.