Lecture 28: Similar Matrices and Jordan Form

Two parts:

A short recap on where positive definite matrices come from, including why $A^\top A$ is positive definite.
Similar matrices. These are matrices that share eigenvalues. The nicest member of each family is its Jordan form.

Part 1: More on positive definite matrices

Throughout, "positive definite" includes "symmetric".

Is $A^{-1}$ positive definite?

Yes. The eigenvalues of $A^{-1}$ are $1/\lambda_i$, and these are positive whenever every $\lambda_i > 0$.

Which test to use. Pivots of $A^{-1}$ are hard to predict, but its eigenvalues are known exactly. So the eigenvalue test is the natural one here.

Is $A + B$ positive definite?

Yes, if $A$ and $B$ both are.

Which tests don't work. We don't know the eigenvalues or pivots of $A + B$.

The energy test works immediately:

x^\top(A + B)x = x^\top Ax + x^\top Bx > 0

General lesson. Keep all four tests in mind and pick whichever one the problem makes easy.

Is $A^\top A$ positive definite? ($A$ rectangular, $m \times n$)

What we already know. $A^\top A$ is square and symmetric, from the least-squares lectures.

Use the energy test, and group the factors well:

x^\top(A^\top A)x = (Ax)^\top(Ax) = \|Ax\|^2 \geq 0

When is it strictly positive? The length $\|Ax\|$ is zero only when $Ax = 0$. To rule that out for every nonzero $x$, $A$ must have no nullspace other than $0$. That means:

independent columns, or equivalently
rank $n$.

\boxed{A^\top A \text{ is positive definite} \iff A \text{ has independent columns}}

Without independent columns, $A^\top A$ is positive semidefinite.

Numerical bonus. Elimination on a positive definite matrix never needs row exchanges. No zero or dangerously small pivots appear.

Part 2: Similar matrices

Definition

Square $n \times n$ matrices $A$ and $B$ are similar if, for some invertible $M$:

\boxed{B = M^{-1}AM}

The model example: diagonalization

S^{-1}AS = \Lambda

So $A$ is similar to $\Lambda$. Here the connecting matrix $M$ is the eigenvector matrix $S$.

Similarity groups matrices into families.

Any invertible $M$ produces another member of $A$'s family.
Most choices of $M$ won't produce a diagonal matrix.
The diagonal $\Lambda$ is the best member of the family.

Numerical example

Start with

A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}

Its eigenvalues are 3 and 1, so it is similar to $\Lambda = \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix}$.

Pick an arbitrary invertible $M$:

M = \begin{bmatrix} 1 & 4 \\ 0 & 1 \end{bmatrix}, \qquad M^{-1} = \begin{bmatrix} 1 & -4 \\ 0 & 1 \end{bmatrix}

Compute $B$:

B = M^{-1}AM = \begin{bmatrix} 1 & -4 \\ 0 & 1 \end{bmatrix}\begin{bmatrix} 2 & 9 \\ 1 & 6 \end{bmatrix} = \begin{bmatrix} -2 & -15 \\ 1 & 6 \end{bmatrix}

Check that $B$ has eigenvalues 3 and 1:

Trace: $-2 + 6 = 4 = 3 + 1$.
Determinant: $-12 + 15 = 3 = 3 \times 1$.

Other members of this family. Any triangular matrix such as $\begin{bmatrix} 3 & * \\ 0 & 1 \end{bmatrix}$. In fact, every $2\times2$ matrix with eigenvalues 3 and 1 belongs to this family.

Key theorem: similar matrices have the same eigenvalues

Proof. Start from $Ax = \lambda x$. Insert $MM^{-1} = I$:

AMM^{-1}x = \lambda x

Multiply on the left by $M^{-1}$:

\underbrace{M^{-1}AM}_{B}\,(M^{-1}x) = \lambda\,(M^{-1}x)

So $\lambda$ is an eigenvalue of $B$, with eigenvector $M^{-1}x$.

What changes and what doesn't.

The eigenvalues stay the same.
The eigenvectors move, from $x$ to $M^{-1}x$.

Diagonalization fits this picture. Going from $A$ to $\Lambda$ keeps the eigenvalues and moves the eigenvectors to the nicest possible ones: $(1,0)$ and $(0,1)$.

Same number of independent eigenvectors. Because $x \mapsto M^{-1}x$ is invertible, independent eigenvectors of $A$ map to independent eigenvectors of $B$.

Part 3: The bad case — repeated eigenvalues

Eigenvalues 4 and 4 split into two families

Family 1: $4I$ alone.

M^{-1}(4I)M = 4M^{-1}M = 4I \quad\text{for every } M

So $4I$ is similar only to itself. It is a family of one.

Family 2: everything else with eigenvalues 4, 4. Every member has only one independent eigenvector, so none of them is diagonalizable. If one were diagonalizable, it would be similar to $4I$, which is impossible.

The best member of family 2:

J = \begin{bmatrix} 4 & 1 \\ 0 & 4 \end{bmatrix}

This is the Jordan form: as close to diagonal as possible, with a single 1 above the diagonal.

Other members of family 2. Any matrix with trace 8 and determinant 16, other than $4I$. For example:

\begin{bmatrix} 5 & 1 \\ -1 & 3 \end{bmatrix}, \qquad \begin{bmatrix} 4 & 0 \\ 17 & 4 \end{bmatrix}

The entry 17 could be any nonzero number.

Counting eigenvectors is not enough

Consider two $4\times4$ matrices. Both have eigenvalues $0, 0, 0, 0$, rank 2, and therefore 2 independent eigenvectors:

J_1 = \begin{bmatrix} 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{bmatrix} \quad (\text{blocks } 3 + 1), \qquad J_2 = \begin{bmatrix} 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 \\ 0 & 0 & 0 & 0 \end{bmatrix} \quad (\text{blocks } 2 + 2)

Changing an off-diagonal entry doesn't matter. Replacing one of the 1s in $J_1$ with, say, 7 keeps the rank at 2 and gives a matrix similar to $J_1$. Jordan's choice of 1s is just the cleanest version.

But $J_1$ and $J_2$ are not similar. The block sizes differ.

Why not (my justification; Strang just states it). If $B = M^{-1}AM$, then $B^2 = M^{-1}A^2M$, so the squares must also be similar and have the same rank.

$J_1^2 \neq 0$: it has rank 1.
$J_2^2 = 0$.

So the two matrices cannot be similar.

Part 4: Jordan's theorem

Jordan blocks

A Jordan block has:

a single repeated eigenvalue $\lambda_i$ on the diagonal,
1s directly above the diagonal,
zeros everywhere else.

J_i = \begin{bmatrix} \lambda_i & 1 & & \\ & \lambda_i & \ddots & \\ & & \ddots & 1 \\ & & & \lambda_i \end{bmatrix}

Each block has exactly one eigenvector.

The theorem

Every square matrix $A$ is similar to a Jordan matrix
$J = \begin{bmatrix} J_1 & & \\ & \ddots & \\ & & J_d \end{bmatrix}$

Number of blocks. $d$ equals the number of independent eigenvectors, since each block contributes one.

The good case. If all $n$ eigenvalues are distinct, then $d = n$ and every block is $1\times1$. So $J = \Lambda$ and $A$ is diagonalizable.

What Jordan's theorem achieves. It completes diagonalization for every matrix, including the defective ones.

Strang's verdict

Jordan form used to be the climax of 18.06. When Strang took the course, the course built up to it.

It no longer is. Jordan form is numerically fragile:

It depends on eigenvalues being exactly equal and ranks being exactly known.
The slightest perturbation changes the eigenvalues or the rank, and the whole structure collapses.

What matters more now:

positive definite matrices,
the SVD, which is next lecture.

Big ideas from Lecture 28

Positive definite is preserved:
- $A^{-1}$ is positive definite (eigenvalue test).
- $A + B$ is positive definite (energy test).
- $A^\top A$ is positive definite exactly when $A$ has independent columns, because $x^\top A^\top Ax = \|Ax\|^2$.
Similar: $B = M^{-1}AM$. Similar matrices have the same eigenvalues, and the eigenvectors map as $x \mapsto M^{-1}x$.
Families: similarity groups matrices into families. The best member of a diagonalizable family is $\Lambda$.
Repeated eigenvalues: $\lambda I$ is alone in its family. All other matrices with the same repeated eigenvalue are defective.
Jordan form: every matrix is similar to a block-diagonal $J$ with one block per independent eigenvector. Block sizes matter, not just counts.
Practicality: Jordan form is theoretically complete but numerically unstable.

Connection to ML and interpretability.

Similarity is a change of basis. $A$ and $M^{-1}AM$ describe the same linear map in different coordinates.
What survives a basis change. Eigenvalues, trace, determinant, rank, and Jordan structure are basis-independent. Individual matrix entries are not.
Why that matters for weight matrices. A model's weights can often be reparameterized by an invertible $M$ without changing the function the model computes. So spectral quantities are meaningful to analyze, while specific coordinates may not be.