Lecture 22: Diagonalization and Powers of $A$

Goal: Use eigenvalues and eigenvectors to diagonalize $A$. Then use the diagonal form to compute powers $A^k$ and solve difference equations $u_{k+1} = Au_k$. The running example is the Fibonacci sequence.

1. Diagonalization: $S^{-1}AS = \Lambda$

Assumption. $A$ has $n$ independent eigenvectors $x_1, \dots, x_n$.

The eigenvector matrix. Put the eigenvectors in the columns of $S$:

S = \begin{bmatrix} x_1 & x_2 & \cdots & x_n \end{bmatrix}

Multiply $A$ by $S$, one column at a time. Each column is an eigenvector, so $Ax_i = \lambda_i x_i$:

AS = \begin{bmatrix} \lambda_1 x_1 & \cdots & \lambda_n x_n \end{bmatrix}

Pull the $\lambda$'s out. Each $\lambda_i$ multiplies column $i$, so it can be moved into a diagonal matrix on the right:

AS = \begin{bmatrix} x_1 & \cdots & x_n \end{bmatrix}\begin{bmatrix} \lambda_1 & & \\ & \ddots & \\ & & \lambda_n \end{bmatrix} = S\Lambda

$\Lambda$ (capital lambda) is the eigenvalue matrix, with the eigenvalues on its diagonal.

Two equivalent forms. Independence of the eigenvectors is exactly what makes $S$ invertible:

\boxed{S^{-1}AS = \Lambda} \qquad\Longleftrightarrow\qquad \boxed{A = S\Lambda S^{-1}}

A new factorization. It joins $A = LU$ (elimination) and $A = QR$ (Gram–Schmidt).

Memory aid. $A$ multiplies its eigenvectors, so $A$ sits next to $S$, not next to $S^{-1}$.

2. Powers of $A$

Eigenvector argument. If $Ax = \lambda x$, multiply both sides by $A$:

A^2x = \lambda Ax = \lambda^2 x

Same result from the factorization. The inner $S^{-1}S$ cancels:

A^2 = S\Lambda S^{-1}\,S\Lambda S^{-1} = S\Lambda^2 S^{-1}

For any power $k$:

\boxed{A^k = S\Lambda^k S^{-1}}

What happens to eigenvalues and eigenvectors:

Eigenvectors: unchanged.
Eigenvalues: raised to the $k$-th power, $\lambda_i^k$.

Why this is powerful. With $LU$, the product $(LU)(LU)\cdots(LU)$ gets you nowhere, because the pivots tell you nothing about powers. With $S\Lambda S^{-1}$, all the interior factors cancel.

Stability theorem

A^k \to 0 \text{ as } k \to \infty \quad\text{when}\quad |\lambda_i| < 1 \text{ for all } i

Absolute values are needed because eigenvalues can be negative or complex.

Scope of the proof. Strang's argument goes through $S\Lambda^kS^{-1}$, so it assumes $n$ independent eigenvectors. The statement remains true even when $A$ is not diagonalizable, though that requires a different proof.

3. Which matrices are diagonalizable?

Distinct eigenvalues guarantee it.

If all $\lambda$'s are different, the eigenvectors are automatically independent. Strang refers the proof to the textbook.
A random matrix, such as MATLAB's eig(rand(10)), almost surely has distinct eigenvalues, so it is diagonalizable.

Repeated eigenvalues require a closer look. You may or may not get enough eigenvectors.

Matrix	Eigenvalues	Independent eigenvectors	Diagonalizable?
$I_{10}$	$1$ (repeated 10 times)	10, since every vector is an eigenvector	Yes; it's already diagonal
$\begin{bmatrix} 2 & 1 \\ 0 & 2 \end{bmatrix}$	$2, 2$	only 1, namely $(1, 0)$	No

The failing example

Eigenvalues. The matrix is triangular, so the eigenvalues are on the diagonal:

\det(A - \lambda I) = (2 - \lambda)^2 \;\Rightarrow\; \lambda = 2, 2

Eigenvectors. $A - 2I = \begin{bmatrix} 0 & 1 \\ 0 & 0 \end{bmatrix}$. Its nullspace is one-dimensional, spanned by $(1, 0)$. The vector $(0, 1)$ is not in it.

Two kinds of multiplicity

Algebraic multiplicity. How many times $\lambda$ is a root of $\det(A - \lambda I)$. Here it is 2.

Geometric multiplicity. The number of independent eigenvectors for $\lambda$, which is $\dim N(A - \lambda I)$. Here it is 1.

When they differ. If geometric multiplicity is less than algebraic multiplicity, there are not enough eigenvectors, and $A$ cannot be diagonalized.

4. Difference equations $u_{k+1} = Au_k$

The setup. Start from a given vector $u_0$ and multiply by $A$ at each step:

u_1 = Au_0, \quad u_2 = A^2u_0, \quad \dots, \quad u_k = A^ku_0

Strang calls this a first-order system: first-order because each step goes up only one level, a system because $u$ is a vector.

$u_k = A^k u_0$ is compact, but not yet a usable answer.

How to actually compute $u_k$

Step 1. Expand $u_0$ in eigenvectors:

u_0 = c_1x_1 + c_2x_2 + \cdots + c_nx_n = Sc

Step 2. Each eigenvector component evolves independently. Strang says each one "goes its own merry way":

Au_0 = c_1\lambda_1x_1 + \cdots + c_n\lambda_nx_n

Step 3. After $k$ steps:

\boxed{u_k = A^ku_0 = c_1\lambda_1^kx_1 + \cdots + c_n\lambda_n^kx_n = S\Lambda^kc}

This is exactly the formula $A^k = S\Lambda^kS^{-1}$ applied to $u_0$, with $c = S^{-1}u_0$.

Static vs. dynamic problems.

$Ax = b$ is a static problem.
$u_{k+1} = Au_k$ is dynamic: things evolve in time.
The eigenvalues decide whether the solution blows up, decays, or stays bounded.

5. Example: Fibonacci numbers

The sequence: $0, 1, 1, 2, 3, 5, 8, 13, 21, 34, \dots$

The rule: $F_{k+2} = F_{k+1} + F_k$.

Questions:

What is $F_{100}$?
How fast does the sequence grow?

Convert to a first-order system

The trick. The Fibonacci rule is a single second-order equation. Turn it into a $2\times2$ first-order system by making the unknown a vector:

u_k = \begin{bmatrix} F_{k+1} \\ F_k \end{bmatrix}

Add a trivial second equation, $F_{k+1} = F_{k+1}$. Together the two equations give:

u_{k+1} = \begin{bmatrix} F_{k+2} \\ F_{k+1} \end{bmatrix} = \underbrace{\begin{bmatrix} 1 & 1 \\ 1 & 0 \end{bmatrix}}_{A}\begin{bmatrix} F_{k+1} \\ F_k \end{bmatrix}

$A$ is symmetric, so its eigenvalues will be real.

Eigenvalues

Quick checks. $\lambda_1 + \lambda_2 = \text{trace} = 1$ and $\lambda_1\lambda_2 = \det A = -1$.

Characteristic equation:

\det(A - \lambda I) = \lambda^2 - \lambda - 1 = 0

This mirrors the Fibonacci rule itself, $F_{k+2} - F_{k+1} - F_k = 0$.

Roots:

\lambda_1 = \frac{1 + \sqrt5}{2} \approx 1.618, \qquad \lambda_2 = \frac{1 - \sqrt5}{2} \approx -0.618

Consequences.

The eigenvalues are distinct, so $A$ is diagonalizable.
$\lambda_1 > 1$, so the solution grows.
$|\lambda_2| < 1$, so that component decays.

Growth rate

The $\lambda_1^k$ term dominates, while $\lambda_2^k \to 0$ very fast:

F_{100} \approx c_1\left(\frac{1+\sqrt5}{2}\right)^{100}

Each Fibonacci number is about 1.618 times the previous one. This ratio is the golden ratio.

Eigenvectors

$A - \lambda I = \begin{bmatrix} 1 - \lambda & 1 \\ 1 & -\lambda \end{bmatrix}$. Try $x = (\lambda, 1)$:

The second row gives $\lambda - \lambda = 0$.
The first row gives $(1-\lambda)\lambda + 1 = -\lambda^2 + \lambda + 1$, which is zero precisely because $\lambda$ satisfies $\lambda^2 = \lambda + 1$.

So the eigenvectors are

x_1 = \begin{bmatrix} \lambda_1 \\ 1 \end{bmatrix}, \qquad x_2 = \begin{bmatrix} \lambda_2 \\ 1 \end{bmatrix}

Finishing the computation (Strang runs out of time; completed here)

Initial vector. $u_0 = (F_1, F_0) = (1, 0)$.

Solve $c_1x_1 + c_2x_2 = u_0$:

The second components give $c_1 + c_2 = 0$.
The first components give $c_1\lambda_1 + c_2\lambda_2 = 1$.

So $c_1 = \dfrac{1}{\lambda_1 - \lambda_2} = \dfrac{1}{\sqrt5}$ and $c_2 = -\dfrac{1}{\sqrt5}$.

Read off $F_k$ from the second component of $u_k$:

F_k = \frac{\lambda_1^k - \lambda_2^k}{\sqrt5}

This is Binet's formula. In particular:

F_{100} \approx \frac{1}{\sqrt5}\left(\frac{1+\sqrt5}{2}\right)^{100}

Big ideas from Lecture 22

Diagonalization: $A = S\Lambda S^{-1}$. This requires $n$ independent eigenvectors so that $S$ is invertible.
Powers: $A^k = S\Lambda^kS^{-1}$. The eigenvectors stay the same and the eigenvalues become $\lambda^k$.
Stability: $A^k \to 0$ exactly when all $|\lambda_i| < 1$.
Distinct eigenvalues imply diagonalizable. With repeated eigenvalues, compare algebraic and geometric multiplicity.
Difference equations: $u_k = \sum c_i\lambda_i^kx_i$. Expand $u_0$ in eigenvectors, then let each component scale by its own $\lambda$.
Fibonacci: the growth rate is the largest eigenvalue, $\frac{1+\sqrt5}{2} \approx 1.618$.

Connection to ML. The rule that the largest $|\lambda|$ governs $A^k$ is exactly why repeated matrix multiplication causes exploding or vanishing signals, for example in RNN gradients or deep linear networks. The same rule governs convergence of Markov chains, which Strang covers in Lecture 24 and which is directly relevant to Markov-chain sequence tasks.

Lecture 22: Diagonalization and Powers of \(A\)

1. Diagonalization: \(S^{-1}AS = \Lambda\)

2. Powers of \(A\)