Lecture 21: Eigenvalues and Eigenvectors

This summary is based on the same lecture's transcript from an OCW mirror. Parts of the transcript are scrambled, so I've put the examples back in the order Strang develops them.

This opens the eigenvalue part of the course, which runs to the end. This lecture covers what eigenvalues and eigenvectors are and how to find them. Later lectures cover how to use them.

1. Definition

The idea. A matrix acts on vectors: $x$ goes in, $Ax$ comes out. For most $x$, the output $Ax$ points in a new direction. Eigenvectors are the special vectors whose output points in the same direction as the input:

\boxed{Ax = \lambda x}

Names. $x \neq 0$ is an eigenvector, and the number $\lambda$ is its eigenvalue.

What $\lambda$ can be. Positive, negative, zero, or even complex. "Same direction" includes the reverse direction.

Eigenvalue $\lambda = 0$

Meaning. $\lambda = 0$ means $Ax = 0$. So the eigenvectors for $\lambda = 0$ are exactly the nonzero vectors of the nullspace.

Consequence. $A$ is singular exactly when $\lambda = 0$ is an eigenvalue.

2. Examples you can solve by inspection

Projection matrix $P$ (onto a plane in $\mathbb{R}^3$)

Think about what projecting does to different vectors.

A typical $b$ is not an eigenvector. Its projection $Pb$ points in a different direction.

Any $x$ in the plane. Projection leaves it unchanged: $Px = x$, so $\lambda = 1$. This gives a whole plane of eigenvectors.

Any $x$ perpendicular to the plane. Projection sends it to zero: $Px = 0$, so $\lambda = 0$. These vectors form the nullspace.

Total. Three independent eigenvectors, two in the plane and one perpendicular, with eigenvalues $1, 1, 0$.

Permutation matrix

A = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}

This matrix swaps the two components.

Swapping leaves $x$ unchanged. $x = (1, 1)$ gives $Ax = x$, so $\lambda = 1$.

Swapping flips the sign of $x$. $x = (-1, 1)$ gives $Ax = (1, -1) = -x$, so $\lambda = -1$.

A nonnegative matrix can have a negative eigenvalue.

3. Two quick checks: trace and determinant

Trace. The sum of the eigenvalues equals the trace, the sum of the diagonal entries:

\lambda_1 + \cdots + \lambda_n = a_{11} + a_{22} + \cdots + a_{nn}

Determinant. The product of the eigenvalues equals the determinant:

\lambda_1 \lambda_2 \cdots \lambda_n = \det A

Using them in the $2\times2$ case. Knowing one eigenvalue gives the other immediately. For the permutation matrix, the trace is 0, so once you know $\lambda_1 = 1$, you get $\lambda_2 = -1$.

4. The systematic method

Why elimination alone fails. $Ax = \lambda x$ has two unknowns, $\lambda$ and $x$, multiplied together.

Rewrite the equation:

(A - \lambda I)x = 0

The key step. For a nonzero $x$ to exist, $A - \lambda I$ must be singular. A singular matrix has determinant zero, so:

\boxed{\det(A - \lambda I) = 0}

This is the characteristic equation, also called the eigenvalue equation.

The procedure:

Find $\lambda$ first. $\det(A - \lambda I)$ is a polynomial of degree $n$ in $\lambda$, so there are $n$ roots. Some may repeat.
Then find $x$. For each $\lambda$, find the nullspace of $A - \lambda I$ by elimination. Set the free variable to 1.

Strang's warning: repeated eigenvalues are "the source of all trouble in 18.06."

Worked example

A = \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix}

$A$ is symmetric and has a constant diagonal.

Characteristic equation:

\det(A - \lambda I) = (3 - \lambda)^2 - 1 = \lambda^2 - 6\lambda + 8

What the coefficients mean.

The $6$ is the trace, $3 + 3$.
The $8$ is the determinant, $9 - 1$.
For any $2\times2$ matrix: $\det(A - \lambda I) = \lambda^2 - (\text{trace})\,\lambda + \det A$

Eigenvalues. $\lambda^2 - 6\lambda + 8 = (\lambda - 4)(\lambda - 2)$, so $\lambda_1 = 4$ and $\lambda_2 = 2$.

Eigenvectors.

For $\lambda_1 = 4$: $A - 4I = \begin{bmatrix} -1 & 1 \\ 1 & -1\end{bmatrix}$. This is singular, as it must be. Its nullspace gives $x_1 = (1, 1)$.
For $\lambda_2 = 2$: $A - 2I = \begin{bmatrix} 1 & 1 \\ 1 & 1\end{bmatrix}$. Its nullspace gives $x_2 = (-1, 1)$.

Each eigenvalue gives a whole line of eigenvectors. "The" eigenvector really means one basis vector for that line.

Preview of symmetric matrices. Here the eigenvalues are real and the eigenvectors are perpendicular: $(1,1)\cdot(-1,1) = 0$. The permutation matrix behaved the same way. This always happens for symmetric matrices, which is a theme for later lectures.

5. Adding a multiple of $I$ shifts every eigenvalue

Compare the two examples. $\begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix}$ is the permutation matrix plus $3I$.

The eigenvalues moved from $1, -1$ to $4, 2$. Each one went up by 3.
The eigenvectors did not change.

Why.

Ax = \lambda x \;\Longrightarrow\; (A + 3I)x = \lambda x + 3x = (\lambda + 3)x

Caution: eigenvalues do not add or multiply in general

The tempting argument. Suppose $Ax = \lambda x$ and $B$ has eigenvalue $\alpha$. It seems that adding gives $(A + B)x = (\lambda + \alpha)x$.

Why it fails. It assumes $x$ is also an eigenvector of $B$. Usually it isn't, because $B$ has its own, different eigenvectors.

Conclusion.

The eigenvalues of $A + B$ are generally not sums of eigenvalues of $A$ and $B$.
The eigenvalues of $AB$ are generally not products.
$B = cI$ is the exception, because every vector is an eigenvector of $cI$.

6. Trouble case 1: complex eigenvalues (rotation)

A $90°$ rotation:

Q = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix}

Expect trouble. Every vector gets rotated by $90°$, so no real vector keeps its direction.

The numbers confirm it.

Trace $= 0$, so $\lambda_1 + \lambda_2 = 0$.
Determinant $= 1$, so $\lambda_1 \lambda_2 = 1$.
No two real numbers satisfy both: summing to 0 forces opposite signs, which makes the product negative.

Solve the characteristic equation:

\det(Q - \lambda I) = \lambda^2 + 1 = 0 \;\Longrightarrow\; \lambda = i,\; -i

Observations.

A real matrix can have complex eigenvalues. When it does, they come in complex conjugate pairs.
This $Q$ is antisymmetric: $Q^\top = -Q$. It is as far from symmetric as possible, and antisymmetric matrices have purely imaginary eigenvalues.
Symmetric matrices have real eigenvalues.
Matrices in between, with both a symmetric and an antisymmetric part, fall in between.

7. Trouble case 2: too few eigenvectors (repeated eigenvalue)

A = \begin{bmatrix} 3 & 1 \\ 0 & 3 \end{bmatrix}

Eigenvalues. $A$ is triangular, so its eigenvalues are its diagonal entries:

\det(A - \lambda I) = (3 - \lambda)^2 \;\Longrightarrow\; \lambda = 3, 3

This is a repeated eigenvalue.

Eigenvectors.

$A - 3I = \begin{bmatrix} 0 & 1 \\ 0 & 0 \end{bmatrix}$. Its nullspace is spanned by $x_1 = (1, 0)$.
The second $\lambda = 3$ gives the same nullspace. No second independent eigenvector exists.

Name. Such a matrix is called degenerate.

It is $2\times2$ but has only one line of eigenvectors.
A repeated eigenvalue opens the possibility of a shortage of eigenvectors.
The next lecture handles the good case, where there are enough eigenvectors.

Summary of examples

Matrix	Eigenvalues	Eigenvectors	Lesson
Projection $P$	$1, 0$	the plane; the perpendicular line	$\lambda = 0$ gives the nullspace
$\begin{bmatrix} 0&1\\1&0\end{bmatrix}$	$1, -1$	$(1,1)$, $(-1,1)$	trace check
$\begin{bmatrix} 3&1\\1&3\end{bmatrix}$	$4, 2$	$(1,1)$, $(-1,1)$	adding $3I$ shifts $\lambda$ by 3; symmetric gives real $\lambda$ and perpendicular $x$
$\begin{bmatrix} 0&-1\\1&0\end{bmatrix}$	$i, -i$	complex	rotations give complex $\lambda$
$\begin{bmatrix} 3&1\\0&3\end{bmatrix}$	$3, 3$	only $(1,0)$	repeated $\lambda$ can mean too few eigenvectors

Big ideas from Lecture 21

$Ax = \lambda x$: eigenvectors keep their direction, and $\lambda$ is the stretching factor.
Method: solve $\det(A - \lambda I) = 0$ for $\lambda$, then find $N(A - \lambda I)$ for $x$.
Checks: the sum of the eigenvalues is the trace, and the product is the determinant.
Shift: $A + cI$ has the same eigenvectors and eigenvalues $\lambda + c$. There is no such rule for $A + B$ or $AB$ in general.
Special structure:
- Triangular matrices have their eigenvalues on the diagonal.
- Symmetric matrices have real eigenvalues and perpendicular eigenvectors.
- Antisymmetric matrices have imaginary eigenvalues.
Two failure modes: complex eigenvalues, and missing eigenvectors when an eigenvalue repeats.

Matrix	Eigenvalues	Eigenvectors	Lesson
Projection \(P\)	\(1, 0\)	the plane; the perpendicular line	\(\lambda = 0\) gives the nullspace
\(\begin{bmatrix} 0&1\\1&0\end{bmatrix}\)	\(1, -1\)	\((1,1)\), \((-1,1)\)	trace check
\(\begin{bmatrix} 3&1\\1&3\end{bmatrix}\)	\(4, 2\)	\((1,1)\), \((-1,1)\)	adding \(3I\) shifts \(\lambda\) by 3; symmetric gives real \(\lambda\) and perpendicular \(x\)
\(\begin{bmatrix} 0&-1\\1&0\end{bmatrix}\)	\(i, -i\)	complex	rotations give complex \(\lambda\)
\(\begin{bmatrix} 3&1\\0&3\end{bmatrix}\)	\(3, 3\)	only \((1,0)\)	repeated \(\lambda\) can mean too few eigenvectors