Lecture 24: Markov Matrices; Fourier Series

Two applications:

Markov matrices. Powers of \(A\), with a steady state that comes from the eigenvalue \(\lambda = 1\).
Fourier series. Projection onto an orthogonal basis, carried over to an infinite-dimensional space of functions.

Part 1: Markov matrices

Definition

A Markov matrix has two properties:

All entries are \(\ge 0\). They are probabilities.
Every column sums to 1. Everything is accounted for.

Strang's invented \(3\times3\) example, written column by column:

A = \begin{bmatrix} .1 & .01 & .3 \\ .2 & .99 & .3 \\ .7 & 0 & .4 \end{bmatrix}

Powers stay Markov. Both properties survive multiplication, so every power \(A^k\) is also a Markov matrix.

Key facts

\(\lambda = 1\) is always an eigenvalue.
Every other eigenvalue has \(|\lambda| \le 1\). No eigenvalue is ever larger than 1. Strang mentions that exceptional cases with \(|\lambda| = 1\) can occur. He does not prove this fact.
The eigenvector \(x_1\) for \(\lambda = 1\) has no negative components. This is the steady state. Strang does not prove this fact either.

Why the steady state appears. Expand \(u_0\) in eigenvectors:

u_k = A^ku_0 = c_1(1)^kx_1 + c_2\lambda_2^kx_2 + \cdots

When \(|\lambda_i| < 1\) for \(i \ge 2\), those terms die out, and \(u_k \to c_1x_1\). As always, this argument assumes a complete set of eigenvectors.

Contrast with Lecture 23.

For powers \(A^k\), the steady state comes from \(\lambda = 1\).
For \(e^{At}\), it came from \(\lambda = 0\).
For powers, a zero eigenvalue's component vanishes after a single step.

Proof that \(\lambda = 1\) is an eigenvalue

Step 1: subtract \(I\). The columns of \(A\) sum to 1, so the columns of \(A - I\) sum to 0.

Step 2: the rows of \(A - I\) are dependent. "Every column sums to 0" means the sum of all the rows is the zero row:

(1, 1, \dots, 1)\,(A - I) = 0

Step 3: conclude. A square matrix with dependent rows is singular, so \(\det(A - I) = 0\) and \(\lambda = 1\) is an eigenvalue.

Interpretation. \((1, \dots, 1)\) lies in the left nullspace of \(A - I\). Equivalently, it is an eigenvector of \(A^\top\) with \(\lambda = 1\).

Finding the eigenvector of \(A\) itself. The steady state \(x_1\) lies in the nullspace of \(A - I\). It is generally a different vector from \((1, \dots, 1)\).

Side fact: \(A\) and \(A^\top\) have the same eigenvalues

\det(A^\top - \lambda I) = \det\big((A - \lambda I)^\top\big) = \det(A - \lambda I)

This uses Property 10, \(\det M^\top = \det M\), together with \((\lambda I)^\top = \lambda I\).

Eigenvalues match, eigenvectors don't. The same \(\lambda\)'s appear, but the eigenvectors are generally different.

Steady state for the \(3\times3\) example

Strang finds it on the spot. Solving \((A - I)x = 0\) gives

x_1 = (0.6,\; 33,\; 0.7)

Check.

Row 3: \(0.7(0.6) - 0.6(0.7) = 0\).
Row 1: \(-0.54 + 0.33 + 0.21 = 0\).
Row 2: \(0.12 - 0.33 + 0.21 = 0\).

All components are positive, as the theory predicts.

Normalized so the components sum to 1: approximately \((0.017,\; 0.962,\; 0.020)\). Nearly everything ends in state 2, which keeps 99% of its population each step.

The California–Massachusetts example

Setup. Each year, people move between two states:

\begin{bmatrix} u_{\text{Cal}} \\ u_{\text{Mass}} \end{bmatrix}_{k+1} = \begin{bmatrix} .9 & .2 \\ .1 & .8 \end{bmatrix}\begin{bmatrix} u_{\text{Cal}} \\ u_{\text{Mass}} \end{bmatrix}_k

Reading the columns.

Column 1: 90% of Californians stay and 10% move to Massachusetts.
Column 2: 80% of the people in Massachusetts stay and 20% move to California.

Total population is conserved. No one is gained or lost.

Limitation. The same matrix applies every year.

Start. \(u_0 = (0, 1000)\), with everyone in Massachusetts.

After one step. \(u_1 = 1000 \times (\text{column 2}) = (200, 800)\). The total is still 1000.

Eigenvalues

\(\lambda_1 = 1\), as for every Markov matrix.

\(\lambda_2 = 0.7\), from the trace: \(1.7 - 1\). The determinant confirms it: \(0.72 - 0.02 = 0.7\).

Eigenvectors

For \(\lambda_1 = 1\): \(A - I = \begin{bmatrix} -.1 & .2 \\ .1 & -.2 \end{bmatrix}\), so \(x_1 = (2, 1)\). It is positive.

For \(\lambda_2 = 0.7\): \(A - 0.7I = \begin{bmatrix} .2 & .2 \\ .1 & .1 \end{bmatrix}\), so \(x_2 = (-1, 1)\).

The solution

General form:

u_k = c_1(1)^k\begin{bmatrix} 2 \\ 1 \end{bmatrix} + c_2(0.7)^k\begin{bmatrix} -1 \\ 1 \end{bmatrix}

Constants from \(u_0 = (0, 1000)\): \(c_1 = \tfrac{1000}{3}\) and \(c_2 = \tfrac{2000}{3}\).

Check \(u_1\): \(\left(\tfrac{2000}{3} - \tfrac{1400}{3},\; \tfrac{1000}{3} + \tfrac{1400}{3}\right) = (200, 800)\). ✓

Steady state:

u_\infty = \tfrac{1000}{3}(2, 1) \approx (667, 333)

The \((0.7)^k\) term decays away.

Row-vector convention

Many texts use row vectors. In many engineering and probability texts, including most ML references, the distribution is a row vector multiplied from the left.

Effect on the matrix. That uses the transpose, so the rows sum to 1 instead of the columns. The stationary distribution then satisfies \(\pi P = \pi\).

Part 2: Projection with an orthonormal basis

Setup. \(q_1, \dots, q_n\) is an orthonormal basis of \(\mathbb{R}^n\). Expand any vector \(v\):

v = x_1q_1 + x_2q_2 + \cdots + x_nq_n

Finding \(x_1\). Take the dot product of both sides with \(q_1\). Every other term gives 0, and \(q_1^\top q_1 = 1\):

\boxed{x_i = q_i^\top v}

In matrix form. The expansion is \(Qx = v\), so

x = Q^{-1}v = Q^\top v

No system needs to be solved, because \(Q^{-1} = Q^\top\).

Part 3: Fourier series

The expansion. Write a \(2\pi\)-periodic function as

f(x) = a_0 + a_1\cos x + b_1\sin x + a_2\cos 2x + b_2\sin 2x + \cdots

Fourier's idea. This is the same problem as the vector expansion above, but now:

the vectors are functions,
the basis is infinite: \(1, \cos x, \sin x, \cos 2x, \sin 2x, \dots\),
the space is infinite-dimensional.

Inner product of functions

Vectors	Functions
\(v^\top w = v_1w_1 + \cdots + v_nw_n\)	\(\displaystyle\langle f, g\rangle = \int_0^{2\pi} f(x)\,g(x)\,dx\)

What changes.

Multiplying component by component becomes multiplying the functions at every \(x\).
Summation becomes integration.
The interval is \([0, 2\pi]\) because every basis function is periodic with period \(2\pi\), and \(f(x + 2\pi) = f(x)\).

The basis is orthogonal. For example:

\int_0^{2\pi}\sin x\cos x\,dx = \left[\tfrac12\sin^2 x\right]_0^{2\pi} = 0

Every other pair of distinct basis functions also integrates to 0, with the help of trig identities.

Finding the coefficients

Same trick as for vectors. Multiply both sides by \(\cos x\) and integrate. Every term but one gives 0:

\int_0^{2\pi} f(x)\cos x\,dx = a_1\int_0^{2\pi}\cos^2 x\,dx = a_1\,\pi

\boxed{a_1 = \frac{1}{\pi}\int_0^{2\pi} f(x)\cos x\,dx}

The constant term. \(a_0\) is the average value of \(f\) over a period.

A precision note. Strang calls the basis "orthonormal". Strictly it is only orthogonal:

\(\int_0^{2\pi}\cos^2 x\,dx = \pi\), not 1.
\(\int_0^{2\pi} 1\,dx = 2\pi\), not 1.

This is why the formula divides by \(\pi\), and by \(2\pi\) for \(a_0\). Normalizing the basis would make the formula exactly \(x_i = q_i^\top v\).

Big ideas from Lecture 24

Markov matrices have nonnegative entries and columns summing to 1.
\(\lambda = 1\) is always an eigenvalue. The columns of \(A - I\) sum to 0, so \((1, \dots, 1)\) is in its left nullspace.
Other eigenvalues satisfy \(|\lambda| \le 1\), and the \(\lambda = 1\) eigenvector is nonnegative. That eigenvector is the steady state.
\(\text{eig}(A) = \text{eig}(A^\top)\), but the eigenvectors are generally different.
Orthonormal expansion: \(x_i = q_i^\top v\), since \(Q^{-1} = Q^\top\).
Fourier series are the same expansion in function space, with \(\langle f, g\rangle = \int fg\). Each coefficient is an inner product divided by the basis function's squared length.

Connection to your research.

Spectral gap. The rate of convergence to the stationary distribution is set by \(|\lambda_2|\), here \(0.7^k\). The gap \(1 - |\lambda_2|\) is what controls mixing time in Markov-chain sequence tasks.
When convergence fails. The caveat "\(|\lambda| \le 1\)" is exactly where convergence can fail. A periodic chain such as \(\begin{bmatrix} 0 & 1 \\ 1 & 0\end{bmatrix}\) has \(\lambda = -1\) and oscillates forever. With all entries strictly positive, every other eigenvalue satisfies \(|\lambda| < 1\) (Perron–Frobenius), and convergence is guaranteed.