Lecture 23: Differential Equations and $e^{At}$

Goal. Solve first-order linear systems with constant coefficients:

\frac{du}{dt} = Au

Key idea. Solutions are exponentials. Finding what goes in the exponent and what multiplies it is a linear algebra problem, solved by eigenvalues and eigenvectors. This lecture runs completely parallel to Lecture 22: powers $\lambda^k$ there become exponentials $e^{\lambda t}$ here.

1. Worked example

\frac{du_1}{dt} = -u_1 + 2u_2, \qquad \frac{du_2}{dt} = u_1 - 2u_2, \qquad A = \begin{bmatrix} -1 & 2 \\ 1 & -2 \end{bmatrix}, \quad u(0) = \begin{bmatrix} 1 \\ 0 \end{bmatrix}

What to expect. At the start everything is in $u_1$. Since $du_2/dt > 0$ while $u_1 > 0$, material flows from $u_1$ into $u_2$ over time.

Step 1: Eigenvalues

Spot them without computing.

$A$ is singular: column 2 is $-2 \times$ column 1. So $\lambda_1 = 0$.
The trace is $-3$, so $\lambda_2 = -3$.

Check with the characteristic polynomial:

\det(A - \lambda I) = \lambda^2 + 3\lambda = \lambda(\lambda + 3)

Step 2: Eigenvectors

For $\lambda_1 = 0$. The eigenvector is in the nullspace of $A$: $x_1 = (2, 1)$.

For $\lambda_2 = -3$. The eigenvector is in the nullspace of $A + 3I = \begin{bmatrix} 2 & 2 \\ 1 & 1 \end{bmatrix}$: $x_2 = (1, -1)$.

Step 3: Pure exponential solutions

Each eigenvalue–eigenvector pair gives a solution:

u = e^{\lambda t}x

Check. Substitute into the equation:

Left side: $\dfrac{d}{dt}\big(e^{\lambda t}x\big) = \lambda e^{\lambda t}x$.
Right side: $A\big(e^{\lambda t}x\big) = e^{\lambda t}Ax = \lambda e^{\lambda t}x$.
They match because $Ax = \lambda x$.

General solution:

u(t) = c_1e^{\lambda_1t}x_1 + c_2e^{\lambda_2t}x_2 = c_1\begin{bmatrix} 2 \\ 1 \end{bmatrix} + c_2e^{-3t}\begin{bmatrix} 1 \\ -1 \end{bmatrix}

Comparison with Lecture 22:

Difference equation $u_{k+1} = Au_k$	Differential equation $du/dt = Au$
$u_k = \sum c_i\,\lambda_i^k\,x_i$	$u(t) = \sum c_i\,e^{\lambda_it}\,x_i$

Step 4: Coefficients from the initial condition

At $t = 0$ every exponential equals 1:

c_1\begin{bmatrix} 2 \\ 1 \end{bmatrix} + c_2\begin{bmatrix} 1 \\ -1 \end{bmatrix} = \begin{bmatrix} 1 \\ 0 \end{bmatrix}

Solve. $c_1 = c_2 = \tfrac13$.

This is $Sc = u(0)$, where $S$ is the eigenvector matrix. You decide at the start how much of each pure exponential is present, and then each one evolves on its own.

Step 5: Steady state

The $e^{-3t}$ term decays to zero, leaving only the constant part:

u(\infty) = \tfrac13\begin{bmatrix} 2 \\ 1 \end{bmatrix} = \begin{bmatrix} 2/3 \\ 1/3 \end{bmatrix}

Where steady states come from. A zero eigenvalue gives a constant term, $e^{0t} = 1$. With every other eigenvalue decaying, that constant term is the steady state.

A conserved quantity. The total $u_1 + u_2$ stays equal to 1 throughout. The columns of $A$ sum to zero, so $\frac{d}{dt}(u_1 + u_2) = 0$.

2. Stability, steady state, and blow-up

Behavior	Condition on the eigenvalues
Stable: $u(t) \to 0$	$\text{Re}(\lambda) < 0$ for every $\lambda$
Steady state	one $\lambda = 0$, and all others have $\text{Re}(\lambda) < 0$
Blow-up	any $\lambda$ with $\text{Re}(\lambda) > 0$

Only the real part matters

Example: $\lambda = -3 + 6i$. Split the exponential:

e^{\lambda t} = e^{-3t}\,e^{6it}, \qquad e^{6it} = \cos 6t + i\sin 6t

Size. $|e^{6it}| = \sqrt{\cos^2 6t + \sin^2 6t} = 1$. This factor just runs around the unit circle. So $|e^{\lambda t}| = e^{-3t}$, which decays.

Interpretation. The real part decides growth or decay. The imaginary part only produces oscillation.

Reversing the sign of $A$

$-A$ has eigenvalues $0$ and $+3$. The $e^{+3t}$ term then blows up.

3. The $2\times2$ stability test

For $A = \begin{bmatrix} a & b \\ c & d \end{bmatrix}$, both eigenvalues have negative real part exactly when:

\boxed{\text{trace} = a + d < 0 \quad\text{and}\quad \det A > 0}

Why both conditions.

Trace $= \lambda_1 + \lambda_2$. If both are negative, the sum is negative. For a complex pair, the imaginary parts cancel in the sum.
Determinant $= \lambda_1\lambda_2$. If both are negative, the product is positive.

A negative trace alone is not enough. $\begin{bmatrix} -2 & 0 \\ 0 & 1 \end{bmatrix}$ has trace $-1$, but its eigenvalue $+1$ blows up. Its determinant is $-2 < 0$, so the test correctly rejects it.

4. Uncoupling the system with $S$

Change variables. Write $u = Sv$, where $S$ is the eigenvector matrix:

S\frac{dv}{dt} = ASv \;\;\Longrightarrow\;\; \frac{dv}{dt} = S^{-1}AS\,v = \Lambda v

The equations are now separate:

\frac{dv_i}{dt} = \lambda_iv_i

Each one is a scalar equation, so $v_i(t) = e^{\lambda_it}v_i(0)$.

The original coupling disappears. $A$ mixed $u_1$ and $u_2$. In the eigenvector basis, $\Lambda$ is diagonal and nothing is mixed.

Back in the original variables:

u(t) = S\,e^{\Lambda t}\,S^{-1}u(0)

5. The matrix exponential $e^{At}$

Definition. Copy the Taylor series for $e^x$:

e^{At} = I + At + \frac{(At)^2}{2!} + \frac{(At)^3}{3!} + \cdots + \frac{(At)^n}{n!} + \cdots

This series always converges, for every matrix $A$ and every $t$. The factorials grow fast enough to overwhelm any powers.

Side comment: the geometric series

(I - At)^{-1} = I + At + (At)^2 + (At)^3 + \cdots

This one does not always converge. It requires every eigenvalue of $At$ to have magnitude less than 1.

Practical use. For small $t$, $(I - At)^{-1} \approx I + At$.

Connecting $e^{At}$ to $S$ and $\Lambda$

Substitute $A^n = S\Lambda^nS^{-1}$ into the series, and write the identity as $I = SS^{-1}$:

e^{At} = SS^{-1} + S\Lambda S^{-1}t + \frac{S\Lambda^2S^{-1}t^2}{2} + \cdots = S\Big(I + \Lambda t + \frac{(\Lambda t)^2}{2} + \cdots\Big)S^{-1}

\boxed{e^{At} = S\,e^{\Lambda t}\,S^{-1}}

The exponential of a diagonal matrix is diagonal:

e^{\Lambda t} = \begin{bmatrix} e^{\lambda_1t} & & \\ & \ddots & \\ & & e^{\lambda_nt} \end{bmatrix}

This formula needs $A$ to be diagonalizable. For the few degenerate matrices without $n$ independent eigenvectors, $S^{-1}$ doesn't exist. The series definition still works, and Strang notes these matrices can still be made triangular instead of diagonal.

The solution in one line:

u(t) = e^{At}u(0)

6. Two stability regions in the complex plane

	Powers $A^k \to 0$	Exponentials $e^{At} \to 0$
Condition	$\lvert\lambda\rvert < 1$	$\text{Re}(\lambda) < 0$
Region	inside the unit circle	the left half-plane

7. A higher-order equation as a first-order system

One second-order equation:

\[y'' + by' + ky = 0\]

The trick from Fibonacci. Make the unknown a vector, $u = (y', y)$, and add the trivial equation $y' = y'$:

u' = \begin{bmatrix} y'' \\ y' \end{bmatrix} = \begin{bmatrix} -b & -k \\ 1 & 0 \end{bmatrix}\begin{bmatrix} y' \\ y \end{bmatrix}

Structure of the matrix.

The first row holds the coefficients of the equation.
The rows below put a 1 just below the diagonal. Each such row encodes a trivial equation.

Generalization. A 5th-order equation becomes a $5\times5$ system with the same pattern: the equation's coefficients across the first row and four rows of trivial equations below. Its eigenvalues are tied directly to the original equation.

Big ideas from Lecture 23

Pure solutions of $du/dt = Au$ are $e^{\lambda t}x$, built from an eigenvalue and its eigenvector.
General solution: $u(t) = \sum c_ie^{\lambda_it}x_i$, with the coefficients from $Sc = u(0)$.
Matrix exponential: $e^{At} = Se^{\Lambda t}S^{-1}$, where $e^{\Lambda t}$ is diagonal with entries $e^{\lambda_it}$.
Stability: every $\text{Re}(\lambda) < 0$. A steady state comes from $\lambda = 0$ with all other eigenvalues decaying.
$2\times2$ test: trace $< 0$ and determinant $> 0$.
Powers vs. exponentials: powers need $|\lambda| < 1$, exponentials need $\text{Re}(\lambda) < 0$.
Higher-order equations become first-order systems with this coefficient-row-plus-ones matrix pattern.

Connection to ML.

Continuous-time Markov chains. The matrix $A$ in this example, with columns summing to 0 and a conserved total, is a continuous-time Markov chain generator. Its steady state $(2/3, 1/3)$ is the stationary distribution.
Structured state-space models. Models like S4 and Mamba discretize $du/dt = Au$ through $e^{A\Delta t}$. Keeping every $\text{Re}(\lambda) < 0$ is exactly how they stay stable over long sequences.

Lecture 23: Differential Equations and \(e^{At}\)