Lecture 26: Complex Matrices; Fast Fourier Transform

Topics:

How lengths, inner products, symmetry, and orthogonality change when vectors and matrices are complex.
The Fourier matrix, the most important complex matrix.
The Fast Fourier Transform (FFT), which multiplies by an $n \times n$ Fourier matrix in about $\tfrac12 n\log_2 n$ steps instead of $n^2$.

1. Complex vectors: length and inner product

A complex vector $z = (z_1, \dots, z_n)$ lives in $\mathbb{C}^n$ instead of $\mathbb{R}^n$.

$z^\top z$ is the wrong length. Take $z = (1, i)$. Then

z^\top z = 1 + i^2 = 0

But a nonzero vector should not have length zero.

The fix: conjugate one factor.

\|z\|^2 = \bar{z}^\top z = |z_1|^2 + \cdots + |z_n|^2

For $z = (1, i)$, this gives $1 + (-i)(i) = 2$, so $\|z\| = \sqrt2$.

Rule: whenever you transpose a complex object, also take the complex conjugate.

Notation. One symbol does both operations:

z^H = \bar{z}^\top

The H honors Hermite. The adjective is "Hermitian".

Inner product.

Real vectors: $y^\top x$.
Complex vectors: $y^Hx$.

Note:

The inner product is now usually a complex number.
The inner product of a vector with itself is its length squared, $z^Hz$, which is real and positive.

2. The real-to-complex dictionary

Real ($\mathbb{R}^n$)	Complex ($\mathbb{C}^n$)
$x^\top x = \Vert x\Vert ^2$	$z^Hz = \Vert z\Vert ^2$
$y^\top x$	$y^Hx$
Symmetric: $A^\top = A$	Hermitian: $A^H = A$
Orthogonal: $Q^\top Q = I$	Unitary: $Q^HQ = I$

Hermitian matrices

The diagonal must be real. Each diagonal entry has to equal its own conjugate.

Off-diagonal entries are conjugates of each other. Example:

A = \begin{bmatrix} 2 & 3 + i \\ 3 - i & 5 \end{bmatrix}

Same good properties as symmetric matrices. Hermitian matrices have real eigenvalues and perpendicular eigenvectors. This is the result proved in Lecture 25.

Unitary matrices

Orthonormal columns, complex version.

q_i^Hq_j = \begin{cases} 0 & i \neq j \\ 1 & i = j \end{cases} \qquad\Longleftrightarrow\qquad Q^HQ = I

The inverse is free. $Q^{-1} = Q^H$.

3. The Fourier matrix

Definition

Electrical engineers index from 0, so rows and columns run from $0$ to $n-1$. Strang adopts that convention for this lecture.

F_n = \begin{bmatrix} 1 & 1 & 1 & \cdots & 1 \\ 1 & w & w^2 & \cdots & w^{n-1} \\ 1 & w^2 & w^4 & \cdots & w^{2(n-1)} \\ \vdots & & & & \vdots \\ 1 & w^{n-1} & w^{2(n-1)} & \cdots & w^{(n-1)^2} \end{bmatrix}, \qquad (F_n)_{jk} = w^{jk}

Properties of $F_n$.

Every entry is a power of a single number $w$.
No entry is zero.
$F_n$ is symmetric, in the plain transpose sense.

The number $w$

w = e^{2\pi i/n} = \cos\frac{2\pi}{n} + i\sin\frac{2\pi}{n}

Location. $w$ lies on the unit circle, at $\frac1n$ of the full turn.

Its powers.

Every power of $w$ also lies on the unit circle. Squaring keeps the absolute value at 1 and doubles the angle.
$w^n = 1$.
So the powers $1, w, w^2, \dots, w^{n-1}$ are the $n$ roots of 1, evenly spaced around the circle.

Polar form is the right form for powers. For example, $w^{10} = e^{20\pi i/n}$ is immediate. Real and imaginary parts would hide the pattern.

Example, $n = 6$. $w$ is at 60°, $w^2$ at 120°, and so on, with $w^6 = 1$.

Sign convention. Many texts use $e^{-2\pi i/n}$ for the forward transform. This changes nothing structurally.

The $4\times4$ case

Here $w = e^{2\pi i/4} = i$. Its powers are $i,\ -1,\ -i,\ 1$.

F_4 = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & i & i^2 & i^3 \\ 1 & i^2 & i^4 & i^6 \\ 1 & i^3 & i^6 & i^9 \end{bmatrix} = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & i & -1 & -i \\ 1 & -1 & 1 & -1 \\ 1 & -i & -1 & i \end{bmatrix}

The exponent in each position is (row number) × (column number).

The columns are orthogonal (with conjugation)

Test columns 1 and 3, which are $(1, i, -1, -i)$ and $(1, -i, -1, i)$.

Without conjugating the result looks wrong:

\[1 + 1 + 1 + 1 = 4\]

With conjugation (the correct inner product):

1\cdot1 + (-i)(-i) + (-1)(-1) + (i)(i) = 1 - 1 + 1 - 1 = 0

The columns are orthogonal but not unit vectors. Each column has length squared 4, so length 2.

Normalizing makes $F_4$ unitary. $\tfrac12 F_4$ satisfies $(\tfrac12F_4)^H(\tfrac12F_4) = I$.

So the inverse is easy:

F_4^{-1} = \tfrac14\,F_4^H

Consequence. Any fast method for multiplying by $F$ also works for $F^{-1}$. The forward and inverse Fourier transforms are equally fast.

4. The Fast Fourier Transform

The cost of direct multiplication

$F_n$ is full, so multiplying it by a vector takes $n^2$ multiplications.

What Fourier knew. The inverse of $F_n$ is easy, because the columns are orthogonal.

What Fourier missed. $F_n$ factors into pieces that are mostly zeros. Gauss noticed this but didn't make a point of it.

The key connection: $w_{64}^2 = w_{32}$

Squaring $w$ doubles the angle. So the square of the 64th root of 1 is exactly the 32nd root of 1.

This links $F_{64}$ to $F_{32}$:

F_{64} = \begin{bmatrix} I & D \\ I & -D \end{bmatrix}\begin{bmatrix} F_{32} & 0 \\ 0 & F_{32} \end{bmatrix} P

The three factors:

$P$ is a permutation. It puts the even-indexed components first, $x_0, x_2, x_4, \dots$, then the odd-indexed ones, $x_1, x_3, \dots$. It costs nothing, since it can be built into the program.
The middle factor applies $F_{32}$ separately to the even part and the odd part. Half of this block matrix is zero.
The left factor is the fix-up. It uses $D = \text{diag}(1, w, w^2, \dots, w^{31}), \qquad w = w_{64}$ and costs about 32 multiplications, for $D$.

Cost after one step:

64^2 \;\longrightarrow\; 2\cdot32^2 + 32

Recursion

Repeat the split inside each $F_{32}$. Each $F_{32}$ becomes two copies of $F_{16}$, plus a fix-up and a permutation.

The permutations combine. Together they sort the input into four groups: even-even, even-odd, odd-even, odd-odd. Continuing all the way down, the middle reduces to trivial 1-point transforms.

What remains is the fix-ups.

Each level costs about $n/2 = 32$ multiplications.
There are $\log_2 64 = 6$ levels: $64 \to 32 \to 16 \to 8 \to 4 \to 2 \to 1$.

\boxed{\text{FFT cost} \approx \tfrac12\,n\log_2 n}

The payoff

For $n = 2^{10} = 1024$:

Direct multiplication: $n^2 \approx 1{,}000{,}000$.
FFT: $\tfrac12 \cdot 1024 \cdot 10 = 5 \cdot 1024 \approx 5{,}000$.

That is a speedup of about 200×, gained just by factoring the matrix properly.

Why it matters. Fourier transforms run constantly in scientific computing. Strang notes the FFT transformed whole industries, and it all rests on the structure of one matrix with orthogonal columns.

Big ideas from Lecture 26

Conjugate whenever you transpose: $\|z\|^2 = z^Hz$ and $\langle y, x\rangle = y^Hx$.
Hermitian ($A^H = A$) is the complex version of symmetric. It has real eigenvalues and orthogonal eigenvectors.
Unitary ($Q^HQ = I$) is the complex version of orthogonal, with $Q^{-1} = Q^H$.
Fourier matrix: $(F_n)_{jk} = w^{jk}$ with $w = e^{2\pi i/n}$. Its columns are orthogonal, and $F_n^{-1} = \frac1n F_n^H$.
FFT: factor $F_n$ into two copies of $F_{n/2}$, plus an even/odd permutation and a diagonal fix-up. Recursing gives about $\tfrac12 n\log_2 n$ operations instead of $n^2$.

Connection to ML.

The FFT is what makes long convolutions fast.
The Fourier matrix diagonalizes every circulant (cyclic convolution) matrix. This is why a convolution becomes elementwise multiplication in the frequency domain.
This trick underlies FFT-based convolution layers, the long-convolution kernels in S4-style state-space models, and FNet's Fourier mixing.

Real (\(\mathbb{R}^n\))	Complex (\(\mathbb{C}^n\))
\(x^\top x = \Vert x\Vert ^2\)	\(z^Hz = \Vert z\Vert ^2\)
\(y^\top x\)	\(y^Hx\)
Symmetric: \(A^\top = A\)	Hermitian: \(A^H = A\)
Orthogonal: \(Q^\top Q = I\)	Unitary: \(Q^HQ = I\)