Lecture 31: Change of Basis; Image Compression

Two parts:

1. Applications. Image compression is a change of basis, followed by throwing away small coefficients.
2. Linear algebra. How coordinates change under a new basis, and how the matrix of a transformation changes. The answer: the two matrices are similar.

Part 1: Image compression

The setup

An image is a huge vector.

• A \(512 \times 512\) grayscale image has \(512^2 = 2^{18}\) pixels.
• Each pixel is a gray level from 0 to 255, which takes 8 bits.
• So the image is a vector \(x \in \mathbb{R}^n\) with \(n = 512^2\).
• A color image needs about three times that.

Why compress. Data volumes from images, video, satellites, simulations, and the web are overwhelming. Strang notes that these lecture videos are themselves compressed: the static blackboard comes through clearly, but his movement needs many bits and looks jumpy.

Lossless vs. lossy.

• Lossless compression exploits exact redundancy and loses nothing.
• Lossy compression is the topic here. It discards information the eye won't miss.

The standard basis is a bad basis

What the standard basis does. Each coordinate is one pixel value.

What it ignores. Neighboring pixels are highly correlated. A blank blackboard has nearly identical gray levels everywhere, and so does a stretch of blue shirt. Listing a blank board pixel by pixel wastes almost everything.

Better basis vectors to include:

The stakes. Choosing the basis is "a billion-dollar decision". TV and film industries prefer different bases. Strang calls it a giant political question that reduces to a linear algebra problem.

JPEG: the Fourier basis on \(8\times8\) blocks

JPEG stands for Joint Photographic Experts Group. It works in three steps.

Step 1: change basis (lossless).

• Split the image into \(8\times8\) blocks, each with 64 pixel values.
• Express each block in a Fourier basis of \(\mathbb{R}^{64}\).
• 64 pixels in, 64 coefficients \(c\) out.
• Nothing is lost at this step.

Step 2: threshold (lossy).

• Set every coefficient below a threshold to zero, giving \(\hat{c}\) with many zeros.
• The threshold is chosen so the eye barely notices.
• The constant ("DC", zero-frequency) coefficient is usually large and kept.
• High-frequency coefficients are tiny in smooth images and get dropped.

Step 3: reconstruct.

• The sum now has only a few terms.
• Going from 64 coefficients to about 3 is roughly 21:1 compression.

Other applications

FBI fingerprints.

• Fingerprints were stored on paper in Washington, and a search could take a week.
• Digitizing them didn't solve the problem, because each image is still too large to search.
• After compression, the files were indexed at a new facility in West Virginia. A search now takes minutes.
• The FBI chose a smooth wavelet basis.

Video. Consecutive frames are highly correlated. So encode prediction plus correction: predict that the next frame matches the previous one, then compress only the small correction.

The general principle. Compression works by exploiting correlation in space or time.

Wavelets: the competition to Fourier

The simplest Haar wavelet basis for \(\mathbb{R}^8\) has these eight vectors:

Expressing a Fourier vector in this basis. The fastest-oscillating vector \((1,-1,1,-1,1,-1,1,-1)\) equals \(w_5 + w_6 + w_7 + w_8\).

Changing to the wavelet basis is solving a linear system

One row of pixels. Take eight pixel values \(p = (p_1, \dots, p_8)\), written in the standard basis. Find the coefficients \(c\) in the wavelet basis:

The wavelet matrix. \(W\) has the wavelets as its columns.

What makes a basis good

1. Fast. You must be able to multiply by \(W\) and by \(W^{-1}\) quickly.

• Fourier has the FFT.
• Haar wavelets have a fast wavelet transform.
• Their entries are only \(0\) and \(\pm1\), so multiplication is trivial in binary.
• The vectors are orthogonal. Check: \(w_1\cdot w_2\) has four \(+1\)s and four \(-1\)s, so it is 0. The other pairs work the same way.

Normalizing the wavelets. Divide each vector by its length:

• \(w_1, w_2\) by \(\sqrt8\),
• \(w_3, w_4\) by \(2\),
• \(w_5\) through \(w_8\) by \(\sqrt2\).

Then \(W\) is orthogonal, so \(W^{-1} = W^\top\). Inverting is as fast as multiplying.

2. Compresses well. A few basis vectors should come close to the signal.

• The standard basis is fastest of all, since there is nothing to compute. But it compresses terribly: throwing away 90% of the pixels just darkens the picture.
• With Fourier or wavelets, the dropped coefficients \(c_5, \dots, c_8\) are small blips the eye won't miss.

Standards at the time.

• JPEG used Fourier.
• JPEG 2000 included wavelets.
• The FBI uses smoother wavelets than Haar's sharp jumps.
• Strang says the competition for better bases "is not over yet".

Part 2: The linear algebra of changing basis

Changing coordinates

Let the columns of \(W\) be the new basis vectors. Then the old coordinates \(x\) and new coordinates \(c\) of the same vector are related by

Changing the matrix of a transformation

Setup. Take one linear transformation \(T\) on \(\mathbb{R}^n\).

• In basis \(v_1, \dots, v_n\), its matrix is \(A\).
• In basis \(w_1, \dots, w_n\), its matrix is \(B\).

\(A\) and \(B\) are similar: \(B = M^{-1}AM\), where \(M\) is the change-of-basis matrix.

Strang admits this part of the lecture is "compressed". Here is the one-line reason.

Derivation (my addition).

• Let \(M\) convert new coordinates to old: \(x = Mc\).
• To apply \(T\) in new coordinates, first go to old coordinates: \(Mc\).
• Apply \(A\) there: \(AMc\).
• Come back to new coordinates: \(M^{-1}AMc\).
• So the matrix in the new basis is \(B = M^{-1}AM\).

This explains Lecture 28. Similar matrices are the same transformation seen in two coordinate systems. That is why they share eigenvalues.

Recap: building the matrix \(A\)

Why the basis images are enough. Every \(x = c_1v_1 + \cdots + c_nv_n\), and by linearity

Building the columns. Expand each output in the basis:

The best basis: eigenvectors

Suppose the basis consists of eigenvectors, so \(T(v_i) = \lambda_iv_i\).

Each column is already expanded. The output \(\lambda_iv_i\) is just \(\lambda_i\) times the \(i\)-th basis vector, with zeros for the rest. So column \(i\) has \(\lambda_i\) in position \(i\) and zeros elsewhere.

Why not use eigenvectors in practice. For images this would be the ideal basis. But computing the eigenvectors for each image costs too much. So practitioners use a cheap, nearly-as-good basis such as Fourier or wavelets.

Big ideas from Lecture 31

1. Compression = change of basis + thresholding. The change of basis is lossless. Dropping small coefficients is where the loss happens.
2. The standard basis is poor for images, because it ignores correlation between neighboring pixels.
3. A good basis must be fast, with quick \(W\) and \(W^{-1}\) (orthogonality helps), and must compress well, so that a few coefficients capture most of the signal.
4. Coordinates change by \(x = Wc\), or \(c = W^{-1}x\).
5. Matrices change by similarity: \(B = M^{-1}AM\). The same \(T\) in different bases gives similar matrices.
6. The eigenvector basis makes the matrix diagonal. It is the ideal basis, but usually too expensive to compute.

Connection to your first question. The optimal basis for compressing a dataset in the least-squares sense is the eigenvector basis of its covariance matrix. That is exactly PCA, also called the Karhunen–Loève transform. The MLT lecture's "data compression" was this same idea, with the best possible basis computed from the data. JPEG uses Fourier because it's fast and nearly as good for natural images.

Connection to interpretability. Sparse autoencoders follow the same "few coefficients" principle. They learn an (overcomplete) basis in which activations have sparse coefficients.