Lecture 20: Cramer's Rule, Inverse Matrix, and Volume

The final lecture on determinants: three applications.

An explicit formula for $A^{-1}$.
Cramer's rule for solving $Ax = b$.
$|\det A|$ = the volume of a box.

1. Formula for the inverse

The $2\times2$ case:

\begin{bmatrix} a & b \\ c & d \end{bmatrix}^{-1} = \frac{1}{ad - bc}\begin{bmatrix} d & -b \\ -c & a \end{bmatrix}

The entries are cofactors.

$d$ is the cofactor of $a$.
$-b$ is the cofactor of $c$. Its sign is minus because $c$ sits in position $(2,1)$, and $2+1$ is odd.
The cofactors land in transposed positions.

The general formula:

\boxed{A^{-1} = \frac{1}{\det A}\,C^\top}

Here $C$ is the cofactor matrix, with entries $C_{ij}$.

Degree count. For $3\times3$, $\det A$ is a sum of products of 3 entries, such as $aei$. Each cofactor is a product of 2 entries, such as $ei - fh$. In general, $\det A$ has degree $n$ and the cofactors have degree $n-1$.

What the formula adds. Gauss–Jordan computes $A^{-1}$ numerically, but never shows what the inverse is. This formula gives the algebra, not just the algorithm.

Proof: $AC^\top = (\det A)\,I$

Diagonal entries. Row $i$ of $A$ times its own cofactors gives

a_{i1}C_{i1} + \cdots + a_{in}C_{in} = \det A

This is exactly the cofactor formula from Lecture 19.

Off-diagonal entries are 0. Row $i$ of $A$ times the cofactors of a different row $k$ is the cofactor expansion, along row $k$, of a matrix $A_S$. Here $A_S$ is $A$ with row $k$ replaced by row $i$. Strang calls it "A screwed up."

$A_S$ has two equal rows, so $\det A_S = 0$.

$2\times2$ illustration. Row 1 is $(a, b)$, and the cofactors of row 2 are $(-b, a)$. Their product is $-ab + ba = 0$, which is the expansion of $\det\begin{bmatrix} a & b \\ a & b\end{bmatrix}$.

Payoff. The formula shows how $A^{-1}$ changes when one entry of $A$ changes: you can track the effect on $\det A$ and on each cofactor.

2. Cramer's rule

Start from the inverse formula:

x = A^{-1}b = \frac{1}{\det A}\,C^\top b

The $j$-th component of $C^\top b$:

C_{1j}b_1 + C_{2j}b_2 + \cdots + C_{nj}b_n

This is the cofactor expansion down column $j$ of a new matrix.

Define $B_j$: $A$ with column $j$ replaced by $b$.

\boxed{x_j = \frac{\det B_j}{\det A}}

Example. For $x_1$, use $B_1 = [\,b \;\; a_2 \;\cdots\; a_n\,]$. Expanding down column 1 multiplies $b_1$ by $C_{11}$, $b_2$ by $C_{21}$, and so on. Those are exactly the cofactors of column 1 of $A$.

Is Cramer's rule practical?

No. It requires $n + 1$ determinants. Computing them is hopelessly slow compared with elimination, and MATLAB never uses it.

Strang's anecdote. A popular book he read as a child, Mathematics for the Million, promoted Cramer's rule over elimination. Strang jokes it should have been called Mathematics for the Millionaire, since only a millionaire could pay for the computation.

He attributes the book to Bell. It is actually by Lancelot Hogben; E. T. Bell wrote the similarly popular Men of Mathematics.

Where its value lies. Like the inverse formula, Cramer's rule gives explicit algebraic formulas you can reason with. Use elimination for numbers.

3. Determinant = volume

$|\det A|$ = volume of the box whose edges are the rows of $A$. Using columns gives the same result, since $\det A^\top = \det A$.

The box. For $3\times3$, the three rows are vectors from the origin. Completing them gives a parallelepiped whose faces are parallelograms.

The sign. The sign of $\det A$ records orientation: whether the box is right-handed or left-handed. Swapping two edges flips the orientation but not the volume.

Proof strategy: show volume satisfies Properties 1, 2, 3

If volume obeys the same three defining properties as the determinant, it must equal the determinant (up to sign).

Property 1. $A = I$. The box is the unit cube, with volume 1 and $\det I = 1$.

Bonus: orthogonal $Q$. The box is the unit cube, rotated, so its volume is still 1. Check against the determinant:

Q^\top Q = I \;\Rightarrow\; \det Q^\top\det Q = 1 \;\Rightarrow\; (\det Q)^2 = 1 \;\Rightarrow\; \det Q = \pm1

This uses Properties 9 and 10. Strang notes it is a homework exercise.

Property 2. Row exchange. The box is the same. Only the sign changes, and $|\det A|$ ignores sign.

Property 3A. Scaling a row by $t$. One edge is stretched by $t$, so the volume is multiplied by $t$. For example, doubling one edge stacks two identical boxes.

Property 3B. Linearity in one row. Strang leaves this to the book's figure and does not prove it in lecture.

Area of a parallelogram

The formula. For the parallelogram with edges $(a, b)$ and $(c, d)$:

\text{Area} = |ad - bc|

Why this is good. Base × height requires lengths, square roots, and angles. This formula uses only the coordinates. Strang emphasizes that this formula impresses him far more than Cramer's rule.

Area of a triangle

With one vertex at the origin. The triangle with vertices $(0,0)$, $(a,b)$, $(c,d)$ is half the parallelogram:

\text{Area} = \tfrac12|ad - bc|

With arbitrary vertices $(x_1,y_1)$, $(x_2,y_2)$, $(x_3,y_3)$:

\text{Area} = \frac12\left|\det\begin{bmatrix} x_1 & y_1 & 1 \\ x_2 & y_2 & 1 \\ x_3 & y_3 & 1 \end{bmatrix}\right|

Why it works. Subtract row 1 from rows 2 and 3. This zeroes the 1s in the last column and leaves the coordinate differences. That is the same as translating the triangle so one vertex sits at the origin. Strang moves fast at this point because time runs out; the $\tfrac12$ and the absolute value follow from the case above.

Big ideas from Lecture 20

Inverse formula: $A^{-1} = C^\top/\det A$, which rests on the identity $AC^\top = (\det A)I$.
Off-diagonal zeros: entries of one row times cofactors of another row give the determinant of a matrix with two equal rows, which is 0.
Cramer's rule: $x_j = \det B_j/\det A$, where $B_j$ is $A$ with column $j$ replaced by $b$. It is elegant algebra but useless for computation.
Volume: $|\det A|$ is the volume of the box spanned by the rows. The sign gives orientation.
Orthogonal matrices: $\det Q = \pm1$, so the rotated unit cube has volume 1.
Coordinate formulas: a parallelogram has area $|ad - bc|$, and a triangle has half that, with no lengths or angles needed.

Side question: do determinants matter for ML?

Your question Is this (determinants, cofactors, Cramer's rule) important for ML and AI?

Partly. Two of the three topics matter little for ML, but one matters a lot.

Low priority: Cramer's rule and the cofactor formula for $A^{-1}$

You will essentially never use these in ML.

Computation. Nobody computes inverses or solves systems this way. Libraries use LU, Cholesky, or QR, and good practice avoids forming $A^{-1}$ at all (solve, not inv).
The one indirect use. The cofactor matrix shows up in Jacobi's formula, $d(\det A) = \operatorname{tr}(\text{adj}(A)\,dA)$, which gives $\frac{\partial}{\partial A}\log\det A = A^{-\top}$ That derivative does appear in ML derivations (Gaussian likelihoods, covariance learning). You don't need cofactors to use it, though.

High priority: determinant as volume

This idea is everywhere in probabilistic ML, because $|\det|$ measures how a linear map scales volume, and volume scaling is exactly what changes a probability density.

Change of variables / normalizing flows. If $y = f(x)$, then

p_Y(y) = p_X(x)\,\left|\det J_f(x)\right|^{-1}

Normalizing flows (RealNVP, Glow, etc.) are designed so that $\log|\det J|$ is cheap to compute, typically with triangular Jacobians, where the determinant is the product of the diagonal. That's Property 7 from Lecture 18.

Multivariate Gaussians. The density contains $(\det \Sigma)^{-1/2}$:

$\log\det\Sigma$ appears in the log-likelihood, entropy, and KL divergence between Gaussians (as in VAEs).
It is the main expensive term in Gaussian process marginal likelihoods. It is computed via Cholesky: $\log\det\Sigma = 2\sum_i \log L_{ii}$.

Determinantal point processes (DPPs). These are used for diverse subset selection. The probability of a set is $\det K_S$, which equals the squared volume spanned by the items' feature vectors. Larger volume means more diverse items.

Orthogonal matrices and $\det = \pm1$. Orthogonal matrices preserve volume. This is relevant to orthogonal initialization and to volume-preserving layers.

What matters more for you

Everything from the next lecture onward is core ML material:

Eigenvalues and eigenvectors: PCA, spectral methods, Hessian analysis, training dynamics.
Symmetric and positive definite matrices: covariances, kernels, convexity.
SVD: low-rank structure, compression, and the rank-1 decompositions widely used in interpretability work on weight matrices.

So skim Cramer's rule, keep "det = volume scaling" firmly in mind, and give your full attention to Lectures 21 onward.