Lecture 4: Factorization into $A = LU$

Topics:

Inverse of a product, and inverse of a transpose
$A = LU$, the matrix form of Gaussian elimination
The cost of elimination (operation count)
Permutation matrices, as a preview of row exchanges

1. Inverse of a product

(AB)^{-1} = B^{-1}A^{-1}

The inverses come in reverse order. Check it using associativity:

(AB)(B^{-1}A^{-1}) = A(BB^{-1})A^{-1} = AA^{-1} = I

Strang's analogy: you put on socks, then shoes. To undo this, you take off the shoes first, then the socks.

2. Inverse of a transpose

Start from $AA^{-1} = I$ and transpose both sides. A transpose of a product also reverses order, $(AB)^\top = B^\top A^\top$, and $I^\top = I$. So

(A^{-1})^\top A^\top = I \quad\Longrightarrow\quad (A^\top)^{-1} = (A^{-1})^\top

For a single matrix, transposing and inverting can be done in either order.

3. $A = LU$ for a 2×2 matrix

Take

A = \begin{bmatrix} 2 & 1 \\ 8 & 7 \end{bmatrix}

Strang avoids a 4 in the (2,2) position, because that would make the second pivot 0 and the matrix singular.

Elimination. The multiplier is 4. Subtracting 4 × row 1 from row 2 is the matrix

E_{21} = \begin{bmatrix} 1 & 0 \\ -4 & 1 \end{bmatrix}, \qquad E_{21}A = U = \begin{bmatrix} 2 & 1 \\ 0 & 3 \end{bmatrix}

Move $E$ to the other side. Multiply both sides by $E_{21}^{-1}$:

A = E_{21}^{-1}U = LU, \qquad L = \begin{bmatrix} 1 & 0 \\ 4 & 1 \end{bmatrix}

The inverse has a plus sign, because it adds back what $E_{21}$ removed.

What $L$ and $U$ look like.

$L$ is lower triangular with 1s on the diagonal.
$U$ is upper triangular with the pivots on the diagonal.

The $LDU$ variant

Pull the pivots out of $U$ into a diagonal matrix $D$. Dividing each row of $U$ by its pivot leaves 1s on the diagonal:

A = \begin{bmatrix} 1 & 0 \\ 4 & 1 \end{bmatrix}\begin{bmatrix} 2 & 0 \\ 0 & 3 \end{bmatrix}\begin{bmatrix} 1 & \tfrac12 \\ 0 & 1 \end{bmatrix} = LDU

Now both triangular factors have 1s on the diagonal, and the pivots sit in $D$. Strang mostly uses plain $LU$.

4. $A = LU$ for a 3×3 matrix (no row exchanges)

Forward elimination as matrices:

E_{32}\,E_{31}\,E_{21}\,A = U

Moving the $E$'s to the right. Invert each one, in reverse order:

A = E_{21}^{-1}\,E_{31}^{-1}\,E_{32}^{-1}\,U = LU

Why $L$ is better than $E$

Example. Suppose the multiplier for step (2,1) is 2, step (3,1) needs nothing ($E_{31} = I$), and the multiplier for step (3,2) is 5.

The forward product $E = E_{32}E_{21}$ is messy:

E = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & -5 & 1 \end{bmatrix}\begin{bmatrix} 1 & 0 & 0 \\ -2 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} = \begin{bmatrix} 1 & 0 & 0 \\ -2 & 1 & 0 \\ \mathbf{10} & -5 & 1 \end{bmatrix}

The 10 appears because row 1 affects row 3 indirectly. Two copies of row 1 were removed from row 2, and then five copies of the new row 2 were removed from row 3. In total, 10 copies of row 1 entered row 3.

The inverse product $L = E_{21}^{-1}E_{32}^{-1}$ is clean:

L = \begin{bmatrix} 1 & 0 & 0 \\ 2 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}\begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 5 & 1 \end{bmatrix} = \begin{bmatrix} 1 & 0 & 0 \\ 2 & 1 & 0 \\ 0 & 5 & 1 \end{bmatrix}

In this order, the multipliers don't interfere with each other. No 10 appears.

The key result

If there are no row exchanges, the multipliers go directly into $L$.

Building $L$ takes no extra work: record each multiplier $\ell_{ij}$ in position $(i,j)$.

Practical consequence. As elimination proceeds, you save each new row of $U$ and each multiplier into $L$. After that, you no longer need $A$, because everything is stored in $L$ and $U$.

5. How much does elimination cost?

What counts as one operation: one multiply plus one subtract, as in "multiply row 1 by $\ell$ and subtract it from row $i$".

Cost step by step, for an $n\times n$ matrix with no zeros:

The first stage clears column 1. It touches about $n^2$ entries.
The second stage works on the remaining $(n-1)\times(n-1)$ block, costing about $(n-1)^2$.
The stages keep shrinking down to $1^2$.

Total cost for $A$:

n^2 + (n-1)^2 + \cdots + 1^2 \;\approx\; \int_0^n x^2\,dx = \frac{n^3}{3}

The cost is not the full $n \cdot n^2 = n^3$, because the matrices being worked on get smaller at each stage. That shrinking is where the factor $\tfrac13$ comes from.

Cost for each right-hand side $b$: about $n^2$. This covers carrying $b$ through elimination plus back substitution.

Practical consequence. Factor $A = LU$ once, at a cost of $\sim n^3/3$. Then each new $b$ costs only $\sim n^2$.

Comparison with determinants. Strang gives determinants a "bad mark": computing them by expansion costs on the order of $n!$, which is far worse.

6. Permutation matrices

Row exchanges are needed when a zero appears in a pivot position. A permutation matrix is the identity matrix with its rows reordered.

All 3×3 permutation matrices (there are 6):

The identity $I$ (no exchange).
Three that exchange one pair of rows: $P_{12}$, $P_{13}$, $P_{23}$.
Two that cycle all three rows. For example, $\begin{bmatrix} 0&1&0\\0&0&1\\1&0&0\end{bmatrix}$ moves row 2 up to row 1, row 3 up to row 2, and row 1 down to row 3.

Properties:

The 6 matrices form a group. The product of two permutation matrices is again on the list, and so is the inverse of any one.
The inverse is the transpose: $P^{-1} = P^\top$.
A single exchange undoes itself. For example, $P_{12}^{-1} = P_{12}$.
The count is $n!$. There are 6 for 3×3 and 24 for 4×4.

In the next lecture, permutation matrices are used to handle elimination when row exchanges are needed.

Big ideas from Lecture 4

Reverse order: $(AB)^{-1} = B^{-1}A^{-1}$ and $(AB)^\top = B^\top A^\top$.
Transpose and inverse commute: $(A^\top)^{-1} = (A^{-1})^\top$.
$A = LU$: $L$ is lower triangular with 1s on the diagonal and holds the multipliers. $U$ is upper triangular with the pivots on the diagonal. This is Gaussian elimination written as a factorization.
$L$ is clean, $E$ is not: in $L$ the multipliers sit untouched, while in $E$ they mix (the stray 10).
Cost: about $\tfrac{n^3}{3}$ operations to factor $A$, and about $n^2$ for each right-hand side.
Permutations: there are $n!$ of them, they form a group, and $P^{-1} = P^\top$.

Lecture 4: Factorization into \(A = LU\)

1. Inverse of a product

2. Inverse of a transpose

3. \(A = LU\) for a 2×2 matrix

The \(LDU\) variant

4. \(A = LU\) for a 3×3 matrix (no row exchanges)

Why \(L\) is better than \(E\)

The key result

5. How much does elimination cost?

6. Permutation matrices

Big ideas from Lecture 4