Lecture 2: Elimination with Matrices

Goal: Solve \(Ax = b\) by elimination, the method every software package uses (not determinants). Then rewrite each elimination step as a matrix multiplication.

1. The example system

\begin{aligned} x + 2y + z &= 2 \\ 3x + 8y + z &= 12 \\ 4y + z &= 2 \end{aligned} \qquad A = \begin{bmatrix} 1 & 2 & 1 \\ 3 & 8 & 1 \\ 0 & 4 & 1 \end{bmatrix}, \quad b = \begin{bmatrix} 2 \\ 12 \\ 2 \end{bmatrix}

Strang works with the matrix alone, leaving out the \(x\), \(y\), \(z\) and the plus signs. He handles the right-hand side \(b\) at the end, which he notes is also how MATLAB does it.

2. Forward elimination: from \(A\) to \(U\)

Step (2,1). The first pivot is the 1 in position (1,1). The multiplier is 3. Subtract 3 × row 1 from row 2:

\begin{bmatrix} 1 & 2 & 1 \\ 0 & 2 & -2 \\ 0 & 4 & 1 \end{bmatrix}

The purpose is to eliminate \(x\) from equation 2.

Step (3,1). Position (3,1) is already 0, so the multiplier is 0 and nothing changes. Software would still check this entry unless told in advance.

Step (3,2). The second pivot is the 2 in position (2,2). The multiplier is 2. Subtract 2 × row 2 from row 3:

U = \begin{bmatrix} 1 & 2 & 1 \\ 0 & 2 & -2 \\ 0 & 0 & 5 \end{bmatrix}

Results:

The pivots are 1, 2, 5. \(U\) is upper triangular.
Strang calls going from \(A\) to \(U\) the most common calculation in scientific computing.
Preview of a later topic: the determinant equals the product of the pivots, so \(\det A = 1\cdot 2\cdot 5 = 10\).
Pivots cannot be zero. If a zero appears in a pivot position, that position has no pivot.

3. How elimination can fail

Temporary failure (fixed by a row exchange). A zero lands in a pivot position, but a row below has a nonzero entry in that column. Swap the rows and continue.

If entry (1,1) were 0, you would swap row 1 with a lower row.
If the 8 were a 6, the second pivot would come out 0. Row 3 has a 4 below it, so a row exchange rescues elimination.

Complete failure (singular matrix). A zero lands in a pivot position and every entry below it is also zero.

If the (3,3) entry were −4 instead of 1, the last step would give \(-4 - 2(-2) = 0\).
No row is available to swap, so there is no third pivot.
The matrix is then not invertible.

4. Back substitution

Augmented matrix. Attach \(b\) as an extra column, written \([A \mid b]\), and apply the same row operations to it:

b = \begin{bmatrix} 2\\12\\2\end{bmatrix} \;\to\; \begin{bmatrix} 2\\6\\2\end{bmatrix} \;\to\; c = \begin{bmatrix} 2\\6\\-10\end{bmatrix}

In the first step, \(12 - 3(2) = 6\). In the second step, \(2 - 2(6) = -10\).

Elimination turns \(A\) into \(U\) and \(b\) into \(c\). So \(Ax=b\) becomes \(Ux = c\):

\begin{aligned} x + 2y + z &= 2 \\ 2y - 2z &= 6 \\ 5z &= -10 \end{aligned}

Solve from the bottom up, which works because \(U\) is triangular:

From the last equation: \(z = -2\).
Then \(2y + 4 = 6\), so \(y = 1\).
Then \(x + 2 - 2 = 2\), so \(x = 2\).

Solution: \((x, y, z) = (2, 1, -2)\).

5. Rows vs. columns in multiplication

Product	Result
Matrix × column vector	A combination of the columns (from Lecture 1). Example: \(A\,(3,4,5)^\top = 3\,\text{col}_1 + 4\,\text{col}_2 + 5\,\text{col}_3\)
Row vector × matrix	A combination of the rows. Example: \((1,2,7)\,A = 1\,\text{row}_1 + 2\,\text{row}_2 + 7\,\text{row}_3\)

Elimination works on rows, so the row view is the one needed in this lecture.

6. Elimination matrices

Each step is a matrix. Every elimination step equals multiplying on the left by a simple matrix, written \(E_{ij}\). It is the identity matrix with the negative multiplier placed in position \((i,j)\).

Step (2,1): subtract 3 × row 1 from row 2.

E_{21} = \begin{bmatrix} 1 & 0 & 0 \\ -3 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}

Rows 1 and 3 of \(E_{21}\) are rows of the identity, so those rows of \(A\) are copied unchanged.
Row 2 of \(E_{21}\) is \((-3, 1, 0)\), so the new row 2 is \(-3\,\text{row}_1 + \text{row}_2\).

Checking one entry. Entry (2,3) of the product comes from row 2 of \(E_{21}\) dotted with column 3 of \(A\):

\[(-3)(1) + (1)(1) + (0)(1) = -2\]

This matches the −2 in position (2,3) after the first step.

Step (3,2): subtract 2 × row 2 from row 3.

E_{32} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & -2 & 1 \end{bmatrix}

The identity matrix \(I\) does nothing: \(IA = A\). It plays the role of the number 1 for matrices.

7. All of elimination in one matrix

E_{32}(E_{21}A) = U \quad\Longrightarrow\quad (E_{32}E_{21})\,A = U

Associative law. You can move the parentheses: \(A(BC) = (AB)C\). So the single matrix \(E = E_{32}E_{21}\) performs the whole elimination at once.

Strang says many proofs in linear algebra come down to just moving parentheses.
Proving the law itself requires going through the full details of matrix multiplication.

No commutative law. In general \(AB \neq BA\). You must keep the matrices in their order.

8. Permutation matrices (row exchanges)

Finding \(P\). To build the matrix that exchanges rows, apply the same exchange to the identity matrix:

P = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}, \qquad P\begin{bmatrix} a & b \\ c & d\end{bmatrix} = \begin{bmatrix} c & d \\ a & b \end{bmatrix}

Left vs. right. Exchanging columns requires putting \(P\) on the right:

\begin{bmatrix} a & b \\ c & d\end{bmatrix} P = \begin{bmatrix} b & a \\ d & c \end{bmatrix}

Multiplying on the left performs row operations.
Multiplying on the right performs column operations.

9. Preview: inverses

A better direction. Instead of asking how to go from \(A\) to \(U\), ask how to go from \(U\) back to \(A\). This leads to the LU factorization in the next lectures.

Inverse of an elimination step. \(E_{21}\) subtracts 3 × row 1 from row 2. To undo it, add 3 × row 1 back to row 2:

E_{21}^{-1} = \begin{bmatrix} 1 & 0 & 0 \\ 3 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}, \qquad E_{21}^{-1}E_{21} = I

Big ideas from Lecture 2

Elimination turns \(A\) into the upper triangular \(U\), and turns \(b\) into \(c\). Back substitution then solves \(Ux = c\) from the bottom up.
Pivots are the nonzero diagonal entries of \(U\). Their product is the determinant.
Failure cases:
- A zero in a pivot position with a nonzero below it is fixed by a row exchange.
- A zero in a pivot position with only zeros below means the matrix is singular.
Every row operation is a left multiplication by an elimination matrix \(E_{ij}\) or a permutation matrix \(P\). Column operations multiply on the right.
Matrix multiplication is associative, so the steps combine into one matrix \(E\). It is not commutative, so the order matters.
Inverse of an elimination step: undo it by adding back what was subtracted.

Side question: what is a singular matrix?

Your question What do you mean by singular matrices? Is a singular matrix one whose determinant is zero?

Yes. A singular matrix is a square matrix that is not invertible, and that is equivalent to \(\det A = 0\).

For an \(n \times n\) matrix \(A\), these statements are all equivalent. If one is true, all are:

\(A\) is singular (no \(A^{-1}\) exists).
\(\det A = 0\).
Elimination gives fewer than \(n\) pivots, because a zero appears in a pivot position with nothing nonzero below it.
The columns are linearly dependent: some column is a combination of the others.
\(Ax = 0\) has a nonzero solution \(x\).
\(Ax = b\) has no solution for some \(b\), and infinitely many solutions for others.
\(0\) is an eigenvalue of \(A\).

Example from Lecture 2: changing the (3,3) entry to \(-4\) made the third pivot 0. The pivots would then be \(1, 2, 0\), so \(\det A = 1 \cdot 2 \cdot 0 = 0\). That matrix is singular.