Lecture 27: Positive Definite Matrices and Minima

Strang's three goals:

1. Tests: how to tell whether a symmetric matrix is positive definite.
2. Meaning: why it matters. It decides whether a point is a minimum.
3. Geometry: positive definite matrices give ellipses and ellipsoids. Indefinite ones give hyperbolas and saddles.

The quantity to watch is \(x^\top Ax\). All matrices in this lecture are symmetric.

1. Four equivalent tests

For a symmetric \(2\times2\) matrix \(A = \begin{bmatrix} a & b \\ b & c \end{bmatrix}\), each of these is a complete test:

Test 4 is usually the definition. Strang stars it. Most presentations define positive definite by test 4 and then prove that tests 1–3 are equivalent.

2. One matrix, three values of \(c\)

The test that decides everything is \(\det = 2c - 36 > 0\), which means \(c > 18\).

The borderline case \(c = 18\)

Singular matrix. Row 2 is a multiple of row 1, so the matrix has rank 1.

Eigenvalues. One eigenvalue is 0. The trace is 20, so the other is 20.

Name. Eigenvalues \(\geq 0\) with a zero among them is called positive semidefinite. Here \(x^\top Ax \geq 0\) for all \(x\), with equality along the nullspace.

Eigenvalue signs without computing eigenvalues (\(c = 20\))

• \(\lambda_1\lambda_2 = \det = 4 > 0\), so both eigenvalues have the same sign.
• \(\lambda_1 + \lambda_2 = \text{trace} = 22 > 0\), so they can't both be negative.
• Therefore both are positive.

3. The quadratic form \(x^\top Ax\)

Multiplying out:

What kind of function this is.

• Every term has degree exactly 2. There is no linear part and no constant.
• It is called a quadratic form.

Reading the matrix entries off the form.

• Diagonal entries give the squared terms.
• Each off-diagonal entry appears twice in the cross term.

For \(c = 7\):

At \((x, y) = (1, -1)\):

Graph of \(f(x, y)\)

Indefinite case (\(c = 7\)): a saddle.

• \(f\) goes up along some directions, such as the \(x\)-axis and the \(y\)-axis.
• \(f\) goes down along others, such as \(x = -y\).
• The origin is a maximum in some directions and a minimum in others.
• The best directions to look are the eigenvector directions.

Positive definite case (\(c = 20\)): a bowl. The origin is a true minimum, and \(f > 0\) everywhere else.

The intuition. Positive definiteness means the squared terms \(ax^2\) and \(cy^2\) are large enough to overwhelm the cross term \(2bxy\), which can take either sign.

4. Completing the square = elimination

The goal. Show the bowl really goes up everywhere. The way to do that is to write \(f\) as a sum of squares.

For \(c = 20\):

How it works.

• \(2(x + 3y)^2\) supplies the \(2x^2\) and the \(12xy\).
• It also supplies \(18y^2\).
• The remaining \(20 - 18 = 2\) gives the second square.

Comparing the three cases:

The numbers are elimination's numbers

Eliminate on \(A\):

Where they appear in the squares.

• Outside the squares are the pivots: 2 and 2.
• Inside the squares is the multiplier: \(\ell_{21} = 3\).

Completing the square is Gaussian elimination. The pivots are the weights on the squares. That is why positive pivots give a sum of positive squares, hence \(x^\top Ax > 0\), hence a bowl with a minimum.

It generalizes. Elimination works for any \(n\times n\) matrix. So \(x^\top Ax\) always splits into \(n\) squares weighted by the \(n\) pivots.

5. Connection to calculus: the second-derivative test

The matrix of second derivatives in 2D:

Why it is symmetric. \(f_{xy} = f_{yx}\): mixed partial derivatives can be taken in either order.

The calculus condition, explained.

• Being positive definite here means \(f_{xx} > 0\) and \(f_{xx}f_{yy} > f_{xy}^2\).
• That inequality is just the \(2\times2\) determinant test.
• Positive pure second derivatives are not enough. They must be large enough to overcome the cross derivative.

What linear algebra adds. The same test now works in \(n\) dimensions, using pivots, determinants, or eigenvalues of an \(n\times n\) matrix.

6. A \(3\times3\) example

How the pivots follow from the determinants. Each pivot is the ratio of consecutive determinants: \(2/1\), \(3/2\), \(4/3\).

Checking the eigenvalues.

• Sum: \(6\), which matches the trace.
• Product: \(2\cdot(4 - 2) = 4\), which matches the determinant.

The quadratic form:

Its graph. It is a bowl (a paraboloid) in 4D: three base coordinates plus the height. Completing the square would write it as three squares weighted by the three positive pivots.

7. Geometry: ellipses and ellipsoids

Slice the bowl at height 1. Setting \(x^\top Ax = 1\) gives:

• an ellipse in 2D,
• an ellipsoid in 3D, a lopsided "football".

Slice a saddle at height 1. That gives a hyperbola.

The shape depends on the eigenvalues

Principal axis theorem

• The eigenvectors give the directions of the axes.
• The eigenvalues determine the lengths of the axes.

Precise lengths (Strang only gestures at this). Change coordinates with \(y = Q^\top x\). The equation becomes

For the \(3\times3\) example (my computation).

Big ideas from Lecture 27

1. Four equivalent tests for symmetric \(A\): positive eigenvalues, positive upper-left determinants, positive pivots, and \(x^\top Ax > 0\) for all \(x \neq 0\).
2. Semidefinite: \(\geq\) in place of \(>\). The matrix can be singular, with a zero eigenvalue.
3. Shape of the graph: \(x^\top Ax\) is a bowl (positive definite) or a saddle (indefinite).
4. Completing the square = elimination. The pivots weight the squares and the multipliers sit inside them.
5. Minima: gradient \(= 0\) and second-derivative matrix positive definite. This is the \(n\)-dimensional version of \(f'' > 0\).
6. Ellipsoids: \(x^\top Ax = 1\) has axes along the eigenvectors, with semi-axis lengths \(1/\sqrt{\lambda_i}\).

Connection to ML.

• Local minima. A local minimum requires a positive (semi)definite Hessian. In high-dimensional loss landscapes, critical points are overwhelmingly saddles: some eigenvalues positive, some negative.
• Conditioning. The ratio \(\lambda_{\max}/\lambda_{\min}\), the condition number, sets how elongated the ellipsoidal level sets are. That elongation is what slows gradient descent.
• Back to PCA. The covariance ellipsoid has its long axes along the top principal components.