Revision Notes & Formula Sheet (all 34 lectures, ML lens)

How to use this doc

This doc condenses all 34 lectures of MIT 18.06 Linear Algebra (Spring 2010, Prof. Gilbert Strang) into one place. It is written in the same style as our chat and read through an ML/DL/AI lens. It ends where it started: PCA from your MLT lecture, rebuilt from the course.

Markings used throughout

Notation. Matrices are \(m\times n\) with rank \(r\). \(A^{\mathsf{T}}\) is the transpose, \(\hat{x}\) a least-squares solution, \(Q\) a matrix with orthonormal columns, \(\Lambda\) a diagonal eigenvalue matrix, \(S\) an eigenvector matrix, \(\Sigma\) a singular-value matrix.

If you need X for ML, go to Y

Part I — Solving \(Ax=b\) (L1–L4)

The one idea to keep from Part I: \(Ax\) is a combination of the columns of \(A\) [L1]. Everything else is elimination organized as matrix factorization.

Three pictures of \(Ax=b\) [L1]

• Row picture. Each equation is a line (2D) or plane (3D); the solution is where they meet. Hard to see beyond 3D.

• Column picture (the important one). Find the weights \(x_1,\dots,x_n\) so that \(x_1\,\mathrm{col}_1+\cdots+x_n\,\mathrm{col}_n=b\).

• Matrix form. \(Ax=b\).

Example: \(2x-y=0,\ -x+2y=3\) has solution \((1,2)\), and indeed

ML link. A linear layer \(y=Wx\) outputs a combination of \(W\)’s columns, so every output lives in the column space of \(W\). Its rank caps what the layer can represent.

Elimination and pivots [L2]

• Subtract multiples of a pivot row to create zeros below each pivot. \(A\) goes to upper-triangular \(U\); \(b\) goes to \(c\). Then back substitution solves \(Ux=c\) from the bottom.

• Example: \(A=\left[\begin{smallmatrix}1&2&1\\3&8&1\\0&4&1\end{smallmatrix}\right]\), \(b=(2,12,2)\) \(\to\) pivots \(1,2,5\); solution \((2,1,-2)\).

• Pivots cannot be zero. A zero with a nonzero below it \(\to\) row exchange (temporary failure). A zero with only zeros below \(\to\) singular (complete failure).

• \(\det A=\pm\,\)product of pivots (sign flips per row exchange).

Elimination matrices. Each step is a left-multiplication by \(E_{ij}\): the identity with \(-(\text{multiplier})\) in position \((i,j)\). Example: \(E_{21}=\left[\begin{smallmatrix}1&0&0\\-3&1&0\\0&0&1\end{smallmatrix}\right]\).

• Left multiplication does row operations; right multiplication does column operations.

• Matrix multiplication is associative (move parentheses freely) but not commutative (\(AB\neq BA\)).

• Permutation matrices \(P\) = identity with rows reordered.

Singular, in one place (your question): a square matrix is singular

Five ways to multiply \(AB\) [L3]

ML link. The column \(\times\) row view is the outer-product view behind low-rank factorizations, LoRA (\(W+BA\)), and attention as a sum over keys.

Inverses and Gauss–Jordan [L3]

• For square \(A\), a left inverse is automatically a right inverse.

• \(A\) is singular \(\iff\) some \(x\neq0\) has \(Ax=0\). Proof: if \(A^{-1}\) existed, \(x=A^{-1}0=0\). Example: \(A=\left[\begin{smallmatrix}1&3\\2&6\end{smallmatrix}\right]\) kills \((3,-1)\).

• Gauss–Jordan: row-reduce \([\,A\mid I\,]\) to \([\,I\mid A^{-1}\,]\). This works because the combined elimination matrix \(E\) satisfies \(EA=I\), so \(E=A^{-1}\).

• Example: \(\left[\begin{smallmatrix}1&3\\2&7\end{smallmatrix}\right]^{-1}=\left[\begin{smallmatrix}7&-3\\-2&1\end{smallmatrix}\right]\).

\(A=LU\), cost, and \(PA=LU\) [L4, L5]

• \(\bm{(AB)^{-1}=B^{-1}A^{-1}}\) and \(\bm{(AB)^{\mathsf{T}}=B^{\mathsf{T}}A^{\mathsf{T}}}\) (reverse order: socks then shoes). Also \((A^{\mathsf{T}})^{-1}=(A^{-1})^{\mathsf{T}}\).

• \(A=LU\): \(L\) is lower triangular with 1s on the diagonal and the multipliers sitting untouched below it; \(U\) holds the pivots. The forward product \(E=E_{32}E_{21}\) mixes multipliers (a stray 10 appears); \(L\) does not.

• The \(LDU\) variant pulls the pivots into a diagonal \(D\).

• Cost: factoring \(A\) takes about \(n^3/3\) operations; each new right-hand side then costs about \(n^2\). Factor once, solve many times.

• Row exchanges: \(PA=LU\) covers every invertible \(A\). Software also exchanges rows to avoid small pivots (partial pivoting) for numerical accuracy [L5].

• Permutations: \(n!\) of them, they form a group, and \(P^{-1}=P^{\mathsf{T}}\).

ML link. Every linear solve in ML (Newton steps, Gaussian processes, least squares) uses \(LU\) or its symmetric cousin, Cholesky. Never compute \(A^{-1}\) explicitly; solve.

Part II — Vector spaces and the four subspaces (L5–L13)

The heart of the course: every \(m\times n\) matrix of rank \(r\) carries four subspaces with dimensions \(r,\ n-r,\ r,\ m-r\) [L10].

Vector spaces and subspaces [L5, L6]

• A vector space is closed under addition and scalar multiplication, i.e. under linear combinations. It always contains \(0\).

• Subspaces of \(\mathbb{R}^3\): \(\mathbb{R}^3\), planes through \(0\), lines through \(0\), and \(\{0\}\). A line missing the origin is not a subspace; neither is the first quadrant (it fails for negative scalars).

• The intersection of subspaces is a subspace; the union usually is not [L6]. The sum \(S+U\) is the right replacement [L11].

• Vectors need not be columns: matrices, and solutions of \(y''+y=0\) (basis \(\cos x,\ \sin x\)), form vector spaces too [L11].

Column space and nullspace [L6]

• \(\mathbf{C}(A)\) = all combinations of the columns, a subspace of \(\mathbb{R}^m\). \(Ax=b\) is solvable exactly when \(b\in\mathbf{C}(A)\).

• \(\mathbf{N}(A)\) = all \(x\) with \(Ax=0\), a subspace of \(\mathbb{R}^n\).

• Solutions of \(Ax=b\) with \(b\neq0\) do not form a subspace (\(0\) is not a solution); they form a shifted copy of \(\mathbf{N}(A)\).

• Two ways to describe a subspace: give vectors and take combinations (column space), or give equations (nullspace).

Computing \(\mathbf{N}(A)\): pivots, free variables, rref [L7]

1. Eliminate to echelon form \(U\) (skip a column with no pivot and keep going).

2. Rank \(r\) = number of pivots.

3. Pivot columns give pivot variables; the other \(\bm{n-r}\) are free variables.

4. Special solutions: set one free variable to 1 and the rest to 0, then solve for the pivot variables. \(\mathbf{N}(A)\) = all combinations of them.

Reduced row echelon form \(R=\left[\begin{smallmatrix}I&F\\0&0\end{smallmatrix}\right]\) (pivot columns first). Then the nullspace matrix is \(N=\left[\begin{smallmatrix}-F\\I\end{smallmatrix}\right]\) and \(RN=0\).

Example:

Elimination preserves \(\mathbf{N}(A)\) and the row space, but changes \(\mathbf{C}(A)\).

Which variables can be free? (your question) Not any set: a set of \(n-r\) variables can be free exactly when the other \(r\) columns form a basis of \(\mathbf{C}(A)\). The pivot choice is just the leftmost valid one. The number of free variables (\(n-r\)) and the nullspace itself never depend on the choice.

Complete solution of \(Ax=b\) and the rank table [L8]

Eliminate on \([\,A\mid b\,]\); each zero row of \(A\) gives a condition on \(b\). Then

ML link. Overparameterized models (more parameters than data, \(r=m<n\)) interpolate the data in infinitely many ways; which one training picks is the implicit-bias question (see the pseudoinverse in Part VII).

Independence, basis, dimension [L9]

• Independent: only the zero combination gives \(0\). Columns of \(A\) are independent \(\iff \mathbf{N}(A)=\{0\}\iff r=n\).

• More than \(m\) vectors in \(\mathbb{R}^m\) are always dependent (\(n>m\) forces free variables).

• Span: the space is all their combinations. Basis = independent + spanning.

• Every basis of a space has the same size, its dimension. If \(\dim=d\), any \(d\) independent vectors automatically span.

• \(\mathop{\mathrm{rank}}= \dim\mathbf{C}(A)\); the pivot columns of \(A\) are a basis. \(\dim\mathbf{N}(A)=n-r\); the special solutions are a basis.

• Rank is a property of a matrix; dimension is a property of a space.

The four fundamental subspaces [L10]

• Row rank = column rank. This is why Strang’s \((3,3,8)\) example in L9 failed: two equal rows force dependent columns (see Corrections).

• “Left” nullspace because \(A^{\mathsf{T}}y=0\) is the same as \(y^{\mathsf{T}}A=0^{\mathsf{T}}\).

Matrix spaces and rank 1 [L11]

• \(3\times3\) matrices: dim 9. Symmetric: 6. Upper triangular: 6. Diagonal (their intersection): 3. Formula: \(\dim S+\dim U=\dim(S\cap U)+\dim(S+U)\), i.e. \(6+6=3+9\).

• Every rank-1 matrix is \(uv^{\mathsf{T}}\); a rank-\(r\) matrix is a sum of \(r\) rank-1 matrices.

• Matrices of a fixed rank do not form a subspace; \(\mathop{\mathrm{rank}}(A+B)\le\mathop{\mathrm{rank}}A+\mathop{\mathrm{rank}}B\).

• \(\{v: v_1+v_2+v_3+v_4=0\}=\mathbf{N}\big(\left[\begin{smallmatrix}1&1&1&1\end{smallmatrix}\right]\big)\), dimension 3. Its left nullspace is \(\{0\}\), with dimension 0 and an empty basis.

Graphs and incidence matrices [L12]

• Incidence matrix \(A\) (edges \(\times\) nodes, one \(-1\) and one \(+1\) per row). \(Ax\) gives potential differences across edges.

• \(\mathbf{N}(A)\) = constant potentials, so \(r=n-1\). Ground one node to remove it.

• \(\mathbf{N}(A^{\mathsf{T}})\) = Kirchhoff’s Current Law (flow in = flow out at each node). Basis = currents around independent loops; \(\dim=m-r\) = number of loops.

• Independent edges form a tree. Linear algebra proves Euler’s formula: nodes \(-\) edges \(+\) loops \(=1\).

• The equilibrium equation of applied math is \(A^{\mathsf{T}}CAx=f\), and \(A^{\mathsf{T}}CA\) is symmetric.

ML link. Incidence matrices give the graph Laplacian \(L=A^{\mathsf{T}}A\), the starting point for spectral clustering and graph neural networks. Its nullspace (constants per connected component) is exactly L12’s \(\mathbf{N}(A)\).

Quiz-1 traps worth remembering [L13]

• \(\mathbf{N}(CD)=\mathbf{N}(D)\) when \(C\) is invertible.

• Invertible matrices are not a subspace; neither are singular ones.

• \(B^2=0\) does not imply \(B=0\): \(B=\left[\begin{smallmatrix}0&1\\0&0\end{smallmatrix}\right]\).

• \(m=n\) does not make row space = column space (same example).

• Row exchanges preserve the row space and nullspace, not the column space.

• A vector cannot be both a row of \(A\) and in \(\mathbf{N}(A)\): row \(\cdot\) itself \(=\lVert \text{row}\rVert^2\neq0\). This previews row space \(\perp\) nullspace.

Part III — Orthogonality, projections, least squares (L14–L17)

Least squares is projection: when \(Ax=b\) has no solution, solve \(A^{\mathsf{T}}A\hat{x}=A^{\mathsf{T}}b\). Its answer \(A\hat{x}\) is the projection of \(b\) onto \(\mathbf{C}(A)\) [L15, L16].

Orthogonal vectors and subspaces [L14]

• \(x\perp y\iff x^{\mathsf{T}}y=0\). Proof via Pythagoras: \(\lVert x\rVert^2+\lVert y\rVert^2=\lVert x+y\rVert^2\) expands to \(2x^{\mathsf{T}}y=0\).

• \(\lVert x\rVert^2=x^{\mathsf{T}}x\). The zero vector is orthogonal to everything.

• Subspaces \(S\perp T\): every vector of \(S\) is orthogonal to every vector of \(T\). They can share only \(0\) (a wall and the floor are not orthogonal).

• Row space \(\perp\) nullspace in \(\mathbb{R}^n\), read straight off \(Ax=0\) (each row dotted with \(x\) is 0). Column space \(\perp\) left nullspace in \(\mathbb{R}^m\).

• They are orthogonal complements: dimensions add to \(n\) (or \(m\)), so \(\mathbf{N}(A)\) contains every vector perpendicular to the row space. This is the Fundamental Theorem, Part 2.

The matrix \(A^{\mathsf{T}}A\) [L14, L16]

• \(A^{\mathsf{T}}A\) is square (\(n\times n\)) and symmetric.

• \(\bm{\mathbf{N}(A^{\mathsf{T}}A)=\mathbf{N}(A)}\) and \(\bm{\mathop{\mathrm{rank}}(A^{\mathsf{T}}A)=\mathop{\mathrm{rank}}(A)}\).

• \(A^{\mathsf{T}}A\) is invertible \(\iff\) \(A\) has independent columns. Proof: if \(A^{\mathsf{T}}Ax=0\), then \(x^{\mathsf{T}}A^{\mathsf{T}}Ax=\lVert Ax\rVert^2=0\), so \(Ax=0\), so \(x=0\).

Projection formulas [L15]

Onto a line through \(a\), and onto \(\mathbf{C}(A)\):

• Derivation: the error \(e=b-A\hat{x}\) must be \(\perp\mathbf{C}(A)\), i.e. \(A^{\mathsf{T}}(b-A\hat{x})=0\). So \(\bm{e\in\mathbf{N}(A^{\mathsf{T}})}\), exactly the four-subspace picture.

• \(\bm{P^{\mathsf{T}}=P}\) and \(\bm{P^2=P}\). These two properties characterize projection matrices.

• A projection onto a line has rank 1; its eigenvalues are 1 (on the line) and 0 (perpendicular).

• Do not split \((A^{\mathsf{T}}A)^{-1}=A^{-1}(A^{\mathsf{T}})^{-1}\) for rectangular \(A\): \(A^{-1}\) does not exist. Only if \(A\) is square and invertible does \(P=I\).

• \(I-P\) projects onto \(\mathbf{N}(A^{\mathsf{T}})\); every \(b\) splits as \(b=Pb+(I-P)b\), with the two parts perpendicular [L16].

Least squares, fully worked [L15, L16]

Fit \(b=C+Dt\) to \((1,1),(2,2),(3,2)\):

• \(A=\left[\begin{smallmatrix}1&1\\1&2\\1&3\end{smallmatrix}\right]\), \(b=(1,2,2)\). \(A^{\mathsf{T}}A=\left[\begin{smallmatrix}3&6\\6&14\end{smallmatrix}\right]\), \(A^{\mathsf{T}}b=(5,11)\).

• Solution \(C=\tfrac23\), \(D=\tfrac12\): the best line is \(\bm{b=\tfrac23+\tfrac12 t}\).

• \(p=\big(\tfrac76,\tfrac53,\tfrac{13}6\big)\), \(e=\big(-\tfrac16,\tfrac13,-\tfrac16\big)\). Check: \(p^{\mathsf{T}}e=0\), and \(e\perp\) both columns.

• Calculus agrees: minimizing \(E(C,D)=\sum_i(C+Dt_i-b_i)^2\) with \(\partial E/\partial C=\partial E/\partial D=0\) gives exactly \(A^{\mathsf{T}}A\hat{x}=A^{\mathsf{T}}b\).

• Two pictures: the data plane (points and best line, vertical errors) and \(\mathbb{R}^m\) (\(b\) projected onto the plane \(\mathbf{C}(A)\)). \(C\) and \(D\) are the weights of the columns that give \(p\).

• Outliers: squaring makes one bad point dominate. Least squares is the default, not sacred.

ML link. This is linear regression with MSE loss. The normal equations are the closed form, gradient descent on MSE converges to the same answer, and the residual is orthogonal to every feature column. Ridge adds \(\lambda I\) to \(A^{\mathsf{T}}A\).

Orthonormal columns: \(Q^{\mathsf{T}}Q=I\) [L17]

• Orthonormal vectors are automatically independent and numerically safe (no overflow, no blow-up).

• An orthogonal matrix is a square \(Q\) with \(Q^{\mathsf{T}}Q=I\), so \(\bm{Q^{-1}=Q^{\mathsf{T}}}\). A rectangular \(Q\) still has \(Q^{\mathsf{T}}Q=I\), but \(QQ^{\mathsf{T}}\neq I\).

• Examples: permutations, rotations \(\left[\begin{smallmatrix}\cos\theta&-\sin\theta\\ \sin\theta&\cos\theta\end{smallmatrix}\right]\), Hadamard matrices (\(\pm1\) entries; sizes beyond 2 must be multiples of 4).

• With \(Q\), everything simplifies: \(P=QQ^{\mathsf{T}}\) (no inverse) and \(\bm{\hat{x}_i=q_i^{\mathsf{T}}b}\), so each coefficient is one dot product.

Gram–Schmidt and \(A=QR\) [L17]

Subtract from each new vector its projections onto all earlier orthogonal ones, then normalize:

• Example: \(a=(1,1,1)\), \(b=(1,0,2)\) \(\to\) \(B=(0,-1,1)\); \(q_1=(1,1,1)/\sqrt3\), \(q_2=(0,-1,1)/\sqrt2\).

• Same column space, better basis.

• \(\bm{A=QR}\), with \(\bm{R=Q^{\mathsf{T}}A}\) upper triangular because each \(q_i\) is perpendicular to all earlier columns. Here \(R=\left[\begin{smallmatrix}\sqrt 3&\sqrt 3\\0&\sqrt 2\end{smallmatrix}\right]\).

• Square roots are unavoidable (normalizing divides by lengths).

ML link. \(QR\) is how least squares is solved stably in practice (better conditioned than forming \(A^{\mathsf{T}}A\)). Orthogonal initialization and orthogonality constraints on weights keep signals from exploding or vanishing, because \(\lVert Qx\rVert=\lVert x\rVert\).

Part IV — Determinants (L18–L20)

For ML, keep one idea: \(\bm{|\det A|}\) is the factor by which \(A\) scales volume [L20]. Cramer’s rule and the cofactor inverse are skimmable.

Three defining properties and seven consequences [L18]

• Software computes \(\det A=\pm(\text{product of pivots})\) via elimination.

• \(\bm{\det(tA)=t^n\det A}\), not \(t\det A\) (a factor \(t\) from each row; volume scaling).

• Well-defined because every permutation is either even or odd, never both.

Formulas [L19]

• Big formula: a sum of \(n!\) terms \(\pm\,a_{1\alpha}a_{2\beta}\cdots a_{n\omega}\), one entry from each row and column, with sign = parity of the permutation. The \(3\times3\) “diagonal” mnemonic fails for \(n\ge4\).

• Cofactors: \(C_{ij}=(-1)^{i+j}\det M_{ij}\) (delete row \(i\), column \(j\)). Then \(\det A=\sum_j a_{ij}C_{ij}\) along any row or column.

• Three ways to compute: pivots (practical), big formula (explicit), cofactors (recursive).

• Tridiagonal 1s: \(|A_n|=|A_{n-1}|-|A_{n-2}|\), giving \(1,0,-1,-1,0,1,\dots\) with period 6 (so \(|A_{61}|=1\)). The same recurrence reappears in L24b with eigenvalues \(e^{\pm i\pi/3}\).

Inverse, Cramer, volume [L20]

• \(\bm{A^{-1}=C^{\mathsf{T}}/\det A}\), from the identity \(AC^{\mathsf{T}}=(\det A)\,I\). Off-diagonal entries are 0 because they expand a matrix with two equal rows.

• Cramer’s rule: \(x_j=\det B_j/\det A\), where \(B_j\) is \(A\) with column \(j\) replaced by \(b\). Elegant algebra, useless computation (\(n+1\) determinants).

• \(\bm{|\det A|}\) = volume of the box spanned by the rows; the sign is orientation. For orthogonal \(Q\): \((\det Q)^2=1\), so \(\det Q=\pm1\) (a rotated unit cube).

• Parallelogram with edges \((a,b),(c,d)\): area \(|ad-bc|\). Triangle with vertices \((x_i,y_i)\): area \(=\tfrac12\left|\det\left[\begin{smallmatrix}x_1&y_1&1\\x_2&y_2&1\\x_3&y_3&1\end{smallmatrix}\right]\right|\).

Correction. Strang credits the book Mathematics for the Million to Bell; it is by Lancelot Hogben (E. T. Bell wrote Men of Mathematics).

What actually matters for ML (my addition)

Rule of thumb: never compute \(\det\) for large matrices directly (it under- or overflows); compute \(\bm{\log\det}\) from a factorization.

Part V — Eigenvalues and dynamics (L21–L24b)

Eigenvalues decide what repeated application of \(A\) does: growth, decay, oscillation, or a steady state [L22, L23].

Definition and method [L21]

• \(\bm{Ax=\lambda x}\): \(x\) keeps its direction; \(\lambda\) is the stretch (it can be negative, zero, or complex).

• \(\lambda=0\iff A\) singular; its eigenvectors are \(\mathbf{N}(A)\).

• Method: solve \(\det(A-\lambda I)=0\) for \(\lambda\), then find \(\mathbf{N}(A-\lambda I)\) for \(x\).

• Checks: \(\sum\lambda_i=\mathop{\mathrm{tr}}A\), \(\prod\lambda_i=\det A\). For \(2\times2\): \(\det(A-\lambda I)=\lambda^2-(\mathop{\mathrm{tr}}A)\lambda+\det A\).

• Shift: \(A+cI\) has eigenvalues \(\lambda+c\) and the same eigenvectors. There is no such rule for \(A+B\) or \(AB\) in general (different eigenvectors).

• Triangular \(\Rightarrow\) eigenvalues on the diagonal. Symmetric \(\Rightarrow\) real \(\lambda\), perpendicular \(x\). Antisymmetric \(\Rightarrow\) imaginary \(\lambda\).

Diagonalization and powers [L22]

• This needs \(n\) independent eigenvectors (\(S\) invertible). Distinct eigenvalues guarantee it.

• Repeated \(\lambda\): compare the algebraic multiplicity (root count) with the geometric multiplicity (\(\dim\mathbf{N}(A-\lambda I)\)). If geometric \(<\) algebraic, \(A\) is not diagonalizable. \(I_{10}\) is fine; \(\left[\begin{smallmatrix}2&1\\0&2\end{smallmatrix}\right]\) is not.

• Stability of powers: \(A^k\to0\iff\) all \(|\lambda_i|<1\).

• Recipe: expand \(u_0=Sc\) in eigenvectors; each component scales by its own \(\lambda^k\).

• Fibonacci: \(u_k=(F_{k+1},F_k)\), \(A=\left[\begin{smallmatrix}1&1\\1&0\end{smallmatrix}\right]\), \(\lambda=\frac{1\pm\sqrt5}{2}\approx1.618,\,-0.618\). Binet *(my addition)*xmy completion: \(F_k=\dfrac{\lambda_1^k-\lambda_2^k}{\sqrt5}\). Growth rate = largest eigenvalue.

ML link. Repeated multiplication by the same matrix (RNNs, deep linear nets, backprop through time) is governed by the largest \(|\lambda|\): above 1 explodes, below 1 vanishes. The same rule governs power iteration and PageRank.

Differential equations and \(e^{At}\) [L23]

• Example: \(A=\left[\begin{smallmatrix}-1&2\\1&-2\end{smallmatrix}\right]\) (singular, trace \(-3\)) \(\to\) \(\lambda=0,-3\); \(x=(2,1),(1,-1)\). From \(u(0)=(1,0)\): \(c_1=c_2=\tfrac13\), steady state \(\bm{(\tfrac23,\tfrac13)}\); the total \(u_1+u_2\) is conserved.

• Only \(\mathop{\mathrm{Re}}\lambda\) matters: \(e^{(-3+6i)t}\) has size \(e^{-3t}\); the imaginary part just oscillates.

• Two stability regions: powers need \(|\lambda|<1\) (inside the unit circle); exponentials need \(\mathop{\mathrm{Re}}\lambda<0\) (left half-plane).

• Higher-order ODEs become first-order systems: \(y''+by'+ky=0\ \to\ u=(y',y)\), \(A=\left[\begin{smallmatrix}-b&-k\\1&0\end{smallmatrix}\right]\).

• The geometric series \((I-At)^{-1}=I+At+(At)^2+\cdots\) converges only when the eigenvalues of \(At\) are inside the unit circle; \(e^{At}\) always converges.

ML link. State-space sequence models (S4, Mamba) discretize \(du/dt=Au\) via \(e^{A\Delta t}\) and constrain \(\mathop{\mathrm{Re}}\lambda<0\) for stability. Neural ODEs are the nonlinear version. A matrix whose columns sum to 0 (like the example) is a continuous-time Markov generator.

Markov matrices [L24]

• Entries \(\ge0\), columns sum to 1. Powers stay Markov.

• \(\lambda=1\) is always an eigenvalue: the columns of \(A-I\) sum to 0, so \((1,\dots,1)\in\mathbf{N}\big((A-I)^{\mathsf{T}}\big)\) and \(A-I\) is singular.

• The other eigenvalues have \(|\lambda|\le1\); the \(\lambda=1\) eigenvector is nonnegative and is the steady state.

• \(\mathrm{eig}(A)=\mathrm{eig}(A^{\mathsf{T}})\) (same characteristic polynomial), but the eigenvectors differ.

• California–Massachusetts: \(A=\left[\begin{smallmatrix}.9&.2\\.1&.8\end{smallmatrix}\right]\), \(\lambda=1,\,0.7\); \(x=(2,1),(-1,1)\). From \((0,1000)\):

  \[u_k=\tfrac{1000}{3}\begin{bmatrix}2\\1\end{bmatrix}+\tfrac{2000}{3}(0.7)^k\begin{bmatrix}-1\\1\end{bmatrix}\ \to\ \bm{(667,\,333)}.\]

• \(3\times3\) example steady state: \((0.6,\,33,\,0.7)\approx(0.017,\,0.962,\,0.020)\) normalized.

• Many texts (and most ML) use row vectors: rows sum to 1 and \(\pi P=\pi\).

ML link (your research). Convergence to the stationary distribution goes like \(|\lambda_2|^k\); the spectral gap \(1-|\lambda_2|\) sets the mixing time of Markov-chain sequence tasks. A periodic chain such as \(\left[\begin{smallmatrix}0&1\\1&0\end{smallmatrix}\right]\) has \(\lambda=-1\) and never converges; all entries strictly positive forces every other \(|\lambda|<1\) (Perron–Frobenius) and guarantees convergence.

Orthonormal expansion and Fourier series [L24]

• With an orthonormal basis, \(v=\sum x_iq_i\) has \(\bm{x_i=q_i^{\mathsf{T}}v}\), because \(Q^{-1}=Q^{\mathsf{T}}\).

• Fourier series: \(f(x)=a_0+a_1\cos x+b_1\sin x+\cdots\) in an infinite-dimensional function space with inner product \(\langle f,g\rangle=\int_0^{2\pi}f(x)g(x)\,dx\).

• \(a_1=\frac1\pi\int_0^{2\pi}f(x)\cos x\,dx\); \(a_0\) = average of \(f\).

Correction. Strang calls the Fourier basis orthonormal; it is only orthogonal (\(\int_0^{2\pi}\cos^2x\,dx=\pi\), \(\int_0^{2\pi}1\,dx=2\pi\)), which is why the formulas divide by \(\pi\) and \(2\pi\).

Quiz-2 problems worth keeping [L24b]

• Projection onto \(a=(2,1,2)\): \(P=aa^{\mathsf{T}}/9\), eigenvalues \(1,0,0\), and \(P^k=P\), so \(u_k\) stops changing after one step.

• Line through the origin \(y=Dt\) fitted to \((1,4),(2,5),(3,8)\): \(\hat D=38/14=19/7\). The projection picture is \(b\) onto the line through \((1,2,3)\) in \(\mathbb{R}^3\), not points onto the fitted line.

• \(4\times4\) with eigenvalues \(\lambda_i\): invertible \(\iff\) no \(\lambda_i=0\); \(\det A^{-1}=1/\prod\lambda_i\); \(\mathop{\mathrm{tr}}(A+I)=\sum\lambda_i+4\).

• Tridiagonal recurrence as a system: \(\left[\begin{smallmatrix}1&-1\\1&0\end{smallmatrix}\right]\), \(\lambda=e^{\pm i\pi/3}\), \(|\lambda|=1\), \(\bm{A^6=I}\) (distinct \(\lambda\), so diagonalizable): period 6, matching L19.

Part VI — Symmetric, positive definite, complex (L25–L27)

Symmetric matrices are the most important class in ML: covariances, kernels, Hessians, Laplacians. Their whole story is \(A=Q\Lambda Q^{\mathsf{T}}\) [L25].

Spectral theorem [L25]

A real symmetric \(A\) has real eigenvalues, its eigenvectors can be chosen orthonormal, and a full set always exists:

• Each \(q_iq_i^{\mathsf{T}}\) is a projection onto the line through \(q_i\): a symmetric matrix is an eigenvalue-weighted sum of perpendicular projections.

• Names: spectral theorem (maths), principal axis theorem (mechanics).

• Proof that \(\lambda\) is real: compute \(\bar x^{\mathsf{T}}Ax\) as \(\lambda\,\bar x^{\mathsf{T}}x\) and as \(\bar\lambda\,\bar x^{\mathsf{T}}x\) (this step uses \(A^{\mathsf{T}}=A\)); cancel \(\bar x^{\mathsf{T}}x=\sum|x_i|^2>0\).

• Proof that the \(x\)’s are perpendicular (my addition): \(\lambda_1x_2^{\mathsf{T}}x_1=x_2^{\mathsf{T}}Ax_1=(Ax_2)^{\mathsf{T}}x_1=\lambda_2x_2^{\mathsf{T}}x_1\), so \((\lambda_1-\lambda_2)\,x_2^{\mathsf{T}}x_1=0\).

• Signs of pivots = signs of eigenvalues (symmetric, no row exchanges). Count the positive pivots of \(A-7I\) to count the eigenvalues above 7. Product of pivots = product of eigenvalues = \(\det\).

• Finding the roots of \(\det(A-\lambda I)\) for large \(n\) is numerically unstable; real eigenvalue algorithms work differently.

Positive definite: four equivalent tests [L25, L27]

For symmetric \(A\), each of these is a complete test:

• Checking only the full determinant is not enough: \(\left[\begin{smallmatrix}-1&0\\0&-3\end{smallmatrix}\right]\) has \(\det=3\) but is negative definite.

• Semidefinite: \(\ge\) instead of \(>\) (singular allowed, a zero eigenvalue).

• \(\left[\begin{smallmatrix}2&6\\6&c\end{smallmatrix}\right]\): positive definite for \(c>18\); \(c=18\) is semidefinite (\(\lambda=0,20\)); \(c=7\) is indefinite (\(x=(1,-1)\) gives \(-3\)).

• Completing the square = elimination: \(2x^2+12xy+20y^2=2(x+3y)^2+2y^2\). The pivots sit outside the squares; the multiplier 3 sits inside.

• \(3\times3\) example \(\left[\begin{smallmatrix}2&-1&0\\-1&2&-1\\0&-1&2\end{smallmatrix}\right]\): determinants \(2,3,4\); pivots \(2,\tfrac32,\tfrac43\); eigenvalues \(2-\sqrt2,\ 2,\ 2+\sqrt2\).

• Preserved [L28]: \(A^{-1}\) (eigenvalues \(1/\lambda\)), \(A+B\) (the energy test adds), and \(A^{\mathsf{T}}A\) is positive definite \(\iff\) \(A\) has independent columns (\(x^{\mathsf{T}}A^{\mathsf{T}}Ax=\lVert Ax\rVert^2\)). Positive definite elimination never needs row exchanges.

Minima, saddles, ellipsoids [L27]

• \(x^{\mathsf{T}}Ax>0\) everywhere \(\Rightarrow\) the graph is a bowl (minimum at 0). Indefinite \(\Rightarrow\) saddle.

• Second-derivative test in \(n\) dimensions: gradient \(=0\) and the Hessian \(\left[\begin{smallmatrix}f_{xx}&f_{xy}\\f_{yx}&f_{yy}\end{smallmatrix}\right]\) positive definite. In 2D: \(f_{xx}>0\) and \(f_{xx}f_{yy}>f_{xy}^2\).

• The level set \(x^{\mathsf{T}}Ax=1\) is an ellipse/ellipsoid with axes along the eigenvectors and semi-axis lengths \(1/\sqrt{\lambda_i}\) *(my addition)*xmy precise version; the smallest \(\lambda\) gives the longest axis.

ML link. Covariance matrices and kernel (Gram) matrices are positive semidefinite by construction (they are \(A^{\mathsf{T}}A\)). A local minimum needs a positive semidefinite Hessian; in high dimensions most critical points are saddles (mixed-sign eigenvalues). The condition number \(\lambda_{\max}/\lambda_{\min}\) sets how elongated the loss ellipsoids are. That elongation is what slows gradient descent and what preconditioning (Adam, natural gradient, Newton) corrects.

Complex vectors and matrices [L26]

• \(z=(1,i)\) has \(z^{\mathsf{T}}z=0\) but \(\lVert z\rVert=\sqrt2\): always conjugate.

• Hermitian matrices have real eigenvalues and orthogonal eigenvectors.

Fourier matrix and FFT [L26]

• \((F_n)_{jk}=w^{jk}\), \(w=e^{2\pi i/n}\) (indices from 0).

  \[F_4=\begin{bmatrix}1&1&1&1\\1&i&-1&-i\\1&-1&1&-1\\1&-i&-1&i\end{bmatrix}.\]

• The columns are orthogonal under the complex inner product; \(F_n^{-1}=\frac1n F_n^{\mathsf{H}}\).

• FFT:

  \[F_{64}=\begin{bmatrix}I&D\\I&-D\end{bmatrix}\begin{bmatrix}F_{32}&0\\0&F_{32}\end{bmatrix}P,\]
  with \(P\) the even/odd split and \(D=\mathop{\mathrm{diag}}(1,w,\dots,w^{31})\). Recurse: the cost is \(\tfrac12n\log_2n\) instead of \(n^2\). For \(n=1024\) that is about 5,000 vs 1,000,000 operations, a \(200\times\) speedup.

ML link. The Fourier matrix diagonalizes every circulant (cyclic convolution) matrix, which is why convolution becomes elementwise multiplication in frequency space. It powers FFT-based convolutions, long-convolution kernels in S4-style models, and FNet’s token mixing.

Part VII — Similarity, SVD, transformations, pseudoinverse (L28–L34)

The SVD \(A=U\Sigma V^{\mathsf{T}}\) works for every matrix and gives the best bases for all four subspaces at once [L29]. Most of modern ML’s linear algebra runs through it.

Similar matrices and Jordan form [L28]

• \(\bm{B=M^{-1}AM}\) (\(M\) invertible) \(\Rightarrow\) \(A\) and \(B\) are similar: same eigenvalues; eigenvectors move \(x\to M^{-1}x\). Also \(B^k=M^{-1}A^kM\).

• Proof: \(A(MM^{-1})x=\lambda x\ \Rightarrow\ (M^{-1}AM)(M^{-1}x)=\lambda(M^{-1}x)\).

• Example: \(\left[\begin{smallmatrix}2&1\\1&2\end{smallmatrix}\right]\) with \(M=\left[\begin{smallmatrix}1&4\\0&1\end{smallmatrix}\right]\) gives \(B=\left[\begin{smallmatrix}-2&-15\\1&6\end{smallmatrix}\right]\): trace 4, det 3, eigenvalues 3 and 1.

• Same family = same eigenvalues and same Jordan structure. \(4I\) is alone in its family; every other matrix with eigenvalues \(4,4\) is defective, with best member \(\left[\begin{smallmatrix}4&1\\0&4\end{smallmatrix}\right]\).

• Jordan theorem: every \(A\) is similar to \(J\) = blocks \(J_i\) (\(\lambda_i\) on the diagonal, 1s just above); the number of blocks is the number of independent eigenvectors. Block sizes matter: the \(3+1\) and \(2+2\) nilpotent \(4\times4\) Jordan forms both have rank 2 but are not similar (their squares have ranks 1 and 0, *(my addition)*xmy justification).

• Strang: Jordan form is numerically fragile (it needs exactly equal eigenvalues); the SVD and positive definiteness matter more today.

ML link. Similarity is a change of basis: same linear map, different coordinates. Eigenvalues, trace, det and rank are basis-independent; individual weight entries are not. That is why spectral quantities of weight matrices are meaningful under reparameterization.

SVD [L29, L32]

• Goal: an orthonormal basis \(v_i\) of the row space that \(A\) maps to an orthogonal set \(\sigma_iu_i\) in the column space; complete it with bases of \(\mathbf{N}(A)\) and \(\mathbf{N}(A^{\mathsf{T}})\) (\(\sigma=0\) there).

• \(\bm{V}\) = eigenvectors of \(A^{\mathsf{T}}A\); \(\bm{\sigma_i=\sqrt{\lambda_i(A^{\mathsf{T}}A)}\ge0}\); \(\bm{U}\) = eigenvectors of \(AA^{\mathsf{T}}\) (same nonzero eigenvalues).

• For symmetric positive definite \(A\), the SVD is \(Q\Lambda Q^{\mathsf{T}}\) (\(U=V\)).

• Example \(\left[\begin{smallmatrix}4&4\\-3&3\end{smallmatrix}\right]\): \(A^{\mathsf{T}}A=\left[\begin{smallmatrix}25&7\\7&25\end{smallmatrix}\right]\), \(\sigma=\sqrt{32},\sqrt{18}\), \(v=(1,1)/\sqrt2,\ (1,-1)/\sqrt2\), \(U=\left[\begin{smallmatrix}1&0\\0&-1\end{smallmatrix}\right]\).

• Example \(\left[\begin{smallmatrix}4&3\\8&6\end{smallmatrix}\right]\) (rank 1): \(v_1=(0.8,0.6)\), \(u_1=(1,2)/\sqrt5\), \(\sigma_1=\sqrt{125}=5\sqrt5\).

• Reading \(\Sigma\): \(\mathop{\mathrm{diag}}(3,2)\) is invertible; \(\mathop{\mathrm{diag}}(3,-5)\) is not an SVD (\(\sigma\ge0\)); \(\mathop{\mathrm{diag}}(3,0)\) has rank 1 with \(v_2\) spanning \(\mathbf{N}(A)\).

Correction (SVD sign). Strang chose \(U\) from \(AA^{\mathsf{T}}\) independently and got a row with the wrong sign. Eigenvectors are only defined up to sign, so compute \(u_i=Av_i/\sigma_i\) after choosing \(V\). He fixes this in L32.

ML link. Truncating to the top \(k\) terms \(\sum_{i\le k}\sigma_iu_iv_i^{\mathsf{T}}\) gives the best rank-\(k\) approximation (Eckart–Young): compression, low-rank adapters, denoising. The SVD of weight matrices (e.g. OV/QK circuits, embedding matrices) is a standard interpretability tool, and the singular-value spectrum shows effective rank.

Linear transformations and their matrices [L30]

• Linear: \(T(cv+dw)=cT(v)+dT(w)\), so \(T(0)=0\). Projection, rotation, and \(T(v)=Av\) are linear; the shift \(T(v)=v+v_0\) and \(T(v)=\lVert v\rVert\) are not.

• Knowing \(T\) on a basis determines \(T\) everywhere.

• Matrix of \(T\): column \(j\) = output-basis coordinates of \(T(v_j)\). Then \(A\times(\text{input coords})=(\text{output coords})\).

• The same projection: the eigenvector basis gives \(\left[\begin{smallmatrix}1&0\\0&0\end{smallmatrix}\right]\); the standard basis (45° line) gives \(\left[\begin{smallmatrix}1/2&1/2\\1/2&1/2\end{smallmatrix}\right]\).

• The derivative on \(\{1,x,x^2\}\to\{1,x\}\) is \(\left[\begin{smallmatrix}0&1&0\\0&0&2\end{smallmatrix}\right]\).

• Matrix multiplication = composition; matrix inverse = inverse transformation.

ML link. A linear layer is a linear map; adding a bias (\(Wx+b\)) is the “shift” non-example, i.e. affine, not linear.

Change of basis and compression [L31]

• New basis in the columns of \(W\): \(\bm{x=Wc}\) (\(c=W^{-1}x\)). The same \(T\) in two bases gives similar matrices \(\bm{B=M^{-1}AM}\). The eigenvector basis gives \(\Lambda\).

• Images: \(512^2\) pixels, 8 bits each; the standard basis ignores the correlation between neighbours.

• Compression = lossless change of basis + lossy thresholding + reconstruction (\(64\to\sim3\) coefficients, about 21:1).

• JPEG: Fourier on \(8\times8\) blocks. Haar wavelets for \(\mathbb{R}^8\): \((1^8)\), \((1^4,-1^4)\), \((1,1,-1,-1,0^4)\), \((0^4,1,1,-1,-1)\), then four \((1,-1)\) pairs; orthogonal, so \(W^{-1}=W^{\mathsf{T}}\) after normalizing.

• A good basis is fast (FFT, fast wavelet transform, orthogonality) and compresses well (few coefficients suffice).

• Video: encode a prediction plus a small correction between frames.

ML link. The best basis for compressing a dataset (in least squares) is the covariance eigenbasis: PCA / Karhunen–Loève. Sparse autoencoders follow the same “few coefficients” principle with a learned, overcomplete basis.

Left, right, and pseudoinverse [L33]

• Key fact: restricted to the row space, \(A\) is one-to-one onto the column space. Proof: \(Ax=Ay\) with \(x,y\) in the row space \(\Rightarrow x-y\in\mathbf{N}(A)\cap\mathbf{C}(A^{\mathsf{T}})=\{0\}\).

• \(A^{+}\) inverts that map and kills \(\mathbf{N}(A^{\mathsf{T}})\). \(\Sigma^{+}\) inverts only the nonzero \(\sigma_i\).

• Example (my addition): \(A=\left[\begin{smallmatrix}1&1\\1&1\end{smallmatrix}\right]\ \Rightarrow\ A^{+}=\tfrac14\left[\begin{smallmatrix}1&1\\1&1\end{smallmatrix}\right]\).

ML link (my addition). \(x^{+}=A^{+}b\) is the minimum-norm least-squares solution. Ridge, \((A^{\mathsf{T}}A+\lambda I)^{-1}A^{\mathsf{T}}\to A^{+}\) as \(\lambda\to0\). In overparameterized linear regression, gradient descent from zero converges to \(A^{+}b\): the classic implicit bias toward minimum norm.

Quiz-3 and final review: patterns to recognize [L32, L34]

• Orthogonal eigenvectors \(\iff AA^{\mathsf{T}}=A^{\mathsf{T}}A\) (normal): symmetric, antisymmetric and orthogonal matrices all qualify [L32].

• Antisymmetric \(\left[\begin{smallmatrix}0&-1&0\\1&0&-1\\0&1&0\end{smallmatrix}\right]\): \(\lambda=0,\pm i\sqrt2\); \(du/dt=Au\) is periodic with \(T=\pi\sqrt2\).

• From eigen-data alone (\(\lambda=0,c,2\), orthogonal \(x\)’s): diagonalizable for all \(c\); symmetric for real \(c\); never PD; PSD for \(c\ge0\); never Markov; \(A/2\) is a projection for \(c=0\) or \(2\).

• Final [L34]: “no solution for some \(b\), unique when it exists” \(\Rightarrow r=n<m\). Then \(A^{\mathsf{T}}A\) is invertible, \(AA^{\mathsf{T}}\) only semidefinite, \(\det(A^{\mathsf{T}}A)\neq\det(AA^{\mathsf{T}})\) (non-square), and \(A^{\mathsf{T}}y=c\) has infinitely many solutions.

• Markov \(\left[\begin{smallmatrix}.2&.4&.3\\.4&.2&.3\\.4&.4&.4\end{smallmatrix}\right]\) (\(\mathrm{col}_1+\mathrm{col}_2=2\,\mathrm{col}_3\)): \(\lambda=0,1,-0.2\); from \((0,10,0)\) the steady state is \(\bm{(3,3,4)}\) (total conserved).

• Least squares \(A=\left[\begin{smallmatrix}1&0\\1&1\\1&2\end{smallmatrix}\right]\), \(b=(3,4,1)\): \(\hat{x}=(\tfrac{11}3,-1)\), \(p=(\tfrac{11}3,\tfrac83,\tfrac53)\). A \(b\) with \(\hat{x}=0\) is any \(b\perp\mathbf{C}(A)\), e.g. \((1,-2,1)\).

• Not of the form \(B^{\mathsf{T}}B\): any non-symmetric matrix, or a symmetric one that is not PSD *(my addition)*xmy sharpening.

Capstone — PCA end to end [MLT L5]

PCA is projection onto the top-\(k\) eigenvectors of the covariance matrix, and nearly every part of 18.06 is used to justify it.

The pipeline, with the lecture behind each step

1. Center the data \(x_1,\dots,x_n\in\mathbb{R}^d\) (subtract the mean).

2. Covariance \(C=\frac1n\sum_ix_ix_i^{\mathsf{T}}=\frac1nX^{\mathsf{T}}X\): symmetric and positive semidefinite because it is \(A^{\mathsf{T}}A\) [L5, L28].

3. Represent a point by a line \(w\) (\(\lVert w\rVert=1\)): coefficient \(x_i^{\mathsf{T}}w\), reconstruction \((x_i^{\mathsf{T}}w)w=ww^{\mathsf{T}}x_i\), a rank-1 projection [L15].

4. Best line: minimize the average \(\lVert x_i-(x_i^{\mathsf{T}}w)w\rVert^2\). Since

   \[\lVert x-(x^{\mathsf{T}}w)w\rVert^2=\lVert x\rVert^2-(x^{\mathsf{T}}w)^2,\]
   this is the same as maximizing \(w^{\mathsf{T}}Cw\) (the variance along \(w\)).

5. Solve with a Lagrange multiplier:

   \[\nabla_w\big(w^{\mathsf{T}}Cw-\lambda(w^{\mathsf{T}}w-1)\big)=0\ \Rightarrow\ \bm{Cw=\lambda w},\qquad w^{\mathsf{T}}Cw=\lambda,\]
   so pick the largest eigenvalue [L21].

6. Residues: \(r_i=x_i-(x_i^{\mathsf{T}}w_1)w_1\) are \(\perp w_1\). Their covariance is \(C-\lambda_1w_1w_1^{\mathsf{T}}\) (deflation), so the best line for the residues is the 2nd eigenvector. Repeating gives the orthonormal eigenbasis [L25 spectral theorem].

7. Choose \(k\) by explained variance \(\dfrac{\sum_{i\le k}\lambda_i}{\sum_i\lambda_i}\ge0.95\); the reconstruction error is the sum of the dropped eigenvalues.

8. Compute via the SVD of the centered \(X\): principal directions = right singular vectors \(V\), \(\lambda_i=\sigma_i^2/n\). This is more stable than forming \(C\) [L29].

Your two questions from the start of the chat

Why is the optimal \(w\) an eigenvector?

1. Covariance enters because the loss depends on the data only through \(x_ix_i^{\mathsf{T}}\).

2. At a constrained optimum, the gradient \(Cw\) can have no component perpendicular to \(w\) (otherwise rotating \(w\) would capture more variance), so \(Cw\parallel w\).

3. In the eigenbasis, \(w^{\mathsf{T}}Cw=\sum_i\lambda_ia_i^2\) with \(\sum_ia_i^2=1\) is a weighted average of eigenvalues, maximized by putting all the weight on \(\lambda_1\).

Note: \((x^{\mathsf{T}}w)w\) is the projection, not the eigenvector; the optimal \(w\) is the eigenvector.

Multivariable version: \(k\) directions at once

• Codes \(z_i=W^{\mathsf{T}}x_i\); reconstructions \(WW^{\mathsf{T}}x_i\), with \(P=WW^{\mathsf{T}}\) the projection onto the subspace (\(Q^{\mathsf{T}}Q=I\) makes the general formula collapse) [L15, L17].

• Only the subspace is determined: \(WR\) works for any orthogonal \(R\). The eigenvector basis is the one where the codes are uncorrelated (\(W^{\mathsf{T}}CW\) diagonal).

• Minimum error \(=\lambda_{k+1}+\cdots+\lambda_d\). The truncated SVD of \(X\) is the best rank-\(k\) approximation (Eckart–Young).

Why PCA is the “best basis”

In L31’s language, PCA is the change of basis that compresses this particular dataset best in least squares (Karhunen–Loève). Fourier and wavelets are cheaper, data-independent approximations to it.

Corrections and traps caught along the way

These are the places where the board (or a first instinct) goes wrong; each one is worth remembering.

General traps

• \(\det(A+B)\neq\det A+\det B\); \(\mathrm{eig}(A+B)\neq\mathrm{eig}(A)+\mathrm{eig}(B)\) unless they share eigenvectors (e.g. \(B=cI\)).

• \(\det(AB)=\det(BA)\) only for square matrices; for rectangular \(A\), \(A^{\mathsf{T}}A\) and \(AA^{\mathsf{T}}\) differ (one can be singular).

• “Free variables” are a choice, but not any choice: the remaining columns must be a basis of \(\mathbf{C}(A)\).

• A bias term makes a layer affine, not linear (\(T(0)\neq0\)); likewise, solutions of \(Ax=b\) (\(b\neq0\)) are never a subspace.

Formula sheet

Everything above on one page, with the lecture to revisit.

Source. MIT OpenCourseWare 18.06 Linear Algebra, Spring 2010, Prof. Gilbert Strang via the video lecture page and its transcripts; the ML connections are my additions from our chat.