Lecture 30 · Regression

The Geometry of Linear Regression: a Projection

Stop picturing data points. Picture each feature as one long vector holding that feature's value for every person. All weighted combinations of these vectors form a flat plane. Linear regression finds the point on that plane closest to the label vector $y$: it projects $y$ onto the plane.

Source: this lecture on YouTube (≈13 min). Previous: L29 · Next: L31.

1 · Features as vectors in ℝⁿ

The lecture's small example: $n=3$ people and $d=2$ features, height and weight.

	height	weight	label y
person 1	2	0	1
person 2	1	1	3
person 3	0	2	1

(Made-up, scaled numbers so the picture is easy to read.)

Read the table by columns. The height column $F_1=(2,1,0)$ is a vector in $\mathbb R^3$, one entry per person. The same goes for weight, $F_2=(0,1,2)$, and for the labels $y$. These are not the data points; each is a feature across all the data points.

2 · The plane of all possible predictions

Any weighting of the features, $aF_1+bF_2$, is a vector of predictions for the three people. As $a$ and $b$ vary, these vectors fill a 2-D plane through the origin: the subspace spanned by the features. In matrix form, $aF_1+bF_2=X^\top\alpha$ with $\alpha=(a,b)$.

Usually $y$ is not on this plane, because no weighting of the features explains the labels exactly. The best we can do is find the point on the plane closest to $y$.

y₁ = y₂ = y₃ = rotate view

features F₁, F₂ and their planelabel vector yprojection X⊤w* (closest point)residual y − X⊤w*

–

w* = (XX⊤)⁻¹Xy

–

projection X⊤w*

–

residual · F₁ and residual · F₂

3 · Closest point means a right angle

Call the closest point $X^\top\alpha^\star$. The line from it up to $y$, the residual $y-X^\top\alpha^\star$, must be perpendicular to the plane. If it leaned at all, you could slide along the plane and get closer. Perpendicular to the plane means perpendicular to every vector in it:

$$\big(y-X^\top\alpha^\star\big)^\top X^\top\alpha^\star=0 .$$

Substitute the closed-form solution from L29, $w^\star=(XX^\top)^{+}Xy$, for $\alpha^\star$. Using the fact that $XX^\top$ is symmetric, the equation holds (the lecture leaves the algebra as an exercise). So:

Geometric meaning: $X^\top w^\star$ is the projection of the label vector $y$ onto the subspace spanned by the features. Linear regression is exactly that projection, and $w^\star$ holds the coordinates of the projected point in terms of the features.

The demo checks this numerically: the residual's dot product with both $F_1$ and $F_2$ is 0.

4 · The lecture's exercise: what if y is on the plane?

Then $y$ is its own closest point, so the residual is $0$. Some exact weighting of the features gives every label, and the squared error is zero: the noise-free case from L28. Try it in the demo by setting $y=(2,2,2)$, which is $F_1+F_2$.

Key takeaways

Treat each feature as a vector in $\mathbb R^n$ (one entry per data point). Their combinations form a subspace, a plane here.
Linear regression finds the point on that subspace closest to the label vector $y$.
That point is $X^\top w^\star$, the orthogonal projection of $y$, and the residual is perpendicular to every feature vector.
Zero training error ⇔ $y$ already lies in the feature subspace.