Lecture 33 · Regression: the probabilistic view

Probabilistic View of Linear Regression

So far, squared error was just an intuitive choice. Now assume a probabilistic mechanism generates the labels: $y=w^\top x+\epsilon$ with Gaussian noise. Regression becomes an estimation problem, and maximum likelihood gives back exactly the least-squares solution. Choosing a noise model means choosing an error function, and vice versa.

Source: this lecture on YouTube (≈15 min). Previous: L32 · Next: L34.

1 · Model the labels, not the features

The data is $\{(x_1,y_1),\dots,(x_n,y_n)\}$ with $x_i\in\mathbb R^d$ and $y_i\in\mathbb R$. We don't try to model how the features $x$ are generated. We only model how a label is produced given a feature vector:

$$y\mid x \;=\; \underbrace{w^\top x}_{\text{structure}} \;+\; \underbrace{\epsilon}_{\text{noise}},\qquad \epsilon\sim\mathcal N(0,\sigma^2).$$

$w\in\mathbb R^d$ is unknown but fixed. It's the parameter to estimate.
The noise is Gaussian with mean 0 and a known variance $\sigma^2$.
So each observed $y_i$ is a noisy version of $w^\top x_i$. The only unknown is $w$, which makes this an estimation problem, just like L19–L21 but now in a supervised setting.

2 · The likelihood

Adding a fixed number to a zero-mean Gaussian only shifts its mean (add 5 to $\mathcal N(0,\sigma^2)$ and you get $\mathcal N(5,\sigma^2)$). So

$$y_i\mid x_i\;\sim\;\mathcal N\!\left(w^\top x_i,\;\sigma^2\right).$$

With the usual i.i.d. assumption, the likelihood of $w$ is a product of Gaussian densities:

$$\mathcal L(w;\,x_1,\dots,x_n,\,y_1,\dots,y_n)=\prod_{i=1}^{n}\frac{1}{\sqrt{2\pi}\,\sigma}\exp\!\left(-\frac{(w^\top x_i-y_i)^2}{2\sigma^2}\right).$$

It depends on the data too, but we treat it as a function of $w$ alone and look for the $w$ that makes the observed labels most likely.

3 · Log-likelihood → squared error

Take the log (products become sums, and log cancels exp): $\;\log\mathcal L(w)=\sum_{i=1}^{n}\left[-\dfrac{(w^\top x_i-y_i)^2}{2\sigma^2}+\log\dfrac{1}{\sqrt{2\pi}\,\sigma}\right].$
$\sigma$ is known, so $\log\frac{1}{\sqrt{2\pi}\sigma}$ and the factor $\frac{1}{2\sigma^2}$ are constants. Drop them: $\;\max_w\;\sum_i -(w^\top x_i-y_i)^2.$
Maximising the negative is minimising the positive: $\;\min_w\;\sum_{i=1}^{n}(w^\top x_i-y_i)^2.$
That's exactly the linear regression problem with squared error from L28, so we already know the answer: $\;\boxed{\hat w_{\text{ML}}=(XX^\top)^{+}Xy=w^\star}.$

4 · The conclusion: noise ⇔ error function

Maximum likelihood with zero-mean Gaussian noise is exactly linear regression with squared error. When we picked squared error in L28 "because it looked intuitive", we were implicitly assuming that zero-mean Gaussian noise corrupts the labels.

Both parts matter. Change the noise and you change the loss. For example, with Laplacian noise the density is $\propto e^{-|w^\top x_i-y_i|/b}$, so the negative log-likelihood becomes a sum of absolute errors:

$$\epsilon\sim\text{Laplace}(0,b)\quad\Longrightarrow\quad \hat w_{\text{ML}}=\arg\min_w\sum_{i=1}^{n}\lvert w^\top x_i-y_i\rvert .$$

In general, the loss is the negative log of the noise density: choosing a noise implicitly chooses an error function, and choosing an error function implicitly assumes a noise.

Left: the loss each noise model implies, $-\log p(\epsilon)$ up to constants. Right: 40 points from the true line $y=0.5+0.8x$ (dashed) plus the chosen noise, fitted by minimising squared error and absolute error. (A constant feature $x_0=1$ gives the intercept.)

squared error ⇔ Gaussianabsolute error ⇔ Laplaciantrue line

–

squared-error fit: ‖ŵ − w‖

–

absolute-error fit: ‖ŵ − w‖

Going beyond the video. The absolute-error fit has no closed form; the demo finds it with iteratively reweighted least squares. The squared loss grows quadratically, so one far-off label pulls the fit hard, while the absolute loss grows linearly and barely notices. That's the probabilistic reason absolute error is "robust": a Laplacian has heavier tails, so it expects occasional large deviations.

5 · What else do we gain?

The equivalence is a nice justification for squared error, but the bigger gain is this: once learning is viewed as estimation, $\hat w_{\text{ML}}$ is an estimator, and all the machinery for judging estimators applies. We can ask how good $\hat w_{\text{ML}}$ is, and whether a better estimator exists. That's the next lecture.

Key takeaways

Probabilistic model: $y\mid x=w^\top x+\epsilon$, $\epsilon\sim\mathcal N(0,\sigma^2)$, with $w$ unknown but fixed. Only the labels are modelled.
Then $y_i\mid x_i\sim\mathcal N(w^\top x_i,\sigma^2)$, and maximising the log-likelihood is minimising $\sum_i(w^\top x_i-y_i)^2$.
$\hat w_{\text{ML}}=(XX^\top)^{+}Xy$, the same $w^\star$ as least squares.
Noise model ⇔ loss function: Gaussian ⇔ squared error, Laplacian ⇔ absolute error.
Viewing $\hat w_{\text{ML}}$ as an estimator lets us study its quality (L34).