← All notes · ML revision (DA5400)

Maximum Likelihood Estimation

L19 ended with a question: when many parameter values could have produced the data, is there a principled way to pick one? This lecture's answer is Fisher's principle of maximum likelihood: choose the parameter value under which the observed data was most likely.

Source: this lecture on YouTube (≈27 min). Previous: L19 · Introduction to estimation. Sections marked Going further go beyond the video.

1 · The likelihood curve

Take the coin model from L19 and a small dataset, 1, 0, 1, 1. For every candidate $p$ between 0 and 1, ask: if the true bias were $p$, what's the chance of seeing exactly this data? Plot that chance against $p$ and you get the likelihood curve.

• At $p = 0$ and $p = 1$ the curve is 0, because each extreme makes one of the observed outcomes impossible.
• Somewhere in between it peaks. That peak is the value of $p$ that makes the data most likely.

Click any outcome below to flip it, or add and remove tosses, and watch the peak move.

2 · Fisher's principle of maximum likelihood

Proposed by R. A. Fisher, the idea is simple: among all the parameter values that could have produced the data, bet on the one that makes the data most probable.

The likelihood function is the probability of the observed data, viewed as a function of the parameter:

This is a joint probability: all $n$ outcomes happening together. The two L19 assumptions now do the heavy lifting:

The compact factor $p^{x_i}(1-p)^{1-x_i}$ is just $p$ when $x_i=1$ and $1-p$ when $x_i=0$. For 1, 0, 1, 1 that gives $p\cdot(1-p)\cdot p\cdot p=p^{3}(1-p)$, the curve above.

The maximum likelihood (ML) estimator is the maximiser. The hat says "this is a guess":

3 · Why take the logarithm?

Products are awkward to differentiate; sums are easy. The logarithm turns one into the other: $\log(ab)=\log a+\log b$.

And it's safe, because log is strictly increasing. If $L(\hat p)>L(p)$ for every other $p$, then $\log L(\hat p)>\log L(p)$ too. The curve's height changes, but the location of the peak doesn't:

Applied to the coin:

4 · Solving for the coin: the fraction of heads

The lecturer leaves this as an exercise. Here it is step by step. Write $h=\sum_i x_i$ for the number of heads.

1. Collect the terms: $\displaystyle\log L(p)=h\log p+(n-h)\log(1-p).$
2. Differentiate: $\displaystyle\frac{d}{dp}\log L(p)=\frac{h}{p}-\frac{n-h}{1-p}.$
3. Set it to zero and solve: $h(1-p)=(n-h)p\;\Rightarrow\;h=np\;\Rightarrow\;\boxed{\hat p_{\text{ML}}=\frac{h}{n}=\frac1n\sum_{i=1}^{n}x_i}.$
4. Check it's a maximum: $\displaystyle\frac{d^2}{dp^2}\log L=-\frac{h}{p^2}-\frac{n-h}{(1-p)^2}<0$. The log-likelihood is concave, so this single stationary point is the global maximum.

The answer is the average of the data. Since each $x_i$ is 0 or 1, the sum just counts the 1s, so the average is the fraction of heads. That's exactly the intuitive guess from L19, now backed by a principle. Remember it still rests on the i.i.d. assumption.

5 · Continuous data breaks the recipe

Now suppose the data is the heights of people: real numbers like 162.53 cm, not 0s and 1s. A coin box can't produce these, so we need a different model. The simplest choice is a Gaussian, $x_i\sim\mathcal N(\mu,\sigma^2)$. To keep things simple, assume $\sigma^2$ is known and only the mean $\mu$ is unknown.

Try the same recipe: "the probability of seeing these exact heights". For a continuous distribution, probability is the area under the density curve over an interval. A single exact value is an interval of width zero, so its probability is always zero, whatever $\mu$ is.

That kills the recipe. The product of the probabilities is 0 for every $\mu$, so it can't tell a good $\mu$ from a terrible one. There's nothing to maximise.

6 · Fisher's fix: replace probabilities with densities

Fisher's proposal: instead of the probability of each observation, use the value of the density (PDF) at that observation. The probability of a single point is 0, but the height of the curve there isn't. It's a sensible stand-in, since a taller curve means data near that point is more probable.

• Densities factorise under independence, just like probabilities do.
• A density value is not a probability. It can even be bigger than 1: a Gaussian with σ = 0.1 has a peak height of about 3.99.
• That's why it's called a likelihood and not a probability. It's the likelihood of the data under a parameter, not the chance of it.

7 · The Gaussian mean: maximum likelihood gives the sample mean

Take the log of the Gaussian likelihood:

The first term doesn't involve $\mu$, and the positive constant $2\sigma^2$ doesn't move the peak, so

Differentiate: $\;2\sum_i(x_i-\mu)=0\;\Rightarrow\;\boxed{\hat\mu_{\text{ML}}=\frac1n\sum_{i=1}^{n}x_i}$, the sample mean. Note the middle form: it's the same "minimise the sum of squared distances" problem as Fact 1 in L15.

8 · The estimator depends on the model you assume

The sample mean is the ML answer because we assumed a Gaussian. Change the assumption and the answer changes. Suppose domain knowledge suggests a Laplace distribution instead. It looks like a bell, but with a sharp peak and heavier tails:

Its log-likelihood is $-n\log 2b-\tfrac1b\sum_i|x_i-\mu|$, so maximising it means minimising the sum of absolute distances, $\sum_i|x_i-\mu|$. The minimiser is the sample median.

The lecture's point: the estimator is tied to the model. Maximum likelihood adapts to whatever model you assume, because it works directly with that model's density. The sample mean isn't automatically "the" right answer.

9 · The general recipe, and what's next

1. Assume a model: a family of distributions with unknown parameter θ.
2. Write the likelihood: the product of the PMF (discrete data) or PDF (continuous data) over the observations, using i.i.d.
3. Take the log to turn the product into a sum.
4. Maximise over θ: set the derivative to zero, or use a numerical optimiser when there's no closed form.

Its power is generality. Once you've written down a model, you automatically have a way to estimate its parameters, even when you have no intuition about what a good guess would be. When you do have intuition (the fraction of heads, the average height), it agrees with it. That's why maximum likelihood is so widely used in statistics and machine learning.

A statistics course would go on to prove why it's a good estimator, and that in certain well-defined senses nothing can beat it. This course takes that as given.

10 · Going further: maximum likelihood in ℝᵈ

10.1 Maximising likelihood = minimising a loss

Writing $\hat\theta=\arg\min_\theta\big[-\log L(\theta)\big]$ shows that every model defines a loss function, its negative log-likelihood:

This is how K-means connects too: its squared-distance cost is the Gaussian row, and K-medians is the Laplace row (L15). It's also exactly how loss functions in regression will arise later: squared loss ⇔ Gaussian noise, absolute loss ⇔ Laplace noise.

10.2 Both parameters unknown: the variance estimate is biased

In 1-D with both $\mu$ and $\sigma^2$ unknown, set the derivative with respect to $\sigma^2$ to zero as well:

On average this comes out a little too small: $\mathbb E[\hat\sigma^2_{\text{ML}}]=\frac{n-1}{n}\sigma^2$. That's because $\hat\mu$ was fitted to the same data, which pulls the points closer to it than they are to the true μ. Dividing by $n-1$ instead gives the unbiased estimate. In $\mathbb R^d$ the same holds for $\hat\Sigma=\frac1n\sum_i(x_i-\hat\mu)(x_i-\hat\mu)^\top$, derived on the L19 page.

10.3 A die with m faces (categorical), using a Lagrange multiplier

Let face $j$ have probability $\pi_j$ and appear $n_j$ times. The log-likelihood is $\sum_j n_j\log\pi_j$, subject to $\sum_j\pi_j=1$. Add the constraint with a multiplier $\lambda$:

It's the fraction of times each face came up: the coin result generalised. This gave the $\hat\pi_k=n_k/n$ mixing weights used in BIC in L18.

10.4 How accurate is the ML estimate? Fisher information

For large $n$, the ML estimate is approximately Gaussian around the truth, with variance set by how sharply the log-likelihood curves at its peak:

For the coin: $\frac{\partial^2}{\partial p^2}\big[x\log p+(1-x)\log(1-p)\big]=-\frac{x}{p^2}-\frac{1-x}{(1-p)^2}$. Taking the expectation gives $I(p)=\frac1p+\frac1{1-p}=\frac{1}{p(1-p)}$. So the variance is $\frac{p(1-p)}{n}$, which matches the spread you saw in the L19 sampling-distribution chart. A sharply curved log-likelihood (the narrow curve in section 1 as you add tosses) means high information and a precise estimate. In $\mathbb R^k$, $I(\theta)$ becomes a $k\times k$ matrix and the covariance of $\hat\theta$ is approximately $\big(nI(\theta)\big)^{-1}$. No unbiased estimator can beat this; that's the Cramér–Rao bound, one of the "it's the best you can get" results the lecture alludes to.

10.5 Invariance: estimate once, transform freely

If $\hat\theta$ is the ML estimate of $\theta$, then $g(\hat\theta)$ is the ML estimate of $g(\theta)$. For example, the ML estimate of a coin's odds $p/(1-p)$ is $\hat p/(1-\hat p)$, and the ML estimate of σ is $\sqrt{\hat\sigma^2_{\text{ML}}}$.

Key takeaways

1. The likelihood is the probability (or density) of the observed data, viewed as a function of the parameter.
2. Maximum likelihood: estimate the parameter as the value that makes the data most likely.
3. i.i.d. turns the likelihood into a product; the log turns it into a sum without moving the peak.
4. Coin: $\hat p_{\text{ML}}$ = fraction of heads. Gaussian mean: $\hat\mu_{\text{ML}}$ = sample mean.
5. For continuous data, every exact value has probability 0, so the likelihood uses the density instead.
6. The estimator depends on the model: a Laplace model gives the median, not the mean.
7. Next: using knowledge we have before seeing the data, which leads to Bayesian estimation.

Self-check

For the exact wording of the lecture, open the video on YouTube and use "Show transcript".