Lecture 23 · Fitting a GMM

The Likelihood of a Gaussian Mixture Model

Writing down the GMM likelihood is easy. Maximising it is not: the log can't break apart the sum inside it, so there's no neat formula for the answer. This lecture shows exactly where the difficulty comes from, and sets up a detour into convex functions to get around it.

Source: this lecture on YouTube (≈15 min). Previous: L22 · Gaussian mixture models.

1 · The likelihood

The unknown parameters are all the means, variances and mixing proportions. Call them θ together: $\theta=(\mu_1..\mu_K,\ \sigma^2_1..\sigma^2_K,\ \pi_1..\pi_K)$. Each point is still generated by the same two-step process, independently of the others, so i.i.d. still holds and the likelihood is a product over points:

$$L(\theta)=\prod_{i=1}^{n} f(x_i;\theta)=\prod_{i=1}^{n}\ \sum_{k=1}^{K}\pi_k\,\frac{1}{\sqrt{2\pi}\,\sigma_k}\,e^{-\frac{(x_i-\mu_k)^2}{2\sigma_k^2}} .$$

Why a sum for each point? We don't know which component produced $x_i$ (the die roll is latent). It came from component 1, or 2, or …, and these possibilities can't happen together, so their chances add. Each term is "the die picked $k$" (probability $\pi_k$) and "component $k$ produced $x_i$" (its Gaussian density), and since both must happen, those two multiply.

2 · Who could have generated this point?

The lecture's example has three components, with means −20, 0 and 15. A point at $x = -15$ is closest to component 1, but that doesn't prove it came from there. Every Gaussian gives every point some density, and the mixing proportions matter too. Each component's share of $f(x)$ is its term $\pi_k\,\mathcal N(x;\mu_k,\sigma_k^2)$ divided by the total:

observed x = -15

Try $x = -10$, exactly halfway between components 1 and 2. Both Gaussians give it the same density, so π alone decides which is more likely. Then try π = (0.05, 0.05, 0.9) at $x = -15$: component 3 is picked far more often, but it's so far away that component 1 still wins. Both closeness and π count.

These shares aren't named in this lecture, but they'll reappear soon: they're the "soft" cluster memberships that the GMM fitting algorithm computes.

3 · The bottleneck: a sum inside the log

As usual, take the log to turn the product into a sum:

$$\log L(\theta)=\sum_{i=1}^{n}\log\Bigg(\sum_{k=1}^{K}\pi_k\,\frac{1}{\sqrt{2\pi}\,\sigma_k}\,e^{-\frac{(x_i-\mu_k)^2}{2\sigma_k^2}}\Bigg)\qquad(\star)$$

For a single Gaussian the log did two helpful things: it turned the product into a sum, and it cancelled the $e^{(\cdots)}$, leaving a simple quadratic. Here it only does the first. The inner sum sits inside the log, and $\log(a+b)$ doesn't split into anything simpler. So the exponentials never cancel.

Single Gaussian	Mixture
$\log e^{-(x-\mu)^2/2\sigma^2}=-\dfrac{(x-\mu)^2}{2\sigma^2}$, a simple quadratic in μ	$\log\big(\pi_1e^{(\cdot)}+\pi_2e^{(\cdot)}+\dots\big)$, no simplification
derivative = 0 gives $\hat\mu=$ sample mean	derivative = 0 has no closed-form solution

4 · Why not just differentiate? Two problems

No closed form. Setting the derivatives with respect to the μs (or σs) to zero gives equations that can't be solved analytically. There's no "μ̂ = average"-style formula.
A constraint on π. The πs aren't free: they must be non-negative and sum to 1. So this is a constrained optimisation problem, which is harder still.

A general-purpose method like gradient ascent on $(\star)$ would still work: step uphill until you reach a peak. But it treats $(\star)$ as just some function and ignores the model's structure, the two-step "pick a component, then draw a point" story behind it.

What's next: we want a method that uses that structure. The key ingredient is convex functions (the next part), which will give a way to deal with the log of a sum. This is where the "convexity" in the lecture title comes in.

Key takeaways

GMM likelihood: $L(\theta)=\prod_i\sum_k\pi_k\,\mathcal N(x_i;\mu_k,\sigma_k^2)$. The product comes from i.i.d.; the sum comes from the unknown (latent) component.
A point's likely component depends on both closeness to each mean and the mixing proportions π.
The log-likelihood has a sum inside a log, so the exponentials don't cancel and there's no closed-form maximiser.
π is constrained (π ≥ 0, Σπ = 1). Gradient ascent works but ignores the model's structure. Convexity is the tool for a better method.

Self-check: why does the mixture density have a sum of terms, while a single Gaussian has just one?

We don't observe which component made the point, so we add up the probability of every possible component (they're mutually exclusive), each weighted by how likely the die is to pick it.

Self-check: what goes wrong when you take the log of the GMM likelihood?

The log turns the product over points into a sum, but each point's term is itself a sum, and log(sum) doesn't simplify. So the exponentials stay inside the log and setting derivatives to zero has no closed-form solution.