Lecture 22 · Estimation for clustered data

Gaussian Mixture Models

When data falls into several clumps, one Gaussian can't explain it. A Gaussian mixture model (GMM) explains it with a two-step story: first pick a clump at random, then draw a point from that clump's own Gaussian.

Source: this lecture on YouTube (≈21 min). Previous: L21 · Bayesian estimation.

1 · Why one Gaussian isn't enough

Take 1-D data in three clumps. Fit one Gaussian by maximum likelihood: its mean is the sample mean, which lands in the middle. The fitted curve then says the outer clumps sit in low-density regions, yet that's exactly where lots of data was observed. Maximum likelihood isn't at fault: it found the best Gaussian. The model is too simple. A single Gaussian has only one peak, and this data needs three.

2 · The two-step generative story

To generate one data point $x_i$:

Step 1: pick a mixture component. Roll a $K$-faced die whose faces have probabilities $\pi_1,\dots,\pi_K$. The face it lands on is $z_i$, the component this point comes from:

$$P(z_i=k)=\pi_k,\qquad \pi_k\ge 0,\quad \sum_{k=1}^{K}\pi_k=1 .$$

Step 2: draw from that component's Gaussian. Each component $k$ has its own mean and variance:

$$x_i\mid z_i=k\;\sim\;\mathcal N(\mu_k,\ \sigma_k^2).$$

Repeat both steps, independently, for every data point. The $\pi_k$ are called mixing proportions: with $\pi=(0.5,0.25,0.25)$, about half the points come from component 1 and a quarter from each of the others. Try it:

π₁ π₂ π₃

?Step 1 rolls the die; step 2 samples from that component.

component 1: μ = −4, σ = 0.8component 2: μ = 0, σ = 0.5component 3: μ = 4, σ = 1.0

Colours show which component generated each point. In real data you don't get the colours, only the positions. That's the next idea.

3 · Latent variables

We observe only $x_1,\dots,x_n$. The die rolls $z_1,\dots,z_n$ that decided where each point came from are never seen. Unobserved quantities like these are called latent variables, and a GMM is a latent variable model: the data depends both on parameters and on hidden variables.

$z_i$ plays the same role as the cluster indicator in K-means (L13). The difference: in K-means each point was assigned to a cluster, while here a random die roll chose its component.

4 · What has to be estimated

Parameter	Meaning	Count
$\pi_1,\dots,\pi_K$	mixing proportions; they sum to 1, so the last is determined by the others	$K-1$
$\mu_1,\dots,\mu_K$	component means	$K$
$\sigma_1^2,\dots,\sigma_K^2$	component variances	$K$
Total		$3K-1$ (about $3K$)

Earlier models had one parameter (a coin's bias, or a Gaussian's mean). Now there are many, and just looking at the data doesn't tell you what they should be. But maximum likelihood works for any well-specified probabilistic model, so it's the tool to use next.

Where this leads (a short preview, not in this video). Since $z_i$ is hidden, the density of a single observation adds up all the ways it could have arisen, one for each die face: $$f(x;\pi,\mu,\sigma^2)=\sum_{k=1}^{K}\pi_k\,\mathcal N(x;\mu_k,\sigma_k^2).$$ This weighted sum of bells is the three-peaked curve in section 1. The next lecture writes the likelihood of a GMM from it.

Key takeaways

Clumped data has several peaks; one Gaussian has one, so it's the wrong model, however well it's fitted.
GMM story: roll a die with probabilities π to get $z_i$, then draw $x_i \sim \mathcal N(\mu_{z_i}, \sigma^2_{z_i})$.
The $z_i$ are latent (hidden), so a GMM is a latent variable model. It's the probabilistic cousin of K-means.
Parameters: π, μ and σ², which is $3K-1$ numbers, to be found by maximum likelihood.

Self-check: with π = (0.5, 0.25, 0.25) and n = 400, roughly how many points come from component 2?

About 100, since each point independently has a 25% chance of coming from component 2.

Self-check: how many parameters does a GMM with K = 5 components have in 1-D?

3 × 5 − 1 = 14: four free mixing proportions, five means and five variances.