Lecture 25 · Fitting a GMM, part 2

Estimating GMM Parameters with a Lower Bound

The GMM log-likelihood is hard to maximise because of the log of a sum. The trick: add some extra helper numbers λ, use Jensen to get a lower bound with the sum outside the log, and notice that this bound is easy to maximise one half at a time, with simple formulas for each half.

Source: this lecture on YouTube (≈23 min). Previous: L24 · Convexity and Jensen · L23 · GMM likelihood.

1 · The trick: multiply and divide by λ

For every data point $i$, introduce $K$ new numbers $\lambda_{i1},\dots,\lambda_{iK}$ that behave like probabilities: each is between 0 and 1, and $\sum_k\lambda_{ik}=1$. Then multiply and divide each term inside the log by $\lambda_{ik}$. That changes nothing:

$$\log L(\theta)=\sum_{i=1}^{n}\log\sum_{k=1}^{K}\lambda_{ik}\,\frac{\pi_k\,\mathcal N(x_i;\mu_k,\sigma_k^2)}{\lambda_{ik}} .$$

Now the inside of the log is a weighted average, with weights $\lambda_{ik}$, of the numbers $\pi_k\mathcal N(x_i;\mu_k,\sigma_k^2)/\lambda_{ik}$. Since log is concave, Jensen (L24) lets us pull the sum outside:

$$\log L(\theta)\;\ge\;\underbrace{\sum_{i=1}^{n}\sum_{k=1}^{K}\lambda_{ik}\,\log\frac{\pi_k\,\mathcal N(x_i;\mu_k,\sigma_k^2)}{\lambda_{ik}}}_{\text{modified log-likelihood }\tilde\ell(\theta,\lambda)} .$$

$\tilde\ell$ is a lower bound on the true log-likelihood, and it holds for every valid choice of λ.
What's λ? For now just helper numbers. They'll turn out to mean "how likely point $i$ is to have come from component $k$": a soft cluster assignment.

2 · Why add more unknowns?

This looks backwards: $\tilde\ell$ isn't the function we wanted, and it has $nK$ extra unknowns. It's worth it because of one key property:

Key insight. Fix λ, and maximising over θ is easy. Fix θ, and maximising over λ is easy. Both have closed-form answers.

So instead of one hard problem, we get two easy ones that can take turns. Putting them together into an algorithm is the next lecture.

3 · Fix λ, solve for θ: weighted averages

With λ held fixed, the log in $\tilde\ell$ now contains a product, $\pi_k \times$ Gaussian, so it splits apart and the exponential cancels. The $-\lambda_{ik}\log\lambda_{ik}$ part is now a constant. Setting derivatives to zero gives:

Parameter	Best value for fixed λ	In words
mean $\mu_k$	$\displaystyle\hat\mu_k=\frac{\sum_i\lambda_{ik}\,x_i}{\sum_i\lambda_{ik}}$	weighted mean of all points, weighted by how much each belongs to $k$
variance $\sigma_k^2$	$\displaystyle\hat\sigma_k^2=\frac{\sum_i\lambda_{ik}\,(x_i-\hat\mu_k)^2}{\sum_i\lambda_{ik}}$	weighted variance around that mean
mixing proportion $\pi_k$	$\displaystyle\hat\pi_k=\frac1n\sum_i\lambda_{ik}$	average membership in $k$ (π is constrained, so this one uses Lagrange multipliers)

Sanity check with hard assignments. If every λ is 0 or 1 (each point fully in one cluster), these become the plain mean, variance and fraction of the points in cluster $k$, just like K-means. Soft λ values simply let each point count partially towards several clusters.

4 · Fix θ, solve for λ: Bayes' theorem

With the means, variances and πs held fixed, $\tilde\ell$ splits into a separate small problem for each point. Solving each one, with the constraint $\sum_k\lambda_{ik}=1$ handled by a Lagrange multiplier, gives:

$$\hat\lambda_{ik}=\frac{\pi_k\,\mathcal N(x_i;\mu_k,\sigma_k^2)}{\sum_{l=1}^{K}\pi_l\,\mathcal N(x_i;\mu_l,\sigma_l^2)} .$$

This is exactly Bayes' theorem for the probability that point $i$ came from component $k$, given its value:

$$P(z_i=k\mid x_i)=\frac{P(x_i\mid z_i=k)\,P(z_i=k)}{P(x_i)} .$$

So the best λ is the posterior probability of each component for each point. It's the same "who could have generated this point?" share from the L23 page.

5 · Try the two steps by hand

Two-component data, starting from a bad guess. Each button applies one of the two formulas above. Point colours show λ: blue means component 1, orange means component 2, and mixed shades mean the point is shared between them. Watch the lower bound $\tilde\ell$ stay at or below the true $\log L$.

–

true log L(θ)

–

lower bound ℓ̃(θ, λ)

–

gap (≥ 0)

Going further: why the gap hits 0 after a λ step. With λ set by Bayes, every term inside Jensen's average, $\pi_k\mathcal N(x_i;\cdot)/\lambda_{ik}$, equals the same number, $f(x_i)$. Averaging identical numbers loses nothing, so Jensen holds with equality and the bound touches the true log-likelihood (as in the L24 check, where equal values gave a gap of 0). This is why alternating the two steps keeps pushing the true likelihood up, which the next lecture builds on.

Key takeaways

Introduce soft weights $\lambda_{ik}$ (one probability distribution over components per point), then multiply and divide by them.
Jensen then gives a lower bound $\tilde\ell(\theta,\lambda)\le\log L(\theta)$ with the sum outside the log, valid for any λ.
Fix λ: θ has closed forms: weighted mean, weighted variance, and average λ for π.
Fix θ: λ is the Bayes posterior, $P(z_i=k\mid x_i)$.
Two easy steps instead of one hard problem. Alternating them gives an algorithm for fitting a GMM, next lecture.

Self-check: if all λ for component k were 1/2, what would μ̂ₖ be?

The plain mean of all the data. Every point gets the same weight, so the weighted mean reduces to the ordinary average.

Self-check: why is the λ that maximises the bound the same as P(zᵢ = k | xᵢ)?

The Lagrange-multiplier solution comes out as πₖ𝒩(xᵢ) divided by the sum over all components, which is exactly Bayes' theorem with prior πₖ and likelihood 𝒩(xᵢ; μₖ, σₖ²).