← All notes · ML revision (DA5400)
Bayesian Estimation
Maximum likelihood looks only at the data. But before seeing any data, we often already have a hunch about the parameter. Bayesian estimation turns that hunch into a probability distribution (the prior), then uses Bayes' theorem to update it with the data into a new distribution (the posterior).
Source: this lecture on YouTube (≈37 min). Previous: L20 · Maximum likelihood. Sections marked Going further go beyond the video.
1 · The missing ingredient: what you know before the data
Suppose you haven't tossed the coin even once, and someone asks for your best guess of $p$. Maximum likelihood can't answer: it needs data, and with no data it has nothing to go on.
But in practice you often do know something beforehand. Someone tells you "I believe this coin's bias is close to 1." Or your experience says "it's definitely a trick coin, so $p$ is near 0 or near 1, not near 0.5." These are statements about the parameter, not about any data. The lecture calls them hunches (or domain knowledge, or beliefs).
2 · The key move: treat the parameter as a random variable
In maximum likelihood, $p$ was a fixed, unknown constant. The Bayesian approach instead describes our uncertainty about $p$ with a probability distribution, $P(\theta)$, over all the values it could take. This is called the prior, because it comes before seeing the data.
- Where the prior's density is high, you believe the true value is more likely to be.
- The prior must live where the parameter can live. For a coin, $p$ is between 0 and 1, so the prior must only cover $[0, 1]$.
- Integrating the prior over an interval gives "how much I believe, before seeing data, that the true value falls in this interval".
Different hunches become different shapes:
"p is close to 1"
Most belief near 0.9, little near 0.
"p is not near 0.5"
Belief piled at both ends: a trick coin, one way or the other.
"the mean μ is around 12.5"
For a Gaussian's mean, which can be any real number, a bell shape works.
After seeing the data, the hunch should be updated. If you expected about 90% heads but see 900 tails in 1,000 tosses, your belief must shift a lot. If the data agrees with you, your belief should become more confident. The updated hunch is again a distribution, now conditioned on the data: $P(\theta\mid\text{data})$, the posterior.
3 · Bayes' theorem does the updating
Bayes' rule from probability says $P(A\mid B)=\dfrac{P(B\mid A)\,P(A)}{P(B)}$. Set $A$ = the parameter $\theta$ and $B$ = the data $x_1,\dots,x_n$:
$$\underbrace{P(\theta\mid x_1,\dots,x_n)}_{\text{posterior}}=\frac{\overbrace{P(x_1,\dots,x_n\mid\theta)}^{\text{likelihood}}\;\;\overbrace{P(\theta)}^{\text{prior}}}{\underbrace{P(x_1,\dots,x_n)}_{\text{evidence}}} .$$- The likelihood is the same object from L20: how probable the data is if the parameter were θ.
- The evidence is the overall probability of the data. It doesn't involve θ, so for the shape of the posterior we can ignore it:
Read it as a re-weighting: start with your prior beliefs and multiply each candidate θ by how well it explains the data. Candidates that explain the data well gain weight; the others lose it.
4 · A flexible prior for a coin: the Beta distribution
For a coin we need a distribution on $[0,1]$. A Gaussian won't do, because it spreads over all real numbers. The standard choice is the Beta distribution, with two positive shape parameters $\alpha$ and $\beta$:
$$P(p)=\frac{1}{Z}\,p^{\alpha-1}(1-p)^{\beta-1},\qquad p\in[0,1],\quad \alpha,\beta>0 .$$$Z$ is just the constant that makes the area 1. What matters is the shape, $p^{\alpha-1}(1-p)^{\beta-1}$. Play with $\alpha$ and $\beta$ and see how many different hunches it can express:
- α = 2, β = 2: a hump at 0.5. "Probably roughly fair, but I'm not sure."
- α = 2, β = 5: leans towards 0. A small α with a large β puts belief closer to 0.
- α = β = 0.5: a U shape. "A trick coin, either very low or very high, but not near 0.5."
- α = β = 1: flat, "no idea". Every $p$ is equally plausible. (Going further: this case connects Bayes back to maximum likelihood, section 8.)
- Making α and β both larger while keeping their ratio fixed makes the hump narrower: a more confident hunch. Compare α = β = 2 with α = β = 20.
The average of a Beta distribution is $\dfrac{\alpha}{\alpha+\beta}$.
5 · Beta prior + coin data = Beta posterior
Multiply the Bernoulli likelihood from L20 by the Beta prior. Write $n_H=\sum_i x_i$ for the number of heads and $n_T=\sum_i(1-x_i)$ for the number of tails.
- Likelihood: $\displaystyle\prod_{i=1}^{n}p^{x_i}(1-p)^{1-x_i}=p^{\,n_H}(1-p)^{\,n_T}$.
- Prior: $\;p^{\alpha-1}(1-p)^{\beta-1}$ (ignoring the constant).
- Multiply, adding the exponents of matching bases: $$P(p\mid\text{data})\;\propto\;p^{\,n_H}(1-p)^{\,n_T}\cdot p^{\alpha-1}(1-p)^{\beta-1}=p^{\,(\alpha+n_H)-1}(1-p)^{\,(\beta+n_T)-1}.$$
- That is exactly the shape of another Beta distribution: $$\boxed{\;\text{Beta}(\alpha,\beta)\ \text{prior}\;\xrightarrow{\;n_H\text{ heads},\ n_T\text{ tails}\;}\;\text{Beta}(\alpha+n_H,\ \beta+n_T)\ \text{posterior}\;}$$
The posterior has the same form as the prior; only the parameters change. So updating your belief is just adding counts: heads to α, tails to β. You never need to compute the posterior density point by point or work out the evidence. Once you know the new parameters, you know the whole distribution.
This doesn't happen for every prior. Pick some other distribution on $[0,1]$ and you still get a valid posterior, but usually not one with a neat known form. A prior that does keep its form like this is called a conjugate prior; Beta is conjugate to the coin (Bernoulli) likelihood.
6 · Try it: prior + data → posterior
Set a prior, feed in coin tosses, and watch the posterior form. The two lecture scenarios are one click away: a hunch that $p$ is near 0.9, followed by data that either disagrees (2 heads, 8 tails) or agrees (9 heads, 1 tail).
Prior
Data
What to notice:
- Case A (disagreement): the posterior lands between the prior and the likelihood. The lecture's sketch puts it around 0.27, as a rough illustration; the exact Beta(12, 10) posterior here peaks at 0.55, because a Beta(10, 2) prior is worth 12 ghost tosses and 10 real ones don't outweigh it. Try Clear, then case A twice or three times: as real data accumulates, it wins.
- Case B (agreement): the posterior stays near 0.9 and becomes sharper than the prior. Agreeing data increases confidence.
- A flat prior (1,1): the posterior has exactly the same shape as the likelihood.
7 · The prior as "ghost data"
Compare the two estimates:
$$\hat p_{\text{ML}}=\frac{n_H}{n}\qquad\text{vs.}\qquad \mathbb E[p\mid\text{data}]=\frac{\alpha+n_H}{\alpha+\beta+n}.$$The second is exactly what maximum likelihood would give if, on top of your real data, you had seen α extra heads and β extra tails. So a Beta(α, β) prior acts like α + β imaginary tosses ("ghost" or pseudo-observations) done before the real experiment.
8 · From a distribution to a single number
Maximum likelihood gives one number, a point estimate. The Bayesian approach gives a whole distribution over the parameter, which carries more information: where the value probably is and how unsure we are. If you must commit to one number, you have choices:
| Point estimate | Definition | For a Beta(a, b) posterior |
|---|---|---|
| Posterior mean | the average of the posterior | $\dfrac{a}{a+b}=\dfrac{\alpha+n_H}{\alpha+\beta+n}$ |
| MAP (maximum a posteriori) | the peak of the posterior: like ML, but maximising the posterior instead of the likelihood | $\dfrac{a-1}{a+b-2}=\dfrac{\alpha+n_H-1}{\alpha+\beta+n-2}$ (when $a,b>1$) |
| Others | e.g. draw samples from the posterior and average them | – |
The lecture states that MAP differs slightly from the mean but doesn't give the formula. The derivation is the same as in L20: take the log of the posterior, $(a-1)\log p+(b-1)\log(1-p)$, set its derivative $\frac{a-1}{p}-\frac{b-1}{1-p}$ to zero, and solve to get $p=\frac{a-1}{a+b-2}$.
9 · Two ways to estimate, side by side
| Maximum likelihood (L20) | Bayesian (this lecture) | |
|---|---|---|
| The parameter is… | a fixed, unknown constant | a random variable describing our uncertainty |
| Uses | data only | data and a prior (hunch) |
| With no data | can't give any answer | answers with the prior |
| Output | one number, $\hat\theta_{\text{ML}}$ | a whole distribution, $P(\theta\mid\text{data})$; reduce it to a number if needed (mean, MAP, …) |
| Main tool | maximise the likelihood | Bayes' theorem: posterior ∝ likelihood × prior |
10 · Going further
10.1 The posterior mean is a weighted average of the prior and the data
Rearranging the posterior mean shows exactly how the two are balanced:
$$\frac{\alpha+n_H}{\alpha+\beta+n}=\underbrace{\frac{\alpha+\beta}{\alpha+\beta+n}}_{w}\cdot\underbrace{\frac{\alpha}{\alpha+\beta}}_{\text{prior mean}}\;+\;\underbrace{\frac{n}{\alpha+\beta+n}}_{1-w}\cdot\underbrace{\frac{n_H}{n}}_{\hat p_{\text{ML}}}.$$The prior's weight $w$ shrinks as $n$ grows. With little data the prior dominates; with lots of data the estimate approaches maximum likelihood, and the hunch stops mattering. Watch it happen:
A coin with true p = 0.3 is tossed n times. The prior is Beta(10, 2), a confident but wrong hunch near 0.9.
10.2 Where the prior earns its keep: very little data
Toss a coin 3 times and get 3 heads. Maximum likelihood says $\hat p = 1$: it claims tails are impossible, which is a very strong conclusion from 3 tosses. With even a mild Beta(2, 2) prior, the posterior is Beta(5, 2), with mean $5/7 \approx 0.71$ and MAP $4/5 = 0.8$. The prior stops the estimate from jumping to an extreme on tiny data.
10.3 Uncertainty comes for free
The posterior's spread tells you how sure you are. For a Beta(a, b) posterior the variance is $\dfrac{ab}{(a+b)^2(a+b+1)}$. With a flat prior and 7 heads in 10 tosses, the posterior Beta(8, 4) has a standard deviation of about 0.13. After 70 heads in 100 tosses, Beta(71, 31) has about 0.045. Same proportion of heads, but much more certainty.
Key takeaways
- Maximum likelihood ignores any knowledge you have before the data. Bayesian estimation includes it.
- Encode the hunch as a prior distribution $P(\theta)$ by treating the parameter as a random variable.
- Bayes' theorem: posterior = likelihood × prior ÷ evidence, so posterior ∝ likelihood × prior.
- For a coin, a Beta(α, β) prior gives a Beta(α + n_H, β + n_T) posterior. Updating means adding counts (a conjugate prior).
- The prior acts like α heads and β tails of ghost data.
- The result is a whole distribution. Reduce it to one number with the posterior mean $\frac{\alpha+n_H}{\alpha+\beta+n}$ or the MAP estimate (the posterior's peak).
Self-check
1. Why can't maximum likelihood give an estimate before any data is seen, while the Bayesian approach can?
Maximum likelihood depends only on the data, so with no data there's nothing to maximise. The Bayesian approach starts from the prior, which exists before any data; with no data the posterior simply equals the prior.
2. Why is a Gaussian a poor prior for a coin's p?
A Gaussian puts probability on every real number, including values below 0 and above 1, which are impossible for p. The Beta distribution lives only on [0, 1].
3. Prior Beta(3, 3). Data: 6 heads, 2 tails. What is the posterior, its mean, and its MAP?
Beta(9, 5). Mean = 9/14 ≈ 0.643. MAP = 8/12 ≈ 0.667. (ML would be 6/8 = 0.75.)
4. Why can we ignore the evidence P(data) when finding the posterior's shape?
It doesn't depend on θ. It's the same number for every θ, so it only rescales the curve to have area 1; the shape and the peak stay the same.
5. Two people have priors Beta(2, 2) and Beta(200, 200). Both see 10 heads and 0 tails. Whose belief moves more, and why?
The Beta(2, 2) person: posterior Beta(12, 2), mean ≈ 0.86. The Beta(200, 200) person: posterior Beta(210, 200), mean ≈ 0.51. A prior with α + β = 400 is like 400 ghost tosses, so 10 real tosses barely move it.
6. What prior makes the MAP estimate equal to the ML estimate for a coin?
The flat prior Beta(1, 1). Then the posterior is proportional to the likelihood, so they peak at the same place.
For the exact wording of the lecture, open the video on YouTube and use "Show transcript".