15 Bayesian Statistics
Day 1
Settling In
If you are currently taking STAT 454, sit with at least one other person who is not taking that class. And vice versa: if you are not in STAT 454 right now, find someone who is!
Check Slack for announcements
- revised OH schedule this week
Checkpoint Debrief
At your table, discuss:
What are the primary differences between Frequentist and Bayesian Statistics? Name at least two.
Prior distributions:
- What’s a prior distribution? (mathematically and conceptually)
- In the Beta-Binomial example in the video, why is the Beta distribution a “natural choice” for a prior?
- More generally, what strategies do we have for choosing priors?
Posterior distributions:
- What’s a posterior distribution? (mathematically and conceptually)
- How do we find/derive a posterior distribution?
Practice Problems
Gamma-Exponential
Suppose we have a random sample \(Y_1, \dots, Y_n\) from an Exponential pdf with parameter \(\lambda > 0\), ie \[f_Y(y \mid \lambda) = \lambda e^{-\lambda y}, \quad y \ge 0,\] where our prior distribution for \(\lambda\) is a Gamma distribution with hyper-parameters \(\alpha > 0, \beta > 0\), ie \[f_\Lambda(\lambda) = \frac{\beta^\alpha}{\Gamma(\alpha)} \lambda^{\alpha - 1} e^{-\lambda \beta}, \quad \lambda \ge 0.\] Find the posterior distribution for \(\lambda\).
Normal-Normal
Suppose we have a random sample \(Y_1, \dots, Y_n\) from a Normal pdf with unknown mean \(\mu\) and known variance \(\sigma^2 > 0\), ie \[f_Y(y \mid \mu) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{1}{2\sigma^2}(y - \mu)^2}, \quad -\infty < y < \infty,\] where our prior distribution for \(\mu\) is Normal with hyper-parameters \(\mu_0, \sigma_0^2\), ie \[f(\mu) = \frac{1}{\sqrt{2\pi\sigma_0^2}} e^{-\frac{1}{2\sigma_0^2}(\mu - \mu_0)^2}, \quad -\infty < \mu < \infty.\] Find the posterior distribution for \(\mu\).
HINT: Complete the square!
If you see \(a \mu^2 - 2b \mu = a\left(\mu^2 - 2 \frac{b}{a}\mu \right)\), add \(a \left(\frac{b}{a}\right)^2 - a \left(\frac{b}{a}\right)^2\), so you get \[ \begin{align} a \mu^2 - 2b \mu & = a \mu^2 - 2b \mu + a \left(\frac{b}{a}\right)^2 - a \left(\frac{b}{a}\right)^2 \\ & = a \left[\mu^2 - 2 \left(\frac{b}{a}\right)\mu \right] + a \left(\frac{b}{a}\right)^2 - a \left(\frac{b}{a}\right)^2 \\ & = a \left[ \mu^2 - 2 \left(\frac{b}{a}\right)\mu + \left(\frac{b}{a}\right)^2\right] - a \left(\frac{b}{a}\right)^2 \\ & = a \left[ \mu - \left(\frac{b}{a}\right)\right]^2 - a \left(\frac{b}{a}\right)^2 \\ & \propto a \left[ \mu - \left(\frac{b}{a}\right)\right]^2 \end{align} \]
BONUS
Go back and solve Problems 1 and 2, this time keeping track of the denominator. Confirm that you get the same answers.
Solutions
See today’s board notes.
Wrapping Up
Main Ideas from Today
The posterior distribution is central to Bayesian statistics. We use it to derive estimators (up next), quantify uncertainty, perform inference, etc.
If you’re working with a conjugate prior, the math works out nicely and your posterior will end up in the same family.
In this class, when deriving a posterior distribution, you should:
- ignore the normalizing constant \(f(\mathbf{x})\)
- drop any constants that don’t depend on the unknown parameter
- re-arrange terms until you see the kernel of a recognizable distribution
When a closed form solution for the posterior is not available, we use Markov chain Monte Carlo (MCMC) techniques.
- take STAT 454 to learn more!
After Class
Day 2
Settling In
If you are currently taking STAT 454, sit with at least one other person who is not taking that class. And vice versa: if you are not in STAT 454 right now, find someone who is!
Check Slack for announcements
- reminder: revised OH schedule this week
Bayesian Estimation
Three common ways of coming up with a Bayesian estimator of an unknown parameter \(\theta\)…
Posterior Mean
If you want to minimize squared error loss, \(L(\hat\theta, \theta) = (\hat\theta - \theta)^2\), use the posterior mean: \[\hat\theta_{Bayes} = E[\theta \mid \mathbf{X}]\]
Posterior Median
If you want to minimize absolute error loss, \(L(\hat\theta, \theta) = |\hat\theta - \theta|\), use the posterior median: \[\hat\theta_{Bayes} = m, \text{ where } P(\theta \le m \mid \mathbf{X}) = P(\theta \ge m \mid \mathbf{X}) = 0.5\]
Posterior Mode
If you want to minimize zero-one loss, \(L(\hat\theta, \theta) = 0*\mathbb{1}(\hat\theta - \theta \le K) + L*\mathbb{1}(\hat\theta - \theta > K)\), use the posterior mode: \[\hat\theta_{Bayes} = \arg\max_\theta g(\theta \mid \mathbf{X})\]
Check out Chapter 9 in the MATH/STAT 355 Notes to learn more about loss functions and the area of statistics known as Decision Theory.
Practice Problems
Gamma-Exponential
Derive the Bayes estimator under squared error loss for the Gamma-Exponential scenario we explored last class.
HINT: If \(X \sim \text{Gamma}(a, b)\), then \(E(X) = \frac{a}{b}\).
Show that this estimator can be rewritten as a weighted average of the MLE (\(\hat\lambda_{MLE} = (\frac{1}{n} \sum_{i=1}^n Y_i)^{-1} = 1 / \overline{Y}\)) and the prior mean (\(E(\lambda) = \frac{\alpha}{\beta}\)).
What will happen to this estimator as \(n \rightarrow \infty\)?
Normal-Normal
Derive the Bayes estimator under squared error loss for the Normal-Normal scenario we explored last class. As above, show that this estimator can be rewritten as a weighted average of the MLE and the prior mean.
Show that this is a biased estimator of \(\mu\). What happens to that bias as \(n \rightarrow \infty\)?
The posterior mean, median, and mode are identical in this scenario. Explain why.
Explain how you would go about deriving the posterior median and mode if you didn’t know they were identical to the posterior mean. (You don’t have to actually do it – just explain the steps!)
Solutions
See today’s board notes.
Wrapping Up
Main Ideas from Today
Bayes estimators can frequently be re-written as a weighted combination of the MLE and prior mean, with increasing weight on the data/MLE as \(n \to \infty\).
Although Bayes estimators are usually biased for finite \(n\), they share many of the same asymptotic properties as MLEs!
FYI: Posterior distributions are useful for more than just estimation! Since we have an entire distribution, we can quantify uncertainty and make probability statements about \(\theta\) (including confidence interval and p-value-like statements) without having to rely on the idea of repeated sampling.
- EXAMPLE: \(\Pr(\theta > 0 \mid \mathbf{X})\)
After Class