2 * mean(c(1.52, 7.81, 1.28, 6.90, 1.17))[1] 7.4726 Method Of Moments
Note
We’ll spend two days on this topic!
Day 1
Settling In
- Sit somewhere new:
- Find people born in the same month as you
- Introduce yourselves / check in with each other
- Catch up on any recent Slack messages you might have missed
- Review your notes from CP4 and prepare to discuss
Checkpoint Discussion
NoteCheckpoint 4
At your table, discuss:
What are the usual steps to finding a Method of Moments (MOM) estimate for an unknown parameter \(\theta\)?
How do those steps change if we have two unknown parameters?
Which of these is correct? MLE and MOM always / sometimes / never yield the same estimate.
Which approach (MLE vs MOM) makes more sense to you? Which seems easier to do?
What’s weird about the MOM estimate in the Unif\((0,\theta)\) case?
Solutions
If we just have one unknown parameter, MOM involves the following steps:
- Find the theoretical first moment \(E(Y)\)
- Set \(E(Y)\) equal to the sample first moment \(\frac{1}{n} \sum_{i=1}^n y_i\)
- Solve for \(\theta\)
If you have two unknown parameters, you have two options:
- Version 1: Solve the following system of equations involving the first and second moments \[ \begin{align*} E(Y) &\equiv \frac{1}{n} \sum_{i=1}^n y_i \\ E(Y^2) &\equiv \frac{1}{n} \sum_{i=1}^n y_i^2 \end{align*} \]
- Version 2: Solve a system of equations involving the first and second central moments \[ \begin{align*} E(Y) &\equiv \frac{1}{n} \sum_{i=1}^n y_i \\ E([Y-E(Y)]^2) &\equiv \frac{1}{n} \sum_{i=1}^n (y_i-\bar{y})^2 \end{align*} \]
sometimes
opinions will vary! (but IMO, method of moments is easier)
Recall that \(\hat\theta_{MOM} = 2\bar{Y}\). Suppose we observed the following data: 1.52, 7.81, 1.28, 6.90, 1.17. Then our method of moments estimate is
But this estimate is too small! The parameter \(\theta\) represents the maximum of our distribution. Given that we observed a value of 7.81, we know that \(\theta\) must be at least that big.
Practice Problems
Find some whiteboard space and work through the following exercises with your group.
Suppose \(Y_1, \dots, Y_n \stackrel{iid}{\sim} f_Y(y \mid \theta)\), where \[f_Y(y\mid \theta) = (\theta^2+\theta)y^{\theta-1}(1-y), \quad 0\leq y\leq 1.\] Find the MOM estimator of \(\theta\).
Hint: start by finding \(E[Y]\)
Suppose \(Y_1, \dots, Y_n \stackrel{iid}{\sim} \text{Normal}(\mu, \sigma^2)\). Derive MOM estimates for \(\mu\) and \(\sigma^2\) using “MOM Version #1” from the videos.
Hint: use what you know about the mean and variance of a normal distribution, combined with the fact that \(V(Y) = E(Y^2) - E(Y)^2\)
- Repeat Problem #2, now using “MOM Version #2” from the videos. Show that you get the same estimates.
Suppose \(Y_1, \dots Y_n\) are a random sample from the pdf \[f_Y(y; \theta) = \frac{1}{\theta^2} e^{-y/\theta^2}, \quad y > 0, \theta \neq 0.\] Show that the MOM procedure does not yield a unique estimate for \(\theta\).
Hint: use a distribution table to find \(E(Y)\) (this is an exponential pdf with \(\theta^{-2}\) in place of the usual \(\lambda\))
Suppose \(Y_i \stackrel{iid}{\sim} \text{Unif}(-\theta, \theta)\). Show that the MOM procedure doesn’t work here.
Hint: use a distribution table to find \(E(Y)\)
Solutions
This one’s on your next problem set – check your answer with another group instead :)
\(\hat\mu = \frac{1}{n} \sum_{i=1}^n Y_i\) and \(\hat\sigma^2 = \frac{1}{n} \sum_{i=1}^n (Y_i)^2 - (\bar{Y})^2\)
\(\hat\mu = \frac{1}{n} \sum_{i=1}^n Y_i\) and \(\hat\sigma^2 = \frac{1}{n} \sum_{i=1}^n (Y_i - \bar{Y})^2\), where \[ \begin{align*} \frac{1}{n} \sum_{i=1}^n (Y_i - \bar{Y})^2 & = \frac{1}{n} \sum_{i=1}^n (Y_i^2 - 2Y_i\bar{Y} + \bar{Y}^2) \\ & = \frac{1}{n} \left(\sum_{i=1}^n Y_i^2 - 2\bar{Y}\sum_{i=1}^n Y_i + n \bar{Y}^2 \right) \\ & = \frac{1}{n} \left(\sum_{i=1}^n Y_i^2 - 2\bar{Y} n \bar{Y} + n \bar{Y}^2 \right) \\ & = \frac{1}{n} \left(\sum_{i=1}^n Y_i^2 - n \bar{Y}^2 \right) \\ & = \frac{1}{n} \sum_{i=1}^n Y_i^2 - \bar{Y}^2 \end{align*} \]
From a distribution table, we know \(E(Y) = \theta^2\). (If we wanted to find this expected value “by hand”, we’d need to use integration by parts. Try it at home and confirm you get the same answer!) Thus, the system of equations we need to solve is simply \[\theta^2 \equiv \bar{Y}.\] But this has two solutions: \(\hat\theta = \pm \sqrt{\bar{Y}}\)
If \(Y \sim \text{Unif}(-\theta,\theta)\), then \(E(Y) = \frac{-\theta + \theta}{2} = 0\). To find the method of moments estimator for \(\theta\), we need to solve the system of equations \[0 \equiv \bar{Y},\] but there’s no \(\theta\) to solve for!
Wrapping Up
After class today, please:
Day 2
Settling In
- Catch up on any recent Slack messages you might have missed
- Review your notes from last class and prepare to discuss Practice Problems #2 and #3
Practice Problems (cont)
Let’s do Problems #2 and #3 together to start class…
Useful Techniques for Finding Moments
Use a distribution table 😃
\(\int f_Y(y) dy = 1\) if \(f_Y(y)\) is a pdf (eg Practice Problem #1)
\(V(Y) = E(Y^2) - E(Y)^2\) (eg Practice Problem #2)
Tip
On PS2: if it’s a “known” distribution, you can use the distribution cheatsheet to find the moments; if not, you’ll need to do so “by hand”
On Quiz 2: will need to know how to do both!
Method of Moments: Pros and Cons
Often pretty easy to compute (eg Practice Problem #3)
Not always unique (eg Practice Problem #4)
Doesn’t always work (eg Practice Problem #5)
MOM vs MLE
MOM sometimes (eg Practice Problem #2), but not always (eg \(\text{Unif}(0,\theta)\)), yields the same estimate as MLE
Usually easier than MLE
Could be a good starting value for Newton Raphson/Fisher Scoring
Multiple Unknown Parameters
Two options:
- Version 1: set the theoretical moments \(E(Y)\), \(E(Y^2)\), \(E(Y^3)\), \(\dots\) equal to their sample counterparts and solve
- Version 2: set the theoretical central moments \(\mu := E(Y)\), \(E[(Y-\mu)^2]\), \(E[(Y-\mu)^3]\), \(\dots\) equal to their sample counterparts and solve
Sometimes one of these is easier than the other (eg Practice Problem 3 vs Practice Problem 2)…
… but you get the same solution either way (eg Practice Problems 2 and 3)
Let’s prove this last statement is true in general!
Proof
Claim: The solutions to the system of equations \[ \begin{align} E(Y) & = \frac{1}{n}\sum_{i=1}^n y_i \\ E(Y^2) & = \frac{1}{n} \sum_{i=1}^n y_i^2 \\ \end{align} \] are equivalent to the solutions of the system of equations \[ \begin{align} E(Y) & = \frac{1}{n}\sum_{i=1}^n y_i \\ V(Y) & = \frac{1}{n} \sum_{i=1}^n (y_i-\bar{y})^2 \\ \end{align} \]
Proof
See board notes from today!
This is the first of many proofs we will write in this course! Here are a few general tips to keep in mind on future proofs:
TipProof Tips
Start by figuring out what, exactly, we need to do. Translate this to a mathematical statement, using mathematical notation.
Clearly state your assumptions early in the proof (by assuming they are true). Then, make sure you use those assumptions somewhere later in your proof. (Otherwise, did you need those assumptions?)
Some phrases and shorthand I’ll use throughout the semester:
shorthand meaning WMS we must show IRTS it remains to show as desired the end st such that wrt with respect to \(\Box\) the end \(\forall\) for all \(\exists\) there exsits More to come!
Wrapping Up
After class today: