12 Other Asymptotic Properties
Day 1: Asymptotic Normality
Settling In
- Catch up on recent Slack messages
- Prepare to take notes
Types of Convergence
- Convergence in probability
- A sequence of random variables \(X_1, \dots, X_n\) converges in probability to the random variable \(X\) (denoted \(X_n \stackrel{p}{\to} X\)) if \[\lim_{n \to \infty} \Pr(\lvert X_n - X\rvert < \epsilon) = 1 \quad \forall \epsilon > 0\]
\(X_1, \dots, X_n\) need not be a random sample (iid random variables). This definition holds for any sequenec of random variables.
EXAMPLES:
Weak Law of Large Numbers: Let \(Y_1, \dots, Y_n\) be a random sample from a distribution with mean \(\mu\) and variance \(\sigma^2 < \infty\) and define \(\overline{Y}_n = \frac{1}{n} \sum_{i=1}^n Y_i\). Then, \(\overline{Y}_n \stackrel{p}{\to} \mu\).
(Conceptually, this tells us that the sample mean, \(\overline{Y}_n\), will be very close to the theoretical/population mean, \(E(Y)\), as the sample size gets large. This is what makes \(\overline{Y}\) such a popular estimator!)
Consistency is a statement about convergence in probability! If \(\hat\theta_n\) is a consistent estimator of \(\theta\), then, equivalently,
the sequence of estimators \(\hat\theta_1, \hat\theta_2, \dots\) (where we repeat our estimation procedure for each sample size \(1, 2, \dots\)) converges in probability to \(\theta\), ie \(\hat\theta_n \stackrel{p}{\to} \theta\).
- Convergence almost surely
- (“convergence almost everywhere”, “convergence with probability 1”)
- A sequence of random variables converges almost surely to the random variable \(X\) (denoted \(X_n \stackrel{a.s.}{\to} X\)) if \[\Pr(\lim_{n \to \infty} \lvert X_n - X \rvert < \epsilon) = 1 \quad \forall \epsilon > 0.\]
Although they look very similar, convergence almost surely is stronger than convergence in probability. Here’s a nice discussion of the conceptual difference between the two on Stack Exchange.
We’ll focus on convergence in probability in this class, but you’ll see almost sure convergence again if you go to grad school in statistics.
EXAMPLE:
- Strong Law of Large Numbers: Let \(Y_1, \dots, Y_n\) be a random sample from a distribution with mean \(\mu\) and variance \(\sigma^2 < \infty\) and define \(\overline{Y}_n = \frac{1}{n} \sum_{i=1}^n Y_i\). Then, \(\overline{Y}_n \stackrel{a.s.}{\to} \mu\).
- Convergence in distribution
- (“weak convergence”, “convergence in law”)
- A sequence of random variables \(X_1, \dots, X_n\) converges in distribution to the random variable \(X\) (denoted \(X_n \stackrel{d}{\to} X\)) if \[\lim_{n \to \infty} F_{X_n}(x) = F_X(x)\] for all \(x\) at which \(F_X(x)\) is continuous.
Convergence in distribution is particularly useful for inference, especially constructing confidence intervals and hypothesis tests. We’ll use it a lot in our next unit!
EXAMPLE:
Central Limit Theorem: Let \(Y_1, \dots, Y_n\) be a random sample from a distribution with mean \(\mu\) and variance \(\sigma^2 < \infty\) and define \(\overline{Y}_n = \frac{1}{n} \sum_{i=1}^n Y_i\). Then, \[\sqrt{n}\left(\frac{\overline{Y}_n - \mu}{\sigma}\right) \stackrel{d}{\to} Z,\] where \(Z \sim N(0, 1)\).
This is sometimes written instead as \(\sqrt{n}(\overline{Y}_n - \mu) \stackrel{d}{\to} N(0, \sigma^2)\).
Connections:
- convergence almost surely \(\implies\) convergence in probability \(\implies\) convergence in distribution
- \(X_n \stackrel{d}{\to} c \Longleftrightarrow X_n \stackrel{p}{\to} c\) if \(c\) is a constant
Asymptotic Normality
An estimator \(\hat\theta_n\) is asymptotically normal if it (after appropriate scaling, eg \(\sqrt{n}(\hat\theta_n - \theta)\)) converges in distribution to a normally distributed random variable.
Activity: Proving the CLT
- Moment Generating Functions
- The moment generating function of a random variable \(X\) is defined \[M_X(t) = E[e^{tX}].\]
Fun Facts about MGFs:
\(M_X(0) = 1\), always.
The \(r\)th derivative of the MGF, evaluated at \(t = 0\), can help us find moments.
- For example, \(M_X^{(1)}(0) = \frac{d}{dt} M_X(t) \rvert_{t = 0} = E[X]\).
If two random variables have the same MGF, they have the same probability distribution!
MGFS are useful for figuring out the distribution of transformations of RVs:
- If \(V = aW + b\) then \(M_V(t) = e^{bt} M_W(at)\).
- If \(W = W_1 + W_2 + \dots W_n\), where \(W_1, \dots, W_n\) are independent, then \(M_W(t) = M_{W_1}(t) M_{W_2}(t) \dots M_{W_{n}}(t)\).
- Theorem
- Let \(W_1, \dots, W_n\) be a sequence of random variables with MGFs \(M_{W_1}(t), \dots, M_{W_n}(t)\), and let \(V\) be another random variable with MGF \(M_V(t)\). If \[\lim_{n \to \infty} M_{W_n}(t) = M_V(t) \quad \forall t \text{ in some neighborhood of } 0,\] then \(W_n \stackrel{d}{\to} V\).
Proof
We won’t prove this one :)💡 To show convergence in distribution, we just need to look at the MGFs!
Solutions
See today’s board notes.
Wrapping Up
Day 2: Asymptotics Toolkit
Settling In
- Fill out this Midterm Survey [link]
- Includes questions about the quiz format, how you are engaging with the course material, etc.
- Please share your thoughts!
- Catch up on recent Slack messages
Motivation
The Central Limit Theorem and Weak (or Strong) Law of Large Numbers provide very nice results about the asymptotic behavior of sample averages:
- WLLN: \(\overline{Y}_n \stackrel{p}{\to} \mu\)
- CLT: \(\sqrt{n}(\overline{Y}_n - \mu) \stackrel{d}{\to} N(0, \sigma^2)\)
As we’ve seen throughout the semester, estimators are often sample averages or functions thereof:
\(Y_i \stackrel{iid}{\sim} N(\mu, \sigma^2)\):
- \(\hat\mu_{MLE} = \frac{1}{n}\sum_{i=1}^n Y_i\)
- \(\hat\sigma^2_{MLE} = \frac{1}{n}\sum_{i=1}^n (Y_i - \overline{Y})^2\)
\(Y_i \stackrel{iid}{\sim} \text{Exponential}(\lambda)\):
- \(\hat\lambda_{MLE} = \left( \frac{1}{n}\sum_{i=1}^n Y_i \right)^{-1}\)
\(X_i \stackrel{iid}{\sim} \text{Poisson}(\lambda)\):
- \(\hat\lambda_{MLE} = \frac{1}{n} \sum_{i=1}^n X_i\)
Challenge: Look back at the many estimation problems we’ve done this semester (problem sets, quizzes, videos, in-class practice problems). How many estimators can you find that are not functions of a sample average? There aren’t many!
For some of these estimators (eg \(\hat\mu_{MLE}\), \(\hat\lambda_{MLE}\)), we can simply apply WLLN and CLT to prove they are consistent and asymptotically normal.
But what about the estimators that are functions of sample averages (eg \(\hat\sigma^2_{MLE}, \hat\lambda_{MLE}\))?
Important Theorems
- Continuous Mapping Theorem
- If \(X_n \stackrel{p}{\to} X\) and \(g\) is a continuous function, then \(g(X_n) \stackrel{p}{\to} g(X)\).
Proof
We won’t prove this one. Challenge: think about how you might approach this proof.
- Slutsky’s Theorem
- Suppose \(X_n \stackrel{d}{\to} X\) and \(Y_n \stackrel{p}{\to} c\), where \(c\) is a constant. If \(g(x, y)\) is a jointly continuous function, then \(g(X_n, Y_n) \stackrel{d}{\to} g(X, c)\).
Proof
We won’t prove this one either :)EXAMPLES:
If \(X_n \stackrel{d}{\to} X\) and \(Y_n \stackrel{p}{\to} c\), then \(X_n + Y_n \stackrel{d}{\to} X + c\).
If \(X_n \stackrel{d}{\to} X\) and \(Y_n \stackrel{p}{\to} c\), then \(Y_n X_n \stackrel{d}{\to} cX\).
If \(X_n \stackrel{p}{\to} a\) and \(Y_n \stackrel{p}{\to} b\), then \(X_n + Y_n \stackrel{p}{\to} a + b\) (recall that \(X_n \stackrel{p}{\to} c \Longleftrightarrow X_n \stackrel{d}{\to} c\))
- Delta Method (aka “propagation of error”)
- Suppose \(\sqrt{n}(Y_n - c) \stackrel{d}{\to} N(0, t)\) for some \(t > 0\). If \(g(y)\) is differentiable at \(c\) and \(g'(c) \neq 0\), then \[\sqrt{n}\left[g(Y_n) - g(c)\right] \stackrel{d}{\to} N\left(0, t [g'(c)]^2 \right).\]
Proof
See today’s board notes.
Practice Problems
Suppose \(Y_1, \dots, Y_n\) are a random sample from an exponential distribution, parameterized as follows: \[f_Y(y; \lambda) = \lambda e^{-\lambda y}, \quad y \ge 0, \quad \lambda > 0.\] Show that \(\hat\lambda_{MLE} = 1 / \overline{Y}\) is a consistent estimator for \(\lambda\).
HINT: \(E(Y) = 1 / \lambda\).
Suppose \(\sqrt{n}(Y_n - \mu) \stackrel{d}{\to} N(0, \sigma^2)\), where \(\sigma^2 > 0\).
Find the asymptotic distribution of \(\sqrt{n}(Y_n^2 - \mu^2)\).
What needs to be true about \(\mu\) for this result to hold?
Suppose \(X_1, \dots, X_n\) are a random sample from a bernoulli distribution with unknown probability of success \(p\). Recall that the MLE for \(p\) is \(\hat{p} = \frac{1}{n} \sum_{i=1}^n X_i\).
Find the asymptotic distribution of \(\hat{p}\) using the CLT.
Find the asymptotic distribution of the estimated odds of success, \(\widehat{\text{odds}} = \frac{\hat{p}}{1- \hat{p}}\). What needs to be true about \(p\) for this result to hold?
Solutions
See today’s board notes.
If you finish all of the practice problems above, work on PS4! You’ll use this “asymptotics toolkit” to solve Problems 4–6.
Wrapping Up
We use these theorems a lot when exploring asymptotic properties of estimators. In particular:
- Consistency: WLLN, Continuous Mapping Theorem, (sometimes Slutsky’s)
- Asymptotic Normality: CLT, Slutsky’s, Delta Method
After class today, please work on the following:
Day 3: Asymptotic Properties of MLEs
Settling In
- Catch up on recent Slack messages
- FINAL REMINDER: please fill out the Midterm Survey! [link]
- (haven’t heard from ~15 of you)
Motivation
Question: Why is maximum likelihood estimation so popular?
Answer: MLEs have nice asymptotic properties!
Under certain “regularity conditions” (which are often met), we can prove MLEs:
- always exist
- are unique
- are “strongly” consistent (i.e \(\hat\theta_{MLE} \overset{a.s.}{\to} \theta\))
- are consistent (i.e \(\hat\theta_{MLE} \overset{p}{\to} \theta\))
- are asymptotically normal
- are asymptotically efficient (NEW)
Asymptotic Efficiency
- Asymptotic Variance
- Suppose \(c_n(T_n - \tau(\theta)) \stackrel{d}{\to} N(0, \sigma^2)\) for some \(0 < \sigma^2 < \infty\). Then \(\sigma^2\) is the asymptotic variance of \(T_n\).
- Asymptotically Efficient
- A sequence of estimators \(T_n\) is asymptotically efficient for \(\tau(\theta)\) if \[\sqrt{n}(T_n - \tau(\theta)) \overset{d}{\to} N(0, v(\theta)),\] where \[v(\theta) = \frac{[\tau'(\theta)]^2}{E\left[ \left( \frac{\partial}{\partial\theta} \log f(\mathbf{x} \mid \theta \right)^2 \right]}.\]
💡 In other words, \(T_n\) is asymptotically efficient if its asymptotic variance achieves the CRLB.
Notice that asymptotic efficiency is defined only in the context of asymptotic normality. Thus, \(\hat\theta_n\) is asymptotically efficient \(\implies\) \(\hat\theta_n\) is asymptotically normal.
Proof: Asymptotic Properties of MLEs
Let’s prove that MLEs are asymptotically normal and efficient (and consistent)!
- Theorem
- Suppose \(X_1, \dots, X_n\) are a random sample from a pdf \(f_X(x \mid \theta)\), \(\hat\theta_n\) is the MLE for \(\theta\), and \(\tau(\theta)\) is a continuous function of \(\theta\). If \(f_X(x \mid \theta)\) satisfies certain regularity conditions1, then \[\sqrt{n}(\tau(\hat\theta_n) - \tau(\theta)) \overset{d}{\to} N(0, v(\theta)),\] where \(v(\theta)\) is the CRLB.
Proof
See today’s board notes.
For our proof, we’ll need to use various tools from our Asymptotics Toolkit! Before we start, let’s remind ourselves of the following:
- WLLN
- CLT
- Slutsky’s Theorem
- Delta Method
- Claim
- The asymptotic normality of MLEs implies that they are consistent.
Proof
See today’s board notes.
Wrapping Up
Highlights from today:
- new property: asymptotic efficiency
- connections between properties: asymptotic efficiency \(\implies\) asymptotic normality \(\implies\) consistency
- another example of how to use the various tools from our Asymptotics Toolkit to explore the asymptotic behavior of estimators
- explanation of why MLEs are so popular!
What’s next?
- using the asymptotic behavior of estimators to derive confidence intervals!
After class today, please:
-
- Warning: this is a long one!
among other things, the support cannot depend on \(\theta\)↩︎