9  Efficiency

The Cramér-Rao Lower Bound

Day 1

Settling In

  • Sit anywhere you want
  • Catch up on Slack
  • Reminders:
    • Quiz 2 coming up on Friday!
      • New Content: Method of Moments, Bias, Variance, MSE
      • Retakes: MLE-01, MLE-02, MLE-03, OLS-01
    • Complete all Quiz 1 revisions (in office hours) before Quiz 2.
      • Mon 3:30–4:30, Thu 2–4, or by appointment
    • See #announcements for details on revisions and retakes.


Checkpoint Discussion

NoteCheckpoint 6

CP6 covers the following content:

  • CLRB Intro Video (slides)
  • CRLB Example Video (slides)
  • Chapter 4 in the MATH/STAT 355 Notes (link)

At your table, discuss:

  1. What are the connections and distinctions between the following terms: bias, variance, best, UMVUE, and efficient?

  2. What does the Cramér-Rao Lower Bound represent, conceptually? How is it related to the terms above?

  3. How is the Cramér-Rao Lower Bound defined, mathematically?

  4. What conditions must hold for the Cramér-Rao Lower Bound Theorem to apply? Can you come up with a scenario where these conditions are not met?

Solutions

  1. Concept Map

    UMVUE: Uniformly Minimum-Variance Unbiased Estimator
    \(\hat\theta^*\) is the UMVUE for \(\theta\) if \[\text{Var}(\hat\theta^*) \le \text{Var}(\hat\theta) \quad \forall \hat\theta \in \Theta,\] where \(\Theta\) is the collection of all unbiased estimators of \(\theta\).
    Best
    \(\hat\theta^*\) is the best estimator for \(\theta\) if it is unbiased and it has the smallest variance among all unbiased estimators of \(\theta\). (Best = UMVUE)
    Efficient
    \(\hat\theta^*\) is an efficient estimator for \(\theta\) if it is unbiased and its variance attains the CRLB, ie \[\text{Var}(\hat\theta^*) = 1/I(\theta)\]

    UMVUE/best/efficient estimators allow us to move beyond comparing the variance of two unbiased estimators head-to-head and go directly to find the “best possible” unbiased estimator. But how do we show an estimator is UMVUE/best/efficient? One approach: use the Cramér-Rao Inequality!

  2. The CRLB represents a lower bound on what the variance of an unbiased estimator can be. If the variance of an unbiased estimator is equal to the CRLB, this tells us that it has the smallest variance of all possible unbiased estimators and is thus best/UMVUE (and also, by definition, efficient).

  3. The CRLB is equal to \(\frac{1}{I(\theta)}\), where the Fisher Information \(I(\theta)\) is defined as follows: \[I(\theta) = E\left[\left(\frac{d}{d\theta} \ln L(\theta) \right)^2 \right] = -E\left[\frac{d^2}{d\theta^2} \ln L(\theta) \right].\]

  4. We must have a random sample from a continuous pdf, with continuous first- and second-order derivatives, where the support does not depend on the unknown parameter \(\theta\).

    The \(\text{Uniform}(0,\theta)\) that we have explored many times this semester does NOT meet these conditions: the support depends on \(\theta\).



Cramér-Rao Lower Bound

Theorem

Let \(Y_1, \dots, Y_n\) be a random sample from a continuous pdf \(f_Y(y;\theta)\) with a single unknown parameter \(\theta\), and let \(\hat\theta = g(Y_1, \dots, Y_n)\) be any unbiased estimator for \(\theta\).

Suppose that \(f_Y(y;\theta)\) has continuous first- and second-order derivatives. Also, suppose that the set of \(y\) values where \(f_Y(y;\theta) \neq 0\) (i.e., the support) does not depend on \(\theta\).

Then, \[Var(\hat\theta) \ge 1/I(\theta),\]

where \[I(\theta) = E\left[\left(\frac{d}{d\theta} \ln L(\theta) \right)^2 \right] = -E\left[\frac{d^2}{d\theta^2} \ln L(\theta) \right]\]


Proof

I’ll pass out paper copies of our proof activity now.

Work with the people at your table and ask each other, and me, questions as you go!

Solutions



Wrapping Up

Now that we’ve proven the CRLB Theorem, we can use it to find efficient estimators. More practice with this next class!

After class today, please:




Day 2

Settling In

  • Sit anywhere you want
  • Catch up on Slack
  • Check your Stat 355 Feedback Spreadsheet for recent updates (Quiz 1, PS2, etc.)
  • Note the revised PS3 submission instructions



Properties Recap

Last week, we discussed tools to compare two estimators and decide which is better:

  • smaller bias = better
  • smaller variance = better


Follow-Up Question: Is there a way to prove that an estimator is “best” of all possible estimators, without having to compare it to every other estimator head-to-head?

Answer: Yes! Two helpful techniques for this:

  1. Cramér-Rao Lower Bound:

    If you find an unbiased estimator \(\hat\theta\) whose variance attains the CRLB, you know that no other estimator can have lower variance \(\implies\) it is “best”

  2. Rao-Blackwell-Lehmann-Scheffe:

    Helps us find UMVUEs based on sufficient statistics. More on this next week!



Practice Problems

Let’s practice using the CRLB to find efficient (and thus best/UMVUE), estimators.


  1. Let \(X_1, X_2, \dots, X_n\) denote the outcome of a series of \(n\) independent trials, where \(X_i = 1\) with probability \(p\) and \(X_i = 0\) with probability \(1-p\), for \(i = 1, 2, \dots, n\).

    On PS2, we showed that \(\hat{p}_1 = X_1\) and \(\hat{p}_2 = \frac{X}{n}\) are both unbiased estimators of \(p\), but that \(\hat{p}_2\) has the lower variance of the two. Thus, \(\hat{p}_2\) is a better estimator than \(\hat{p}_1\). But is it the best?

    Use the CRLB to show that it is.


  1. Suppose \(Y_1, \dots, Y_n\) are a random sample from the distribution with pdf \[f_Y(y; \theta) = \frac{2y}{\theta^2}, \quad 0 \le y \le \theta.\]

    1. (Review) Find the MOM estimator for \(\theta\).

    2. Find the variance of \(\hat\theta_{MOM}\) and compare it to the CRLB.
      What do you find?

      Note

      Your answer here will depend on which “version” of the information you use.

      • Version 1: \(I(\theta) = E[\left(\frac{\partial}{\partial\theta} \log \left[\prod_{i=1}^n f_Y(y_i;\theta)\right]\right)^2]\)
      • Version 2: \(I(\theta) = -E[\frac{\partial^2}{\partial\theta^2} \log \left[\prod_{i=1}^n f_Y(y_i;\theta)\right]]\)
      • Version 3: \(I(\theta) = nE\left[\left(\frac{\partial}{\partial\theta} \log f_Y(y;\theta)\right)^2\right]\), where \(f_Y(y;\theta)\) is the pdf for a single observation (not the joint pdf/likelihood)
      • Version 4: \(I(\theta) = -n E[\frac{\partial^2}{\partial\theta^2} \log f_Y(y;\theta)]\)

      Version 1 is the general definition, but Version 2 is often much easier to work with. On PS3, you’ll show that Versions 1 and 2 are equivalent if the CRLB regularity conditions hold. Versions 3 and 4 are equivalent to 1 and 2, respectively, if your observations are iid (and the CRLB regularity conditions hold).

      If you use Version 3, as do Larsen and Marx (2018) in a similar problem, you’ll find that the variance of the MOM estimator is smaller than the Cramér-Rao “Lower Bound.”

    3. Explain why your result in (b) does not contradict the CRLB Theorem.


  1. Let \(X_1, \dots, X_n\) be a random sample from a Poisson distribution with parameter \(\lambda\), where \[f_X(x; \lambda) = \frac{\lambda^x e^{-\lambda}}{x!}.\] Show that \(\hat\lambda = \bar{X}\) is an efficient estimator for \(\lambda\).


Solutions See here.



Efficiency Recap

When proving that an estimator is efficient, remember:

  1. Check if your estimator is unbiased, first.

    If your estimator is biased, there’s a different lower bound on the variance, namely, \(\text{Var}(\hat\theta) = \frac{[m'(\theta)]^2}{I(\theta)}\), where \(m(\theta) = E(\hat\theta)\).

  2. The CRLB doesn’t always apply. Check the conditions!

    Examples: \(Y_i \stackrel{iid}{\sim} \text{Unif}(0, \theta)\), Practice Problem #2

  3. Even when the CRLB does apply, it can’t always be attained.

    Example: \(Y_i \stackrel{iid}{\sim} N(\mu, \sigma^2)\) where both \(\mu\) and \(\sigma^2\) are unknown and we are trying to estimate \(\sigma^2\). The CRLB cannot be attained. (See Casella and Berger (2002) Section 7.3.)


What do we do in Scenarios 2 and 3? Is it still possible to find a UMVUE?

Sometimes! We’ll talk about alternate techniques next week when we cover sufficiency.



Wrapping Up

After class today, please:

    • New Content: Method of Moments, Bias, Variance, MSE
    • Retakes: MLE-01, MLE-02, MLE-03, OLS-01




Casella, George, and Roger L. Berger. 2002. Statistical Inference. 2nd ed. Duxbury Press/Thomson Learning.
Larsen, Richard J., and Morris L. Marx. 2018. An Introduction to Mathematical Statistics and Its Applications. 6th ed. Pearson.