10  Sufficiency

Day 1

Settling In



Concept Review

Pull out a piece of paper and answer the following questions (without looking at your notes, if possible!):

  1. Define the following terms (in words and mathematical notation):

    • unbiased
    • efficient
    • best/UMVUE

  2. Describe the connections between these terms.

  3. What are the typical steps to showing an estimator is efficient?

  4. Are there any scenarios where the “typical steps” don’t work?

Solutions
  1. definitions:
    • unbiased: \(E(\hat\theta) - \theta) = 0\) or \(E(\hat\theta) = \theta\)
    • efficient: unbiased and variance attains CRLB, ie \(E(\hat\theta) = \theta\) and \(\text{Var}(\hat\theta) = \frac{1}{I(\theta)}\), where \(I(\theta) = E\left( \left[ \frac{\partial}{\partial \theta} \log L(\theta) \right]^2 \right)\)
    • best/UMVUE: unbiased and smallest variance among all possible unbiased estimators, ie \(\hat\theta^*\) is a UMVUE for \(\theta\) if \(\text{Var}(\hat\theta^*) \le \text{Var}(\hat\theta) \ \forall \hat\theta \in \Theta\), where \(\Theta = \{\hat\theta \mid E[\hat\theta] = \theta \}\)
  2. connections:
    • must be unbiased to be best/UMVUE or efficient (part of definition!), thus:
      • best/UMVUE \(\implies\) unbiased
      • efficient \(\implies\) unbiased
    • efficient \(\implies\) best/UMVUE, but the reverse is not true
  3. steps to finding efficient (and thus best/UMVUE) estimators:
    • show \(\hat\theta\) is unbiased
    • find the variance of \(\hat\theta\)
    • find the CRLB, \(1/I(\theta)\),
      • note: often easier to use the “second version” of \(I(\theta) = -E\left( \frac{\partial^2}{\partial \theta^2} \log L(\theta) \right)\)
    • confirm that \(\text{Var}(\hat\theta)\) is equivalent to the CRLB
  4. scenarios where these steps don’t work
    • sometimes CRLB “regularity conditions” aren’t met, and thus the “CRLB” is not really a lower bound (eg \(\text{Unif}(0,\theta)\), Efficiency Practice Problem #2)
    • sometimes CRLB can’t be attained (eg estimating \(\sigma^2\) from \(N(\mu, \sigma^2)\) when both \(\mu\) and \(\sigma^2\) are unknown)



Sufficiency

Another common technique for finding UMVUEs, especially when CRLB conditions don’t hold, involves sufficient statistics.

Remember \(\hat\theta_3 = \frac{n+1}{n} X_{(n)} - 1\) from our Simulation Activity?

The CRLB conditions don’t hold here: \(f_X(x; \theta) = \frac{1}{\theta}, x = 1, ..., \theta\). The support depends on our unknown parameter, \(\theta\).

But, we can still prove that \(\hat\theta_3\) is a UMVUE using results about sufficient statistics!

(Sufficiency is a useful property for other reasons, too.)


DEFINITION

Sufficient
Let \(X_1, \dots, X_n\) be a random sample from a distribution with unknown parameter \(\theta\). The statistic \(T(\mathbf{X})\) is a sufficient statistic for \(\theta\) if the conditional distribution of \(\mathbf{X}\) given \(T(\mathbf{X})\) does not depend on \(\theta\).
NoteIntuition

We can think about sufficiency in terms of the information we need in order to estimate an unknown parameter \(\theta\).

If I want to estimate \(\theta\), do I need the full dataset, \(x_1, \dots, x_n\)? Or is knowing, for example, the sample mean, \(\bar{x} = \frac{1}{n} \sum_{i=1}^n x_i\) enough (aka sufficient)?


EXAMPLE

Suppose \(X_1, \dots, X_n\) are a random sample from the Poisson distribution with parameter \(\lambda\). Show that \(T(\mathbf{X}) = \sum_{i=1}^n X_i\) is sufficient for \(\lambda\).

HINT: Recall that \(X_i \stackrel{ind}{\sim} \text{Poisson}(\lambda_i)\) \(\implies\) \(\sum_{i=1}^n X_i \sim \text{Poisson}(\sum_{i=1}^n \lambda_i)\)

Solution See the board notes for today.



Factorization Criterion

(How do we come up with \(T\)?)

Consider a pdf for a random variable \(X\) that depends on an unknown parameter \(\theta\), given by \(f(x \mid \theta)\).

The statistic \(T(X)\) is sufficient for \(\theta\) iff \(f(x \mid \theta)\) factors as \[f(x \mid \theta) = g(T(x) \mid \theta) h(x),\] where \(g(T(x) \mid \theta)\) depends on \(x\) only through \(T(x)\) and \(h(x)\) does not depend on \(\theta\).


EXAMPLE (continued)

Try the example above, this time using the Factorization Criterion.

Solution See the board notes for today.



Rao-Blackwell-Lehmann-Scheffe (RBLS)

(Using sufficiency to find UMVUEs.)

Let \(T(\mathbf{X})\) be a complete and sufficient statistic for unknown parameter \(\theta\), and let \(\tau(\theta)\) be some function of \(\theta\).

We’ll talk about what “complete” means later.

If there exists at least one unbiased estimator \(\tilde\tau(\mathbf{X})\) for \(\tau(\theta)\), then there exists a unique UMVUE for \(\tau(\theta)\) given by \[ \hat\tau(T(\mathbf{X})) = E[\tilde\tau(\mathbf{X}) \mid T(\mathbf{X}) ] \]


Why do we care?

RBLS tells us that if \(T(\mathbf{X})\) is complete and sufficient for \(\theta\), then any function \(\phi(T(\mathbf{X}))\) is the UMVUE of its expectation \(E[\phi(T(\mathbf{X}))]\). So this gives us a very convenient way to find UMVUEs!

  1. Find a complete and sufficient statistic for an unknown parameter.
  2. Functions of that statistic are then the UMVUE for their expectation!


EXAMPLE (continued)

Suppose \(X_i \stackrel{iid}{\sim} \text{Poisson}(\lambda), \ i = 1, \dots, n\). Use RBLS and our work above to propose a UMVUE for \(\lambda\).

We’ll assume, without proof, that \(T(\mathbf{X}) = \sum_{i=1}^n X_i\) is also complete.

Solution See the board notes for today.




Day 2

Reminders

Upcoming deadlines:

  • No class on Friday. Attend Capstone Days instead!
  • PS3 due Friday @ 11:59 pm. Submit via Moodle.
    • Revised office hours schedule this week. Check the calendar on Moodle if you are planning to stop by.
  • CP8 (Consistency) due before class on Monday Wednesday.


From last class:

  • Sufficiency Definition: \(T(\mathbf{X})\) is a sufficient statistic for \(\theta\) if the conditional distribution of \(\mathbf{X}\) given \(T(\mathbf{X})\) does not depend on \(\theta\).

  • Factorization Criterion: \(T(X)\) is sufficient for \(\theta\) iff \(f(x \mid \theta) = g(T(x) \mid \theta) h(x)\).

  • Rao-Blackwell-Lehmann-Scheffe: If \(T(X)\) is sufficient and complete for \(\theta\), then \(\phi(T(X))\) is the UMVUE for its expectation (for any function \(\phi\)).




Practice Problems

Your turn! Find some whiteboard space and work through the following questions:

  1. Suppose \(Y_1, \dots, Y_n \stackrel{iid}{\sim} \text{Gamma}(\alpha, \beta)\). Find a jointly sufficient statistic for \((\alpha, \beta)\).

    RECALL: \(f_Y(y \mid \alpha, \beta) = \frac{\beta^\alpha}{\Gamma(\alpha)} y^{\alpha - 1} e^{-\beta y}, \ y \ge 0, \ \alpha > 0, \beta > 0\)

  2. Suppose \(Y_1, \dots, Y_n \stackrel{iid}{\sim} \text{Unif}(0, \theta)\).

    1. (Review) Can we use the CRLB to find a UMVUE in this scenario? Why/why not?

    2. Show that \(X_{(n)}\) is sufficient for \(\theta\).

      HINT: Write out your pdf using indicator variables.

    3. Assuming that \(X_{(n)}\) is also complete, use RBLS to propose a UMVUE for \(\theta\).

Solutions See the board notes for today.





Wrapping Up

After class today, please:




Day 3

Settling In

  • Sit at a table representing the distance you are traveling for Spring Break! ☀️

    • Staying on campus? Sit in the middle of the room.
    • Going further away? Arrange yourself accordingly.
      (your left/my right = west, towards the screens = north)
    • Check in with the people around you.

  • Turn in your Capstone Reflection

  • Catch up on any recent Slack messages you’ve missed.



Review

Pull out a piece of paper and something to write with. Without looking at your notes, answer the following:

  1. Using mathematical notation, define a sufficient statistic.

  2. In words, what does it mean for a statistic to be sufficient? Use terms that someone who’s taken only STAT 155 could understand!

  3. What are the typical steps to show something is sufficient?

  4. How can we use sufficient statistics to find UMVUEs?

Solutions See today’s board notes.



Practice Problems (revisited)

Poisson

Suppose \(X_1, \dots, X_n \stackrel{iid}{\sim} \text{Poisson}(\lambda)\). Find a sufficient statistic for \(\lambda\).

Solution See today’s board notes.


What do we learn from this example?

  • We can use the factorization criterion instead of the conditional probability distribution definition to find a sufficient statistic.

  • We can usually find more than one sufficient statistic for an unknown parameter. In fact, if \(T(\mathbf{X})\) is a sufficient statistic for \(\theta\) and \(V(\mathbf{X}) = g(T(\mathbf{X})\) where \(g()\) is a one-to-one function, then \(V(\mathbf{X})\) is also sufficient.

    Examples

    • Suppose \(X_1, \dots, X_n \stackrel{iid}{\sim} \text{Poisson}(\lambda)\). Above, we showed \(\sum_{i=1}^n X_i\) is sufficient for \(\lambda\). Thus \(\bar{X} = \frac{1}{n} \sum_{i=1}^n X_i\) is also sufficient for \(\lambda\). (See board notes.)

    • Suppose \(Y_1, \dots, Y_n \stackrel{iid}{\sim} N(\mu, \sigma^2)\). On PS3 you will show that the sample mean and variance, \[\left(\bar{X}, S^2 \right) = \left(\frac{1}{n} \sum X_i, \frac{1}{n-1} \sum (X_i - \bar{X})^2\right),\] are jointly sufficient for \(\mu, \sigma^2\).

      Using the factorization criterion, it’s perhaps easier to show that \((\sum_{i=1}^n X_i, \sum_{i=1}^n X_i^2)\) are sufficient for \(\mu, \sigma^2\). Our PS3 result then follows since \((\sum_{i=1}^n X_i, \sum_{i=1}^n X_i^2) \stackrel{1-1}{\longleftrightarrow} (\bar{X}, S^2)\).

  • Some sufficient statistics reduce the amount of information we need to keep track of more than others.

    Minimal Sufficient
    \(T^*(\mathbf{X})\) is a minimal sufficient statistic for \(\theta\) if
    1. \(T^*(\mathbf{X})\) is sufficient, and
    2. \(\exists \ g\) such that \(T^*(\mathbf{X}) = g(T(\mathbf{X}))\) for every other sufficient statistic \(T(\mathbf{X})\) (i.e., \(T^*\) is a reduction of \(T\) for every other sufficient statistic \(T\))

    Example

    In our Poisson example, the order statistics \((X_{(1)}, \dots, X_{(n)})\) are also sufficient for \(\lambda\). (See board notes.) But we can do better. Our sufficient statistic from earlier, \(\sum X_i\), is a “reduction” of \((X_{(1)}, \dots, X_{(n)})\) and turns out to be minimal sufficient in this case.

    How do we prove something is minimal sufficient? Use the Lehmann-Scheffe Theorem:

    Suppose that a random variable \(X\) has pdf given by \(f(x \mid \theta)\), and that, for every pair of points \((x,y)\), the ratio \(\frac{f(y \mid \theta)}{f(x \mid \theta)}\) does not depend on \(\theta\) iff \(T^*(x) = T^*(y)\). Then \(T^*\) is minimal sufficient for \(\theta\).

    See Section 4.4 of the MATH/STAT 355 Notes for a proof and Section 4.5 (Problem 9) for an example using Lehmann-Scheffe to find a minimal sufficient statistic.



Uniform

Suppose \(Y_1, \dots, Y_n \stackrel{iid}{\sim} \text{Unif}(0, \theta)\). Find a sufficient statistic for \(\theta\). Then, use that result to propose a UMVUE for \(\theta\).

Solution See today’s board notes.


What do we learn from this example?

  • When the support depends on our unknown parameter…

    • use indicator variables when writing down the joint pdf
    • consider order statistics

  • Notice that the sufficient statistic \(X_{(n)}\) is the same as the MLE in this scenario, and that there is also a close relationship between the Poisson sufficient statistic (\(\sum_{i=1}^n X_i\)) and MLE (\(\hat\lambda = \bar{X}\)). This is not a coincidence! In fact, all MLEs are functions of sufficient statistics.

  • If your sufficient statistic \(T(\mathbf{X})\) isn’t itself unbiased for the unknown parameter, find a function \(\phi\) such that \(\phi(T(\mathbf{X}))\) is. That’s your UMVUE!



More Definitions

Sufficient (“Looser Definition”)
\(T(\mathbf{X})\) is a sufficient statistic for \(\theta\) if the distribution of \(\mathbf{X}\) depends on \(\theta\) only through \(T(\mathbf{X})\).


Sufficient (“Bayesian Definition”)
\(T(\mathbf{X})\) is a sufficient statistic for \(\theta\) iff the posterior \(f(\theta \mid \mathbf{X}) \propto f(\mathbf{X} \mid \theta) \pi(\theta)\) depends on the data (\(\mathbf{X}\)) only through \(T\) for every possible prior distribution \(\pi\).


Complete
\(T(\mathbf{X})\) is complete if \(E[g(T(\mathbf{X}))] = 0\) implies that \(g(T(\mathbf{X})) = 0\) (almost everywhere).
Details We won’t get into the details about completeness in this class. Take Stat Theory in grad school if interested :)



Wrapping Up

  • Interested in the proof behind the Rao-Blackwell-Lehmann-Scheffe result? Check out Section 4.4 in the Stat Theory Notes

  • Interested in the proof behind the Factorization Criterion result? Check out Theorem 6.2.6 in Casella and Berger (2002).

  • For Quiz 3, make sure you can:

    • define a sufficient statistic, both in mathematical notation and in common terms
    • describe, in words, the connection between sufficient statistics and UMVUEs
    • use the factorization criterion to find a sufficient statistic
    • use RBLS to find a UMVUE
Casella, George, and Roger L. Berger. 2002. Statistical Inference. 2nd ed. Duxbury Press/Thomson Learning.