Derive a hypothesis test to test the one-sided hypotheses \(H_0: \mu = 8\) vs \(H_1: \mu > 8\) at the \(\alpha = 0.05\) significance level. Assume \(\sigma^2 = 64\) and we suppose observe \(\bar{Y} = 10\) based on a sample of size \(n = 64\).
Calculate a p-value. Do we make the same decision?
Your turn!
Revisit the scenario in Question 1. How will our procedure change if…
we use \(\alpha = 0.01\)?
we use a two-sided alternative hypothesis: \(H_1: \mu \neq 8\)?
we use a different test statistic: \(T_2 = \frac{\bar{Y}-8}{\sigma / \sqrt{n}}\)?
we don’t know \(\sigma^2\)?
Suppose we have a random sample of size \(n\) from a Bernoulli with probability of success \(p\), and we want to test the hypotheses \(H_0: p = 0.5\) vs \(H_1: p \neq 0.5\). Derive a hypothesis test for this scenario, using the test statistic \(T = \frac{\hat{p} - 0.5}{\sqrt{\hat{p}(1-\hat{p})/n}}\) and a significance level of 0.05.
Consider the scenario in Question 3, but suppose the sample size is small. Do you have any concerns about the test you derived above? If so, propose and derive an alternate test.
There are three “classical” hypothesis tests for testing the following hypotheses: \[H_0: \theta = \theta_0 \text{ vs } H_1: \theta \neq \theta_0\]
Suppose \(\hat\theta_{MLE}\) is an MLE for \(\theta\). Then…
Wald Test
The Wald test statistic \(\lambda_W\) is given by \[\lambda_W = \left( \frac{\hat\theta_{MLE} - \theta_0}{se(\hat\theta_{MLE})} \right)^2\]
We reject \(H_0\) for large values of \(\lambda_W\).
Intuition:
\(\theta_0\) close to observed MLE \(\rightarrow\) small numerator \(\rightarrow\) small \(\lambda_W\) (fail to reject \(H_0\))
\(\theta_0\) far from MLE \(\rightarrow\) large numerator \(\rightarrow\) large \(\lambda_W\) (reject \(H_0\))
Likelihood Ratio Test (aka Wilks Test)
The Likelihood Ratio test statistic \(\lambda_{LRT}\) is given by \[\lambda_{LRT} = -2 \log \left( \frac{\sup_{\theta = \theta_0} L(\theta)}{\sup_{\theta \in \Theta}L(\theta)} \right) = -2 \log \left( \frac{L(\theta_0)}{L(\hat\theta_{MLE})} \right),\] where \(\Theta\) is the set of all possible values of \(\theta\).
We reject \(H_0\) for large values of \(\lambda_{LRT}\).
Intuition:
\(\theta_0\) close to MLE \(\rightarrow\) numerator similar to denominator \(\rightarrow\) ratio close to 1 \(\rightarrow\)\(\lambda_{LRT}\) close to 0 (fail to reject \(H_0\))
\(\theta_0\) far from MLE \(\rightarrow\) numerator much smaller than denominator \(\rightarrow\) ratio close to 0 \(\rightarrow\)\(\lambda_{LRT}\) gets really big (reject \(H_0\))
Score Test
The score test statistic \(\lambda_S\) is given by \[\lambda_S = \frac{[S(\theta_0)]^2}{I(\theta_0)},\] where \(S(\theta) = \frac{\partial}{\partial \theta} \log L(\theta)\) and \(I(\theta) = E\left[ \left( \frac{\partial}{\partial \theta} \log L(\theta)\right)^2 \right]\).
As above, reject \(H_0\) for large values of \(\lambda_S\).
Intuition:
\(\theta_0\) close to MLE \(\rightarrow\)\(S(\theta_0)\) close to zero \(\rightarrow\) small \(\lambda_S\) (fail to reject \(H_0\))
\(\theta_0\) far from MLE \(\rightarrow\) large \(\lambda_S\) (reject \(H_0\))
Comparison
Similarities
Asymptotically, these three tests are equivalent!
All three test statistics have the same asymptotic distribution:
You’ll prove this is true for a specific case on PS5!
More generally, we can prove this result using a combination of the asymptotic normality of MLEs + Slutsky’s / Continuous Mapping Theorem + Taylor Expansion
Differences
If they’re “equivalent,” why pick one over another?
Wald:
easy to compute
con: “non-invariant” to reparameterization (PS5)
LRT:
uniformly most powerful for simple alternatives (\(H_0: \theta = \theta_0\) vs \(H_1: \theta = \theta_1\))
Score:
don’t need the MLE! (helpful if MLE has no closed form, eg logistic regression)
Practice Problems
Suppose \(X_1, \dots, X_n\) are a random sample from a Bernoulli distribution with unknown probability of success \(p\), and we want to test the hypotheses \(H_0: p = 0.5\) vs \(H_1: p \neq 0.5\).
Derive the Wald test statistic.
Derive the LRT statistic.
Derive the score test statistic.
Calculate the p-values for all three tests if \(\hat{p} = 0.4\) and \(n = 300\).
Calculate the p-values for all three tests if \(\hat{p} = 0.4\) and \(n = 95\).
What’s the difference between type I error and type II error?
Which type of error is most closely connected to the significance level, \(\alpha\)? How are they related?
What is power? Which type of error is most closely connected to the concept of power? How are they related?
Solutions
type I = reject \(H_0\) when \(H_0\) is true
type II = fail to reject \(H_0\) when \(H_1\) is true
\(\alpha = \Pr(\text{type I error})\)
power = \(\Pr(\text{reject } H_0 \mid H_1 \text{ true} )\)
power = 1 - type II error
Why Care?
Choosing a significance level
Choosing between two or more possible tests
Designing a study (power analyses)
Practice Problems
Will \(n = 45\) be a sufficiently large sample to test \(H_0: \mu = 10\) versus \(H_1: \mu \neq 10\) at the \(\alpha =0.05\) level of significance if the experimenter wants the Type II error probability to be no greater than 0.20 when \(\mu = 12\)? Assume that \(\sigma =4\).
Suppose \(n=36\) observations are taken from a Normal distribution where \(\sigma = 8.0\) for the purpose of testing \(H_0:\mu=60\) versus \(H_1: \mu \not=60\) at the \(\alpha = 0.07\) level of significance. The lead investigator skipped statistics class the day decision rules were being discussed and intends to reject \(H_0\) if \(\bar{y}\) falls in the region \((60 - \bar{y}^*, 60+ \bar{y}^*)\).
Explain why this rejection region doesn’t make much sense.
Find \(\bar{y}^*\) that would give you the right level of significance.
What is the power of the test when \(\mu=62\)?
What would the power of the test be when \(\mu=62\) if the rejection region had been defined the correct way?
Solutions
This question is on your problem set this week. Check your answer with a classmate :)
…
(a) This investigator set up their rejection region such that we will reject the null hypothesis if the estimate, \(\bar{y}\) is close to the null value, 60. This is the opposite of what we would usually do!
So, we need to find \(-z^*, z^*\) between which a standard Normal distribution falls 7% of the time, or, equivalently, outside which a standard Normal distribution falls 93% of the time.
Since this rejection region is symmetric, this implies that we want to find the value \(-z^*\) below which a standard Normal falls \(\frac{93}{2}\)% of the time, and then the distribution will fall above \(z^*\) the other \(\frac{93}{2}\)% of the time:
## find -z* such that a N(0,1) falls below it (93/2)% of the timeqnorm(.93/2, mean =0, sd =1)
[1] -0.08784484
## z*qnorm(.93/2, mean =0, sd =1, lower.tail = F)
[1] 0.08784484
This implies that \(\bar{y}^* = z^*\frac{8}{6} = 0.1171265\).
(c) The power of the test under the alternative \(\mu = 62\) is
where \(\bar{y} \stackrel{\mu = 62}{\sim} N(62, \frac{8^2}{36})\).
Thus, the power is:
## calculate powerpnorm(60+0.117, mean =62, sd =8/6) -pnorm(60-0.117, mean =62, sd =8/6)
[1] 0.02276674
This is very small!
(d) To do this test correctly, we reject \(H_0\) if \(\bar{y} \not\in (60 - \bar{y}^*, 60+ \bar{y}^*)\). To find the power of this test when \(\mu = 62\), we first need to find the value of \(\bar{y}^*\) that gives us the right level of significance:
where, as before, we have \(\frac{\bar{y} - 60}{8/\sqrt{36}} \stackrel{H_0}{\sim} N(0,1)\).
To find \(-z^*,z^*\) we now need to find the values between which a standard Normal falls 93% of the time (or, equivalently, outside which a standard Normal falls 7% of the time).
This means that we need to find \(-z^*\) below which a standard Normal falls \(\frac{7}{2}\)% of the time, and \(z^*\) above which a standard Normal falls the other \(\frac{7}{2}\)% of the time:
# find -z* such that a N(0,1) falls below it (7/2)% of the timeqnorm(0.07/2, mean =0, sd =1)
[1] -1.811911
# find z* such that a N(0,1) falls above it (7/2)% of the timeqnorm(0.07/2, mean =0, sd =1, lower.tail = F)
[1] 1.811911
This implies that our critical value is \(\bar{y}^* = z^*\frac{8}{6} = 2.4158809\).
Now that we have \(\bar{y}^*\), we can find the power of this test under the alternative \(\mu = 62\):
where, as above, \(\bar{y} \stackrel{\mu = 62}{\sim} N(62, \frac{8^2}{36})\). Thus \(\beta\) is
## calculate beta (1 - power)pnorm(60+2.4216, mean =62, sd =8/6) -pnorm(60-2.4216, mean =62, sd =8/6)
[1] 0.6236184
which implies that the power \(1-\beta = 0.3763816\). This is much higher than the power of the initially proposed test!
Wrapping Up
Concept Debrief
A “good” hypothesis test has low type I error, low type II error, and high power.
Type I error is influenced by the significance level (\(\alpha\)).
Power and type II error are influenced by:
the rejection region (which comes from our choice of \(\alpha\))
the true value of the parameter under \(H_1\)
the sample size
the variability of data
In applied analysis, it is common practice to calculate the power of your study before collecting data and use this to help choose a sample size. (But in order to do this, you need to make assumptions about all of the factors listed above!)
What is power? (In words and mathematical notation)
Does a “good” test have large (close to 1) or small (close to 0) power?
Suppose we are interested in testing the hypotheses \(H_0: \theta = \theta_0\) vs \(H_1: \theta \neq \theta_0\). Come up with a test that will have power exactly equal to 1, and another that will have power exactly equal to 0.
Uniformly Most Powerful Tests
Power Function
The power function of a hypothesis test is defined \[
\gamma(\theta)
= \Pr(\mathbf{X} \in R \mid \theta)
= \begin{cases}
\Pr(\text{T1E}) & \text{ if } \theta \in \Theta_0 \\
\text{power} & \text{ if } \theta \in \Theta_0^c
\end{cases}
\]
Uniformly Most Powerful
Let \(\mathcal{C}\) represent the class of all level \(\alpha\) tests, ie \[
\mathcal{C} = \{\gamma(\theta) : \sup_{\theta \in \Theta_0} \gamma(\theta) = \alpha \}.
\] A hypothesis test, with power function \(\gamma^*(\theta)\), is a uniformly most powerful level \(\alpha\) test if \(\gamma^*(\theta) \in \mathcal{C}\) and \(\gamma^*(\theta) \ge \gamma\prime(\theta)\) for all \(\theta \in \Theta_0^C\) and all \(\gamma\prime(\theta) \in \mathcal{C}\).
Intuition: A hypothesis test is “uniformly most powerful” if, when compared to all other tests with the same significance level \(\alpha\), it has higher power for all values of the parameter under the alternative hypothesis. (This should remind you of of UMVUEs!)
Q: Can we always find a uniformly most powerful test?
A: In general, no. But in some scenarios, yes!
Neyman-Pearson Lemma
Suppose we are interested in testing the “simple” hypotheses \(H_0: \theta = \theta_0\) vs \(H_1: \theta = \theta_1\).
The Neyman-Pearson Lemma tells us that the simplified likelihood ratio test with test statistic \(\lambda_{LRT} = \frac{L(\theta_1)}{L(\theta_0)}\) (where we reject \(H_0\) for large values of \(\lambda\)) is a UMP test in this scenario.
Proof
Check out Section 7.4 of the MATH/STAT 355 Notes. This is a fun proof and uses a similar technique to the proof of Chebyshev’s Inequality!
Putting it All Together
The end of the semester is quickly approaching! This is our final day of “required” content. (The next few class periods will cover “bonus” topics.)
We’ve covered a lot this semester. How does it all connect??
Activity
Create a study guide for one of the following topics:
estimation techniques: MLE, least squares, and MOM
finite sample properties: bias, variance, MSE
finite sample properties (cont): efficiency and sufficiency