At your table:
introduce yourselves: name, pronouns, class year, major, hometown, highlights from your winter break, something you’re looking forward to this semester (in Stat Theory or otherwise!), etc.
check in with each other
brainstorm a list of statistical techniques or concepts that you’ve come across in previous courses where the underlying mathematical details were simplified or skipped altogether. which are you particularly curious to learn more about?
Name: call me “Kelsey” or “Professor Grinde”
Pronouns: she/her/hers
Hometown(s):
Major(s):
Things I love about Stat Theory:
Some other things I love (not necessarily in this order!):
In this course we will…
This course will has a different feel and focus than other stats classes you’ve taken at Mac! We will do a lot of math, some computation, and almost no data application.
At your table, discuss:
What’s one thing you’re good at outside of school?
How did you get good at that thing?
Once you’ve discussed as a table, share (and upvote) responses to Question #2 here:
This is a challenging course. To help support you in that challenge, we will:
Devote a lot of class time to practice.
Have lots of opportunities to engage in feedback loops.
Implications for course structure:
Questions about this or anything else on the Syllabus?
The first part of this course will focus on the topic of estimation, exploring different techniques for estimating the value of unknown parameters in probability models.
You’re already familiar with one such technique—least squares estimation—from STAT 155 and STAT 253.
Forced expiratory volume (FEV) is a measure of lung capacity. Higher values indicate higher lung capacity—the ability to blow out more air in one second—and therefore indicate better lung function.
The fev.csv dataset contains data on lung function for a total of 654 children, ages 3-19. Each row in the dataset corresponds to a single child’s visit to a doctor’s clinic. All children participating in this study attended the same pediatric clinic. The exact date of the study is unknown, but we at least know that it took place before 2005. Variables collected include:
age at time of measurement (years)fev (liters per second)height at time of measurement (inches)sex (male/female)smoke (smoker/nonsmoker)head(fev) subjid age fev height sex smoke
1 301 9 1.708 57.0 male nonsmoker
2 451 8 1.724 67.5 male nonsmoker
3 501 7 1.720 54.5 male nonsmoker
4 642 9 1.558 53.0 female nonsmoker
5 901 9 1.895 57.0 female nonsmoker
6 1701 8 2.336 61.0 male nonsmoker
Suppose we wanted to understand the relationship between height and lung function. We could do this by fitting a simple linear regression model
\[E[\text{FEV} \mid \text{height}] = \beta_0 + \beta_1 \text{height},\]
where fev is our outcome and height is our explanatory variable.
We can fit this model in R by running the code below:
model1 <- lm(fev ~ height, data = fev)
coef(model1)(Intercept) height
-5.4326788 0.1319756
As you learned in STAT 155 and STAT 253, R is using least squares estimation to estimate the intercept (\(\hat\beta_0\) = -5.43) and slope (\(\hat\beta_1\) = 0.13).
In the rest of this activity, you’ll derive the least squares estimates by hand!
In fitting this simple linear regression model, our goal is to find the line that provides the “best” fit to our data.
More specifically, we hope to find the values of the intercept \(\beta_0\) and slope \(\beta_1\) that minimize the sum of squared residuals:
\[\sum_{i=1}^n r_i^2 = \sum_{i=1}^n (y_i - \hat{y}_i)^2 = \sum_{i=1}^n (y_i - [\beta_0 + \beta_1 x_i])^2\]
In order to find the least squares estimates for the intercept and slope of our simple linear regression model, we need to minimize a function (the sum of squared residuals).
To do this, we can use techniques from multivariable calculus!
If we want to find the value of \(x\) that minimizes the function \(f(x)\), then we need to…
take the derivative \(f'(x) = \frac{d}{dx} f(x)\)
set the derivative equal to zero and solve for \(x\)
find the second derivative \(f''(x)\), evaluate at the solution to step #2, and check if it’s positive in order to confirm we have a minimum
(we should also check the endpoints!)
To find the least squares estimates \(\hat\beta_0, \hat\beta_1\), we start by finding the partial derivatives \(\frac{\partial}{\partial\beta_0}\) and \(\frac{\partial}{\partial\beta_1}\) of the sum of squared residuals.
\[\frac{\partial}{\partial\beta_0} \sum_{i=1}^n (y_i - [\beta_0 + \beta_1 x_i])^2 = ??\]
Find some space at the whiteboards. You’ll work on the rest of these exercises in small groups from the boards.
\[\begin{align*} \frac{\partial}{\partial\beta_0} \sum_{i=1}^n (y_i - [\beta_0 + \beta_1 x_i])^2 &\stackrel{set}{=} 0 \\ 2n(\beta_0 + \beta_1 \bar{x} - \bar{y}) & = 0, \text{ plugging in the partial derivative from above} \\ \beta_0 + \beta_1 \bar{x} - \bar{y} & = 0 \\ \beta_0 &= \bar{y} - \beta_1 \bar{x} \end{align*}\]
This is as far as we can simplify for now. Let’s switch to looking at the other partial derivative:
\[\begin{align*} \frac{\partial}{\partial\beta_1} \sum_{i=1}^n (y_i - [\beta_0 + \beta_1 x_i])^2 &\stackrel{set}{=} 0 \\ 2(n\beta_0 \bar{x} + \beta_1 \sum_{i=1}^n x_i^2 -\sum_{i=1}^n y_i x_i) &= 0, \text{ plugging in the partial derivative from above} \\ n\beta_0 \bar{x} + \beta_1 \sum_{i=1}^n x_i^2 -\sum_{i=1}^n y_i x_i &= 0 \\ n(\bar{y} - \beta_1 \bar{x}) \bar{x} + \beta_1 \sum_{i=1}^n x_i^2 -\sum_{i=1}^n y_i x_i &= 0, \text{ plugging in } \beta_0 = \bar{y} - \beta_1 \bar{x} \text{ from above} \\ n\bar{y}\bar{x} - n\beta_1 \bar{x}^2 + \beta_1 \sum_{i=1}^n x_i^2 -\sum_{i=1}^n y_i x_i &= 0 \\ n\bar{y}\bar{x} + \beta_1\left(\sum_{i=1}^n x_i^2 - n\bar{x}^2\right) -\sum_{i=1}^n y_i x_i &= 0 \\ \beta_1\left(\sum_{i=1}^n x_i^2 - n\bar{x}^2\right) &= \sum_{i=1}^n y_i x_i - n\bar{y}\bar{x} \\ \implies \hat\beta_1 & = \frac{\sum_{i=1}^n y_i x_i - n\bar{y}\bar{x}}{\sum_{i=1}^n x_i^2 - n\bar{x}^2} \\ \end{align*}\]
From here, we can plug our slope estimate \(\hat\beta_1\) back into the equation for \(\beta_0\) above to get our estimate for the intercept \[\hat\beta_0 = \bar{y} - \hat\beta_1 \bar{x}\]
Done with Exercises 1–3? Take a picture of your board work and post it on the #board-work channel on Slack. Head back to your table for the last two questions.
lm.Here’s one way to do it (among many!):
# set up
y <- fev$fev
x <- fev$height
n <- length(y)
# enter formula to get beta1hat by hand
beta1hat <- (sum(y*x) - n*mean(y)*mean(x))/(sum(x^2) - n * mean(x)^2)
print(beta1hat)[1] 0.1319756# enter formula to get beta0hat by hand
beta0hat <- mean(y) - beta1hat * mean(x)
print(beta0hat)[1] -5.432679We get the same answer! You now know how to derive the estimate of the intercept and slope of a simple linear regression model by hand :)
Scavenger Hunt. Working with your table, take a few minutes to find all of the following. Bookmark any important sites.
If you didn’t finish the exercises during class time today, no problem! You won’t turn these in. Just be sure to complete them outside of class, review the solutions, and ask any outstanding questions on Slack or in office hours.
This class builds on concepts and techniques from previous courses (calc, linear algebra, probability, 155)
Least squares is just one of many techniques for estimating unknown parameters