16  Theory of Linear Models

Settling In

  • Check Slack for announcements



Review: Estimation Techniques

We can write a linear regression model as follows: \[\mathbf{y} = \mathbf{X}\boldsymbol\beta + \boldsymbol\epsilon,\]

where \[ \mathbf{y} = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix}, \ \mathbf{X} = \begin{bmatrix} 1 & x_{11} & x_{21} & \dots & x_{p1} \\ 1 & x_{12} & x_{22} & \dots & x_{p2} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_{1n} & x_{2n} & \dots & x_{pn} \end{bmatrix} \ \boldsymbol\beta = \begin{bmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_p \end{bmatrix}, \ \boldsymbol\epsilon = \begin{bmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_n \end{bmatrix}. \]


Earlier in the semester, we showed that we can estimate the regression coefficients \(\boldsymbol\beta\) using (ordinary) least squares estimation (OLS) or maximum likelihood estimation (MLE), in both cases yielding the estimator

\[\hat{\boldsymbol\beta}_{OLS} = \hat{\boldsymbol\beta}_{MLE} = \left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \mathbf{X}^\top \mathbf{y}.\]


How “good” is this estimator?



Properties of Least Squares Estimators

Bias

Assume that \(\mathbf{y} = \mathbf{X}\boldsymbol\beta + \boldsymbol\epsilon\) and \(E[\boldsymbol\epsilon] = \mathbf{0}\). Then, the least squares estimator \(\hat{\boldsymbol\beta}_{OLS} = \left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \mathbf{X}^\top \mathbf{y}\) is an unbiased estimator of \(\boldsymbol\beta\).

In other words, if we fit the “correct” model and our errors have mean zero, then the OLS estimator will be unbiased. No other assumptions (e.g. normality, independence, homoskedasticity) needed!

Proof See PS6!


What if we fit the wrong model, ie \(\mathbf{y} \neq \mathbf{X}\boldsymbol\beta + \boldsymbol\epsilon\)?

Proof See PS6!


Variance

Assume that \(\mathbf{y} = \mathbf{X}\boldsymbol\beta + \boldsymbol\epsilon\), where \(E[\boldsymbol\epsilon] = \mathbf{0}\) \(\text{Var}[\boldsymbol\epsilon] = \sigma^2 \mathbf{I}_n\). Then, the variance of the least squares estimator is \(\text{Var}[\hat{\boldsymbol\beta}] = \sigma^2\left( \mathbf{X}^\top \mathbf{X}\right)^{-1}\).

In other words, this is the variance of our OLS estimator if we fit the “correct” model and our errors are independent with mean zero and equal variance. No assumptions about normality needed!

Proof

\[ \begin{aligned} \text{Var}[\hat{\boldsymbol\beta}] & = \text{Var}\left[ \left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \mathbf{X}^\top \mathbf{y} \right] \\ & = \left[\left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \mathbf{X}^\top \right] \left[\text{Var}(\mathbf{y})\right] \left[\left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \mathbf{X}^\top\right]^\top, \\ & \qquad \qquad \text{ since } \text{Var}(\mathbf{A}\mathbf{x}) = \mathbf{A}\text{Var}(\mathbf{x}) \mathbf{A}^\top\\ & = \left[\left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \mathbf{X}^\top \right] \left[\text{Var}(\mathbf{X}\boldsymbol\beta + \boldsymbol\epsilon)\right] \left[\left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \mathbf{X}^\top\right]^\top, \\ & \qquad \qquad \text{ since } \mathbf{y} = \mathbf{X}\boldsymbol\beta + \boldsymbol\epsilon \text{ by assumption } \\ & = \left[\left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \mathbf{X}^\top \right] \left[\text{Var}(\boldsymbol\epsilon)\right] \left[\left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \mathbf{X}^\top\right]^\top, \\ & \qquad \qquad \text{ since } \mathbf{X}, \boldsymbol\beta \text{ are fixed } \\ & = \left[\left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \mathbf{X}^\top \right] \left[\sigma^2 \mathbf{I}_n\right] \left[\left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \mathbf{X}^\top\right]^\top, \\ & \qquad \qquad \text{ since } \text{Var}(\mathbf{y}) = \sigma^2 \mathbf{I}_n \text{ by assumption } \\ & = \sigma^2 \left[\left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \mathbf{X}^\top \right] \left[\left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \mathbf{X}^\top\right]^\top, \\ & = \sigma^2 \left[\left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \mathbf{X}^\top \right] \left[\mathbf{X} \left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \right], \\ & \qquad \qquad \text{ since } (\mathbf{A}\mathbf{B})^\top = \mathbf{B}^\top \mathbf{A}^\top \text{ and } \left(\mathbf{A}^{-1}\right)^\top = \left(\mathbf{A}^\top\right)^{-1} \\ & = \sigma^2 \left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \end{aligned} \]


What if \(\text{Var}[\boldsymbol\epsilon] \neq \sigma^2 \mathbf{I}_n\)?

Proof Take STAT 452: Correlated Data to find out!


Gauss-Markov Theorem

Assume that \(\mathbf{y} = \mathbf{X}\boldsymbol\beta + \boldsymbol\epsilon\), where \(E[\boldsymbol\epsilon] = \mathbf{0}\) \(\text{Var}[\boldsymbol\epsilon] = \sigma^2 \mathbf{I}_n\). Then, the least squares estimator \(\hat{\boldsymbol\beta}_{OLS} = \left(\mathbf{X}^\top \mathbf{X}\right)^{-1} \mathbf{X}^\top \mathbf{y}\) is the best linear unbiased estimator (BLUE).

In other words, compared to all other unbiased linear estimators of \(\boldsymbol\beta\), \(\hat{\boldsymbol\beta}_{OLS}\) has the smallest variance!

Proof

Let \(\hat{\boldsymbol\beta} = \mathbf{C}\mathbf{y}\) be another linear estimator of \(\boldsymbol\beta\), with \(\mathbf{C} = (\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top + \mathbf{D}\) for some non-zero matrix \(\mathbf{D}\).

Then,

\[ \begin{aligned} E[\hat{\boldsymbol\beta}] & = E[\mathbf{C}\mathbf{y}] \\ & = E[((\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top + \mathbf{D})\mathbf{y}] \\ & = E[(\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y} + \mathbf{D}\mathbf{y}] \\ & = E[(\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}] + E[\mathbf{D}\mathbf{y}] \\ & = \boldsymbol\beta + E[\mathbf{D}\mathbf{y}], \text{ from above} \\ & = \boldsymbol\beta + \mathbf{D}E[\mathbf{y}] \\ & = \boldsymbol\beta + \mathbf{D}E[\mathbf{X}\boldsymbol\beta + \boldsymbol\epsilon], \text{ since } \mathbf{y} = \mathbf{X}\boldsymbol\beta + \boldsymbol\epsilon \text{ by assumption} \\ & = \boldsymbol\beta + \mathbf{D}(\mathbf{X}\boldsymbol\beta + E[\boldsymbol\epsilon]) \\ & = \boldsymbol\beta + \mathbf{D}(\mathbf{X}\boldsymbol\beta + \mathbf{0}), \text{ since } E[\boldsymbol\epsilon] = \mathbf{0} \text{ by assumption} \\ & = \boldsymbol\beta + \mathbf{D}\mathbf{X}\boldsymbol\beta \\ & = (\mathbf{I} + \mathbf{D}\mathbf{X}) \boldsymbol\beta \end{aligned} \]

In order for \(\hat{\boldsymbol\beta}\) to be unbiased (and thus in the running to be BLUE), we need \(\mathbf{D}\mathbf{X} = \mathbf{0}\).

Next, let’s consider the variance:

\[ \begin{aligned} \text{Var}[\hat{\boldsymbol\beta}] & = \text{Var}[\mathbf{C}\mathbf{y}] \\ & = \mathbf{C} \text{Var}[\mathbf{y}] \mathbf{C}^\top \\ & = \mathbf{C} (\sigma^2 \mathbf{I}) \mathbf{C}^\top, \text{ since } \text{Var}[\mathbf{y}] = \sigma^2 \mathbf{I}, \text{ as shown above} \\ & = \sigma^2 \mathbf{C}\mathbf{C}^\top \\ & = \sigma^2 ((\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top + \mathbf{D}) ((\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top + \mathbf{D})^\top \\ & = \sigma^2 ((\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top + \mathbf{D}) (\mathbf{X}(\mathbf{X}^\top \mathbf{X})^{-1} + \mathbf{D}^\top) \\ & = \sigma^2 \left[(\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top \mathbf{X}(\mathbf{X}^\top \mathbf{X})^{-1} + (\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top \mathbf{D}^\top + \mathbf{D}\mathbf{X}(\mathbf{X}^\top \mathbf{X})^{-1} + \mathbf{D}\mathbf{D}^\top\right] \\ & = \sigma^2 \left[(\mathbf{X}^\top \mathbf{X})^{-1} + (\mathbf{X}^\top \mathbf{X})^{-1} (\mathbf{D} \mathbf{X})^\top + \mathbf{D}\mathbf{X}(\mathbf{X}^\top \mathbf{X})^{-1} + \mathbf{D}\mathbf{D}^\top\right] \end{aligned} \]

But, if \(\hat{\boldsymbol\beta}\) is unbiased, then \(\mathbf{D}\mathbf{X} = \mathbf{0}\) and the variance becomes:

\[ \begin{aligned} \text{Var}[\hat{\boldsymbol\beta}] & =\sigma^2 \left[(\mathbf{X}^\top \mathbf{X})^{-1} + (\mathbf{X}^\top \mathbf{X})^{-1} (\mathbf{D} \mathbf{X})^\top + \mathbf{D}\mathbf{X}(\mathbf{X}^\top \mathbf{X})^{-1} + \mathbf{D}\mathbf{D}^\top\right] \\ & = \sigma^2 \left[(\mathbf{X}^\top \mathbf{X})^{-1} + (\mathbf{X}^\top \mathbf{X})^{-1} (\mathbf{0})^\top + \mathbf{0}(\mathbf{X}^\top \mathbf{X})^{-1} + \mathbf{D}\mathbf{D}^\top\right] \\ & = \sigma^2 \left[(\mathbf{X}^\top \mathbf{X})^{-1} + \mathbf{D}\mathbf{D}^\top\right] \\ & = \sigma^2 (\mathbf{X}^\top \mathbf{X})^{-1} + \sigma^2\mathbf{D}\mathbf{D}^\top \\ & = \text{Var}(\hat{\boldsymbol\beta}_{OLS}) + \sigma^2\mathbf{D}\mathbf{D}^\top, \end{aligned} \]

which exceeds \(\text{Var}(\hat{\boldsymbol\beta}_{OLS})\) provided that \(\mathbf{D}\mathbf{D}^\top\) is positive semi-definite (which it is).

Thus, \(\hat{\boldsymbol\beta}_{OLS}\) is a best linear unbiased estimator of \(\boldsymbol\beta\).



Exercises

Find some whiteboard space and work on PS6 Problems #4 and #5.

NOTE: If \(\mathbf{x}_{n \times 1}\) is a vector of random variables, \(\mathbf{A}_{p \times n}\) is a matrix of constants, and \(\mathbf{b}_{n\times 1}\) is a vector of constants, then…

  • \(E[\mathbf{A}] = \mathbf{A}\)
  • \(E[\mathbf{b}] = \mathbf{b}\)
  • \(E[\mathbf{A}\mathbf{x}] = \mathbf{A} E[\mathbf{x}]\)
  • \(E[\mathbf{x} + \mathbf{b}] = E[\mathbf{x}] + \mathbf{b}\)



Wrapping Up