--- title: "Overfitting (Notes)" subtitle: "Stat 253" author: "Your Name" format: html: toc: true toc-depth: 2 embed-resources: true --- ```{r part2-setup} #| include: false #| eval: true knitr::opts_chunk$set( collapse = TRUE, warning = FALSE, message = FALSE, error = TRUE, fig.height = 2.75, fig.width = 4.25, fig.env='figure', fig.pos = 'h', fig.align = 'center') ``` # Learning Goals {-} - Explain why training/in-sample model evaluation metrics can provide a misleading view of true test/out-of-sample performance - Implement testing and training sets in R using the `tidymodels` package \ \ # Notes {-} ## Overfitting {.unnumbered .smaller} When we add more and more predictors into a model, it can become *overfit* to the noise in our sample data: - our model loses the broader trend / big picture - thus does not generalize to *new* data - thus results in bad predictions and a bad understanding of the relationship among the new data points \ **Preventing overfitting: training and testing** - **In-sample** metrics, i.e. measures of how well the model performs on the same sample data that we used to build it, tend to be overly optimistic and lead to overfitting. - Instead, we should *build* and *evaluate*, or **train** and **test**, our model using different data. \ \ ## R Code {.unnumbered .smaller} :::{.callout-note} This section is for **future** reference. It is a summary of code you'll learn below for creating and applying training and testing data. You don't need to (and in fact should not) run the code in this section---just use this as example code for future reference. ::: Suppose we wish to build and evaluate a linear regression model of `y` vs `x1` and `x2` using our `sample_data`. ```{r eval = FALSE} # Load packages library(tidyverse) library(tidymodels) ``` **Split the sample data into training and test sets** ```{r eval = FALSE} # Set the random number seed set.seed(___) # Split the sample_data # "prop" is the proportion of data assigned to the training set # it must be some number between 0 and 1 data_split <- initial_split(sample_data, strata = y, prop = ___) # Get the training data from the split data_train <- data_split %>% training() # Get the testing data from the split data_test <- data_split %>% testing() ``` **Build a training model** ```{r eval = FALSE} # STEP 1: model specification lm_spec <- linear_reg() %>% set_mode("regression") %>% set_engine("lm") # STEP 2: model estimation using the training data model_train <- lm_spec %>% fit(y ~ x1 + x2, data = data_train) ``` **Use the training model to make predictions for the test data** ```{r eval = FALSE} # Make predictions model_train %>% augment(new_data = data_test) ``` **Evaluate the training model using the test data** ```{r eval = FALSE} # Calculate the test MAE model_train %>% augment(new_data = data_test) %>% mae(truth = y, estimate = .pred) ``` \ \ # Exercises (Part 2) {-} ## Directions {-} - Open the Part 2 QMD file - Same directions as before: - Be kind to yourself/each other - Collaborate - We will not discuss these exercises as a class. Be sure to ask questions as I walk around the room! \ ## Questions {.unnumbered .smaller} The following exercises are inspired by Chapter 5.3.1 of ISLR. ```{r} # Load packages # NOTE: You might first need to install the ISLR package library(tidyverse) library(tidymodels) library(ISLR) # Load data data(Auto) # Select three variables from Auto data # Save as a data frame called "cars" cars <- Auto %>% dplyr::select(mpg, horsepower, year) ``` \ Let's use the `cars` data to compare three **linear regression models** of fuel efficiency in miles per gallon (`mpg`) by engine power (`horsepower`): ```{r} # Raw data cars_plot <- ggplot(cars, aes(x = horsepower, y = mpg)) + geom_point() cars_plot ``` ```{r} # model 1: 1 predictor (y = b0 + b1 x) cars_plot + geom_smooth(method = "lm", se = FALSE) ``` ```{r} # model 2: 2 predictors (y = b0 + b1 x + b2 x^2) cars_plot + geom_smooth(method = "lm", se = FALSE, formula = y ~ poly(x, 2)) ``` ```{r} # model 3: 19 predictors (y = b0 + b1 x + b2 x^2 + ... + b19 x^19) cars_plot + geom_smooth(method = "lm", se = FALSE, formula = y ~ poly(x, 19)) ``` \ **Goal** Let's evaluate and compare these models by **training** and **testing** them using different data. \ 1. **155 review: set.seed()** Run the two chunks below *multiple* times each. Afterward, summarize what `set.seed()` does and why it's important to being able to *reproduce* a random sample. ```{r} sample_n(cars, 2) ``` ```{r} set.seed(253) sample_n(cars, 2) ``` \ 2. **Training and test sets** Let's *randomly* split our original 392 sample cars into two separate pieces: select 80% of the cars to **train (build)** the model and the other 20% to **test (evaluate)** the model. ```{r} # Set the random number seed set.seed(8) # Split the cars data into 80% / 20% # Ensure that the sub-samples are similar with respect to mpg cars_split <- initial_split(cars, strata = mpg, prop = 0.8) ``` ```{r} # Check it out # What do these numbers mean? cars_split ``` ```{r} # Get the training data from the split cars_train <- cars_split %>% training() # Get the testing data from the split cars_test <- cars_split %>% testing() ``` ```{r} # The original data has 392 cars nrow(cars) # How many cars are in cars_train? # How many cars are in cars_test? ``` \ 3. **Reflect on the above code** a. Why do we want the training and testing data to be similar with respect to `mpg` (`strata = mpg`)? What if they *weren't*? b. Why did we need all this new code instead of just using the *first* 80% of cars in the sample for training and the *last* 20% for testing? \ 4. **Build the training model** ```{r eval = FALSE} # STEP 1: model specification lm_spec <- linear_reg() %>% set_mode("regression") %>% set_engine("lm") # STEP 2: model estimation using the training data # Construct the 19th order polynomial model using the TRAINING data model_19_train <- ___ %>% ___(mpg ~ poly(horsepower, 19), data = ___) ``` ```{r} ``` \ 5. **Evaluate the training model** ```{r eval = FALSE} # How well does the TRAINING model predict the TRAINING data? # Calculate the training (in-sample) MAE model_19_train %>% augment(new_data = ___) %>% mae(truth = mpg, estimate = .pred) ``` ```{r} ``` ```{r eval = FALSE} # How well does the TRAINING model predict the TEST data? # Calculate the test MAE model_19_train %>% augment(new_data = ___) %>% mae(truth = mpg, estimate = .pred) ``` ```{r} ``` \ 6. **Punchline** The table below summarizes your results for `train_model_19` as well as the other two models of interest. (You should confirm the other two model results outside of class!) Model Training MAE Testing MAE ---------------------------- ------------- ------------- `mpg ~ horsepower` 3.78 4.00 `mpg ~ poly(horsepower, 2)` 3.20 3.49 `mpg ~ poly(horsepower, 19)` 2.99 6.59 Answer the following and reflect on why each answer *makes sense*: a. *Within* each model, how do the training errors compare to the testing errors? (This isn't *always* the case, but is common.) b. What about the training and test errors for the third model suggest that it is *overfit* to our sample data? c. Which model seems the best with respect to the *training* errors? d. Which model is the best with respect to the *testing* errors? e. Which model would you choose? \ 7. **Final reflection** a. The training / testing procedure provided a more honest evaluation and comparison of our model predictions. How might we *improve* upon this procedure? What problems can you anticipate in splitting our data into 80% / 20%? b. Summarize the key themes from today in your own words. \ 8. **STAT 155 REVIEW: data drill** a. Construct and interpret a plot of `mpg` vs `horsepower` and `year`. b. Calculate the average `mpg`. c. Calculate the average `mpg` for each `year`. HINT: `group_by()` d. Plot the *average* `mpg` by `year`. \ 9. **Digging deeper (optional)** Check out the online solutions for exercise 6. Instead of calculating MAE from scratch for 3 different models, I wrote a function `calculate_MAE()` to automate the process. After picking through this code, adapt the function so that it also returns the $R^2$ value of each model. \ \ # Done! - Render your notes. - Check the solutions in the course website (Solution drop downs). - If you finish all that during class, start your homework!