--- title: "Model Evaluation Experiment (Notes)" subtitle: "Stat 253" author: "Your Name" format: html: toc: true toc-depth: 2 embed-resources: true --- ```{r setup} #| include: false #| eval: true knitr::opts_chunk$set( collapse = TRUE, warning = FALSE, message = FALSE, error = TRUE, fig.height = 2.75, fig.width = 4.25, fig.env='figure', fig.pos = 'h', fig.align = 'center') ``` # Small Group Discussion {-} To start class today, we're going to do a Model Evaluation Experiment! First, we'll review some model evaluation ideas from last class. \ ## Directions {.unnumbered .smaller} Let's *build* and *evaluate* a predicted model of an adult's height ($y$) using some predictors $x_i$ (e.g., age, weight, etc.). - Introduce yourself in whatever way you feel appropriate and check in with each other as human beings - Come up with a team name - Work through the steps below as a group - Each group will be given a different sample of 40 adults - Start by predicting `height` (in) using `hip` circumference (cm) - Evaluate the model on your sample. - **Be prepared to share your answers to:** - How good is your simple model? - What would happen if we added more predictors? \ ## Questions {.unnumbered .smaller} ![](https://mac-stat.github.io/STAT253/images/MLdiagram1.jpg){width=100%} **Goal** - Let's *build* and *evaluate* a predictive model of an adult's height ($y$) using some predictors $x_i$ (eg: age, height, etc). - Since $y$ is *quantitative* this is a **regression task**. - There are countless possible models of $y$ vs $x$. We'll utilize a **linear regression model**: $$y = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p + \varepsilon$$ - And after building this model, we'll **evaluate** it. \ **Data:** Each group will be given a different sample of 40 adults. ```{r packages} # Load packages needed for this analysis library(tidyverse) library(tidymodels) ``` Using the following starter code, fill in the blank in the URL with the appropriate number depending on your group number. - Group 1: 50 - Group 2: 143 - Group 3: 86 - Group 4: 160 - Groups 5 & 6: 174 **REMINDER:** _Do not edit the starter code directly. Instead, copy-paste the code into an empty code chunk below. Then, make edits (eg fill in blanks) in that second code chunk._ ```{r data-starter} #| eval: false # Load data humans <- read.csv("https://Mac-Stat.github.io/data/bodyfat___.csv") %>% filter(ankle < 30) %>% rename(body_fat = fatSiri) ``` ```{r data-yourturn} ``` \ Check out a density plot of your outcome: ```{r plot-starter} #| eval: false # Plot data ggplot(humans, aes(x = ___)) + geom____() ``` ```{r plot-yourturn} ``` \ **Model building:** Build a linear regression model of `height` (in) by `hip` circumference (cm). ```{r step1-starter} #| eval: false # STEP 1: model specification lm_spec <- ___() %>% set_mode(___) %>% set_engine(___) ``` ```{r step1-yourturn} ``` ```{r step2-starter} #| eval: false # STEP 2: model estimation model_1 <- ___ %>% ___(height ~ hip, data = humans) ``` ```{r step2-yourturn} ``` ```{r step3-coef} # Check out the coefficients # Do all groups have the same coefficients? Should they? ``` \ **Model evaluation:** How *good* is our model? ```{r calculate-r2} # Calculate the R^2 for model_1 ``` ```{r predict-starter} #| eval: false # Use your model to predict height for your subjects # Just print the first 6 results model_1 %>% ___(new_data = ___) %>% head() ``` ```{r predict-yourturn} ``` ```{r mae-starter} #| eval: false # Calculate the MAE, i.e. typical prediction error, for your model model_1 %>% augment(new_data = humans) %>% ___(truth = ___, estimate = ___) ``` ```{r mae-yourturn} ``` \ **Reflection** In addition to `hip` circumference, suppose we incorporated more predictors into our model of `height`. What would happen to $R^2$? To the MAE? \ **Prepare to discuss:** - Make sure everyone in your group is on the same page regarding your answers to the questions above. - Send one group member to write your model's MAE and $R^2$ on the board. - Pick another group member to share your group's answers, questions, etc. during our full class discussion. \ \ 🛑 **STOP HERE.** Don't move on to the next section until we've discussed these questions as a class. *Still waiting?* Discuss the following BONUS questions with your group: - What other model evaluation techniques could you use to evaluate how "good" this simple model is that you didn't use above? - Try implementing those techniques. ```{r bonus} ``` \ \ # Exercises (Part 1) {-} ## Directions {.unnumbered .smaller} - Take **5 minutes** to complete exercises 1 and 2 (choosing one of three models). - We'll pause for a few minutes to discuss each group's answers to these exercises. - Then, and only then, you can finish exercises 3 - 5. REMINDERS: - Be kind to yourself/each other. You will make mistakes! - Collaborate: - actively contribute to discussion (don't work on your own) - actively include all group members in discussion - create a space where others feel comfortable making mistakes and sharing their ideas - stay in sync \ ## Questions {.unnumbered .smaller} 1. **Select a model** Consider 3 different models of `height`, estimated below. As a group, use *your* data to choose which is the best predictive model of `height`. Calculate the MAE for this model. ```{r fit-models} # height vs hip model_1 <- lm_spec %>% fit(height ~ hip, data = humans) model_1 %>% tidy() # height vs hip & weight model_2 <- lm_spec %>% fit(height ~ hip + weight, data = humans) model_2 %>% tidy() # height vs a lot of predictors (AND some interaction terms) model_3 <- lm_spec %>% fit(height ~ chest * age * weight * body_fat * abdomen + hip + thigh + knee + ankle + biceps + forearm + wrist, data = humans) model_3 %>% tidy() ``` ```{r} # Calculate the MAE for your model ___ %>% augment(new_data = humans) %>% mae(truth = height, estimate = .pred) ``` \ 2. **Share your results** Only when you're done with exercise 1: - Open this ["Top Model Competition" Google Doc](https://docs.google.com/spreadsheets/d/1dWCdwXNjNvrD3-aADaMQxh87euDePvKK1dROSMycp_U/edit?usp=sharing). - Find the tab corresponding to your section. - Record your team name in the appropriate row. (`GROUP NAME`) - Record which model you chose (1, 2, or 3). (`CHOSEN MODEL`) - Record the MAE for your model. (`MAE: YOUR SAMPLE`) \ 🛑 **WAIT.** Don't keep going. \ \ \ \ \ \ **Don't peek** What do you know?! 40 new people just walked into the doctor's office and the doctor wants to predict their `height`: ```{r} # Import the new data new_patients <- read.csv("https://Mac-Stat.github.io/data/bodyfat182.csv") %>% filter(ankle < 30) %>% rename(body_fat = fatSiri) ``` \ \ 3. **Intuition** Consider using *your* model to predict `height` for these 40 *new* subjects. On average, do you think these predictions will be _better_ or _worse_ than for your original patients? Why? \ \ 4. **How well does your model do in the real world?** Use *your* model to predict `height` for the *new* patients and calculate the typical prediction error (MAE). **Record this in the Google sheet.** (`MAE: NEW PATIENTS`) ```{r} ___ %>% augment(new_data = new_patients) %>% mae(truth = height, estimate = .pred) ``` \ \ 5. **Reflection** In summary, which model seems best? What's the central theme here? \ \ **Now open and save the QMD for Part 2. ** \ \ # Done! - Render your notes. - Check the solutions in the course website (Solution drop downs). - If you finish all that during class, start your homework!