class: center, middle, inverse, title-slide .title[ # Classification ] .subtitle[ ## Part 1a ] .author[ ### Prof. Ryan Weldzius ] .institute[ ### Villanova University ] --- <style type="text/css"> .small .remark-code { /*Change made here*/ font-size: 85% !important; } .tiny .remark-code { /*Change made here*/ font-size: 50% !important; } </style> # Agenda 1. Classification 2. Fortnite gaming (i.e., Prof's desperate attempt to be relevant) ``` r require(tidyverse) fn <- read_rds('../Data/fn_cleaned_final.rds') ``` --- # Definitions - *Classification:* predicting the **class** of given data points via **predictive modeling** -- - *Class*: AKA targets, labels, or **categories** -- - *Predictive Modeling*: Approximate mapping function `\(f: X \rightarrow Y\)` -- - `\(X\)`: predictor variables - `\(Y\)`: outcome variable - `\(f\)`: ?? --- # Mapping Functions - We have already used mapping functions! -- - Linear Regression -- - `\(f\)`: `\(Y = \alpha + \beta X + \varepsilon\)` -- - Underlying idea: `\(X\)` contain information about `\(Y\)` --- # It is in the `\(Y\)` - If `\(Y\)` is continuous, we use OLS (ordinary least squares) regression -- - If `\(Y\)` is **binary**, we use "logistic" regression (AKA "logit") -- - As always, this is a **deep** area of study for those interested -- - Today, using OLS for binary `\(Y\)` -- - Next few classes: replacing OLS regression with logit --- # Fortnite <center><img src="https://cdn2.unrealengine.com/blade-2560x1440-2560x1440-d4e556fb8166.jpg" width="80%"></center> --- # Fortnite - Goal is to win (i.e., be the last player alive) -- - Professional e-sports teams want to maximize this probability -- - .blue[Research Question]: How can we increase the number of victories? -- - **NB**: we are moving out of the **.blue[Research]** camp now, and into the **.red[Prediction]** world -- - We don't care so much about *why* a relationship exists, we just want to get accurate predictions -- - Theory can still help us, but want to start with the data to get our thinking started --- # The Data ``` r glimpse(fn) ``` ``` ## Rows: 957 ## Columns: 24 ## $ placed <dbl> 17, 41, 36, 28, 3, 15, 9, 29,… ## $ mental_state <chr> "sober", "sober", "high", "hi… ## $ eliminations <dbl> 2, 0, 3, 1, 3, 0, 2, 3, 4, 1,… ## $ assists <dbl> 0, 2, 0, 4, 2, 1, 2, 2, 0, 2,… ## $ revives <dbl> 0, 0, 0, 0, 1, 0, 0, 0, 1, 0,… ## $ accuracy <dbl> 0.19371429, 0.32400265, 0.336… ## $ hits <dbl> 10, 17, 38, 22, 49, 4, 43, 14… ## $ head_shots <dbl> 1, 0, 0, 3, 18, 3, 2, 3, 13, … ## $ distance_traveled <dbl> 226, 370, 725, 266, 938, 148,… ## $ materials_gathered <dbl> 0, 0, 0, 358, 305, 0, 1286, 1… ## $ materials_used <dbl> 0, 38, 0, 61, 234, 170, 195, … ## $ damage_taken <dbl> 282, 203, 206, 262, 437, 151,… ## $ damage_to_players <dbl> 372, 354, 206, 286, 823, 122,… ## $ damage_to_structures <dbl> 538, 1403, 260, 3841, 1470, 4… ## $ won <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,… ## $ player <int> -5, -5, -5, -5, -5, -5, -5, -… ## $ gameId <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10… ## $ startTime <dttm> 2020-04-10 16:46:06, 2020-04… ## $ sessionId <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,… ## $ lagSess <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,… ## $ delta <dbl> 928.2763, 212.1983, 529.5711,… ## $ startTime2 <dttm> NA, 2020-04-10 17:05:06, 202… ## $ gameIdSession <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10… ## $ gameIdSession2 <dbl> 1, 4, 9, 16, 25, 36, 49, 64, … ``` --- # The Data - Start with the basics: -- 1. What is the unit of analysis? 2. Which variables are we interested in? --- # Prediction `$$Y = \alpha + \beta_1 X_1 + \beta_2 X_2 + \dots + \varepsilon$$` -- - `\(Y\)`: victory (`won`) -- - `\(X\)`: ?? -- - In prediction, we don't care about **theory** or **research questions** - Just want to maximize **accuracy**...which `\(X\)`'s are the "best"? - But theory can still help us make sensible choices about which `\(X\)`'s to use -- - Look at univariate & conditional relationships --- # The Data - Outcome `\(Y\)`: `won` ``` r require(scales) fn %>% summarise(`Win %` = percent(mean(won))) ``` ``` ## # A tibble: 1 × 1 ## `Win %` ## <chr> ## 1 30% ``` -- - Multivariate analysis? --- # Which `\(X\)`? ``` r fn %>% group_by(mental_state) %>% summarise(pr_win = mean(won)) ``` ``` ## # A tibble: 2 × 2 ## mental_state pr_win ## <chr> <dbl> ## 1 high 0.234 ## 2 sober 0.370 ``` --- # Which `\(X\)`? ``` r fn %>% group_by(gameIdSession) %>% summarise(pr_win = mean(won)) ``` ``` ## # A tibble: 44 × 2 ## gameIdSession pr_win ## <int> <dbl> ## 1 1 0.0588 ## 2 2 0.0588 ## 3 3 0.206 ## 4 4 0.147 ## 5 5 0.147 ## 6 6 0.0588 ## 7 7 0.206 ## 8 8 0.265 ## 9 9 0.412 ## 10 10 0.618 ## # ℹ 34 more rows ``` --- # Which `\(X\)`? ``` r fn %>% group_by(gameIdSession) %>% summarise(pr_win = mean(won)) %>% ggplot(aes(x = gameIdSession,y = pr_win)) + geom_point() ``` <img src="12a_ClassificationPart1_files/figure-html/unnamed-chunk-8-1.png" style="display: block; margin: auto;" /> --- # Which `\(X\)`? ``` r fn %>% ggplot(aes(x = hits,y = won)) + geom_point() ``` <img src="12a_ClassificationPart1_files/figure-html/unnamed-chunk-9-1.png" style="display: block; margin: auto;" /> --- # Which `\(X\)`? ``` r fn %>% ggplot(aes(x = hits,y = won)) + geom_jitter() ``` <img src="12a_ClassificationPart1_files/figure-html/unnamed-chunk-10-1.png" style="display: block; margin: auto;" /> --- # Heatmaps - Look at 3-dimensions of data -- - Done this before by tweaking `fill`, `color`, or `size` -- - `geom_tile()`: create a heatmap ``` r p <- fn %>% mutate(accuracy_decile = ntile(hits,n=10)) %>% # Bin hits by decile (10%) group_by(accuracy_decile,mental_state) %>% # Calculate average winning by mental state and accuracy summarise(pr_win = mean(won), .groups = 'drop') %>% ggplot(aes(x = factor(mental_state), y = factor(accuracy_decile), # Both x and y-axes are factors fill = pr_win)) + # Fill by third dimension geom_tile() + # Creates rectangles scale_fill_gradient(limits = c(0,1)) # Set fill color (can do much more here) ``` --- # Heatmaps ``` r p ``` <img src="12a_ClassificationPart1_files/figure-html/unnamed-chunk-12-1.png" style="display: block; margin: auto;" /> --- # Simplest Predictions - Remember: regression is just fancier conditional means ``` r fn <- fn %>% mutate(hits_decile = ntile(hits,n=10)) %>% # Bin hits by decile (10%) group_by(hits_decile,mental_state) %>% # Calculate average winning by mental state and accuracy mutate(prob_win = mean(won)) %>% # use mutate() instead of summarise() to avoid collapsing the data mutate(pred_win = ifelse(prob_win > .5,1,0)) %>% # If the probability is greater than 50-50, predict a win ungroup() ``` --- # Simplest Predictions - Conditional means ``` r fn %>% group_by(won,pred_win) %>% summarise(nGames=n(),.groups = 'drop') ``` ``` ## # A tibble: 4 × 3 ## won pred_win nGames ## <dbl> <dbl> <int> ## 1 0 0 625 ## 2 0 1 41 ## 3 1 0 241 ## 4 1 1 50 ``` -- - How good is this? Think about the underlying goal...we want a model that accurately predicts whether a game is won or not - The `won` column is the **truth**...it tells us whether the game was won or not - The `pred_win` column is our **prediction** --- # Accuracy - What is "accuracy"? -- - Proportion "correct" predictions -- - For a binary outcome, "accuracy" has two dimensions -- - Proportion of correct `1`s: **Sensitivity** - Proportion of correct `0`s: **Specificity** --- # Accuracy ``` r (sumTab <- fn %>% group_by(won) %>% mutate(total_games = n()) %>% group_by(won,pred_win,total_games) %>% summarise(nGames=n(),.groups = 'drop') %>% mutate(prop = nGames / total_games)) ``` ``` ## # A tibble: 4 × 5 ## won pred_win total_games nGames prop ## <dbl> <dbl> <int> <int> <dbl> ## 1 0 0 666 625 0.938 ## 2 0 1 666 41 0.0616 ## 3 1 0 291 241 0.828 ## 4 1 1 291 50 0.172 ``` -- - Overall accuracy: (625+50) / (666+291) = 71% - But we are doing **great** at predicting losses (94%)... - ...and **terribly** at predicting wins (17%) --- # Conclusion - `Classification` is just a type of prediction - `Accuracy` measures the proportion of "correct" predictions