class: center, middle, inverse, title-slide .title[ # Intro to <code>R</code> (Part 3) ] .author[ ### Prof. Weldzius ] .institute[ ### Villanova University ] --- <style type="text/css"> .small .remark-code { /*Change made here*/ font-size: 85% !important; } .tiny .remark-code { /*Change made here*/ font-size: 50% !important; } </style> # Load the data again ``` r library(tidyverse) df<-read_rds("https://github.com/rweldzius/PSC4175/raw/main/static/data/sc_debt.Rds") ``` --- # Manipulating the Data -- - These data are cool! -- - But TMI at first -- - I want to know... -- - Where is `Villanova University`? -- - Who is the most selective? -- - Which schools produce the richest grads? -- - There are `tidyverse` functions to answer all of these questions --- # Manipulating with `tidyverse` - The code process of `tidyverse` relies on a "pipe" symbol: `%>%` -- - Think of it as a "chain" that **links code together** - Or maybe a "do" symbol because it tells `R` what to do -- - The basic grammar of `R` is: object, `%>%`, verb ``` r object %>% # This is the object function() # This is the verb ``` --- # Manipulating with `tidyverse` - `tidyverse` has many useful "verbs" (i.e., functions) - `filter()`: subsets **rows** - `select()`: subsets **columns** - `arrange()`: sorts **rows** based on **columns** - `summarise()`: collapses **rows** - `group_by()`: groups **rows** by **columns** --- # Manipulating: `filter()` - So let's look at Villanova -- - `filter` will select **rows** of the data based on some criteria -- ``` r df %>% filter(instnm == "Villanova University") # Only select rows with Villanova ``` ``` ## # A tibble: 1 × 16 ## unitid instnm stabbr grad_debt_mdn control region preddeg ## <int> <chr> <chr> <int> <chr> <chr> <chr> ## 1 216597 Villan… PA 26000 Private North… Bachel… ## # ℹ 9 more variables: openadmp <int>, adm_rate <dbl>, ## # ccbasic <int>, sat_avg <int>, md_earn_wne_p6 <int>, ## # ugds <int>, costt4_a <int>, selective <dbl>, ## # research_u <dbl> ``` --- # Manipulating: `select()` - Still TMI! -- - I only care about the admissions rate (`adm_rate`), the SAT scores (`sat_avg`), and the future earnings (`md_earn_wne_p6`) -- - `select` will select **columns** ``` r df %>% filter(instnm == "Villanova University") %>% select(instnm,adm_rate,sat_avg,md_earn_wne_p6) # Only select four columns ``` ``` ## # A tibble: 1 × 4 ## instnm adm_rate sat_avg md_earn_wne_p6 ## <chr> <dbl> <int> <int> ## 1 Villanova University 0.282 1422 62600 ``` --- # Manipulating: `arrange()` - How does Villanova compare...? -- - to other schools in terms of SAT scores? - to other schools in terms of future earnings? - to other schools in terms of admissions rates? -- - `arrange` will sort the data based on a column (ascending!) ``` r df %>% arrange(sat_avg) %>% # Sort data by SAT scores select(instnm,sat_avg) # Only look at name and SAT scores ``` ``` ## # A tibble: 2,546 × 2 ## instnm sat_avg ## <chr> <int> ## 1 Morgan State University 737 ## 2 Saint Augustine's University 847 ## 3 Albany State University 849 ## 4 Holy Names University 851 ## 5 Livingstone College 854 ## 6 Virginia Union University 855 ## 7 Manor College 861 ## 8 Saint Louis Christian College 865 ## 9 Bacone College 875 ## 10 Paine College 876 ## # ℹ 2,536 more rows ``` --- # Manipulating: `arrange()` - Villanova is not in the bottom 10 schools ``` r df %>% arrange(sat_avg) %>% # Sort data by SAT scores select(instnm,sat_avg) # Only look at name and SAT scores ``` ``` ## # A tibble: 2,546 × 2 ## instnm sat_avg ## <chr> <int> ## 1 Morgan State University 737 ## 2 Saint Augustine's University 847 ## 3 Albany State University 849 ## 4 Holy Names University 851 ## 5 Livingstone College 854 ## 6 Virginia Union University 855 ## 7 Manor College 861 ## 8 Saint Louis Christian College 865 ## 9 Bacone College 875 ## 10 Paine College 876 ## # ℹ 2,536 more rows ``` --- # Manipulating: `arrange()` - Use `desc()` to order in descending Villanova not in top 10 either ``` r df %>% arrange(desc(sat_avg)) %>% # Sort data by SAT scores (descending) select(instnm,sat_avg) # Only look at name and SAT scores ``` ``` ## # A tibble: 2,546 × 2 ## instnm sat_avg ## <chr> <int> ## 1 California Institute of Technology 1557 ## 2 Massachusetts Institute of Technology 1547 ## 3 University of Chicago 1528 ## 4 Harvey Mudd College 1526 ## 5 Duke University 1522 ## 6 Franklin W Olin College of Engineering 1522 ## 7 Washington University in St Louis 1520 ## 8 Rice University 1520 ## 9 Yale University 1517 ## 10 Harvard University 1517 ## # ℹ 2,536 more rows ``` --- # Manipulating: `arrange()` - What if we look only at "selective" schools (i.e., those who accept less than 10% of applicants)? -- ``` r df %>% filter(adm_rate < .1) %>% # Only look at schools who accept less than 10% arrange(sat_avg,adm_rate) %>% # Sort data by SAT scores AND THEN admissions rates (breaks ties) select(instnm,adm_rate,sat_avg) # Only look at name, admissions rate, and SAT scores ``` ``` ## # A tibble: 25 × 3 ## instnm adm_rate sat_avg ## <chr> <dbl> <int> ## 1 Colby College 0.0967 1456 ## 2 Swarthmore College 0.0893 1469 ## 3 Pomona College 0.074 1480 ## 4 Dartmouth College 0.0793 1500 ## 5 Stanford University 0.0434 1503 ## 6 Northwestern University 0.0905 1506 ## 7 Columbia University in the City of New … 0.0545 1511 ## 8 Brown University 0.0707 1511 ## 9 University of Pennsylvania 0.0766 1511 ## 10 Vanderbilt University 0.0912 1515 ## # ℹ 15 more rows ``` --- # How does Villanova compare? -- - `arrange` in descending order ``` r df %>% filter(adm_rate < .1) %>% arrange(desc(sat_avg),adm_rate) %>% select(instnm,adm_rate,sat_avg) ``` ``` ## # A tibble: 25 × 3 ## instnm adm_rate sat_avg ## <chr> <dbl> <int> ## 1 California Institute of Technology 0.0642 1557 ## 2 Massachusetts Institute of Technology 0.067 1547 ## 3 University of Chicago 0.0617 1528 ## 4 Duke University 0.076 1522 ## 5 Rice University 0.0872 1520 ## 6 Harvard University 0.0464 1517 ## 7 Princeton University 0.0578 1517 ## 8 Yale University 0.0608 1517 ## 9 Vanderbilt University 0.0912 1515 ## 10 Columbia University in the City of New … 0.0545 1511 ## # ℹ 15 more rows ``` --- # More complicated? More `%>%`! -- - Less selective schools by SAT with debt and state ``` r df %>% filter(adm_rate > .2 & adm_rate < .3) %>% # Less selective schools (accept between 20% and 30%) arrange(stabbr,desc(sat_avg)) %>% # Sort by state name, then by SAT scores select(instnm,sat_avg,grad_debt_mdn,stabbr) # Only look at some columns ``` ``` ## # A tibble: 37 × 4 ## instnm sat_avg grad_debt_mdn stabbr ## <chr> <int> <int> <chr> ## 1 Heritage Christian Universi… NA NA AL ## 2 University of California-Sa… 1370 15000 CA ## 3 California Polytechnic Stat… 1342 19501 CA ## 4 University of California-Ir… 1306 15488 CA ## 5 California Institute of the… NA 27000 CA ## 6 University of Miami 1371 17125 FL ## 7 Georgia Institute of Techno… 1418 23000 GA ## 8 Point University 986 26000 GA ## 9 Grinnell College 1457 17500 IA ## 10 St Luke's College NA 17750 IA ## # ℹ 27 more rows ``` --- # A quick aside on missingness -- - Some rows have `NA` in some columns -- - `NA` is the standard code for **missing data** in `R` - Data can be missing for many different reasons (i.e., some schools don't require SAT scores or record them) - We can use a base `R` function called `is.na()` which will be `TRUE` if the value is `NA` or `FALSE` otherwise -- - And we can combine `is.na()` with the `filter()` function from `tidyverse` -- - We will return to this in the lectures on **data wrangling** -- - For now, how many schools don't report SAT scores? ``` r df %>% filter(is.na(sat_avg)) %>% # Only look at schools that DON'T report SATs select(instnm,stabbr) # Only look at the name and the state ``` ``` ## # A tibble: 1,317 × 2 ## instnm stabbr ## <chr> <chr> ## 1 Amridge University AL ## 2 Central Alabama Community College AL ## 3 Athens State University AL ## 4 Chattahoochee Valley Community College AL ## 5 Coastal Alabama Community College AL ## 6 Gadsden State Community College AL ## 7 George C Wallace State Community College-Selma AL ## 8 Heritage Christian University AL ## 9 Jefferson State Community College AL ## 10 Lurleen B Wallace Community College AL ## # ℹ 1,307 more rows ``` --- # Stepping back -- - Thus far, lots of .red[data] -- - Not a lot of .blue[science] -- - But remember the .blue[Research] camp! -- 1. .red[Observation] &rarr; .blue[Question] 2. .blue[Theory] &rarr; .blue[Hypothesis] 3. .red[Data Collection / Wrangling] &rarr; .red[Analysis] 4. .red[Results] &rarr; .blue[Conclusion] -- - We have been doing lots of .red[Observation]! -- - Do we have any good .blue[Research questions]? --- # Stepping back - .blue[RQ]: How might admissions and SAT scores be **related**? -- - .blue[Theory]: selective schools have stricter criteria -- - .blue[Hypothesis]: admissions and SAT scores should be **negatively** related -- - How can we test this hypothesis? --- # Summarizing Data: `summarise()` + `mean()` -- - We can combine base `R` functions with `tidyverse` functions! -- - Base `R`: `mean()` - `tidyverse`: `summarise()` (aka `summarize()`) - Overall average SAT scores ``` r df %>% summarise(mean_sat = mean(sat_avg,na.rm=T)) # Average SAT scores for entire data ``` ``` ## # A tibble: 1 × 1 ## mean_sat ## <dbl> ## 1 1141. ``` --- # Summarizing Data -- - Let's unpack this ``` r df %>% summarise(mean_sat = mean(sat_avg,na.rm=T)) ``` -- - Create new variable `mean_sat` that contains the `mean()` of every school's average SAT score -- - `na.rm=T` means we want to ignore missing data. If not? -- ``` r df %>% summarise(mean_sat = mean(sat_avg)) ``` ``` ## # A tibble: 1 × 1 ## mean_sat ## <dbl> ## 1 NA ``` --- # Summarizing Data -- - Recall we want see if more selective schools have higher SAT scores -- ``` r df %>% filter(adm_rate < .1) %>% # Only look at schools who accept less than 10% of applicants summarise(mean_sat_LT10 = mean(sat_avg,na.rm=T)) # Calculate the average SAT score ``` ``` ## # A tibble: 1 × 1 ## mean_sat_LT10 ## <dbl> ## 1 1510. ``` ``` r df %>% filter(adm_rate > .1) %>% # Only look at schools who accept more than 10% of applicants summarise(mean_sat_GT10 = mean(sat_avg,na.rm=T)) # Calculate the average SAT score ``` ``` ## # A tibble: 1 × 1 ## mean_sat_GT10 ## <dbl> ## 1 1135. ``` --- # Summarizing Data: `group_by()` -- - One final `tidyverse` function: `group_by()` -- - There is a column called `selective` which is either 1 or 0 -- - 1: the admissions rate is less than 10% - 0: otherwise -- ``` r df %>% select(instnm,selective,adm_rate) ``` ``` ## # A tibble: 2,546 × 3 ## instnm selective adm_rate ## <chr> <dbl> <dbl> ## 1 Alabama A & M University 0 0.918 ## 2 University of Alabama at Birmingham 0 0.737 ## 3 Amridge University NA NA ## 4 University of Alabama in Huntsville 0 0.826 ## 5 Alabama State University 0 0.969 ## 6 The University of Alabama 0 0.827 ## 7 Central Alabama Community College NA NA ## 8 Athens State University NA NA ## 9 Auburn University at Montgomery 0 0.904 ## 10 Auburn University 0 0.807 ## # ℹ 2,536 more rows ``` --- # Summarizing Data: `group_by()` - Instead of running two separate `filter()` commands, use `group_by()` -- ``` r df %>% group_by(selective) %>% # Group the data by selective (either 1 or 0) summarise(mean_sat = mean(sat_avg,na.rm=T)) # Calculate average SAT for each group ``` ``` ## # A tibble: 3 × 2 ## selective mean_sat ## <dbl> <dbl> ## 1 0 1135. ## 2 1 1510. ## 3 NA NaN ``` --- # Results - Do more selective schools have higher SAT scores? -- - Yes -- - This .red[Result] **confirms** our .blue[Hypothesis] and **answers** our .blue[Research Question] --- # Conclusion -- - What we've done today is a microcosm of data science -- 1. Opened .red[data] (`readRDS`) -- 2. Looked at .red[data] (`tidyverse` + `select()`, `filter()`, `arrange()`) -- 3. Generated .blue[hypotheses] (Admissions versus SAT scores) -- 4. .red[Tested] .blue[hypotheses] (`summarise()` + `mean()`)