--- title: "Univariate Analysis and Conditional Relationships" subtitle: "Homework 5" author: "Your Name" institute: "Villanova University" output: html_document: default pdf_document: default --- ```{r setup, include=FALSE} knitr::opts_chunk$set(echo = TRUE,error=TRUE,warning = FALSE,message = FALSE) ``` # Agenda - Basics of univariate analysis - Conditional data: when a variable varies with respect to some other variable. - How does the value of the outcome of interest vary _depending_ on the value of another variable of interest? - Typically: outcome of interest (dependent variable), Y-axis. - Other variables possibly related to the outcome (independent variables): X-axis Our tools depend on the **type of variables** we are trying to graph. # Univariate Data Analysis Univariate is pretty much what it sounds like: one variable. When undertaking univariate data analysis, we need first and foremost to figure what type of variable it is that we're working with. Once we do that, we can choose the appropriate use of the variable, either as an outcome or as a possible predictor. ## Motivating Question We'll be working with data from every NBA player who was active during the 2018-19 season. Here's the data: ```{r} require(tidyverse) nba<-read_rds("https://github.com/rweldzius/PSC4175/raw/main/static/data/nba_players_2018.Rds") ``` This data contains the following variables: # Codebook for NBA Data | Name | Definition | |--------------------|------------------------------------:| | namePlayer | Player name | | idPlayer | Unique player id | | slugSeason | Season start and end | | numberPlayerSeason | Which season for this player | | isRookie | Rookie season, true or false | | slugTeam | Team short name | | idTeam | Unique team id | | gp | Games Played | | gs | Games Started | | fgm | Field goals made | | fga | Field goals attempted | | pctFG | Percent of field goals made | | fg3m | 3 point field goals made | | fg3a | 3 point field goals attempted | | pctFG3 | Percent of 3 point field goals made | | pctFT | Free Throw percentage | | fg2m | 2 point field goals made | | fg2a | 2 point field goals attempted | | pctFG2 | Percent of 2 point field goals made | | agePlayer | Player age | | minutes | Minutes played | | ftm | Free throws made | | fta | Free throws attempted | | oreb | Offensive rebounds | | dreb | Defensive rebounds | | treb | Total rebounds | | ast | Assists | | blk | Blocks | | tov | Turnovers | | pf | Personal fouls | | pts | Total points | | urlNBAAPI | Source url | We're interested in the following questions: - Do certain colleges produce players that have more field goals? What about free throw percentage above a certain level? Are certain colleges in the east or the west more likely to produce higher scorers? How does this vary as a player has more seasons? To answer these questions we need to look at the following variables: - Field goals - Free throw percentage above .25 - Colleges - Player seasons - Region We're going to go through a pretty standard set of steps for each variable. First, examine some cases. Second, based on our examination, we'll try either a plot or a table. Once we've seen the plot or the table, we'll think a bit about ordering, and then choose an appropriate measure of central tendency, and maybe variation. ## Types of Variables It's really important to understand the types of variables you're working with. Many times analysts are indifferent to this step particularly with larger datasets. This can lead to a great deal of confusion down the road. Below are the variable types we'll be working with this semester and the definition of each. ### Continuous Variables A continuous variable can theoretically be subdivided at any arbitrarily small measure and can still be identified. You may have encountered further subdivision of continuous variables into "interval" or "ratio" data in other classes. We RARELY use these distinctions in practice. The distinction between a continuous and a categorical variable is hugely consequential, but the distinction between interval and ratio is not really all that important in practice. The mean is the most widely used measure of central tendency for a continuous variable. If the distribution of the variable isn't very symmetric or there are large outliers, then the median is a much better measure of central tendency. ### Categorical Variables A categorical variables divides the sample up into a set of mutually exclusive and exhaustive categories. Mutually exclusive means that each case can only be one, and exhaustive means that the categories cover every possible option. Categorical is sort of the "top" level classification for variables of this type. Within the broad classification of categorical there are multiple types of other variables. #### Categorical: ordered an ordered categorical variable has-- you guessed it-- some kind of sensible order that can be applied. For instance, the educational attainment of an individual: high school diploma, associates degree, bachelor's degree, graduate degree-- is an ordered categorical variable. Ordered categorical variables should be arranged in the order of the variable, with proportions or percentages associated with each order. The mode, or the category with the highest proportion, is a reasonable measure of central tendency, but with fewer than ten categories the analyst should generally just show the proportion in each category. ##### Categorical: ordered, binary An ordered binary variable has just two levels, but can be ordered. For instance, is a bird undertaking its first migration: yes or no? A "no" means that the bird has more than one. The mean of a binary variable is exactly the same thing as the proportion of the sample with that characteristic. So, the mean of a binary variable for "first migration" where 1="yes" will give the proportion of birds migrating for the first time. An ordered binary variable coded as 0 or 1 can be summarized using the mean which is the same thing as the proportion of the sample with that characteristic. #### Categorical: unordered An unordered categorical variable has no sensible ordering that can be applied. Think about something like college major. There's no "number" we might apply to philosophy that has any meaningful distance from a number we might apply to chemical engineering. Unlike an ordered variable, an unordered categorical variable should be ordered in terms of the proportions falling into each of the categories. As with an unordered variable, it's best just to show the proportions in each category for variables with less than ten levels. The mode is a reasonable single variable summary of an unordered categorical variable. ##### Categorical: unordered, binary This kind of variable has no particular order, but can be just binary. A "1" means that the case has that characteristics, a "0" means the case does not have that characteristic. For instance, whether a tree is deciduous or not. An unordered binary variable coded as 0 or 1 can also be summarized by the mean, which is the same thing as the proportion of the sample with that characteristic. ### Formats for categorical variables In R, categorical variables CAN be stored as text, numbers or even logicals. Don't count on the data to help you out-- you as the analyst need to figure this out. ## Factors We probably need to talk about factors. In R, a factor is a way of storing categorical variables. The factor provides additional information, including an ordering of the variable and a number assigned to each "level" of the factor. A categorical variable is a general term that's understood across statistics. A factor variable is a specific R term. Most of the time it's best not to have a categorical variable structured as a factor unless you know you want it to be a factor. More on this later . . . ## The Process: #TrustTheProcess I'm going to walk you through how an analyst might typically decide what type of variables they're working with. It generally works like this: 1. Take a look at a few observations and form a guess as to what type of variable it is. 2. Based on that guess, create an appropriate plot or table. 3. If the plot or table looks as expected, calculate some summary measures. If not, go back to 1. ## "Glimpse" to start: what's in here anyway? The first thing we're going to do with any dataset is just to take a quick look. We can call the data itself, but that will just show the first few cases and the first few variables. Far better is the glimpse command, which shows us all variables and the first few observations for all of the variables. Here's a link to the codebook for this dataset: The six variables we're going to think about are field goals, free throw percentage, seasons played, rookie season, college attended, and conference played in. ```{r} glimpse(nba) ``` ## Continuous Let's start by taking a look at field goals. It seems pretty likely that this is a continuous variable. Let's take a look at the top 50 spots. ```{r} nba%>% ## Start with the dataset select(namePlayer,slugTeam,fgm)%>% ## and then select a few variables arrange(-fgm)%>% ## arrange in reverse order of field goals print(n=50) ## print out the top 50 ``` So what I'm seeing here is that field goals aren't "clumped" at certain levels. Let's confirm that by looking at a kernel density plot. ```{r} nba%>% ggplot(aes(x=fgm))+ geom_density() ``` We can also use a histogram to figure out much the same thing. ```{r} nba%>% ggplot(aes(x=fgm))+ geom_histogram() ``` Now, technically field goals don't meet the definition I set out above as being a continuous variable because they aren't divisible below a certain amount. Usually in practice though we just ignore this-- this variable is "as good as" continuous, given that it varies smoothly over the range and isn't confined to a relatively small set of possible values. **Quick Exercise 1**: Do the same thing for field goal percentage and think about what kind of variable it is. ```{r} # INSERT CODE HERE ``` ## Measures for Continuous Variables The mean is used most of the time for continuous variables, but it's VERY sensitive to outliers. The median (50th percentile) is usually better, but it can be difficult to explain to general audiences. ```{r} nba%>% summarize(mean_fgm=mean(fgm)) nba%>% summarize(median_fgm=median(fgm)) ``` In this case I'd really prefer the mean as a single measure of field goal production, but depending on the audience I still might just go ahead and use the median. **Quick Exercise 2** What measure would you prefer for field goal percentage? Calculate that measure. ```{r} # INSERT CODE HERE ``` ## Categorical: ordered Let's take a look at player seasons. ```{r} nba%>% select(namePlayer,numberPlayerSeason)%>% arrange(-numberPlayerSeason)%>% print(n=50) ``` Looks like it might be continuous? Let's plot it: ```{r} nba%>% ggplot(aes(x=numberPlayerSeason))+ geom_histogram(binwidth = 1) ``` Nope. See how it falls into a small set of possible categories? This is an ordered categorical variable. That means we should calculate the proportions in each category ```{r} nba%>% group_by(numberPlayerSeason)%>% count(name="total_in_group")%>% ungroup()%>% mutate(proportion=total_in_group/sum(total_in_group)) ``` What does this tell us? **Quick Exercise 3** Create a histogram for player age. What does that tell us about the NBA? ```{r} # INSERT CODE HERE ``` ## Categorical: ordered, binary Let's take a look at the variable for Rookie season. ```{r} nba%>%select(namePlayer,isRookie) ``` Okay, so that's set to a logical. In R, TRUE or FALSE are special values that indicate the result of a logical question. In this it's whether or not the player is a rookie. Usually we want a binary variable to have at least one version that's structured so that 1= TRUE and 2=FALSE. This makes data analysis much easier. Let's do that with this variable. This code uses `ifelse` to create a new variable called `isRookiebin` that's set to 1 if the `isRookie` variable is true, and 0 otherwise. ```{r} nba<-nba%>% mutate(isRookie_bin=ifelse(isRookie==TRUE,1,0)) ``` Now that it's coded 0,1 we can calculate the mean, which is the same thing as the proportion of the players that are rookies. ```{r} nba%>%summarize(mean=mean(isRookie_bin)) ``` ## Categorical: unordered Let's take a look at which college a player attended, which is a good example of an unordered categorical variable. The `org` variable tells us which organization the player was in before playing in the NBA. ```{r} nba%>% select(org)%>% glimpse() ``` This look like team or college names, so this would be a categorical variable. Let's take a look at the counts of players from different organizations: ```{r} nba%>% group_by(org)%>% count()%>% arrange(-n)%>% print(n=50) ``` Here we have a problem. If we're interested just in colleges, we're going to need to structure this a bit more. The code below filters out three categories that we don't want: missing data, anything classified as others, and sports teams from other countries. The last is incomplete-- I probably missed some! If I were doing this for real, I would use a list of colleges and only include those names. What I do below is to negate the `str_detect` variable by placing the `!` in front of it. This means I want all of the cases that don't match the pattern the supplied. The pattern makes heavy use of the OR operator `|`. I'm saying I don't want to include players whose organization included the letters `CB` r `KK` and so on (these are common prefixes for sports organizations in other countries, I definitely did not look that up on Wikipedia. Ok, I did.). ```{r} nba%>% filter(!is.na(org))%>% filter(!org=="Other")%>% filter(!str_detect(org,"CB|KK|rytas|FC|B.C.|S.K.|Madrid"))%>% group_by(org)%>% count()%>% arrange(-n)%>% print(n=50) ``` That looks better. Which are the most common colleges and universities that send players to the NBA? **Quick Exercise 4** Arrange the number of players by team in descending order. ```{r} # INSERT CODE HERE ``` ## Categorical: unordered, binary There are two conference in the NBA, eastern and western. Let's take a look at the variable that indicates which conference the payer played in that season. ```{r} nba%>%select(idConference)%>% glimpse() ``` It looks like conference is structured as numeric, but a "1" or a "2". Because it's best to have binary variables structured as "has the characteristic" or "doesn't have the characteristic" we're going to create a variable for western conference that's set to 1 if the player was playing in the western conference and 0 if the player was not (this is the same as playing in the eastern conference). ```{r} nba<-nba%>% mutate(west_conference=ifelse(idConference==1,1,0)) ``` Once we've done that, we can see how many players played in each conference. ```{r} nba%>% summarize(mean(west_conference)) ``` Makes sense! **Quick Exercise 5**:* create a variable for whether or not the player is from the USA. Calculate the proportion of players from the USA in the NBA. The coding on country is ... decidedy US-centric, so you'll need to think about this one a bit. ```{r} # INSERT CODE HERE ``` ## Analysis Ok, now that we know how this works, we can do some summary analysis. First of all, what does the total number of field goals made look like by college? We know that field goals are continuous (sort of) so let's summarize them via the mean. We know that college is a categorical variable, so we'll use that to group the data. This is one of our first examples of a conditiona mean, which we'll use a lot. ## Top 50 Colleges by Total FG ```{r} nba%>% filter(!is.na(org))%>% filter(!org=="Other")%>% filter(!str_detect(org,"CB|KK|rytas|FC|B.C.|S.K.|Madrid"))%>% group_by(org)%>% summarize(mean_fg=sum(fgm))%>% arrange(-mean_fg)%>% print(n=50) ``` Next, what about field goal percentage? ## Top 50 Colleges by Average Field Goal Percent ```{r} nba%>% filter(!is.na(org))%>% filter(!org=="Other")%>% filter(!str_detect(org,"CB|KK|rytas|FC|B.C.|S.K.|Madrid"))%>% group_by(org)%>% summarize(mean_ftp=mean(pctFT))%>% arrange(-mean_ftp)%>% print(n=50) ``` **Quick Exercise 6** Calculate field goals made by player season. ```{r} # INSERT CODE HERE ``` **Quick Exercise 7** Calculate free throw percent made by player season. ```{r} # INSERT CODE HERE ``` # Conditional relationships: The "gender" gap in electoral politics Conditional variation involves examining how the values of two or more variables are related to one another. Earlier we made these comparisons by creating different tibbles and then comparing across tibbles, but we can also make comparisons without creating multiple tibbles. So load in the Michigan 2020 Exit Poll Data. ```{r message=FALSE} library(tidyverse) library(scales) mi_ep <- read_rds("https://github.com/rweldzius/PSC4175/raw/main/static/data/MI2020_ExitPoll_small.rds") MI_final_small <- mi_ep %>% filter(preschoice=="Donald Trump, the Republican" | preschoice=="Joe Biden, the Democrat") %>% mutate(BidenVoter=ifelse(preschoice=="Joe Biden, the Democrat",1,0), TrumpVoter=ifelse(BidenVoter==1,0,1), AGE10=ifelse(AGE10==99,NA,AGE10)) ``` We learned that if we `count` using multiple variables that R will count within values. Can we use this to analyze how this varies by groups? Let's see! ```{r} MI_final_small %>% filter(AGE10==1) %>% count(preschoice,SEX) %>% mutate(PctSupport = n/sum(n), PctSupport = round(PctSupport, digits=2)) ``` Here we have broken everything out by both `preschoice` and `SEX` but the `PctSupport` is not quite what we want because it is the fraction of responses (out of 1) that are in each row rather than the proportion of support for each candidate **by** sex. To correct this and to perform the functions within a value we need to use the `group_by` function. We can use the `group_by` command to organize our data a bit better. What `group_by` does is to run all subsequent code separately according to the defined group. So instead of running a count or summarize separately for both Males and Females as we did above, we can `group_by` the variable `SEX.chr` (or `FEMALE` or `SEX` -- it makes no difference as they are all equivalent) and then preform the subsequent commands. So here we are going to filter to select those who are 24 and below and then we are going to count the number of Biden and Trump supporters within each value of `SEX.chr` ```{r} MI_final_small %>% filter(AGE10==1) %>% group_by(SEX) %>% count(preschoice) ``` Note that any functions of the data are also now organized by that grouping, so if we were to manually compute the proportions using the mutation approach discussed above we would get: ```{r} MI_final_small %>% filter(AGE10==1) %>% group_by(SEX) %>% count(preschoice) %>% mutate(PctSupport = n/sum(n), PctSupport = round(PctSupport, digits=2)) ``` So you can see that `PctSupport` sums to 2.0 because it sums to 1.0 within each value of the grouping variable `SEX`. If we wanted the fraction of voters who are in each unique category - so that the percentage of all the categories sum to 1.0 -- we would want to `ungroup` before doing the mutation that calculates the percentage. So here we are doing the functions after the `group_by()` separately for each value of the grouping variables (here `SEX`) and then we are going to then undo that and return to the entire dataset. ```{r} MI_final_small %>% filter(AGE10==1) %>% group_by(SEX) %>% count(preschoice) %>% ungroup() %>% mutate(PctSupport = n/sum(n), PctSupport = round(PctSupport, digits=2)) ``` If we are just interested in the proportion and we do not care about the number of respondents in each value (although here it seems relevant!) we could also `group_by` and then `summarize` as follows: ```{r} MI_final_small %>% filter(AGE10==1) %>% group_by(SEX) %>% summarize(PctBiden = mean(BidenVoter), PctTrump = mean(TrumpVoter)) %>% mutate(PctBiden = round(PctBiden, digits =2), PctTrump = round(PctTrump, digits =2)) ``` Because we have already filtered to focus only on Biden and Trump voters, we don't actually need both since `PctBiden = 1 - PctTrump` and `PctTrump = 1 - PctBiden`. Note that we can have multiple groups. So if we want to group by age and by sex we can do the following... ```{r} MI_final_small %>% group_by(SEX, AGE10) %>% summarize(PctBiden = mean(BidenVoter)) %>% mutate(PctBiden = round(PctBiden, digits =2)) ``` We can also save it for later analysis and then filter or select the results. For example: ```{r} SexAge <- MI_final_small %>% group_by(SEX, AGE10) %>% summarize(PctBiden = mean(BidenVoter)) %>% mutate(PctBiden = round(PctBiden, digits =2)) %>% drop_na() ``` So if we want to look at the Biden support by age among females (i.e., `SEX==2`) we can look at: ```{r} SexAge %>% filter(SEX == 2) ``` **Quick Exercise 8** What is the Biden support by age among males? ```{r} # INSERT CODE HERE ``` # Dicrete Variable By Discrete Variable (Barplot) If we are working with discrete/categorical/ordinal/data --- i.e., variables that take on a finite (and small) number of unique values then we are interested in how to compare across bar graphs. Before we used `geom_bar` to plot the number of observations associated with each value of a variable. But we often want to know how the number of observations may vary according to a second variable. For example, we care not only about why voters reported that they supported Biden or Trump in 2020 but we are also interested in knowing whether Biden and Trump voters were voting for similar or different reasons. Did voters differ in terms of why they were voting for a candidate in addition to who they were voting for? If so, this may suggest something about what each set of voters were looking for in a candidate. Let's first plot the barplot and then plot the barplot by presidential vote choice for the Michigan Exit Poll we were just analyzing. We are interested in the distribution of responses to the variable `Quality` and we only care about voters who voted for either Biden or Trump (`preschoice`) so let's select those variables and `filter` using `preschoice` to select those respondents. We have an additional complication that the question was only asked of half of the respondents and some that were asked refused to answer. To remove these respondents we want to `drop_na` (note that this will drop every observation with a missing value -- this is acceptable because we have used `select` to focus on the variables we are analyzing, but if we did not use `select` it would have dropped an observation with missing data in **any** variable. We could get around this using `drop_na(Quality)` if we wanted). A final complication is that some respondents did not answer the question they were asked so we have to use `filter` to remove respondents with missing observations. Now we include labels -- note how we are suppressing the x-label because the value labels are self-explanatory in this instance and add the `geom_bar` as before. ```{r} mi_ep %>% select(Quality,preschoice) %>% filter(preschoice == "Joe Biden, the Democrat" | preschoice == "Donald Trump, the Republican") %>% drop_na() %>% filter(Quality != "[DON'T READ] Don’t know/refused") %>% ggplot(aes(x= Quality)) + labs(y = "Number of Voters", x = "", title = "Michigan 2020 Exit Poll: Reasons for voting for a candidate") + geom_bar(color="black") ``` Note that if we add `coord_flip` that we will flip the axes of the graph. (We could also have done this by changing `aes(y= Quality)`, but then we would also have to change the associated labels.) ```{r} mi_ep %>% select(Quality,preschoice) %>% filter(preschoice == "Joe Biden, the Democrat" | preschoice == "Donald Trump, the Republican") %>% drop_na() %>% filter(Quality != "[DON'T READ] Don’t know/refused") %>% ggplot(aes(x= Quality)) + labs(y = "Number of Voters", x = "", title = "Michigan 2020 Exit Poll: Reasons for voting for a candidate") + geom_bar(color="black") + coord_flip() ``` So enough review, lets add another dimension to the data. To show how the self-reported reasons for voting for a presidential candidate varied by vote choice we are going to use the `fill` of the graph to create different color bars depending on the value of the character or factor variable that is used to `fill`. So we are going to include as a `ggplot` aesthetic a character or factor variable as a `fill` (here `fill=preschoice`) and then we are going to also include `fill` in the `labs` function to make sure that we label the meaning of the values being plotted. The other change we have made is in `geom_bar` where we used `position=dodge` to make sure that the bars are plotted next to one-another rather than on top of one another. ```{r} mi_ep %>% select(Quality,preschoice) %>% filter(preschoice == "Joe Biden, the Democrat" | preschoice == "Donald Trump, the Republican") %>% drop_na() %>% filter(Quality != "[DON'T READ] Don’t know/refused") %>% ggplot(aes(x= Quality, fill = preschoice)) + labs(y = "Number of Voters", x = "", title = "Michigan 2020 Exit Poll: Reasons for voting for a candidate", fill = "Self-Reported Vote") + geom_bar(color="black", position="dodge") + coord_flip() ``` For fun, see what happens when you do not use `postion=dodge`. Also see what happens if you do not flip the coordinates using `coord_flip`. It is important to note that the `fill` variable has to be a character or a factor. If we want to graph self-reported vote by sex, for example, we need to redefine the variable for the purposes of `ggplot` as follows. Note that because we are not mutating it and we are only defining it to be a factor within the ``ggplot` object, this redefinition will not stick. Note also the problem caused by uninformative values in `SEX` -- can you change it. ```{r} mi_ep %>% filter(preschoice == "Joe Biden, the Democrat" | preschoice == "Donald Trump, the Republican") %>% ggplot(aes(x= preschoice, fill = factor(SEX))) + labs(y = "Number of Respondents", x = "", title = "Vote by Respondent Sex", fill = "Sex") + geom_bar(color="black", position="dodge") + coord_flip() ``` **Quick Exercise 9** The barplot we just produced does not satisfy our principles of visualization because the fill being used is uninterpretable to those unfamiliar with the dataset. Redo the code to use a `fill` variable that produces an informative label. Hint: don't overthink. ```{r} # INSERT CODE HERE ``` # A new question Suppose we were concerned with whether some polls might give different answers because of variation in who the poll is able to reach using that method. People who take polls via landline phones (do you even know what that is?) might differ from those who take surveys online. Or people contacted using randomly generated phone numbers (RDD) may differ from those contacted from a voter registration list that has had telephone numbers merged onto it. Polls were done using lots of different methods in 2020. # Loading the data ```{r} require(tidyverse) Pres2020.PV <- read_rds(file="https://github.com/rweldzius/PSC4175/raw/main/static/data/Pres2020_PV.Rds") Pres2020.PV <- Pres2020.PV %>% mutate(Trump = Trump/100, Biden = Biden/100, margin = Biden - Trump) ``` ```{r} Pres2020.PV %>% count(Mode) ``` This raises the question of -- how do we visualization variation in a variable by another variable? More specifically, how can we visualize how the `margin` we get using one type of survey compares to the `margin` from another type of poll? (We cannot use a scatterplot because the data is from different observations (here polls).) We could do this using earlier methods by `select`ing polls with a specific interview method ("mode") and then plotting the `margin` (or `Trump` or `Biden`), but that will produce a bunch of separate plots that may be hard to directly compare. (In addition to having more things to look at we would want to make sure that the scale of the x-axis and y-axis are similar.) We can plot another "layer" of data in ``ggplot` using the `fill` paramter. Previously we used it to make the graphs look nice by choosing a particular color. But if we set `fill` to be a variable in our `tibble` then `ggplot` will plot the data seperately for each unique value in the named variable. So if we want to plot the histogram of `margin` for two types of polls we can use the `fill` argument in `ggplot` to tell R to produce different fills depending on the value of that variable. ```{r} Pres2020.PV %>% filter(Mode == "IVR/Online" | Mode == "Live phone - RDD") %>% ggplot(aes(x= margin, fill = Mode)) + labs(y = "Number of Polls", x = "Biden- Trump Margin", title = "Biden-Trump Margin for Two Types of Polls", fill = "Mode of Interview") + geom_histogram(bins=10, color="black", position="dodge") + scale_x_continuous(breaks=seq(-.1,.2,by=.05), labels= scales::percent_format(accuracy = 1)) ``` **Quick Exercise 10** Try running the code without the `filter`. What do you observe? How useful is this? Why or why not? ```{r} # INSERT CODE ``` While informative, it can be hard to compare the distribution of more than two categories using such methods. To compare the variation across more types of surveys we need to use a different visualization that summarizes the variation in the variable of interest a bit more. One common visualization is the `boxplot` which reports the mean, 25th percentile (i.e., the value of the data if we sort the data from lowest to highest and take the value of the observation that is 25\% of the way through), the 75th percentile, the range of values, and notable outliers. Let's see what the `boxplot` of survey mode looks like after we first drop surveys that were conducted using modes that were hardly used (or missing). ```{r} Pres2020.PV %>% filter(Mode != "IVR" & Mode != "Online/Text" & Mode != "Phone - unknown" & Mode != "NA") %>% ggplot(aes(x = Mode, y = margin)) + labs(x = "Mode of Survey Interview", y = "Biden- Trump Margin", title = "2020 Popular Vote Margin by Type of Poll") + geom_boxplot(fill = "slateblue") + scale_y_continuous(breaks=seq(-.1,.2,by=.05), labels= scales::percent_format(accuracy = 1)) ``` We can also flip the graph if we think it makes more sense to display it in a different orientation using `coord_flip`. (We could, of course, also redefine the x and y variables in the ``ggplot` object, but it is useful to have a command to do this to help you determine which orientiation is most useful). ```{r} Pres2020.PV %>% filter(Mode != "IVR" & Mode != "Online/Text" & Mode != "Phone - unknown" & Mode != "NA") %>% ggplot(aes(x = Mode, y = margin)) + labs(x = "Mode of Survey Interview", y = "Biden- Trump Margin", title = "2020 Popular Vote Margin by Type of Poll") + geom_boxplot(fill = "slateblue") + scale_y_continuous(breaks=seq(-.1,.2,by=.05), labels= scales::percent_format(accuracy = 1)) + coord_flip() ``` A downside of the boxplot is that it can be hard to tell how the data varies within each box. Is it equally spread out? How much data are contained in the lines (which are simply 1.5 times the height of the box)? To get a better handle on this we can use a "violin" plot that dispenses with a standard box and instead tries to plot the distribution of data within each category. ```{r} Pres2020.PV %>% filter(Mode != "IVR" & Mode != "Online/Text" & Mode != "Phone - unknown" & Mode != "NA") %>% ggplot(aes(x=Mode, y=margin)) + xlab("Mode") + ylab("Biden- Trump Margin") + geom_violin(fill="slateblue") ``` It is also hard to know **how much** data is being plotted. If some modes have 1000 polls and others have only 5 that seems relevant. **Quick Exercise 11** We have looked at the difference in `margin`. How about differences in the percent who report supporting `Biden` and `Trump`? What do you observe. Does this suggest that the different ways of contacting respondents may matter in terms of who responds? Is there something else that may explain the differences (i.e., what are we assuming when making this comparison)? ```{r} # INSERT CODE HERE ``` **Quick Exercise 12** Some claims have been made that polls that used multiple ways of contacting respondents were better than polls that used just one. Can you evaluate whether there were differences in so-called "mixed-mode" surveys compared to single-mode surveys? (This requires you to define a new variable based on `Mode` indicating whether survey is mixed-mode or not.) ```{r} # INSERT CODE HERE ``` # Continuous Variable By Continuous Variable (Scatterplot) When we have two continuous variables we use a scatterplot to visualize the relationship. A scatterplot is simply a graph of every point in (x,y) where x is the value associated with the x-variable and y is the value associated with the y-variable. For example, we may want to see how support for Trump and Biden within a poll varies. So each observation is a poll of the national popular vote and we are going to plot the percentage of respondents in each poll supporting Biden against the percentage who support Trump. To include two variables we are going to change our aesthetic to define both an x variable and a y variable -- here `aes(x = Biden, y = Trump)` and we are going to label and scale the axes appropriately. ```{r} Pres2020.PV %>% ggplot(aes(x = Biden, y = Trump)) + labs(title="Biden and Trump Support in 2020 National Popular Vote", y = "Trump Support", x = "Biden Support") + geom_point(color="purple") + scale_y_continuous(breaks=seq(0,1,by=.05), labels= scales::percent_format(accuracy = 1)) + scale_x_continuous(breaks=seq(0,1,by=.05), labels= scales::percent_format(accuracy = 1)) ``` The results are intriguing! First the data seems like it falls along a grid. This is because of how poll results are reported in terms of percentage points and it highlights that even continuous variables may be reported in discrete values. This is consequential because it is hard to know how many polls are associated with each point on the graph. How many polls are at the point (Biden 50\%, Trump 45\%)? This matters for trying to determine what the relationship might be. Second, it is clear that there are some questions that need to be asked -- why doesn't `Biden + Trump = 100\%`? To try to display how many observations are located at each point we have two tools at our disposal. First, we can alter the "alpha transparency" by setting `alpha-.5` in the `geom_point` call. By setting a low level of transparency, this means that the point will become less transparent as more points occur at the same coordinate. Thus, a faint point indicates that only a single poll (observation) is located at a coordinate whereas a solid point indicates that there are many polls. When we apply this to the scatterplot you can immediately see that most of the polls are located in the neighborhood of Biden 50\%, Trump 42\%. ```{r} Pres2020.PV %>% ggplot(aes(x = Biden, y = Trump)) + labs(title="Biden and Trump Support in 2020 National Popular Vote", y = "Trump Support", x = "Biden Support") + geom_point(color="purple",alpha = .3) + scale_y_continuous(breaks=seq(0,1,by=.05), labels= scales::percent_format(accuracy = 1)) + scale_x_continuous(breaks=seq(0,1,by=.05), labels= scales::percent_format(accuracy = 1)) ``` However, the grid-like nature of the plot is still somewhat hard to interpret as it can be hard to discern variations in color gradient. Another tool is to add a tiny bit of randomness to the x and y values associated with each plot. Instead of values being constrained to vary by a full percentage point, for example, the jitter allows it to vary by less. To do so we replace `geom_point` with `geom_jitter`. ```{r} Pres2020.PV %>% ggplot(aes(x = Biden, y = Trump)) + labs(title="Biden and Trump Support in 2020 National Popular Vote", y = "Trump Support", x = "Biden Support") + geom_jitter(color="purple",alpha = .5) + scale_y_continuous(breaks=seq(0,1,by=.05), labels= scales::percent_format(accuracy = 1)) + scale_x_continuous(breaks=seq(0,1,by=.05), labels= scales::percent_format(accuracy = 1)) ``` Note how much the visualization changes. Whereas before the eye was focused on -- and arguably distracted by -- the grid-like orientation imposed by the measurement, once we jitter the points we are immediately made aware of the relationship between the two variables. While we are indeed slightly changing our data by adding random noise, the payoff is that the visualization arguably better highlights the nature of the relationship. Insofar the goal of visualization is communication, this trade-off seems worthwhile in this instance. But here again is where data science is sometimes art as much as science. The decision of which visualization to use depends on what you think most effectively communicates the nature of the relationship to the reader. We can also look at the accuracy of a poll as a function of the sample size. This is also a relationship between two continuous variables -- hence a scatterplot! Are polls with more respondents more accurate? There is one poll with nearly 80,000 respondents that we will filter out to me able to show a reasonable scale. Note that we are going to use `labels = scales::comma` when plotting the x-axis to report numbers with commas for readability. ```{r} Pres2020.PV %>% filter(SampleSize < 50000) %>% mutate(TrumpError = Trump - RepCertVote/100, BidenError = Biden - DemCertVote/100) %>% ggplot(aes(x = SampleSize, y = TrumpError)) + labs(title="Trump Polling Error in 2020 National Popular Vote as a function of Sample Size", y = "Error: Trump Poll - Trump Certified Vote", x = "Sample Size in Poll") + geom_jitter(color="purple",alpha = .5) + scale_y_continuous(breaks=seq(-.2,1,by=.05), labels= scales::percent_format(accuracy = 1)) + scale_x_continuous(breaks=seq(0,30000,by=5000), labels= scales::comma) ``` In sum, we have tested Trump's theory that the MSM was biased against him. We found that polls that underpredicted Trump **also** underpredicted Biden. This is not what we would expect if the polls favored one candidate over another. ```{r} Pres2020.PV %>% ggplot(aes(x = Biden, y = Trump)) + labs(title="Biden and Trump Support in 2020 National Popular Vote", y = "Trump Support", x = "Biden Support") + geom_jitter(color="purple",alpha = .5) + scale_y_continuous(breaks=seq(0,1,by=.05), labels= scales::percent_format(accuracy = 1)) + scale_x_continuous(breaks=seq(0,1,by=.05), labels= scales::percent_format(accuracy = 1)) ``` What is an alternative explanation for these patterns? Why would polls underpredict *both* Trump and Biden? Perhaps they were fielded earlier in the year, when more people were interested in third party candidates, or hadn't made up their mind. We'll turn to testing this theory next time!