Fix typos
This commit is contained in:
parent
149c27ef61
commit
3378a1de5c
30
EDA.Rmd
30
EDA.Rmd
|
@ -10,13 +10,13 @@ This chapter will show you how to use visualisation and transformation to explor
|
||||||
|
|
||||||
1. Use what you learn to refine your questions and or generate new questions.
|
1. Use what you learn to refine your questions and or generate new questions.
|
||||||
|
|
||||||
EDA is not a formal process with a strict set of rules. More than anything, EDA is a state of mind. During the initial phases of EDA you should feel be free to investigate every idea that occurs to you. Some of these ideas will pan out, and some will be dead ends. As your exploration continues you will hone in on a few particularly productive areas that you'll eventually write up and communicate to others.
|
EDA is not a formal process with a strict set of rules. More than anything, EDA is a state of mind. During the initial phases of EDA you should feel be free to investigate every idea that occurs to you. Some of these ideas will pan out, and some will be dead ends. As your exploration continues, you will hone in on a few particularly productive areas that you'll eventually write up and communicate to others.
|
||||||
|
|
||||||
EDA is an important part of any data analysis, even if the questions are handed to you on a platter, because you always need to investigate the quality of your data. Data cleaning is just one application of EDA: you're ask questions about whether your data meets your expectations or not. To do data cleaning, you'll need to deploy all the tools of EDA: visualisation, transformation, and modelling.
|
EDA is an important part of any data analysis, even if the questions are handed to you on a platter, because you always need to investigate the quality of your data. Data cleaning is just one application of EDA: you ask questions about whether your data meets your expectations or not. To do data cleaning, you'll need to deploy all the tools of EDA: visualisation, transformation, and modelling.
|
||||||
|
|
||||||
### Prerequisites
|
### Prerequisites
|
||||||
|
|
||||||
In this chapter we'll combine what you've learned about dplyr and ggplot2 to iteratively ask questions, answer them with data, and then ask new questions.
|
In this chapter we'll combine what you've learned about dplyr and ggplot2 to interactively ask questions, answer them with data, and then ask new questions.
|
||||||
|
|
||||||
```{r setup, message = FALSE}
|
```{r setup, message = FALSE}
|
||||||
library(ggplot2)
|
library(ggplot2)
|
||||||
|
@ -174,14 +174,14 @@ Many of the questions above will prompt you to explore a relationship *between*
|
||||||
|
|
||||||
### Unusual values
|
### Unusual values
|
||||||
|
|
||||||
Outliers are observations that are unusual; data points that are don't seem to fit the pattern. Sometimes outliers are data entry errors; other times outliers suggest important new science. When you have a lot of data, outliers are sometimes difficult to see in a histogram. For example, take the distribution of the `x` variable from the diamonds dataset. The only evidence of outliers is the unusually wide limits on the x-axis.
|
Outliers are observations that are unusual; data points that don't seem to fit the pattern. Sometimes outliers are data entry errors; other times outliers suggest important new science. When you have a lot of data, outliers are sometimes difficult to see in a histogram. For example, take the distribution of the `y` variable from the diamonds dataset. The only evidence of outliers is the unusually wide limits on the y-axis.
|
||||||
|
|
||||||
```{r}
|
```{r}
|
||||||
ggplot(diamonds) +
|
ggplot(diamonds) +
|
||||||
geom_histogram(aes(x = y), binwidth = 0.5)
|
geom_histogram(aes(x = y), binwidth = 0.5)
|
||||||
```
|
```
|
||||||
|
|
||||||
There are so many observations in the common bins that the rare bins are so short that you can't see them (although maybe if you stare intently at 0 you'll spot something). To make it easy to see the unusual vaues, we need to zoom into to small values of the y-axis with `coord_cartesian()`:
|
There are so many observations in the common bins that the rare bins are so short that you can't see them (although maybe if you stare intently at 0 you'll spot something). To make it easy to see the unusual values, we need to zoom into to small values of the y-axis with `coord_cartesian()`:
|
||||||
|
|
||||||
```{r}
|
```{r}
|
||||||
ggplot(diamonds) +
|
ggplot(diamonds) +
|
||||||
|
@ -200,9 +200,9 @@ unusual <- diamonds %>%
|
||||||
unusual
|
unusual
|
||||||
```
|
```
|
||||||
|
|
||||||
The `y` variable measures one of the three dimensions of these diamonds, in mm. We know that diamonds can't have a width of 0mm, so these values must be incorrect.. We might also suspect that measurements of 32mm and 59mm are implausible: those diamonds are over an inch long, but don't cost hundreds of thousands of dollars!
|
The `y` variable measures one of the three dimensions of these diamonds, in mm. We know that diamonds can't have a width of 0mm, so these values must be incorrect. We might also suspect that measurements of 32mm and 59mm are implausible: those diamonds are over an inch long, but don't cost hundreds of thousands of dollars!
|
||||||
|
|
||||||
When you discover an outlier it's a good idea to trace it back as far as possible. You'll be in a much stronger analytical position if you can figure out why it happened. If you can't figure it out, and want to just move on with your analysis, replace it with a missing value, which we'll discuss in the next section.
|
When you discover an outlier, it's a good idea to trace it back as far as possible. You'll be in a much stronger analytical position if you can figure out why it happened. If you can't figure it out, and want to just move on with your analysis, replace it with a missing value, which we'll discuss in the next section.
|
||||||
|
|
||||||
### Exercises
|
### Exercises
|
||||||
|
|
||||||
|
@ -261,7 +261,7 @@ ggplot(data = diamonds2, mapping = aes(x = x, y = y)) +
|
||||||
geom_point(na.rm = TRUE)
|
geom_point(na.rm = TRUE)
|
||||||
```
|
```
|
||||||
|
|
||||||
Other times you want to understand what makes observations with missing values different to observations with recorded values. For example, in `nycflights13::flights`, missing value in the `dep_time` variable indicate that the flight was cancelled. So you might want to compare the scheduled departure times for cancelled and non-cancelled times. You can do by making a new variable with `is.na()`.
|
Other times you want to understand what makes observations with missing values different to observations with recorded values. For example, in `nycflights13::flights`, missing values in the `dep_time` variable indicate that the flight was cancelled. So you might want to compare the scheduled departure times for cancelled and non-cancelled times. You can do by making a new variable with `is.na()`.
|
||||||
|
|
||||||
```{r}
|
```{r}
|
||||||
nycflights13::flights %>%
|
nycflights13::flights %>%
|
||||||
|
@ -311,7 +311,7 @@ ggplot(data = diamonds, mapping = aes(x = price, y = ..density..)) +
|
||||||
geom_freqpoly(aes(colour = cut), binwidth = 500)
|
geom_freqpoly(aes(colour = cut), binwidth = 500)
|
||||||
```
|
```
|
||||||
|
|
||||||
There's something rather surprising about this plot - it appears that fair diamonds (the lowest quality) have the highest average cut! But maybe that's because frequency polygons are a little hard to interpret - there's a lot going on in this plot.
|
There's something rather surprising about this plot - it appears that fair diamonds (the lowest quality) have the highest average price! But maybe that's because frequency polygons are a little hard to interpret - there's a lot going on in this plot.
|
||||||
|
|
||||||
Another alternative to display the distribution of a continuous variable broken down by a categorical variable is the boxplot. A **boxplot** is a type of visual shorthand for a distribution of values that is popular among statisticians. Each boxplot consists of:
|
Another alternative to display the distribution of a continuous variable broken down by a categorical variable is the boxplot. A **boxplot** is a type of visual shorthand for a distribution of values that is popular among statisticians. Each boxplot consists of:
|
||||||
|
|
||||||
|
@ -340,7 +340,7 @@ ggplot(data = diamonds, mapping = aes(x = cut, y = price)) +
|
||||||
geom_boxplot()
|
geom_boxplot()
|
||||||
```
|
```
|
||||||
|
|
||||||
We see much less information about the distribution, but the boxplots are much more compact so we can more easily compare them (and fit more on one plot). It supports the counterintuive finding that better quality diamonds are cheaper on average! In the exercises, you'll be challenged to figure out why.
|
We see much less information about the distribution, but the boxplots are much more compact so we can more easily compare them (and fit more on one plot). It supports the counterintuitive finding that better quality diamonds are cheaper on average! In the exercises, you'll be challenged to figure out why.
|
||||||
|
|
||||||
`cut` is an ordered factor: fair is worse than good, which is worse than very good and so on. Most factors are unordered, so it's fair game to reorder to display the results better. For example, take the `class` variable in the `mpg` dataset. You might be interested to know how highway mileage varies across classes:
|
`cut` is an ordered factor: fair is worse than good, which is worse than very good and so on. Most factors are unordered, so it's fair game to reorder to display the results better. For example, take the `class` variable in the `mpg` dataset. You might be interested to know how highway mileage varies across classes:
|
||||||
|
|
||||||
|
@ -369,19 +369,21 @@ ggplot(data = mpg) +
|
||||||
1. Use what you've learned to improve the visualisation of the departure times
|
1. Use what you've learned to improve the visualisation of the departure times
|
||||||
of cancelled vs. non-cancelled flights.
|
of cancelled vs. non-cancelled flights.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
1. What variable in the diamonds dataset is most important for predicting
|
1. What variable in the diamonds dataset is most important for predicting
|
||||||
the price of a diamond? How is that variable correlated with cut?
|
the price of a diamond? How is that variable correlated with cut?
|
||||||
Why does the combination of those two relationships lead to lower quality
|
Why does the combination of those two relationships lead to lower quality
|
||||||
diamonds being more expensive?
|
diamonds being more expensive?
|
||||||
|
|
||||||
1. Install the ggstance pacakge, and create a horizontal boxplot.
|
1. Install the ggstance package, and create a horizontal boxplot.
|
||||||
How does this compare to using `coord_flip()`?
|
How does this compare to using `coord_flip()`?
|
||||||
|
|
||||||
1. One problem with boxplots is that they were developed in an era of
|
1. One problem with boxplots is that they were developed in an era of
|
||||||
much smaller datasets and tend to display a prohibitively large
|
much smaller datasets and tend to display a prohibitively large
|
||||||
number of "outlying values". One approach to remedy this problem is
|
number of "outlying values". One approach to remedy this problem is
|
||||||
the letter value plot. Install the lvplot package, and try using
|
the letter value plot. Install the lvplot package, and try using
|
||||||
`geom_lvplot()` to display the distribution of price vs cut. What
|
`geom_lv()` to display the distribution of price vs cut. What
|
||||||
do you learn? How do you interpret the plots?
|
do you learn? How do you interpret the plots?
|
||||||
|
|
||||||
1. Compare and contrast `geom_violin()` with a facetted `geom_histogram()`,
|
1. Compare and contrast `geom_violin()` with a facetted `geom_histogram()`,
|
||||||
|
@ -419,7 +421,7 @@ diamonds %>%
|
||||||
geom_tile(aes(fill = n))
|
geom_tile(aes(fill = n))
|
||||||
```
|
```
|
||||||
|
|
||||||
If the categorical variables are unordered, you might want to use the seriation package to simultaneously reorder the rows and columns in order to more clearly reveal interesting patterns. For larger plots, you might want to try the d3heatmap or heatmaply packages which creative interactive plots.
|
If the categorical variables are unordered, you might want to use the seriation package to simultaneously reorder the rows and columns in order to more clearly reveal interesting patterns. For larger plots, you might want to try the d3heatmap or heatmaply packages, which create interactive plots.
|
||||||
|
|
||||||
#### Exercises
|
#### Exercises
|
||||||
|
|
||||||
|
@ -482,7 +484,7 @@ ggplot(data = smaller, mapping = aes(x = carat, y = price)) +
|
||||||
|
|
||||||
1. Instead of summarising the conditional distribution with a boxplot, you
|
1. Instead of summarising the conditional distribution with a boxplot, you
|
||||||
could use a frequency polygon. What do you need to consider when using
|
could use a frequency polygon. What do you need to consider when using
|
||||||
`cut_width()` vs `cut_number()`? How does that impact a visualiation of
|
`cut_width()` vs `cut_number()`? How does that impact a visualisation of
|
||||||
the 2d distribution of `carat` and `price`?
|
the 2d distribution of `carat` and `price`?
|
||||||
|
|
||||||
1. Visualise the distribution of carat, partitioned by price.
|
1. Visualise the distribution of carat, partitioned by price.
|
||||||
|
|
Loading…
Reference in New Issue