Logical and number tweaking
This commit is contained in:
parent
33af7eabb9
commit
9239e50ccc
179
logicals.Rmd
179
logicals.Rmd
|
@ -150,7 +150,7 @@ flights |>
|
||||||
filter(dep_time == NA)
|
filter(dep_time == NA)
|
||||||
```
|
```
|
||||||
|
|
||||||
Instead we'll need a new too: `is.na()`.
|
Instead we'll need a new tool: `is.na()`.
|
||||||
|
|
||||||
### `is.na()`
|
### `is.na()`
|
||||||
|
|
||||||
|
@ -248,7 +248,14 @@ flights |>
|
||||||
filter(month %in% c(11, 12))
|
filter(month %in% c(11, 12))
|
||||||
```
|
```
|
||||||
|
|
||||||
Note the `%in%` obeys different rules for `NA` to `==`.
|
Note that `%in%` obeys different rules for `NA` to `==`.
|
||||||
|
|
||||||
|
```{r}
|
||||||
|
c(1, 2, NA) == NA
|
||||||
|
c(1, 2, NA) %in% NA
|
||||||
|
```
|
||||||
|
|
||||||
|
This can make for a useful shortcut:
|
||||||
|
|
||||||
```{r}
|
```{r}
|
||||||
flights |>
|
flights |>
|
||||||
|
@ -260,30 +267,39 @@ flights |>
|
||||||
The rules for missing values in Boolean algebra are a little tricky to explain because they seem inconsistent at first glance:
|
The rules for missing values in Boolean algebra are a little tricky to explain because they seem inconsistent at first glance:
|
||||||
|
|
||||||
```{r}
|
```{r}
|
||||||
NA & c(TRUE, FALSE, NA)
|
df <- tibble(x = c(TRUE, FALSE, NA))
|
||||||
NA | c(TRUE, FALSE, NA)
|
|
||||||
|
df |>
|
||||||
|
mutate(
|
||||||
|
and = x & NA,
|
||||||
|
or = x | NA
|
||||||
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
To understand what's going on, think about `NA | TRUE`.
|
To understand what's going on, think about `NA | TRUE`.
|
||||||
If a logical is `NA`, than means it could either be `TRUE` or `FALSE`.
|
A missing value means that the value could either be `TRUE` or `FALSE`.
|
||||||
`TRUE | TRUE` and `FALSE | TRUE` are both `TRUE`, so `NA | TRUE` must also be `TRUE`.
|
`TRUE | TRUE` and `FALSE | TRUE` are both `TRUE`, so `NA | TRUE` must also be `TRUE`.
|
||||||
Similar reasoning applies with `NA & FALSE`.
|
Similar reasoning applies with `NA & FALSE`.
|
||||||
|
|
||||||
### Exercises
|
### Exercises
|
||||||
|
|
||||||
1. Find all flights where `arr_delay` is missing but `dep_delay` is not. Find all flights where neither `arr_time` nor `sched_arr_time` are missing, but `arr_delay` is.
|
1. Find all flights where `arr_delay` is missing but `dep_delay` is not. Find all flights where neither `arr_time` nor `sched_arr_time` are missing, but `arr_delay` is.
|
||||||
2. How many flights have a missing `dep_time`? What other variables are missing? What might these rows represent?
|
2. How many flights have a missing `dep_time`? What other variables are missing in these rows? What might these rows represent?
|
||||||
3. How could you use `arrange()` to sort all missing values to the start? (Hint: use `!is.na()`).
|
3. Look at the number of cancelled flights per day. Is there a pattern? Is the proportion of cancelled flights related to the average delay?
|
||||||
4. Come up with another approach that will give you the same output as `not_cancelled |> count(dest)` and `not_cancelled |> count(tailnum, wt = distance)` (without using `count()`).
|
|
||||||
5. Look at the number of cancelled flights per day. Is there a pattern? Is the proportion of cancelled flights related to the average delay?
|
|
||||||
|
|
||||||
## Summaries {#logical-summaries}
|
## Summaries {#logical-summaries}
|
||||||
|
|
||||||
There are four particularly useful summary functions for logical vectors: they all take a vector of logical values and return a single value, making them a good fit for use in `summarise()`.
|
While, you can summarize logical variables directly with functions that work only with logicals, there are two other important summaries.
|
||||||
|
Numeric summaries like `sum()` and `mean()`, and using summaries as inline filters.
|
||||||
|
|
||||||
`any()` and `all()` --- `any()` will return if there's at least one `TRUE`, `all()` will return `TRUE` if all values are `TRUE`.
|
### Logical summaries
|
||||||
|
|
||||||
|
There are two important logical summaries: `any()` and `all()`.
|
||||||
|
`any(x)` is the equivalent of `|`; it'll return `TRUE` if there are any `TRUE`'s in `x`.
|
||||||
|
`all(x)` is equivalent of `&`; it'll return `TRUE` only if all values of `x` are `TRUE`'s.
|
||||||
Like all summary functions, they'll return `NA` if there are any missing values present, and like usual you can make the missing values go away with `na.rm = TRUE`.
|
Like all summary functions, they'll return `NA` if there are any missing values present, and like usual you can make the missing values go away with `na.rm = TRUE`.
|
||||||
We could use this to see if there were any days where every flight was delayed:
|
|
||||||
|
For example, we could use `all()` to find out if there were days where every flight was delayed:
|
||||||
|
|
||||||
```{r}
|
```{r}
|
||||||
not_cancelled <- flights |>
|
not_cancelled <- flights |>
|
||||||
|
@ -291,18 +307,31 @@ not_cancelled <- flights |>
|
||||||
|
|
||||||
not_cancelled |>
|
not_cancelled |>
|
||||||
group_by(year, month, day) |>
|
group_by(year, month, day) |>
|
||||||
filter(all(arr_delay >= 0))
|
summarise(
|
||||||
|
all_delayed = all(arr_delay >= 0),
|
||||||
|
any_delayed = any(arr_delay >= 0),
|
||||||
|
.groups = "drop"
|
||||||
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
`sum()` and `mean()` are particularly useful with logical vectors because when you use a logical vector in a numeric context, `TRUE` becomes 1 and `FALSE` becomes 0.
|
In most cases, however, `any()` and `all()` are a little too crude, and it would be nice to be able to get a little more detail about how many values are `TRUE` or `FALSE`.
|
||||||
That means that `sum(x)` gives the number of `TRUE`s in `x` and `mean(x)` gives the proportion of `TRUE`s.
|
That leads us to the numeric summaries.
|
||||||
That lets us find the day with the highest proportion of delayed flights:
|
|
||||||
|
### Numeric summaries
|
||||||
|
|
||||||
|
When you use a logical vector in a numeric context, `TRUE` becomes 1 and `FALSE` becomes 0.
|
||||||
|
This makes `sum()` and `mean()` are particularly useful with logical vectors because `sum(x)` will give the number of `TRUE`s and `mean(x)` gives the proportion of `TRUE`s.
|
||||||
|
That lets us see the distribution of delays across the days of the year:
|
||||||
|
|
||||||
```{r}
|
```{r}
|
||||||
not_cancelled |>
|
not_cancelled |>
|
||||||
group_by(year, month, day) |>
|
group_by(year, month, day) |>
|
||||||
summarise(prop_delayed = mean(arr_delay > 0)) |>
|
summarise(
|
||||||
arrange(desc(prop_delayed))
|
prop_delayed = mean(arr_delay > 0),
|
||||||
|
.groups = "drop"
|
||||||
|
) |>
|
||||||
|
ggplot(aes(prop_delayed)) +
|
||||||
|
geom_histogram(binwidth = 0.05)
|
||||||
```
|
```
|
||||||
|
|
||||||
Or we could ask how many flights left before 5am, which usually are flights that were delayed from the previous day:
|
Or we could ask how many flights left before 5am, which usually are flights that were delayed from the previous day:
|
||||||
|
@ -310,13 +339,37 @@ Or we could ask how many flights left before 5am, which usually are flights that
|
||||||
```{r}
|
```{r}
|
||||||
not_cancelled |>
|
not_cancelled |>
|
||||||
group_by(year, month, day) |>
|
group_by(year, month, day) |>
|
||||||
summarise(n_early = sum(dep_time < 500)) |>
|
summarise(
|
||||||
|
n_early = sum(dep_time < 500),
|
||||||
|
.groups = "drop"
|
||||||
|
) |>
|
||||||
arrange(desc(n_early))
|
arrange(desc(n_early))
|
||||||
```
|
```
|
||||||
|
|
||||||
There's another useful way to use logical vectors with summaries: to reduce variables to a subset of interest.
|
### Logical subsetting
|
||||||
This makes use of the base `[` (pronounced subset) operator.
|
|
||||||
You'll learn more about this in Section \@ref(vector-subsetting), but this usage works in a similar way to a `filter()` except that instead of applying to entire data frame it applies to a single variable.
|
There's one final use for logical vectors in summaries: you can use a logical vector to filter a single variable to a subset of interest.
|
||||||
|
This makes use of the base `[` (pronounced subset) operator, which you'll learn more about this in Section \@ref(vector-subsetting).
|
||||||
|
|
||||||
|
Imagine we wanted to look at the average delay just for flights that were actually delayed.
|
||||||
|
One way to do so would be to first filter the flights:
|
||||||
|
|
||||||
|
```{r}
|
||||||
|
not_cancelled |>
|
||||||
|
filter(arr_delay > 0) |>
|
||||||
|
group_by(year, month, day) |>
|
||||||
|
summarise(
|
||||||
|
ahead = mean(arr_delay),
|
||||||
|
n = n(),
|
||||||
|
.groups = "drop"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
This works, but what if we wanted to also compute the average delay for flights that left early?
|
||||||
|
We'd need to perform a separate filter step, and then figure out how to combine the two data frames together (which we'll cover in Chapter \@ref(relational-data)).
|
||||||
|
Instead you could use `[` to perform an inline filtering: `arr_delay[arr_delay > 0]` will yield only the positive arrival delays.
|
||||||
|
|
||||||
|
This leads to:
|
||||||
|
|
||||||
```{r}
|
```{r}
|
||||||
not_cancelled |>
|
not_cancelled |>
|
||||||
|
@ -324,15 +377,19 @@ not_cancelled |>
|
||||||
summarise(
|
summarise(
|
||||||
ahead = mean(arr_delay[arr_delay > 0]),
|
ahead = mean(arr_delay[arr_delay > 0]),
|
||||||
behind = mean(arr_delay[arr_delay < 0]),
|
behind = mean(arr_delay[arr_delay < 0]),
|
||||||
|
n = n(),
|
||||||
|
.groups = "drop"
|
||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Also note the difference in the group size: in the first chunk `n` gives the number of delayed flights per day; in the second, `n` gives the total number of flights.
|
||||||
|
|
||||||
### Exercises
|
### Exercises
|
||||||
|
|
||||||
1. For each plane, count the number of flights before the first delay of greater than 1 hour.
|
1. What will `sum(is.na(x))` tell you? How about `mean(is.na(x))`?
|
||||||
2. What does `prod()` return when applied to a logical vector? What logical summary function is it equivalent to? What does `min()` return applied to a logical vector? What logical summary function is it equivalent to?
|
2. What does `prod()` return when applied to a logical vector? What logical summary function is it equivalent to? What does `min()` return applied to a logical vector? What logical summary function is it equivalent to? Read the documentation and perform a few experiments.
|
||||||
|
|
||||||
## Conditonal transformations
|
## Conditional transformations
|
||||||
|
|
||||||
One of the most powerful features of logical vectors are their use for conditional transformations, i.e. returning one value for true values, and a different value for false values.
|
One of the most powerful features of logical vectors are their use for conditional transformations, i.e. returning one value for true values, and a different value for false values.
|
||||||
We'll see a couple of different ways to do this, and the
|
We'll see a couple of different ways to do this, and the
|
||||||
|
@ -371,53 +428,70 @@ Instead, you can switch to `case_when()` instead.
|
||||||
|
|
||||||
`case_when()` has a special syntax that unfortunately looks like nothing else you'll use in the tidyverse.
|
`case_when()` has a special syntax that unfortunately looks like nothing else you'll use in the tidyverse.
|
||||||
it takes pairs that look like `condition ~ output`.
|
it takes pairs that look like `condition ~ output`.
|
||||||
`condition` must evaluate to a logical vector; when it's `TRUE`, output will be used.
|
`condition` must make a logical a logical vector; when it's `TRUE`, `output` will be used.
|
||||||
|
|
||||||
```{r}
|
```{r}
|
||||||
df |>
|
flights |>
|
||||||
mutate(
|
mutate(
|
||||||
status = case_when(
|
status = case_when(
|
||||||
balance == 0 ~ "no money",
|
is.na(arr_delay) ~ "cancelled",
|
||||||
balance < 0 ~ "overdraft",
|
arr_delay > 60 ~ "very late",
|
||||||
balance > 0 ~ "ok"
|
arr_delay > 15 ~ "late",
|
||||||
)
|
abs(arr_delay) <= 15 ~ "on time",
|
||||||
|
arr_delay < -15 ~ "early",
|
||||||
|
arr_delay < -30 ~ "very early",
|
||||||
|
),
|
||||||
|
.keep = "used"
|
||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
(Note that I usually add spaces to make the outputs line up so it's easier to scan)
|
(Note that I usually add spaces to make the outputs line up so it's easier to scan)
|
||||||
|
|
||||||
If none of the cases match, the output will be missing:
|
To explain how `case_when()` works, lets pull it out of the mutate and create some simple dummy data.
|
||||||
|
|
||||||
```{r}
|
```{r}
|
||||||
x <- 1:10
|
x <- 1:10
|
||||||
case_when(
|
case_when(
|
||||||
x %% 2 == 0 ~ "even",
|
x < 5 ~ "small",
|
||||||
|
x >= 5 ~ "big"
|
||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
You can create a catch all value by using `TRUE` as the condition:
|
- If none of the cases match, the output will be missing:
|
||||||
|
|
||||||
```{r}
|
```{r}
|
||||||
case_when(
|
case_when(
|
||||||
x %% 2 == 0 ~ "even",
|
x %% 2 == 0 ~ "even",
|
||||||
TRUE ~ "odd"
|
)
|
||||||
)
|
```
|
||||||
```
|
|
||||||
|
|
||||||
If multiple conditions are `TRUE`, the first is used:
|
- You can create a catch all value by using `TRUE` as the condition:
|
||||||
|
|
||||||
```{r}
|
```{r}
|
||||||
case_when(
|
case_when(
|
||||||
x < 5 ~ "< 5",
|
x %% 2 == 0 ~ "even",
|
||||||
x < 3 ~ "< 3",
|
TRUE ~ "odd"
|
||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- If multiple conditions are `TRUE`, the first is used:
|
||||||
|
|
||||||
|
```{r}
|
||||||
|
case_when(
|
||||||
|
x < 5 ~ "< 5",
|
||||||
|
x < 3 ~ "< 3",
|
||||||
|
TRUE ~ "big"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
The simple examples I've shown you here all use just a single variable, but the logical conditions can use any number of variables.
|
||||||
|
And you can use variables on the right hand side.
|
||||||
|
|
||||||
## Cumulative tricks
|
## Cumulative tricks
|
||||||
|
|
||||||
Before we move on to the next chapter, I want to show you a grab bag of tricks that make use of cumulative functions (i.e. functions that depending on every previous value of a vector in some way).
|
Before we move on to the next chapter, I want to show you a grab bag of tricks that make use of cumulative functions (i.e. functions that depending on every previous value of a vector).
|
||||||
These all feel a bit magical, and I'm torn on whether or not they should be included in this book.
|
These all feel a bit magical, and I'm torn on whether or not they should be included in this book.
|
||||||
But in the end, some of them are just so useful I think it's important to mention them --- they don't help with that many problems, but when they do, they provide a substantial advantage.
|
But in the end, some of them are just so useful I think it's important to mention them --- they're not particularly easy to understand and don't help with that many problems, but when they do, they provide a substantial advantage.
|
||||||
|
|
||||||
<!-- TODO: illustration of accumulating function -->
|
<!-- TODO: illustration of accumulating function -->
|
||||||
|
|
||||||
|
@ -454,9 +528,12 @@ df |> filter(cumall(!(balance < 0)))
|
||||||
```{r}
|
```{r}
|
||||||
df |>
|
df |>
|
||||||
mutate(
|
mutate(
|
||||||
flip = (balance < 0) != lag(balance < 0),
|
negative = balance < 0,
|
||||||
|
flip = negative != lag(negative),
|
||||||
group = cumsum(coalesce(flip, FALSE))
|
group = cumsum(coalesce(flip, FALSE))
|
||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
##
|
### Exercises
|
||||||
|
|
||||||
|
1. For each plane, count the number of flights before the first delay of greater than 1 hour.
|
||||||
|
|
|
@ -85,6 +85,10 @@ There are a couple of related counts that you might find useful:
|
||||||
### Exercises
|
### Exercises
|
||||||
|
|
||||||
1. How can you use `count()` to count the number rows with a missing value for a given variable?
|
1. How can you use `count()` to count the number rows with a missing value for a given variable?
|
||||||
|
2. Expand the following calls to `count()` to use the core verbs of dplyr:
|
||||||
|
1. `flights |> count(dest, sort = TRUE)`
|
||||||
|
|
||||||
|
2. `flights |> count(tailnum, wt = distance)`
|
||||||
|
|
||||||
## Numeric transformations
|
## Numeric transformations
|
||||||
|
|
||||||
|
@ -341,7 +345,7 @@ flights |>
|
||||||
The chief advantage of `first()` and `nth()` over `[` is that you can set a default value if that position does not exist (i.e. you're trying to get the 3rd element from a group that only has two elements).
|
The chief advantage of `first()` and `nth()` over `[` is that you can set a default value if that position does not exist (i.e. you're trying to get the 3rd element from a group that only has two elements).
|
||||||
The chief advantage of `last()` over `[`, is writing `last(x)` rather than `x[length(x)]`.
|
The chief advantage of `last()` over `[`, is writing `last(x)` rather than `x[length(x)]`.
|
||||||
|
|
||||||
Additioanlly, if the rows aren't ordered, but there's a variable that defines the order, you can use `order_by` argument.
|
Additionally, if the rows aren't ordered, but there's a variable that defines the order, you can use `order_by` argument.
|
||||||
You can do this with `[` + `order_by()` but it requires a little thought.
|
You can do this with `[` + `order_by()` but it requires a little thought.
|
||||||
|
|
||||||
Computing positions is complementary to filtering on ranks.
|
Computing positions is complementary to filtering on ranks.
|
||||||
|
@ -482,7 +486,7 @@ We've seen a few variants of different functions
|
||||||
| `sum` | `cumsum` | `+` |
|
| `sum` | `cumsum` | `+` |
|
||||||
| `prod` | `cumprod` | `*` |
|
| `prod` | `cumprod` | `*` |
|
||||||
| `all` | `cumall` | `&` |
|
| `all` | `cumall` | `&` |
|
||||||
| `any` | `cumany` | `\|` |
|
| `any` | `cumany` | `|` |
|
||||||
| `min` | `cummin` | `pmin` |
|
| `min` | `cummin` | `pmin` |
|
||||||
| `max` | `cummax` | `pmax` |
|
| `max` | `cummax` | `pmax` |
|
||||||
|
|
||||||
|
|
Loading…
Reference in New Issue