335 lines
19 KiB
HTML
335 lines
19 KiB
HTML
<section data-type="chapter" id="chp-missing-values">
|
||
<h1><span id="sec-missing-values" class="quarto-section-identifier d-none d-lg-block"><span class="chapter-title">Missing values</span></span></h1>
|
||
<section id="introduction" data-type="sect1">
|
||
<h1>
|
||
Introduction</h1>
|
||
<p>You’ve already learned the basics of missing values earlier in the book. You first saw them in <a href="#sec-summarize" data-type="xref">#sec-summarize</a> where they interfered with computing summary statistics, and you learned about their infectious nature and how to check for their presence in <a href="#sec-na-comparison" data-type="xref">#sec-na-comparison</a>. Now we’ll come back to them in more depth, so you can learn more of the details.</p>
|
||
<p>We’ll start by discussing some general tools for working with missing values recorded as <code>NA</code>s. We’ll then explore the idea of implicitly missing values, values are that are simply absent from your data, and show some tools you can use to make them explicit. We’ll finish off with a related discussion of empty groups, caused by factor levels that don’t appear in the data.</p>
|
||
|
||
<section id="prerequisites" data-type="sect2">
|
||
<h2>
|
||
Prerequisites</h2>
|
||
<p>The functions for working with missing data mostly come from dplyr and tidyr, which are core members of the tidyverse.</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">library(tidyverse)</pre>
|
||
</div>
|
||
</section>
|
||
</section>
|
||
|
||
<section id="explicit-missing-values" data-type="sect1">
|
||
<h1>
|
||
Explicit missing values</h1>
|
||
<p>To begin, let’s explore a few handy tools for creating or eliminating missing explicit values, i.e. cells where you see an <code>NA</code>.</p>
|
||
|
||
<section id="last-observation-carried-forward" data-type="sect2">
|
||
<h2>
|
||
Last observation carried forward</h2>
|
||
<p>A common use for missing values is as a data entry convenience. When data is entered by hand, missing values sometimes indicate that the value in the previous row has been repeated (or carried forward):</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">treatment <- tribble(
|
||
~person, ~treatment, ~response,
|
||
"Derrick Whitmore", 1, 7,
|
||
NA, 2, 10,
|
||
NA, 3, NA,
|
||
"Katherine Burke", 1, 4
|
||
)</pre>
|
||
</div>
|
||
<p>You can fill in these missing values with <code><a href="https://tidyr.tidyverse.org/reference/fill.html">tidyr::fill()</a></code>. It works like <code><a href="https://dplyr.tidyverse.org/reference/select.html">select()</a></code>, taking a set of columns:</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">treatment |>
|
||
fill(everything())
|
||
#> # A tibble: 4 × 3
|
||
#> person treatment response
|
||
#> <chr> <dbl> <dbl>
|
||
#> 1 Derrick Whitmore 1 7
|
||
#> 2 Derrick Whitmore 2 10
|
||
#> 3 Derrick Whitmore 3 10
|
||
#> 4 Katherine Burke 1 4</pre>
|
||
</div>
|
||
<p>This treatment is sometimes called “last observation carried forward”, or <strong>locf</strong> for short. You can use the <code>.direction</code> argument to fill in missing values that have been generated in more exotic ways.</p>
|
||
</section>
|
||
|
||
<section id="fixed-values" data-type="sect2">
|
||
<h2>
|
||
Fixed values</h2>
|
||
<p>Some times missing values represent some fixed and known value, most commonly 0. You can use <code><a href="https://dplyr.tidyverse.org/reference/coalesce.html">dplyr::coalesce()</a></code> to replace them:</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">x <- c(1, 4, 5, 7, NA)
|
||
coalesce(x, 0)
|
||
#> [1] 1 4 5 7 0</pre>
|
||
</div>
|
||
<p>Sometimes you’ll hit the opposite problem where some concrete value actually represents a missing value. This typically arises in data generated by older software that doesn’t have a proper way to represent missing values, so it must instead use some special value like 99 or -999.</p>
|
||
<p>If possible, handle this when reading in the data, for example, by using the <code>na</code> argument to <code><a href="https://readr.tidyverse.org/reference/read_delim.html">readr::read_csv()</a></code>. If you discover the problem later, or your data source doesn’t provide a way to handle on it read, you can use <code><a href="https://dplyr.tidyverse.org/reference/na_if.html">dplyr::na_if()</a></code>:</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">x <- c(1, 4, 5, 7, -99)
|
||
na_if(x, -99)
|
||
#> [1] 1 4 5 7 NA</pre>
|
||
</div>
|
||
</section>
|
||
|
||
<section id="nan" data-type="sect2">
|
||
<h2>
|
||
NaN</h2>
|
||
<p>Before we continue, there’s one special type of missing value that you’ll encounter from time to time: a <code>NaN</code> (pronounced “nan”), or <strong>n</strong>ot <strong>a</strong> <strong>n</strong>umber. It’s not that important to know about because it generally behaves just like <code>NA</code>:</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">x <- c(NA, NaN)
|
||
x * 10
|
||
#> [1] NA NaN
|
||
x == 1
|
||
#> [1] NA NA
|
||
is.na(x)
|
||
#> [1] TRUE TRUE</pre>
|
||
</div>
|
||
<p>In the rare case you need to distinguish an <code>NA</code> from a <code>NaN</code>, you can use <code>is.nan(x)</code>.</p>
|
||
<p>You’ll generally encounter a <code>NaN</code> when you perform a mathematical operation that has an indeterminate result:</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">0 / 0
|
||
#> [1] NaN
|
||
0 * Inf
|
||
#> [1] NaN
|
||
Inf - Inf
|
||
#> [1] NaN
|
||
sqrt(-1)
|
||
#> Warning in sqrt(-1): NaNs produced
|
||
#> [1] NaN</pre>
|
||
</div>
|
||
</section>
|
||
</section>
|
||
|
||
<section id="sec-missing-implicit" data-type="sect1">
|
||
<h1>
|
||
Implicit missing values</h1>
|
||
<p>So far we’ve talked about missing values that are <strong>explicitly</strong> missing, i.e. you can see an <code>NA</code> in your data. But missing values can also be <strong>implicitly</strong> missing, if an entire row of data is simply absent from the data. Let’s illustrate the difference with a simple data set that records the price of some stock each quarter:</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">stocks <- tibble(
|
||
year = c(2020, 2020, 2020, 2020, 2021, 2021, 2021),
|
||
qtr = c( 1, 2, 3, 4, 2, 3, 4),
|
||
price = c(1.88, 0.59, 0.35, NA, 0.92, 0.17, 2.66)
|
||
)</pre>
|
||
</div>
|
||
<p>This dataset has two missing observations:</p>
|
||
<ul><li><p>The <code>price</code> in the fourth quarter of 2020 is explicitly missing, because its value is <code>NA</code>.</p></li>
|
||
<li><p>The <code>price</code> for the first quarter of 2021 is implicitly missing, because it simply does not appear in the dataset.</p></li>
|
||
</ul><p>One way to think about the difference is with this Zen-like koan:</p>
|
||
<blockquote class="blockquote">
|
||
<p>An explicit missing value is the presence of an absence.<br/></p>
|
||
<p>An implicit missing value is the absence of a presence.</p>
|
||
</blockquote>
|
||
<p>Sometimes you want to make implicit missings explicit in order to have something physical to work with. In other cases, explicit missings are forced upon you by the structure of the data and you want to get rid of them. The following sections discuss some tools for moving between implicit and explicit missingness.</p>
|
||
|
||
<section id="pivoting" data-type="sect2">
|
||
<h2>
|
||
Pivoting</h2>
|
||
<p>You’ve already seen one tool that can make implicit missings explicit and vice versa: pivoting. Making data wider can make implicit missing values explicit because every combination of the rows and new columns must have some value. For example, if we pivot <code>stocks</code> to put the <code>quarter</code> in the columns, both missing values become explicit:</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">stocks |>
|
||
pivot_wider(
|
||
names_from = qtr,
|
||
values_from = price
|
||
)
|
||
#> # A tibble: 2 × 5
|
||
#> year `1` `2` `3` `4`
|
||
#> <dbl> <dbl> <dbl> <dbl> <dbl>
|
||
#> 1 2020 1.88 0.59 0.35 NA
|
||
#> 2 2021 NA 0.92 0.17 2.66</pre>
|
||
</div>
|
||
<p>By default, making data longer preserves explicit missing values, but if they are structurally missing values that only exist because the data is not tidy, you can drop them (make them implicit) by setting <code>values_drop_na = TRUE</code>. See the examples in <a href="#sec-tidy-data" data-type="xref">#sec-tidy-data</a> for more details.</p>
|
||
</section>
|
||
|
||
<section id="complete" data-type="sect2">
|
||
<h2>
|
||
Complete</h2>
|
||
<p><code><a href="https://tidyr.tidyverse.org/reference/complete.html">tidyr::complete()</a></code> allows you to generate explicit missing values by providing a set of variables that define the combination of rows that should exist. For example, we know that all combinations of <code>year</code> and <code>qtr</code> should exist in the <code>stocks</code> data:</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">stocks |>
|
||
complete(year, qtr)
|
||
#> # A tibble: 8 × 3
|
||
#> year qtr price
|
||
#> <dbl> <dbl> <dbl>
|
||
#> 1 2020 1 1.88
|
||
#> 2 2020 2 0.59
|
||
#> 3 2020 3 0.35
|
||
#> 4 2020 4 NA
|
||
#> 5 2021 1 NA
|
||
#> 6 2021 2 0.92
|
||
#> # … with 2 more rows</pre>
|
||
</div>
|
||
<p>Typically, you’ll call <code><a href="https://tidyr.tidyverse.org/reference/complete.html">complete()</a></code> with names of existing variables, filling in the missing combinations. However, sometimes the individual variables are themselves incomplete, so you can instead provide your own data. For example, you might know that the <code>stocks</code> dataset is supposed to run from 2019 to 2021, so you could explicitly supply those values for <code>year</code>:</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">stocks |>
|
||
complete(year = 2019:2021, qtr)
|
||
#> # A tibble: 12 × 3
|
||
#> year qtr price
|
||
#> <dbl> <dbl> <dbl>
|
||
#> 1 2019 1 NA
|
||
#> 2 2019 2 NA
|
||
#> 3 2019 3 NA
|
||
#> 4 2019 4 NA
|
||
#> 5 2020 1 1.88
|
||
#> 6 2020 2 0.59
|
||
#> # … with 6 more rows</pre>
|
||
</div>
|
||
<p>If the range of a variable is correct, but not all values are present, you could use <code>full_seq(x, 1)</code> to generate all values from <code>min(x)</code> to <code>max(x)</code> spaced out by 1.</p>
|
||
<p>In some cases, the complete set of observations can’t be generated by a simple combination of variables. In that case, you can do manually what <code><a href="https://tidyr.tidyverse.org/reference/complete.html">complete()</a></code> does for you: create a data frame that contains all the rows that should exist (using whatever combination of techniques you need), then combine it with your original dataset with <code><a href="https://dplyr.tidyverse.org/reference/mutate-joins.html">dplyr::full_join()</a></code>.</p>
|
||
</section>
|
||
|
||
<section id="joins" data-type="sect2">
|
||
<h2>
|
||
Joins</h2>
|
||
<p>This brings us to another important way of revealing implicitly missing observations: joins. You’ll learn more about joins in <a href="#chp-joins" data-type="xref">#chp-joins</a>, but we wanted to quickly mention them to you here since you can often only know that values are missing from one dataset when you compare it another.</p>
|
||
<p><code>dplyr::anti_join(x, y)</code> is a particularly useful tool here because it selects only the rows in <code>x</code> that don’t have a match in <code>y</code>. For example, we can use two <code><a href="https://dplyr.tidyverse.org/reference/filter-joins.html">anti_join()</a></code>s reveal to reveal that we’re missing information for four airports and 722 planes mentioned in <code>flights</code>:</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">library(nycflights13)
|
||
|
||
flights |>
|
||
distinct(faa = dest) |>
|
||
anti_join(airports)
|
||
#> Joining with `by = join_by(faa)`
|
||
#> # A tibble: 4 × 1
|
||
#> faa
|
||
#> <chr>
|
||
#> 1 BQN
|
||
#> 2 SJU
|
||
#> 3 STT
|
||
#> 4 PSE
|
||
|
||
flights |>
|
||
distinct(tailnum) |>
|
||
anti_join(planes)
|
||
#> Joining with `by = join_by(tailnum)`
|
||
#> # A tibble: 722 × 1
|
||
#> tailnum
|
||
#> <chr>
|
||
#> 1 N3ALAA
|
||
#> 2 N3DUAA
|
||
#> 3 N542MQ
|
||
#> 4 N730MQ
|
||
#> 5 N9EAMQ
|
||
#> 6 N532UA
|
||
#> # … with 716 more rows</pre>
|
||
</div>
|
||
</section>
|
||
|
||
<section id="exercises" data-type="sect2">
|
||
<h2>
|
||
Exercises</h2>
|
||
<ol type="1"><li>Can you find any relationship between the carrier and the rows that appear to be missing from <code>planes</code>?</li>
|
||
</ol></section>
|
||
</section>
|
||
|
||
<section id="factors-and-empty-groups" data-type="sect1">
|
||
<h1>
|
||
Factors and empty groups</h1>
|
||
<p>A final type of missingness is the empty group, a group that doesn’t contain any observations, which can arise when working with factors. For example, imagine we have a dataset that contains some health information about people:</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">health <- tibble(
|
||
name = c("Ikaia", "Oletta", "Leriah", "Dashay", "Tresaun"),
|
||
smoker = factor(c("no", "no", "no", "no", "no"), levels = c("yes", "no")),
|
||
age = c(34L, 88L, 75L, 47L, 56L),
|
||
)</pre>
|
||
</div>
|
||
<p>And we want to count the number of smokers with <code><a href="https://dplyr.tidyverse.org/reference/count.html">dplyr::count()</a></code>:</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">health |> count(smoker)
|
||
#> # A tibble: 1 × 2
|
||
#> smoker n
|
||
#> <fct> <int>
|
||
#> 1 no 5</pre>
|
||
</div>
|
||
<p>This dataset only contains non-smokers, but we know that smokers exist; the group of non-smoker is empty. We can request <code><a href="https://dplyr.tidyverse.org/reference/count.html">count()</a></code> to keep all the groups, even those not seen in the data by using <code>.drop = FALSE</code>:</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">health |> count(smoker, .drop = FALSE)
|
||
#> # A tibble: 2 × 2
|
||
#> smoker n
|
||
#> <fct> <int>
|
||
#> 1 yes 0
|
||
#> 2 no 5</pre>
|
||
</div>
|
||
<p>The same principle applies to ggplot2’s discrete axes, which will also drop levels that don’t have any values. You can force them to display by supplying <code>drop = FALSE</code> to the appropriate discrete axis:</p>
|
||
<div>
|
||
<pre data-type="programlisting" data-code-language="r">ggplot(health, aes(smoker)) +
|
||
geom_bar() +
|
||
scale_x_discrete()
|
||
|
||
ggplot(health, aes(smoker)) +
|
||
geom_bar() +
|
||
scale_x_discrete(drop = FALSE)</pre>
|
||
<div class="cell quarto-layout-panel">
|
||
<div class="quarto-layout-row quarto-layout-valign-top">
|
||
<div class="cell-output-display quarto-layout-cell" style="flex-basis: 50.0%;justify-content: center;">
|
||
<p><img src="missing-values_files/figure-html/unnamed-chunk-17-1.png" class="img-fluid" alt="A bar chart with a single value on the x-axis, "no". The same bar chart as the last plot, but now with two values on the x-axis, "yes" and "no". There is no bar for the "yes" category." width="288"/></p>
|
||
</div>
|
||
<div class="cell-output-display quarto-layout-cell" style="flex-basis: 50.0%;justify-content: center;">
|
||
<p><img src="missing-values_files/figure-html/unnamed-chunk-17-2.png" class="img-fluid" alt="A bar chart with a single value on the x-axis, "no". The same bar chart as the last plot, but now with two values on the x-axis, "yes" and "no". There is no bar for the "yes" category." width="288"/></p>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<p>The same problem comes up more generally with <code><a href="https://dplyr.tidyverse.org/reference/group_by.html">dplyr::group_by()</a></code>. And again you can use <code>.drop = FALSE</code> to preserve all factor levels:</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">health |>
|
||
group_by(smoker, .drop = FALSE) |>
|
||
summarise(
|
||
n = n(),
|
||
mean_age = mean(age),
|
||
min_age = min(age),
|
||
max_age = max(age),
|
||
sd_age = sd(age)
|
||
)
|
||
#> Warning: There were 2 warnings in `summarise()`.
|
||
#> The first warning was:
|
||
#> ℹ In argument `min_age = min(age)`.
|
||
#> ℹ In group 1: `smoker = yes`.
|
||
#> Caused by warning in `min()`:
|
||
#> ! no non-missing arguments to min; returning Inf
|
||
#> ℹ Run `dplyr::last_dplyr_warnings()` to see the 1 remaining warning.
|
||
#> # A tibble: 2 × 6
|
||
#> smoker n mean_age min_age max_age sd_age
|
||
#> <fct> <int> <dbl> <dbl> <dbl> <dbl>
|
||
#> 1 yes 0 NaN Inf -Inf NA
|
||
#> 2 no 5 60 34 88 21.6</pre>
|
||
</div>
|
||
<p>We get some interesting results here because when summarizing an empty group, the summary functions are applied to zero-length vectors. There’s an important distinction between empty vectors, which have length 0, and missing values, each of which has length 1.</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r"># A vector containing two missing values
|
||
x1 <- c(NA, NA)
|
||
length(x1)
|
||
#> [1] 2
|
||
|
||
# A vector containing nothing
|
||
x2 <- numeric()
|
||
length(x2)
|
||
#> [1] 0</pre>
|
||
</div>
|
||
<p>All summary functions work with zero-length vectors, but they may return results that are surprising at first glance. Here we see <code>mean(age)</code> returning <code>NaN</code> because <code>mean(age)</code> = <code>sum(age)/length(age)</code> which here is 0/0. <code><a href="https://rdrr.io/r/base/Extremes.html">max()</a></code> and <code><a href="https://rdrr.io/r/base/Extremes.html">min()</a></code> return -Inf and Inf for empty vectors so if you combine the results with a non-empty vector of new data and recompute you’ll get the minimum or maximum of the new data<span data-type="footnote">In other words, <code>min(c(x, y))</code> is always equal to <code>min(min(x), min(y))</code>.</span>.</p>
|
||
<p>Sometimes a simpler approach is to perform the summary and then make the implicit missings explicit with <code><a href="https://tidyr.tidyverse.org/reference/complete.html">complete()</a></code>.</p>
|
||
<div class="cell">
|
||
<pre data-type="programlisting" data-code-language="r">health |>
|
||
group_by(smoker) |>
|
||
summarise(
|
||
n = n(),
|
||
mean_age = mean(age),
|
||
min_age = min(age),
|
||
max_age = max(age),
|
||
sd_age = sd(age)
|
||
) |>
|
||
complete(smoker)
|
||
#> # A tibble: 2 × 6
|
||
#> smoker n mean_age min_age max_age sd_age
|
||
#> <fct> <int> <dbl> <int> <int> <dbl>
|
||
#> 1 yes NA NA NA NA NA
|
||
#> 2 no 5 60 34 88 21.6</pre>
|
||
</div>
|
||
<p>The main drawback of this approach is that you get an <code>NA</code> for the count, even though you know that it should be zero.</p>
|
||
</section>
|
||
|
||
<section id="summary" data-type="sect1">
|
||
<h1>
|
||
Summary</h1>
|
||
<p>Missing values are weird! Sometimes they’re recorded as an explicit <code>NA</code> but other times you only notice them by their absence. This chapter has given you some tools for working with explicit missing values, tools for uncovering implicit missing values, and discussed some of the ways that implicit can become explicit and vice versa.</p>
|
||
<p>In the next chapter, we tackle the final chapter in this part of the book: joins. This is a bit of a change from the chapters so far because we’re going to discuss tools that work with data frames as a whole, not something that you put inside a data frame.</p>
|
||
|
||
|
||
</section>
|
||
</section>
|