In this chapter, you'll learn the art of data **rectangling**, taking data that is fundamentally tree-like and converting it into a rectangular data frames made up of rows and columns.
This is important because hierarchical data is surprisingly common, especially when working with data that comes from a web API.
To learn about rectangling, you'll first learn about lists, the data structure that makes hierarchical data possible in R.
Then you'll learn about two crucial tidyr functions: `tidyr::unnest_longer()`, which converts children in rows, and `tidyr::unnest_wider()`, which converts children into columns.
We'll then show you a few case studies, applying these simple function multiple times to solve real complex problems.
We'll finish off by talking about JSON, the most frequent source of hierarchical datasets and common format for data exchange on the web.
We'll also use repurrrsive to supply some interesting datasets to practice your rectangling skills, and we'll finish up with a little jsonlite, which we'll use to read JSON files into R lists.
The viewer starts by showing just the top level of the list, but you can interactively expand any of the components to see more, as in @fig-view-expand-1.
This is a powerful idea because it allows you to store arbitrarily complex objects in a data frame; even things that wouldn't typically belong there.
This idea is used a lot in tidymodels, because it allows you to store things like models or resamples in a data frame.
And those things are carried along like any other column:
```{r}
df |>
filter(x == 1)
```
The default print method just displays a rough summary of the contents.
The list column could be arbitrarily complex, so there's no good way to print it.
If you want to see it, you'll need to pull the list-column out and apply of the techniques that you learned above:
```{r}
df |>
filter(x == 1) |>
pull(z) |>
str()
```
Similarly, if you `View()` a data frame in RStudio, you'll get the standard tabular view, which doesn't allow you to selectively expand list columns.
To explore those fields you'll need to `pull()` and view, e.g.
`View(pull(df, z))`
::: callout-note
## Base R
It's possible to put a list in a column of a `data.frame`, but it's a lot fiddlier.
List-columns are implicit in the definition of the data frame: a data frame is a named list of equal length vectors.
A list is a vector, so it's always been legitimate to use a list as a column of a data frame.
However, base R doesn't make it easy to create list-columns because `data.frame()`treats a list as a list of columns:
```{r}
data.frame(x = list(1:3, 3:5))
```
You can prevent `data.frame()` from doing this with `I()`, but the result doesn't print particularly well:
```{r}
data.frame(
x = I(list(1:3, 3:5)),
y = c("1, 2", "3, 4, 5")
)
```
Tibbles make it easier to work with list-columns because `tibble()` doesn't modify its inputs and the print method is designed with lists in mind.
:::
## Unnesting
Now that you've learned the basics of lists and how you can use them as a column of a data frame, lets start to see how you can turn them back into regular columns and rows so you can use them with the tidyverse functions you've already learned about.
We'll start with very simple sample data so you can get the idea of how things work, and then in the next section switch to more realistic examples.
These two cases correspond to two tools from tidyr: `unnest_wider()` and `unnest_longer()`.
Their suffixes have the same meaning as `pivot_wider()` and `pivot_longer()`: `_wider()` adds more columns and `_longer()` adds more rows.
If your situation isn't as clear cut as these cases, you'll still need to use one of `unnest_longer()` and `unnest_wider()`; you'll just need to do a bit more thinking and experimentation to figure out which one is best.
When each row has the same number of elements with the same names, like `df1`, it's natural to put each component into its own column with `unnest_wider()`:
By default, the names of the new columns come exclusively from the names of the list, but you can use the `names_sep` argument to request that they combine the original column with the new column.
As you'll learn in the next section, this is useful for disambiguating repeated names.
What happens if you attempt to unnest a column that doesn't contain only one type of thing.
For example, what happens if we take this data set and unnest into rows?
`y` will contain two numbers, a factor, a logical, which can't normally be mixed in a single column:
```{r}
df4 <- tribble(
~x, ~y,
"a", list(1, "a"),
"b", list(TRUE, factor("a"), 5)
)
```
An important invariant for `unnest_longer()` is that the columns say the same but the number of rows change.
So what happens?
How does `unnest_longer()` produce five rows while keeping everything in `y`?
```{r}
df4 |> unnest_longer(y)
```
We still get a list-column, but every element of the list-column contains a single element.
When `unnest_longer()` can't find a common type, it keeps the original types by using a list-column.
You might wonder if this breaks the commandment that every element of a column must be the same type --- not quite, because every element is a still a list, but each component of a list can contain something different.
What happens if you find this problem in a dataset you're trying to rectangle?
I think there are two basic options.
You could try and coerce to a class that is meaningful for all the rows using the `transform` argument.
It's not particularly useful here because there's only really one class that these five class can be converted to: character.
```{r}
df4 |> unnest_longer(y, transform = as.character)
```
Another option would be to filter down to the rows that have values of a specific type:
There are few other useful rectangling functions that we're not going to talk about here:
- `unnest_auto()` automatically picks between `unnest_longer()` and `unnest_wider()`based on the structure of the list-column. It's a great for rapid exploration, but I think it's ultimately a bad idea because it doesn't force you to understand how your data is structured, and makes your code harder to understand.
- `unnest()` modifies rows and columns simultaneously. It's useful when you have a list-column that contains a 2d structure like a data frame (which we often call a nested data frame), which we don't otherwise use in this book.
- `hoist()` allows you to reach into a deeply nested list and extract just the components that you need. It's mostly equivalent to repeated invocations of `unnest_wider()` + `select()` so you should read up on it if there's just a couple of important variables that you want to pull out, embedded in a bunch of data that you don't care about.
1. From time-to-time you encounter data frames with multiple list-columns with aligned values.
For example, in the following data frame, the values of `y` and `z` are aligned (i.e. `y` and `z` will always have the same length within a row, and the first value of `y` corresponds to the first value of `z`).
What happens if you apply two `unnest_longer()` calls to this data frame?
How can you preserve the relationship between `x` and `y`?
Now that you understand the basics of `unnest_wider()` and `unnest_longer()` lets use them to tackle some real rectangling challenges.
These challenges share the common feature that they're mostly just a sequence of multiple `unnest_wider()` and/or `unnest_longer()` calls, with a little dash of dplyr where needed.
We'll start with `gh_repos` --- this is some data about GitHub repositories retrived from GitHub API. It's a very deeply nested list so it's hard for me to display in this book; you might want to explore a little on your own with `View(gh_repos)` before we continue.
To make it more manageable I'm going to put it in a tibble in a column called `json` (for reasons we'll get to later)
At first glance, it might seem like we haven't improved the situation --- while we have more rows now (176 instead of 6) it seems like each element of `json` is still a list.
However, there's an important difference: now each element is a **named** list so we can use `unnamed_wider()` to put each element into its own column:
```{r}
repos |>
unnest_longer(json) |>
unnest_wider(json)
```
This is a bit overwhelming --- there are so many columns that tibble doesn't even print all of them!
We'll finish off with an that is very deeply nested and requires repeated rounds of `unnest_wider()` and `unnest_longer()` to unravel: `gmaps_cities`.
This is a two column tibble containing five cities names and the results of using Google's [geocoding API](https://developers.google.com/maps/documentation/geocoding) to determine their location:
```{r}
gmaps_cities
```
`json` is list-column with internal names, so we start with an `unnest_wider()`:
```{r}
gmaps_cities |>
unnest_wider(json)
```
This gives us a status column and the actual results.
We'll drop the status column since they're all `OK`.
In a real analysis, you'd also want separately capture all the rows where `status != "OK"` so you could figure out what went wrong.
`results` is an unnamed list, with either one or two elements.
We'll figure to out why shortly.
```{r}
gmaps_cities |>
unnest_wider(json) |>
select(-status) |>
unnest_longer(results)
```
Now results is a named list, so we'll `unnest_wider()`:
```{r}
locations <- gmaps_cities |>
unnest_wider(json) |>
select(-status) |>
unnest_longer(results) |>
unnest_wider(results)
locations
```
Now we can see why Washington and Arlington got two results: Washington matched both the state and the city (DC), and Arlington matched Arlington Virginia and Arlington Texas.
There are few different places we could go from here.
We might want to determine the exact location of the match stored in the `geometry` list-column:
```{r}
locations |>
select(city, formatted_address, geometry) |>
unnest_wider(geometry)
```
That gives us new `bounds` (which gives a rectangular region) and the midpoint in `location`, which we can unnest to get latitude (`lat`) and longitude (`lng`):
```{r}
locations |>
select(city, formatted_address, geometry) |>
unnest_wider(geometry) |>
unnest_wider(location)
```
Extracting the bounds requires a few more steps
```{r}
locations |>
select(city, formatted_address, geometry) |>
unnest_wider(geometry) |>
# focus on the variables of interest
select(!location:viewport) |>
unnest_wider(bounds)
```
I then rename `southwest` and `northeast` (the corners of the rectangle) so I can use `names_sep` to create short but evocative names:
```{r}
locations |>
select(city, formatted_address, geometry) |>
unnest_wider(geometry) |>
select(!location:viewport) |>
unnest_wider(bounds) |>
rename(ne = northeast, sw = southwest) |>
unnest_wider(c(ne, sw), names_sep = "_")
```
Note that I take advantage of the fact that you can unnest multiple columns at a time by supplying a vector of variable names to `unnest_wider()`.
This one place where `hoist()`, which we mentioned briefly above can be useful.
Once you've discovered the path to get to the components you're interested in, you can extract them directly using `hoist()`:
JSON, short for javascript object notation, is a data format that grew out of the javascript programming language and has become an extremely common way of representing data.
