Polishing strings

This commit is contained in:
Hadley Wickham 2022-01-19 19:35:16 -06:00
parent 9d67d622ad
commit d17da6d661
1 changed files with 90 additions and 83 deletions

View File

@ -10,25 +10,25 @@ So far, you've used a bunch of strings without learning much about the details.
Now it's time to dive into them, learning what makes strings tick, and mastering some of the powerful string manipulation tool you have at your disposal.
We'll begin with the details of creating strings and character vectors.
You'll then learn a grab bag of handy string functions before we dive into creating strings from data.
You'll then dive into creating strings from data.
Next, we'll discuss the basics of regular expressions, a powerful tool for describing patterns in strings, then use those tools to extract data from strings.
The chapter finishes up with a brief discussion where English language expectations might steer you wrong when working with text from other languages.
The chapter finishes up with functions that work with individual letters, a brief discussion of where your expectations from English might steer you wrong when working with other languages, and a few useful non-stringr functions.
This chapter is paired with two other chapters.
Regular expression are a big topic, so we'll come back to them again in Chapter \@ref(regular-expressions).
We'll come back to strings again in Chapter \@ref(programming-with-strings) where we'll think about them about more from a programming perspective than a data analysis perspective.
We'll also come back to strings again in Chapter \@ref(programming-with-strings) where we'll look at them from a programming perspective rather than a data analysis perspective.
### Prerequisites
In this chapter, we'll use functions from the stringr package.
The equivalent functionality is available in base R (through functions like `grepl()`, `gsub()`, and `regmatches()`) but we think you'll find stringr easier to use because it's been carefully designed to be as consistent as possible.
We'll also work with the babynames data since it provides some fun strings to manipulate.
In this chapter, we'll use functions from the stringr package which is part of the core tidyverse.
We'll also use the babynames data since it provides some fun strings to manipulate.
```{r setup, message = FALSE}
library(tidyverse)
library(babynames)
```
Similar functionality is available in base R (through functions like `grepl()`, `gsub()`, and `regmatches()`) but we think you'll find stringr easier to use because it's been carefully designed to be as consistent as possible.
You can easily tell when you're using a stringr function because all stringr functions start with `str_`.
This is particularly useful if you use RStudio, because typing `str_` will trigger autocomplete, allowing you jog your memory of which functions are available.
@ -39,7 +39,7 @@ knitr::include_graphics("screenshots/stringr-autocomplete.png")
## Creating a string
We've created strings in passing earlier in the book, but didn't discuss the details.
First, there are two basic ways to create a string: using either single quotes (`'`) or double quotes (`"`).
First, you can create a string using either single quotes (`'`) or double quotes (`"`).
Unlike other languages, there is no difference in behavior, but in the interests of consistency the [tidyverse style guide](https://style.tidyverse.org/syntax.html#character-vectors) recommends using `"`, unless the string contains multiple `"`.
```{r}
@ -65,7 +65,7 @@ double_quote <- "\"" # or '"'
single_quote <- '\'' # or "'"
```
So if you want to include a literal backslash in your string, you'll need to double it up: `"\\"`:
And if you want to include a literal backslash in your string, you'll need to double it up: `"\\"`:
```{r}
backslash <- "\\"
@ -109,15 +109,15 @@ But if your string contains `)"` you can instead use `r"[]"` or `r"{}"`, and if
### Other special characters
As well as `\"`, `\'`, and `\\` there are a handful of other special characters that may come in handy. The most common are `\n`, newline, and `\t`, tab. You'll also sometimes see strings containing Unicode escapes that start with `\u` or `\U`. This is a way of writing non-English characters that works on all systems:
As well as `\"`, `\'`, and `\\` there are a handful of other special characters that may come in handy. The most common are `\n`, newline, and `\t`, tab. You'll also sometimes see strings containing Unicode escapes that start with `\u` or `\U`. This is a way of writing non-English characters that works on all systems. You can see the complete list of other special characters in `?'"'`.
```{r}
x <- c("\u00b5", "\U0001f604")
x <- c("one\ntwo", "one\ttwo", "\u00b5", "\U0001f604")
x
str_view(x)
```
You can see the complete list of other special characters in `?'"'`.
Note that `str_view()` shows special whitespace characters (i.e. everything except spaces and newlines) with a blue background to make them easier to spot.
### Vectors
@ -129,17 +129,16 @@ x
```
Technically, a string is a length-1 character vector, but this doesn't have much bearing on your data analysis life.
We'll come back to this idea is more detail when we think about vectors from more of a programming perspective in Chapter \@ref(vectors).
Now that you've learned the basics of creating strings by "hand", we'll go into the details of creating strings from other strings.
We'll come back to this idea is more detail when we think about vectors as a programming tool in Chapter \@ref(vectors).
### Exercises
## Creating strings from data
It's a common problem to generate strings from other strings, typically by combining fixed strings that you write with variable strings that come from the data.
Now that you've learned the basics of creating strings by "hand", we'll go into the details of creating strings from other strings.
It's a common problem: you often have some fixed strings that you wrote that you want to combine some varying strings that come from the data.
For example, to create a greeting you might combine "Hello" with a `name` variable.
First, we'll discuss two techniques that make this easy.
First, we'll discuss two functions that make this easy.
Then we'll talk about a slightly different scenario where you want to summarise a character vector, collapsing any number of strings into one.
### `str_c()`
@ -155,7 +154,7 @@ str_c("x", "y", "z")
str_c("Hello ", c("John", "Susan"))
```
`str_c()` is designed to be used with `mutate()` so it obeys the usual tidyverse rules for recycling and missing values:
`str_c()` is designed to be used with `mutate()` so it obeys the usual rules for recycling and missing values:
```{r}
df <- tibble(name = c("Timothy", "Dewey", "Mable", NA))
@ -175,7 +174,7 @@ df %>% mutate(
If you are mixing many fixed and variable strings with `str_c()`, you'll notice that you have to type `""` repeatedly, and this can make it hard to see the overall goal of the code.
An alternative approach is provided by the [glue package](https://glue.tidyverse.org) via `str_glue()`[^strings-4] .
You give it a single string containing `{}`. Anything inside `{}` will be evaluated like it's outside of the string:
You give it a single string containing `{}` and anything inside `{}` will be evaluated like it's outside of the string:
[^strings-4]: If you're not using stringr, you can also access it directly with `glue::glue().`
@ -189,7 +188,8 @@ As you can see above, `str_glue()` currently converts missing values to the stri
We'll hopefully fix that by the time the book is printed: <https://github.com/tidyverse/glue/issues/246>
You also might wonder what happens if you need to include a regular `{` or `}` in your string.
Here we use a slightly different escaping technique; instead of prefixing with special character like `\`, you just double up the `{` or `}`:
You might expect that you'll need to escape it, and you'd be right.
But glue uses a slightly different escaping technique; instead of prefixing with special character like `\`, you just double up the `{` and `}`:
```{r}
df %>% mutate(greeting = str_glue("{{Hi {name}!}}"))
@ -197,11 +197,11 @@ df %>% mutate(greeting = str_glue("{{Hi {name}!}}"))
### `str_flatten()`
`str_c()` and `glue()` work well with `mutate()` because the output is the same length as the input.
`str_c()` and `glue()` work well with `mutate()` because their output is the same length as their inputs.
What if you want a function that works well with `summarise()`, i.e. something that always returns a single string?
That's the job of `str_flatten()`:[^strings-5] it takes a character vector and combines each element of the vector into a single string:
That's the job of `str_flatten()`[^strings-5]: it takes a character vector and combines each element of the vector into a single string:
[^strings-5]: The base R equivalent is `paste()` with the `collapse` argument set.
[^strings-5]: The base R equivalent is `paste()` used with the `collapse` argument.
```{r}
str_flatten(c("x", "y", "z"))
@ -235,33 +235,36 @@ df %>%
str_c(letters[1:2], letters[1:3])
```
2. Convert between `str_c()` and `glue()`
2. Convert the following expressions from `str_c()` to `glue()` or vice versa:
3. How to make `{{{{` with glue?
a. `str_c("The price of ", food, " is ", price)`
b. `glue("I'm {age} years old and live in {country}")`
c. `str_c("\\section{", title, "}")`
## Working with patterns
Before we can discuss the opposite problem of extracting data out of strings, we need to take a quick digression to talk about **regular expressions**.
Regular expressions are a very concise language for describing patterns in strings.
Regular expressions can be overwhelming at first, and you'll think a cat walked across your keyboard.
Fortunately, as your understanding improves they'll soon start to make sense.
Regular expression is a bit of a mouthful, and the term isn't that useful as it refers to the underlying body of computer science theory where the meanings of both "regular" and "expression" are somewhat distant to their day-to-day meaning.
In practice, most people abbreviate to "regexs" or "regexps".
It's probably even more useful to be able to extract data from string than create strings from data, but before we can tackle that, we need to take a brief digression to talk about **regular expressions**.
Regular expressions are a very concise language that describes patterns in strings.
For example, `"^The"` is shorthand for any string that starts with "The", and `a.+e` is a shorthand for "a" followed by one or more other characters, followed by an "e".
We'll start by using `str_detect()` which answers a simple question: "does this pattern occur anywhere in my vector?".
We'll then ask progressively more complex questions by learning more about regular expressions and the functions that use them.
We'll then ask progressively more complex questions by learning more about regular expressions and the stringr functions that use them.
### Detect matches
To learn about regular expressions, we'll start with probably the simplest function that uses them: `str_detect()`.
It takes a character vector and a pattern, and returns a logical vector that says if the pattern was found at each element of the pattern:
The term "regular expression" is a bit of a mouthful, so most people abbreviate to "regex"[^strings-6] or "regexp".
To learn about regexes, we'll start with the simplest function that uses them: `str_detect()`. It takes a character vector and a pattern, and returns a logical vector that says if the pattern was found at each element of the vector.
The following code shows the simplest type of pattern, an exact match.
[^strings-6]: With a hard g, sounding like "reg-x".
```{r}
x <- c("apple", "banana", "pear")
str_detect(x, "e")
str_detect(x, "b")
str_detect(x, "x")
str_detect(x, "e") # does the word contain an e?
str_detect(x, "b") # does the word contain a b?
str_detect(x, "ear") # does the word contain "ear"?
```
`str_detect()` returns a logical vector the same length as the first argument, so it pairs well with `filter()`.
@ -271,8 +274,8 @@ For example, this code finds all names that contain a lower-case "x":
babynames %>% filter(str_detect(name, "x"))
```
We can also use `str_detect()` to summarize by remembering that when you use a logical vector in a numeric context, `FALSE` becomes 0 and `TRUE` becomes 1.
That means `sum(str_detect(x, pattern))` will tell you the number of observations that match the pattern, and `mean(str_detect(x, pattern))` will tell you the proportion that match.
We can also use `str_detect()` with `summarize()` by remembering that when you use a logical vector in a numeric context, `FALSE` becomes 0 and `TRUE` becomes 1.
That means `sum(str_detect(x, pattern))` will tell you the number of observations that match, while `mean(str_detect(x, pattern))` tells you the proportion of observations that match.
For example, the following snippet computes and visualizes the proportion of baby names that contain "x", broken down by year:
```{r, fig.alt = "A timeseries showing the proportion of baby names that contain the letter x. The proportion declines gradually from 8 per 1000 in 1880 to 4 per 1000 in 1980, then increases rapidly to 16 per 1000 in 2019."}
@ -294,14 +297,14 @@ str_detect(c("x", "X"), "x")
str_detect(c("xyz", "xza"), "xy")
```
In general, any letter or number will match exactly, but punctuation characters like `.`, `+`, `*`, `[`, `]`, `?`, often have special meanings[^strings-6].
In general, any letter or number will match exactly, but punctuation characters like `.`, `+`, `*`, `[`, `]`, `?`, often have special meanings[^strings-7].
For example, `.`
will match any character[^strings-7], so `"a."` will match any string that contains an a followed by another character
will match any character[^strings-8], so `"a."` will match any string that contains an "a" followed by another character
:
[^strings-6]: You'll learn how to escape this special behaviour in Section \@ref(regexp-escaping)
[^strings-7]: You'll learn how to escape this special behaviour in Section \@ref(regexp-escaping)
[^strings-7]: Well, any character apart from `\n`.
[^strings-8]: Well, any character apart from `\n`.
```{r}
str_detect(c("a", "ab", "ae", "bd", "ea", "eab"), "a.")
@ -348,7 +351,9 @@ str_view_all(names, "[^aeiou]")
str_view_all(names, "[^aeiou]+")
```
Lets practice our regular expression usage with some other useful stringr functions.
Regular expressions are very compact and use a lot of punctuation characters, so they can seem overwhelming at first, and you'll think a cat has walked across your keyboard.
So don't worry if they're hard to understand at first; you'll get better with practice.
Lets start that practice with some other useful stringr functions.
### Count matches
@ -359,6 +364,15 @@ x <- c("apple", "banana", "pear")
str_count(x, "p")
```
Note that regular expression matches never overlap so `str_count()` only starts looking for a new match after the end of the last match.
For example, in `"abababa"`, how many times will the pattern `"aba"` match?
Regular expressions say two, not three:
```{r}
str_count("abababa", "aba")
str_view_all("abababa", "aba")
```
It's natural to use `str_count()` with `mutate()`.
The following example uses `str_count()` with character classes to count the number of vowels and consonants in each name.
@ -372,30 +386,22 @@ babynames %>%
```
If you look closely, you'll notice that there's something off with our calculations: "Aaban" contains three "a"s, but our summary reports only two vowels.
That's because I've forgotten that regular expressions are case sensitive.
That's because I've forgotten to tell you that regular expressions are case sensitive.
There are three ways we could fix this:
- Add the upper case vowels to the character class: `str_count(name, "[aeiouAEIOUS]")`.
- Add the upper case vowels to the character class: `str_count(name, "[aeiouAEIOU]")`.
- Tell the regular expression to ignore case: `str_count(regex(name, ignore.case = TRUE), "[aeiou]")`. We'll talk about this next.
- Use `str_lower()` to convert the names to lower case: `str_count(to_lower(name), "[aeiou]")`. We'll come back to this function in Section \@ref(other-languages).
This is pretty typical when working with strings --- there are often multiple ways to reach your goal, either making your pattern more complicated or by doing some preprocessing on your string.
If you get stuck trying one approach, it can often be useful to switch gears and tackle the problem from a different perspective.
Note that regular expression matches never overlap, and `str_count()` only starts looking for a new match after the end of the last match.
For example, in `"abababa"`, how many times will the pattern `"abaµ"` match?
Regular expressions say two, not three:
```{r}
str_count("abababa", "aba")
str_view_all("abababa", "aba")
```
### Replace matches
Sometimes there are inconsistencies in the formatting that are easier to fix before you start extracting; easier to make the data more regular and check your work than coming up with a more complicated regular expression in `str_*` and friends.
`str_replace_all()` allows you to replace a match with the text of your choosing.
This can be particularly useful if you need to standardize a vector.
Unlike the regexp functions we've encountered so far, `str_replace_all()` takes three arguments: a character vector, a pattern, and a replacement.
`str_replace_all()` allow you to replace matches with new strings.
The simplest use is to replace a pattern with a fixed string:
```{r}
@ -403,7 +409,16 @@ x <- c("apple", "pear", "banana")
str_replace_all(x, "[aeiou]", "-")
```
With `str_replace_all()` you can perform multiple replacements by supplying a named vector.
`str_remove_all()` is a short cut for `str_replace_all(x, pattern, "")` --- it removes matching patterns from a string.
Use in `mutate()`
Using pipe inside mutate.
Recommendation to make a function, and think about testing it --- don't need formal tests, but useful to build up a set of positive and negative test cases as you.
### Advanced replacements
You can also perform multiple replacements by supplying a named vector.
The name gives a regular expression to match, and the value gives the replacement.
```{r}
@ -411,21 +426,14 @@ x <- c("1 house", "1 person has 2 cars", "3 people")
str_replace_all(x, c("1" = "one", "2" = "two", "3" = "three"))
```
`str_remove_all()` is a short cut for `str_replace_all(x, pattern, "")` --- it removes matching patterns from a string.
`pattern` as a function.
Come back to that in Chapter \@ref(programming-with-strings).
Alternatively, you can provide a replacement function: it's called with a vector of matches, and should return what to replacement them with.
We'll come back to this powerful tool in Chapter \@ref(programming-with-strings).
```{r}
x <- c("1 house", "1 person has 2 cars", "3 people")
str_replace_all(x, "[aeiou]+", str_to_upper)
```
Use in `mutate()`
Using pipe inside mutate.
Recommendation to make a function, and think about testing it --- don't need formal tests, but useful to build up a set of positive and negative test cases as you.
### Pattern control
Now that you've learn about regular expressions, you might be worried about them working when you don't want them to.
@ -435,7 +443,7 @@ You can opt-out of the regular expression rules by using `fixed()`:
str_view(c("", "a", "."), fixed("."))
```
Note that both fixed strings and regular expressions are case sensitive by default.
Both fixed strings and regular expressions are case sensitive by default.
You can opt out by setting `ignore_case = TRUE`.
```{r}
@ -474,7 +482,8 @@ Waiting on: <https://github.com/tidyverse/tidyups/pull/15>
So far all of our examples have been using English.
The details of the many ways other languages are different to English are too diverse to detail here, but I wanted to give a quick outline of the functions who's behavior differs based on your **locale**, the set of settings that vary from country to country.
The locale is specified with a two or three letter lower-case language abbreviation, optionally followed by a `_` and a upper region identifier.
Locale is specified with lower-case language abbreviation, optionally followed by a `_` and a upper-case region identifier.
For example, "en" is English, "en_GB" is British English, and "en_US" is American English.
If you don't already know the code for your language, [Wikipedia](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) has a good list, and you can see which are supported with `stringi::stri_locale_list()`.
@ -494,15 +503,15 @@ Fortunately there are three sets of functions where the locale matters:
str_to_upper(c("i", "ı"), locale = "tr")
```
- **Comparing strings**: `str_equal()` lets you compare if two strings are equally optionally ignoring case:
- **Comparing strings**: `str_equal()` lets you compare if two strings are equal, optionally ignoring case:
```{r}
str_equal("i", "I", ignore_case = TRUE)
str_equal("i", "I", ignore_case = TRUE, locale = "tr")
```
- **Sorting strings**: `str_sort()` and `str_order()` sort vector alphabetically, but the alphabet is not the same in every language[^strings-8].
Here's an example: in Czech, "ch" is a digraph that appears after `h` in the alphabet.
- **Sorting strings**: `str_sort()` and `str_order()` sort vectors alphabetically, but the alphabet is not the same in every language[^strings-9]!
Here's an example: in Czech, "ch" is a compound letter that appears after `h` in the alphabet.
```{r}
str_sort(c("a", "c", "ch", "h", "z"))
@ -510,28 +519,28 @@ Fortunately there are three sets of functions where the locale matters:
```
Danish has a similar problem.
Normally, characters with diacritic sorts after the plain character.
But in Danish ø and å are letters that come at the end of the alphabet:
Normally, characters with diacritics (e.g. à, á, â) sort after the plain character (e.g. a).
But in Danish ø and å are their own letters that come at the end of the alphabet:
```{r}
str_sort(c("a", "å", "o", "ø", "z"))
str_sort(c("a", "å", "o", "ø", "z"), locale = "da")
```
TODO after dplyr 1.1.0: discuss `arrange()`
This also comes up when sorting strings with `dplyr::arrange()` which is why it also has a `locale` argument.
[^strings-8]: Sorting in languages that don't have an alphabet (like Chinese) is more complicated still.
[^strings-9]: Sorting in languages that don't have an alphabet (like Chinese) is more complicated still.
## Letters
Functions that work with the letters inside of the string.
Functions that work with the components of strings called **code points**.
Depending on the language involved, this might be a letter (like in most European languages), a syllable (like Japanese), or a logogram (like in Chinese).
It might be something more exotic like an accent, or a special symbol used to join two emoji together.
But to keep things simple, I'll call these letters.
### Length
`str_length()` tells you the number of characters in the string[^strings-9]:
[^strings-9]: The number of characters turns out to be a surprisingly complicated concept when you look across more languages.
We're not going to get into the details here, but you'll need to learn more about this if you want work with non-European languages.
`str_length()` tells you the number of letters in the string:
```{r}
str_length(c("a", "R for data science", NA))
@ -612,8 +621,6 @@ The are a bunch of other places you can use regular expressions outside of strin
- `matches()`: as you can tell from it's lack of `str_` prefix, this isn't a stringr fuction.
It's a "tidyselect" function, a fucntion that you can use anywhere in the tidyverse when selecting variables (e.g. `dplyr::select()`, `rename_with()`, `across()`, ...).
- `str_locate()`, `str_match()`, `str_split()`; useful for programming with strings.
- `apropos()` searches all objects available from the global environment.
This is useful if you can't quite remember the name of the function.