You learned the basics of regular expressions in Chapter \@ref(strings), but regular expressions are fairly rich language so it's worth spending some extra time on the details.
The chapter starts by expanding your knowledge of patterns, to cover six important new topics (escaping, anchoring, character classes, shorthand classes, quantifiers, and alternation).
Here we'll focus mostly on the language itself, not the functions that use it.
That means we'll mostly work with toy character vectors, showing the results with `str_view()` and `str_view_all()`.
You'll need to take what you learn here and apply it to data frames with tidyr functions or by combining dplyr and stringr functions.
Next we'll talk about the important concepts of "grouping" and "capturing" which give you new ways to extract variables out of strings using `tidyr::separate_group()`.
Grouping also allows you to use back references which allow you do things like match repeated patterns.
We'll finish by discussing the various "flags" that allow you to tweak the operation of regular expressions and cover a few final details about how regular expressions work.
These aren't particularly important in day-to-day usage, but at little extra understanding of the underlying tools is often helpful.
It's worth noting that the regular expressions used by stringr are very slightly different to those of base R.
That's because stringr is built on top of the [stringi package](https://stringi.gagolewski.com), which is in turn built on top of the [ICU engine](https://unicode-org.github.io/icu/userguide/strings/regexp.html), whereas base R functions (like `gsub()` and `grepl()`) use either the [TRE engine](https://github.com/laurikari/tre) or the [PCRE engine](https://www.pcre.org).
Fortunately, the basics of regular expressions are so well established that you'll encounter few variations when working with the patterns you'll learn in this book (and I'll point them out where important).
You only need to be aware of the difference when you start to rely on advanced features like complex Unicode character ranges or special features that use the `(?…)` syntax.
You can learn more about these advanced features in `vignette("regular-expressions", package = "stringr")`.
You learned the very basics of the regular expression pattern language in Chapter \@ref(strings), and now its time to dig into more of the details.
First, we'll start with **escaping**, which allows you to match characters that the pattern language otherwise treats specially.
Next you'll learn about **anchors**, which allow you to match the start or end of the string.
Then you'll learn about **character classes** and their shortcuts, which allow you to match any character from a set.
We'll finish up with **quantifiers**, which control how many times a pattern can match, and **alternation**, which allows you to match either *this* or *that.*
In general, look at punctuation characters with suspicion; if your regular expression isn't matching what you think it should, check if you've used any of these characters.
To remember which is which, try this mnemonic which I learned from [Evan Misshula](https://twitter.com/emisshula/status/323863393167613953): if you begin with power (`^`), you end up with money (`$`).
In Chapter \@ref(strings) you learned about `?` (0 or 1 matches), `+` (1 or more matches), and `*` (0 or more matches).
For example, `colou?r` will match American or British spelling, `\d+` will match one or more digits, and `\s?` will optionally match a single whitespace.
Does it match "a" followed by one or more "b"s, or does it match "ab" repeated any number of times?
What does `^a|b$` match?
Does it match the complete string a or the complete string b, or does it match a string starting with a or a string starting with "b"?
The answer to these questions is determined by operator precedence, similar to the PEMDAS or BEDMAS rule you might have learned in school.
The question comes down to whether `ab+` is equivalent to `a(b+)` or `(ab)+` and whether `^a|b$` is equivalent to `(^a)|(b$)` or `^(a|b)$`.
Alternation has low precedence which means it affects many characters, whereas quantifiers have high precedence which means it affects few characters.
Quantifiers apply to the preceding pattern, regardless of how many letters define it: `a+` matches one or more "a"s, `\d+` matches one or more digits, and `[aeiou]+` matches one or more vowels.
You can use parentheses to apply a quantifier to a compound pattern.
For example, `([aeiou].)+` matches a vowel followed by any letter, repeated any number of times.
```{r}
str_view(words, "^([aeiou].)+$", match = TRUE)
```
`|` has very low precedence which means that everything to the left or right is included in the group.
For example if you want to match only a complete string, `^apple|pear|banana$` won't work because it will match apple at the start of the string, pear anywhere, and banana at the end.
Instead, you need `^(apple|pear|banana)$`.
Technically the escape, character classes, and parentheses are all operators that have some relative precedence.
But these tend to be less likely to cause confusion, for example you experience with escapes in other situations means it's unlikely that you'd think to write `\(s|d)` to mean `(\s)|(\d)`.
3. Create regular expressions that match the British or American spellings of the following words: grey/gray, modelling/modeling, summarize/summarise, aluminium/aluminum, defence/defense, analog/analogue, center/centre, sceptic/skeptic, aeroplane/airplane, arse/ass, doughnut/donut.
7. Describe in words what these regular expressions match: (read carefully to see if I'm using a regular expression or a string that defines a regular expression.)
The are a number of settings, called **flags**, that you can use to control some of the details of the pattern language.
In stringr, you can supply these by instead of passing a simple string as a pattern, by passing the object created by `regex()`:
```{r, eval = FALSE}
# The regular call:
str_view(fruit, "nana")
# Is shorthand for
str_view(fruit, regex("nana"))
```
This is useful because it allows you to pass additional arguments to control the details of the match the most useful is probably `ignore_case = TRUE` because it allows characters to match either their uppercase or lowercase forms:
If you're doing a lot of work with multiline strings (i.e. strings that contain `\n`), `multiline` and `dotall` can also be useful.
`dotall = TRUE` allows `.` to match everything, including `\n`:
```{r}
x <- "Line 1\nLine 2\nLine 3"
str_view_all(x, ".L")
str_view_all(x, regex(".L", dotall = TRUE))
```
And `multiline = TRUE` allows `^` and `$` to match the start and end of each line rather than the start and end of the complete string:
```{r}
x <- "Line 1\nLine 2\nLine 3"
str_view_all(x, "^Line")
str_view_all(x, regex("^Line", multiline = TRUE))
```
If you're writing a complicated regular expression and you're worried you might not understand it in the future, `comments = TRUE` can be super useful.
It allows you to use comments and white space to make complex regular expressions more understandable.
Spaces and new lines are ignored, as is everything after `#`.
(Note that I'm using a raw string here to minimise the number of escapes needed)
```{r}
phone <- regex(r"(
\(? # optional opening parens
(\d{3}) # area code
[) -]? # optional closing parens, space, or dash
(\d{3}) # another three numbers
[ -]? # optional space or dash
(\d{3}) # three more numbers
)", comments = TRUE)
str_match("514-791-8141", phone)
```
If you're using comments and want to match a space, newline, or `#`, you'll need to escape it:
```{r}
str_view("x x #", regex("x #", comments = TRUE))
str_view("x x #", regex(r"(x\ \#)", comments = TRUE))
To put these ideas in practice we'll solve a few semi-authentic problems using the `words` and `sentences` datasets built into stringr.
`words` is a list of common English words and `sentences` is a set of simple sentences originally used for testing voice transmission.
```{r}
str_view(head(words))
str_view(head(sentences))
```
Let's find all sentences that start with the:
```{r}
str_view(sentences, "^The", match = TRUE)
str_view(sentences, "^The\\b", match = TRUE)
```
All sentences that use a pronoun:
Modify to create simple set of positive and negative examples (if you later get more into programming and learn about unit tests, I highly recommend unit testing your regular expressions. This doesn't guarantee you won't get it wrong, but it ensures that you'll never make the same mistake twice.)
```{r}
str_view_all(sentences, "\\b(he|she|it)\\b", match = TRUE)
str_view_all(head(sentences), "\\b(he|she|it)\\b", match = FALSE)
str_view_all(sentences, regex("\\b(he|she|it)\\b", ignore_case = TRUE), match = TRUE)
```
All words that only contain consonants:
```{r}
str_view(words, "[^aeiou]+", match = TRUE)
str_view(words, "^[^aeiou]+$", match = TRUE)
```
This is a case where flipping the problem around can make it easier to solve.
Instead of looking for words that containing only consonant, we could look for words that don't contain any vowels:
```{r}
words[!str_detect(words, "[aeiou]")]
```
Can we find evidence for or against the rule "i before e except after c"?
To look for words that support this rule we want i follows e following any letter that isn't c, i.e. `[^c]ie`.
The opposite branch is `cei`:
```{r}
str_view(words, "[^c]ie|cei", match = TRUE)
```
To look for words that don't follow this rule, we just switch the i and the e:
```{r}
str_view(words, "[^c]ei|cie", match = TRUE)
```
Consist only of vowel-consonant or consonant-vowel pairs?
```{r}
str_view(words, "^([aeiou][^aeiou])+$", match = TRUE)
str_view(words, "^([^aeiou][aeiou])+$", match = TRUE)
```
Could combine in two ways: by making one complex regular expression or using `str_detect()` with Boolean operators:
```{r}
str_view(words, "^((([aeiou][^aeiou])+)|([^aeiou][aeiou]+))$", match = TRUE)
vc <- str_detect(words, "^([aeiou][^aeiou])+$")
cv <- str_detect(words, "^([^aeiou][aeiou])+$")
words[cv | vc]
```
This only handles words with even number of letters?
If we wanted to require the words to be at least four characters long we could modify the regular expressions switching `+` for `{2,}` or we could combine the results with `str_length()`:
```{r}
words[(cv | vc) & str_length(words) >= 4]
```
Do any words contain all vowels?
```{r}
str_view(words, "a.*e.*i.*o.*u", match = TRUE)
str_view(words, "e.*a.*u.*o.*i", match = TRUE)
```
```{r}
words[
str_detect(words, "a") &
str_detect(words, "e") &
str_detect(words, "i") &
str_detect(words, "o") &
str_detect(words, "u")
]
```
All sentences that contain a color:
```{r}
str_view(sentences, "\\b(red|green|blue)\\b", match = TRUE)
