441 lines
17 KiB
Plaintext
441 lines
17 KiB
Plaintext
# Regular expressions
|
|
|
|
```{r, results = "asis", echo = FALSE}
|
|
status("restructuring")
|
|
```
|
|
|
|
## Introduction
|
|
|
|
You learned the basics of regular expressions in Chapter \@ref(strings), but because regular expressions are a miniature language it's worth spending some extra time on the details.
|
|
|
|
The chapter starts by expanding your knowledge of patterns, to cover six important new topics (escaping, anchoring, character classes, shorthand classes, quantifiers, and alternation).
|
|
Here we'll focus mostly on the language itself, not the functions that use it.
|
|
That means we'll mostly work with toy character vectors, showing the results with `str_view()` and `str_view_all()`.
|
|
You'll need to take what you learn here and apply it to data frames with tidyr functions or by combining dplyr and stringr functions.
|
|
|
|
Next we'll talk about the important concepts of "grouping" and "capturing" which give you new ways to extract variables out of strings using `tidyr::separate_group()`.
|
|
Grouping also allows you to use back references which allow you do things like match repeated patterns.
|
|
|
|
We'll finish by discussing the various "flags" that allow you to tweak the operation of regular expressions and then cover a details about how regular expressions work that , and then discuss some useful strategies .
|
|
|
|
### Prerequisites
|
|
|
|
This chapter will use regular expressions as provided by the **stringr** package.
|
|
|
|
```{r setup, message = FALSE}
|
|
library(tidyverse)
|
|
```
|
|
|
|
It's worth noting that the regular expressions used by stringr are very slightly different to those of base R.
|
|
That's because stringr is built on top of the [stringi package](https://stringi.gagolewski.com), which is in turn built on top of the [ICU engine](https://unicode-org.github.io/icu/userguide/strings/regexp.html), whereas base R functions (like `gsub()` and `grepl()`) use either the [TRE engine](https://github.com/laurikari/tre) or the [PCRE engine](https://www.pcre.org).
|
|
Fortunately, the basics of regular expressions are so well established that you're unlikely to encounter any differences when working with the patterns you'll learn in this book.
|
|
You only need to be aware of the difference when you start to rely on advanced features like complex Unicode character ranges or special features that use the `(?…)` syntax.
|
|
You can learn more about these advanced features in `vignette("regular-expressions", package = "stringr")`.
|
|
|
|
Another useful reference is [https://www.regular-expressions.info/](https://www.regular-expressions.info/tutorial.html).
|
|
It's not R specific, but it includes a lot more information about how regular expressions actually work.
|
|
|
|
### Exercises
|
|
|
|
1. Explain why each of these strings don't match a `\`: `"\"`, `"\\"`, `"\\\"`.
|
|
|
|
2. How would you match the sequence `"'\`?
|
|
|
|
3. What patterns will the regular expression `\..\..\..` match?
|
|
How would you represent it as a string?
|
|
|
|
## Pattern language
|
|
|
|
You learned the very basics of the regular expression pattern language in Chapter \@ref(strings), and now its time to dig into more of the details.
|
|
First, we'll start with **escaping**, which allows you to match characters that the pattern language otherwise treats specially.
|
|
Next you'll learn about **anchors**, which allow you to match the start or end of the string.
|
|
Then you'll learn about **character classes** and their shortcuts, which allow you to match any character from a set.
|
|
We'll finish up with **quantifiers**, which control how many times a pattern can match, and **alternation**, which allows you to match either *this* or *that.*
|
|
|
|
The terms I use here are the technical names for each component.
|
|
They're not always the most evocative of their purpose, but it's very helpful to know the correct terms if you later want to Google for more details.
|
|
|
|
### Escaping {#regexp-escaping}
|
|
|
|
In Chapter \@ref(strings), you'll learned how to match a literal `.` by using `fixed(".")`.
|
|
What if you want to match a literal `.` as part of a regular expression?
|
|
You'll need to use an escape, which tells the regular expression you want it to match exactly, not use its special behavior.
|
|
Like strings, regexps use the backslash, `\`, to escape special behavior.
|
|
So to match a `.`, you need the regexp `\.`.
|
|
Unfortunately this creates a problem.
|
|
We use strings to represent regular expressions, and `\` is also used as an escape symbol in strings.
|
|
So to create the regular expression `\.` we need the string `"\\."`.
|
|
|
|
```{r}
|
|
# To create the regular expression \., we need to use \\.
|
|
dot <- "\\."
|
|
|
|
# But the expression itself only contains one:
|
|
str_view(dot)
|
|
|
|
# And this tells R to look for an explicit .
|
|
str_view(c("abc", "a.c", "bef"), "a\\.c")
|
|
```
|
|
|
|
In this book, I'll write regular expression as `\.` and strings that represent the regular expression as `"\\."`.
|
|
|
|
If `\` is used as an escape character in regular expressions, how do you match a literal `\`?
|
|
Well you need to escape it, creating the regular expression `\\`.
|
|
To create that regular expression, you need to use a string, which also needs to escape `\`.
|
|
That means to match a literal `\` you need to write `"\\\\"` --- you need four backslashes to match one!
|
|
|
|
```{r}
|
|
x <- "a\\b"
|
|
str_view(x)
|
|
str_view(x, "\\\\")
|
|
```
|
|
|
|
Alternatively, you might find it easier to use the raw strings you learned about in Section \@ref(raw-strings)).
|
|
That allows you to avoid one layer of escaping:
|
|
|
|
```{r}
|
|
str_view(x, r"(\\)")
|
|
```
|
|
|
|
The full set of characters with special meanings that need to be escaped is `.^$\|*+?{}[]()`.
|
|
In general, look at punctuation character with suspicion; if your regular expression isn't matching what you think it should, check if you've used any of these characters.
|
|
|
|
### Anchors
|
|
|
|
By default, regular expressions will match any part of a string.
|
|
If you want to match at the start of end you need to **anchor** the regular expression using `^` or `$`.
|
|
|
|
- `^` to match the start of the string.
|
|
- `$` to match the end of the string.
|
|
|
|
```{r}
|
|
x <- c("apple", "banana", "pear")
|
|
str_view(x, "a") # match "a" anywhere
|
|
str_view(x, "^a") # match "a" at start
|
|
str_view(x, "a$") # match "a" at end
|
|
```
|
|
|
|
To remember which is which, try this mnemonic which I learned from [Evan Misshula](https://twitter.com/emisshula/status/323863393167613953): if you begin with power (`^`), you end up with money (`$`).
|
|
It's tempting to put `$` at the start, because that's how we write sums of money, but it's not what regular expressions want.
|
|
|
|
To force a regular expression to only match the full string, anchor it with both `^` and `$`:
|
|
|
|
```{r}
|
|
x <- c("apple pie", "apple", "apple cake")
|
|
str_view(x, "apple")
|
|
str_view(x, "^apple$")
|
|
```
|
|
|
|
You can also match the boundary between words with `\b`.
|
|
I don't often use this in my R code, but I'll sometimes use it when I'm doing a search in RStudio.
|
|
It's use to find the name of a function that's a component of other functions.
|
|
For example, I'll search for `\bsum\b` to avoid matching `summarise`, `summary`, `rowsum` and so on:
|
|
|
|
```{r}
|
|
x <- c("summary(x)", "summarise(df)", "rowsum(x)", "sum(x)")
|
|
str_view(x, "sum")
|
|
str_view(x, "\\bsum\\b")
|
|
```
|
|
|
|
### Character classes
|
|
|
|
You can also create your own collections of characters using `[]`:
|
|
|
|
- `[abc]`: matches a, b, or c.
|
|
- `[a-z]`: matches every character between a and z. `[0-9]` matches any number.
|
|
- `[^abc]`: matches anything except a, b, or c.
|
|
- `[\^\-]`: matches `^` or `-`.
|
|
|
|
Remember that these are case sensitive.
|
|
If you want to match any Latin letter or number, you could write `[a-zA-Z0-9]`.
|
|
|
|
A character class containing a single character can be a nice alternative to escapes when you want to include a single special character (i.e. `$` `.` `|` `?` `*` `+` `(` `)` `[` `{` `}`, but not `]` `\` `^`).
|
|
This can be more readable because there are fewer slashes, but it does require a deeper understanding of regular expressions.
|
|
|
|
```{r}
|
|
# Look for a literal character that normally has special meaning in a regex
|
|
str_view(c("abc", "a.c", "a*c", "a c"), "a[.]c")
|
|
str_view(c("abc", "a.c", "a*c", "a c"), ".[*]c")
|
|
str_view(c("abc", "a.c", "a*c", "a c"), "a[ ]")
|
|
```
|
|
|
|
### Shorthand character classes
|
|
|
|
There are a few character classes that are used so commonly that they get their own shortcut.
|
|
You've already seen `.`, which matches any character apart from a newline.
|
|
There are three other useful pairs:
|
|
|
|
- `\d`: matches any digit; `\D` matches anything that isn't a digit.
|
|
- `\s`: matches any whitespace (e.g. space, tab, newline); `\S` matches anything that isn't whitespace.
|
|
- `\w` matches any "word" character, i.e. letters and numbers; `\W`, matches any non-word character.
|
|
|
|
Remember, to create a regular expression containing `\d` or `\s`, you'll need to escape the `\` for the string, so you'll type `"\\d"` or `"\\s"`.
|
|
The following code demonstrates the different matches with a selection of letters, numbers, and punctuation characters.
|
|
|
|
```{r}
|
|
str_view_all("abcd12345!@#%. ", "\\d+")
|
|
str_view_all("abcd12345!@#%. ", "\\D+")
|
|
str_view_all("abcd12345!@#%. ", "\\w+")
|
|
str_view_all("abcd12345!@#%. ", "\\W+")
|
|
str_view_all("abcd12345!@#%. ", "\\s+")
|
|
str_view_all("abcd12345!@#%. ", "\\S+")
|
|
```
|
|
|
|
### Quantifiers
|
|
|
|
The next step up in power involves controlling how many times a pattern matches, the so called **quantifiers**.
|
|
We discussed `?` (0 or 1 matches), `+` (1 or more matches), and `*` (0 or more matches) in the last chapter.
|
|
Note that the precedence of these operators is high, so you can write: `colou?r` to match either American or British spellings.
|
|
That means most uses will need parentheses, like `bana(na)+`.
|
|
|
|
You can also specify the number of matches precisely:
|
|
|
|
- `{n}`: exactly n
|
|
- `{n,}`: n or more
|
|
- `{n,m}`: between n and m
|
|
|
|
```{r}
|
|
x <- "1888 is the longest year in Roman numerals: MDCCCLXXXVIII"
|
|
str_view(x, "C{2}")
|
|
str_view(x, "C{2,}")
|
|
str_view(x, "C{1,3}")
|
|
str_view(x, "C{2,3}")
|
|
```
|
|
|
|
By default these matches are **greedy**: they will match the longest string possible.
|
|
You can make them **lazy**, matching the shortest string possible by putting a `?` after them.
|
|
This is an advanced feature of regular expressions, but it's useful to know that it exists:
|
|
|
|
```{r}
|
|
str_view(x, 'C{2,3}?')
|
|
str_view(x, 'C+[LX]+')
|
|
str_view(x, 'C+[LX]+?')
|
|
```
|
|
|
|
### Alternation
|
|
|
|
You can use **alternation** to pick between one or more alternative patterns.
|
|
This is a more general form of character classes that's not limited to match single characters.
|
|
I recommend always pairing `|` with parentheses, to make it very clear what the alternatives are.
|
|
Here are a few examples:
|
|
|
|
- Match apple, pear, or banana: `"(apple)|(pear)|(banana)"`
|
|
- Match 3 letters or two digits: `"(\\w{3})|(\\d{3})"`
|
|
|
|
We'll come back to parentheses very shortly in more detail.
|
|
|
|
For example, `abc|def` will match either `"abcef"`, or `"abdef"`.
|
|
Note that the precedence for `|` is low, so you'll often need to use it with parentheses: `(abc)|(def)` will match either `"abc"`, or `"def"`.
|
|
`abc|xyz` matches `abc` or `xyz` not `abcyz` or `abxyz`.
|
|
Like with mathematical expressions, if precedence ever gets confusing, use parentheses to make it clear what you want:
|
|
|
|
```{r}
|
|
str_view(c("grey", "gray"), "gr(e|a)y")
|
|
```
|
|
|
|
### Exercises
|
|
|
|
1. How would you match the literal string `"$^$"`?
|
|
|
|
2. Given the corpus of common words in `stringr::words`, create regular expressions that find all words that:
|
|
|
|
a. Start with "y".
|
|
b. Don't start with "y".
|
|
c. End with "x".
|
|
d. Are exactly three letters long. (Don't cheat by using `str_length()`!)
|
|
e. Have seven letters or more.
|
|
|
|
Since `words` is long, you might want to use the `match` argument to `str_view()` to show only the matching or non-matching words.
|
|
|
|
3. What strings will `$a` match?
|
|
|
|
4. Create regular expressions to find all words that:
|
|
|
|
a. Start with a vowel.
|
|
b. That only contain consonants. (Hint: thinking about matching "not"-vowels.)
|
|
c. End with `ed`, but not with `eed`.
|
|
d. End with `ing` or `ise`.
|
|
|
|
5. Empirically verify the rule "i before e except after c".
|
|
|
|
6. Is "q" always followed by a "u"?
|
|
|
|
7. Write a regular expression that matches a `word` if it's probably written in British English, not American English.
|
|
|
|
8. Create a regular expression that will match telephone numbers as commonly written in your country.
|
|
|
|
9. Describe the equivalents of `?`, `+`, `*` in `{m,n}` form.
|
|
|
|
10. Describe in words what these regular expressions match: (read carefully to see if I'm using a regular expression or a string that defines a regular expression.)
|
|
|
|
a. `^.*$`
|
|
b. `"\\{.+\\}"`
|
|
c. `\d{4}-\d{2}-\d{2}`
|
|
d. `"\\\\{4}"`
|
|
|
|
11. Create regular expressions to find all words that:
|
|
|
|
a. Start with three consonants.
|
|
b. Have three or more vowels in a row.
|
|
c. Have two or more vowel-consonant pairs in a row.
|
|
|
|
12. Solve the beginner regexp crosswords at <https://regexcrossword.com/challenges/beginner>.
|
|
|
|
## Grouping and capturing
|
|
|
|
Earlier, you learned about parentheses as a way to disambiguate complex expressions.
|
|
Parentheses also create a numbered capturing group (number 1, 2 etc.).
|
|
A capturing group stores the part of the string matched by the part of the regular expression inside the parentheses.
|
|
You can refer to the same text as previously matched by a capturing group with **backreferences**, like `\1`, `\2` etc.
|
|
|
|
For example, the following regular expression finds all fruits that have a repeated pair of letters.
|
|
|
|
```{r}
|
|
str_view(fruit, "(..)\\1", match = TRUE)
|
|
```
|
|
|
|
### Replacement
|
|
|
|
You can also use backreferences when replacing.
|
|
The following code will switch the order of the second and third words:
|
|
|
|
```{r}
|
|
sentences %>%
|
|
str_replace("(\\w+) (\\w+) (\\w+)", "\\1 \\3 \\2") %>%
|
|
head(5)
|
|
```
|
|
|
|
Names that start and end with the same letter.
|
|
Implement with `str_sub()` instead.
|
|
|
|
### str_match()
|
|
|
|
```{r}
|
|
sentences %>%
|
|
str_view("the (\\w+) (\\w+)", match = TRUE) %>%
|
|
head()
|
|
```
|
|
|
|
### Non-capturing groups
|
|
|
|
Occasionally, you'll want to use parentheses without creating matching groups.
|
|
You can create a non-capturing group with `(?:)`.
|
|
Typically, however, you'll find it easier to just ignore that result in the output of `str_match()`.
|
|
|
|
```{r}
|
|
x <- c("a gray cat", "a grey dog")
|
|
str_match(x, "(gr(e|a)y)")
|
|
str_match(x, "(gr(?:e|a)y)")
|
|
```
|
|
|
|
### Exercises
|
|
|
|
1. Describe, in words, what these expressions will match:
|
|
|
|
a. `(.)\1\1`
|
|
b. `"(.)(.)\\2\\1"`
|
|
c. `(..)\1`
|
|
d. `"(.).\\1.\\1"`
|
|
e. `"(.)(.)(.).*\\3\\2\\1"`
|
|
|
|
2. Construct regular expressions to match words that:
|
|
|
|
a. Start and end with the same character.
|
|
b. Contain a repeated pair of letters (e.g. "church" contains "ch" repeated twice.)
|
|
c. Contain one letter repeated in at least three places (e.g. "eleven" contains three "e"s.)
|
|
|
|
## Flags
|
|
|
|
The are a number of settings, called **flags**, that you can use to control some of the details of the pattern language.
|
|
In stringr, you can supply these by instead of passing a simple string as a pattern, by passing the object created by `regex()`:
|
|
|
|
```{r, eval = FALSE}
|
|
# The regular call:
|
|
str_view(fruit, "nana")
|
|
# Is shorthand for
|
|
str_view(fruit, regex("nana"))
|
|
```
|
|
|
|
This is useful because it allows you to pass additional arguments to control the details of the match the most useful is probably `ignore_case = TRUE` because it allows characters to match either their uppercase or lowercase forms:
|
|
|
|
```{r}
|
|
bananas <- c("banana", "Banana", "BANANA")
|
|
str_view(bananas, "banana")
|
|
str_view(bananas, regex("banana", ignore_case = TRUE))
|
|
```
|
|
|
|
If you're doing a lot of work with multiline strings (i.e. strings that contain `\n`), `multiline` and `dotall` can also be useful.
|
|
`dotall = TRUE` allows `.` to match everything, including `\n`:
|
|
|
|
```{r}
|
|
x <- "Line 1\nLine 2\nLine 3"
|
|
str_view_all(x, ".L")
|
|
str_view_all(x, regex(".L", dotall = TRUE))
|
|
```
|
|
|
|
And `multiline = TRUE` allows `^` and `$` to match the start and end of each line rather than the start and end of the complete string:
|
|
|
|
```{r}
|
|
x <- "Line 1\nLine 2\nLine 3"
|
|
str_view_all(x, "^Line")
|
|
str_view_all(x, regex("^Line", multiline = TRUE))
|
|
```
|
|
|
|
If you're writing a complicated regular expression and you're worried you might not understand it in the future, `comments = TRUE` can be super useful.
|
|
It allows you to use comments and white space to make complex regular expressions more understandable.
|
|
Spaces and new lines are ignored, as is everything after `#`.
|
|
(Note that I'm using a raw string here to minimise the number of escapes needed)
|
|
|
|
```{r}
|
|
phone <- regex(r"(
|
|
\(? # optional opening parens
|
|
(\d{3}) # area code
|
|
[) -]? # optional closing parens, space, or dash
|
|
(\d{3}) # another three numbers
|
|
[ -]? # optional space or dash
|
|
(\d{3}) # three more numbers
|
|
)", comments = TRUE)
|
|
|
|
str_match("514-791-8141", phone)
|
|
```
|
|
|
|
If you're using comments and want to match a space, newline, or `#`, you'll need to escape it:
|
|
|
|
```{r}
|
|
str_view("x x #", regex("x #", comments = TRUE))
|
|
str_view("x x #", regex(r"(x\ \#)", comments = TRUE))
|
|
```
|
|
|
|
## Some details
|
|
|
|
### Overlapping
|
|
|
|
Matches never overlap, and the regular expression engine only starts looking for a new match after the end of the last match.
|
|
For example, in `"abababa"`, how many times will the pattern `"aba"` match?
|
|
Regular expressions say two, not three:
|
|
|
|
```{r}
|
|
str_count("abababa", "aba")
|
|
str_view_all("abababa", "aba")
|
|
```
|
|
|
|
### Zero width matches
|
|
|
|
It's possible for a regular expression to match no character, i.e. the space between too characters.
|
|
This typically happens when you use a quantifier that allows zero matches:
|
|
|
|
```{r}
|
|
str_view_all("abcdef", "c?")
|
|
```
|
|
|
|
But anchors also create zero-width matches:
|
|
|
|
```{r}
|
|
str_view_all("this is a sentence", "\\b")
|
|
str_view_all("this is a sentence", "^")
|
|
```
|
|
|
|
### Greediness
|
|
|
|
Regular expressions always attempt to match the longest possible string.
|