r4ds/functions.Rmd

---
layout: default
title: Expressing yourself in code
---

# Expressing yourself in code

```{r, include = FALSE}
knitr::opts_chunk$set(
  cache = TRUE,
  fig.path = "figures/functions/"
)
```

Code is a means of communication, not just to the computer, but to other people. This is important because every project you undertake is fundamentally collaborative, and even if you're not working with other people you'll definitely be working with future-you. 

After solving a data analysis challenge, it's often worth looking at your code and thinking about whether or not it's obvious what you've done. If you spend a little time rewriting your code while the ideas are fresh, you can save a lot of time later trying to recreate what your code did.

To me, this is what mastering R as a programming language is all about: making it easier to express yourself, so that over time your becomes more and more clear, and easier to write.  In this chapter, you'll learn some of the most important skills, but to learn more you need to study R as a programming language, not just an interactive environment for data science. We have written two books that will help you do so:

* [Hands on programming with R](http://shop.oreilly.com/product/0636920028574.do),
  by Garrett Grolemund. This is an introduction to R as a programming language 
  and is a great place to start if R is your first programming language.
  
* [Advanced R](http://adv-r.had.co.nz) by Hadley Wickham. This dives into the
  details of R the programming language. This is a great place to start if
  you've programmed in other languages and you want to learn what makes R 
  special, different, and particularly well suited to data analysis.

You get better very slowly if you don't consciously practice, so this chapter brings together a number of ideas that we mention elsewhere into one focussed chapter on code as communication.

```{r}
library(magrittr)
```

This chapter is not comprehensive, but it will illustrate some patterns that in the long-term that will help you write clear and comprehensive code. 

The goal is not just to write better funtions or to do things that you couldn't do before, but to code with more "ease".

## Piping

```R
foo_foo <- little_bunny()
```

There are a number of ways that you could write this:

1.  Function composition:

    ```R
    bop_on(
      scoop_up(
        hop_through(foo_foo, forest),
        field_mouse
      ), 
      head
    )
    ```
    
    The disadvantage is that you have to read from inside-out, from 
    right-to-left, and that the arguments end up spread far apart 
    (sometimes called the 
    [dagwood sandwhich](https://en.wikipedia.org/wiki/Dagwood_sandwich)
    problem).
    
1.  Intermediate state:

    ```R
    foo_foo_1 <- hop_through(foo_foo, forest)
    foo_foo_2 <- scoop_up(foo_foo_1, field_mouse)
    foo_foo_3 <- bop_on(foo_foo_2, head)
    ```
    
    This avoids the nesting, but you now have to name each intermediate element.
    If there are natural names, use this form. But if you're just numbering
    them, I don't think it's that useful. Whenever I write code like this, 
    I invariably write the wrong number somewhere and then spend 10 minutes 
    scratching my head and trying to figure out what went wrong with my code.
    
    You may also worry that this form creates many intermediate copies of your
    data and takes up a lot of memory. First, in R, I don't think worrying about
    memory is a useful way to spend your time: worry about it when it becomes
    a problem (i.e. you run out of memory), not before. Second, R isn't stupid:
    it will reuse the shared columns in a pipeline of data frame transformations.
    
    You can see that using `pryr::object_size()` (unfortunately the built-in
    `object.size()` doesn't have quite enough smarts to show you this super 
    important feature of R):
    
    ```{R}
    diamonds <- ggplot2::diamonds
    pryr::object_size(diamonds)
    
    diamonds2 <- dplyr::mutate(diamonds, price_per_carat = price / carat)
    pryr::object_size(diamonds2)
    
    pryr::object_size(diamonds, diamonds2)
    ```
    
    `diamonds` is 3.46 MB, and `diamonds2` is 3.89 MB, but the total size of 
    `diamonds` and `diamonds2` is only 3.89 MB. How does that work?
    only 3.89 MB

1.  Overwrite the original:

    ```R
    foo_foo <- hop_through(foo_foo, forest)
    foo_foo <- scoop_up(foo_foo, field_mouse)
    foo_foo <- bop_on(foo_foo, head)
    ```
    
    This is a minor variation of the previous form, where instead of giving
    each intermediate element its own name, you use the same name, replacing
    the previous value at each step. This is less typing (and less thinking),
    so you're less likely to make mistakes. However, it can make debugging
    painful, because if you make a mistake you'll need to start from
    scratch again. Also, I think the reptition of the object being transformed
    (here we've repeated `foo_foo` six times!) obscures the intent of the code.
    
1.  Use the pipe 

    ```R
    foo_foo %>%
      hop_through(forest) %>%
      scoop_up(field_mouse) %>%
      bop_on(head)
    ```
    
    This is my favourite form. The downside is that you need to understand
    what the pipe does, but once you've mastered that simple task, you can
    read this series of function compositions like it's a set of imperative
    actions. 
    
    (Behind the scenes magrittr converts this call to the previous form,
    using `.` as the name of the object. This makes it easier to debug than
    the first form because it avoids deeply nested fuction calls.)

## Useful intermediates

*   Whenever you write your own function that is used primarily for its 
    side-effects, you should always return the first argument invisibly, e.g.
    `invisible(x)`: that way it can easily be used in a pipe. 

    If a function doesn't follow this contract (e.g. `plot()` which returns 
    `NULL`), you can still use it with magrittr by using the "tee" operator. 
    `%T>%` works like `%>%` except instead it returns the LHS instead of the 
    RHS:

    ```{r}
    library(magrittr)
    rnorm(100) %>%
      matrix(ncol = 2) %>%
      plot() %>%
      str()
    
    rnorm(100) %>%
      matrix(ncol = 2) %T>%
      plot() %>%
      str()
    ```

*   When you run a pipe interactively, it's easy to see if something
    goes wrong. When you start writing pipes that are used in production, i.e.
    they're run automatically and a human doesn't immediately look at the output
    it's a really good idea to include some assertions that verify the data 
    looks like expect. One great way to do this is the ensurer package, 
    writen by Stefan Milton Bache (the author of magrittr). 
  
    <http://www.r-statistics.com/2014/11/the-ensurer-package-validation-inside-pipes/>

*   If you're working with functions that don't have a dataframe based API  
    (i.e. you pass them individual vectors, not a data frame and expressions 
    to be evaluated in the context of that data frame), you might find `%$%` 
    useful. It "explodes" out the variables in a data frame so that you can 
    refer to them explicitly. This is useful when working with many functions 
    in base R:
    
    ```{r}
    mtcars %$%
      cor(disp, mpg)
    ```

## When not to use the pipe

The pipe is a powerful tool, but it's not the only tool at your disposal, and it doesn't solve every problem! Generally, you should reach for another tool when:

* Your pipes get longer than five or six lines. It's a good idea to create 
  intermediate objects with meaningful names. That helps with debugging,
  because it's easier to figure out when things went wrong. It also helps
  understand the problem, because a good name can be very evocative of the
  purpose.
  
* You have multiple inputs or outputs.  

* Instead of creating a linear pipeline where you're primarily transforming
  one object, you're starting to create a directed graphs with a complex
  dependency structure. Pipes are fundamentally linear and expressing 
  complex relationships with them does not often yield clear code.

*   For assignment. magrittr provides the `%<>%` operator which allows you to
    replace code like:
  
    ```R
    mtcars <- mtcars %>% transform(cyl = cyl * 2)
    ```
    
    with
     
    ```R
    mtcars %<>% transform(cyl = cyl * 2)
    ```
    
    I'm not a fan of this operator because I think assignment is such a 
    special operation that it should always be clear when it's occuring.
    In my opinion, a little bit of duplication (i.e. repeating the 
    name of the object twice), is fine in return for making assignment
    more explicit.
    
    I think it also gives you a better mental model of how assignment works
    in R. The above code does not modify `mtcars`: it instead creates a 
    modified copy and then replaces the old version (this may seem like a 
    subtle point but I think it's quite important).  

## Duplication

As you become a better R programmer, you'll learn more techniques for reducing various types of duplication. This allows you to do more with less, and allows you to express yourself more clearly by taking advantage of powerful programming constructs.

Two main tools for reducing duplication are functions and for-loops. You tend to use for-loops less often in R than in other programming languages because R is a functional programming language. That means that you can extract out common patterns of for loops and put them in a function.

### Extracting out a function

Whenever you've copied and pasted code more than twice, you need to take a look at it and see if you can extract out the common components and make a function. For example, take a look at this code. What does it do?

```{r}
df <- data.frame(
  a = rnorm(10),
  b = rnorm(10),
  c = rnorm(10),
  d = rnorm(10)
)

df$a <- (df$a - min(df$a, na.rm = TRUE)) / 
  (max(df$a, na.rm = TRUE) - min(df$a, na.rm = TRUE))
df$b <- (df$b - min(df$b, na.rm = TRUE)) / 
  (max(df$a, na.rm = TRUE) - min(df$b, na.rm = TRUE))
df$c <- (df$c - min(df$c, na.rm = TRUE)) / 
  (max(df$c, na.rm = TRUE) - min(df$c, na.rm = TRUE))
df$d <- (df$d - min(df$d, na.rm = TRUE)) / 
  (max(df$d, na.rm = TRUE) - min(df$d, na.rm = TRUE))
```

You might be able to puzzle out that this rescales each column to 0--1. Did you spot the mistake? I made an error when updating the code for `df$b`, and I forgot to change an `a` to a `b`. Extracting repeated code out into a function is a good idea because it helps make your code more understandable (because you can name the operation), and it prevents you from making this sort of update error.

To write a function you need to first analyse the operation. How many inputs does it have?

```{r, eval = FALSE}
(df$a - min(df$a, na.rm = TRUE)) /
  (max(df$a, na.rm = TRUE) - min(df$a, na.rm = TRUE))
```

It's often a good idea to rewrite the code using some temporary values. Here this function only takes one input, so I'll call it `x`:

```{r}
x <- 1:10
(x - min(x, na.rm = TRUE)) / (max(x, na.rm = TRUE) - min(x, na.rm = TRUE))
```

We can also see some duplication in this code: I'm computing the `min()` and `max()` multiple times, and I could instead do that in one step:

```{r}
rng <- range(x, na.rm = TRUE)
(x - rng[1]) / (rng[2] - rng[1])
```

Now that I've simplified the code, and made sure it works, I can turn it into a function:

```{r}
rescale01 <- function(x) {
  rng <- range(x, na.rm = TRUE)
  (x - rng[1]) / (rng[2] - rng[1])
}
```

Always make sure your code works on a simple test case before creating the function!

Now we can use that to simplify our original example:

```{r}
df$a <- rescale01(df$a)
df$b <- rescale01(df$b)
df$c <- rescale01(df$c)
df$d <- rescale01(df$d)
```

This makes it more clear what we're doing, and avoids one class of copy-and-paste errors. However, we still have quite a bit of duplication: we're doing the same thing to each column.

### Common looping patterns

Before we tackle the problem of rescaling each column, lets start with a simpler case. Imagine we want to summarise each column with its median. One way to do that is to use a for loop. Every for loop has three main components:

1. Creating the space for the output.
2. The sequence to loop over.
3. The body of the loop.

```{r}
medians <- vector("numeric", ncol(df))
for (i in 1:ncol(df)) {
  medians[i] <- median(df[[i]])
}
medians
```

If you do this a lot, you should probably make a function for it:

```{r}
col_medians <- function(df) {
  out <- vector("numeric", ncol(df))
  for (i in 1:ncol(df)) {
    out[i] <- median(df[[i]])
  }
  out
}
col_medians(df)
```

Now imagine that you also want to compute the interquartile range of each column? How would you change the function? What if you also wanted to calculate the min and max?

```{r}
col_min <- function(df) {
  out <- vector("numeric", ncol(df))
  for (i in 1:ncol(df)) {
    out[i] <- min(df[[i]])
  }
  out
}
col_max <- function(df) {
  out <- vector("numeric", ncol(df))
  for (i in 1:ncol(df)) {
    out[i] <- max(df[[i]])
  }
  out
}
```

I've now copied-and-pasted this function three times, so it's time to think about how to generalise it. If you look at these functions, you'll notice that they are very similar: the only difference is the function that gets called.

I mentioned earlier that R is a functional programming language. Practically, what this means is that you can not only pass vectors and data frames to functions, but you can also pass other functions. So you can generalise these `col_*` functions by adding an additional argument:

```{r}
col_summary <- function(df, fun) {
  out <- vector("numeric", ncol(df))
  for (i in 1:ncol(df)) {
    out[i] <- fun(df[[i]])
  }
  out
}
col_summary(df, median)
col_summary(df, min)
```

We can take this one step further and use another cool feature of R functions: "`...`". "`...`" just takes any additional arguments and allows you to pass them on to another function:

```{r}
col_summary <- function(df, fun, ...) {
  out <- vector("numeric", ncol(df))
  for (i in 1:ncol(df)) {
    out[i] <- fun(df[[i]], ...)
  }
  out
}
col_summary(df, median, na.rm = TRUE)
```

If you've used R for a bit, the behaviour of function might seem familiar: it looks like the `lapply()` or `sapply()` functions. Indeed, all of the apply function in R abstract over common looping patterns.

There are two main differences with `lapply()` and `col_summary()`:

* `lapply()` returns a list. This allows it to work with any R function, not 
  just those that return numeric output.
  
* `lapply()` is written in C, not R. This gives some very minor performance
  improvements.
  
As you learn more about R, you'll learn more functions that allow you to abstract over common patterns of for loops.

### Exercises

1.   Adapt `col_summary()` so that it only applies to numeric inputs.
     You might want to start with an `is_numeric()` function that returns
     a logical vector that has a TRUE corresponding to each numeric column.

1.   How do `sapply()` and `vapply()` differ from `col_summary()`?

## Robust and readable functions

(This is an advanced topic. You shouldn't worry too much about it when you first start writing functions. Instead you should focus on getting a function that works right for the easiest 80% of the problem. Then in time, you'll learn how to get to 99% with minimal extra effort. The defaults in this book should steer you in the right direction: we avoid teaching you functions with major suprises.)

In this section you'll learn an important principle that lends itself to reliable and readable code: favour code that can be understood with a minimum of context. On one extreme, take this code:

```{r, eval = FALSE}
baz <- foo(bar, qux)
```

What does it do? You can glean only a little from the context: `foo()` is a function that takes (at least) two arguments, and it returns a result we store in `baz`. But apart from that, you have no idea. To understand what this function does, you need to read the definitions of `foo()`, `bar`, and `qux`. Using better variable names helps a lot:

```{r, eval = FALSE}
df2 <- arrange(df, qux)
```

It's now much easier to see what's going on! Function and variable names are important because they tell you about (or at least jog your memory of) what the code does. That helps you understand code in isolation, even if you don't completely understand all the details. Unfortunately naming things is hard, and its hard to give concrete advice apart from giving objects short but evocative names.  As autocomplete in RStudio has gotten better, I've tended to use longer names that are more descriptive. Short names are faster to type, but you write code relatively infrequently compared to the number of times that you read it.

The idea of minimising the context needed to understand your code goes beyond just good naming. You also want to favour functions with predictable behaviour and few surprises. If a function does radically different things when its inputs differ slightly, you'll need to carefully read the surrounding context in order to predict what it will do. The goal of this section is to educate you about the most common ways R functions can be surprising and to provide you with unsurprising alternatives.

There are three common classes of surprises in R:

1.  Unstable types: What what will `df[, x]` return? You can assume that `df` 
    is a data frame and `x` is a vector because of their names. But you don't 
    know whether this code will return a data frame or a vector because the 
    behaviour of `[` depends on the length of x.
    
1.  Non-standard evaluation: What will `filter(df, x == y)` do? It depends on 
    whether `x` or `y` or both are variable in `df` or variables in the current 
    environment.

1.  Hidden arguments: What sort of variable will `data.frame(x = "a")` 
    create? It will be either a character vector or a factor depending on 
    the value of the global `stringsAsFactors` option. 
    
Avoiding these three types of functions helps you to write code that you is easily understand and fails obviously with unexpected input. If this behaviour is so important, why do any functions behave differently? It's because R is not just a programming language, but it's also an environment for interactive data analysis. Some things make sense for interactive use (where you quickly check the output and guessing what you want is ok) but don't make sense for programming (where you want errors to arise as quickly as possible). 
You might notice that these issues revolve around data frames. That's unfortunate because data frames are the data structure you'll use most commonly. It's ironic, the most frustrating things about programming in R are features that were originally designed to make your data analysis easier! Data frames try very hard to be helpful:

```{r}
df <- data.frame(xy = c("x", "y"))
# Character vectors work hard to work with for a long time, so R
# helpfully converts to a factor for you:
class(df$xy)

# If you're only selecting a single column, R tries to be helpful
# and give you that column, rather than giving you a single column
# data frame
class(df[, "xy"])

# If you have long variable names, R is "helpful" and lets you select
# them with a unique prefix
df$x
```

These features all made sense at the time they were added to R, but computing environments have changed a lot, and these features now tend to cause a lot of problems. dplyr disables them for you:

```{r, error = TRUE}
df <- dplyr::data_frame(xy = c("x", "y"))
class(df$xy)
class(df[, "xy"])
df$x
```

### Unpredictable types

One of the most frustrating for programming is they way `[` returns a vector if the result has a single column, and returns a data frame otherwise. In other words, if you see code like `df[x, ]` you can't predict what it will return without knowing the value of `x`. This can trip you up in surprising ways. For example, imagine you've written this function to return the last row of a data frame:

```{r}
last_row <- function(df) {
  df[nrow(df), ]
}
```

It's not always going to return a row! If you give it a single column data frame, it will return a single number:

```{r}
df <- data.frame(x = 1:3)
last_row(df)
```

There are two ways to avoid this problem:

* Use `drop = FALSE`: `df[x, , drop = FALSE]`.
* Subset the data frame like a list: `df[x]`.

Using one of those techniques for `last_row()` makes it more predictable: you know it will always return a data frame.

```{r}
last_row <- function(df) {
  df[nrow(df), , drop = FALSE]
}
last_row(df)
```

Another common cause of problems is the `sapply()` function. If you've never heard of it before, feel free to skip this bit: just remember to avoid it! The problem with `sapply()` is that it tries to guess what the simplest form of output is, and it always succeeds. 

The following code shows how `sapply()` can produce three different types of data depending on the input.

```{r}
df <- data.frame(
  a = 1L,
  b = 1.5,
  y = Sys.time(),
  z = ordered(1)
)


df[1:4] %>% sapply(class) %>% str()
df[1:2] %>% sapply(class) %>% str()
df[3:4] %>% sapply(class) %>% str()
```

In the next chapter, you'll learn about the purrr package which provides a variety of alternatives. In this case, you could use `map_chr()` which always returns a character vector: if it can't, it will throw an error. Another option is the base `vapply()` function which takes a third argument indicating what the output should look like.

This doesn't make `sapply()` bad and `vapply()` and `map_chr()` good. `sapply()` is nice because you can use it interactively without having to think about what `f` will return. 95% of the time it will do the right thing, and if it doesn't you can quickly fix it. `map_chr()` is more important when your programming because a clear error message is more valuable when an operation is buried deep inside a tree of function calls. At this point its worth thinking more about 

### Non-standard evaluation

You've learned a number of functions that implement special lookup rules:

```{r, eval = FALSE}
ggplot(mpg, aes(displ, cty)) + geom_point()
filter(mpg, displ > 10)
```

These are called "non-standard evaluation", or NSE for short, because the usual lookup rules don't apply. In both cases above neither `displ` nor `cty` are present in the global environment. Instead both ggplot2 and dplyr look for them first in a data frame. This is great for interactive use, but can cause problems inside a function because they'll fall back to the global environment if the variable isn't found.

[Talk a little bit about the standard scoping rules]

For example, take this function:

```{r}
big_x <- function(df, threshold) {
  dplyr::filter(df, x > threshold)
}
```

There are two ways in which this function can fail:

1.  `df$x` might not exist. There are two potential failure modes:
      
    ```{r, error = TRUE}
    big_x(mtcars, 10)
    
    x <- 1
    big_x(mtcars, 10)
    ```
    
    The second failure mode is particularly pernicious because it doesn't 
    throw an error, but instead silently returns an incorrect result. It
    works because by design `filter()` looks in both the data frame and
    the parent environment. 
    
    It is unlikely that the variable you care about will both be missing where
    you expect it, and present where you don't expect it. But I think it's 
    worth weighing heavily in your analysis of potential failure modes because 
    it's a failure that's easy to miss (since it just silently gives a bad 
    result), and hard to track down (since you need to read a lot of context).

1.  `df$threshold` might exist:
    
    ```{r}
    df <- dplyr::data_frame(x = 1:10, threshold = 100)
    big_x(df, 5)
    ```
    
    Again, this is bad because it silently gives an unexpected result.
    
How can you avoid this problem?  Currently, you need to do this:

```{r}
big_x <- function(df, threshold) {
  if (!"x" %in% names(df)) 
    stop("`df` must contain variable called `x`.", call. = FALSE)
  
  if ("threshold" %in% names(df))
    stop("`df` must not contain variable called `threshold`.", call. = FALSE)
  
  dplyr::filter(df, x > threshold)
}
```

Because dplyr currently has no way to force a name to be interpreted as either a local or parent variable, as I've only just realised that's really you should avoid NSE.  In a future version you should be able to do:

```{r}
big_x <- function(df, threshold) {
  dplyr::filter(df, .this$x > .parent$threshold)
}
```

Another option is to implement it yourself using base subsetting:

```{r}
big_x <- function(df, threshold) {
  rows <- df$x > threshold
  df[!is.na(rows) & rows, , drop = FALSE]
}
```

The challenge is remembering that `filter()` also drops missing values, and you also need to remember to use `drop = FALSE`! 

### Relying on global options

Functions are easiest to reason about if they have two properties:

1.  Their output only depends on their inputs.
1.  They don't affect the outside world except through their return value.

The first property is particularly important. If a function has hidden additional inputs, it's very difficult to even know where the important context is!

The biggest breaker of this rule in base R are functions that create data frames. Most of these functions have a `stringsAsFactors` argument that defaults to `getOption("stringsAsFactors")`. This means that a global option affects the operation of a very large number of functions, and you need to be aware that depending on an external state a function might produce either a character vector or a factor. In this book, we steer you away from that problem by recommnding functions like `readr::read_csv()` and `dplyr::data_frame()` that don't rely on this option. But be aware of it!  Generally if a function is affected by a global option, you should avoid setting it.

Only use `options()` to control side-effects of a function. The value of an option should never affect the return value of a function. There are only three violations of this rule in base R: `stringsAsFactors`, `encoding`, `na.action`. For example, base R lets you control the number of digits printed in default displays with (e.g.) `options(digits = 3)`. This is a good use of an option because it's something that people frequently want control over, but doesn't affect the computation of a result, just its display. Follow this principle with your own use of options.

### Trying too hard

Another class of problems is functions that try really really hard to always return a useful result. Unfortunately they try so hard that they never throw error messages so you never find out if the input is really really weird.

### Exercises

1.  Look at the `encoding` argument to `file()`, `url()`, `gzfile()` etc. 
    What is the default value? Why should you avoid setting the default
    value on a global level?