Prerequisites

In this chapter, you’ll learn how to load flat files in R with the readr package, which is part of the core tidyverse.

library(tidyverse)

Practical advice

Once you read data in, the first step usually involves transforming it in some way to make it easier to work with in the rest of your analysis. Let’s take another look at the students data with that in mind.

In the favourite.food column, there are a bunch of food items and then the character string N/A, which should have been an real NA that R will recognize as “not available”. This is something we can address using the na argument.

students <- read_csv("data/students.csv", na = c("N/A", ""))

students
#> # A tibble: 6 × 5
#>   `Student ID` `Full Name`      favourite.food     mealPlan            AGE  
#>          <dbl> <chr>            <chr>              <chr>               <chr>
#> 1            1 Sunil Huffmann   Strawberry yoghurt Lunch only          4    
#> 2            2 Barclay Lynn     French fries       Lunch only          5    
#> 3            3 Jayendra Lyne    <NA>               Breakfast and lunch 7    
#> 4            4 Leon Rossini     Anchovies          Lunch only          <NA> 
#> 5            5 Chidiegwu Dunkel Pizza              Breakfast and lunch five 
#> 6            6 Güvenç Attila    Ice cream          Lunch only          6

You might also notice that the Student ID and Full Name columns are surrounded by back ticks. That’s because they contain spaces, breaking R’s usual rules for variable names. To refer to them, you need to use those back ticks:

students |> 
  rename(
    student_id = `Student ID`,
    full_name = `Full Name`
  )
#> # A tibble: 6 × 5
#>   student_id full_name        favourite.food     mealPlan            AGE  
#>        <dbl> <chr>            <chr>              <chr>               <chr>
#> 1          1 Sunil Huffmann   Strawberry yoghurt Lunch only          4    
#> 2          2 Barclay Lynn     French fries       Lunch only          5    
#> 3          3 Jayendra Lyne    <NA>               Breakfast and lunch 7    
#> 4          4 Leon Rossini     Anchovies          Lunch only          <NA> 
#> 5          5 Chidiegwu Dunkel Pizza              Breakfast and lunch five 
#> 6          6 Güvenç Attila    Ice cream          Lunch only          6

An alternative approach is to use janitor::clean_names() to use some heuristics to turn them all into snake case at onceThe janitor package is not part of the tidyverse, but it offers handy functions for data cleaning and works well within data pipelines that uses |>..

students |> janitor::clean_names()
#> # A tibble: 6 × 5
#>   student_id full_name        favourite_food     meal_plan           age  
#>        <dbl> <chr>            <chr>              <chr>               <chr>
#> 1          1 Sunil Huffmann   Strawberry yoghurt Lunch only          4    
#> 2          2 Barclay Lynn     French fries       Lunch only          5    
#> 3          3 Jayendra Lyne    <NA>               Breakfast and lunch 7    
#> 4          4 Leon Rossini     Anchovies          Lunch only          <NA> 
#> 5          5 Chidiegwu Dunkel Pizza              Breakfast and lunch five 
#> 6          6 Güvenç Attila    Ice cream          Lunch only          6

Another common task after reading in data is to consider variable types. For example, meal_type is a categorical variable with a known set of possible values, which in R should be represent as factor:

students |>
  janitor::clean_names() |>
  mutate(
    meal_plan = factor(meal_plan)
  )
#> # A tibble: 6 × 5
#>   student_id full_name        favourite_food     meal_plan           age  
#>        <dbl> <chr>            <chr>              <fct>               <chr>
#> 1          1 Sunil Huffmann   Strawberry yoghurt Lunch only          4    
#> 2          2 Barclay Lynn     French fries       Lunch only          5    
#> 3          3 Jayendra Lyne    <NA>               Breakfast and lunch 7    
#> 4          4 Leon Rossini     Anchovies          Lunch only          <NA> 
#> 5          5 Chidiegwu Dunkel Pizza              Breakfast and lunch five 
#> 6          6 Güvenç Attila    Ice cream          Lunch only          6

Note that the values in the meal_type variable has stayed exactly the same, but the type of variable denoted underneath the variable name has changed from character (<chr>) to factor (<fct>). You’ll learn more about factors in #chp-factors.

Before you move on to analyzing these data, you’ll probably want to fix the age column as well: currently it’s a character variable because of the one observation that is typed out as five instead of a numeric 5. We discuss the details of fixing this issue in #chp-spreadsheets.

students <- students |>
  janitor::clean_names() |>
  mutate(
    meal_plan = factor(meal_plan),
    age = parse_number(if_else(age == "five", "5", age))
  )

students
#> # A tibble: 6 × 5
#>   student_id full_name        favourite_food     meal_plan             age
#>        <dbl> <chr>            <chr>              <fct>               <dbl>
#> 1          1 Sunil Huffmann   Strawberry yoghurt Lunch only              4
#> 2          2 Barclay Lynn     French fries       Lunch only              5
#> 3          3 Jayendra Lyne    <NA>               Breakfast and lunch     7
#> 4          4 Leon Rossini     Anchovies          Lunch only             NA
#> 5          5 Chidiegwu Dunkel Pizza              Breakfast and lunch     5
#> 6          6 Güvenç Attila    Ice cream          Lunch only              6

Other arguments

There are a couple of other important arguments that we need to mention, and they’ll be easier to demonstrate if we first show you a handy trick: read_csv() can read csv files that you’ve created in a string:

read_csv(
  "a,b,c
  1,2,3
  4,5,6"
)
#> # A tibble: 2 × 3
#>       a     b     c
#>   <dbl> <dbl> <dbl>
#> 1     1     2     3
#> 2     4     5     6

Usually read_csv() uses the first line of the data for the column names, which is a very common convention. But sometime there are a few lines of metadata at the top of the file. You can use skip = n to skip the first n lines or use comment = "#" to drop all lines that start with (e.g.) #:

read_csv(
  "The first line of metadata
  The second line of metadata
  x,y,z
  1,2,3",
  skip = 2
)
#> # A tibble: 1 × 3
#>       x     y     z
#>   <dbl> <dbl> <dbl>
#> 1     1     2     3

read_csv(
  "# A comment I want to skip
  x,y,z
  1,2,3",
  comment = "#"
)
#> # A tibble: 1 × 3
#>       x     y     z
#>   <dbl> <dbl> <dbl>
#> 1     1     2     3

In other cases, the data might not have column names. You can use col_names = FALSE to tell read_csv() not to treat the first row as headings, and instead label them sequentially from X1 to Xn:

read_csv(
  "1,2,3
  4,5,6",
  col_names = FALSE
)
#> # A tibble: 2 × 3
#>      X1    X2    X3
#>   <dbl> <dbl> <dbl>
#> 1     1     2     3
#> 2     4     5     6

Alternatively you can pass col_names a character vector which will be used as the column names:

read_csv(
  "1,2,3
  4,5,6",
  col_names = c("x", "y", "z")
)
#> # A tibble: 2 × 3
#>       x     y     z
#>   <dbl> <dbl> <dbl>
#> 1     1     2     3
#> 2     4     5     6

These arguments are all you need to know to read the majority of CSV files that you’ll encounter in practice. (For the rest, you’ll need to carefully inspect your .csv file and carefully read the documentation for read_csv()’s many other arguments.)

Other file types

Once you’ve mastered read_csv(), using readr’s other functions is straightforward; it’s just a matter of knowing which function to reach for:

• read_csv2() reads semicolon separated files. These use ; instead of , to separate fields, and are common in countries that use , as the decimal marker.

• read_tsv() reads tab delimited files.

• read_delim() reads in files with any delimiter, attempting to automatically guess the delimited if you don’t specify it.

• read_fwf() reads fixed width files. You can specify fields either by their widths with fwf_widths() or their position with fwf_positions().

• read_table() reads a common variation of fixed width files where columns are separated by white space.

• read_log() reads Apache style log files.

Exercises

1. What function would you use to read a file where fields were separated with “|”?

2. Apart from file, skip, and comment, what other arguments do read_csv() and read_tsv() have in common?

3. What are the most important arguments to read_fwf()?

4. Sometimes strings in a CSV file contain commas. To prevent them from causing problems they need to be surrounded by a quoting character, like " or '. By default, read_csv() assumes that the quoting character will be ". What argument to read_csv() do you need to specify to read the following text into a data frame?

   "x,y\n1,'a,b'"

5. Identify what is wrong with each of the following inline CSV files. What happens when you run the code?

   read_csv("a,b\n1,2,3\n4,5,6")
   read_csv("a,b,c\n1,2\n1,2,3,4")
   read_csv("a,b\n\"1")
   read_csv("a,b\n1,2\na,b")
   read_csv("a;b\n1;3")

6. Practice referring to non-syntactic names in the following data frame by:

   1. Extracting the variable called 1.
   2. Plotting a scatterplot of 1 vs 2.
   3. Creating a new column called 3 which is 2 divided by 1.
   4. Renaming the columns to one, two and three.

   annoying <- tibble(
     `1` = 1:10,
     `2` = `1` * 2 + rnorm(length(`1`))
   )

Guessing types

readr uses a heuristic to figure out the column types. For each column, it pulls the values of 1,000You can override the default of 1000 with the guess_max argument. rows spaced evenly from the first row to the last, ignoring an missing values. It then works through the following questions:

• Does it contain only F, T, FALSE, or TRUE (ignoring case)? If so, it’s a logical.
• Does it contain only numbers (e.g. 1, -4.5, 5e6, Inf)? If so, it’s a number.
• Does it match match the ISO8601 standard? If so, it’s a date or date-time. (We’ll come back to date/times in more detail in #sec-creating-datetimes).
• Otherwise, it must be a string.

You can see that behavior in action in this simple example:

read_csv("
  logical,numeric,date,string
  TRUE,1,2021-01-15,abc
  false,4.5,2021-02-15,def
  T,Inf,2021-02-16,ghi"
)
#> Rows: 3 Columns: 4
#> ── Column specification ─────────────────────────────────────────────────────
#> Delimiter: ","
#> chr  (1): string
#> dbl  (1): numeric
#> lgl  (1): logical
#> date (1): date
#> 
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
#> # A tibble: 3 × 4
#>   logical numeric date       string
#>   <lgl>     <dbl> <date>     <chr> 
#> 1 TRUE        1   2021-01-15 abc   
#> 2 FALSE       4.5 2021-02-15 def   
#> 3 TRUE      Inf   2021-02-16 ghi

This heuristic works well if you have a clean dataset, but in real life you’ll encounter a selection of weird and wonderful failures.

Missing values, column types, and problems

The most common way column detection fails is that a column contains unexpected values and you get a character column instead of a more specific type. One of the most common causes for this a missing value, recorded using something other than the NA that stringr expects.

Take this simple 1 column CSV file as an example:

csv <- "
  x
  10
  .
  20
  30"

If we read it without any additional arguments, x becomes a character column:

df <- read_csv(csv)
#> Rows: 4 Columns: 1
#> ── Column specification ─────────────────────────────────────────────────────
#> Delimiter: ","
#> chr (1): x
#> 
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

In this very small case, you can easily see the missing value .. But what happens if you have thousands of rows with only a few missing values represented by .s speckled amongst them? One approach is to tell readr that x is a numeric column, and then see where it fails. You can do that with the col_types argument, which takes a named list:

df <- read_csv(csv, col_types = list(x = col_double()))
#> Warning: One or more parsing issues, call `problems()` on your data frame for
#> details, e.g.:
#>   dat <- vroom(...)
#>   problems(dat)

Now read_csv() reports that there was a problem, and tells us we can find out more with problems():

problems(df)
#> # A tibble: 1 × 5
#>     row   col expected actual file                                    
#>   <int> <int> <chr>    <chr>  <chr>                                   
#> 1     3     1 a double .      /private/tmp/RtmpZYGhlj/file9e8176037b8c

This tells us that there was a problem in row 3, col 1 where readr expected a double but got a .. That suggests this dataset uses . for missing values. So then we set na = ".", the automatic guessing succeeds, giving us the numeric column that we want:

df <- read_csv(csv, na = ".")
#> Rows: 4 Columns: 1
#> ── Column specification ─────────────────────────────────────────────────────
#> Delimiter: ","
#> dbl (1): x
#> 
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

Column types

readr provides a total of nine column types for you to use:

• col_logical() and col_double() read logicals and real numbers. They’re relatively rarely needed (except as above), since readr will usually guess them for you.
• col_integer() reads integers. We distinguish because integers and doubles in this book because they’re functionally equivalent, but reading integers explicitly can occasionally be useful because they occupy half the memory of doubles.
• col_character() reads strings. This is sometimes useful to specify explicitly when you have a column that is a numeric identifier, i.e. long series of digits that identifies some object, but it doesn’t make sense to (e.g.) divide it in half.
• col_factor(), col_date() and col_datetime() create factors, dates and date-time respectively; you’ll learn more about those when we get to those data types in #chp-factors and #chp-datetimes.
• col_number() is a permissive numeric parser that will ignore non-numeric components, and is particularly useful for currencies. You’ll learn more about it in #chp-numbers.
• col_skip() skips a column so it’s not included in the result.

It’s also possible to override the default column by switching from list() to cols():

csv <- "
x,y,z
1,2,3"

read_csv(csv, col_types = cols(.default = col_character()))
#> # A tibble: 1 × 3
#>   x     y     z    
#>   <chr> <chr> <chr>
#> 1 1     2     3

Another useful helper is cols_only() which will read in only the columns you specify:

read_csv(
  "x,y,z
  1,2,3",
  col_types = cols_only(x = col_character())
)
#> # A tibble: 1 × 1
#>   x    
#>   <chr>
#> 1 1