r4ds/oreilly/data-import.html

596 lines
40 KiB
HTML
Raw Normal View History

<section data-type="chapter" id="chp-data-import">
2022-11-19 01:55:22 +08:00
<h1><span id="sec-data-import" class="quarto-section-identifier d-none d-lg-block"><span class="chapter-title">Data import</span></span></h1>
<section id="introduction" data-type="sect1">
<h1>
Introduction</h1>
2023-01-13 07:22:57 +08:00
<p>Working with data provided by R packages is a great way to learn data science tools, but you want to apply what youve learned to your own data at some point. In this chapter, youll learn the basics of reading data files into R.</p>
<p>Specifically, this chapter will focus on reading plain-text rectangular files. Well start with practical advice for handling features like column names, types, and missing data. You will then learn about reading data from multiple files at once and writing data from R to a file. Finally, youll learn how to handcraft data frames in R.</p>
<section id="prerequisites" data-type="sect2">
<h2>
Prerequisites</h2>
<p>In this chapter, youll learn how to load flat files in R with the <strong>readr</strong> package, which is part of the core tidyverse.</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">library(tidyverse)</pre>
</div>
</section>
</section>
<section id="reading-data-from-a-file" data-type="sect1">
<h1>
Reading data from a file</h1>
2023-01-13 07:22:57 +08:00
<p>To begin, well focus on the most rectangular data file type: CSV, which is short for comma-separated values. Here is what a simple CSV file looks like. The first row, commonly called the header row, gives the column names, and the following six rows provide the data.</p>
<div class="cell">
<pre><code>#&gt; Student ID,Full Name,favourite.food,mealPlan,AGE
#&gt; 1,Sunil Huffmann,Strawberry yoghurt,Lunch only,4
#&gt; 2,Barclay Lynn,French fries,Lunch only,5
#&gt; 3,Jayendra Lyne,N/A,Breakfast and lunch,7
#&gt; 4,Leon Rossini,Anchovies,Lunch only,
#&gt; 5,Chidiegwu Dunkel,Pizza,Breakfast and lunch,five
#&gt; 6,Güvenç Attila,Ice cream,Lunch only,6</code></pre>
</div>
<p><a href="#tbl-students-table" data-type="xref">#tbl-students-table</a> shows a representation of the same data as a table.</p>
<div class="cell">
<div class="cell-output-display">
<div id="tbl-students-table" class="anchored">
<table class="table table-sm table-striped"><caption>Table 8.1: Data from the students.csv file as a table.</caption>
<colgroup><col style="width: 15%"/><col style="width: 23%"/><col style="width: 26%"/><col style="width: 27%"/><col style="width: 6%"/></colgroup><thead><tr class="header"><th style="text-align: right;">Student ID</th>
<th style="text-align: left;">Full Name</th>
<th style="text-align: left;">favourite.food</th>
<th style="text-align: left;">mealPlan</th>
<th style="text-align: left;">AGE</th>
</tr></thead><tbody><tr class="odd"><td style="text-align: right;">1</td>
<td style="text-align: left;">Sunil Huffmann</td>
<td style="text-align: left;">Strawberry yoghurt</td>
<td style="text-align: left;">Lunch only</td>
<td style="text-align: left;">4</td>
</tr><tr class="even"><td style="text-align: right;">2</td>
<td style="text-align: left;">Barclay Lynn</td>
<td style="text-align: left;">French fries</td>
<td style="text-align: left;">Lunch only</td>
<td style="text-align: left;">5</td>
</tr><tr class="odd"><td style="text-align: right;">3</td>
<td style="text-align: left;">Jayendra Lyne</td>
<td style="text-align: left;">N/A</td>
<td style="text-align: left;">Breakfast and lunch</td>
<td style="text-align: left;">7</td>
</tr><tr class="even"><td style="text-align: right;">4</td>
<td style="text-align: left;">Leon Rossini</td>
<td style="text-align: left;">Anchovies</td>
<td style="text-align: left;">Lunch only</td>
<td style="text-align: left;">NA</td>
</tr><tr class="odd"><td style="text-align: right;">5</td>
<td style="text-align: left;">Chidiegwu Dunkel</td>
<td style="text-align: left;">Pizza</td>
<td style="text-align: left;">Breakfast and lunch</td>
<td style="text-align: left;">five</td>
</tr><tr class="even"><td style="text-align: right;">6</td>
<td style="text-align: left;">Güvenç Attila</td>
<td style="text-align: left;">Ice cream</td>
<td style="text-align: left;">Lunch only</td>
<td style="text-align: left;">6</td>
</tr></tbody></table></div>
</div>
</div>
2022-11-19 00:30:32 +08:00
<p>We can read this file into R using <code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_csv()</a></code>. The first argument is the most important: its the path to the file.</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">students &lt;- read_csv("data/students.csv")
#&gt; Rows: 6 Columns: 5
#&gt; ── Column specification ─────────────────────────────────────────────────────
#&gt; Delimiter: ","
#&gt; chr (4): Full Name, favourite.food, mealPlan, AGE
#&gt; dbl (1): Student ID
#&gt;
#&gt; Use `spec()` to retrieve the full column specification for this data.
#&gt; Specify the column types or set `show_col_types = FALSE` to quiet this message.</pre>
</div>
2023-01-13 07:22:57 +08:00
<p>When you run <code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_csv()</a></code>, it prints out a message telling you the number of rows and columns of data, the delimiter that was used, and the column specifications (names of columns organized by the type of data the column contains). It also prints out some information about retrieving the full column specification and how to quiet this message. This message is an integral part of readr, and well return to it in <a href="#sec-col-types" data-type="xref">#sec-col-types</a>.</p>
<section id="practical-advice" data-type="sect2">
<h2>
Practical advice</h2>
<p>Once you read data in, the first step usually involves transforming it in some way to make it easier to work with in the rest of your analysis. Lets take another look at the <code>students</code> data with that in mind.</p>
2023-01-13 07:22:57 +08:00
<p>In the <code>favourite.food</code> column, there are a bunch of food items, and then the character string <code>N/A</code>, which should have been a real <code>NA</code> that R will recognize as “not available”. This is something we can address using the <code>na</code> argument.</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">students &lt;- read_csv("data/students.csv", na = c("N/A", ""))
students
#&gt; # A tibble: 6 × 5
#&gt; `Student ID` `Full Name` favourite.food mealPlan AGE
#&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt;
#&gt; 1 1 Sunil Huffmann Strawberry yoghurt Lunch only 4
#&gt; 2 2 Barclay Lynn French fries Lunch only 5
#&gt; 3 3 Jayendra Lyne &lt;NA&gt; Breakfast and lunch 7
#&gt; 4 4 Leon Rossini Anchovies Lunch only &lt;NA&gt;
#&gt; 5 5 Chidiegwu Dunkel Pizza Breakfast and lunch five
#&gt; 6 6 Güvenç Attila Ice cream Lunch only 6</pre>
</div>
2023-01-13 07:22:57 +08:00
<p>You might also notice that the <code>Student ID</code> and <code>Full Name</code> columns are surrounded by backticks. Thats because they contain spaces, breaking Rs usual rules for variable names. To refer to them, you need to use those backticks:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">students |&gt;
rename(
student_id = `Student ID`,
full_name = `Full Name`
)
#&gt; # A tibble: 6 × 5
#&gt; student_id full_name favourite.food mealPlan AGE
#&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt;
#&gt; 1 1 Sunil Huffmann Strawberry yoghurt Lunch only 4
#&gt; 2 2 Barclay Lynn French fries Lunch only 5
#&gt; 3 3 Jayendra Lyne &lt;NA&gt; Breakfast and lunch 7
#&gt; 4 4 Leon Rossini Anchovies Lunch only &lt;NA&gt;
#&gt; 5 5 Chidiegwu Dunkel Pizza Breakfast and lunch five
#&gt; 6 6 Güvenç Attila Ice cream Lunch only 6</pre>
</div>
2022-11-19 00:30:32 +08:00
<p>An alternative approach is to use <code><a href="https://rdrr.io/pkg/janitor/man/clean_names.html">janitor::clean_names()</a></code> to use some heuristics to turn them all into snake case at once<span data-type="footnote">The <a href="http://sfirke.github.io/janitor/">janitor</a> package is not part of the tidyverse, but it offers handy functions for data cleaning and works well within data pipelines that uses <code>|&gt;</code>.</span>.</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">students |&gt; janitor::clean_names()
#&gt; # A tibble: 6 × 5
#&gt; student_id full_name favourite_food meal_plan age
#&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt;
#&gt; 1 1 Sunil Huffmann Strawberry yoghurt Lunch only 4
#&gt; 2 2 Barclay Lynn French fries Lunch only 5
#&gt; 3 3 Jayendra Lyne &lt;NA&gt; Breakfast and lunch 7
#&gt; 4 4 Leon Rossini Anchovies Lunch only &lt;NA&gt;
#&gt; 5 5 Chidiegwu Dunkel Pizza Breakfast and lunch five
#&gt; 6 6 Güvenç Attila Ice cream Lunch only 6</pre>
</div>
2023-01-13 07:22:57 +08:00
<p>Another common task after reading in data is to consider variable types. For example, <code>meal_type</code> is a categorical variable with a known set of possible values, which in R should be represented as a factor:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">students |&gt;
janitor::clean_names() |&gt;
mutate(
meal_plan = factor(meal_plan)
)
#&gt; # A tibble: 6 × 5
#&gt; student_id full_name favourite_food meal_plan age
#&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;fct&gt; &lt;chr&gt;
#&gt; 1 1 Sunil Huffmann Strawberry yoghurt Lunch only 4
#&gt; 2 2 Barclay Lynn French fries Lunch only 5
#&gt; 3 3 Jayendra Lyne &lt;NA&gt; Breakfast and lunch 7
#&gt; 4 4 Leon Rossini Anchovies Lunch only &lt;NA&gt;
#&gt; 5 5 Chidiegwu Dunkel Pizza Breakfast and lunch five
#&gt; 6 6 Güvenç Attila Ice cream Lunch only 6</pre>
</div>
2023-01-13 07:22:57 +08:00
<p>Note that the values in the <code>meal_type</code> variable have stayed the same, but the type of variable denoted underneath the variable name has changed from character (<code>&lt;chr&gt;</code>) to factor (<code>&lt;fct&gt;</code>). Youll learn more about factors in <a href="#chp-factors" data-type="xref">#chp-factors</a>.</p>
<p>Before you analyze these data, youll probably want to fix the <code>age</code> column. Currently, its a character variable because one of the observations is typed out as <code>five</code> instead of a numeric <code>5</code>. We discuss the details of fixing this issue in <a href="#chp-spreadsheets" data-type="xref">#chp-spreadsheets</a>.</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">students &lt;- students |&gt;
janitor::clean_names() |&gt;
mutate(
meal_plan = factor(meal_plan),
age = parse_number(if_else(age == "five", "5", age))
)
students
#&gt; # A tibble: 6 × 5
#&gt; student_id full_name favourite_food meal_plan age
#&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;fct&gt; &lt;dbl&gt;
#&gt; 1 1 Sunil Huffmann Strawberry yoghurt Lunch only 4
#&gt; 2 2 Barclay Lynn French fries Lunch only 5
#&gt; 3 3 Jayendra Lyne &lt;NA&gt; Breakfast and lunch 7
#&gt; 4 4 Leon Rossini Anchovies Lunch only NA
#&gt; 5 5 Chidiegwu Dunkel Pizza Breakfast and lunch 5
#&gt; 6 6 Güvenç Attila Ice cream Lunch only 6</pre>
</div>
</section>
<section id="other-arguments" data-type="sect2">
<h2>
Other arguments</h2>
2023-01-13 07:22:57 +08:00
<p>There are a couple of other important arguments that we need to mention, and theyll be easier to demonstrate if we first show you a handy trick: <code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_csv()</a></code> can read CSV files that youve created in a string:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">read_csv(
"a,b,c
1,2,3
4,5,6"
)
#&gt; # A tibble: 2 × 3
#&gt; a b c
#&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
#&gt; 1 1 2 3
#&gt; 2 4 5 6</pre>
</div>
2023-01-13 07:22:57 +08:00
<p>Usually, <code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_csv()</a></code> uses the first line of the data for the column names, which is a very common convention. But its not uncommon for a few lines of metadata to be included at the top of the file. You can use <code>skip = n</code> to skip the first <code>n</code> lines or use <code>comment = "#"</code> to drop all lines that start with (e.g.) <code>#</code>:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">read_csv(
"The first line of metadata
The second line of metadata
x,y,z
1,2,3",
skip = 2
)
#&gt; # A tibble: 1 × 3
#&gt; x y z
#&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
#&gt; 1 1 2 3
read_csv(
"# A comment I want to skip
x,y,z
1,2,3",
comment = "#"
)
#&gt; # A tibble: 1 × 3
#&gt; x y z
#&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
#&gt; 1 1 2 3</pre>
</div>
2023-01-13 07:22:57 +08:00
<p>In other cases, the data might not have column names. You can use <code>col_names = FALSE</code> to tell <code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_csv()</a></code> not to treat the first row as headings and instead label them sequentially from <code>X1</code> to <code>Xn</code>:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">read_csv(
"1,2,3
4,5,6",
col_names = FALSE
)
#&gt; # A tibble: 2 × 3
#&gt; X1 X2 X3
#&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
#&gt; 1 1 2 3
#&gt; 2 4 5 6</pre>
</div>
2023-01-13 07:22:57 +08:00
<p>Alternatively, you can pass <code>col_names</code> a character vector which will be used as the column names:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">read_csv(
"1,2,3
4,5,6",
col_names = c("x", "y", "z")
)
#&gt; # A tibble: 2 × 3
#&gt; x y z
#&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
#&gt; 1 1 2 3
#&gt; 2 4 5 6</pre>
</div>
2023-01-13 07:22:57 +08:00
<p>These arguments are all you need to know to read the majority of CSV files that youll encounter in practice. (For the rest, youll need to carefully inspect your <code>.csv</code> file and read the documentation for <code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_csv()</a></code>s many other arguments.)</p>
</section>
<section id="other-file-types" data-type="sect2">
<h2>
Other file types</h2>
2022-11-19 00:30:32 +08:00
<p>Once youve mastered <code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_csv()</a></code>, using readrs other functions is straightforward; its just a matter of knowing which function to reach for:</p>
2023-01-13 07:22:57 +08:00
<ul><li><p><code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_csv2()</a></code> reads semicolon-separated files. These use <code>;</code> instead of <code>,</code> to separate fields and are common in countries that use <code>,</code> as the decimal marker.</p></li>
<li><p><code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_tsv()</a></code> reads tab-delimited files.</p></li>
<li><p><code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_delim()</a></code> reads in files with any delimiter, attempting to automatically guess the delimiter if you dont specify it.</p></li>
<li><p><code><a href="https://readr.tidyverse.org/reference/read_fwf.html">read_fwf()</a></code> reads fixed-width files. You can specify fields by their widths with <code><a href="https://readr.tidyverse.org/reference/read_fwf.html">fwf_widths()</a></code> or by their positions with <code><a href="https://readr.tidyverse.org/reference/read_fwf.html">fwf_positions()</a></code>.</p></li>
<li><p><code><a href="https://readr.tidyverse.org/reference/read_table.html">read_table()</a></code> reads a common variation of fixed-width files where columns are separated by white space.</p></li>
<li><p><code><a href="https://readr.tidyverse.org/reference/read_log.html">read_log()</a></code> reads Apache-style log files.</p></li>
</ul></section>
<section id="exercises" data-type="sect2">
<h2>
Exercises</h2>
<ol type="1"><li><p>What function would you use to read a file where fields were separated with “|”?</p></li>
2022-11-19 00:30:32 +08:00
<li><p>Apart from <code>file</code>, <code>skip</code>, and <code>comment</code>, what other arguments do <code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_csv()</a></code> and <code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_tsv()</a></code> have in common?</p></li>
<li><p>What are the most important arguments to <code><a href="https://readr.tidyverse.org/reference/read_fwf.html">read_fwf()</a></code>?</p></li>
<li>
2023-01-13 07:22:57 +08:00
<p>Sometimes strings in a CSV file contain commas. To prevent them from causing problems, they need to be surrounded by a quoting character, like <code>"</code> or <code>'</code>. By default, <code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_csv()</a></code> assumes that the quoting character will be <code>"</code>. To read the following text into a data frame, what argument to <code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_csv()</a></code> do you need to specify?</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">"x,y\n1,'a,b'"</pre>
</div>
</li>
<li>
<p>Identify what is wrong with each of the following inline CSV files. What happens when you run the code?</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">read_csv("a,b\n1,2,3\n4,5,6")
read_csv("a,b,c\n1,2\n1,2,3,4")
read_csv("a,b\n\"1")
read_csv("a,b\n1,2\na,b")
read_csv("a;b\n1;3")</pre>
</div>
</li>
<li>
<p>Practice referring to non-syntactic names in the following data frame by:</p>
<ol type="a"><li>Extracting the variable called <code>1</code>.</li>
2023-01-13 07:22:57 +08:00
<li>Plotting a scatterplot of <code>1</code> vs. <code>2</code>.</li>
<li>Creating a new column called <code>3</code>, which is <code>2</code> divided by <code>1</code>.</li>
<li>Renaming the columns to <code>one</code>, <code>two</code>, and <code>three</code>.</li>
</ol><div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">annoying &lt;- tibble(
`1` = 1:10,
`2` = `1` * 2 + rnorm(length(`1`))
)</pre>
</div>
</li>
</ol></section>
</section>
<section id="sec-col-types" data-type="sect1">
<h1>
Controlling column types</h1>
2023-01-13 07:22:57 +08:00
<p>A CSV file doesnt contain any information about the type of each variable (i.e., whether its a logical, number, string, etc.), so readr will try to guess the type. This section describes how the guessing process works, how to resolve some common problems that cause it to fail, and, if needed, how to supply the column types yourself. Finally, well mention a few general strategies that are useful if readr is failing catastrophically and you need to get more insight into the structure of your file.</p>
<section id="guessing-types" data-type="sect2">
<h2>
Guessing types</h2>
2023-01-13 07:22:57 +08:00
<p>readr uses a heuristic to figure out the column types. For each column, it pulls the values of 1,000<span data-type="footnote">You can override the default of 1000 with the <code>guess_max</code> argument.</span> rows spaced evenly from the first row to the last, ignoring missing values. It then works through the following questions:</p>
<ul><li>Does it contain only <code>F</code>, <code>T</code>, <code>FALSE</code>, or <code>TRUE</code> (ignoring case)? If so, its a logical.</li>
2023-01-13 07:22:57 +08:00
<li>Does it contain only numbers (e.g., <code>1</code>, <code>-4.5</code>, <code>5e6</code>, <code>Inf</code>)? If so, its a number.</li>
<li>Does it match the ISO8601 standard? If so, its a date or date-time. (Well return to date-times in more detail in <a href="#sec-creating-datetimes" data-type="xref">#sec-creating-datetimes</a>).</li>
<li>Otherwise, it must be a string.</li>
</ul><p>You can see that behavior in action in this simple example:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">read_csv("
logical,numeric,date,string
TRUE,1,2021-01-15,abc
false,4.5,2021-02-15,def
T,Inf,2021-02-16,ghi"
)
#&gt; Rows: 3 Columns: 4
#&gt; ── Column specification ─────────────────────────────────────────────────────
#&gt; Delimiter: ","
#&gt; chr (1): string
#&gt; dbl (1): numeric
#&gt; lgl (1): logical
#&gt; date (1): date
#&gt;
#&gt; Use `spec()` to retrieve the full column specification for this data.
#&gt; Specify the column types or set `show_col_types = FALSE` to quiet this message.
#&gt; # A tibble: 3 × 4
#&gt; logical numeric date string
#&gt; &lt;lgl&gt; &lt;dbl&gt; &lt;date&gt; &lt;chr&gt;
#&gt; 1 TRUE 1 2021-01-15 abc
#&gt; 2 FALSE 4.5 2021-02-15 def
#&gt; 3 TRUE Inf 2021-02-16 ghi</pre>
</div>
2023-01-13 07:22:57 +08:00
<p>This heuristic works well if you have a clean dataset, but in real life, youll encounter a selection of weird and beautiful failures.</p>
</section>
<section id="missing-values-column-types-and-problems" data-type="sect2">
<h2>
Missing values, column types, and problems</h2>
2023-01-13 07:22:57 +08:00
<p>The most common way column detection fails is that a column contains unexpected values, and you get a character column instead of a more specific type. One of the most common causes for this is a missing value, recorded using something other than the <code>NA</code> that stringr expects.</p>
<p>Take this simple 1 column CSV file as an example:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">csv &lt;- "
x
10
.
20
30"</pre>
</div>
<p>If we read it without any additional arguments, <code>x</code> becomes a character column:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">df &lt;- read_csv(csv)
#&gt; Rows: 4 Columns: 1
#&gt; ── Column specification ─────────────────────────────────────────────────────
#&gt; Delimiter: ","
#&gt; chr (1): x
#&gt;
#&gt; Use `spec()` to retrieve the full column specification for this data.
#&gt; Specify the column types or set `show_col_types = FALSE` to quiet this message.</pre>
</div>
2023-01-13 07:22:57 +08:00
<p>In this very small case, you can easily see the missing value <code>.</code>. But what happens if you have thousands of rows with only a few missing values represented by <code>.</code>s speckled among them? One approach is to tell readr that <code>x</code> is a numeric column, and then see where it fails. You can do that with the <code>col_types</code> argument, which takes a named list:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">df &lt;- read_csv(csv, col_types = list(x = col_double()))
#&gt; Warning: One or more parsing issues, call `problems()` on your data frame for
#&gt; details, e.g.:
#&gt; dat &lt;- vroom(...)
#&gt; problems(dat)</pre>
</div>
2022-11-19 00:30:32 +08:00
<p>Now <code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_csv()</a></code> reports that there was a problem, and tells us we can find out more with <code><a href="https://readr.tidyverse.org/reference/problems.html">problems()</a></code>:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">problems(df)
#&gt; # A tibble: 1 × 5
2023-01-13 07:22:57 +08:00
#&gt; row col expected actual file
#&gt; &lt;int&gt; &lt;int&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt;
#&gt; 1 3 1 a double . /private/tmp/Rtmp1nE0XP/file11b88112257a4</pre>
</div>
<p>This tells us that there was a problem in row 3, col 1 where readr expected a double but got a <code>.</code>. That suggests this dataset uses <code>.</code> for missing values. So then we set <code>na = "."</code>, the automatic guessing succeeds, giving us the numeric column that we want:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">df &lt;- read_csv(csv, na = ".")
#&gt; Rows: 4 Columns: 1
#&gt; ── Column specification ─────────────────────────────────────────────────────
#&gt; Delimiter: ","
#&gt; dbl (1): x
#&gt;
#&gt; Use `spec()` to retrieve the full column specification for this data.
#&gt; Specify the column types or set `show_col_types = FALSE` to quiet this message.</pre>
</div>
</section>
<section id="column-types" data-type="sect2">
<h2>
Column types</h2>
<p>readr provides a total of nine column types for you to use:</p>
<ul><li>
2022-11-19 00:30:32 +08:00
<code><a href="https://readr.tidyverse.org/reference/parse_atomic.html">col_logical()</a></code> and <code><a href="https://readr.tidyverse.org/reference/parse_atomic.html">col_double()</a></code> read logicals and real numbers. Theyre relatively rarely needed (except as above), since readr will usually guess them for you.</li>
<li>
2023-01-13 07:22:57 +08:00
<code><a href="https://readr.tidyverse.org/reference/parse_atomic.html">col_integer()</a></code> reads integers. We distinguish integers and doubles in this book because theyre functionally equivalent, but reading integers explicitly can occasionally be useful because they occupy half the memory of doubles.</li>
<li>
2022-11-19 00:30:32 +08:00
<code><a href="https://readr.tidyverse.org/reference/parse_atomic.html">col_character()</a></code> reads strings. This is sometimes useful to specify explicitly when you have a column that is a numeric identifier, i.e. long series of digits that identifies some object, but it doesnt make sense to (e.g.) divide it in half.</li>
<li>
2023-01-13 07:22:57 +08:00
<code><a href="https://readr.tidyverse.org/reference/parse_factor.html">col_factor()</a></code>, <code><a href="https://readr.tidyverse.org/reference/parse_datetime.html">col_date()</a></code>, and <code><a href="https://readr.tidyverse.org/reference/parse_datetime.html">col_datetime()</a></code> create factors, dates, and date-times respectively; youll learn more about those when we get to those data types in <a href="#chp-factors" data-type="xref">#chp-factors</a> and <a href="#chp-datetimes" data-type="xref">#chp-datetimes</a>.</li>
<li>
2022-11-19 00:30:32 +08:00
<code><a href="https://readr.tidyverse.org/reference/parse_number.html">col_number()</a></code> is a permissive numeric parser that will ignore non-numeric components, and is particularly useful for currencies. Youll learn more about it in <a href="#chp-numbers" data-type="xref">#chp-numbers</a>.</li>
<li>
2022-11-19 00:30:32 +08:00
<code><a href="https://readr.tidyverse.org/reference/col_skip.html">col_skip()</a></code> skips a column so its not included in the result.</li>
</ul><p>Its also possible to override the default column by switching from <code><a href="https://rdrr.io/r/base/list.html">list()</a></code> to <code><a href="https://readr.tidyverse.org/reference/cols.html">cols()</a></code>:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">csv &lt;- "
x,y,z
1,2,3"
read_csv(csv, col_types = cols(.default = col_character()))
#&gt; # A tibble: 1 × 3
#&gt; x y z
#&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt;
#&gt; 1 1 2 3</pre>
</div>
2022-11-19 00:30:32 +08:00
<p>Another useful helper is <code><a href="https://readr.tidyverse.org/reference/cols.html">cols_only()</a></code> which will read in only the columns you specify:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">read_csv(
"x,y,z
1,2,3",
col_types = cols_only(x = col_character())
)
#&gt; # A tibble: 1 × 1
#&gt; x
#&gt; &lt;chr&gt;
#&gt; 1 1</pre>
</div>
</section>
</section>
<section id="sec-readr-directory" data-type="sect1">
<h1>
Reading data from multiple files</h1>
2022-11-19 00:30:32 +08:00
<p>Sometimes your data is split across multiple files instead of being contained in a single file. For example, you might have sales data for multiple months, with each months data in a separate file: <code>01-sales.csv</code> for January, <code>02-sales.csv</code> for February, and <code>03-sales.csv</code> for March. With <code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_csv()</a></code> you can read these data in at once and stack them on top of each other in a single data frame.</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">sales_files &lt;- c("data/01-sales.csv", "data/02-sales.csv", "data/03-sales.csv")
read_csv(sales_files, id = "file")
#&gt; Rows: 19 Columns: 6
#&gt; ── Column specification ─────────────────────────────────────────────────────
#&gt; Delimiter: ","
#&gt; chr (1): month
#&gt; dbl (4): year, brand, item, n
#&gt;
#&gt; Use `spec()` to retrieve the full column specification for this data.
#&gt; Specify the column types or set `show_col_types = FALSE` to quiet this message.
#&gt; # A tibble: 19 × 6
#&gt; file month year brand item n
#&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
#&gt; 1 data/01-sales.csv January 2019 1 1234 3
#&gt; 2 data/01-sales.csv January 2019 1 8721 9
#&gt; 3 data/01-sales.csv January 2019 1 1822 2
#&gt; 4 data/01-sales.csv January 2019 2 3333 1
#&gt; 5 data/01-sales.csv January 2019 2 2156 9
#&gt; 6 data/01-sales.csv January 2019 2 3987 6
#&gt; # … with 13 more rows</pre>
</div>
<p>With the additional <code>id</code> parameter we have added a new column called <code>file</code> to the resulting data frame that identifies the file the data come from. This is especially helpful in circumstances where the files youre reading in do not have an identifying column that can help you trace the observations back to their original sources.</p>
2022-11-19 00:30:32 +08:00
<p>If you have many files you want to read in, it can get cumbersome to write out their names as a list. Instead, you can use the base <code><a href="https://rdrr.io/r/base/list.files.html">list.files()</a></code> function to find the files for you by matching a pattern in the file names. Youll learn more about these patterns in <a href="#chp-regexps" data-type="xref">#chp-regexps</a>.</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">sales_files &lt;- list.files("data", pattern = "sales\\.csv$", full.names = TRUE)
sales_files
#&gt; [1] "data/01-sales.csv" "data/02-sales.csv" "data/03-sales.csv"</pre>
</div>
</section>
<section id="sec-writing-to-a-file" data-type="sect1">
<h1>
Writing to a file</h1>
2022-11-19 00:30:32 +08:00
<p>readr also comes with two useful functions for writing data back to disk: <code><a href="https://readr.tidyverse.org/reference/write_delim.html">write_csv()</a></code> and <code><a href="https://readr.tidyverse.org/reference/write_delim.html">write_tsv()</a></code>. Both functions increase the chances of the output file being read back in correctly by using the standard UTF-8 encoding for strings and ISO8601 format for date-times.</p>
<p>The most important arguments are <code>x</code> (the data frame to save), and <code>file</code> (the location to save it). You can also specify how missing values are written with <code>na</code>, and if you want to <code>append</code> to an existing file.</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">write_csv(students, "students.csv")</pre>
</div>
<p>Now lets read that csv file back in. Note that the type information is lost when you save to csv:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">students
#&gt; # A tibble: 6 × 5
#&gt; student_id full_name favourite_food meal_plan age
#&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;fct&gt; &lt;dbl&gt;
#&gt; 1 1 Sunil Huffmann Strawberry yoghurt Lunch only 4
#&gt; 2 2 Barclay Lynn French fries Lunch only 5
#&gt; 3 3 Jayendra Lyne &lt;NA&gt; Breakfast and lunch 7
#&gt; 4 4 Leon Rossini Anchovies Lunch only NA
#&gt; 5 5 Chidiegwu Dunkel Pizza Breakfast and lunch 5
#&gt; 6 6 Güvenç Attila Ice cream Lunch only 6
write_csv(students, "students-2.csv")
read_csv("students-2.csv")
#&gt; # A tibble: 6 × 5
#&gt; student_id full_name favourite_food meal_plan age
#&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt;
#&gt; 1 1 Sunil Huffmann Strawberry yoghurt Lunch only 4
#&gt; 2 2 Barclay Lynn French fries Lunch only 5
#&gt; 3 3 Jayendra Lyne &lt;NA&gt; Breakfast and lunch 7
#&gt; 4 4 Leon Rossini Anchovies Lunch only NA
#&gt; 5 5 Chidiegwu Dunkel Pizza Breakfast and lunch 5
#&gt; 6 6 Güvenç Attila Ice cream Lunch only 6</pre>
</div>
2023-01-13 07:22:57 +08:00
<p>This makes CSVs a little unreliable for caching interim results—you need to recreate the column specification every time you load in. There are two main alternative:</p>
<ol type="1"><li>
2022-11-19 00:30:32 +08:00
<p><code><a href="https://readr.tidyverse.org/reference/read_rds.html">write_rds()</a></code> and <code><a href="https://readr.tidyverse.org/reference/read_rds.html">read_rds()</a></code> are uniform wrappers around the base functions <code><a href="https://rdrr.io/r/base/readRDS.html">readRDS()</a></code> and <code><a href="https://rdrr.io/r/base/readRDS.html">saveRDS()</a></code>. These store data in Rs custom binary format called RDS:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">write_rds(students, "students.rds")
read_rds("students.rds")
#&gt; # A tibble: 6 × 5
#&gt; student_id full_name favourite_food meal_plan age
#&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;fct&gt; &lt;dbl&gt;
#&gt; 1 1 Sunil Huffmann Strawberry yoghurt Lunch only 4
#&gt; 2 2 Barclay Lynn French fries Lunch only 5
#&gt; 3 3 Jayendra Lyne &lt;NA&gt; Breakfast and lunch 7
#&gt; 4 4 Leon Rossini Anchovies Lunch only NA
#&gt; 5 5 Chidiegwu Dunkel Pizza Breakfast and lunch 5
#&gt; 6 6 Güvenç Attila Ice cream Lunch only 6</pre>
</div>
</li>
<li>
2023-01-13 07:22:57 +08:00
<p>The arrow package allows you to read and write parquet files, a fast binary file format that can be shared across programming languages. Well return to arrow in more depth in <a href="#chp-arrow" data-type="xref">#chp-arrow</a>.</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">library(arrow)
write_parquet(students, "students.parquet")
read_parquet("students.parquet")
#&gt; # A tibble: 6 × 5
#&gt; student_id full_name favourite_food meal_plan age
#&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;fct&gt; &lt;dbl&gt;
#&gt; 1 1 Sunil Huffmann Strawberry yoghurt Lunch only 4
#&gt; 2 2 Barclay Lynn French fries Lunch only 5
#&gt; 3 3 Jayendra Lyne NA Breakfast and lunch 7
#&gt; 4 4 Leon Rossini Anchovies Lunch only NA
#&gt; 5 5 Chidiegwu Dunkel Pizza Breakfast and lunch 5
#&gt; 6 6 Güvenç Attila Ice cream Lunch only 6</pre>
</div>
</li>
2023-01-13 07:22:57 +08:00
</ol><p>Parquet tends to be much faster than RDS and is usable outside of R, but does require the arrow package.</p>
</section>
<section id="data-entry" data-type="sect1">
<h1>
Data entry</h1>
2022-11-19 00:30:32 +08:00
<p>Sometimes youll need to assemble a tibble “by hand” doing a little data entry in your R script. There are two useful functions to help you do this which differ in whether you layout the tibble by columns or by rows. <code><a href="https://tibble.tidyverse.org/reference/tibble.html">tibble()</a></code> works by column:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">tibble(
x = c(1, 2, 5),
y = c("h", "m", "g"),
z = c(0.08, 0.83, 0.60)
)
#&gt; # A tibble: 3 × 3
#&gt; x y z
#&gt; &lt;dbl&gt; &lt;chr&gt; &lt;dbl&gt;
#&gt; 1 1 h 0.08
#&gt; 2 2 m 0.83
#&gt; 3 5 g 0.6</pre>
</div>
<p>Note that every column in tibble must be same size, so youll get an error if theyre not:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">tibble(
x = c(1, 2),
y = c("h", "m", "g"),
z = c(0.08, 0.83, 0.6)
)
#&gt; Error:
#&gt; ! Tibble columns must have compatible sizes.
#&gt; • Size 2: Existing data.
#&gt; • Size 3: Column `y`.
#&gt; Only values of size one are recycled.</pre>
</div>
2022-11-19 00:30:32 +08:00
<p>Laying out the data by column can make it hard to see how the rows are related, so an alternative is <code><a href="https://tibble.tidyverse.org/reference/tribble.html">tribble()</a></code>, short for <strong>tr</strong>ansposed t<strong>ibble</strong>, which lets you lay out your data row by row. <code><a href="https://tibble.tidyverse.org/reference/tribble.html">tribble()</a></code> is customized for data entry in code: column headings start with <code>~</code> and entries are separated by commas. This makes it possible to lay out small amounts of data in an easy to read form:</p>
<div class="cell">
2022-11-19 01:26:25 +08:00
<pre data-type="programlisting" data-code-language="r">tribble(
~x, ~y, ~z,
"h", 1, 0.08,
"m", 2, 0.83,
"g", 5, 0.60,
)
#&gt; # A tibble: 3 × 3
#&gt; x y z
#&gt; &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt;
#&gt; 1 h 1 0.08
#&gt; 2 m 2 0.83
#&gt; 3 g 5 0.6</pre>
</div>
2022-11-19 00:30:32 +08:00
<p>Well use <code><a href="https://tibble.tidyverse.org/reference/tibble.html">tibble()</a></code> and <code><a href="https://tibble.tidyverse.org/reference/tribble.html">tribble()</a></code> later in the book to construct small examples to demonstrate how various functions work.</p>
</section>
<section id="summary" data-type="sect1">
<h1>
Summary</h1>
2023-01-13 07:22:57 +08:00
<p>In this chapter, youve learned how to load CSV files with <code><a href="https://readr.tidyverse.org/reference/read_delim.html">read_csv()</a></code> and to do your own data entry with <code><a href="https://tibble.tidyverse.org/reference/tibble.html">tibble()</a></code> and <code><a href="https://tibble.tidyverse.org/reference/tribble.html">tribble()</a></code>. Youve learned how csv files work, some of the problems you might encounter, and how to overcome them. Well come to data import a few times in this book: <a href="#chp-spreadsheets" data-type="xref">#chp-spreadsheets</a> from Excel and googlesheets, <a href="#chp-databases" data-type="xref">#chp-databases</a> will show you how to load data from databases, <a href="#chp-arrow" data-type="xref">#chp-arrow</a> from parquet files, <a href="#chp-rectangling" data-type="xref">#chp-rectangling</a> from JSON, and <a href="#chp-webscraping" data-type="xref">#chp-webscraping</a> from websites.</p>
<p>Now that youre writing a substantial amount of R code, its time to learn more about organizing your code into files and directories. In the next chapter, youll learn all about the advantages of scripts and projects, and some of the many tools that they provide to make your life easier.</p>
</section>
</section>