Data science is an exciting discipline that allows you to transform raw data into understanding, insight, and knowledge. The goal of “R for Data Science” is to help you learn the most important tools in R that will allow you to do data science efficiently and reproducibly. After reading this book, you’ll have the tools to tackle a wide variety of data science challenges, using the best parts of R.
What you will learn
Data science is a huge field, and there’s no way you can master it all by reading a single book. The goal of this book is to give you a solid foundation in the most important tools, and enough knowledge to find the resources to learn more when necessary. Our model of the tools needed in a typical data science project looks something like #fig-ds-diagram.
In our model of the data science process you start with data import and tidying. Next you understand your data with an iterative cycle of transforming, visualizing, and modeling. You finish the process by communicating your results to other humans.
First you must import your data into R. This typically means that you take data stored in a file, database, or web application programming interface (API), and load it into a data frame in R. If you can’t get your data into R, you can’t do data science on it!
Once you’ve imported your data, it is a good idea to tidy it. Tidying your data means storing it in a consistent form that matches the semantics of the dataset with the way it is stored. In brief, when your data is tidy, each column is a variable, and each row is an observation. Tidy data is important because the consistent structure lets you focus your efforts on answering questions about the data, not fighting to get the data into the right form for different functions.
Once you have tidy data, a common next step is to transform it. Transformation includes narrowing in on observations of interest (like all people in one city, or all data from the last year), creating new variables that are functions of existing variables (like computing speed from distance and time), and calculating a set of summary statistics (like counts or means). Together, tidying and transforming are called wrangling, because getting your data in a form that’s natural to work with often feels like a fight!
Once you have tidy data with the variables you need, there are two main engines of knowledge generation: visualisation and modelling. These have complementary strengths and weaknesses so any real analysis will iterate between them many times.
Visualisation is a fundamentally human activity. A good visualisation will show you things that you did not expect, or raise new questions about the data. A good visualisation might also hint that you’re asking the wrong question, or that you need to collect different data. Visualisations can surprise you and they don’t scale particularly well because they require a human to interpret them.
The last step of data science is communication, an absolutely critical part of any data analysis project. It doesn’t matter how well your models and visualisation have led you to understand the data unless you can also communicate your results to others.
Surrounding all these tools is programming. Programming is a cross-cutting tool that you use in nearly every part of a data science project. You don’t need to be an expert programmer to be a successful data scientist, but learning more about programming pays off, because becoming a better programmer allows you to automate common tasks, and solve new problems with greater ease.
You’ll use these tools in every data science project, but for most projects they’re not enough. There’s a rough 80-20 rule at play; you can tackle about 80% of every project using the tools that you’ll learn in this book, but you’ll need other tools to tackle the remaining 20%. Throughout this book, we’ll point you to resources where you can learn more.
How this book is organised
The previous description of the tools of data science is organised roughly according to the order in which you use them in an analysis (although of course you’ll iterate through them multiple times). In our experience, however, learning data ingest and tidying first is sub-optimal, because 80% of the time it’s routine and boring, and the other 20% of the time it’s weird and frustrating. That’s a bad place to start learning a new subject! Instead, we’ll start with visualisation and transformation of data that’s already been imported and tidied. That way, when you ingest and tidy your own data, your motivation will stay high because you know the pain is worth the effort.
Within each chapter, we try and adhere to a similar pattern: start with some motivating examples so you can see the bigger picture, and then dive into the details. Each section of the book is paired with exercises to help you practice what you’ve learned. Although it can be tempting to skip the exercises, there’s no better way to learn than practicing on real problems.
What you won’t learn
There are a number of important topics that this book doesn’t cover. We believe it’s important to stay ruthlessly focused on the essentials so you can get up and running as quickly as possible. That means this book can’t cover every important topic.
Modeling
To learn more about modeling, we highly recommend Tidy Modeling with R, by our colleagues Max Kuhn and Julia Silge. This book will teach you the tidymodels family of packages, which, as you might guess from the name, share many conventions with the tidyverse packages we use in this book.
Big data
This book proudly focuses on small, in-memory datasets. This is the right place to start because you can’t tackle big data unless you have experience with small data. The tools you learn in this book will easily handle hundreds of megabytes of data, and with a little care, you can typically use them to work with 1-2 Gb of data. If you’re routinely working with larger data (10-100 Gb, say), you should learn more about data.table. This book doesn’t teach data.table because it has a very concise interface that offers fewer linguistic cues, which makes it harder to learn. However, if you’re working with large data, the performance payoff is well worth the effort required to learn it.
If your data is bigger than this, carefully consider whether your big data problem is actually a small data problem in disguise. While the complete data set might be big, often the data needed to answer a specific question is small. You might be able to find a subset, subsample, or summary that fits in memory and still allows you to answer the question that you’re interested in. The challenge here is finding the right small data, which often requires a lot of iteration.
Another possibility is that your big data problem is actually a large number of small data problems in disguise. Each individual problem might fit in memory, but you have millions of them. For example, you might want to fit a model to each person in your dataset. This would be trivial if you had just 10 or 100 people, but instead you have a million. Fortunately, each problem is independent of the others (a setup that is sometimes called embarrassingly parallel), so you just need a system (like Hadoop or Spark) that allows you to send different datasets to different computers for processing. Once you’ve figured out how to answer your question for a single subset using the tools described in this book, you can learn new tools like sparklyr to solve it for the full dataset.
Python, Julia, and friends
In this book, you won’t learn anything about Python, Julia, or any other programming language useful for data science. This isn’t because we think these tools are bad. They’re not! And in practice, most data science teams use a mix of languages, often at least R and Python.
However, we strongly believe that it’s best to master one tool at a time. You will get better faster if you dive deep, rather than spreading yourself thinly over many topics. This doesn’t mean you should only know one thing, just that you’ll generally learn faster if you stick to one thing at a time. You should strive to learn new things throughout your career, but make sure your understanding is solid before you move on to the next interesting thing.
We think R is a great place to start your data science journey because it is an environment designed from the ground up to support data science. R is not just a programming language, it is also an interactive environment for doing data science. To support interaction, R is a much more flexible language than many of its peers. This flexibility comes with its downsides, but the big upside is how easy it is to evolve tailored grammars for specific parts of the data science process. These mini languages help you think about problems as a data scientist, while supporting fluent interaction between your brain and the computer.
Prerequisites
We’ve made a few assumptions about what you already know in order to get the most out of this book. You should be generally numerically literate, and it’s helpful if you have some programming experience already. If you’ve never programmed before, you might find Hands on Programming with R by Garrett to be a useful adjunct to this book.
There are four things you need to run the code in this book: R, RStudio, a collection of R packages called the tidyverse, and a handful of other packages. Packages are the fundamental units of reproducible R code. They include reusable functions, the documentation that describes how to use them, and sample data.
R
To download R, go to CRAN, the comprehensive Rarchive network. CRAN is composed of a set of mirror servers distributed around the world and is used to distribute R and R packages. Don’t try and pick a mirror that’s close to you: instead use the cloud mirror, https://cloud.r-project.org, which automatically figures it out for you.
A new major version of R comes out once a year, and there are 2-3 minor releases each year. It’s a good idea to update regularly. Upgrading can be a bit of a hassle, especially for major versions, which require you to re-install all your packages, but putting it off only makes it worse. You’ll need at least R 4.1.0 for this book.
RStudio
RStudio is an integrated development environment, or IDE, for R programming. Download and install it from https://www.rstudio.com/download. RStudio is updated a couple of times a year. When a new version is available, RStudio will let you know. It’s a good idea to upgrade regularly so you can take advantage of the latest and greatest features. For this book, make sure you have at least RStudio 2022.02.0.
When you start RStudio, #fig-rstudio-console, you’ll see two key regions in the interface: the console pane, and the output pane. For now, all you need to know is that you type R code in the console pane, and press enter to run it. You’ll learn more as we go along!
The RStudio IDE has two key regions: type R code in the console pane on the left, and look for plots in the output pane on the right.
The tidyverse
You’ll also need to install some R packages. An R package is a collection of functions, data, and documentation that extends the capabilities of base R. Using packages is key to the successful use of R. The majority of the packages that you will learn in this book are part of the so-called tidyverse. All packages in the tidyverse share a common philosophy of data and R programming, and are designed to work together naturally.
You can install the complete tidyverse with a single line of code:
install.packages("tidyverse")
On your own computer, type that line of code in the console, and then press enter to run it. R will download the packages from CRAN and install them on to your computer. If you have problems installing, make sure that you are connected to the internet, and that https://cloud.r-project.org/ isn’t blocked by your firewall or proxy.
You will not be able to use the functions, objects, or help files in a package until you load it with library(). Once you have installed a package, you can load it using the library() function:
This tells you that tidyverse is loading eight packages: ggplot2, tibble, tidyr, readr, purrr, dplyr, stringr, and forcats packages. These are considered to be the core of the tidyverse because you’ll use them in almost every analysis.
Packages in the tidyverse change fairly frequently. You can check whether updates are available, and optionally install them, by running tidyverse_update().
Other packages
There are many other excellent packages that are not part of the tidyverse, because they solve problems in a different domain, or are designed with a different set of underlying principles. This doesn’t make them better or worse, just different. In other words, the complement to the tidyverse is not the messyverse, but many other universes of interrelated packages. As you tackle more data science projects with R, you’ll learn new packages and new ways of thinking about data.
In this book we’ll use three data packages from outside the tidyverse:
These packages provide data on airline flights, world development, and baseball that we’ll use to illustrate key data science ideas.
Running R code
The previous section showed you several examples of running R code. Code in the book looks like this:
1 + 2
#> [1] 3
If you run the same code in your local console, it will look like this:
> 1 + 2
[1] 3
There are two main differences. In your console, you type after the >, called the prompt; we don’t show the prompt in the book. In the book, output is commented out with #>; in your console it appears directly after your code. These two differences mean that if you’re working with an electronic version of the book, you can easily copy code out of the book and into the console.
Throughout the book, we use a consistent set of conventions to refer to code:
Functions are displayed in a code font and followed by parentheses, like sum(), or mean().
Other R objects (such as data or function arguments) are in a code font, without parentheses, like flights or x.
Sometimes, to make it clear which package an object comes from, we’ll use we’ll use the package name followed by two colons, like dplyr::mutate(), or nycflights13::flights. This is also valid R code.
Acknowledgements
This book isn’t just the product of Hadley, Mine, and Garrett, but is the result of many conversations (in person and online) that we’ve had with many people in the R community. There are a few people we’d like to thank in particular, because they have spent many hours answering our questions and helping us to better think about data science:
Jenny Bryan and Lionel Henry for many helpful discussions around working with lists and list-columns.
Yihui Xie for his work on the bookdown package, and for tirelessly responding to my feature requests.
Bill Behrman for his thoughtful reading of the entire book, and for trying it out with his data science class at Stanford.
The #rstats Twitter community who reviewed all of the draft chapters and provided tons of useful feedback.
This book was written in the open, and many people contributed pull requests to fix minor problems. Special thanks goes to everyone who contributed via GitHub:
A big thank you to all 212 people who contributed specific improvements via GitHub pull requests (in alphabetical order by username): Alex (@ALShum), A. s. (@Adrianzo), @AlanFeder, Antti Rask (@AnttiRask), Oluwafemi OYEDELE (@BB1464), Brian G. Barkley (@BarkleyBG), Bianca Peterson (@BinxiePeterson), Birger Niklas (@BirgerNi), David Clark (@DDClark), @DSGeoff, Edwin Thoen (@EdwinTh), Eric Kitaif (@EricKit), Gerome Meyer (@GeroVanMi), Josh Goldberg (@GoldbergData), Iain (@Iain-S), Jeffrey Stevens (@JeffreyRStevens), 蒋雨蒙 (@JeldorPKU), @MJMarshall, Kara de la Marck (@MarckK), Matt Wittbrodt (@MattWittbrodt), Jakub Nowosad (@Nowosad), Y. Yu (@PursuitOfDataScience), Jajo (@RIngyao), Richard Knight (@RJHKnight), Ranae Dietzel (@Ranae), @ReeceGoding, Robin (@Robinlovelace), Rod Mazloomi (@RodAli), Rohan Alexander (@RohanAlexander), Romero Morais (@RomeroBarata), Shannon Ellis (@ShanEllis), Christian Heinrich (@Shurakai), Steven M. Mortimer (@StevenMMortimer), @a-rosenberg, Tim Becker (@a2800276), Adam Gruer (@adam-gruer), adi pradhan (@adidoit), Andrea Gilardi (@agila5), Ajay Deonarine (@ajay-d), @aleloi, pete (@alonzi), Andrew M. (@amacfarland), Andrew Landgraf (@andland), Angela Li (@angela-li), @ariespirgel, @august-18, Michael Henry (@aviast), Azza Ahmed (@azzaea), Steven Moran (@bambooforest), Mara Averick (@batpigandme), Brent Brewington (@bbrewington), Bill Behrman (@behrman), Ben Herbertson (@benherbertson), Ben Marwick (@benmarwick), Ben Steinberg (@bensteinberg), Benjamin Yeh (@bentyeh), Betul Turkoglu (@betulturkoglu), Brandon Greenwell (@bgreenwell), Brett Klamer (@bklamer), @boardtc, Christian (@c-hoh), Camille V Leonard (@camillevleonard), Christian Mongeau (@chrMongeau), Cooper Morris (@coopermor), Colin Gillespie (@csgillespie), Rademeyer Vermaak (@csrvermaak), Chris Saunders (@ctsa), Abhinav Singh (@curious-abhinav), Curtis Alexander (@curtisalexander), Christian G. Warden (@cwarden), Charlotte Wickham (@cwickham), Kenny Darrell (@darrkj), David Rubinger (@davidrubinger), Derwin McGeary (@derwinmcgeary), Daniel Gromer (@dgromer), @djbirke, Zhuoer Dong (@dongzhuoer), Devin Pastoor (@dpastoor), Julian During (@duju211), Dylan Cashman (@dylancashman), Dirk Eddelbuettel (@eddelbuettel), Ahmed El-Gabbas (@elgabbas), Henry Webel (@enryH), Eric Watt (@ericwatt), Erik Erhardt (@erikerhardt), Etienne B. Racine (@etiennebr), Everett Robinson (@evjrob), @fellennert, Flemming Miguel (@flemmingmiguel), Floris Vanderhaeghe (@florisvdh), @funkybluehen, @gabrivera, Garrick Aden-Buie (@gadenbuie), bahadir cankardes (@gridgrad), Gustav W Delius (@gustavdelius), Hao Chen (@hao-trivago), Harris McGehee (@harrismcgehee), @hendrikweisser, Hengni Cai (@hengnicai), Ian Sealy (@iansealy), Ian Lyttle (@ijlyttle), Ivan Krukov (@ivan-krukov), Jacob Kaplan (@jacobkap), Jazz Weisman (@jazzlw), John Blischak (@jdblischak), John D. Storey (@jdstorey), Jeff Boichuk (@jeffboichuk), Gregory Jefferis (@jefferis), Jennifer (Jenny) Bryan (@jennybc), Jen Ren (@jenren), Jeroen Janssens (@jeroenjanssens), Janet Wesner (@jilmun), Jim Hester (@jimhester), JJ Chen (@jjchern), Jacek Kolacz (@jkolacz), Joanne Jang (@joannejang), John Sears (@johnsears), @jonathanflint, Jon Calder (@jonmcalder), Jonathan Page (@jonpage), JooYoung Seo (@jooyoungseo), Justinas Petuchovas (@jpetuchovas), Jordan (@jrdnbradford), Jeffrey Arnold (@jrnold), Jose Roberto Ayala Solares (@jroberayalas), @juandering, Julia Stewart Lowndes (@jules32), Sonja (@kaetschap), Kara Woo (@karawoo), Katrin Leinweber (@katrinleinweber), Karandeep Singh (@kdpsingh), Kevin Perese (@kevinxperese), Kevin Ferris (@kferris10), Kirill Sevastyanenko (@kirillseva), @koalabearski, Kirill Müller (@krlmlr), Rafał Kucharski (@kucharsky), Noah Landesberg (@landesbergn), Lawrence Wu (@lawwu), @lindbrook, Luke W Johnston (@lwjohnst86), Kunal Marwaha (@marwahaha), Matan Hakim (@matanhakim), Mauro Lepore (@maurolepore), Mark Beveridge (@mbeveridge), @mcewenkhundi, Matt Herman (@mfherman), Michael Boerman (@michaelboerman), Mitsuo Shiota (@mitsuoxv), Matthew Hendrickson (@mjhendrickson), Mohammed Hamdy (@mmhamdy), Maxim Nazarov (@mnazarov), Maria Paula Caldas (@mpaulacaldas), Mustafa Ascha (@mustafaascha), Nelson Areal (@nareal), Nate Olson (@nate-d-olson), Nathanael (@nateaff), @nattalides, Nick Clark (@nickclark1000), @nickelas, Nirmal Patel (@nirmalpatel), Nischal Shrestha (@nischalshrestha), Nicholas Tierney (@njtierney), @olivier6088, Pablo E. Garcia (@pabloedug), Paul Adamson (@padamson), Peter Hurford (@peterhurford), Patrick Kennedy (@pkq), Pooya Taherkhani (@pooyataher), Radu Grosu (@radugrosu), Rayna M Harris (@raynamharris), Robin Gertenbach (@rgertenbach), Riva Quiroga (@rivaquiroga), Richard Zijdeman (@rlzijdeman), @robertchu03, Emily Robinson (@robinsones), Rob Tenorio (@robtenorio), Albert Y. Kim (@rudeboybert), Saghir (@saghirb), Hojjat Salmasian (@salmasian), Jonas (@sauercrowd), Vebash Naidoo (@sciencificity), Seamus McKinsey (@seamus-mckinsey), @seanpwilliams, Luke Smith (@seasmith), Matthew Sedaghatfar (@sedaghatfar), Sebastian Kraus (@sekR4), Sam Firke (@sfirke), @shoili, S’busiso Mkhondwane (@sibusiso16), Jakob Krigovsky (@sonicdoe), Stéphane Guillou (@stragu), Sergiusz Bleja (@svenski), Tal Galili (@talgalili), Tim Broderick (@timbroderick), Tim Waterhouse (@timwaterhouse), TJ Mahr (@tjmahr), Thomas Klebel (@tklebel), Tom Prior (@tomjamesprior), Terence Teo (@tteo), @twgardner2, Ulrik Lyngs (@ulyngs), Martin Van der Linden (@vanderlindenma), Walter Somerville (@waltersom), Will Beasley (@wibeasley), Yihui Xie (@yihui), Yiming (Paul) Li (@yimingli), Hiroaki Yutani (@yutannihilation), Yu Yu Aung (@yuyu-aung), Zach Bogart (@zachbogart), @zeal626, Zeki Akyol (@zekiakyol).
Colophon
An online version of this book is available at https://r4ds.hadley.nz. It will continue to evolve in between reprints of the physical book. The source of the book is available at https://github.com/hadley/r4ds. The book is powered by Quarto which makes it easy to write books that combine text and executable code.
