r4ds/datetimes.qmd

788 lines
28 KiB
Plaintext
Raw Normal View History

# Dates and times {#sec-dates-and-times}
```{r}
#| results: "asis"
#| echo: false
source("_common.R")
2023-01-23 21:55:51 +08:00
status("complete")
2022-08-12 06:32:49 +08:00
# https://github.com/tidyverse/lubridate/issues/1058
options(warnPartialMatchArgs = FALSE)
```
2015-12-17 01:21:42 +08:00
2016-07-19 21:01:50 +08:00
## Introduction
This chapter will show you how to work with dates and times in R.
At first glance, dates and times seem simple.
You use them all the time in your regular life, and they don't seem to cause much confusion.
2022-11-08 07:06:29 +08:00
However, the more you learn about dates and times, the more complicated they seem to get!
2016-07-28 03:51:24 +08:00
2022-11-08 07:06:29 +08:00
To warm up think about how many days there are in a year, and how many hours there are in a day.
2022-08-12 06:32:49 +08:00
You probably remembered that most years have 365 days, but leap years have 366.
Do you know the full rule for determining if a year is a leap year[^datetimes-1]?
2022-11-08 07:06:29 +08:00
The number of hours in a day is a little less obvious: most days have 24 hours, but in places that use daylight saving time (DST), one day each year has 23 hours and another has 25.
2016-07-28 03:51:24 +08:00
2022-08-12 06:32:49 +08:00
[^datetimes-1]: A year is a leap year if it's divisible by 4, unless it's also divisible by 100, except if it's also divisible by 400.
In other words, in every set of 400 years, there's 97 leap years.
2016-07-28 03:51:24 +08:00
Dates and times are hard because they have to reconcile two physical phenomena (the rotation of the Earth and its orbit around the sun) with a whole raft of geopolitical phenomena including months, time zones, and DST.
This chapter won't teach you every last detail about dates and times, but it will give you a solid grounding of practical skills that will help you with common data analysis challenges.
2016-04-21 05:11:04 +08:00
2022-11-08 07:06:29 +08:00
We'll begin by showing you how to create date-times from various inputs, and then once you've got a date-time, how you can extract components like year, month, and day.
We'll then dive into the tricky topic of working with time spans, which come in a variety of flavors depending on what you're trying to do.
We'll conclude with a brief discussion of the additional challenges posed by time zones.
2016-07-19 21:01:50 +08:00
### Prerequisites
This chapter will focus on the **lubridate** package, which makes it easier to work with dates and times in R.
As of the latest tidyverse release, lubridate is part of core tidyverse so.
We will also need nycflights13 for practice data.
2016-07-19 21:01:50 +08:00
```{r}
#| message: false
2016-10-04 01:30:24 +08:00
library(tidyverse)
2016-04-21 05:11:04 +08:00
library(nycflights13)
```
## Creating date/times {#sec-creating-datetimes}
2016-04-21 05:11:04 +08:00
2016-08-05 21:53:01 +08:00
There are three types of date/time data that refer to an instant in time:
2016-07-28 03:51:24 +08:00
- A **date**.
Tibbles print this as `<date>`.
2016-07-28 03:51:24 +08:00
- A **time** within a day.
Tibbles print this as `<time>`.
2016-07-28 03:51:24 +08:00
- A **date-time** is a date plus a time: it uniquely identifies an instant in time (typically to the nearest second).
Tibbles print this as `<dttm>`.
2022-08-12 06:32:49 +08:00
Base R calls these POSIXct, but doesn't exactly trip off the tongue.
2016-07-28 03:51:24 +08:00
2022-08-12 06:32:49 +08:00
In this chapter we are going to focus on dates and date-times as R doesn't have a native class for storing times.
If you need one, you can use the **hms** package.
You should always use the simplest possible data type that works for your needs.
That means if you can use a date instead of a date-time, you should.
Date-times are substantially more complicated because of the need to handle time zones, which we'll come back to at the end of the chapter.
2016-07-28 03:51:24 +08:00
2016-07-29 04:39:52 +08:00
To get the current date or date-time you can use `today()` or `now()`:
2016-04-21 05:11:04 +08:00
```{r}
2016-07-28 03:51:24 +08:00
today()
now()
2016-04-21 05:11:04 +08:00
```
2016-07-28 03:51:24 +08:00
Otherwise, the following sections describe the four ways you're likely to create a date/time:
2016-07-29 04:39:52 +08:00
- While reading a file with readr.
- From a string.
- From individual date-time components.
- From an existing date/time object.
2016-07-29 04:39:52 +08:00
### During import
If your CSV contains an ISO8601 date or date-time, you don't need to do anything; readr will automatically recognize it:
```{r}
#| message: false
csv <- "
date,datetime
2022-01-02,2022-01-02 05:12
"
read_csv(csv)
```
2023-02-28 21:35:57 +08:00
If you haven't heard of **ISO8601** before, it's an international standard[^datetimes-2] for writing dates where the components of a date are organized from biggest to smallest separated by `-`. For example, in ISO8601 March 5 2022 is `2022-05-03`. ISO8601 dates can also include times, where hour, minute, and second are separated by `:`, and the date and time components are separated by either a `T` or a space.
For example, you could write 4:26pm on March 5 2022 as either `2022-05-03 16:26` or `2022-05-03T16:26`.
[^datetimes-2]: <https://xkcd.com/1179/>
For other date-time formats, you'll need to use `col_types` plus `col_date()` or `col_datetime()` along with a date-time format.
The date-time format used by readr is a standard used across many programming languages, describing a date component with a `%` followed by a single character.
For example, `%Y-%m-%d` specifies a date that's a year, `-`, month (as number) `-`, day.
Table @tbl-date-formats lists all the options.
| Type | Code | Meaning | Example |
|-------|-------|--------------------------------|-----------------|
| Year | `%Y` | 4 digit year | 2021 |
| | `%y` | 2 digit year | 21 |
| Month | `%m` | Number | 2 |
| | `%b` | Abbreviated name | Feb |
| | `%B` | Full name | Februrary |
| Day | `%d` | Two digits | 02 |
| | `%e` | One or two digits | 2 |
| Time | `%H` | 24-hour hour | 13 |
| | `%I` | 12-hour hour | 1 |
| | `%p` | AM/PM | pm |
| | `%M` | Minutes | 35 |
| | `%S` | Seconds | 45 |
| | `%OS` | Seconds with decimal component | 45.35 |
| | `%Z` | Time zone name | America/Chicago |
| | `%z` | Offset from UTC | +0800 |
| Other | `%.` | Skip one non-digit | : |
| | `%*` | Skip any number of non-digits | |
: All date formats understood by readr {#tbl-date-formats}
2023-02-26 22:59:15 +08:00
And this code shows a few options applied to a very ambiguous date:
```{r}
#| messages: false
csv <- "
date
01/02/15
"
read_csv(csv, col_types = cols(date = col_date("%m/%d/%y")))
read_csv(csv, col_types = cols(date = col_date("%d/%m/%y")))
read_csv(csv, col_types = cols(date = col_date("%y/%m/%d")))
```
Note that no matter how you specify the date format, it's always displayed the same way once you get it into R.
If you're using `%b` or `%B` and working with non-English dates, you'll also need to provide a `locale()`.
See the list of built-in languages in `date_names_langs()`, or create your own with `date_names()`,
2016-08-13 01:25:51 +08:00
2016-07-28 03:51:24 +08:00
### From strings
The date-time specification language is powerful, but requires careful analysis of the date format.
An alternative approach is to use lubridate's helpers which attempt to automatically determine the format once you specify the order of the component.
To use them, identify the order in which year, month, and day appear in your dates, then arrange "y", "m", and "d" in the same order.
That gives you the name of the lubridate function that will parse your date.
For example:
2016-04-21 05:11:04 +08:00
```{r}
2016-07-29 04:39:52 +08:00
ymd("2017-01-31")
2016-04-21 05:11:04 +08:00
mdy("January 31st, 2017")
2016-07-29 04:39:52 +08:00
dmy("31-Jan-2017")
```
`ymd()` and friends create dates.
To create a date-time, add an underscore and one or more of "h", "m", and "s" to the name of the parsing function:
2016-04-21 05:11:04 +08:00
```{r}
ymd_hms("2017-01-31 20:11:59")
mdy_hm("01/31/2017 08:01")
```
2016-07-29 22:18:05 +08:00
You can also force the creation of a date-time from a date by supplying a timezone:
```{r}
2022-08-12 06:32:49 +08:00
ymd("2017-01-31", tz = "UTC")
2016-07-29 22:18:05 +08:00
```
2016-07-28 03:51:24 +08:00
### From individual components
2016-04-21 05:11:04 +08:00
Instead of a single string, sometimes you'll have the individual components of the date-time spread across multiple columns.
This is what we have in the `flights` data:
2016-04-21 05:11:04 +08:00
```{r}
2022-02-24 03:15:52 +08:00
flights |>
2016-07-28 03:51:24 +08:00
select(year, month, day, hour, minute)
2016-04-21 05:11:04 +08:00
```
2016-08-13 01:25:51 +08:00
To create a date/time from this sort of input, use `make_date()` for dates, or `make_datetime()` for date-times:
2016-04-21 05:11:04 +08:00
```{r}
2022-02-24 03:15:52 +08:00
flights |>
select(year, month, day, hour, minute) |>
2016-07-28 03:51:24 +08:00
mutate(departure = make_datetime(year, month, day, hour, minute))
2016-04-21 05:11:04 +08:00
```
Let's do the same thing for each of the four time columns in `flights`.
The times are represented in a slightly odd format, so we use modulus arithmetic to pull out the hour and minute components.
2022-08-10 00:43:12 +08:00
Once we've created the date-time variables, we focus in on the variables we'll explore in the rest of the chapter.
2016-04-21 05:11:04 +08:00
```{r}
2016-07-28 03:51:24 +08:00
make_datetime_100 <- function(year, month, day, time) {
make_datetime(year, month, day, time %/% 100, time %% 100)
}
2016-04-21 05:11:04 +08:00
2022-02-24 03:15:52 +08:00
flights_dt <- flights |>
filter(!is.na(dep_time), !is.na(arr_time)) |>
2016-07-28 03:51:24 +08:00
mutate(
dep_time = make_datetime_100(year, month, day, dep_time),
arr_time = make_datetime_100(year, month, day, arr_time),
sched_dep_time = make_datetime_100(year, month, day, sched_dep_time),
sched_arr_time = make_datetime_100(year, month, day, sched_arr_time)
2022-02-24 03:15:52 +08:00
) |>
2016-07-28 03:51:24 +08:00
select(origin, dest, ends_with("delay"), ends_with("time"))
2016-04-21 05:11:04 +08:00
2016-07-28 03:51:24 +08:00
flights_dt
2016-04-21 05:11:04 +08:00
```
2022-08-12 06:32:49 +08:00
With this data, we can visualize the distribution of departure times across the year:
2016-07-29 04:39:52 +08:00
```{r}
2022-08-12 06:32:49 +08:00
#| fig.alt: >
#| A frequency polyon with departure time (Jan-Dec 2013) on the x-axis
#| and number of flights on the y-axis (0-1000). The frequency polygon
#| is binned by day so you see a time series of flights by day. The
#| pattern is dominated by a weekly pattern; there are fewer flights
#| on weekends. The are few days that stand out as having a surprisingly
#| few flights in early Februrary, early July, late November, and late
#| December.
2022-02-24 03:15:52 +08:00
flights_dt |>
2022-12-05 16:12:12 +08:00
ggplot(aes(x = dep_time)) +
2016-07-29 04:39:52 +08:00
geom_freqpoly(binwidth = 86400) # 86400 seconds = 1 day
```
Or within a single day:
```{r}
2022-08-12 06:32:49 +08:00
#| fig.alt: >
#| A frequency polygon with departure time (6am - midnight Jan 1) on the
#| x-axis, number of flights on the y-axis (0-17), binned into 10 minute
#| increments. It's hard to see much pattern because of high variability,
#| but most bins have 8-12 flights, and there are markedly fewer flights
#| before 6am and after 8pm.
2022-02-24 03:15:52 +08:00
flights_dt |>
filter(dep_time < ymd(20130102)) |>
2022-12-05 16:12:12 +08:00
ggplot(aes(x = dep_time)) +
2016-07-29 04:39:52 +08:00
geom_freqpoly(binwidth = 600) # 600 s = 10 minutes
```
Note that when you use date-times in a numeric context (like in a histogram), 1 means 1 second, so a binwidth of 86400 means one day.
For dates, 1 means 1 day.
2016-07-29 04:39:52 +08:00
2016-07-28 03:51:24 +08:00
### From other types
2016-04-21 05:11:04 +08:00
You may want to switch between a date-time and a date.
That's the job of `as_datetime()` and `as_date()`:
2016-07-29 04:39:52 +08:00
```{r}
as_datetime(today())
2016-07-29 04:39:52 +08:00
as_date(now())
```
2016-04-21 05:11:04 +08:00
Sometimes you'll get date/times as numeric offsets from the "Unix Epoch", 1970-01-01.
If the offset is in seconds, use `as_datetime()`; if it's in days, use `as_date()`.
2016-08-15 22:53:14 +08:00
```{r}
as_datetime(60 * 60 * 10)
as_date(365 * 10 + 2)
2016-08-15 22:53:14 +08:00
```
2016-07-28 03:51:24 +08:00
### Exercises
2016-04-21 05:11:04 +08:00
2016-07-28 03:51:24 +08:00
1. What happens if you parse a string that contains invalid dates?
2016-04-21 05:11:04 +08:00
```{r}
#| eval: false
2016-07-28 03:51:24 +08:00
ymd(c("2010-10-10", "bananas"))
```
2016-04-21 05:11:04 +08:00
2. What does the `tzone` argument to `today()` do?
Why is it important?
2016-04-21 05:11:04 +08:00
3. For each of the following date-times show how you'd parse it using a readr column-specification and a lubridate function.
2016-04-21 05:11:04 +08:00
2016-07-28 03:51:24 +08:00
```{r}
d1 <- "January 1, 2010"
d2 <- "2015-Mar-07"
d3 <- "06-Jun-2017"
d4 <- c("August 19 (2015)", "July 1 (2015)")
d5 <- "12/30/14" # Dec 30, 2014
t1 <- "1705"
t2 <- "11:15:10.12 PM"
2016-07-28 03:51:24 +08:00
```
2016-04-21 05:11:04 +08:00
2016-07-29 04:39:52 +08:00
## Date-time components
2016-04-21 05:11:04 +08:00
Now that you know how to get date-time data into R's date-time data structures, let's explore what you can do with them.
This section will focus on the accessor functions that let you get and set individual components.
The next section will look at how arithmetic works with date-times.
2016-04-21 05:11:04 +08:00
2016-07-28 03:51:24 +08:00
### Getting components
2016-04-21 05:11:04 +08:00
You can pull out individual parts of the date with the accessor functions `year()`, `month()`, `mday()` (day of the month), `yday()` (day of the year), `wday()` (day of the week), `hour()`, `minute()`, and `second()`.
2016-04-21 05:11:04 +08:00
```{r}
2022-08-12 06:32:49 +08:00
datetime <- ymd_hms("2026-07-08 12:34:56")
2016-04-21 05:11:04 +08:00
2016-07-28 03:51:24 +08:00
year(datetime)
month(datetime)
mday(datetime)
2016-04-21 05:11:04 +08:00
2016-07-28 03:51:24 +08:00
yday(datetime)
wday(datetime)
2016-04-21 05:11:04 +08:00
```
For `month()` and `wday()` you can set `label = TRUE` to return the abbreviated name of the month or day of the week.
Set `abbr = FALSE` to return the full name.
2016-04-21 05:11:04 +08:00
```{r}
2016-07-28 03:51:24 +08:00
month(datetime, label = TRUE)
2016-07-29 22:18:05 +08:00
wday(datetime, label = TRUE, abbr = FALSE)
2016-04-21 05:11:04 +08:00
```
2016-07-29 04:39:52 +08:00
2016-08-13 01:25:51 +08:00
We can use `wday()` to see that more flights depart during the week than on the weekend:
2016-04-21 05:11:04 +08:00
```{r}
2022-08-12 06:32:49 +08:00
#| fig-alt: >
#| A bar chart with days of the week on the x-axis and number of
#| flights on the y-axis. Monday-Friday have roughly the same number of
#| flights, ~48,0000, decreasingly slightly over the course of the week.
#| Sunday is a little lower (~45,000), and Saturday is much lower
#| (~38,000).
2022-02-24 03:15:52 +08:00
flights_dt |>
mutate(wday = wday(dep_time, label = TRUE)) |>
2016-07-28 03:51:24 +08:00
ggplot(aes(x = wday)) +
2022-11-24 01:55:08 +08:00
geom_bar()
2016-04-21 05:11:04 +08:00
```
There's an interesting pattern if we look at the average departure delay by minute within the hour.
It looks like flights leaving in minutes 20-30 and 50-60 have much lower delays than the rest of the hour!
2016-04-21 05:11:04 +08:00
```{r}
2022-08-12 06:32:49 +08:00
#| fig-alt: >
#| A line chart with minute of actual departure (0-60) on the x-axis and
#| average delay (4-20) on the y-axis. Average delay starts at (0, 12),
#| steadily increases to (18, 20), then sharply drops, hitting at minimum
#| at ~23 minute past the hour and 9 minutes of delay. It then increases
#| again to (17, 35), and sharply decreases to (55, 4). It finishes off
#| with an increase to (60, 9).
2022-02-24 03:15:52 +08:00
flights_dt |>
mutate(minute = minute(dep_time)) |>
group_by(minute) |>
summarize(
2021-04-18 22:02:33 +08:00
avg_delay = mean(dep_delay, na.rm = TRUE),
2022-11-24 01:55:08 +08:00
n = n()
) |>
2022-12-05 16:12:12 +08:00
ggplot(aes(x = minute, y = avg_delay)) +
2022-11-24 01:55:08 +08:00
geom_line()
2016-04-21 05:11:04 +08:00
```
Interestingly, if we look at the *scheduled* departure time we don't see such a strong pattern:
2016-04-21 05:11:04 +08:00
2016-07-29 04:39:52 +08:00
```{r}
2022-08-12 06:32:49 +08:00
#| fig-alt: >
#| A line chart with minute of scheduled departure (0-60) on the x-axis
#| and average delay (4-16). There is relatively little pattern, just a
#| small suggestion that the average delay decreases from maybe 10 minutes
#| to 8 minutes over the course of the hour.
2022-02-24 03:15:52 +08:00
sched_dep <- flights_dt |>
mutate(minute = minute(sched_dep_time)) |>
group_by(minute) |>
summarize(
2016-07-28 03:51:24 +08:00
avg_delay = mean(arr_delay, na.rm = TRUE),
2022-11-24 01:55:08 +08:00
n = n()
)
2016-04-21 05:11:04 +08:00
2022-12-05 16:12:12 +08:00
ggplot(sched_dep, aes(x = minute, y = avg_delay)) +
2016-07-28 03:51:24 +08:00
geom_line()
2016-04-21 05:11:04 +08:00
```
So why do we see that pattern with the actual departure times?
2022-11-24 01:55:08 +08:00
Well, like much data collected by humans, there's a strong bias towards flights leaving at "nice" departure times, as @fig-human-rounding shows.
Always be alert for this sort of pattern whenever you work with data that involves human judgement!
2016-04-21 05:11:04 +08:00
```{r}
2022-11-24 01:55:08 +08:00
#| label: fig-human-rounding
#| fig-cap: >
#| A frequency polygon showing the number of flights scheduled to
#| depart each hour. You can see a strong preference for round numbers
#| like 0 and 30 and generally for numbers that are a multiple of five.
2022-08-12 06:32:49 +08:00
#| fig-alt: >
#| A line plot with departure minute (0-60) on the x-axis and number of
#| flights (0-60000) on the y-axis. Most flights are scheduled to depart
#| on either the hour (~60,000) or the half hour (~35,000). Otherwise,
#| all most all flights are scheduled to depart on multiples of five,
#| with a few extra at 15, 45, and 55 minutes.
2022-11-24 01:55:08 +08:00
#| echo: false
2022-12-05 16:12:12 +08:00
ggplot(sched_dep, aes(x = minute, y = n)) +
2016-07-28 03:51:24 +08:00
geom_line()
2016-04-21 05:11:04 +08:00
```
2016-07-28 03:51:24 +08:00
### Rounding
2016-04-21 05:11:04 +08:00
An alternative approach to plotting individual components is to round the date to a nearby unit of time, with `floor_date()`, `round_date()`, and `ceiling_date()`.
Each function takes a vector of dates to adjust and then the name of the unit round down (floor), round up (ceiling), or round to.
This, for example, allows us to plot the number of flights per week:
2016-04-21 05:11:04 +08:00
```{r}
2022-08-12 06:32:49 +08:00
#| fig-alt: >
#| A line plot with week (Jan-Dec 2013) on the x-axis and number of
#| flights (2,000-7,000) on the y-axis. The pattern is fairly flat from
#| February to November with around 7,000 flights per week. There are
#| far fewer flights on the first (approximately 4,500 flights) and last
#| weeks of the year (approximately 2,500 flights).
2022-02-24 03:15:52 +08:00
flights_dt |>
count(week = floor_date(dep_time, "week")) |>
2022-12-05 16:12:12 +08:00
ggplot(aes(x = week, y = n)) +
2022-08-12 06:32:49 +08:00
geom_line() +
geom_point()
```
You can use rounding to show the distribution of flights across the course of a day by computing the difference between `dep_time` and the earliest instant of that day:
```{r}
#| fig-alt: >
#| A line plot with depature time on the x-axis. This is units of seconds
#| since midnight so it's hard to interpret.
flights_dt |>
mutate(dep_hour = dep_time - floor_date(dep_time, "day")) |>
2022-12-05 16:12:12 +08:00
ggplot(aes(x = dep_hour)) +
2022-11-24 01:55:08 +08:00
geom_freqpoly(binwidth = 60 * 30)
2016-04-21 05:11:04 +08:00
```
Computing the difference between a pair of date-times yields a difftime (more on that in @sec-intervals).
We can convert that to an `hms` object to get a more useful x-axis:
2022-08-12 06:32:49 +08:00
```{r}
#| fig-alt: >
#| A line plot with depature time (midnight to midnight) on the x-axis
#| and number of flights on the y-axis (0 to 15,000). There are very few
#| (<100) flights before 5am. The number of flights then rises rapidly
#| to 12,000 / hour, peaking at 15,000 at 9am, before falling to around
#| 8,000 / hour for 10am to 2pm. Number of flights then increases to
#| around 12,000 per hour until 8pm, when they rapidly drop again.
flights_dt |>
mutate(dep_hour = hms::as_hms(dep_time - floor_date(dep_time, "day"))) |>
2022-12-05 16:12:12 +08:00
ggplot(aes(x = dep_hour)) +
2022-11-24 01:55:08 +08:00
geom_freqpoly(binwidth = 60 * 30)
2022-08-12 06:32:49 +08:00
```
2016-07-29 22:18:05 +08:00
2022-08-12 06:32:49 +08:00
### Modifying components
2016-07-28 03:51:24 +08:00
2022-11-24 01:55:08 +08:00
You can also use each accessor function to modify the components of a date/time.
This doesn't come up much in data analysis, but can be useful when cleaning data that has clearly incorrect dates.
2016-04-21 05:11:04 +08:00
```{r}
2022-08-12 06:32:49 +08:00
(datetime <- ymd_hms("2026-07-08 12:34:56"))
2016-07-29 22:18:05 +08:00
2022-08-12 06:32:49 +08:00
year(datetime) <- 2030
2016-07-28 03:51:24 +08:00
datetime
month(datetime) <- 01
datetime
hour(datetime) <- hour(datetime) + 1
2017-05-03 20:42:23 +08:00
datetime
2016-04-21 05:11:04 +08:00
```
2022-11-28 01:21:04 +08:00
Alternatively, rather than modifying an existing variable, you can create a new date-time with `update()`.
2022-08-12 06:32:49 +08:00
This also allows you to set multiple values in one step:
2016-04-21 05:11:04 +08:00
```{r}
2022-08-12 06:32:49 +08:00
update(datetime, year = 2030, month = 2, mday = 2, hour = 2)
2016-04-21 05:11:04 +08:00
```
2016-07-28 03:51:24 +08:00
If values are too big, they will roll-over:
2016-04-21 05:11:04 +08:00
```{r}
2022-08-12 06:32:49 +08:00
update(ymd("2023-02-01"), mday = 30)
update(ymd("2023-02-01"), hour = 400)
2016-04-21 05:11:04 +08:00
```
2016-07-28 03:51:24 +08:00
### Exercises
2016-04-21 05:11:04 +08:00
1. How does the distribution of flight times within a day change over the course of the year?
2. Compare `dep_time`, `sched_dep_time` and `dep_delay`.
Are they consistent?
Explain your findings.
3. Compare `air_time` with the duration between the departure and arrival.
2016-07-29 22:18:05 +08:00
Explain your findings.
(Hint: consider the location of the airport.)
2016-07-29 22:18:05 +08:00
4. How does the average delay time change over the course of a day?
Should you use `dep_time` or `sched_dep_time`?
Why?
2016-07-29 04:39:52 +08:00
5. On what day of the week should you leave if you want to minimise the chance of a delay?
2016-07-29 04:39:52 +08:00
6. What makes the distribution of `diamonds$carat` and `flights$sched_dep_time` similar?
2016-07-29 22:18:05 +08:00
2022-11-24 01:55:08 +08:00
7. Confirm our hypothesis that the early departures of flights in minutes 20-30 and 50-60 are caused by scheduled flights that leave early.
Hint: create a binary variable that tells you whether or not a flight was delayed.
2016-04-21 05:11:04 +08:00
2016-07-29 04:39:52 +08:00
## Time spans
2016-04-21 05:11:04 +08:00
Next you'll learn about how arithmetic with dates works, including subtraction, addition, and division.
Along the way, you'll learn about three important classes that represent time spans:
2016-04-21 05:11:04 +08:00
- **Durations**, which represent an exact number of seconds.
- **Periods**, which represent human units like weeks and months.
- **Intervals**, which represent a starting and ending point.
How do you pick between duration, periods, and intervals?
As always, pick the simplest data structure that solves your problem.
If you only care about physical time, use a duration; if you need to add human times, use a period; if you need to figure out how long a span is in human units, use an interval.
2016-04-21 05:11:04 +08:00
2016-07-29 04:39:52 +08:00
### Durations
2016-04-21 05:11:04 +08:00
2016-07-29 04:39:52 +08:00
In R, when you subtract two dates, you get a difftime object:
2016-04-21 05:11:04 +08:00
```{r}
2016-07-29 04:39:52 +08:00
# How old is Hadley?
2022-08-12 06:32:49 +08:00
h_age <- today() - ymd("1979-10-14")
2016-07-29 04:39:52 +08:00
h_age
2016-04-21 05:11:04 +08:00
```
A difftime class object records a time span of seconds, minutes, hours, days, or weeks.
This ambiguity can make difftimes a little painful to work with, so lubridate provides an alternative which always uses seconds: the **duration**.
2016-04-21 05:11:04 +08:00
2016-04-22 01:26:19 +08:00
```{r}
2016-07-29 04:39:52 +08:00
as.duration(h_age)
2016-04-22 01:26:19 +08:00
```
2016-04-21 05:11:04 +08:00
2016-08-13 01:25:51 +08:00
Durations come with a bunch of convenient constructors:
2016-04-22 01:26:19 +08:00
```{r}
2016-07-28 03:51:24 +08:00
dseconds(15)
dminutes(10)
2016-07-29 04:39:52 +08:00
dhours(c(12, 24))
ddays(0:5)
2016-07-28 03:51:24 +08:00
dweeks(3)
dyears(1)
2016-04-22 01:26:19 +08:00
```
Durations always record the time span in seconds.
Larger units are created by converting minutes, hours, days, weeks, and years to seconds: 60 seconds in a minute, 60 minutes in an hour, 24 hours in a day, and 7 days in a week.
Larger time units are more problematic.
2023-01-23 22:35:09 +08:00
A year uses the "average" number of days in a year, i.e. 365.25.
There's no way to convert a month to a duration, because there's just too much variation.
2016-07-29 22:18:05 +08:00
You can add and multiply durations:
2016-04-22 01:26:19 +08:00
```{r}
2016-07-29 04:39:52 +08:00
2 * dyears(1)
dyears(1) + dweeks(12) + dhours(15)
2016-04-22 01:26:19 +08:00
```
2016-04-21 05:11:04 +08:00
2016-07-29 04:39:52 +08:00
You can add and subtract durations to and from days:
2016-04-21 05:11:04 +08:00
2016-04-22 01:26:19 +08:00
```{r}
2016-07-29 04:39:52 +08:00
tomorrow <- today() + ddays(1)
last_year <- today() - dyears(1)
2016-04-22 01:26:19 +08:00
```
2016-04-21 05:11:04 +08:00
2016-07-29 04:39:52 +08:00
However, because durations represent an exact number of seconds, sometimes you might get an unexpected result:
2016-04-21 05:11:04 +08:00
2016-04-22 01:26:19 +08:00
```{r}
2023-01-23 22:35:09 +08:00
one_am <- ymd_hms("2026-03-08 01:00:00", tz = "America/New_York")
2016-04-22 01:26:19 +08:00
2023-01-23 22:35:09 +08:00
one_am
one_am + ddays(1)
2016-04-22 01:26:19 +08:00
```
2023-01-23 22:35:09 +08:00
Why is one day after 1am March 8, 2am March 9?
If you look carefully at the date you might also notice that the time zones have changed.
2023-01-23 22:35:09 +08:00
March 8 only has 23 hours because it's when DST starts, so if we add a full days worth of seconds we end up with a different time.
2016-07-29 04:39:52 +08:00
### Periods
2016-04-22 01:26:19 +08:00
To solve this problem, lubridate provides **periods**.
Periods are time spans but don't have a fixed length in seconds, instead they work with "human" times, like days and months.
That allows them to work in a more intuitive way:
2016-04-22 01:26:19 +08:00
```{r}
2023-01-23 22:35:09 +08:00
one_am
one_am + days(1)
2016-04-22 01:26:19 +08:00
```
Like durations, periods can be created with a number of friendly constructor functions.
2016-04-22 01:26:19 +08:00
2016-07-29 04:39:52 +08:00
```{r}
hours(c(12, 24))
days(7)
months(1:6)
```
2016-04-22 01:26:19 +08:00
2016-07-29 04:39:52 +08:00
You can add and multiply periods:
```{r}
10 * (months(6) + days(1))
days(50) + hours(25) + minutes(2)
```
2016-04-22 01:26:19 +08:00
And of course, add them to dates.
Compared to durations, periods are more likely to do what you expect:
2016-04-22 01:26:19 +08:00
```{r}
2016-07-29 04:39:52 +08:00
# A leap year
2022-08-12 06:32:49 +08:00
ymd("2024-01-01") + dyears(1)
ymd("2024-01-01") + years(1)
2016-07-29 04:39:52 +08:00
# Daylight Savings Time
2023-01-23 22:35:09 +08:00
one_am + ddays(1)
one_am + days(1)
2016-04-22 01:26:19 +08:00
```
Let's use periods to fix an oddity related to our flight dates.
Some planes appear to have arrived at their destination *before* they departed from New York City.
2016-04-22 01:26:19 +08:00
```{r}
2022-02-24 03:15:52 +08:00
flights_dt |>
2016-07-28 03:51:24 +08:00
filter(arr_time < dep_time)
2016-04-22 01:26:19 +08:00
```
These are overnight flights.
We used the same date information for both the departure and the arrival times, but these flights arrived on the following day.
We can fix this by adding `days(1)` to the arrival time of each overnight flight.
2016-04-22 01:26:19 +08:00
```{r}
2022-02-24 03:15:52 +08:00
flights_dt <- flights_dt |>
2016-07-28 03:51:24 +08:00
mutate(
overnight = arr_time < dep_time,
2022-08-12 06:32:49 +08:00
arr_time = arr_time + days(if_else(overnight, 0, 1)),
2016-07-28 03:51:24 +08:00
sched_arr_time = sched_arr_time + days(overnight * 1)
)
2016-04-22 01:26:19 +08:00
```
2016-07-28 03:51:24 +08:00
Now all of our flights obey the laws of physics.
2016-04-22 01:26:19 +08:00
```{r}
2022-02-24 03:15:52 +08:00
flights_dt |>
2016-07-28 03:51:24 +08:00
filter(overnight, arr_time < dep_time)
2016-04-22 01:26:19 +08:00
```
2022-08-12 06:32:49 +08:00
### Intervals {#sec-intervals}
2016-04-22 01:26:19 +08:00
2016-07-29 22:18:05 +08:00
It's obvious what `dyears(1) / ddays(365)` should return: one, because durations are always represented by a number of seconds, and a duration of a year is defined as 365 days worth of seconds.
2016-04-22 01:26:19 +08:00
What should `years(1) / days(1)` return?
Well, if the year was 2015 it should return 365, but if it was 2016, it should return 366!
There's not quite enough information for lubridate to give a single clear answer.
2022-08-10 04:57:18 +08:00
What it does instead is give an estimate:
2016-04-22 01:26:19 +08:00
```{r}
2016-07-28 03:51:24 +08:00
years(1) / days(1)
2016-04-22 01:26:19 +08:00
```
If you want a more accurate measurement, you'll have to use an **interval**.
2022-08-10 04:57:18 +08:00
An interval is a pair of starting and ending date times, or you can think of it as a duration with a starting point.
You can create an interval by writing `start %--% end`:
2016-04-22 01:26:19 +08:00
```{r}
2022-08-12 06:32:49 +08:00
y2023 <- ymd("2023-01-01") %--% ymd("2024-01-01")
y2024 <- ymd("2024-01-01") %--% ymd("2025-01-01")
y2023
y2024
2016-04-22 01:26:19 +08:00
```
2022-08-12 06:32:49 +08:00
You could then divide it by `days()` to find out how many days fit in the year:
2016-04-22 01:26:19 +08:00
```{r}
2022-08-12 06:32:49 +08:00
y2023 / days(1)
y2024 / days(1)
2016-04-22 01:26:19 +08:00
```
2016-07-29 04:39:52 +08:00
### Exercises
2016-04-21 05:11:04 +08:00
2021-04-18 21:43:06 +08:00
1. Explain `days(overnight * 1)` to someone who has just started learning R.
How does it work?
2016-07-29 22:18:05 +08:00
2021-04-18 21:43:06 +08:00
2. Create a vector of dates giving the first day of every month in 2015.
Create a vector of dates giving the first day of every month in the *current* year.
2016-04-21 05:11:04 +08:00
2021-04-18 21:43:06 +08:00
3. Write a function that given your birthday (as a date), returns how old you are in years.
2016-07-29 04:39:52 +08:00
2021-04-18 21:43:06 +08:00
4. Why can't `(today() %--% (today() + years(1))) / months(1)` work?
2016-04-21 05:11:04 +08:00
## Time zones
2016-04-21 05:11:04 +08:00
Time zones are an enormously complicated topic because of their interaction with geopolitical entities.
Fortunately we don't need to dig into all the details as they're not all important for data analysis, but there are a few challenges we'll need to tackle head on.
2016-04-21 05:11:04 +08:00
2022-08-12 06:32:49 +08:00
<!--# https://www.ietf.org/timezones/tzdb-2018a/theory.html -->
The first challenge is that everyday names of time zones tend to be ambiguous.
For example, if you're American you're probably familiar with EST, or Eastern Standard Time.
However, both Australia and Canada also have EST!
To avoid confusion, R uses the international standard IANA time zones.
2022-08-12 06:32:49 +08:00
These use a consistent naming scheme `{area}/{location}`, typically in the form `{continent}/{city}` or `{ocean}/{city}`.
Examples include "America/New_York", "Europe/Paris", and "Pacific/Auckland".
2016-07-29 04:39:52 +08:00
You might wonder why the time zone uses a city, when typically you think of time zones as associated with a country or region within a country.
This is because the IANA database has to record decades worth of time zone rules.
2022-08-12 06:32:49 +08:00
Over the course of decades, countries change names (or break apart) fairly frequently, but city names tend to stay the same.
Another problem is that the name needs to reflect not only the current behavior, but also the complete history.
For example, there are time zones for both "America/New_York" and "America/Detroit".
These cities both currently use Eastern Standard Time but in 1969-1972 Michigan (the state in which Detroit is located), did not follow DST, so it needs a different name.
It's worth reading the raw time zone database (available at <https://www.iana.org/time-zones>) just to read some of these stories!
2016-04-22 01:26:19 +08:00
2016-07-29 22:18:05 +08:00
You can find out what R thinks your current time zone is with `Sys.timezone()`:
2016-04-22 01:26:19 +08:00
```{r}
2016-07-28 03:51:24 +08:00
Sys.timezone()
2016-04-22 01:26:19 +08:00
```
2016-10-04 22:01:12 +08:00
(If R doesn't know, you'll get an `NA`.)
2016-07-29 22:18:05 +08:00
And see the complete list of all time zone names with `OlsonNames()`:
2016-04-22 01:26:19 +08:00
```{r}
2016-07-29 22:18:05 +08:00
length(OlsonNames())
head(OlsonNames())
2016-04-21 05:11:04 +08:00
```
In R, the time zone is an attribute of the date-time that only controls printing.
For example, these three objects represent the same instant in time:
2016-04-22 01:26:19 +08:00
```{r}
2022-08-12 06:32:49 +08:00
x1 <- ymd_hms("2024-06-01 12:00:00", tz = "America/New_York")
x1
x2 <- ymd_hms("2024-06-01 18:00:00", tz = "Europe/Copenhagen")
x2
x3 <- ymd_hms("2024-06-02 04:00:00", tz = "Pacific/Auckland")
x3
2016-04-22 01:26:19 +08:00
```
2016-08-13 01:25:51 +08:00
You can verify that they're the same time using subtraction:
2016-04-22 01:26:19 +08:00
```{r}
2016-07-29 22:18:05 +08:00
x1 - x2
x1 - x3
2016-04-22 01:26:19 +08:00
```
Unless otherwise specified, lubridate always uses UTC.
2022-08-12 06:32:49 +08:00
UTC (Coordinated Universal Time) is the standard time zone used by the scientific community and is roughly equivalent to GMT (Greenwich Mean Time).
It does not have DST, which makes a convenient representation for computation.
Operations that combine date-times, like `c()`, will often drop the time zone.
In that case, the date-times will display in your local time zone:
2016-04-22 01:26:19 +08:00
```{r}
2016-07-28 03:51:24 +08:00
x4 <- c(x1, x2, x3)
x4
2016-04-22 01:26:19 +08:00
```
2016-07-29 22:18:05 +08:00
You can change the time zone in two ways:
2016-04-22 01:26:19 +08:00
- Keep the instant in time the same, and change how it's displayed.
Use this when the instant is correct, but you want a more natural display.
2016-04-22 01:26:19 +08:00
```{r}
2016-07-28 03:51:24 +08:00
x4a <- with_tz(x4, tzone = "Australia/Lord_Howe")
x4a
x4a - x4
2016-04-22 01:26:19 +08:00
```
(This also illustrates another challenge of times zones: they're not all integer hour offsets!)
- Change the underlying instant in time.
Use this when you have an instant that has been labelled with the incorrect time zone, and you need to fix it.
2016-04-22 01:26:19 +08:00
```{r}
2016-07-28 03:51:24 +08:00
x4b <- force_tz(x4, tzone = "Australia/Lord_Howe")
x4b
x4b - x4
2016-04-22 01:26:19 +08:00
```
## Summary
This chapter has introduced you to the tools that lubridate provides to help you work with date-time data.
Working with dates and times can seem harder than necessary, but hopefully this chapter has helped you see why --- date-times are more complex than they seem at first glance, and handling every possible situation adds complexity.
Even if your data never crosses a day light savings boundary or involves a leap year, the functions need to be able to handle it.
The next chapter gives a round up of missing values.
You've seen them in a few places and have no doubt encounter in your own analysis, and it's how time to provide a grab bag of useful techniques for dealing with them.