r4ds/datetimes.Rmd

# Dates and times

## Introduction

This chapter will show you how to work with dates and times in R. At first glance, dates and times seem simple. You use them all the time in every day, and generally have too many problems. However, the more you learn about dates and times, the more complicated the get. For example:

* Does every year have 365 days?
* Does every day have 24 hours?
* Does every minute have 60 seconds?

I'm sure you remembered that there are leap years that have 365 days (but do you know the full rule for determining if a year is a leap year?). You might have remembered that many parts of the world use daylight savings time, so that some days have 23 hours, and others have 25. You probably didn't know that some minutes have 61 seconds because occassionally leap seconds are added to keep things in synch.  Read <http://www.creativedeletion.com/2015/01/28/falsehoods-programmers-date-time-zones.html> for even more things that you probably believe that are not true.

Dates and times are hard because they have to reconcile two physical phenonmen (the rotation of the Earth and its orbit around the sun) with a whole raft of cultural phenonmeon including months and time zones. This chapter won't teach you everything about dates and times, but it will give you a solid grounding of practical skills that will help you with common data analysis challenges.

### Prerequisites

This chapter will focus on the __lubridate__ package, which makes it easier to work with dates and times in R. We will use nycflights13 for practice data, and some packages for EDA.

```{r setup, message = FALSE}
library(lubridate)

library(nycflights13)
library(dplyr)
library(ggplot2)
```

## Creating date/times

There are three important 

* A __date__. Number of days since Jan 1, 1970.  `<date>`

* A __date-time__ is a date plus a time.  POSIXct. (We'll come back to POSIXlt
  later - but generally you should avoid it.). Number of seconds since Jan 1, 1970.
  `<dttm>`

* A __time__, the number of seconds. A date + a time = a date-time. Not 
  discussed furher in this chapter. `<time>`

When I want to talk about them collectively I'll use date/times.

If you can use a date, you should. Avoids all the time zome issues you'll learn about later on.

Note that historical dates (before ~1800) are tricky because the world hadn't yet agreed on a standard calendar. Time zones prior to 1970 are hard because the data is not available. If you're working with historical dates/times you'll need to think this through carefully.

There are four ways you are likely to create a date time:

* From a character vector
* From numeric vectors of each component
* From an existing date/time object

There are two special dates/times that are often useful:

```{r}
today()
now()
```

### From strings

Time data normally comes as character strings. You've seen one approach to parsing date times with readr package, in [date-times](#readr-datetimes). Another approach is to use the lubridate helpers. These automatically work out the format once you tell it the order of the day, month, and year components. To use them, identify the order in which the year, month, and day appears in your dates. Now arrange "y", "m", and "d" in the same order. This is the name of the function in lubridate that will parse your dates. For example:

```{r}
ymd("20170131")
mdy("January 31st, 2017")
dmy("31-1-2017")
```

If you have a date-time that also contains hours, minutes, or seconds, add an underscore and then one or more of "h", "m", and "s" to the name of the parsing function.

```{r}
ymd_hms("2017-01-31 20:11:59")
mdy_hm("01/31/2017 08:01")
```

Lubridate's parsing functions handle a wide variety of formats and separators, which simplifies the parsing process. 

### From individual components

Sometimes you'll have the component of a date-time spread across multiple columns, as in the flights data:

```{r}
flights %>% 
  select(year, month, day, hour, minute)
```

To combine separate numbers into a single date-time, use `make_datetime()`:

```{r}
flights %>% 
  select(year, month, day, hour, minute) %>% 
  mutate(departure = make_datetime(year, month, day, hour, minute))
```

Let's do the same thing for every date-time column in `flights`. The times are represented in a slightly odd format, so we use modulus arithmetic to pull out the hour and minute components.  Once that's done, we can drop the old `year`, `month`, and `day`, `hour` and `minute` columns. I've rearrange the variables a bit so they print nicely.

```{r}
make_datetime_100 <- function(year, month, day, time) {
  make_datetime(year, month, day, time %/% 100, time %% 100)
}

flights_dt <- flights %>% 
  filter(!is.na(dep_time), !is.na(arr_time)) %>% 
  mutate(
    dep_time = make_datetime_100(year, month, day, dep_time),
    arr_time = make_datetime_100(year, month, day, arr_time),
    sched_dep_time = make_datetime_100(year, month, day, sched_dep_time),
    sched_arr_time = make_datetime_100(year, month, day, sched_arr_time)
  ) %>% 
  select(origin, dest, ends_with("delay"), ends_with("time"))

flights_dt
```

### From other types

Converting back and forth.

### Exercises

1.  What happens if you parse a string that contains invalid dates?

    ```{r, eval = FALSE}
    ymd(c("2010-10-10", "bananas"))
    ```

1.  What does the `tzone` argument to `today()` do? Why is it important?

1.  Use lubridate to parse each of the following dates:

    ```{r}
    d1 <- "January 1, 2010"
    d2 <- "2015-Mar-07"
    d3 <- "06-Jun-2017"
    d4 <- c("August 19 (2015)", "July 1 (2015)")
    d5 <- "12/30/14" # Dec 30, 2014
    ```

## Date components

Now that we have the scheduled arrival and departure times at date times, let's look at the patterns. We could plot a histogram of flights throughout the year:

```{r}
flights_dt %>% 
  ggplot(aes(dep_time)) + 
  geom_freqpoly(binwidth = 86400) # 86400 seconds = 1 day
```

These are important to know whenever you use a date time in a numeric context. For example, the `binwidth` of a histogram gives the number of seconds for a date-time, and the number of days for a date.  Adding an integer to a date-time vs. adding integer to date.

That's not terribly informative because the pattern is dominated by day of week effects - there are fewer flights of Saturday.

Let's instead group flights by day of the week, to see which week days are the busiest, and by hour to see which times of the day are busiest. To do this we will need to extract the day of the week and hour that each flight was scheduled to depart.

### Getting components

You can pull out individual parts of the date with the acccessor functions `year()`, `month()`, `mday()` (day of the month), `yday()` (day of the year)`, `wday()` (day of the week), `hour()`, `minute()`, `second()`. 

```{r}
datetime <- ymd_hms("2007-08-09 12:34:56")

year(datetime)
month(datetime)
mday(datetime)

yday(datetime)
wday(datetime)
```

For both `month()` and `wday()` you can set `label = TRUE` to return the name of the month or day of the week. Set `abbr = TRUE` to return an abbreviated version of the name, which can be helpful in plots.

```{r}
month(datetime, label = TRUE)
wday(datetime, label = TRUE, abbr = TRUE)
```
We can use the `wday()` accessor to see that more flights depart on weekdays than weekend days. 

```{r}
flights_dt %>% 
  mutate(wday = wday(dep_time, label = TRUE)) %>% 
  ggplot(aes(x = wday)) +
    geom_bar()
```

The `hour()` accessor reveals that scheduled departures follow a bimodal distribution throughout the day. There is a morning and evening peak in departures.

```{r}
flights_dt %>% 
  mutate(hour = hour(dep_time)) %>% 
  ggplot(aes(x = hour)) +
    geom_freqpoly(binwidth = 1)
```

When should you depart if you want to minimize your chance of delay? The results are striking. On average, flights that left on a Saturday arrived ahead of schedule.

```{r, warning = FALSE}
flights_dt %>% 
  mutate(wday = wday(dep_time, label = TRUE)) %>% 
  group_by(wday) %>% 
  summarise(avg_delay = mean(arr_delay, na.rm = TRUE)) %>% 
  ggplot(aes(wday, avg_delay)) +
    geom_bar(stat = "identity")
```

There's an interesting pattern if we look at the average departure delay by minute. It looks like flights leaving around 20-30 and 50-60 generally have much lower delays that you'd expect!

```{r}
flights_dt %>% 
  mutate(minute = minute(dep_time)) %>% 
  group_by(minute) %>% 
  summarise(
    avg_delay = mean(arr_delay, na.rm = TRUE),
    n = n()) %>% 
  ggplot(aes(minute, avg_delay)) +
    geom_line()
```

Interestingly, if we look at the _scheduled_ departure time we don't see such a strong pattern:

```{r, fig.align = "default", out.width = "50%"}
sched_dep <- flights_dt %>% 
  mutate(minute = minute(sched_dep_time)) %>% 
  group_by(minute) %>% 
  summarise(
    avg_delay = mean(arr_delay, na.rm = TRUE),
    n = n())

ggplot(sched_dep , aes(minute, avg_delay)) +
  geom_line()
```

So we do we see such a strong pattern in the delays of actual departure times? Well, like much data collected by humans, there's a strong bias towards flights leaving at "nice" departure times:

```{r}
ggplot(sched_dep , aes(minute, n)) +
  geom_line()
```

So what we're probably seeing is the impact of scheduled flights that leave a few minutes early.

### Rounding

An alternative approach to plotting individual components is to round the date, using `floor_date()`, `round_date()`, and `ceiling_date()` to round (or move) a date to a nearby unit of time. Each function takes a vector of dates to adjust and then the name of the time unit to floor, ceiling, or round them to.

```{r}
flights_dt %>% 
  count(week = floor_date(dep_time, "week")) %>% 
  ggplot(aes(week, n)) +
    geom_line()
```

### Setting components

You can also use each accessor function to set the components of a date or date-time.

```{r}
datetime
year(datetime) <- 2001
datetime
month(datetime) <- 01
datetime
hour(datetime) <- hour(datetime) + 1
```

You can set more than one component at once with `update()`.

```{r}
update(datetime, year = 2002, month = 2, mday = 2, hour = 2)
```

If values are too big, they will roll-over:

```{r}
ymd("2015-02-01") %>% update(mday = 30)
ymd("2015-02-01") %>% update(hour = 400)
```

### Exercises

1.  Confirm my hypthosese that the early departures of flights from 20-30 and 
    50-60 are caused by scheduled flights that leave early.  Hint: create a
    a new categorical variable that tells you whether or not the flight
    was delayed, and group by that.


## Arithmetic with dates

Next we'll learn how to perform

Along the way, you'll learn about three important classes that represent time spaces:

* __durations__, which record an exact number of seconds.
* __periods__, which capture human units like weeks and months.
* __intervals__, which capture a starting and ending point.

### Subtraction

A difftime class object records a span of time in one of seconds, minutes, hours, days, or weeks. R creates a difftime whenever you subtract two dates or two date-times.

```{r}
(day1 <- lubridate::ymd("2000-01-01") - lubridate::ymd("1999-12-31"))
(day2 <- as.difftime(24, units = "hours"))
```

Difftimes come with base R, but they have some rough edges. For example, the value of a difftime depends on the difftime's units attribute. If this attribute is dropped, as it is when you combine difftimes with `c()`, the value becomes uninterpretable. Consider what happens when I combine these two difftimes that have the same length.

```{r}
c(day1, day2)
```

You can avoid these rough edges by using lubridate's version of difftimes, known as durations.

### Addition with durations

Durations behave like difftimes, but are a little more user friendly. To make a duration, choose a unit of time, make it plural, and then place a "d" in front of it. This is the name of the function in lubridate that will make your duration, i.e.

```{r}
dseconds(15)
dminutes(10)
dhours(12)
ddays(7)
dweeks(3)
dyears(1)
```

This makes it easy to arithmetic with date-times. 

Durations always contain a time span measured in seconds. Larger units are estimated by converting minutes, hours, days, weeks, and years to seconds at the standard rate. This makes durations very precise, but it can lead to unexpected results when the timeline is non-contiguous, as with during daylight savings transitions.

Technically, the timeline also misbehaves during __leap seconds__, extra seconds that are added to the timeline to account for changes in the Earth's movement. In practice, most operating systems ignore leap seconds, and R follows the behavior of the operating system. If you are curious about when leap seconds occur, R lists them under `.leap.seconds`.

### Addition with periods

You can use lubridate's period class to handle irregularities in the timeline. Periods are time spans that are generalized to work with clock times, the "name" of a date-time that you would see on a clock, like "2016-03-13 00:00:00." Periods have no fixed length, which lets them work in an intuitive, human friendly way. When you add a one day period to "2000-03-13 00:00:00" the result will be "2000-03-14 00:00:00" whether there were 86400 seconds in March 13, 2000 or 82800 seconds (due to Daylight Savings Time).

To make a period object, call the name of the unit you wish to use, make it plural, and pass it the number of units to use as an argument. 

```{r}
seconds(1)
minutes(1)
hours(1)
days(1)
weeks(1)
months(1)
years(1)
```

You can add periods together to make larger periods.

```{r}
days(50) + hours(25) + minutes(2)
```

To see how periods work, compare the performance of durations and periods during Daylight Savings Time and a leap year.

```{r}
# Daylight Savings Time
ymd_hms("2016-03-13 00:00:00", tz = "America/New_York") + days(1)
ymd_hms("2016-03-13 00:00:00", tz = "America/New_York") + ddays(1)

# A leap year
ymd_hms("2016-01-01 00:00:00") + years(1)
ymd_hms("2016-01-01 00:00:00") + dyears(1)
```

The period always returns the "expected" clock time, as if the irregularity had not happened. The duration always returns the time that is exactly 86,400 seconds (in the case of a day) or 31,536,000 seconds later (in the case of a year).

When the timeline behaves normally, the results of a period and a duration will agree.

```{r}
# Not Daylight Savings Time
ymd_hms("2016-03-14 00:00:00") + days(1)
ymd_hms("2016-03-14 00:00:00") + ddays(1)
```

When should you use a period and when should you use a duration?

* Use durations whenever you need to calculate physical properties or compare exact timespans, such as the life of two different batteries.

* Use periods whenever you need to model human events, such as the opening of the stock market, or the close of the business day.

Periods also let you model date-times that reoccur on a monthly basis in a way that would be impossible with durations. Consider that some of the months below are 31 days, some have 30, and one has 29.

```{r}
mdy("January 1st, 2016") + months(0:11)
```

Let's use periods to fix an oddity related to our flight dates. Some planes appear to have arrived at their destination _before_ they departed from New York City.

```{r}
flights_dt %>% 
  filter(arr_time < dep_time) 
```

These are overnight flights. We used the same date information for both the departure and the arrival times, but these flights arrived on the following day. We can fix this by adding `days(1)` to the arrival time of each overnight flight.

```{r}
flights_dt <- flights_dt %>% 
  mutate(
    overnight = arr_time < dep_time,
    arr_time = arr_time + days(overnight * 1),
    sched_arr_time = sched_arr_time + days(overnight * 1)
  )
```

Now all of our flights obey the laws of physics.

```{r}
flights_dt %>% 
  filter(overnight, arr_time < dep_time) 
```

### Division

It's obvious what `dyears(1) / ddays(365)` should return. It should return one because durations are always represented by seconds, an a duration of a year is defined as 365 days worth of seconds.

What should `years(1) / days(1)` return? Well, if the year was 2015 it should return 365, but if it was 366, it should return 366! There's not quite enough information for lubridate to give a single clear answer. What it does instead is give an estimate, with a warning:

```{r}
years(1) / days(1)
```

If you want a more accurate measurement, you'll have to use an __interval__ instead of a a duration. An interval is a duration with a starting point - that makes it precise so you can determine exactly how long it is:

```{r}
next_year <- today() + years(1)
(today() %--% next_year) / ddays(1)
```

To find out how many periods fall into an interval, you need to use integer division:

```{r}
(today() %--% next_year) %/% days(1)
```

### Summary

Addition

Subtraction

Division

* Duration / Duration =  Number
* Duration / Period =    Error
* Period   / Duration =  Error
* Period   / Period =    Estimated value
* Interval / Period =    Integer with warning
* Interval / Duration =  Number

## Time zones 

Time zones are an enormously complicated topic because of their interaction with geopolitical entities. Fortunately we don't need to dig into all the details as they're not all important for data analysis, but there are a few challenges we'll need to tackle head on.

### Time zone names

The first challange is that the names of time zones that you're familiar with are not very general. For example, if you're an American you're probably familiar with EST, or Eastern Standard Time. However, both Australia and Canada also have Eastern standard times which mean different things! To avoid confusion R uses the international standard IANA time zones. These don't have a terribly consistent naming scheme, but tend to fall in one of three camps:

*   "Country/Region" and "Country", e.g. "US/Central", "Canada/Central",
    "Australia/Sydney", "Japan". These are generally easiest to use if the
    time zone you want is present in the database.

*   "Continent/City", e.g. "America/Chicago", "Europe/Paris", "Australia/NSW".
    Sometimes there are three parts if there have been multiple rules over time 
    for a smaller region (e.g. "America/North_Dakota/New_Salem" 
    vs"America/North_Dakota/Beulah"). Note that Australia is both a continent 
    and a country which makes things confusing.  Fortunately this type is
    rarely relevant for 
  
*   Other, e.g. "CET", "EST".  These are best avoided as they are confusing
    and ambiguous.

You can see a complete list of all time zone names that R knows about with `OlsonNames()`:

```{r}
length(OlsonNames())
head(OlsonNames())
```

And find out what R thinks your current time zone is with `Sys.timezone()`:

```{r}
Sys.timezone()
```

### Daylight Savings Time

An additional complication of time zones is daylight savings time (DST): many time zones shift by an hour during summer time.  For example, the same instants may be the same time or difference times in Denver and Phoenix over the course of the year:

```{r}
x1 <- ymd_hm("2015-01-10 13:00", "2015-05-10 13:00")
with_tz(x1, tzone = "America/Denver")
with_tz(x1, tzone = "America/Phoenix")
```

DST is also challening because it creates discontinuities. What is one day after 1am on  March 13 in New York city? There are two possibilities!

```{r}
nyc <- function(x) {
  ymd_hms(x, tz = "America/New_York")
}
nyc("2016-03-13 01:00:00") + ddays(1)
nyc("2016-03-13 01:00:00") + days(1)
```

This also creates a challenge for determining how much time has elapsed between two date-times. Lubridate also offers solution for this: the __interval__, which you can coerce into either a duration or a period:

```{r}
inst <- nyc("2016-03-13 01:00:00") %--% nyc("2016-03-14 01:00:00")
as.duration(inst)
as.period(inst)
```

### Changing the time zone

In R, time zone is an attribute of the date-time that only controls printing. For example, these three objects represent the same instant in time:

```{r}
x1 <- ymd_hms("2015-06-01 12:00:00", tz = "America/New_York")
x2 <- ymd_hms("2015-06-01 18:00:00", tz = "Europe/Copenhagen")
x3 <- ymd_hms("2015-06-02 04:00:00", tz = "Pacific/Auckland")
```

If you don't specify the time zone, lubridate always assumes UTC.

You can check that's true by subtracting them (we'll talk more about that in the next section)

```{r}
x1 - x2
x1 - x3
```

Operations that drop attributes, such as `c()` will drop the time zone attribute from your date-times. In that case, the date-times will display in your local time zone:

```{r}
x4 <- c(x1, x2, x3)
x4
```

There are two ways to change the time zone:

*   Keep the instant in time the same, and change how it's displayed.
  
    ```{r}
    x4a <- with_tz(x4, tzone = "Australia/Lord_Howe")
    x4a
    x4a - x4
    ```
    
    (This nicely illustrates another possible incorrect believe you might hold:
    that time zones are always whole number changes.)

*   Change the underlying instant in time:

    ```{r}
    x4b <- force_tz(x4, tzone = "Australia/Lord_Howe")
    x4b
    x4b - x4
    ```

### UTC

If you do not set the time zone, lubridate will automatically assign the date-time to Coordinated Universal Time (UTC). Coordinated Universal Time is the standard time zone used by the scientific community and roughly equates to its predecessor, Greenwich Meridian Time. Since Coordinated Universal time does not follow Daylight Savings Time, it is straightforward to work with times saved in this time zone.

```{r}
ymd_hms("2015-06-02 04:00:00")
```