---
title: "`fivethirtyeight` Package"
author: "Albert Y. Kim, Chester Ismay, and Jennifer Chunn"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: |
  %\VignetteIndexEntry{Motivation, Principles, and Data Sets Included}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, message=FALSE, warning=FALSE, echo=FALSE}
library(fivethirtyeight)
# tidyverse includes ggplot2, tibble, tidyr, readr, purrr, dplyr:
library(tidyverse)
library(knitr)
```


## Motivation

We are involved in statistics and data science education, in particular at the
introductory undergraduate level. As such, we are always looking for data sets that 
balance being

1. **Rich enough** to answer meaningful questions with, **real enough** to ensure that there is context, and **realistic enough** to convey to students that data as it exists "in the wild" often needs processing. 
1. Easily and quickly accessible to novices, so that we [minimize the prerequisites to research](https://arxiv.org/abs/1507.05346). 

It has been our experience that many data sets that exist in R packages, such as the 
 [`nycflights13`](https://github.com/hadley/nycflights13), 
[`babynames`](https://github.com/hadley/babynames), and [`gapminder`](https://github.com/jennybc/gapminder)
packages, are of great pedagogical value as they:

* Satisfy the above two goals.
* Are in standardized format as they fit into the [tidy tools](https://CRAN.R-project.org/package=tidyverse/vignettes/manifesto.html) ecosystem.
* Are really fun to play with!

It is along these lines that we present `fivethirtyeight`: an R package of data and code behind the stories and interactives at [FiveThirtyEight.com](http://fivethirtyeight.com/), a data-driven journalism website founded by Nate Silver and owned by ESPN. FiveThirtyEight has been very forward thinking in making the data used in many of their articles open and accessible on [GitHub](https://github.com/fivethirtyeight/data), a web-based repository for collaboration on code and data. 

With consultation from [Andrew Flowers](http://fivethirtyeight.com/contributors/andrew-flowers/) and [Andrei Scheinkman](http://fivethirtyeight.com/contributors/andrei-scheinkman/) of FiveThirtyEight, we go one step further by:

1. Doing just enough pre-processing so that statistics and data science novices can sink their teeth into the data right away.
2. Packaging it all in an easy to load format: package installation instead of working with CSV files.
3. Providing easily accessible documentation: The help file for each data set includes a thorough description of the observational unit and all variables, a link to the original article, and (if listed) the data sources.






## Guidelines

In order to make the data easily accessible to R novices, we pre-process the
original data sets as they exist in the [538 GitHub 
repository](https://github.com/fivethirtyeight/data) to adhere to the following 
guidelines:

1. **Naming conventions for data frame and variable names**:
    1. Whenever possible, all names should be no more than 20 characters long. Exceptions to this rule exist when shortening the names to less than 20 characters would lead to a loss of information.
    1. Use only lower case characters and replace all spaces with underscores. This format is known as `snake_case` and is an alternative to `camelCase`.
    1. In the case of variable (column) names within a data frame, use underscores instead of spaces.
1. **Variables identifying observational units**:
    1. Any variables uniquely identifying each observational unit should be in the left-hand columns. 
1. **Dates**:
    1. If only a `year` variable exists, then it should be represented as a numerical variable.
    1. If there are `year` and `month` variables, then convert them to `Date` objects as `year-month-01`. In other words, associate all observations from the same month to have a `day` of `01` so that a correct `Date` object can be assigned.
    1. If there are `year`, `month`, and `day` variables, then convert them to `Date` objects as `year-month-day`.
1. **Ordered Factors, Factors, Characters, and Logicals**:
    1. Ordinal categorical variables are represented as `ordered` factors.
    1. Categorical variables with a fixed and known set of levels are represented as regular `factor`s.
    1. Categorical variables whose possible levels are either unknown or of a very large number are represented as `character`s.
    1. Any "yes/no" character encoding of binary variables is converted to `TRUE/FALSE` logical variables. 
1. **Tidy data format**: 
    1. Whenever possible, save all data frames in ["tidy" data](http://vita.had.co.nz/papers/tidy-data.html) format:
        a) Each variable forms a column.
        a) Each observation forms a row.
        a) Each type of observational unit forms a table.
    1. If converting the raw data to "tidy" data format alters the dataset too much, then make the code to convert to tidy format easily accessible in the help file. 

**Note**: The code used to pre-process the data can be found on the [GitHub repository](https://github.com/rudeboybert/fivethirtyeight/tree/master/data-raw) for the package in the `process_data_sets.R` files. These can serve as data manipulation/wrangling examples and exercises for more advanced students.




## Data Sets

```{r, message=FALSE, warning=FALSE, echo=FALSE}
datasets <- read_csv("https://goo.gl/OT8iHa") %>% 
  filter(!is.na(DATAFRAME_NAME)) %>% 
  mutate(DATAFRAME_NAME = paste("`", DATAFRAME_NAME, "`", sep="")) %>% 
  transmute(
    `Data Frame Name` = DATAFRAME_NAME,
    `Article Title` = ARTICLE_TITLE,
    `Filed Under` = ARTICLE_FILED_UNDER,
    `Author 1` = ARTICLE_AUTHOR_1,
    `Author 2` = ARTICLE_AUTHOR_2,
    `Author 3` = ARTICLE_AUTHOR_3,
    Date = ARTICLE_DATE,
    URL = ARTICLE_URL
  ) %>% 
  arrange(`Data Frame Name`)
kable(datasets)
```


