---
title: "Using Multiple APIs"
date: "`r Sys.Date()`"
output: 
  rmarkdown::html_vignette:
    df_output: paged
    fig_width: 5
vignette: >
  %\VignetteIndexEntry{Using Multiple APIs}
  %\VignetteEngine{knitr::rmarkdown}
  \usepackage[utf8]{inputenc}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(collapse = T, comment = "#>")
options(tibble.print_min = 4L, tibble.print_max = 10L)
```

Let's take a look at a few HTTP APIs that transmit data in JSON format, and then get that data into tidy tibbles with tidyjson.

```{r load, echo=TRUE, results='hide', message=FALSE}
library(dplyr)
library(tidyr)
library(jsonlite)
library(tidyjson)
library(ggplot2)
library(lubridate)
```

# Github

The tidyverse is used heavily for data cleansing, so let's explore some tidyverse repository data through Github's APIs.  We are going to grab the data directly and then explore the structure of the JSON with `json_schema`.

```{r gitapi, echo=TRUE}
baseurl <- 'https://api.github.com/repos/tidyverse/dplyr/issues'
dplyr_issues <- as.tbl_json(baseurl)

dplyr_issues %>% json_schema %>% prettify
```

After exploring the structure of the data, we decide we want to look at a high level overview of the isssues we have.  Note that we can grab nested object detail by declaring a more complex path like `jstring('assignee','login')`.  This avoids the tendency to use `enter_object()` where it is not necessary.

```{r gitapi_highlevel, echo=TRUE}

highlevel <- dplyr_issues %>% gather_array('index') %>% 
  spread_values(id=jnumber('id')
                , assignee=jstring('assignee','login')
                , comments=jnumber('comments')
                , title=jstring('title')
                , state=jstring('state')
                , number=jnumber('number')
                )

print(highlevel)

```

And perhaps we want to look at a few different summaries.  We notice that there are only 30 issues here, but anyone familiar with `dplyr` will know that the repo is much more popular than that.  Github's API is paginated, so we only got the first 30 issues back from the API.

```{r gitapi_summarize, echo=TRUE}

highlevel %>% group_by(assignee) %>% summarize(nissues=n())

highlevel %>% group_by(comments) %>% summarize(nissues=n(), issues=paste(number,collapse=',')) %>% 
  ungroup() %>% arrange(desc(comments))

highlevel %>% group_by(state) %>% summarize(nissues=n())

```

Let's aggregate a few more api calls.  Documentation can be found at the [github API docs](https://developer.github.com/guides/traversing-with-pagination/) and in particular [here](https://developer.github.com/v3/issues/#list-issues).

```{r gitapi_many, echo=TRUE}
manyissues <- lapply(c(1:7), function(x){as.tbl_json(paste0(baseurl,'?state=all&per_page=50&page=',x))})

## Collapse into one tbl_json
manyissues <- tidyjson::bind_rows(manyissues)

## Summarize status & users that create issues
manyissues %>% gather_array('issue') %>% spread_values(
  login=jstring('user','login')
  , comments=jnumber('comments')
  , issuenum = jnumber('number')
  , state = jstring('state')
) %>% group_by(login, state) %>% summarize(issuecount=n()) %>% ungroup() %>%
  spread(state, issuecount, fill=0) %>%
  mutate(total=closed+open) %>%
  arrange(desc(total), desc(open)) %>% head(10)
```

# CitiBike NYC

This is a static public API that shows location, status, and current availability for bikes in NYC bike sharing. 

```{r citibike_init, echo=TRUE}
citibike <- as.tbl_json("http://citibikenyc.com/stations/json")

## We see what we have is an object
citibike %>% json_types()

## So let's explore that object
citibike %>% gather_object()
```

Let's explore the array, but store executionTime for later reference:

```{r citibike_prep, echo=TRUE}
citibike_list <- citibike %>% 
  spread_values(execution=jstring(executionTime)) %>%
  enter_object('stationBeanList') %>% gather_array('arrayid')

citibike_list %>%
  filter(arrayid==1) %>%
  json_schema() %>% prettify()

```
### Availability
The percentage availablity of bikes should be linearly correlated.  I.e. 25% bikes available means 75% of docks available.
```{r citibike_available, echo=TRUE}
citibike_available <- citibike_list %>% 
  spread_values(id=jnumber(id)
                , location=jstring(location)
                , lastCommunication=jstring(lastCommunicationTime)
                , availableBikes=jnumber(availableBikes)
                , availableDocks=jnumber(availableDocks)
                , totalDocks=jnumber(totalDocks)) %>%
  mutate(openDockPct=availableDocks / totalDocks
         , bikeDockPct=availableBikes / totalDocks
         , timeSinceUpdateMinutes=as.integer(as_datetime(execution)-as_datetime(lastCommunication))/60
         , timeSinceUpdateBin=cut(timeSinceUpdateMinutes
                                  ,c(0,1,15,60,6*60,24*60,Inf)
                                  , labels=c('0-1 Min','1-15 Min'
                                             , '15 Min - 1 Hr'
                                             , '1-6 Hr'
                                             , '6-24 Hr'
                                             , '24+ Hr'))
         )

## Expect generally linear behavior
ggplot(citibike_available, aes(openDockPct, bikeDockPct)) + geom_point()
```

And if we are in the process of exploring New York City, we probably care about how many actual bikes / docks are available, and how up-to-date that information is.
```{r citibike_count, echo=TRUE}
ggplot(citibike_available, aes(availableBikes, availableDocks, col=timeSinceUpdateBin)) +
  geom_point()

```

### Mapping
Remember that our object is still a tbl_json object, so we can go back and grab additional keys if necessary.  What if we wanted to map the data for easier use while we explore the city?
```{r citibike_map_prep, ECHO=TRUE}
citibike_map <- citibike_available %>%
  spread_values(lat=jnumber(latitude)
                , long=jnumber(longitude))

citibike_map %>% group_by(is.na(lat),is.na(long)) %>% summarize(n())
```

It looks like the data are populated, so we should be good to go!!  This is a feature we plan to add to this vignette in the future.  Data analysis is always more fun with quality visualizations.

### Consistent Behavior

One last point of note.  What if we got a bad response and our pipeline above was automated?

```{r citibike_error_test, ECHO=TRUE}
citibike_list_0 <- '{}' %>% 
  spread_values(execution=jstring(executionTime)) %>%
  enter_object('stationBeanList') %>% gather_array('arrayid')

citibike_available_0 <- citibike_list_0 %>% 
  spread_values(id=jnumber(id)
                , location=jstring(location)
                , lastCommunication=jstring(lastCommunicationTime)
                , availableBikes=jnumber(availableBikes)
                , availableDocks=jnumber(availableDocks)
                , totalDocks=jnumber(totalDocks)) %>%
  mutate(openDockPct=availableDocks / totalDocks
         , bikeDockPct=availableBikes / totalDocks
         , timeSinceUpdateMinutes=as.integer(as_datetime(execution)-as_datetime(lastCommunication))/60
         , timeSinceUpdateBin=cut(timeSinceUpdateMinutes
                                  ,c(0,1,15,60,6*60,24*60,Inf)
                                  , labels=c('0-1 Min','1-15 Min'
                                             , '15 Min - 1 Hr'
                                             , '1-6 Hr'
                                             , '6-24 Hr'
                                             , '24+ Hr'))
  )

ggplot(citibike_available_0, aes(availableBikes, availableDocks, col=timeSinceUpdateBin)) +
  geom_point()
```

While some may prefer an error (and it would be easy enough to check and implement an error of our own using a package like `assertthat`), this is a powerful feature of the `tidyjson` package that allows us to _be sure_ of the structure of data that we receive from parsing the JSON object.  

So if the API changes its schema, or if the response you receive does not have sufficient data, you can rest assurred that the resulting data structure will conform to the specifications you provide and _stay tidy_.  For further information on this, see documentation on `spread_values` (which explicitly defines the data structure you will create) and `spread_all` (which is easier to use when interactively exploring).
