---
title: "CDS functionality"
author: "Reto Stauffer, Koen Hufkens"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{CDS functionality}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

# load the library
library(ncdf4)
library(raster)
library(maps)
```

## Downloading Data Sets from Copernicus's Climate Data Store

[Copernicus.eu](https://www.copernicus.eu/en) provides a set of interesting data sets for research, education, and applied earth sciences on their [Climate Data Store](https://cds.climate.copernicus.eu) (CDS). Among the different data sets there is the latest [ECMWF](https://www.ecmwf.int) high-resolution reanalysis data set which replaces ERA Interim (which was ERA-4; version 4). The R package `ecmwfr` provides a convenient interface to download these data sets.

## Before Downloading Data

Before you will be able to download any data you need to get a free personal
account. 

* [Register yourself for CDS services](https://cds.climate.copernicus.eu/user/register)
* [Terms and conditions](https://cds.climate.copernicus.eu/disclaimer-privacy)

To retrieve data `ecmwfr` use `wf_set_key()`to add the login details to your local keyring (requires the _R_ package [`keyring`](https://CRAN.R-project.org/package=keyring)).

Once you are in possession of your personal user (namely your user ID and a secret key) `ecmwfr` allows to send requests to CDS and/or download the data.

## The Request Syntax

CDS data retrievals are based on a `list` object which specifies the data set to be downloaded.  These definitions are called `requests` (for those who are familiar with _mars_: these are basically _mars_ requests). A `request` defined the type of the `dataset`, the `variables` to be downloaded, the time period, output `format`, `target` location, a custom `area` extent, and other details.

The request syntax is available for a range of different CDS data sets. Check the [CDS Dataset](https://cds.climate.copernicus.eu/cdsapp#!/search?type=dataset) website to see a list of available datasets and to check whether API requests are allowed or not (go to _Download Data_, select some data, show request by clicking _Show API Request_ (red button, bottom of page).

The conversion from a python based string to the list format can be automated if you use the RStudio based Addin. By selecting and using Addin -> python to list (or 'Mars to list') you dynamically convert queries copied from either ECMWF or CDS based services.

![](https://user-images.githubusercontent.com/1354258/56429618-def12080-62c3-11e9-90e4-0954e10d06f3.gif)

### An ERA-5 example

```{r demo request, echo = TRUE}
# Specify the data set
request <- list("dataset_short_name" = "reanalysis-era5-pressure-levels",
                "product_type"   = "reanalysis",
                "variable"       = "temperature",
                "pressure_level" = "850",
                "year"           = "2000",
                "month"          = "04",
                "day"            = "04",
                "time"           = "00:00",
                "area"           = "70/-20/30/60",
                "format"         = "netcdf",
                "target"         = "era5-demo.nc")
```

This is the "demo" request and it specifies the following:

* `dataset`: downloading ERA-5 reanalysis on pressure level(s)
* `product_type`: (deterministic) reanalysis data
* `variable`/`pressure_level`: requesting temperature on 850 hectopascal
* `year`/`month`/`day`: April 4, 2000 (one day in this example)
* `time`: valid at 00:00 UTC (date/time always in UTC),
* `area`: custom subset covering northern Europe
* `format`: output format NetCDF
* `target`: local output file `era5-demo.nc`

The data set as specified above can be downloaded calling the
`wf_request` function:

```{r spatial-request, echo = TRUE, eval = FALSE}
# Start downloading the data, the path of the file
# will be returned as a variable (ncfile)
ncfile <- wf_request(user = "2088",
                      request = request,   
                      transfer = TRUE,  
                      path = "~",
                      verbose = FALSE)
```

Depending on the request (the amount of data you are asking for) the request function may tike a while! Please note: if you try to download larger amounts of data it is suggested to split the data sets, e.g., download year-by-year, or even month-by-month, if you are trying to download several varaiables/fields.

Once the retrieval has finished you should now be the owner of a NetCDF containing the requested information located in the current working directory, called `era5-demo.nc`. 

Quick check:
```{r echo = FALSE}
ncfile <- system.file(package = "ecmwfr","extdata/cds.nc")
```

```{r spatial-plot, echo = TRUE, figure = TRUE, fig.width = 8, fig.height = 6}
# Open NetCDF file and plot
r <- raster::raster(ncfile)
raster::plot(r, main = "ERA-5 Reanalysis Demo (2m Temperature 850 hPa)")
maps::map("world", add = TRUE)
```

## Date period specification

For those familiar to ECMWF _mars_ syntax: CDS does not
accept `date = "2000-01-01/to/2000-12-31"` specifications at the moment. It is possible to specify one specific date via `date = "2000-01-01"` or multiple days via `date = ["2000-01-01","2000-01-02","2000-10-20"]` but
not via `".../to/..."`.
