---
title: "Subset a Data Frame"
author: "David Gerbing"
output: 
  rmarkdown::html_vignette:
    toc: true
vignette: >
  %\VignetteIndexEntry{1. Data: Subset a Data Frame}
  %\VignetteEngine{knitr::rmarkdown}
  \usepackage[utf8]{inputenc}
---

```{r}
library("lessR")
```

## Subset a Data Frame with __Base R__ `Extract`

The most general way to subset a data frame by rows and/or columns is the base R `Extract` function, called by `d[rows, columms]`, where _d_ is the data frame. To use this function, for the _rows_ parameter, pass the row names of the selected rows, the indices or actual names, or pass a logical statement that, when evaluated, results in these names. For the _cols_ parameter, pass the column indices of the selected columns, or pass a list of variable names that reduces to these indices. 

In the logical expression of rows, use the standard R operators. 

operator  | meaning
-------- | --------
&  | and
\| | or
! | not
== | is equal to
!= | is not equal to
%in% | is in a vector

For the column specification, specify a variable range of contiguous variables with a colon, `:`. 

## Annoying Features of __Base R__ `Extract`

When `Extract` evaluates the row or column specifications to obtain the indices, there are several limitations.

1. `rows`: Any reference to the variables in the data frame for this specification must contain the name of the data frame followed by a \$. But this name has already been specified in the function call, and now is redundant, repeated for every variable reference.
2. `cols`: Usually specified with a vector of variable names.
  + All variable names in in the provided vector of names must be quoted. 
  + No variable ranges.
3. No character strings that store the values passed to `rows` and `cols`. Instead directly enter the conditions for both rows and columns, which can make the entire expression quite large.


## More Flexible Use of `Extract`

To address the first two deficiencies, one possibility is the base R `subset()` function. To use `Extract` directly,  __lessR__ provides the function `.()` for obtaining the indices of selected rows and of selected columns. This function is only callable within the base R `Extract` function, with what R refers to as non-standard evaluation. That basically means that the annoying restrictions are removed, though in some advanced uses the `.()` may not apply.

The general form of the subsetting follows.

```
d[.(rows), .(columns)]
```

That is, call the `Extract` function as before, but now wrap the row and column expressions with `.()`.

To illustrate, use the Employee data set contained in __lessR__, here read into the _d_ data frame.

```{r}
d <- Read("Employee")
```

Subset the data frame by only listing observations with a _Gender_ of "M" with scores on _Post_ larger than 90. Only list columns for the variables in the range from _Years_ to _Salary_, and _Post_. Referring back to the output of `Read()`, the variable range includes _Years_, _Gender_, _Dept_, and _Salary_.

```{r}
d[.(Gender=="M" & Post>90), .(Years:Salary, Post)]
```

Following is the traditional R call to `Extract` for subsetting.

```{r}
d[d$Gender=="M" & d$Post>90, c("Years", "Gender", "Dept", "Salary", "Post")]
```

To negate a row selection, add a `!` to the beginning of the logical condition passed to `.()`, within the call to `.()`. To exclude the specified variables, place a `-`, in front of the call to `.()`.

```{r}
d[.(!(Gender=="M" & Post>90)), -.(Dept:Plan, Pre)]
```

Can still provide the indices directly for one or both of the expressions as the base R `Extract` function is unmodified.

```{r}
d[1:3, .(Years:Salary, Post)]
d[.(Gender=="M" & Post>90), 1:3]
```

To enhance readability, store the specified row or column conditions as character strings. Each string must be named `rows` or `cols`. Because the entire expression is a character string, differentiate between single and double quotes as needed. For example, use single quotes within the string and double quotes to define the entire string, as illustrated next.

```{r}
rows <- "Gender=='M' & Post>93"
cols <- "Gender:Salary, Post"
```

To subset, pass the respective character strings, `rows` and `cols`, to `.()`, respectively.

```{r}
d[.(rows), .(cols)]
```

To negate, as with the literal expressions, use `!` for the rows and `-` for the columns. Notice their placement.

```{r}
d[.(!rows), -.(cols)]
```

## Missing Data

The variable _Dept_ is missing for the fourth row of data.

```{r}
d[1:5,]
```

Here with the traditional use of `Extract`, specify rows of data only when the value of _Dept_ is _ADMN_. 

```{r}
d[d$Dept=="ADMN", c('Gender', 'Dept', 'Salary')]
```

The result provides what was requested, and also when _Dept_ is `<NA>`, which is not requested. The requested value of _ADMN_ is not the same as `<NA>`.

Use the `.()` function to obtain what is requested, rows of data in which the value of _Dept_ is _ADMN_.

```{r}
d[.(Dept=="ADMN"), .(Gender:Salary)]
```

If rows with the value of the variable missing are desired, then `.()` provides that information only when requested, such as with the base R function `is.na()`.
```{r}
d[.(Dept=="ADMN" | is.na(Dept)), .(Gender:Salary)]
```


## Random Selection of Rows

The function `.()` also provides for random selection of rows. To randomly select the specified number of rows from the data frame to subset, specify the `random()` function for the logical criterion of the `rows`. The value passed to `random()` can either be the actual number of rows to select, or the proportion of rows to select.

Here randomly select five rows of data from the _d_ data frame.

```{r}
d[.(random(5)), .(Years:Salary)]
```

Here specify a proportion of rows to select.

```{r}
d[.(random(0.1)), .(Years:Salary)]
```

