<!--

    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.

-->

# Chapter 7: TsFile

## Usage

Now, you are ready to start doing some awesome things with TsFile. This section demonstrates the detailed usages of TsFile.

### Time-series Data
A time-series is considered as a sequence of quadruples. A quadruple is defined as (device, measurement, time, value).

* **measurement**: A physical or formal measurement that a time-series is taking, e.g., the temperature of a city, the 
sales number of some goods or the speed of a train at different times. As a traditional sensor (like a thermometer) also
 takes a single measurement and produce a time-series, we will use measurement and sensor interchangeably below.

* **device**: A device refers to an entity that is taking several measurements (producing multiple time-series), e.g., 
a running train monitors its speed, oil meter, miles it has run, current passengers each is conveyed to a time-series.

Table 1 illustrates a set of time-series data. The set showed in the following table contains one device named "device\_1" 
with three measurements named "sensor\_1", "sensor\_2" and "sensor\_3". 

<center>
<table style="text-align:center">
	<tr><th colspan="6">device_1</th></tr>
	<tr><th colspan="2">sensor_1</th><th colspan="2">sensor_2</th><th colspan="2">sensor_3</th></tr>
	<tr><th>time</th><th>value</td><th>time</th><th>value</td><th>time</th><th>value</td>
	<tr><td>1</td><td>1.2</td><td>1</td><td>20</td><td>2</td><td>50</td></tr>
	<tr><td>3</td><td>1.4</td><td>2</td><td>20</td><td>4</td><td>51</td></tr>
	<tr><td>5</td><td>1.1</td><td>3</td><td>21</td><td>6</td><td>52</td></tr>
	<tr><td>7</td><td>1.8</td><td>4</td><td>20</td><td>8</td><td>53</td></tr>
</table>
<span>A set of time-series data</span>
</center>

**One Line of Data**: In many industrial applications, a device normally contains more than one sensor and these sensors
 may have values at a same timestamp, which is called one line of data. 

Formally, one line of data consists of a `device_id`, a timestamp which indicates the milliseconds since January 1,
 1970, 00:00:00, and several data pairs composed of `measurement_id` and corresponding `value`. All data pairs in one 
 line belong to this `device_id` and have the same timestamp. If one of the `measurements` does not have a `value` 
 in the `timestamp`, use a space instead(Actually, TsFile does not store null values). Its format is shown as follow:

```
device_id, timestamp, <measurement_id, value>...
```

An example is illustrated as follow. In this example, the data type of two measurements are  `INT32`, `FLOAT` respectively.

```
device_1, 1490860659000, m1, 10, m2, 12.12
```


### Writing TsFile

#### Generate a TsFile File.
A TsFile can be generated by following three steps and the complete code will be given in the section "Example for writing TsFile".

* First, construct a `TsFileWriter` instance.
    
    Here are the available constructors:
    
    * Without pre-defined schema
    ```
    public TsFileWriter(File file) throws IOException
    ```
    * With pre-defined schema
	```
	public TsFileWriter(File file, FileSchema schema) throws IOException
	```
	This one is for using the HDFS file system. `TsFileOutput` can be an instance of class `HDFSOutput`.
	
	```
	public TsFileWriter(TsFileOutput output, FileSchema schema) throws IOException 
    ```
	**Parameters:**
	
	* file : The TsFile to write
	
	* schema : The file schemas, will be introduced in next part.

* Second, add measurements
	
	Or you can make an instance of class `FileSchema` first and pass this to the constructor of class `TsFileWriter`
	
	The class `FileSchema` contains a map whose key is the name of one measurement schema, and the value is the schema itself.
	
	Here are the interfaces:
	```
	// Create an empty FileSchema or from an existing map
	public FileSchema()
	public FileSchema(Map<String, MeasurementSchema> measurements)
    
	// Use this two interfaces to add measurements
	public void registerMeasurement(MeasurementSchema descriptor)
	public void registerMeasurements(Map<String, MeasurementSchema> measurements)
    
	// Some useful getter and checker
	public TSDataType getMeasurementDataType(String measurementId)
	public MeasurementSchema getMeasurementSchema(String measurementId)
	public Map<String, MeasurementSchema> getAllMeasurementSchema()
	public boolean hasMeasurement(String measurementId)
	```
	
	You can always use the following interface in `TsFileWriter` class to add additional measurements: 
        
    ```
    public void addMeasurement(MeasurementSchema measurementSchema) throws WriteProcessException
    ```
	
	The class `MeasurementSchema` contains the information of one measurement, there are several constructors:
	```
	public MeasurementSchema(String measurementId, TSDataType type, TSEncoding encoding)
    
	public MeasurementSchema(String measurementId, TSDataType type, TSEncoding encoding, CompressionType compressionType)
	
	public MeasurementSchema(String measurementId, TSDataType type, TSEncoding encoding, CompressionType compressionType, 
	Map<String, String> props)
    ```
    
    **Parameters:**
    	
    * measurementID: The name of this measurement, typically the name of the sensor.
    	
    * type: The data type, now support six types: `BOOLEAN`, `INT32`, `INT64`, `FLOAT`, `DOUBLE`, `TEXT`;
    
    * encoding: The data encoding. See [Chapter 2-3](../2-Concept%20Key%20Concepts%20and%20Terminology/3-Encoding.md).
    
    * compression: The data compression. Now supports `UNCOMPRESSED` and `SNAPPY`.
    
    * props: Properties for special data types.Such as `max_point_number` for `FLOAT` and `DOUBLE`, `max_string_length` for
    `TEXT`. Use as string pairs into a map such as ("max_point_number", "3").
	
    > **Notice:** Although one measurement name can be used in multiple deltaObjects, the properties cannot be changed. I.e. 
        it's not allowed to add one measurement name for multiple times with different type or encoding.
        Here is a bad example:
        
     	// The measurement "sensor_1" is float type
     	addMeasurement(new MeasurementSchema("sensor_1", TSDataType.FLOAT, TSEncoding.RLE));
     	
     	// This call will throw a WriteProcessException exception
     	addMeasurement(new MeasurementSchema("sensor_1", TSDataType.INT32, TSEncoding.RLE));
* Third, insert and write data continually.
	
	Use this interface to create a new `TSRecord`(a timestamp and device pair).
	
	```
	public TSRecord(long timestamp, String deviceId)
	```
	Then create a `DataPoint`(a measurement and value pair), and use the addTuple method to add the DataPoint to the correct
	TsRecord.
	
	Use this method to write
	
	```
	public void write(TSRecord record) throws IOException, WriteProcessException
	```
	
* Finally, call `close` to finish this writing process. 
	
	```
	public void close() throws IOException
	```

#### Example for writing a TsFile

You should install TsFile to your local maven repository.

See reference: [Installation](./1-Installation.md)

A more thorough example can be found at `/tsfile/example/src/main/java/org/apache/iotdb/tsfile/TsFileWrite.java`

```java
package org.apache.iotdb.tsfile;

import java.io.File;
import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding;
import org.apache.iotdb.tsfile.write.TsFileWriter;
import org.apache.iotdb.tsfile.write.record.TSRecord;
import org.apache.iotdb.tsfile.write.record.datapoint.DataPoint;
import org.apache.iotdb.tsfile.write.record.datapoint.FloatDataPoint;
import org.apache.iotdb.tsfile.write.record.datapoint.IntDataPoint;
import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
/**
 * An example of writing data to TsFile
 * It uses the interface:
 * public void addMeasurement(MeasurementSchema MeasurementSchema) throws WriteProcessException
 */
public class TsFileWrite {

  public static void main(String args[]) {
    try {
      String path = "test.tsfile";
      File f = new File(path);
      if (f.exists()) {
        f.delete();
      }
      TsFileWriter tsFileWriter = new TsFileWriter(f);

      // add measurements into file schema
      tsFileWriter
              .addMeasurement(new MeasurementSchema("sensor_1", TSDataType.FLOAT, TSEncoding.RLE));
      tsFileWriter
              .addMeasurement(new MeasurementSchema("sensor_2", TSDataType.INT32, TSEncoding.TS_2DIFF));
      tsFileWriter
              .addMeasurement(new MeasurementSchema("sensor_3", TSDataType.INT32, TSEncoding.TS_2DIFF));
      // construct TSRecord
      TSRecord tsRecord = new TSRecord(1, "device_1");
      DataPoint dPoint1 = new FloatDataPoint("sensor_1", 1.2f);
      DataPoint dPoint2 = new IntDataPoint("sensor_2", 20);

     //For time 1 in device_1, the data will be 1.2, 20, null
      tsRecord.addTuple(dPoint1);
      tsRecord.addTuple(dPoint2);

      // write a TSRecord to TsFile
      tsFileWriter.write(tsRecord);
      // close TsFile
      tsFileWriter.close();
    } catch (Throwable e) {
      e.printStackTrace();
      System.out.println(e.getMessage());
    }
  }
}

```

### Interface for Reading TsFile

#### Before the Start

The set of time-series data in section "Time-series Data" is used here for a concrete introduction in this section. The set showed in the following table contains one deltaObject named "device\_1" with three measurements named "sensor\_1", "sensor\_2" and "sensor\_3". And the measurements has been simplified to do a simple illustration, which contains only 4 time-value pairs each.

<center>
<table style="text-align:center">
	<tr><th colspan="6">device_1</th></tr>
	<tr><th colspan="2">sensor_1</th><th colspan="2">sensor_2</th><th colspan="2">sensor_3</th></tr>
	<tr><th>time</th><th>value</td><th>time</th><th>value</td><th>time</th><th>value</td>
	<tr><td>1</td><td>1.2</td><td>1</td><td>20</td><td>2</td><td>50</td></tr>
	<tr><td>3</td><td>1.4</td><td>2</td><td>20</td><td>4</td><td>51</td></tr>
	<tr><td>5</td><td>1.1</td><td>3</td><td>21</td><td>6</td><td>52</td></tr>
	<tr><td>7</td><td>1.8</td><td>4</td><td>20</td><td>8</td><td>53</td></tr>
</table>
<span>A set of time-series data</span>
</center>

#### Definition of Path

A path is a dot-separated string which uniquely identifies a time-series in TsFile, e.g., "root.area_1.device_1.sensor_1". 
The last section "sensor_1" is called "measurementId" while the remaining parts "root.area_1.device_1" is called deviceId. 
As mentioned above, the same measurement in different devices has the same data type and encoding, and devices are also unique.

In read interfaces, The parameter ```paths``` indicates the measurements to be selected.

Path instance can be easily constructed through the class ```Path```. For example:

```
Path p = new Path("device_1.sensor_1");
```

We will pass an ArrayList of paths for final query call to support multiple paths.

```
List<Path> paths = new ArrayList<Path>();
paths.add(new Path("device_1.sensor_1"));
paths.add(new Path("device_1.sensor_3"));
```

> **Notice:** When constructing a Path, the format of the parameter should be a dot-separated string, the last part will
 be recognized as measurementId while the remaining parts will be recognized as deviceId.


#### Definition of Filter

##### Usage Scenario
Filter is used in TsFile reading process to select data satisfying one or more given condition(s). 

##### IExpression
The `IExpression` is a filter expression interface and it will be passed to our final query call.
We create one or more filter expressions and may use binary filter operators to link them to our final expression.

* **Create a Filter Expression**
	
	There are two types of filters.
	
	 * TimeFilter: A filter for `time` in time-series data.
	 	```
	 	IExpression timeFilterExpr = new GlobalTimeExpression(TimeFilter);
	 	```
	 	Use the following relationships to get a `TimeFilter` object (value is a long int variable).
	    <center>
        <table style="text-align:center">
        	<tr><th>Relationship</th><th>Description</td></tr>
        	<tr><td>TimeFilter.eq(value)</td><td>Choose the time equal to the value</td>
        	<tr><td>TimeFilter.lt(value)</td><td>Choose the time less than the value</td>
        	<tr><td>TimeFilter.gt(value)</td><td>Choose the time greater than the value</td>
        	<tr><td>TimeFilter.ltEq(value)</td><td>Choose the time less than or equal to the value</td>
        	<tr><td>TimeFilter.gtEq(value)</td><td>Choose the time greater than or equal to the value</td>
        	<tr><td>TimeFilter.notEq(value)</td><td>Choose the time not equal to the value</td>
        	<tr><td>TimeFilter.not(TimeFilter)</td><td>Choose the time not satisfy another TimeFilter</td>
        </table>
        </center>
	 	
	 * ValueFilter: A filter for `value` in time-series data.
	 	
	 	```
	 	IExpression valueFilterExpr = new SingleSeriesExpression(Path, ValueFilter);
	 	```
		The usage of  `ValueFilter` is the same as using `TimeFilter`, just to make sure that the type of the value
		equal to the measurement's(defined in the path).

* **Binary Filter Operators**

	Binary filter operators can be used to link two single expressions.

	 * BinaryExpression.and(Expression, Expression): Choose the value satisfy for both expressions.
	 * BinaryExpression.or(Expression, Expression): Choose the value satisfy for at least one expression.
	 

##### Filter Expression Examples

* **TimeFilterExpression Examples**

	```
	IExpression timeFilterExpr = new GlobalTimeExpression(TimeFilter.eq(15)); // series time = 15

	```
	```
	IExpression timeFilterExpr = new GlobalTimeExpression(TimeFilter.ltEq(15)); // series time <= 15

	```
	```
	IExpression timeFilterExpr = new GlobalTimeExpression(TimeFilter.lt(15)); // series time < 15

	```
	```
	IExpression timeFilterExpr = new GlobalTimeExpression(TimeFilter.gtEq(15)); // series time >= 15

	```
	```
	IExpression timeFilterExpr = new GlobalTimeExpression(TimeFilter.notEq(15)); // series time != 15

	```
	```
	IExpression timeFilterExpr = BinaryExpression.and(new GlobalTimeExpression(TimeFilter.gtEq(15L)),
                                             new GlobalTimeExpression(TimeFilter.lt(25L))); // 15 <= series time < 25
	```
	```
	IExpression timeFilterExpr = BinaryExpression.or(new GlobalTimeExpression(TimeFilter.gtEq(15L)),
                                             new GlobalTimeExpression(TimeFilter.lt(25L))); // series time >= 15 or series time < 25
	```
#### Read Interface

First, we open the TsFile and get a `ReadOnlyTsFile` instance from a file path string `path`.

```
TsFileSequenceReader reader = new TsFileSequenceReader(path);
   
ReadOnlyTsFile readTsFile = new ReadOnlyTsFile(reader);
```
Next, we prepare the path array and query expression, then get final `QueryExpression` object by this interface:

```
QueryExpression queryExpression = QueryExpression.create(paths, statement);
```

The ReadOnlyTsFile class has two `query` method to perform a query.
* **Method 1**

	```
	public QueryDataSet query(QueryExpression queryExpression) throws IOException
	```

* **Method 2**

	```
	public QueryDataSet query(QueryExpression queryExpression, long partitionStartOffset, long partitionEndOffset) throws IOException
	```

	This method is designed for advanced applications such as the TsFile-Spark Connector.

	* **params** : For method 2, two additional parameters are added to support partial query:
		*  ```partitionStartOffset```: start offset for a TsFile
		*  ```partitionEndOffset```: end offset for a TsFile

		> **What is Partial Query ?**
		>
		> In some distributed file systems(e.g. HDFS), a file is split into severval parts which are called "Blocks" and stored in different nodes. Executing a query paralleled in each nodes involved makes better efficiency. Thus Partial Query is needed. Paritial Query only selects the results stored in the part split by ```QueryConstant.PARTITION_START_OFFSET``` and ```QueryConstant.PARTITION_END_OFFSET``` for a TsFile.

#### QueryDataset Interface

The query performed above will return a `QueryDataset` object.

Here's the useful interfaces for user.


* `bool hasNext();`

    Return true if this dataset still has elements.
* `List<Path> getPaths()`

    Get the paths in this data set.
* `List<TSDataType> getDataTypes();` 

   Get the data types. The class TSDataType is an enum class, the value will be one of the following:
   
       BOOLEAN,
       INT32,
       INT64,
       FLOAT,
       DOUBLE,
       TEXT;
 * `RowRecord next() throws IOException;`
 
    Get the next record.
    
    The class `RowRecord` consists of a `long` timestamp and a `List<Field>` for data in different sensors,
     we can use two getter methods to get them.
     
    ```
    long getTimestamp();
    List<Field> getFields();
    ```
    
    To get data from one Field, use these methods:
    
    ```
    TSDataType getDataType();
    Object getObjectValue();
    ```

#### Example for reading an existing TsFile


You should install TsFile to your local maven repository.

See reference: [Installation](./1-Installation.md)

A more thorough example with query statement can be found at 
`/tsfile/example/src/main/java/org/apache/iotdb/tsfile/TsFileRead.java`

```java
package org.apache.iotdb.tsfile;
import java.io.IOException;
import java.util.ArrayList;
import org.apache.iotdb.tsfile.read.ReadOnlyTsFile;
import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
import org.apache.iotdb.tsfile.read.common.Path;
import org.apache.iotdb.tsfile.read.expression.IExpression;
import org.apache.iotdb.tsfile.read.expression.QueryExpression;
import org.apache.iotdb.tsfile.read.expression.impl.BinaryExpression;
import org.apache.iotdb.tsfile.read.expression.impl.GlobalTimeExpression;
import org.apache.iotdb.tsfile.read.expression.impl.SingleSeriesExpression;
import org.apache.iotdb.tsfile.read.filter.TimeFilter;
import org.apache.iotdb.tsfile.read.filter.ValueFilter;
import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;

/**
 * The class is to show how to read TsFile file named "test.tsfile".
 * The TsFile file "test.tsfile" is generated from class TsFileWrite.
 * Run TsFileWrite to generate the test.tsfile first
 */
public class TsFileRead {
  private static void queryAndPrint(ArrayList<Path> paths, ReadOnlyTsFile readTsFile, IExpression statement)
          throws IOException {
    QueryExpression queryExpression = QueryExpression.create(paths, statement);
    QueryDataSet queryDataSet = readTsFile.query(queryExpression);
    while (queryDataSet.hasNext()) {
      System.out.println(queryDataSet.next());
    }
    System.out.println("------------");
  }

  public static void main(String[] args) throws IOException {

    // file path
    String path = "test.tsfile";

    // create reader and get the readTsFile interface
    TsFileSequenceReader reader = new TsFileSequenceReader(path);
    ReadOnlyTsFile readTsFile = new ReadOnlyTsFile(reader);
    // use these paths(all sensors) for all the queries
    ArrayList<Path> paths = new ArrayList<>();
    paths.add(new Path("device_1.sensor_1"));
    paths.add(new Path("device_1.sensor_2"));
    paths.add(new Path("device_1.sensor_3"));

    // no query statement
    queryAndPrint(paths, readTsFile, null);

    //close the reader when you left
    reader.close();
  }
}

```

### User-specified config file path

Default config file `tsfile-format.properties.template` is located at `/tsfile/src/main/resources` directory. If you want to use your own path, you can:
```
System.setProperty(TsFileConstant.TSFILE_CONF, "your config file path");
```
and then call:
```
TSFileConfig config = TSFileDescriptor.getInstance().getConfig();
```


