<!--

    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
    
        http://www.apache.org/licenses/LICENSE-2.0
    
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.

-->

## Cluster Setup

This article is the setup process of IoTDB Cluster (1.0.0).

## Environments

1. JDK>=1.8.

2. Max open file 65535.

3. Disable the swap memory.

## Get the binary distribution

### Download

Download the binary distribution from website [Download Page](https://iotdb.apache.org/Download/).

### Compiled from source code

Download the source code:

```
git clone https://github.com/apache/iotdb.git
```

The default branch is master, you should checkout to the release tag:

```
git checkout v1.0.0
```

Under the source root folder:

```
mvn clean package -pl distribution -am -DskipTests
```

Then you will get the binary distribution under **distribution/target**, in which the **all-bin** contains ConfigNode and DataNode, and DataNode contains the Cli.

## Binary Distribution Content

| **Folder**              | **Description**                                                                            |
|-------------------------|--------------------------------------------------------------------------------------------|
| conf                    | Configuration files folder, contains configuration files of ConfigNode and DataNode        |
| data                    | Data files folder, contains data files of ConfigNode and DataNode                          |       |
| grafana-metrics-example | Grafana metric page module                                                                 |
| lib                     | Jar files folder                                                                           |
| licenses                | Licenses files folder                                                                      |
| logs                    | Logs files folder, contains logs files of ConfigNode and DataNode                          |
| sbin                    | Shell files folder, contains start/stop/remove shell of ConfigNode and DataNode, cli shell |
| tools                   | System tools                                                                               |

## Start the Cluster

Users could start a cluster which contains multiple ConfigNode and DataNode.
A cluster need at least one ConfigNode and no less than the number of data/schema_replication_factor DataNodes.

The total process are three steps:

* Start the first ConfigNode
* Add ConfigNode (Optional)
* Add DataNode

### Start the first ConfigNode

Please set the important parameters in conf/iotdb-confignode.properties and conf/iotdb-common.properties:

iotdb-confignode.properties:

| **Configuration**              | **Description**                                                                              |
|--------------------------------|----------------------------------------------------------------------------------------------|
| cn\_internal\_address          | Internal rpc service address of ConfigNode                                                   |
| cn\_internal\_port             | Internal rpc service port of ConfigNode                                                      |
| cn\_consensus\_port            | ConfigNode replication consensus protocol communication port                                 |
| cn\_target\_config\_node\_list | Target ConfigNode address, if the current ConfigNode is the first one, then set its own address:port |

iotdb-common.properties:

| **Configuration**                          | **Description**                                                                                      |
|--------------------------------------------|------------------------------------------------------------------------------------------------------|
| data\_replication\_factor                  | Data replication factor, no more than DataNode number                                                |
| data\_region\_consensus\_protocol\_class   | Consensus protocol of data replicas                                                                  |
| schema\_replication\_factor                | Schema replication factor, no more than DataNode number                                              |
| schema\_region\_consensus\_protocol\_class | Consensus protocol of schema replicas                                                                |

Start on Linux:
```
# Foreground
bash ./sbin/start-confignode.sh

# Background
nohup bash ./sbin/start-confignode.sh >/dev/null 2>&1 &
```

Start on Windows:
```
sbin\start-confignode.bat
```

More details  [ConfigNode Configurations](https://iotdb.apache.org/UserGuide/Master/Reference/ConfigNode-Config-Manual.html).

### Add ConfigNode (Optional)

This will add the replication factor of ConfigNode, except for the ports that couldn't conflict with, make sure other configurations are the same with existing ConfigNode in Cluster, and set parameter cn\_target\_config\_nodes\_list as an active ConfigNode in Cluster.

The adding ConfigNode also use the start-confignode.sh/bat.

### Start DataNode

You could add any number of DataNode.

Please set the important parameters in iotdb-datanode.properties:

| **Configuration**                   | **Description**                                  |
|-------------------------------------|--------------------------------------------------|
| dn\_rpc\_address                    | Client RPC Service address                       |
| dn\_rpc\_port                       | Client RPC Service port                          |
| dn\_internal\_address               | Control flow address of DataNode inside cluster  |
| dn\_internal\_port                  | Control flow port of DataNode inside cluster     |
| dn\_mpp\_data\_exchange\_port       | Data flow port of DataNode inside cluster        |
| dn\_data\_region\_consensus\_port   | Data replicas communication port for consensus   |
| dn\_schema\_region\_consensus\_port | Schema replicas communication port for consensus |
| dn\_target\_config\_node\_list      | Running ConfigNode of the Cluster                |

Start on Linux:
```
# Foreground
bash ./sbin/start-datanode.sh

# Background
nohup bash ./sbin/start-datanode.sh >/dev/null 2>&1 &
```

Start on Windows:
```
sbin\start-datanode.bat
```

More details are in [DataNode Configurations](https://iotdb.apache.org/UserGuide/Master/Reference/DataNode-Config-Manual.html).

### Stop IoTDB
When you meet problem, and want to stop IoTDB ConfigNode and DataNode directly, our shells can help you do this.

In Windows:

```
sbin\stop-datanode.bat
```
```
sbin\stop-confignode.bat
```
In Linux:
```
bash sbin/stop-datanode.sh
```
```
bash sbin/stop-confignode.sh
```
Be careful not to miss the "sudo" label, because some port info's acquisition may require root authority. If you can't sudo, just
use "jps" or "ps aux | grep iotdb" to get the process's id, then use "kill -9 <process-id>" to stop the process.  

## Start StandAlone
If you just want to setup your IoTDB locally, 
You can quickly init 1C1D (i.e. 1 Confignode and 1 Datanode) environment by our shells.

This will work well if you don't change our default settings.

Start on Windows:
```
sbin\start-standalone.bat
```
Start on Linux:
```
bash sbin/start-standalone.sh
```

Besides, with our shell, you can also directly kill these processes.

Stop on Windows:
```
sbin\stop-standalone.bat
```
Stop on Linux:
```
bash sbin/stop-standalone.sh
```

Note: On Linux, the 1C1D processes both launches in the background, and you can see the logs for details. 

The stop-standalone.sh may not work well without sudo, since IoTDB's port numbers may be invisible without permission. 
If stop-standalone.sh meets some error, you can use "jps" or "ps aux | grep iotdb" to obtain the process ids,
and use "sudo kill -9 <process-id>" to manually stop the processes.

## Start Cli

Cli shell is in sbin folder.

Start on Linux:
```
./datanode/sbin/start-cli.sh
```

Start on Windows:
```
datanode\sbin\start-cli.bat
```

## Shrink the Cluster

### Remove ConfigNode

Execute the remove-confignode shell on an active ConfigNode, and make sure that there is at least one active ConfigNode in Cluster after removing.

Remove on Linux:
```
# Remove the ConfigNode with confignode_id
./confignode/sbin/remove-confignode.sh <confignode_id>

# Remove the ConfigNode with address:port
./confignode/sbin/remove-confignode.sh <internal_address>:<internal_port>
```

Remove on Windows:
```
# Remove the ConfigNode with confignode_id
confignode\sbin\remove-confignode.bat <confignode_id>

# Remove the ConfigNode with address:port
confignode\sbin\remove-confignode.bat <internal_address>:<internal_port>
```

### Remove DataNode

Execute the remove-datanode shell on an active DataNode, and make sure that the number of active DataNodes are no less than the number of data/schema_replication_factor in Cluster after removing.

Remove on Linux:
```
# Remove the DataNode with datanode_id
bash ./datanode/sbin/remove-datanode.sh <datanode_id>

# Remove the DataNode with rpc address:port
bash ./datanode/sbin/remove-datanode.sh <rpc_address>:<rpc_port>
```

Remove on Windows:
```
# Remove the DataNode with datanode_id
datanode\sbin\remove-datanode.bat <datanode_id>

# Remove the DataNode with rpc address:port
datanode\sbin\remove-datanode.bat <rpc_address>:<rpc_port>
```

## Quick Start

This section uses a local environment as an example to 
illustrate how to start, expand, and shrink a IoTDB Cluster.

### 1. Prepare the Start Environment

Unzip the apache-iotdb-1.0.0-all-bin.zip file to cluster0 folder.

### 2. Start a Minimum Cluster

Start the Cluster version with one ConfigNode and one DataNode(1C1D), and
the default number of replicas is one.
```
./cluster0/sbin/start-confignode.sh
./cluster0/sbin/start-datanode.sh
```

### 3. Verify the Minimum Cluster

+ If everything goes well, the minimum cluster will start successfully. Then, we can start the Cli for verification.
```
./cluster0/sbin/start-cli.sh
```

+ Execute the [show cluster](https://iotdb.apache.org/UserGuide/Master/Maintenance-Tools/Maintenance-Command.html#show-all-node-information)
  command on the Cli. The result is shown below:
```
IoTDB> show cluster
+------+----------+-------+---------------+------------+
|NodeID|  NodeType| Status|InternalAddress|InternalPort|
+------+----------+-------+---------------+------------+
|     0|ConfigNode|Running|      127.0.0.1|       22277|
|     1|  DataNode|Running|      127.0.0.1|        9003|
+------+----------+-------+---------------+------------+
Total line number = 2
It costs 0.160s
```

### 4. Prepare the Expanding Environment

Unzip the apache-iotdb-1.0.0-all-bin.zip file to cluster1 and cluster2 folder.

### 5. Modify the Node Configuration file

For folder cluster1:

+ Modify ConfigNode configurations:

| **configuration item**         | **value**       |
|--------------------------------|-----------------|
| cn\_internal\_address          | 127.0.0.1       |
| cn\_internal\_port             | 22279           |
| cn\_consensus\_port            | 22280           |
| cn\_target\_config\_node\_list | 127.0.0.1:22277 |

+ Modify DataNode configurations:

| **configuration item**              | **value**       |
|-------------------------------------|-----------------|
| dn\_rpc\_address                    | 127.0.0.1       |
| dn\_rpc\_port                       | 6668            |
| dn\_internal\_address               | 127.0.0.1       |
| dn\_internal\_port                  | 9004            |
| dn\_mpp\_data\_exchange\_port       | 8778            |
| dn\_data\_region\_consensus\_port   | 40011           |
| dn\_schema\_region\_consensus\_port | 50011           |
| dn\_target\_config\_node\_list      | 127.0.0.1:22277 |

For folder cluster2:

+ Modify ConfigNode configurations:

| **configuration item**         | **value**       |
|--------------------------------|-----------------|
| cn\_internal\_address          | 127.0.0.1       |
| cn\_internal\_port             | 22281           |
| cn\_consensus\_port            | 22282           |
| cn\_target\_config\_node\_list | 127.0.0.1:22277 |

+ Modify DataNode configurations:

| **configuration item**              | **value**       |
|-------------------------------------|-----------------|
| dn\_rpc\_address                    | 127.0.0.1       |
| dn\_rpc\_port                       | 6669            |
| dn\_internal\_address               | 127.0.0.1       |
| dn\_internal\_port                  | 9005            |
| dn\_mpp\_data\_exchange\_port       | 8779            |
| dn\_data\_region\_consensus\_port   | 40012           |
| dn\_schema\_region\_consensus\_port | 50012           |
| dn\_target\_config\_node\_list      | 127.0.0.1:22277 |

### 6. Expanding the Cluster

Expanding the Cluster to three ConfigNode and three DataNode(3C3D).
The following commands can be executed in arbitrary order.

```
./cluster1/sbin/start-confignode.sh
./cluster1/sbin/start-datanode.sh
./cluster2/sbin/start-confignode.sh
./cluster2/sbin/start-datanode.sh
```

### 7. Verify Cluster expansion

Execute the show cluster command, then the result is shown below:
```
IoTDB> show cluster
+------+----------+-------+---------------+------------+
|NodeID|  NodeType| Status|InternalAddress|InternalPort|
+------+----------+-------+---------------+------------+
|     0|ConfigNode|Running|      127.0.0.1|       22277|
|     2|ConfigNode|Running|      127.0.0.1|       22279|
|     3|ConfigNode|Running|      127.0.0.1|       22281|
|     1|  DataNode|Running|      127.0.0.1|        9003|
|     4|  DataNode|Running|      127.0.0.1|        9004|
|     5|  DataNode|Running|      127.0.0.1|        9005|
+------+----------+-------+---------------+------------+
Total line number = 6
It costs 0.012s
```

### 8. Shrinking the Cluster

+ Remove a ConfigNode:
```
./cluster0/sbin/remove-confignode.sh 127.0.0.1:22279
```

+ Remove a DataNode:
```
./cluster0/sbin/remove-datanode.sh 127.0.0.1:6668
```

### 9. Verify Cluster shrinkage

Execute the show cluster command, then the result is shown below:
```
IoTDB> show cluster
+------+----------+-------+---------------+------------+
|NodeID|  NodeType| Status|InternalAddress|InternalPort|
+------+----------+-------+---------------+------------+
|     0|ConfigNode|Running|      127.0.0.1|       22277|
|     3|ConfigNode|Running|      127.0.0.1|       22281|
|     1|  DataNode|Running|      127.0.0.1|        9003|
|     5|  DataNode|Running|      127.0.0.1|        9005|
+------+----------+-------+---------------+------------+
Total line number = 4
It costs 0.007s
```
