<!--
 ! Licensed to the Apache Software Foundation (ASF) under one
 ! or more contributor license agreements.  See the NOTICE file
 ! distributed with this work for additional information
 ! regarding copyright ownership.  The ASF licenses this file
 ! to you under the Apache License, Version 2.0 (the
 ! "License"); you may not use this file except in compliance
 ! with the License.  You may obtain a copy of the License at
 !
 !   http://www.apache.org/licenses/LICENSE-2.0
 !
 ! Unless required by applicable law or agreed to in writing,
 ! software distributed under the License is distributed on an
 ! "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 ! KIND, either express or implied.  See the License for the
 ! specific language governing permissions and limitations
 ! under the License.
 !-->

## <a id="Lifecycle_management_statements">Lifecycle Management Statements</a>

    CreateStatement ::= "CREATE" ( DatabaseSpecification
                                 | TypeSpecification
                                 | DatasetSpecification
                                 | IndexSpecification
                                 | SynonymSpecification
                                 | FunctionSpecification )

    QualifiedName       ::= Identifier ( "." Identifier )?
    DoubleQualifiedName ::= Identifier "." Identifier ( "." Identifier )?

The CREATE statement is used for creating dataverses as well as other persistent artifacts in a dataverse.
It can be used to create new dataverses, datatypes, datasets, indexes, and user-defined query functions.

### <a id="Dataverses"> Dataverses</a>

    DatabaseSpecification ::= "DATAVERSE" Identifier IfNotExists

The CREATE DATAVERSE statement is used to create new dataverses.
To ease the authoring of reusable query scripts, an optional IF NOT EXISTS clause is included to allow
creation to be requested either unconditionally or only if the dataverse does not already exist.
If this clause is absent, an error is returned if a dataverse with the indicated name already exists.

The following example creates a new dataverse named TinySocial if one does not already exist.

##### Example

    CREATE DATAVERSE TinySocial IF NOT EXISTS;

### <a id="Types"> Types</a>

    TypeSpecification    ::= "TYPE" FunctionOrTypeName IfNotExists "AS" ObjectTypeDef
    FunctionOrTypeName   ::= QualifiedName
    IfNotExists          ::= ( <IF> <NOT> <EXISTS> )?
    TypeExpr             ::= ObjectTypeDef | TypeReference | ArrayTypeDef | MultisetTypeDef
    ObjectTypeDef        ::= ( <CLOSED> | <OPEN> )? "{" ( ObjectField ( "," ObjectField )* )? "}"
    ObjectField          ::= Identifier ":" ( TypeExpr ) ( "?" )?
    NestedField          ::= Identifier ( "." Identifier )*
    IndexField           ::= NestedField ( ":" TypeReference )?
    TypeReference        ::= Identifier
    ArrayTypeDef         ::= "[" ( TypeExpr ) "]"
    MultisetTypeDef      ::= "{{" ( TypeExpr ) "}}"

The CREATE TYPE statement is used to create a new named datatype.
This type can then be used to create stored collections or utilized when defining one or more other datatypes.
Much more information about the data model is available in the [data model reference guide](../datamodel.html).
A new type can be a object type, a renaming of another type, an array type, or a multiset type.
A object type can be defined as being either open or closed.
Instances of a closed object type are not permitted to contain fields other than those specified in the create type statement.
Instances of an open object type may carry additional fields, and open is the default for new types if neither option is specified.

The following example creates a new object type called GleambookUser type.
Since it is defined as (defaulting to) being an open type,
instances will be permitted to contain more than what is specified in the type definition.
The first four fields are essentially traditional typed name/value pairs (much like SQL fields).
The friendIds field is a multiset of integers.
The employment field is an array of instances of another named object type, EmploymentType.

##### Example

    CREATE TYPE GleambookUserType AS {
      id:         int,
      alias:      string,
      name:       string,
      userSince: datetime,
      friendIds: {{ int }},
      employment: [ EmploymentType ]
    };

The next example creates a new object type, closed this time, called MyUserTupleType.
Instances of this closed type will not be permitted to have extra fields,
although the alias field is marked as optional and may thus be NULL or MISSING in legal instances of the type.
Note that the type of the id field in the example is UUID.
This field type can be used if you want to have this field be an autogenerated-PK field.
(Refer to the Datasets section later for more details on such fields.)

##### Example

    CREATE TYPE MyUserTupleType AS CLOSED {
      id:         uuid,
      alias:      string?,
      name:       string
    };

### <a id="Datasets"> Datasets</a>

    DatasetSpecification ::= ( <INTERNAL> )? <DATASET> QualifiedName "(" QualifiedName ")" IfNotExists
                               PrimaryKey ( <ON> Identifier )? ( <HINTS> Properties )?
                               ( "USING" "COMPACTION" "POLICY" CompactionPolicy ( Configuration )? )?
                               ( <WITH> <FILTER> <ON> Identifier )?
                              |
                               <EXTERNAL> <DATASET> QualifiedName "(" QualifiedName ")" IfNotExists <USING> AdapterName
                               Configuration ( <HINTS> Properties )?
                               ( <USING> <COMPACTION> <POLICY> CompactionPolicy ( Configuration )? )?
    AdapterName          ::= Identifier
    Configuration        ::= "(" ( KeyValuePair ( "," KeyValuePair )* )? ")"
    KeyValuePair         ::= "(" StringLiteral "=" StringLiteral ")"
    Properties           ::= ( "(" Property ( "," Property )* ")" )?
    Property             ::= Identifier "=" ( StringLiteral | IntegerLiteral )
    FunctionSignature    ::= FunctionOrTypeName "@" IntegerLiteral
    PrimaryKey           ::= <PRIMARY> <KEY> NestedField ( "," NestedField )* ( <AUTOGENERATED> )?
    CompactionPolicy     ::= Identifier

The CREATE DATASET statement is used to create a new dataset.
Datasets are named, multisets of object type instances;
they are where data lives persistently and are the usual targets for queries.
Datasets are typed, and the system ensures that their contents conform to their type definitions.
An Internal dataset (the default kind) is a dataset whose content lives within and is managed by the system.
It is required to have a specified unique primary key field which uniquely identifies the contained objects.
(The primary key is also used in secondary indexes to identify the indexed primary data objects.)

Internal datasets contain several advanced options that can be specified when appropriate.
One such option is that random primary key (UUID) values can be auto-generated by declaring the field to be UUID and putting "AUTOGENERATED" after the "PRIMARY KEY" identifier.
In this case, unlike other non-optional fields, a value for the auto-generated PK field should not be provided at insertion time by the user since each object's primary key field value will be auto-generated by the system.

Another advanced option, when creating an Internal dataset, is to specify the merge policy to control which of the
underlying LSM storage components to be merged.
(The system supports Log-Structured Merge tree based physical storage for Internal datasets.)
Currently the system supports four different component merging policies that can be chosen per dataset:
no-merge, constant, prefix, and correlated-prefix.
The no-merge policy simply never merges disk components.
The constant policy merges disk components when the number of components reaches a constant number k that can be configured by the user.
The prefix policy relies on both component sizes and the number of components to decide which components to merge.
It works by first trying to identify the smallest ordered (oldest to newest) sequence of components such that the sequence does not contain a single component that exceeds some threshold size M and that either the sum of the component's sizes exceeds M or the number of components in the sequence exceeds another threshold C.
If such a sequence exists, the components in the sequence are merged together to form a single component.
Finally, the correlated-prefix policy is similar to the prefix policy, but it delegates the decision of merging the disk components of all the indexes in a dataset to the primary index.
When the correlated-prefix policy decides that the primary index needs to be merged (using the same decision criteria as for the prefix policy), then it will issue successive merge requests on behalf of all other indexes associated with the same dataset.
The system's default policy is the prefix policy except when there is a filter on a dataset, where the preferred policy for filters is the correlated-prefix.

Another advanced option shown in the syntax above, related to performance and mentioned above, is that a **filter** can optionally be created on a field to further optimize range queries with predicates on the filter's field.
Filters allow some range queries to avoid searching all LSM components when the query conditions match the filter.
(Refer to [Filter-Based LSM Index Acceleration](../filters.html) for more information about filters.)

An External dataset, in contrast to an Internal dataset, has data stored outside of the system's control.
Files living in HDFS or in the local filesystem(s) of a cluster's nodes are currently supported.
External dataset support allows queries to treat foreign data as though it were stored in the system,
making it possible to query "legacy" file data (for example, Hive data) without having to physically import it.
When defining an External dataset, an appropriate adapter type must be selected for the desired external data.
(See the [Guide to External Data](../externaldata.html) for more information on the available adapters.)

The following example creates an Internal dataset for storing FacefookUserType objects.
It specifies that their id field is their primary key.

#### Example

    CREATE INTERNAL DATASET GleambookUsers(GleambookUserType) PRIMARY KEY id;

The next example creates another Internal dataset (the default kind when no dataset kind is specified) for storing MyUserTupleType objects.
It specifies that the id field should be used as the primary key for the dataset.
It also specifies that the id field is an auto-generated field,
meaning that a randomly generated UUID value should be assigned to each incoming object by the system.
(A user should therefore not attempt to provide a value for this field.)
Note that the id field's declared type must be UUID in this case.

#### Example

    CREATE DATASET MyUsers(MyUserTupleType) PRIMARY KEY id AUTOGENERATED;

The next example creates an External dataset for querying LineItemType objects.
The choice of the `hdfs` adapter means that this dataset's data actually resides in HDFS.
The example CREATE statement also provides parameters used by the hdfs adapter:
the URL and path needed to locate the data in HDFS and a description of the data format.

#### Example

    CREATE EXTERNAL DATASET LineItem(LineItemType) USING hdfs (
      ("hdfs"="hdfs://HOST:PORT"),
      ("path"="HDFS_PATH"),
      ("input-format"="text-input-format"),
      ("format"="delimited-text"),
      ("delimiter"="|"));


### <a id="Indices">Indices</a>

    IndexSpecification ::= (<INDEX> Identifier IfNotExists <ON> QualifiedName
                           "(" ( IndexField ) ( "," IndexField )* ")" (<TYPE> IndexType)? (<ENFORCED>)?)
                           |
                           <PRIMARY> <INDEX> Identifier? IfNotExists <ON> QualifiedName (<TYPE> <BTREE>)?
    IndexType          ::= <BTREE> | <RTREE> | <KEYWORD> | <NGRAM> "(" IntegerLiteral ")"

The CREATE INDEX statement creates a secondary index on one or more fields of a specified dataset.
Supported index types include `BTREE` for totally ordered datatypes, `RTREE` for spatial data,
and `KEYWORD` and `NGRAM` for textual (string) data.
An index can be created on a nested field (or fields) by providing a valid path expression as an index field identifier.

An indexed field is not required to be part of the datatype associated with a dataset if the dataset's datatype
is declared as open **and** if the field's type is provided along with its name and if the `ENFORCED` keyword is
specified at the end of the index definition.
`ENFORCING` an open field introduces a check that makes sure that the actual type of the indexed field
(if the optional field exists in the object) always matches this specified (open) field type.

The following example creates a btree index called gbAuthorIdx on the authorId field of the GleambookMessages dataset.
This index can be useful for accelerating exact-match queries, range search queries, and joins involving the author-id
field.

#### Example

    CREATE INDEX gbAuthorIdx ON GleambookMessages(authorId) TYPE BTREE;

The following example creates an open btree index called gbSendTimeIdx on the (non-declared) `sendTime` field of the GleambookMessages dataset having datetime type.
This index can be useful for accelerating exact-match queries, range search queries, and joins involving the `sendTime` field.
The index is enforced so that records that do not have the `sendTime` field or have a mismatched type on the field
cannot be inserted into the dataset.

#### Example

    CREATE INDEX gbSendTimeIdx ON GleambookMessages(sendTime: datetime?) TYPE BTREE ENFORCED;

The following example creates an open btree index called gbReadTimeIdx on the (non-declared) `readTime`
field of the GleambookMessages dataset having datetime type.
This index can be useful for accelerating exact-match queries, range search queries,
and joins involving the `readTime` field.
The index is not enforced so that records that do not have the `readTime` field or have a mismatched type on the field
can still be inserted into the dataset.

#### Example

    CREATE INDEX gbReadTimeIdx ON GleambookMessages(readTime: datetime?);

The following example creates a btree index called crpUserScrNameIdx on screenName,
a nested field residing within a object-valued user field in the ChirpMessages dataset.
This index can be useful for accelerating exact-match queries, range search queries,
and joins involving the nested screenName field.
Such nested fields must be singular, i.e., one cannot index through (or on) an array-valued field.

#### Example

    CREATE INDEX crpUserScrNameIdx ON ChirpMessages(user.screenName) TYPE BTREE;

The following example creates an rtree index called gbSenderLocIdx on the sender-location field of the GleambookMessages dataset. This index can be useful for accelerating queries that use the [`spatial-intersect` function](functions.html#spatial-intersect) in a predicate involving the sender-location field.

#### Example

    CREATE INDEX gbSenderLocIndex ON GleambookMessages("sender-location") TYPE RTREE;

The following example creates a 3-gram index called fbUserIdx on the name field of the GleambookUsers dataset. This index can be used to accelerate some similarity or substring maching queries on the name field. For details refer to the document on [similarity queries](similarity.html#NGram_Index).

#### Example

    CREATE INDEX fbUserIdx ON GleambookUsers(name) TYPE NGRAM(3);

The following example creates a keyword index called fbMessageIdx on the message field of the GleambookMessages dataset. This keyword index can be used to optimize queries with token-based similarity predicates on the message field. For details refer to the document on [similarity queries](similarity.html#Keyword_Index).

#### Example

    CREATE INDEX fbMessageIdx ON GleambookMessages(message) TYPE KEYWORD;

The following example creates a special secondary index which holds only the primary keys.
This index is useful for speeding up aggregation queries which involve only primary keys.
The name of the index is optional. If the name is not specified, the system will generate
one. When the user would like to drop this index, the metadata can be queried to find the system-generated name.

#### Example

    CREATE PRIMARY INDEX gb_pk_idx ON GleambookMessages;

An example query that can be accelerated using the primary-key index:

    SELECT COUNT(*) FROM GleambookMessages;

To look up the the above primary-key index, issue the following query:

    SELECT VALUE i
    FROM Metadata.`Index` i
    WHERE i.DataverseName = "TinySocial" AND i.DatasetName = "GleambookMessages";

The query returns:

    [ { "DataverseName": "TinySocial", "DatasetName": "GleambookMessages", "IndexName": "GleambookMessages", "IndexStructure": "BTREE", "SearchKey": [ [ "messageId" ] ], "IsPrimary": true, "Timestamp": "Wed Nov 07 17:25:11 PST 2018", "PendingOp": 0 }
    , { "DataverseName": "TinySocial", "DatasetName": "GleambookMessages", "IndexName": "gb_pk_idx", "IndexStructure": "BTREE", "SearchKey": [  ], "IsPrimary": false, "Timestamp": "Wed Nov 07 17:25:11 PST 2018", "PendingOp": 0 }
     ]

Remember that `CREATE PRIMARY INDEX` creates a secondary index.
That is the reason the `IsPrimary` field is false.
The primary-key index can be identified by the fact that the `SearchKey` field is empty since it only contains primary key fields.