////
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements.  See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
////
[[development-environment]]
= Development Environment

TinkerPop is fairly large body of code spread across many modules and covering multiple programming languages. Despite
this complexity, it remains relatively straightforward a project to build. This following subsections explain how to
configure a development environment for TinkerPop.

[[system-configuration]]
== System Configuration

At a minimum, development of TinkerPop requires link:http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html[Java 1.8.0_40+]
and link:https://maven.apache.org/download.cgi[Maven 3.2.5+]. Maven is used as the common build system, which even
controls the builds of non-JVM link:http://tinkerpop.apache.org/docs/current/tutorials/gremlin-language-variants/[GLVs]
such as `gremlin-python`. Java and Maven are described as a "minimum" for a development environment, because they
will only build JVM portions of TinkerPop and many integration tests will not fire with this simple setup. It is
possible to get a clean and successful build with this minimum, but it will not be possible to build non-JVM aspects
of the project and those will go untested.

To gain the ability to execute all aspects of the TinkerPop build system, other environmental configurations must be
established. Those prerequisites are defined in the following subsections.

IMPORTANT: For those who intend to offer a contribution, building with a minimal configuration may not be sufficient
when submitting a pull request. Consider setting up the full environment.

NOTE: For those using Windows, efforts have been made to keep the build OS independent, but, in practice, it is likely
that TinkerPop's build system will only allow for a minimum build at best.

[[documentation-environment]]
=== Documentation Environment

The documentation generation process is not Maven-based and uses shell scripts to process the project's asciidoc. The
scripts should work on Mac and Linux.

To generate documentation, it is required that link:https://hadoop.apache.org[Hadoop 2.7.x] is running in
link:https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SingleCluster.html#Pseudo-Distributed_Operation[pseudo-distributed]
mode. Be sure to set the `HADOOP_GREMLIN_LIBS` environment variable as described in the
link:http://tinkerpop.apache.org/docs/current/reference/#hadoop-gremlin[reference documentation]. It is also important
to set the `CLASSPATH` to point at the directory containing the Hadoop configuration files, like `mapred-site.xml`.

Also note that link:http://www.grymoire.com/Unix/Awk.html[awk] version `4.0.1` is required for documentation generation.

Documentation can be generated locally with:

[source,text]
bin/process-docs.sh

Documentation is generated to the `target/docs` directory. It is also possible to generate documentation locally with
Docker. `docker/build.sh -d`.

To generate the web site locally, there is no need for any of the above infrastructure. Site generation is a simple
shell script:

[source,text]
bin/generate-home.sh

The site will be generated to the `target/site/home` directory.

[[python-environment]]
=== Python Environment

As of TinkerPop 3.2.2, the build optionally requires link:https://www.python.org/[Python 2.x] to work with the
`gremlin-python` module. If Python is not installed, TinkerPop will still build with Maven, but native Python tests and
Java tests that require Python code will be skipped. Developers should also install link:https://pypi.python.org/pypi/pip[pip]
and link:https://virtualenv.pypa.io/en/stable/[virtualenv] (version 15.0.2 - older versions may cause build failures).

Once the Python environment is established, the full building and testing of `gremlin-python` may commence. It can be
done manually from the command line with:

[source,text]
mvn clean install -Pglv-python

which enables the "glv-python" Maven profile or in a more automated fashion simply add a `.glv` file to the root of the
`gremlin-python` module which will signify to Maven that the environment is Python-ready. The `.glv` file need not have
any contents and is ignored by Git. A standard `mvn clean install` will then build `gremlin-python` in full.

As of TinkerPop 3.2.5, the build also requires Python to execute `gremlin-console` integration tests. The integration
test is configured by a "console-integration-tests" Maven profile. This profile can be activated manually or can more
simply piggy-back on the `.glv` file in `gremlin-python`. Note that unlike `gremlin-python` the tests are actually
integration tests and therefore must be actively switched on with `-DskipIntegrationTests=false`:

[source,text]
mvn clean install -pl gremlin-console -DskipIntegrationTests=false

[[dotnet-environment]]
=== DotNet Environment

The build optionally requires link:https://www.microsoft.com/net/core[.NET Core SDK] (>=2.1.101) to work with the
`gremlin-dotnet` module. If .NET Core SDK is not installed, TinkerPop will still build with Maven, but .NET projects
will be skipped.

`gremlin-dotnet` can be build and tested from the command line with:

[source,text]
mvn clean install -Pgremlin-dotnet

which enables the "gremlin-dotnet" Maven profile or in a more automated fashion simply add a `.glv` file to the `src`
and `test` directories of the `gremlin-dotnet` module  which will signify to Maven that the environment is .NET-ready.
The `.glv` file need not have any contents and is ignored by Git. A standard `mvn clean install` will then build
`gremlin-dotnet` in full.

For those who will release TinkerPop, it is also necessary to install link:http://www.mono-project.com/[Mono]. The
release process is known to work with 5.0.1, so it is best to probably install that version. Release managers should
probably also do an install of link:https://dist.nuget.org/win-x86-commandline/v3.4.4/nuget.exe[nuget 3.4.4] as it
will help with environmental setup. To get an environment ready to deploy to NuGet, it is necessary to have a
NuGet API key (PMC members who have NuGet accounts can help with that). The API key should be added to `NuGet.Config`
with the following:

[source,text]
----
mono nuget.exe setApiKey [your-api-key]
----

This should update `~/.config/NuGet/NuGet.Config` a file with an entry containing the encrypted API key. On
`mvn deploy`, this file will be referenced on the automated `nuget push`.

See release documentation for more information on configuration for release.


[[nodejs-environment]]
=== JavaScript Environment
When building `gremlin-javascript`, mvn command will include a local copy of Node.js runtime and npm inside your project
using `com.github.eirslett:frontend-maven-plugin` plugin. This copy of the Node.js runtime will not affect any
other existing Node.js runtime instances in your machine.

The release manager should have the authentication token set, for more information see the Release Environment
section below.


[[release-environment]]
=== Release Environment

This section is only useful to TinkerPop release managers and describes prerequisites related to deploying an official
release of TinkerPop. All Apache releases must be signed. Please see link:http://www.apache.org/dev/release-signing.html[this guide]
in the Apache documentation for instructions on to set up gpg. Keys should be added to KEYS files in both the
link:https://dist.apache.org/repos/dist/dev/tinkerpop/KEYS[development] and
link:https://dist.apache.org/repos/dist/release/tinkerpop/KEYS[release] distribution directories and committed
using Apache Subversion (SVN).

Uploading to pypi uses link:https://pypi.python.org/pypi/twine[twine] which is automatically installed by the build
process in maven. Twine refers to `HOME/.pypirc` file for configuration on the pypi deploy environments and username
and password combinations. The file typically looks like this:

[source,text]
----
[distutils]
index-servers=
    pypi
    pypitest

[pypitest]
repository = https://testpypi.python.org/pypi
username = <username>
password =

[pypi]
repository = https://pypi.python.org/pypi
username = <username>
password =
----

The release manager shall use the project's pypi credentials, which are available in the PMC repository.
The `password` should be left blank so the deployment process in Maven will prompt for it at deployment time.

To deploy `gremlin-javascript` on the link:https://www.npmjs.com[npm registry], the release manager must set the
authentication information on the ~/.npmrc file. The easiest way to do that is to use the `npm adduser` command. This
must be done only once, as the auth token doesn't have an expiration date and it's stored on your file system.

[[building-testing]]
== Building and Testing

The following commands are a mix of Maven flags and shell scripts that handle different build operations

* Build project: `mvn clean install`
** Build a specific module (e.g. `gremlin-server`) within the project: `mvn clean install -pl gremlin-server`
** Specify specific tests in a TinkerPop Suite to run with the `GREMLIN_TESTS` environment variable, along with the
Maven project list argument, e.g.:
+
----
export GREMLIN_TESTS='org.apache.tinkerpop.gremlin.process.traversal.step.map.PathTest$Traversals,org.apache.tinkerpop.gremlin.process.traversal.PathTest'
mvn -Dmaven.javadoc.skip=true --projects tinkergraph-gremlin test
----
** Clean the `.groovy/grapes/org.apache.tinkerpop` directory on build: `mvn clean install -DcleanGrapes`
** Turn off "heavy" logging in the "process" tests: `mvn clean install -DargLine="-DmuteTestLogs=true"`
** The test suite for `neo4j-gremlin` is disabled by default - to turn it on: `mvn clean install -DincludeNeo4j`
* Regenerate test data (only necessary given changes to IO classes): `mvn clean install -Dio` from `tinkergraph-gremlin` directory
** If there are changes to the Gryo format, it may be necessary to generate the Grateful Dead dataset from GraphSON (see `IoDataGenerationTest.shouldWriteGratefulDead`)
* Check license headers are present: `mvn apache-rat:check`
* Build AsciiDocs (see <<documentation-environment,Documentation Environment>>): `bin/process-docs.sh`
** Build AsciiDocs (but don't evaluate code blocks): `bin/process-docs.sh --dryRun`
** Build AsciiDocs (but don't evaluate code blocks in specific files): `bin/process-docs.sh --dryRun docs/src/reference/the-graph.asciidoc,docs/src/tutorial/getting-started,...`
** Build AsciiDocs (but evaluate code blocks only in specific files): `bin/process-docs.sh --fullRun docs/src/reference/the-graph.asciidoc,docs/src/tutorial/getting-started,...`
** Process a single AsciiDoc file: +pass:[docs/preprocessor/preprocess-file.sh `pwd`/gremlin-console/target/apache-tinkerpop-gremlin-console-*-standalone "" "*" `pwd`/docs/src/xyz.asciidoc]+
* Build JavaDocs: `mvn process-resources -Djavadoc`
* Check for Apache License headers: `mvn apache-rat:check`
* Check for newer dependencies: `mvn versions:display-dependency-updates` or `mvn versions:display-plugin-updates`
* Deploy JavaDocs/AsciiDocs: `bin/publish-docs.sh svn-username`
* Integration Tests: `mvn verify -DskipIntegrationTests=false`
** Execute with the `-DincludeNeo4j` option to include transactional tests.
** Execute with the `-DuseEpoll` option to try to use Netty native transport (works on Linux, but will fallback to Java NIO on other OS).
* Benchmarks: `mvn verify -DskipBenchmarks=false`
** Reports are generated to the console and to `gremlin-tools/gremlin-benchmark/target/reports/benchmark`.
* Test coverage report: `mvn clean install -Dcoverage` - note that the `install` is necessary because report aggregation is bound to that part of the lifecycle.
** Reports are generated to `gremlin-tools/gremlin-coverage/target/site`.
* `cd site`
** Generate web site locally: `bin/generate-home.sh`
** Publish web site: `bin/publish-home.sh <username>`

[[docker-integration]]
== Docker Integration

TinkerPop provides a shell script, that can start several build tasks within a Docker container. The
required Docker images will be built automatically if they don't exist yet. Thus the first invocation
of the Docker script is expected to take some time.

The script can be found under `PROJECT_HOME/docker/build.sh`. The following tasks are currently
supported:

* run standard test suite
* run integration tests
* build Java docs
* build user docs

A list of command line options is provided by `docker/build.sh --help`. The container will install,
configure and start all required dependencies, such as Hadoop.

Options can be passed to Docker by setting the `TINKERPOP_DOCKER_OPTS` environment variable. A speed boost can
be gained at the expense of memory by using tmpfs and the special directory `/usr/src/tinkermem`.

[source,bash]
.Build in-memory
----
TINKERPOP_DOCKER_OPTS="--tmpfs /usr/src/tinkermem:exec,mode=0755,rw,noatime,size=2000m"
----

[source,bash]
.Disable IPv6 for Hadoop
----
TINKERPOP_DOCKER_OPTS="--sysctl net.ipv6.conf.all.disable_ipv6=1 --sysctl net.ipv6.conf.default.disable_ipv6=1"
----

A custom maven settings.xml can be supplied, for example, to point to a local proxy. Copy the `settings.xml` to the
`PROJECT_HOME/` directory. The Docker script will detect and copy it to the running container.

If the container is used to generate the user docs, it will start a web server and show the URL that
is used to host the HTML docs.

After finishing all tasks, the script will immediately destroy the container.


== IDE Setup with Intellij

This section refers specifically to setup within Intellij.  TinkerPop has a module called `gremlin-shaded` which
contains shaded dependencies for some libraries that are widely used and tend to introduce conflicts.  To ensure
that Intellij properly interprets this module after importing the Maven `pom.xml` perform the following steps:

. Build `gremlin-shaded` from the command line with `mvn clean install`.
. Right-click on the `gremlin-shaded` module in the project viewer of Intellij and select "Remove module". If this menu
option is not available (as is the case in newer versions of Intellij - first noticed in 13.1.5), then open the "Maven
Projects" side panel, right click the `gremlin-shaded` module and select "Ignore Project".
. In the "Maven Projects" Tool window and click the tool button for "Reimport All Maven projects" (go to
`View | Tool Windows | Maven Projects` on the main menu if this panel is not activated).
. At this point it should be possible to compile and run the tests within Intellij, but in the worst case, use
`File | Invalidate Caches/Restart` to ensure that indices properly rebuild.

Note that it may be necessary to re-execute these steps if the `gremlin-shaded` `pom.xml` is ever updated.

Developers working on the `neo4j-gremlin` module should enabled the `include-neo4j` Maven profile in Intellij.
This will ensure that tests will properly execute within the IDE.

If Intellij complains about "duplicate sources" for the Groovy files when attempting to compile/run tests, then
install the link:http://plugins.jetbrains.com/plugin/7442?pr=idea[GMavenPlus Intellij plugin].

The `gremlin-core` module uses a Java annotation processor to help support DSLs. To support this capability be sure
that:

. `File | Settings | Compiler | Annotation Processors` has the checkbox with the "Enable annotation processing" checked.
Intellij should be able to detect the processor automatically on build.
. The `gremlin-core/target` directory should not be hidden and `target/classes`, `target/generated-sources` and
`target/generated-test-sources should be marked as "Generated Sources Root". If they are not setup that way by
Intellij by default then simply right-click on them use the "Mark Directory with" option to make the appropriate
selections.
