// Copyright 2021 the original author or authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

[[jvm_test_suite_plugin]]
= The JVM Test Suite Plugin

The JVM Test Suite plugin (plugin id: `jvm-test-suite`) provides a DSL and API to model multiple groups of automated tests into test suites in JVM-based projects.  Tests suites are intended to grouped by their purpose and can have separate dependencies and use different testing frameworks.

For instance, this plugin can be used to define a group of Integration Tests, which might run much longer than unit tests and have different environmental requirements.
[[sec:jvm_test_suite_usage]]
== Usage

This plugin is applied automatically by the `java` plugin but can be additionally applied explicitly if desired. The plugin cannot be used without a JVM language plugin applied as it relies on several conventions of the `java` plugin.

.Applying the JVM Test Suite plugin
====
include::sample[dir="snippets/testing/test-suite-plugin/kotlin",files="build.gradle.kts[tags=apply-jvm-test-suite]"]
include::sample[dir="snippets/testing/test-suite-plugin/groovy",files="build.gradle[tags=apply-jvm-test-suite]"]
====

The plugins adds the following objects to the project:

 * A `testing` extension (type: link:{groovyDslPath}/org.gradle.testing.base.TestingExtension.html[TestingExtension]) to the project used to configure test suites.

When used with the <<java_plugin#java_plugin,Java Plugin>>:

 * A test suite named `test` (type: link:{groovyDslPath}/org.gradle.api.plugins.jvm.JvmTestSuite.html[JvmTestSuite]).
 * A `test` link:{groovyDslPath}/org.gradle.api.tasks.SourceSet.html[SourceSet].
 * Several configurations derived from the `test` SourceSet name: `testImplementation`, `testCompileOnly`, `testRuntimeOnly`
 * A single test suite target backed by a task named `test`.

The `test` task, SourceSet and derived configurations are identical in name and function to those used in prior Gradle releases.

[[sec:jvm_test_suite_tasks]]
== Tasks

The JVM Test Suite plugin adds the following task to the project:

`test` — link:{groovyDslPath}/org.gradle.api.tasks.testing.Test.html[Test]::
_Depends on_: `testClasses` from the `java` plugin, and all tasks which produce the test runtime classpath
+
Runs the tests using the framework configured for the default test suite.

Additional instances of link:{groovyDslPath}/org.gradle.api.tasks.testing.Test.html[Test] tasks will be automatically created for each test suite added via the `testing` extension.

[[sec:jvm_test_suite_configuration]]
== Configuration

See the link:{groovyDslPath}/org.gradle.testing.base.TestingExtension.html[TestingExtension] class in the API documentation.

[[sec:jvm_test_suite_terminology]]
=== Terminology and Modeling
The JVM Test Suite Plugin introduces some modeling concepts backed by new APIs.  Here are their definitions.

==== Test Suite ====
A test suite is a collection of JVM-based tests.

==== Test Suite Target ====
For the initial release of this plugin, each test suite has a single target.  This results in a 1:1:1 relationship between test suite, test suite target and a matching link:{groovyDslPath}/org.gradle.api.tasks.testing.Test.html[Test] task.  The name of the `Test` task is derived from the suite name.  Future iterations of the plugin will allow defining multiple targets based other attributes, such as a particular JDK/JRE runtime.

Each test suite has some configuration that is common across for all tests contained in the suite:

* Testing framework
* Sources
* Dependencies

In the future, other properties may be specified in the test suite which may influence the toolchains selected to compile and run tests.

The `Test` task associated with the target inherits its name from the suite.  Other properties of the `Test` task are configurable.

==== Test Suite Type ====
Each test suite must be assigned a type.
Types can be used to group related test suites across multiple Gradle projects within a build.

The type of a test suite can be configured using the suites's link:{groovyDslPath}/org.gradle.api.plugins.jvm.JvmTestSuite.html#getTestType[test type property].
The type *must be unique* across all test suites in the same Gradle project.
By convention, the type is set to the name of the test suite, converted to dash-case - with the exception of the built-in test suite, which uses the value link:{javadocPath}/org/gradle/api/attributes/TestSuiteType.html#UNIT_TEST--['unit-test'].

Common values are available as constants in link:{javadocPath}/org/gradle/api/attributes/TestSuiteType.html[TestSuiteType].

[[sec:jvm_test_suite_examples]]
[discrete]
== Configuration Examples

Here are several examples to illustrate the configurability of test suites.

[[sec:declare_an_additional_test_suite]]
== Declare an additional test suite
====
include::sample[dir="snippets/testing/test-suite-plugin/kotlin",files="build.gradle.kts[tags=configure-testing-extension]"]
include::sample[dir="snippets/testing/test-suite-plugin/groovy",files="build.gradle[tags=configure-testing-extension]"]
====

<1> Configure all test suites for this project.
<2> Configure the built-in `test` suite.  This suite is automatically created for backwards compatibility.  You *must* specify a testing framework to use in order to run these tests (e.g. JUnit 4, JUnit Jupiter).  This suite is the only suite that will automatically have access to the production source's `implementation` dependencies, all other suites must explicitly declare these.
<3> Declare this test suite uses JUnit Jupiter.  The framework's dependencies are automatically included.  It is always necessary to explicitly configure the built-in `test` suite even if JUnit4 is desired.
<4> Define a new suite called `integrationTest`.  Note that all suites other than the built-in `test` suite will by convention work as if `useJUnitJupiter()` was called.  You do *not* have to explicitly configure the testing framework on these additional suites, unless you wish to change to another framework.
<5> Add a dependency on the production code of the project to the `integrationTest` suite targets.  By convention, only the built-in `test` suite will automatically have a dependency on the production code of the project.
<6> Configure all targets of this suite.  By convention, test suite targets have no relationship to other `Test` tasks.  This example shows that the slower integration test suite targets should run after all targets of the `test` suite are complete.
<7> Configure the `check` task to depend on all `integrationTest` targets.  By convention, test suite targets are not associated with the `check` task.

Invoking the `check` task on the above configured build should show output similar to:

[listing]
----
include::{snippetsPath}/testing/test-suite-plugin/tests/checkTaskOutput.out[]
----

Note that the `integrationTest` test suite does not run until after the `test` test suite completes.

[[sec:configuring_the_built_in_test_suite]]
== Configure the built-in `test` suite
====
include::sample[dir="snippets/testing/test-suite-configure-default-suite/kotlin",files="build.gradle.kts[tags=configure-default-suite]"]
include::sample[dir="snippets/testing/test-suite-configure-default-suite/groovy",files="build.gradle[tags=configure-default-suite]"]
====

<1> Declare the `test` test suite uses the TestNG test framework.
<2> Lazily configure the test task of all targets of the suite; note the return type of `testTask` is `TaskProvider<Test>`.
<3> Configure more detailed test framework options.  The `options` will be a subclass of `org.gradle.api.tasks.testing.TestFrameworkOptions`, in this case it is `org.gradle.api.tasks.testing.testng.TestNGOptions`.


== Configure dependencies of a test suite
====
include::sample[dir="snippets/testing/test-suite-configure-suite-dependencies/kotlin",files="build.gradle.kts[tags=configure-suite-dependencies]"]
include::sample[dir="snippets/testing/test-suite-configure-suite-dependencies/groovy",files="build.gradle[tags=configure-suite-dependencies]"]
====

<1> Configure the built-in `test` test suite.
<2> Add the assertj library to the test's compile and runtime classpaths.  The `dependencies` block within a test suite is already scoped for that test suite.  Instead of having to know the global name of the configuration, test suites have a consistent name that you use in this block for declaring `implementation`, `compileOnly`, `runtimeOnly` and `annotationProcessor` dependencies.
<3> Add the Auto Value annotation processor to the suite's annotation processor classpath so that it will be run when compiling the tests.

== Configure dependencies of a test suite to reference project outputs
====
include::sample[dir="snippets/testing/test-suite-configure-suite-dependencies-project/kotlin",files="build.gradle.kts[tags=configure-suite-dependencies-project]"]
include::sample[dir="snippets/testing/test-suite-configure-suite-dependencies-project/groovy",files="build.gradle[tags=configure-suite-dependencies-project]"]
====

<1> Add a production dependency which is used as part of the library's public API.
<2> Add a production dependency which is only used internally and is not exposed as part of this project's public classes.
<3> Configure a new test suite which adds a `project()` dependency to the suite's compile and runtime classpaths. This dependency provides access to the project's outputs as well as any dependencies declared on its `api` and `compileOnlyApi` configurations.

== Configure source directories of a test suite
====
include::sample[dir="snippets/testing/test-suite-configure-source-dir/kotlin",files="build.gradle.kts[tags=configure-source-dir]"]
include::sample[dir="snippets/testing/test-suite-configure-source-dir/groovy",files="build.gradle[tags=configure-source-dir]"]
====

<1> Declare and configure a suite named `integrationTest`.  The `SourceSet` and synthesized `Test` tasks will be based on this name.
<2> Configure the `sources` of the test suite.
<3> Configure the `java` SourceDirectorySet of the test suite.
<4> Overwrite the `srcDirs` property, replacing the conventional `src/integrationTest/java` location with `src/it/java`.

== Configure the type of a test suite

====
include::sample[dir="snippets/testing/test-suite-configure-type/kotlin",files="build.gradle.kts[tags=configure-type]"]
include::sample[dir="snippets/testing/test-suite-configure-type/groovy",files="build.gradle[tags=configure-type]"]
====

<1> This suite would use a type value of 'secondary-test' by default.  This explicitly sets the type to 'integration-test'.

== Configure the `Test` task for a test suite
====
include::sample[dir="snippets/testing/test-suite-configure-source-dir/kotlin",files="build.gradle.kts[tags=configure-test-task]"]
include::sample[dir="snippets/testing/test-suite-configure-source-dir/groovy",files="build.gradle[tags=configure-test-task]"]
====

<1> Configure the `integrationTest` task created by declaring a suite of the same name.
<2> Configure the `Test` task properties.

`Test` tasks associated with a test suite target can also be configured directly, by name.  It is not necessary to configure via the test suite DSL.

== Sharing configuration between multiple test suites

There are several ways to share configuration between multiple test suites to avoid the duplication of dependencies or other configuration.  Each method is a tradeoff between flexibility and the use of a declarative vs. imperative configuration style.

1. Use the `configureEach` method on the `suites` container to configure every test suite the same way.
2. Use `withType` and `matching` with `configureEach` to filter test suites and configure a subset of them.
3. Extract the configuration block to a local variable and apply it only to the desired test suites.

=== Method 1: Use `configureEach`

This is the most straightforward way to share configuration across _every_ test suite.  The configuration is applied to each test suite.

====
include::sample[dir="snippets/testing/test-suite-multi-configure-each/kotlin",files="build.gradle.kts[tags=multi-configure]"]
include::sample[dir="snippets/testing/test-suite-multi-configure-each/groovy",files="build.gradle[tags=multi-configure]"]
====

<1> Configure every JVM test suite
<2> Provide the dependencies to be shared by all test suites
<3> Define the additional test suites which will be configured upon creation
<4> Add dependencies specific to the `functionalTest` test suite

=== Method 2: Use `withType`, `matching` and `configureEach`

This method adds filtering commands to only apply the configuration to a subset of the test suites, as identified by name.

====
include::sample[dir="snippets/testing/test-suite-multi-configure-each-matching/kotlin",files="build.gradle.kts[tags=multi-configure]"]
include::sample[dir="snippets/testing/test-suite-multi-configure-each-matching/groovy",files="build.gradle[tags=multi-configure]"]
====

<1> Configure every JVM test suite matching the given criteria
<2> Use a different testing framework for the `functionalTest` test suite
<3> Add dependencies specific to the `functionalTest` test suite


=== Method 3: Extract a custom configuration block

This method is the most flexible, but also the most imperative.

====
include::sample[dir="snippets/testing/test-suite-multi-configure-each-extracted/kotlin",files="build.gradle.kts[tags=multi-configure]"]
include::sample[dir="snippets/testing/test-suite-multi-configure-each-extracted/groovy",files="build.gradle[tags=multi-configure]"]
====

<1> Define a closure which takes a single `JvmTestSuite` parameter and configures it
<2> Apply the closure to a test suite, using the default (`test`) test suite
<3> Alternate means of applying a configuration closure to a test suite outside of its declaration, using the `integrationTest` test suite

[[sec:differences_with_top_level_dependencies]]
== Differences between the test suite `dependencies` and the top-level `dependencies` blocks

Gradle 7.6 changed the API of the test suite's `dependencies` block to be more strongly-typed.

Each dependency scope returns a link:{groovyDslPath}/org.gradle.api.artifacts.dsl.DependencyAdder.html[`DependencyAdder`] that provides strongly-typed methods to add and configure dependencies.

There is also a link:{javadocPath}/org/gradle/api/artifacts/dsl/DependencyFactory.html[`DependencyFactory`] with factory methods to create new dependencies from different notations.
Dependencies can be created lazily using these factory methods, as shown below.

This API differs from the top-level `dependencies` block in the following ways:

* Dependencies must be declared using a `String`, an instance of `Dependency`, a `FileCollection`, a `Provider` of `Dependency`, or a `ProviderConvertible` of `MinimalExternalModuleDependency`.
* Outside of Gradle build scripts, you must explicitly call a getter for the `DependencyAdder` and `add`.
    ** `dependencies.add("implementation", x)` becomes `getImplementation().add(x)`
* You cannot declare dependencies with the `Map` notation from Kotlin and Java.
  Use multi-argument methods instead in Kotlin and Java.
    ** Kotlin: `compileOnly(mapOf("group" to "foo", "name" to "bar"))` becomes `compileOnly(module(group = "foo", name = "bar"))`
    ** Java: `compileOnly(Map.of("group", "foo", "name", "bar"))` becomes `getCompileOnly().add(module("foo", "bar", null))`
* You cannot add a dependency with an instance of `Project`. You must turn it into a `ProjectDependency` first.
* You cannot add version catalog bundles directly.
  Instead, use the `bundle` method on each configuration.
    ** Kotlin and Groovy: `implementation(libs.bundles.testing)` becomes `implementation.bundle(libs.bundles.testing)`
* You cannot use providers for non-`Dependency` types directly.
  Instead, map them to a `Dependency` using the `DependencyFactory`.
    ** Kotlin and Groovy: `implementation(myStringProvider)` becomes `implementation(myStringProvider.map { dependencyFactory.create(it) })`
    ** Java: `implementation(myStringProvider)` becomes `getImplementation().add(myStringProvider.map(getDependencyFactory()::create)`

The `dependencies` block within a test suite may not provide access to the same methods as the link:{groovyDslPath}/org.gradle.api.artifacts.dsl.DependencyHandler.html[top-level `dependencies` block].
To access these methods, you need to use the top-level `dependencies` block.

[NOTE]
====
Adding dependencies via the test suite's own `dependencies` block is the preferred and recommended way. You can also access the configurations of a test suite in a top-level `dependencies` block after creating the test suite, but the exact names of the configurations used by test suites should be considered an implementation detail subject to change.
====

== Differences between similar methods on link:{groovyDslPath}/org.gradle.api.plugins.jvm.JvmTestSuite.html[JvmTestSuite] and link:{groovyDslPath}/org.gradle.api.tasks.testing.Test.html[Test] task types

Instances of JvmTestSuite have methods link:{javadocPath}/org/gradle/api/plugins/jvm/JvmTestSuite.html#useJUnit--[useJUnit()] and link:{javadocPath}/org/gradle/api/plugins/jvm/JvmTestSuite.html#useJUnitJupiter--[useJUnitJupiter()], which are similar in name to methods on the link:{groovyDslPath}/org.gradle.api.tasks.testing.Test.html[Test] task type:  link:{javadocPath}/org/gradle/api/tasks/testing/Test.html#useJUnit--[useJUnit()] and link:{javadocPath}/org/gradle/api/tasks/testing/Test.html#useJUnitPlatform--[useJUnitPlatform()].  However, there are important differences.

Unlike the methods on the link:{groovyDslPath}/org.gradle.api.tasks.testing.Test.html[Test] task, link:{groovyDslPath}/org.gradle.api.plugins.jvm.JvmTestSuite.html[JvmTestSuite] methods perform two additional pieces of configuration:

. link:{javadocPath}/org/gradle/api/plugins/jvm/JvmTestSuite.html#useJUnit--[JvmTestSuite#useJUnit()], link:{javadocPath}/org/gradle/api/plugins/jvm/JvmTestSuite.html#useJUnitJupiter--[#useJUnitJupiter()] and other framework-specific methods automatically apply the relevant testing framework libraries to the compile and runtime classpaths of the suite targets.  Note the overloaded methods like link:{javadocPath}/org/gradle/api/plugins/jvm/JvmTestSuite.html#useJUnit-java.lang.String-[JvmTestSuite#useJUnit(String)] and link:{javadocPath}/org/gradle/api/plugins/jvm/JvmTestSuite.html#useJUnitJupiter-java.lang.String-[#useJUnitJupiter(String)] allow you to provide specific versions of the framework dependencies.
. link:{javadocPath}/org/gradle/api/plugins/jvm/JvmTestSuite.html#useJUnit--[JvmTestSuite#useJUnit()] and link:{javadocPath}/org/gradle/api/plugins/jvm/JvmTestSuite.html#useJUnitJupiter--[#useJUnitJupiter()] automatically configure the suite targets' link:{groovyDslPath}/org.gradle.api.tasks.testing.Test.html[Test] tasks to execute using the specified framework.

An example of this behavior is shown in the first configuration example above, <<#sec:configuring_the_built_in_test_suite,Configuring the built-in `test` suite>>.

Note that unlike the link:{groovyDslPath}/org.gradle.api.tasks.testing.Test.html[Test] task, the aforementioned methods on link:{groovyDslPath}/org.gradle.api.plugins.jvm.JvmTestSuite.html[JvmTestSuite] do not accept a closure or action for configuring the framework.  These framework configuration options can be set on the individual targets.

[[sec:outgoing_variants]]
== Outgoing Variants
Each test suite creates an outgoing variant containing its test execution results.  These variants are designed for consumption by the <<test_report_aggregation_plugin#test_report_aggregation_plugin, Test Report Aggregation Plugin>>.

The attributes will resemble the following.  User-configurable attributes are highlighted below the sample.

.outgoingVariants task output
[source,text]
----
--------------------------------------------------
Variant testResultsElementsForTest (i)
--------------------------------------------------
Description = Directory containing binary results of running tests for the test Test Suite's test target.

Capabilities
    - org.gradle.sample:list:1.0.2 (default capability)
Attributes
    - org.gradle.category              = verification
    - org.gradle.testsuite.name        = test           // <1>
    - org.gradle.testsuite.target.name = test           // <2>
    - org.gradle.testsuite.type        = unit-test      // <3>
    - org.gradle.verificationtype      = test-results

Artifacts
    - build/test-results/test/binary (artifactType = directory)
----
<1> TestSuiteName attribute; value is derived from link:{javadocPath}/org/gradle/api/plugins/jvm/JvmTestSuite.html#getName--[JvmTestSuite#getName()].
<2> TestSuiteTargetName attribute; value is derived from link:{javadocPath}/org/gradle/api/plugins/jvm/JvmTestSuiteTarget.html#getName--[JvmTestSuiteTarget#getName()].
<3> TestSuiteType attribute; value is derived from link:{javadocPath}/org/gradle/api/plugins/jvm/JvmTestSuite.html#getTestType--[JvmTestSuite#getTestType()].
