= Testing Gradle plugins

Testing plays a crucial role in the development process as it ensures reliable and high-quality software.
The same principles apply to build code and more specifically Gradle plugins.
In this section you will learn effective techniques for testing plugin code.

This section assumes you have:

- Basic understanding of software engineering practices
- Knowledge of <<implementing_gradle_plugins.adoc#implemention_plugins,Gradle plugin implementation techniques>>
- Working knowledge in writing Java code

== The sample project

All discussions in this section are centered around a sample project called _URL verifier plugin_.
The plugin creates a task named `verifyUrl` that checks whether a given URL can be resolved via HTTP GET.
The end user can provide the URL via an extension named `verification`.

The following build script assumes that the plugin JAR file has been published to a binary repository.
In a nutshell, the script demonstrates how to apply the plugin to the project and configure its exposed extension.

====
include::sample[dir="snippets/developingPlugins/testingPlugins/kotlin/include-plugin-build",files="build.gradle.kts[]"]
include::sample[dir="snippets/developingPlugins/testingPlugins/groovy/include-plugin-build",files="build.gradle[]"]
====
<1> Applies the plugin to the project
<2> Configures the URL to be verified through the exposed extension

Executing the task renders a success message if the HTTP GET call to the configured URL returns with a 200 response code.

[[verify-url-cmd-output]]
[source,shell]
----
$ gradle verifyUrl

include::{snippetsPath}/developingPlugins/testingPlugins/tests/verifyUrl.out[]
----

Before diving into the code, let's first revisit the different types of tests and the tooling that supports implementing them.

== On the importance of testing

Testing is a foundational activity in the software development life cycle.
Appropriate testing ensures that the software works on a functional and non-functional level before it is released to the end user.
As a by product, automated testing also enables the development team to refactor and evolve the code without fearing to introduce regressions in the process.

[[testing-pyramid]]
=== The testing pyramid

image::testing-pyramid.png[float=right]

Probably the easiest way to test software is to manually exercise it.
**Manual testing** can occur at any time and is not bound to writing automation code.
However, manual testing is error-prone and cumbersome as it requires a human to walk through a set of predefined test cases.
Manually testing Gradle plugins requires consuming the plugin binary in a build script.

Other types of tests can be fully automated and exercised with every change to the source code.
The testing pyramid introduced by Mike Cohen in his book link:https://www.mountaingoatsoftware.com/books/succeeding-with-agile-software-development-using-scrum[Succeeding with Agile: Software Development Using Scrum] describes three types of automated tests.

**Unit testing** aims to verify the smallest unit of code.
In Java-based projects this unit is a method.
Unit tests usually do not interact with other parts of the system e.g.
a database or the file system.
Interactions with other parts of the system are usually cut off with the help of Stubs or Mocks.
You will find that POJOs and utility classes are good candidates for unit tests as they are self-contained and do not use the Gradle API.

**Integration testing** verifies that multiple classes or components work together as a whole.
The code under test may reach out to external subsystems.

**Functional testing** is used to test the system from the end user's perspective.
End-to-end tests for Gradle plugins stand up a build script, apply the plugin under test and execute the build with a specific task.
The outcome of the build (e.g.
standard output/error or generated artifacts) verifies the correctness of the functionality.

=== Tooling support

Implementing manual and automated testing for Gradle plugins is straight forward - it just requires the right tooling.
The table below gives you a brief overview on how to approach each test type.
Please be aware that you have the free choice of using the test framework you are most familiar with.
For a detailed discussion and code example please refer to the dedicated section further down.

[cols="20%,80%",options="header"]
|======================
|Test type                               |Tooling support
|<<manual-tests,Manual tests>>           |<<composite_builds.adoc#composite_builds,Gradle composite builds>>
|<<unit-tests,Unit tests>>               |Any JVM-based test framework
|<<integration-tests,Integration tests>> |Any JVM-based test framework
|<<functional-tests,Functional tests>>   |Any JVM-based test framework and <<test_kit.adoc#test_kit,Gradle TestKit>>
|======================

[[manual-tests]]
== Setting up manual tests

The <<composite_builds.adoc#composite_builds,composite builds>> feature of Gradle makes it very easy to test a plugin manually.
The standalone plugin project and the consuming project can be combined together into a single unit making it much more straight forward to try out or debug changes without the hassle of re-publishing the binary file.

----
.
├── include-plugin-build   // <1>
│   ├── build.gradle
│   └── settings.gradle
└── url-verifier-plugin    // <2>
    ├── build.gradle
    ├── settings.gradle
    └── src
----
<1> Consuming project that includes the plugin project
<2> The plugin project

There are two ways to include a plugin project into a consuming project.

1.
By using the command line option `--include-build`.
2.
By using the method `includeBuild` in `settings.gradle`.

The following code snippet demonstrates the use of the settings file.

====
include::sample[dir="snippets/developingPlugins/testingPlugins/kotlin/include-plugin-build",files="settings.gradle.kts[tags=include-build]"]
include::sample[dir="snippets/developingPlugins/testingPlugins/groovy/include-plugin-build",files="settings.gradle[tags=include-build]"]
====

The command line output of task `verifyUrl` from the project `include-plugin-build` <<verify-url-cmd-output,looks exactly the same as shown in the introduction>> except that it now executed as part of a composite build.

Manual testing has its place in the development process.
By no means is it a replacement for automated testing.
Next up, you'll learn how to organize and implement automated tests for Gradle plugins.

== Setting up automated tests

Setting up a suite of tests earlier on is crucial to the success of your plugin.
You will encounter various situations that make your tests an invaluable safety net you can rely on e.g.
when upgrading the plugin to a new Gradle version and enhancing or refactoring the code.

=== Organizing test source code

We recommend to implement a good distribution of unit, integration and functional tests to cover the most important use cases.
Separating the source code for each test type automatically results in a project that is more maintainable and manageable.

By default the Java project already creates a convention for organizing unit tests, the directory `src/test/java`.
Additionally, if you apply the Groovy plugin source code under the directory `src/test/groovy` is taken under consideration for compilation.
Consequently, source code directories for other test types should follow a similar pattern.
Below you can find an exemplary project layout for a plugin project that chooses to use a Groovy-based testing approach.

----
.
└── src
    ├── functionalTest
    │   └── groovy      // <1>
    ├── integrationTest
    │   └── groovy      // <2>
    ├── main
    │   ├── java        // <3>
    └── test
        └── groovy      // <4>
----
<1> Source directory containing functional tests
<2> Source directory containing integration tests
<3> Source directory containing production source code
<4> Source directory containing unit tests

NOTE: The directories `src/integrationTest/groovy` and `src/functionalTest/groovy` are not based on an existing standard convention for Gradle projects.
You are free to choose any project layout that works best for you.

In the next section, you will learn how to configure those source directories for compilation and test execution.
You can also rely on third-party plugins for convience e.g.
the link:https://github.com/nebula-plugins/nebula-project-plugin#nebula-facet-plugin[Nebula Facet plugin] or the link:https://github.com/unbroken-dome/gradle-testsets-plugin[TestSets plugin].

=== Modeling test types

[NOTE]
====
A new configuration DSL for modeling the below `integrationTest` suite is available via the incubating <<jvm_test_suite_plugin.adoc#jvm_test_suite_plugin,JVM Test Suite>> plugin.
====

Gradle models source code directories with the help of the <<java_plugin.adoc#source_sets,source set concept>>.
By pointing an instance of a source set to one or many source code directories, _Gradle will automatically create a corresponding compilation task out-of-the-box_.
A pre-configured source set can be created with one line of build script code.
The source set automatically registers _configurations_ to define dependencies for the sources of the source set.
We use that to define an `integrationTestImplementation` dependency to the project itself, which represents the "main" variant of our project (i.e. the compiled plugin code).

====
include::sample[dir="snippets/developingPlugins/testingPlugins/kotlin/url-verifier-plugin",files="build.gradle.kts[tags=test-source-set]"]
include::sample[dir="snippets/developingPlugins/testingPlugins/groovy/url-verifier-plugin",files="build.gradle[tags=test-source-set]"]
====

Source sets are only responsible for compiling source code, but do not deal with executing the byte code.
For the purpose of test execution, a corresponding task of type link:{javadocPath}/org/gradle/api/tasks/testing/Test.html[Test] needs to be established.
The following listing shows the setup for executing integration tests.
As you can see below, the task references the classes and runtime classpath of the integration test source set.

====
include::sample[dir="snippets/developingPlugins/testingPlugins/kotlin/url-verifier-plugin",files="build.gradle.kts[tags=test-task]"]
include::sample[dir="snippets/developingPlugins/testingPlugins/groovy/url-verifier-plugin",files="build.gradle[tags=test-task]"]
====

=== Configuring a test framework

Gradle does not dictate the use of a specific test framework.
Popular choices include link:https://junit.org/[JUnit], link:https://testng.org/[TestNG] and link:http://spockframework.org/[Spock].
Once you choose an option, you have to add its dependency to the compile classpath for your tests.
The following code snippet shows how to use Spock for implementing tests.
We choose to use it for all our three test types (_test_, _integrationTest_ and _functionalTest_) and thus define a dependency for each of them.

====
include::sample[dir="snippets/developingPlugins/testingPlugins/kotlin/url-verifier-plugin",files="build.gradle.kts[tags=test-framework]"]
include::sample[dir="snippets/developingPlugins/testingPlugins/groovy/url-verifier-plugin",files="build.gradle[tags=test-framework]"]
====

NOTE: Spock is a Groovy-based BDD test framework that even includes APIs for creating Stubs and Mocks.
The Gradle team prefers Spock over other options for its expressiveness and conciseness.

== Implementing automated tests

This section discusses representative implementation examples for unit, integration and functional tests.
All test classes are based on the use of Spock though it should be relatively easy to adapt the code to a different test framework.
Please revisit the section <<testing-pyramid,the testing pyramid>> for a formal discussion of the definition of each test type.

[[unit-tests]]
=== Implementing unit tests

The URL verifier plugin emits HTTP GET calls to check if a URL can be resolved successfully.
The method `DefaultHttpCaller.get(String)` is responsible for calling a given URL and returns with an instance of type `HttpResponse`.
`HttpResponse` is a POJO containing information about the HTTP response code and message.

.HttpResponse.java
[source,java]
----
include::{snippetsPath}/developingPlugins/testingPlugins/groovy/url-verifier-plugin/src/main/java/org/myorg/http/HttpResponse.java[]
----

The class `HttpResponse` represents a good candidate to be tested by a unit test.
It does not reach out to any other classes nor does it use the Gradle API.

.HttpResponseTest.groovy
[source,groovy]
----
include::{snippetsPath}/developingPlugins/testingPlugins/groovy/url-verifier-plugin/src/test/groovy/org/myorg/http/HttpResponseTest.groovy[]
----

IMPORTANT: When writing unit tests, it's important to test boundary conditions and various forms of invalid input.
Furthermore, try to extract as much logic as possible from classes that use the Gradle API to make it testable as unit tests.
It will buy you the benefit of maintainable code and faster test execution.

[[integration-tests]]
=== Implementing integration tests

Let's have a look at a class that reaches out to another system, the piece of code that emits the HTTP calls.
At the time of executing a test for the class `DefaultHttpCaller`, the runtime environment needs to be able to reach out to the internet.

.DefaultHttpCaller.java
[source,java]
----
include::{snippetsPath}/developingPlugins/testingPlugins/groovy/url-verifier-plugin/src/main/java/org/myorg/http/DefaultHttpCaller.java[]
----

Implementing an integration test for `DefaultHttpCaller` doesn't look much different from the unit test shown in the previous section.

.DefaultHttpCallerIntegrationTest.groovy
[source,groovy]
----
include::{snippetsPath}/developingPlugins/testingPlugins/groovy/url-verifier-plugin/src/integrationTest/groovy/org/myorg/http/DefaultHttpCallerIntegrationTest.groovy[]
----

[[functional-tests]]
=== Implementing functional tests

Functional tests verify the correctness of the plugin end-to-end.
In practice that means applying, configuring and executing the functionality of the plugin implementation represented by the class `UrlVerifierPlugin`.
As you can see, the implementation exposes an extension and a task instance that uses the URL value configured by the end user.

.UrlVerifierPlugin.java
[source,java]
----
include::{snippetsPath}/developingPlugins/testingPlugins/groovy/url-verifier-plugin/src/main/java/org/myorg/UrlVerifierPlugin.java[]
----

Every Gradle plugin project should apply the <<java_gradle_plugin.adoc#java_gradle_plugin,plugin development plugin>> to reduce boilerplate code.
By applying the plugin development plugin, the test source set is preconfigured for the use with TestKit.
If we want to use a custom source set for functional tests and leave the default test source set for only unit tests, we can configure the plugin development plugin to look for TestKit tests elsewhere.

====
include::sample[dir="snippets/developingPlugins/testingPlugins/kotlin/url-verifier-plugin",files="build.gradle.kts[tags=source-set-config]"]
include::sample[dir="snippets/developingPlugins/testingPlugins/groovy/url-verifier-plugin",files="build.gradle[tags=source-set-config]"]
====

Functional tests for Gradle plugins use an instance of `GradleRunner` to execute the build under test.
`GradleRunner` is an API provided by TestKit which internally uses the Tooling API to execute the build.
The following example applies the plugin to the build script under test, configures the extension and executes the build with the task `verifyUrl`.
Please see the <<test_kit.adoc#test_kit,TestKit documentation>> to get more familiar with the functionality of TestKit.

.UrlVerifierPluginFunctionalTest.groovy
[source,groovy]
----
include::{snippetsPath}/developingPlugins/testingPlugins/groovy/url-verifier-plugin/src/functionalTest/groovy/org/myorg/UrlVerifierPluginFunctionalTest.groovy[]
----

==== IDE integration

TestKit determines the plugin classpath by running a specific Gradle task.
You will need to execute the `assemble` task to initially generate the plugin classpath or to reflect changes to it even when running TestKit-based functional tests from the IDE.

Some IDEs provide a convenience option to delegate the "test classpath generation and execution" to the build.
In IntelliJ you can find this option under _Preferences... > Build, Execution, Deployment > Build Tools > Gradle > Runner > Delegate IDE build/run actions to gradle_.

image::intellij-delegate-to-build.png[]
