# Testing Gradle plugins

version 9.0.0

Testing plays a crucial role in the development process by ensuring reliable and high-quality software. This principle applies to build code, including Gradle plugins.

## The sample project

This section revolves around a sample project called the "URL verifier plugin". This plugin creates a task named `verifyUrl` that checks whether a given URL can be resolved via HTTP GET. The end user can provide the URL via an extension named `verification`.

The following build script assumes that the plugin JAR file has been published to a binary repository. The script demonstrates how to apply the plugin to the project and configure its exposed extension:

build.gradle.kts

    plugins {
        id("org.myorg.url-verifier")        (1)
    }

    verification {
        url = "https://www.google.com/"  (2)
    }

build.gradle

    plugins {
        id 'org.myorg.url-verifier'         (1)
    }

    verification {
        url = 'https://www.google.com/'     (2)
    }

1 | Applies the plugin to the project
2 | Configures the URL to be verified through the exposed extension

Executing the `verifyUrl` task renders a success message if the HTTP GET call to the configured URL returns with a 200 response code:

    $ gradle verifyUrl

    > Task :verifyUrl
    Successfully resolved URL 'https://www.google.com/'

    BUILD SUCCESSFUL in 0s
    5 actionable tasks: 5 executed

Before diving into the code, let’s first revisit the different types of tests and the tooling that supports implementing them.

## The importance of testing

Testing is a crucial part of the software development life cycle, ensuring that software functions correctly and meets quality standards before release. Automated testing allows developers to refactor and improve code with confidence.

### The testing pyramid

Manual Testing

  While manual testing is straightforward, it is error-prone and requires human effort. For Gradle plugins, manual testing involves using the plugin in a build script.

Automated Testing

  Automated testing includes unit, integration, and functional testing.

The testing pyramid introduced by Mike Cohen in his book Succeeding with Agile: Software Development Using Scrum describes three types of automated tests:

1. Unit Testing: Verifies the smallest units of code, typically methods, in isolation. It uses Stubs or Mocks to isolate code from external dependencies.

2. Integration Testing: Validates that multiple units or components work together.

3. Functional Testing: Tests the system from the end user’s perspective, ensuring correct functionality. End-to-end tests for Gradle plugins simulate a build, apply the plugin, and execute specific tasks to verify functionality.

### Tooling support

Testing Gradle plugins, both manually and automatically, is simplified with the appropriate tools. The table below provides a summary of each testing approach. You can choose any test framework you’re comfortable with.

For detailed explanations and code examples, refer to the specific sections below:

Test type | Tooling support
--- | ---
Manual tests | Gradle composite builds
Unit tests | Any JVM-based test framework
Integration tests | Any JVM-based test framework
Functional tests | Any JVM-based test framework and Gradle TestKit

## Setting up manual tests

The composite builds feature of Gradle makes it easy to test a plugin manually. The standalone plugin project and the consuming project can be combined into a single unit, making it straightforward to try out or debug changes without re-publishing the binary file:

    .
    ├── include-plugin-build   (1)
    │   ├── build.gradle
    │   └── settings.gradle
    └── url-verifier-plugin    (2)
        ├── build.gradle
        ├── settings.gradle
        └── src

1 | Consuming project that includes the plugin project
2 | The plugin project

There are two ways to include a plugin project in a consuming project:

1. By using the command line option `--include-build`.

2. By using the method `includeBuild` in `settings.gradle`.

The following code snippet demonstrates the use of the settings file:

settings.gradle.kts

    pluginManagement {
        includeBuild("../url-verifier-plugin")
    }

settings.gradle

    pluginManagement {
        includeBuild '../url-verifier-plugin'
    }

The command line output of the `verifyUrl` task from the project `include-plugin-build` looks exactly the same as shown in the introduction, except that it now executes as part of a composite build.

Manual testing has its place in the development process, but it is not a replacement for automated testing.

## Setting up automated tests

Setting up a suite of tests early on is crucial to the success of your plugin. Automated tests become an invaluable safety net when upgrading the plugin to a new Gradle version or enhancing/refactoring the code.

### Organizing test source code

We recommend implementing a good distribution of unit, integration, and functional tests to cover the most important use cases. Separating the source code for each test type automatically results in a project that is more maintainable and manageable.

By default, the Java project creates a convention for organizing unit tests in the directory `src/test/java`. Additionally, if you apply the Groovy plugin, source code under the directory `src/test/groovy` is considered for compilation (with the same standard for Kotlin under the directory `src/test/kotlin`). Consequently, source code directories for other test types should follow a similar pattern:

    .
    └── src
        ├── functionalTest
        │   └── groovy      (1)
        ├── integrationTest
        │   └── groovy      (2)
        ├── main
        │   ├── java        (3)
        └── test
            └── groovy      (4)

1 | Source directory containing functional tests
2 | Source directory containing integration tests
3 | Source directory containing production source code
4 | Source directory containing unit tests

| The directories `src/integrationTest/groovy` and `src/functionalTest/groovy` are not based on an existing standard convention for Gradle projects. You are free to choose any project layout that works best for you.

You can configure the source directories for compilation and test execution.

The Test Suite plugin provides a DSL and API to model multiple groups of automated tests into test suites in JVM-based projects. You can also rely on third-party plugins for convenience, such as the Nebula Facet plugin or the TestSets plugin.

### Modeling test types

| A new configuration DSL for modeling the below `integrationTest` suite is available via the incubating JVM Test Suite plugin.

In Gradle, source code directories are represented using the concept of source sets. A source set is configured to point to one or more directories containing source code. When you define a source set, Gradle automatically sets up compilation tasks for the specified directories.

A pre-configured source set can be created with one line of build script code. The source set automatically registers configurations to define dependencies for the sources of the source set:

    // Define a source set named 'test' for test sources
    sourceSets {
        test {
            java {
                srcDirs = ['src/test/java']
            }
        }
    }
    // Specify a test implementation dependency on JUnit
    dependencies {
        testImplementation 'junit:junit:4.12'
    }

We use that to define an `integrationTestImplementation` dependency to the project itself, which represents the "main" variant of our project (i.e., the compiled plugin code):

build.gradle.kts

    val integrationTest by sourceSets.creating

    dependencies {
        "integrationTestImplementation"(project)
    }

build.gradle

    def integrationTest = sourceSets.create("integrationTest")

    dependencies {
        integrationTestImplementation(project)
    }

Source sets are responsible for compiling source code, but they do not deal with executing the bytecode. For test execution, a corresponding task of type Test needs to be established. The following setup shows the execution of integration tests, referencing the classes and runtime classpath of the integration test source set:

build.gradle.kts

    val integrationTestTask = tasks.register<Test>("integrationTest") {
        description = "Runs the integration tests."
        group = "verification"
        testClassesDirs = integrationTest.output.classesDirs
        classpath = integrationTest.runtimeClasspath
        mustRunAfter(tasks.test)
    }
    tasks.check {
        dependsOn(integrationTestTask)
    }

build.gradle

    def integrationTestTask = tasks.register("integrationTest", Test) {
        description = 'Runs the integration tests.'
        group = "verification"
        testClassesDirs = integrationTest.output.classesDirs
        classpath = integrationTest.runtimeClasspath
        mustRunAfter(tasks.named('test'))
    }
    tasks.named('check') {
        dependsOn(integrationTestTask)
    }

### Configuring a test framework

Gradle does not dictate the use of a specific test framework. Popular choices include JUnit, TestNG and Spock. Once you choose an option, you have to add its dependency to the compile classpath for your tests.

The following code snippet shows how to use Spock for implementing tests:

build.gradle.kts

    repositories {
        mavenCentral()
    }

    dependencies {
        testImplementation(platform("org.spockframework:spock-bom:2.3-groovy-4.0"))
        testImplementation("org.spockframework:spock-core")
        testRuntimeOnly("org.junit.platform:junit-platform-launcher")

        "integrationTestImplementation"(platform("org.spockframework:spock-bom:2.3-groovy-4.0"))
        "integrationTestImplementation"("org.spockframework:spock-core")
        "integrationTestRuntimeOnly"("org.junit.platform:junit-platform-launcher")

        "functionalTestImplementation"(platform("org.spockframework:spock-bom:2.3-groovy-4.0"))
        "functionalTestImplementation"("org.spockframework:spock-core")
        "functionalTestRuntimeOnly"("org.junit.platform:junit-platform-launcher")
    }

    tasks.withType<Test>().configureEach {
        // Using JUnitPlatform for running tests
        useJUnitPlatform()
    }

build.gradle

    repositories {
        mavenCentral()
    }

    dependencies {
        testImplementation platform("org.spockframework:spock-bom:2.3-groovy-4.0")
        testImplementation 'org.spockframework:spock-core'
        testRuntimeOnly 'org.junit.platform:junit-platform-launcher'

        integrationTestImplementation platform("org.spockframework:spock-bom:2.3-groovy-4.0")
        integrationTestImplementation 'org.spockframework:spock-core'
        integrationTestRuntimeOnly 'org.junit.platform:junit-platform-launcher'

        functionalTestImplementation platform("org.spockframework:spock-bom:2.3-groovy-4.0")
        functionalTestImplementation 'org.spockframework:spock-core'
        functionalTestRuntimeOnly 'org.junit.platform:junit-platform-launcher'
    }

    tasks.withType(Test).configureEach {
        // Using JUnitPlatform for running tests
        useJUnitPlatform()
    }

| Spock is a Groovy-based BDD test framework that even includes APIs for creating Stubs and Mocks. The Gradle team prefers Spock over other options for its expressiveness and conciseness.

## Implementing automated tests

This section discusses representative implementation examples for unit, integration, and functional tests. All test classes are based on the use of Spock, though it should be relatively easy to adapt the code to a different test framework.

### Implementing unit tests

The URL verifier plugin emits HTTP GET calls to check if a URL can be resolved successfully. The method `DefaultHttpCaller.get(String)` is responsible for calling a given URL and returns an instance of type `HttpResponse`. `HttpResponse` is a POJO containing information about the HTTP response code and message:

HttpResponse.java

    package org.myorg.http;

    public class HttpResponse {
        private int code;
        private String message;

        public HttpResponse(int code, String message) {
            this.code = code;
            this.message = message;
        }

        public int getCode() {
            return code;
        }

        public String getMessage() {
            return message;
        }

        @Override
        public String toString() {
            return "HTTP " + code + ", Reason: " + message;
        }
    }

The class `HttpResponse` represents a good candidate for a unit test. It does not reach out to any other classes nor does it use the Gradle API.

HttpResponseTest.groovy

    package org.myorg.http

    import spock.lang.Specification

    class HttpResponseTest extends Specification {

        private static final int OK_HTTP_CODE = 200
        private static final String OK_HTTP_MESSAGE = 'OK'

        def "can access information"() {
            when:
            def httpResponse = new HttpResponse(OK_HTTP_CODE, OK_HTTP_MESSAGE)

            then:
            httpResponse.code == OK_HTTP_CODE
            httpResponse.message == OK_HTTP_MESSAGE
        }

        def "can get String representation"() {
            when:
            def httpResponse = new HttpResponse(OK_HTTP_CODE, OK_HTTP_MESSAGE)

            then:
            httpResponse.toString() == "HTTP $OK_HTTP_CODE, Reason: $OK_HTTP_MESSAGE"
        }
    }

| When writing unit tests, it’s important to test boundary conditions and various forms of invalid input. Try to extract as much logic as possible from classes that use the Gradle API to make it testable as unit tests. It will result in maintainable code and faster test execution.

You can use the ProjectBuilder class to create Project instances to use when you test your plugin implementation.

src/test/java/org/example/GreetingPluginTest.java

    public class GreetingPluginTest {
        @Test
        public void greeterPluginAddsGreetingTaskToProject() {
            Project project = ProjectBuilder.builder().build();
            project.getPluginManager().apply("org.example.greeting");

            assertTrue(project.getTasks().getByName("hello") instanceof GreetingTask);
        }
    }

### Implementing integration tests

Let’s look at a class that reaches out to another system, the piece of code that emits the HTTP calls. At the time of executing a test for the class `DefaultHttpCaller`, the runtime environment needs to be able to reach out to the internet:

DefaultHttpCaller.java

    package org.myorg.http;

    import java.io.IOException;
    import java.net.HttpURLConnection;
    import java.net.URI;
    import java.net.URISyntaxException;

    public class DefaultHttpCaller implements HttpCaller {
        @Override
        public HttpResponse get(String url) {
            try {
                HttpURLConnection connection = (HttpURLConnection) new URI(url).toURL().openConnection();
                connection.setConnectTimeout(5000);
                connection.setRequestMethod("GET");
                connection.connect();

                int code = connection.getResponseCode();
                String message = connection.getResponseMessage();
                return new HttpResponse(code, message);
            } catch (IOException e) {
                throw new HttpCallException(String.format("Failed to call URL '%s' via HTTP GET", url), e);
            } catch (URISyntaxException e) {
                throw new RuntimeException(e);
            }
        }
    }

Implementing an integration test for `DefaultHttpCaller` doesn’t look much different from the unit test shown in the previous section:

DefaultHttpCallerIntegrationTest.groovy

    package org.myorg.http

    import spock.lang.Specification
    import spock.lang.Subject

    class DefaultHttpCallerIntegrationTest extends Specification {
        @Subject HttpCaller httpCaller = new DefaultHttpCaller()

        def "can make successful HTTP GET call"() {
            when:
            def httpResponse = httpCaller.get('https://www.google.com/')

            then:
            httpResponse.code == 200
            httpResponse.message == 'OK'
        }

        def "throws exception when calling unknown host via HTTP GET"() {
            when:
            httpCaller.get('https://www.example.invalid/')

            then:
            def t = thrown(HttpCallException)
            t.message == "Failed to call URL 'https://www.example.invalid/' via HTTP GET"
            t.cause instanceof UnknownHostException
        }
    }

### Implementing functional tests

Functional tests verify the correctness of the plugin end-to-end. In practice, this means applying, configuring, and executing the functionality of the plugin implementation. The `UrlVerifierPlugin` class exposes an extension and a task instance that uses the URL value configured by the end user:

UrlVerifierPlugin.java

    package org.myorg;

    import org.gradle.api.Plugin;
    import org.gradle.api.Project;
    import org.myorg.tasks.UrlVerify;

    public class UrlVerifierPlugin implements Plugin<Project> {
        @Override
        public void apply(Project project) {
            UrlVerifierExtension extension = project.getExtensions().create("verification", UrlVerifierExtension.class);
            UrlVerify verifyUrlTask = project.getTasks().create("verifyUrl", UrlVerify.class);
            verifyUrlTask.getUrl().set(extension.getUrl());
        }
    }

Every Gradle plugin project should apply the plugin development plugin to reduce boilerplate code. By applying the plugin development plugin, the test source set is preconfigured for the use with TestKit. If we want to use a custom source set for functional tests and leave the default test source set for only unit tests, we can configure the plugin development plugin to look for TestKit tests elsewhere.

build.gradle.kts

    gradlePlugin {
        testSourceSets(functionalTest)
    }

build.gradle

    gradlePlugin {
        testSourceSets(sourceSets.functionalTest)
    }

Functional tests for Gradle plugins use an instance of `GradleRunner` to execute the build under test. `GradleRunner` is an API provided by TestKit, which internally uses the Tooling API to execute the build.

The following example applies the plugin to the build script under test, configures the extension and executes the build with the task `verifyUrl`. Please see the TestKit documentation to get more familiar with the functionality of TestKit.

UrlVerifierPluginFunctionalTest.groovy

    package org.myorg

    import org.gradle.testkit.runner.GradleRunner
    import spock.lang.Specification
    import spock.lang.TempDir

    import static org.gradle.testkit.runner.TaskOutcome.SUCCESS

    class UrlVerifierPluginFunctionalTest extends Specification {
        @TempDir File testProjectDir
        File buildFile

        def setup() {
            buildFile = new File(testProjectDir, 'build.gradle')
            buildFile << """
                plugins {
                    id 'org.myorg.url-verifier'
                }
            """
        }

        def "can successfully configure URL through extension and verify it"() {
            buildFile << """
                verification {
                    url = 'https://www.google.com/'
                }
            """

            when:
            def result = GradleRunner.create()
                .withProjectDir(testProjectDir)
                .withArguments('verifyUrl')
                .withPluginClasspath()
                .build()

            then:
            result.output.contains("Successfully resolved URL 'https://www.google.com/'")
            result.task(":verifyUrl").outcome == SUCCESS
        }
    }

#### IDE integration

TestKit determines the plugin classpath by running a specific Gradle task. You will need to execute the `assemble` task to initially generate the plugin classpath or to reflect changes to it even when running TestKit-based functional tests from the IDE.

Some IDEs provide a convenience option to delegate the "test classpath generation and execution" to the build. In IntelliJ, you can find this option under Preferences…​ > Build, Execution, Deployment > Build Tools > Gradle > Runner > Delegate IDE build/run actions to Gradle.