Testing on Android using JUnit 5

JUnit 4 has always been the standard unit testing framework for Android development, however, a production version of JUnit 5 has been available for quite a while. JUnit 5 is the next generation of JUnit testing and has quite a new structure. From a user-perspective it offers a number of useful new features and various other benefits. Often Android developers are used to waiting a while for new things from the Java-world to be an option on Android. In the case of JUnit 5, with some changes to your Gradle dependencies, you can use it right now from your Java and Kotlin unit tests.

The architecture of JUnit 5 is quite a significant change from before, with the framework being split into three major components.

• JUnit Platform is the underlying layer on which testing frameworks can run on the JVM and offers up the API for different test engines.
• JUnit Jupiter defines how we write JUnit 5 tests and then contains an engine for running these tests on the platform.
• JUnit Vintage gives us an engine for running our previous JUnit 4 tests. We don't need to worry about having to update all of our tests in one go, as we can use the vintage engine to run these and have both JUnit 4 and 5 tests within our project.

As users of the testing framework, it's likely that the most interesting part is the new features we get access to. Nested tests allow us to put our tests into groups, increasing readability and allowing us to reduce repetition within test names. Parameterised tests are really powerful and can also reduce duplicating tests to give different inputs, the parameters can even be provided through a variety of mechanisms. Dynamic tests offer an API to generate tests on the fly using a test factory, rather than hard-coding each of the tests.

Parameterised tests run tests with different inputs

The JUnit 5 extension model allows us to extend the test framework and is the evolution of concepts from JUnit 4 such as @Rule and custom runners. The extension model will allow JUnit 5 to evolve with all developers and tool makers having the same public extension API. There are many more features beyond those listed here, all of which can be found in the user guide.

Now, let's get JUnit 5 running in our Android project.

Configuring Gradle

Thanks to Marcel Schnelle (mannodermaus) there is an easy-to-use Gradle plugin that will configure all the testing tasks to use JUnit 5. The plugin has a few minimum requirements that our project needs to meet, at the time of writing this is the Android Gradle plugin 3.2.0+, Gradle 4.7+ and Java 8+.

We will start by adding the plugin to our root build.gradle file using the latest version, which at time of writing is 1.3.2.0.

→ /build.gradle

buildscript {
  dependencies {
    classpath "de.mannodermaus.gradle.plugins:android-junit5:1.3.2.0"
  }
}

After a successful Gradle sync, we will be ready to apply the plugin to any Android modules we wish to use JUnit 5 in.

→ /app/build.gradle

apply plugin: 'de.mannodermaus.android-junit5'

dependencies {
  testImplementation "org.junit.jupiter:junit-jupiter-api:5.3.2"
  testRuntimeOnly "org.junit.jupiter:junit-jupiter-engine:5.3.2"
  testImplementation "org.junit.jupiter:junit-jupiter-params:5.3.2"
}

If there are JUnit 4 tests in the project we are configuring, then we will also need to keep the JUnit 4 dependency and enable the JUnit vintage test engine mentioned above.

testImplementation "junit:junit:4.12"
testRuntimeOnly "org.junit.vintage:junit-vintage-engine:5.3.2"

Once the plugin and the Gradle dependencies have been configured, we can run tests in the same way as we were previously using the same Gradle tasks and run configurations in Android Studio.

If we need them, there are some options for configuring the test environment within Gradle. Our settings go in a junitPlatform block within the existing testOptions, where we may already be doing some configuration. The most common use-case would be for filtering which tests are executed. We could use the filters to divide our tests into unit and integration and then have them run for different variants, as an example.

android {
  testOptions {
    junitPlatform {
      filters {
        includePattern "^(tests_to_include_regex)$"
        excludePattern "^(tests_to_exclude_regex)$"

        includeTags "slow"
        excludeTags "integration"
      }
      debugFilters {
        includeTags "integration"
      }
    }
  }
}

For any more information on the plugin, all the setup and usage instructions can be found on GitHub.

JVM tests only

If our unit tests are only going to be ran on the JVM, no Robolectric or Android required, then we can get away without the Gradle plugin. To do this we can remove the plugin classpath entry from the root build.gradle and the line where the plugin is applied within the app module build.gradle. Instead we need to add some configuration to testOptions to enable running with JUnit 5.

→ /app/build.gradle

android {
  testOptions {
    unitTests.all {
      useJUnitPlatform()
    }
  }
}

We lose the per-variant filters that were demonstrated above if we go down this route. However, for most Android projects that aren't using Robolectric this should definitely be sufficient.

Writing the tests

Let's write some tests to explore the new features of JUnit 5 and how we should go about using it in our apps. We will be using AssertJ for our assertions, mainly for readability within the samples. We will be testing a very simple repository that allows us to look up contacts using their IDs, starting with the JUnit 4 version.

import org.assertj.core.api.Assertions.assertThat
import org.junit.Test

class ContactsRepositoryTest {
  private val repository = ContactsRepository()

  @Test
  fun findContact_givenExistingId() {
    val id = 2
    val expectedContact = Contact(id, "Melanie")

    val actualContact = repository.findContact(id)

    assertThat(actualContact)
        .isEqualTo(expectedContact)
  }
}

Converting this test to JUnit 5 is as simple as changing the import for @Test. Projects with many more tests, using more features of JUnit 4, may have more changes that are required. As mentioned above, we can start new tests in JUnit 5 and then either leave the JUnit 4 ones as is, or convert them gradually.

// Before
import org.junit.Test

// After
import org.junit.jupiter.api.Test

Descriptive test names

A common structure for tests is breaking them into given, when and then.

• The given clause involves setting up the conditions for the test, such as mocking or configuring the component we are about to test.
• Within the when clause, we execute the behaviour we are looking to verify.
• It is in the then clause that we check the behaviour is working correctly with any assertions.

This structure is fairly clear within the body of the test function, however, in the naming of the function some issues may start to arise.

@Test
fun givenContactsLoadedAndIdMatches_whenFindContact_thenContactReturned()

@Test
fun givenContactsNotLoadedAndIdMatches_whenFindContact_thenNoContactReturned()

Some approaches could be taken to reduce the length, such as using shorter wording in the given clauses and potentially removing the then clause from the name. This does result in losing some detail, which can be really useful in test names to quickly see in the test report what is going on and to demonstrate what a test is supposed to do. It is also much easier to read a descriptive sentence than the camel-case seen in function names.

JUnit 5 introduces the @DisplayName annotation, which can be used to provide a descriptive name for the test report. The same result can be achieved in Kotlin through the use of back-ticks around the function name. The advantage of the @DisplayName approach is that we keep the searchable and familiar function name, whereas the back-tick approach avoids the need to maintain both the function name and annotation.

@Test
@DisplayName(
  "Given contacts are loaded but the ID is invalid, When we find a contact, Then no contact is returned."
)
fun findContact_givenExistingId()

@Test
fun `Given contacts are loaded but ID invalid, When we find a contact, Then no contact returned`()

Structuring tests

A common reason for test names growing in length is having multiple tests with the same starting condition. Beyond naming, a similar starting condition can cause test bodies to grow in length as well! To solve the issue, JUnit 5 introduces @Nested, allowing you to group a set of tests into an inner class. The group can have a shared starting state and can also have a display name specified, reducing the length of individual test function names. The nested structure is shown within the test report, making it very readable.

@Nested
@DisplayName("Given valid contact ID")
inner class ValidContactId  {
  private val id = 2

  @Test
  @DisplayName("When find contact, then correct contact returned")
  fun findContact_givenExistingId() {
    val expectedContact = Contact(id, "Melanie")

    val actualContact = repository.findContact(id)

    assertThat(actualContact)
        .isEqualTo(expectedContact)
  }

  ...
}

After fleshing out the tests for this repository more, with a few tests in each group, we can peek at the test report to see how it all looks. 👀

The code we have written is available on GitHub.

Instrumentation tests

We have been discussing unit tests only and as we all know instrumentation tests, usually using Espresso, are also very important. Currently, our Espresso tests will be using JUnit 4, which can be left as it was previously. This means using the JUnit 4 test runner, as opposed to adding the JUnit 5 vintage engine and using the new infrastructure.

Instrumentation tests using the JUnit 5 Gradle plugin are still considered experimental

The Gradle plugin we used to introduce JUnit 5 on Android has support for instrumentation tests, which at the time of writing is considered experimental. From checking the GitHub issues page for the plugin, many people have had problems using JUnit 5 for their instrumentation test suite. When the situation changes this article will be updated with the current details of using it.

Since JUnit 5 is built upon Java 8, to use it we need a minSdkVersion of 26 or above. Before we start freaking out, this can be achieved by running the tests with a product flavour that has the minimum SDK increased to this level.

To write and run our instrumentation tests using the JUnit 5 framework, we will need some Gradle configuration. A little bit more is required than for unit tests, due to needing to tell the system to to use the plugin's runner builder and ensuring tests running on the device work correctly.

→ /app/build.gradle

android {
  defaultConfig {
    testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
    testInstrumentationRunnerArgument "runnerBuilder", "de.mannodermaus.junit5.AndroidJUnit5Builder"
  }
}

dependencies {
  androidTestImplementation "org.junit.jupiter:junit-jupiter-api:5.3.2"
  androidTestImplementation "de.mannodermaus.junit5:android-instrumentation-test:0.2.2"

  androidTestRuntimeOnly "org.junit.jupiter:junit-jupiter-engine:5.3.2"
  androidTestRuntimeOnly "org.junit.platform:junit-platform-runner:1.3.2"
  androidTestRuntimeOnly "de.mannodermaus.junit5:android-instrumentation-test-runner:0.2.2"
}

It is noteworthy that instrumentation tests tend to have less need for JUnit 5, due to there being less tests and fewer variants of similar tests, in general. This will of course depend heavily from project to project though! To find out more, please check out the plugin's GitHub page.

Wrap up

We have only considered a few small aspects of writing JUnit 5 tests, as there is simply so much to look at. To find out more, there is a detailed user guide covering all parts of the framework and how to go about writing tests and customising the test process.

By using JUnit 5 in our Android app testing we are using the latest evolution of the JUnit framework. It gives us access to more features, helps us make tests as readable as possible, reduce duplication within our tests and extending the test framework has never been easier. Hopefully it will catch on more in the Android community and become the "standard" for Android development sooner rather than later.

I hope the article was useful. If you have any feedback or questions please feel free to reach out.

Thanks for reading!