Android Library UI Tests & Firebase Test Lab

This article assumes prior knowledge about both UI testing and Firebase Test Lab. It isn't required to have a great depth of knowledge on either, but this article won't discuss how to setup a project on Firebase, which is a prerequisite for running any sort of Test Lab tests.

UI tests are an invaluable tool in ensuring that our Android applications are functioning properly and that new changes to our codebase aren’t breaking existing behavior. In today’s modular applications, we may even want to write UI tests per module that launches Fragments and Activities directly without having to do complicated screen interactions to test functionality.

One tool at our disposal when it comes to UI testing is Firebase Test Lab. It allows us to run our UI tests on both virtual and physical devices and can be integrated into CI/CD pipelines with the gcloud command-line interface.

Problems with Test Lab & Library Module Tests

An issue that you may face with Test Lab is trying to test library builds. Since Test Lab only supports .apk and .aab files, the library's generated .aar files cannot be uploaded to Test Lab, even when you try to upload them manually instead of via command line. Trying to do so will just give a general error:

Since the generated androidTest .apk file is used to run library module tests locally, it can be inferred that it contains both the regular library source code and the test code from the androidTest directory. UI tests added in the androidTest directory of a library module still generates a test APK file even when the library's regular output is an AAR file as shown below.

However, even if this APK contains the proper source code, just trying to upload the APK file twice as both the test and app APK results in an error from Test Lab stating that the “uploaded test APK is the same as the uploaded application APK”. This issue will also come via the command line interface. In other words, Test Lab must have 2 separate APK files in order to run the test. Specifically, it appears that the uploaded test and application APK files must have different package names.

One possible way to fix this issue would be creating a sample application module for our given library module. This works for standalone library projects but may be overly cumbersome in cases in which a large number of libraries are maintained or multi-module projects where there is a desire to keep the test cases in the same modules as the class-under-test.

A Simple Solution

Since Test Lab requires 2 separate test and app APK files, the androidTest APK can be built twice, with different package names each time. LibraryTestExample is a project I have created to show how this is done.

The first thing that needs to be done is to add the testApplicationId property to the module-level build.gradle file. This property will be used to update the package name of the instrumentation test APK creating when running instrumented tests. By default, if this property is not defined, the module's package name is appended with .test for the test APK. For example, for a module with package name me.jdvp.librarytestexample, the instrumented test APK will have the following package name: me.jdvp.librarytestexample.test.

For the property’s value, System.getProperty will be used:

android {
    defaultConfig {
        testApplicationId System.getProperty("testApplicationId", "me.jdvp.librarytestexample.test")

        //... other default config items
    }

By using System.getProperty, the testApplicationId can be dynamically changed when running the androidTest assemble tasks. System.getProperty takes two parameters itself. The first is the specified property key and the second is the default value if no system property is found. In the example, above me.jdvp.librarytestexample.test since it is the value that would have been used for the sample project if no testApplicationId value had been provided.

After making this change to build.gradle, running the command to assemble the test APK as shown below will generate an APK with a package name of me.jdvp.librarytestexample.test:

./gradlew assembleDebugAndroidTest

However, if using the following command in which the testApplicationId property is defined, the test APK will have a package name of me.jdvp.librarytestexample:

./gradlew assembleDebugAndroidTest -DtestApplicationId=me.jdvp.librarytestexample

Manually uploading these separate files to Test Lab should allow your tests to run!

Uploading via command line

By default, the gradle commands given above will generate files with the same name. That means that if you run the second command immediately after the first, you will only have 1 APK file remaining in the output directory.

One solution to this is to simply rename the file between each gradle command. If using this approach, the configuration can be handled by CI/CD scripts wherever they live without having to update the project’s gradle files any further.

As an example, the sample project leverages GitHub Actions to run the UI tests as a form of commit validation on pushes to master. The build script for this is commit_validation.yml.

Inside the script, the following steps are run which generate the test APK twice and rename it each time. The second time the assemble task is run, a new test application ID is provided in order to ensure distinct package names.

- name: Assemble test apk
  run: ./gradlew assembleDebugAndroidTest
- name: Rename test apk
  run: mv ./libtestingexample/build/outputs/apk/androidTest/debug/libtestingexample-debug-androidTest.apk ./test.apk
- name: Assemble app apk
  run: ./gradlew assembleDebugAndroidTest -DtestApplicationId=me.jdvp.librarytestexample
- name: Rename app apk
  run: mv ./libtestingexample/build/outputs/apk/androidTest/debug/libtestingexample-debug-androidTest.apk ./app.apk

Later, When running gcloud firebase test android to run the instrumentation tests, the renamed files are referenced.

- name: Run test lab tests
  run: |-
    gcloud firebase test android run \
      --type instrumentation \
      --app ./app.apk \
      --test ./test.apk \
      --results-history-name='libtestingexample results' \
      --device model=Pixel2,version=29,locale=en,orientation=portrait

An example of the commit_validation workflow running can be found here.

If the logic for renaming these APKs is to be placed in build.gradle rather than in CI/CD scripts, a potential solution could be renaming the file based on the same system property as used for the package name. This is not used for the sample project but might look like the following.

android {
    testVariants.all { testVariant ->
        testVariant.outputs.all { output ->
            outputFileName = System.getProperty("testApplicationId", "me.jdvp.librarytestexample.test") + "-androidTest.apk"
        }
    }
}

Wrapping Up

UI testing can be an important part of our development processes and allow us to ensure that our code is functioning properly. Although Firebase Test Lab doesn’t support testing library-module aar files out of the box, the workaround specified in this article can help test libraries on Firebase Test Lab when the need arises. If you have any comments or questions, feel free to leave them below. If you want to see this in action, take a look at the sample project!

Originally published at https://jdvp.me on September 6, 2021.