Write Once, Test Everywhere: Cross-Library Screenshot Testing with AndroidUiTestingUtils 2.0.0

As observed in the previous blog post of this series, screenshot tests have a lot in common regardless of the library they're written with, namely:

1. Setting the device configuration (e.g. Locale, Ui Mode, Font scale...)

2. Setting the Subject Under Test (SUT): e.g. invoking the composable

3. Recording/Verifying the screenshot of the SUT.

In this blog post, we'll see, in broad terms, how AndroidUitestingUtils 2.0.0 uses those findings to support writing cross-library screenshot tests.

Defining a common interface

So we'll define a common interface for setting up the device configuration and the SUT, as well as for recording/verifying.

This interface must have no direct dependency on any screenshot library: we don't want cross-library screenshot tests to depend on libraries we might never use.

// 1: Define the common device configuration.
data class ScreenshotConfigForComposable(
   val locale: String? = null,
   val fontScale: FontSize? = null,
   ...
)

// 1: Pass common device configuration. 
// To be set by each library internally
abstract class ScreenshotTestRuleForComposable( 
   open val config: ScreeenshotConfigForComposable,
): TestWatcher() {

    // 2 & 3: Set the SUT & Record/verify snapshots
    abstract fun snapshot(
        name: String? = null,
        composable: @Composable () -> Unit,
    )
}

Implementing the interface for each library

Now we'll implement that interface for Paparazzi, Dropshots & Roborazzi ¹.

To maintain dependency separation, each implementation is enclosed within its own module, e.g. :paparazzi, :dropshots, :roborazzi, etc. and only depends on its corresponding screenshot library — :paparazzi on Paparazzi, :dropshots on Dropshots, etc., — ensuring no cross-dependencies among them.

¹ AndroidUiTestingUtils provides these implementations + one for Shot

Paparazzi

To access Paparazzi's classes, we declare its dependency in the gradle file of :paparazzi module.

dependencies {
    api 'app.cash.paparazzi:paparazzi:1.3.1'
}

And use them in our TestRule

class PaparazziScreenshotTestRuleForComposable( 
   override val config: ScreeenshotConfigForComposable,
): ScreenshotTestRuleForComposable(config) {
    // 1. Set the device configuration 
    private paparazzi: Paparazzi by lazy {
        // Class in AndroidUiTestingUtils that 
        // converts screenshotConfigForComposable
        // into something that Paparazzi understands  
        PaparazziForComposableTestRuleBuilder()
            .applyScreenshotConfig(config) 
            .build()
    } 

    override fun apply(
        base: Statement?, 
        description: Description?,
    ) : Statement =
        paparazzi.apply(statement, base)

    // 2 & 3: Set the SUT & record/verify snapshots
    override fun snapshot(
        name: String? = null,
        composable: @Composable () -> Unit,
    ){
        paparazzi.snapshot(name, composable)
    }  
}

Check out the full code here

Dropshots

Same as before, we declare Dropshots dependency in the gradle file of :dropshots module

dependencies {
    api 'com.dropbox.dropshots:dropshots:0.4.1'
}

class DropshotsScreenshotTestRuleForComposable( 
   override val config: ScreeenshotConfigForComposable,
): ScreenshotTestRuleForComposable(config) {
    // 1. Set the configuration via TestRule
    // This is provided by AndroidUiTestingUtils
    private val activityScenarioRule by lazy { 
        ActivityScenarioForComposableRule(
            config = config.toComposableConfig() 
        )            
    }

    // Used for recording/verifying
    private val dropshots: Dropshots by lazy { 
        Dropshots()
    } 

    // Ensure both rules apply with RuleChain
    override fun apply(
        base: Statement?, 
        description: Description?,
    ) : Statement = 
        RuleChain
            .outerRule(dropshots)
            .around(activityScenarioRule)
            .apply(base, description)
    }  

    // 2 & 3: Set the SUT & record/verify snapshots
    override fun snapshot(
        name: String? = null,
        composable: @Composable () -> Unit,
    ){
        // 2: Set the SUT and return it as ComposeView
        val composeView =
            activityScenarioRule
                .setContent(composable)
                .composeView

        // 3: Record/verify the ComposeView
        dropshots.assertSnapshot(
            view = composeView, 
            name = name,
        )
    }  
}

Check out the full code here

Roborazzi

Analogue to Paparazzi and Dropshots

dependencies { 
    api 'io.github.takahirom.roborazzi:roborazzi:1.5.0-rc-1' 
}

class RoborazziScreenshotTestRuleForComposable( 
   override val config: ScreeenshotConfigForComposable,
): ScreenshotTestRuleForComposable(config) {
    // 1. Set the configuration via TestRule
    // This is provided by AndroidUiTestingUtils
    // and contains a ComposeRule too
    private val activityScenarioRule by lazy { 
        RobolectricActivityScenarioForComposableRule(
            config = config.toComposableConfig() 
        )            
    }

    private val filePathGenerator: FilePathGenerator = 
        FilePathGenerator()

    override fun apply(
        base: Statement?,
        description: Description?,
    ): Statement =
        activityScenarioRule.apply(base, description)

    // 2 & 3: Set the SUT & record/verify snapshots
    override fun snapshot(
        name: String? = null,
        composable: @Composable () -> Unit,
    ){
        val screenshotPath = 
            filePathGenerator(
                parent = DEFAULT_ROBORAZZI_OUTPUT_DIR_PATH,
                fileName = name,
            )

        // 2: Set the SUT
        activityScenarioRule
            .setContent(composable)

        // 3: Record/verify with the ComposeRule
            .composeRule
            .onRoot()
            .captureRoboImage(
                filePath = screenshotPath,
            )
    }  
}

Check out the full code here

Unfortunately, that's not enough to support Roborazzi. We still need to address 2 points:

1. Enable Robolectric Native Graphics mode

2. Use the RobolectricTestRunner

Supporting Robolectric Native Graphics (RNG)

The documented way to enable RNG is through Robolectric's @GraphicsMode(NATIVE) annotation.

That means, cross-library tests would require this annotation even though Roborazzi or any other RNG-based libraries aren't used. All tests would depend on Robolectric.

This is suboptimal, and a better solution exists: one can enable RNG via gradle¹ instead, like this

testOptions {
    unitTests.all {
        systemProperty 'robolectric.graphicsMode', 'NATIVE'
    }
}

In doing so, we no longer require the @GraphicsMode(NATIVE) annotation, avoiding a direct dependency on Robolectric.

¹ Beware this enables RNG for all your Robolectric tests, though

Supporting RobolectricTestRunner

The need for the RobolectricTestRunner creates a direct dependency on Robolectric in common code.

Google's AndroidJUnitRunner supports shared tests: tests that can run on both the JVM and emulators/physical devices.
For that, it detects in which environment the tests run, and delegates to the RobolectricTestRunner on the JVM and to the AndroidJUnit4ClassRunner otherwise.

Interestingly, it uses reflection to sidestep a direct dependency on RobolectricTestRunner and, therefore, Robolectric.

Thus, the issue is that we cannot use the AndroidJUnitRunner directly because Paparazzi is not compatible with the RobolectricTestRunner ¹.

Thus, we need to customize Google's AndroidJUnitRunner accordingly:

1. On the JVM

   1. With Paparazzi, use the standard JUnit4 TestRunner BlockJUnit4ClassRunner

   2. With Roborazzi, use the RobolectricTestRunner

2. On-device

   1. Use the AndroidJUnit4ClassRunner

So we create our own CrossLibraryScreenshotTestRunner based on Google's AndroidJUnitRunner, and tweak the necessary parts as follows:

private static boolean runsOnJvm() {
    return !System.getProperty("java.runtime.name")
            .toLowerCase()
            .contains("android")
}

private static String getRunnerClassName() {
    ...
    if (runsOnJvm()) {
        ...
        // If the project has Paparazzi as a dependency, we
        // assume it is used for JVM screenshot testing:
        // Robolectric could be used for standard unit tests too.
        if (hasClass("app.cash.paparazzi.Paparazzi")) {
            return "org.junit.runners.BlockJUnit4ClassRunner";
        }
        if (hasClass("org.robolectric.RobolectricTestRunner")) {
            return "org.robolectric.RobolectricTestRunner";
        }
    }
    return "androidx.test.internal.runner.junit4.AndroidJUnit4ClassRunner";
}

Links with the full code below:
- Google's AndroidJUnitRunner
- AndroidUiTestingUtils CrossLibraryScreenshotTestRunner

This TestRunner belongs to cross-library common code, but we circumvent direct dependencies on Robolectric and Paparazzi through reflection.

Now, we can make cross-library screenshot tests work by using @RunWith(CrossLibraryScreenshotTestRunner::java.class)

¹ Paparazzi is not compatible with RobolectricTestRunner #1089, #425

Deciding which library executes the tests

So far we've seen that, to achieve cross-library screenshot tests, we need:

1. An implementation of the ScreenshotTestRuleForComposable for each library we want to use

2. Use @RunWith(CrossLibraryScreenshotTestRunner::java.class)

The last step is to create a special ScreenshotTestRuleForComposable that enables us to choose which of the previous ScreenshotTestRuleForComposable implementations we delegate to.

The ScreenshotLibraryTestRuleForComposable of AndroidUiTestingUtils simplifies that by encapsulating the delegation boilerplate code and providing the getScreenshotLibraryTestRule() to decide the ScreenshotTestRuleForComposable that executes the tests under the hood

class MyScreenshotLibraryTestRuleForComposable(
    override val config: ScreenshotConfigForComposable,
) : ScreenshotLibraryTestRuleForComposable(config) {

    fun getScreenshotLibraryTestRule(
        config: ScreenshotConfigForComposable,
    ): ScreenshotTestRuleForComposable =
        // A ScreenshotTestRuleForComposable that we'll
        // use to run our screenshot tests, e.g.
        roborazziTestRuleForComposable
}

Check out ScreenshotLibraryTestRuleForComposable full code here

Let's see how to use it!

A cross-library screenshot test example

With all the pieces in place, writing a cross-library screenshot test is as easy as this

// Required due to RNG-based libraries, e.g Roborazzi 
@RunWith(CrossLibraryScreenshotTestRunner::class)
class CrossLibraryScreenshotTest {
   @get:Rule
   val rule =
      // This TestRule also chooses
      // the library to be used 
      MyScreenshotLibraryTestRuleForComposable(
          config = ScreenshotConfigForComposable(
              locale = "es",
              fontScale = FontSize.HUGE,
              ...
        )
   )

   @Test
   fun snapComposable() {
      rule.snapshot("myComposable") {
          AppTheme { MyComposable() }
      }
   }
}

To execute the test, run the corresponding record/verify commands of the library that MyScreenshotLibraryTestRuleForComposable picks under the hood.

But, what if I want to use another screenshot library?

Executing tests with another library

To execute your screenshot tests with a distinct library, follow these steps¹:

1. Configure the plugin of the new library as described in its README.md.

2. Implement ScreenshotTestRuleForComposable with that library or import the module(s) from AndroidUiTestingUtils that contain it.

3. Make getScreenshotLibraryTestRule() inside ScreenshotLibraryTestRuleForComposable return the new ScreenshotTestRuleForComposable.

4. Ensure all cross-library screenshot tests are placed in the right test folder:

   1. unitTests (JVM tests, e.g. for Paparazzi & Roborazzi)

   2. androidTests (on-device tests, e.g. for Dropshots & Shot)

That's it. As you can see, the screenshot tests don't need any rewriting!

¹ It's also possible to pick the library dynamically, so steps 3-4 wouldn't be required.
See the corresponding section in FAQ

Conclusions

Writing cross-library screenshot tests brings some benefits if:

1. You are brand new to screenshot tests and want to experiment with libraries to see which ones fit your project's needs the best

2. You've got hundreds of screenshot tests and want to migrate to another library/solution. This new library might not be what you need, and you don't want to go through the trouble of rewriting all those tests if it doesn't work out.

We've seen that providing a cross-library solution comes with its difficulties, like supporting RNG-based libraries like Roborazzi.

Thankfully, AndroidUiTestingUtils tackles these challenges for us, so that we can concentrate on the writing of cross-library tests instead of worrying about choosing the "right" library.

FAQ

I don't use Jetpack Compose. What about the "old" Android View System?

AndroidUiTestingUtils also provides ScreenshotTestRuleForView using the very same concept.
Have a look at DropshotsScreenshotTestRuleForView and this test example to see how it is done.

What about supporting specific library configurations, like Paparazzi's rendering mode?

I've skipped libraries' configurations for simplicity's sake.

For on-device libraries, it's very straightforward, as you can see in DropshotsConfig and DropshotsScreenshotTestRuleForComposable

But Roborazzi and Paparazzi need extra caring if used in shared tests: dependencies on Paparazzi and Roborazzi in shared code (i.e. debugImplementation) will throw runtime errors when running shared tests on devices/emulators with e.g. Shot or Dropshots. AndroidUiTestingUtils uses extra mapper modules to define those libraries' config, for instance

dependencies {
    // :mapper-paparazzi in shared test code (debug) 
    // :paparazzi only in JVM (test)
    debugImplementation 'com.github.sergio-sastre.AndroidUiTestingUtils:mapper-paparazzi:2.0.0'
    testImplementation 'com.github.sergio-sastre.AndroidUiTestingUtils:paparazzi:2.0.0'
}

Check out PaparazziConfig in :mapper-paparazzi and PaparazziForComposableTestRuleBuilder in :paparazzi for further details.

What if I want to use several libraries in the same project/module? How can I pick one library dynamically?

That's likely the case for shared screenshot tests: run them locally with Roborazzi/Paparazzi for fast feedback but with Shot/Dropshots on CI for solid testing.

To use 1 JVM and 1 on-device library, check SharedScreenshotLibraryTestRuleForComposable as well as this test rule, this gradle file and this example.

It's also possible to use 2+ JVM or 2+ on-device libraries, but you'll need to run the record/verify command together with a custom gradle property that selects the ScreenshotTestRule used to execute the screenshot tests at runtime. Check out this test rule, this gradle file and this example for more info.

What about parameterized cross-library screenshot tests?

It's possible too. However, RNG-based libraries need to run with ParameterizedRobolectricTestRunner. This TestRunner uses @ParameterizedRobolectricTestRunner.Parameters instead of the standard @Parameterized.Parameters from JUnit4, what makes it strongly coupled to Robolectric.
Nevertheless, AndroidUiTestingUtils also solves this issue.
Just use @RunWith(ParameterizedCrossLibraryScreenshotTestRunner::class) together with JUnit4 @Parameterized.Parameters and you're done. You can check it out here

What if I use my own screenshot testing library? How to support it?

1. Provide an implementation of the ScreenshotTestRuleForComposable that uses your own library

2. Make ScreenshotLibraryTestRuleForComposable return that implementation in its getScreenshotLibraryTestRule(config: ScreenshotConfigForComposable)

And you're done!

Further resources

👨‍🏫 Mastering Screenshot Testing Libraries for Android: A Hands-on Workshop(Droidcon Berlin 2023) - Workshop analysing pros and cons of many libraries

📸 AndroidUiTestingUtils - A library with utilities to fully support Cross-Library screenshot testing.

🛝 Android screenshot testing playground - Screenshot test examples with different libraries, as well as Cross-library screenshot tests for Compose and Android Views.

✍️ Blog post series: The road to effective snapshot testing - Several articles to introduce screenshot testing and its benefits, as well as solutions to problems you might come across.

Special thanks to Pablo Gallego for reviewing this blog post!

If you are interested in the content I publish, follow me on Twitter or LinkedIn!