Android Test Butler WhiteGlove Service for Automated Mobile Tests

At LinkedIn, automated testing is a key component of our new strategy for releasing mobile applications with greater frequency. As we developed the new LinkedIn Android app that we launched late last year, we found that our tests had a major problem: our testing environment itself was unreliable, so our tests failed intermittently. 

We needed a solution that would let us rely on our tests to inform us when there was a problem with the app. In short, we wanted our engineers to focus on worrying about finding bugs in our apps, not the testing environment. 

Many developers probably know from experience that UI testing on Android can be unstable for a variety of reasons: tests could depend on being run in a certain order, or depend on some shared application state that may not be reset between tests. These types of flakiness are largely up to app developers to address, but the more insidious issues can occur within the Android device itself.

If you’ve ever tried to run a large number of UI tests on Android, you may be familiar with some of the ways the emulator can be unreliable. Animations must be disabled, so that Espresso tests can run reliably. Tests may fail because the emulator CPU randomly goes to sleep, WiFi turns off unexpectedly, or rogue accelerometer data causes the device to change orientations. At LinkedIn, we’ve even seen cases where the lock screen on the emulator is randomly triggered, causing tests to fail. 

This inconsistent behavior was causing our developers to lose trust in our tests and question why we were even writing them in the first place.

For this reason, we created and open sourced Test Butler, a reliable Android testing tool. LinkedIn runs over one million tests each day using Test Butler and we believe that it can provide a benefit to anyone running Android tests. 

Test Butler heavily inspired by the Google presentation “Going Green: Cleaning up the Toxic Mobile Environment” which outlines the tools that the Google testing team had developed to manage their own tests on Android. We were fortunate enough to meet up with members of this team to get their input on the direction we were taking as we developed Test Butler. 

Test Butler includes an Android library that your test code can depend on, as well as a companion Android app APK that you can install on your Android emulator before running tests. It provides a number of features that will make your testing safer and more reliable:

- Signature-level permissions: Test Butler is signed with the system keystore for the stock Android emulator, which means it will automatically be granted any signature-level permissions it requests without the need to grant those permissions through Android Debug Bridge (ADB).

- Disable crash & App Not Responding (ANR) dialogs: Since it’s a system app, Test Butler can install a custom IActivityController on the emulator and suppress system crash & ANR dialogs from appearing and causing test failures.

- Disable animations: Test Butler handles all the steps of disabling animations with a single line of code in your app!

- Lock screen, WiFi, CPU: Test Butler holds locks on the keyguard, WiFi radio, and CPU to ensure they won’t go to sleep and cause test failures.

In addition to system stability enhancements, Test Butler can also help handle other testing tasks that require extra permissions for your app:

- Enable/disable WiFi: Test Butler allows tests to simulate a situation where WiFi is not connected or changes availability at some point.

- Change device orientation: Tests can manually set the orientation of the device during test execution.

- Set location services mode: Test Butler lets your code simulate different location services modes, like battery saver or high accuracy.

- Set application locale: Tests can set a custom Locale object for their application to simulate running the app in another language.

We’re happy to announce that Test Butler has been released under the Apache 2.0 license and the code is available on GitHub. Please take a look at our code, and let us know what you do and don’t like. Contributions and suggestions are welcome!