Mobile automation testing tips
Tuesday, February 7, 2017
|
Adam Croxen |
As mobile application development matures, the range of features increases, but so does the effort required to test for regressions before shipping updates. A robust automation testing strategy can help in this situation but it is important to look at some key factors when setting up test suite infrastructure.
An automated UI test suite can quickly run through all your app’s key scenarios, interacting with the device just as a user would, and give you confidence that your changes have not broken anything. Of course, this does not replace unit testing, which is a key part of any software development process. While your unit tests help you design and maintain your app’s low-level building blocks, automation tests are great for ensuring everything is integrated correctly.
Automated UI tests can save your team hours of repetitive testing across devices with different form factors, screen sizes and OS levels.
Yet there are many choices and decisions to make when setting up a implementing a comprehensive automation suite. And with such a broad landscape of test tools and frameworks to choose from, each with their own pros and cons, even getting started can be difficult.
Before starting, it’s worth thinking about some aims for how your test suite will work, and how developers and testers will interact with it.
Most technical leads will be familiar with the challenge of getting developers into the habit of making code testable. Therefore, top of your list of objectives should be “ease of use”. Tests should be easy to write, and more importantly, trivially easy to run, by your development team. Ideally, all developers should be able to run all tests, in a single click, right from in their IDE. No excuses!
A key decision then is the choice of tooling. Automated test tooling falls broadly into two categories: the platform’s native automation tooling, and cross-platform “wrapper” frameworks. Native tooling is supported directly by Apple and Google, and requires a level of expertise in each. Wrapper frameworks, meanwhile, add layers of abstraction but communicate with native tools under the hood.
Appium and Calabash are two popular “wrapper” frameworks which run on both iOS and Android. Tests are written in the industry-standard Cucumber “Given/When/Then” form, which will be familiar to all agile team members, technical or not. Cucumber test steps then need to be implemented, and this is done by writing code to drive interactions with the app.
In theory, these frameworks allow a single test suite to run on both iOS and Android. This sounds very appealing, however in practice you are likely to find that differences in your UI layout between platforms will mean this is unrealistic.
These frameworks also have several moving parts to build, configure and maintain, such as web servers and custom app builds, making it a difficult process to set the tests up for developers, testers and on your continuous integration server.
Finally, tests on wrapper frameworks are notoriously flaky and slow to run, with arbitrary “waits” required throughout your test code to allow the app to catch up with the test tools.
Native test tools on iOS and Android have really come of age in their latest generations. Appium or Calabash may have been a fine choice in the early days of mobile development, when native options were less attractive. But XCUITest for iOS and Espresso for Android are robust, well-supported and flexible.
Tests run right inside Xcode and Android Studio alongside your existing unit tests, and can be run easily on devices or emulators, by developers or on your Jenkins server. Native tests also tend to be faster to run than wrapper frameworks.
Finally, they offer greater scope for flexibility as we will see in the next section.
So much for test tooling. Your next consideration should be ensuring your app’s dependencies can be stubbed and mocked easily. Many apps rely on a high level of interaction with a server, which is fine (but slow) if you are testing general read-only scenarios.
However, consider the situation when you are testing server error conditions. Or perhaps your app has a user registration and login flow. How can your app create a fresh user in an automated test every time? You are likely to want to run your tests on many devices simultaneously.
My mantra here is: remember that you’re testing your application, not your server.
The solution is to stub your app’s network data: specifying within your test code exactly what response you want to network calls made by your app. On iOS, we have had great success with our XCUITests using the technique described in this blog.
Over on Android, Espresso’s flexibility means a wide range of options is available. When testing projects which use our cross-platform app framework, Kirin, code to stub the network on Android automated tests can be shared easily with unit tests created for the app’s core logic.
So, you’ve selected your test technology, and you’re writing your tests to be flexible and independent of any external conditions. Great! Now, full steam ahead and get your tests in place to cover all your high-impact app scenarios.
There are many more bells and whistles you can add to get even more value from your tests (cloud testing services, for example). And while it’s important to make the right choices, it’s equally important not to become paralyzed by choice; the sooner you start, the sooner you’ll reap the benefits.
This content is made possible by a guest author, or sponsor; it is not written by and does not necessarily reflect the views of App Developer Magazine's editorial staff.
An automated UI test suite can quickly run through all your app’s key scenarios, interacting with the device just as a user would, and give you confidence that your changes have not broken anything. Of course, this does not replace unit testing, which is a key part of any software development process. While your unit tests help you design and maintain your app’s low-level building blocks, automation tests are great for ensuring everything is integrated correctly.
Automated UI tests can save your team hours of repetitive testing across devices with different form factors, screen sizes and OS levels.
Decisions, Decisions
Yet there are many choices and decisions to make when setting up a implementing a comprehensive automation suite. And with such a broad landscape of test tools and frameworks to choose from, each with their own pros and cons, even getting started can be difficult.
Before starting, it’s worth thinking about some aims for how your test suite will work, and how developers and testers will interact with it.
Tooled Up
Most technical leads will be familiar with the challenge of getting developers into the habit of making code testable. Therefore, top of your list of objectives should be “ease of use”. Tests should be easy to write, and more importantly, trivially easy to run, by your development team. Ideally, all developers should be able to run all tests, in a single click, right from in their IDE. No excuses!
A key decision then is the choice of tooling. Automated test tooling falls broadly into two categories: the platform’s native automation tooling, and cross-platform “wrapper” frameworks. Native tooling is supported directly by Apple and Google, and requires a level of expertise in each. Wrapper frameworks, meanwhile, add layers of abstraction but communicate with native tools under the hood.
Wrappers Unwrapped
Appium and Calabash are two popular “wrapper” frameworks which run on both iOS and Android. Tests are written in the industry-standard Cucumber “Given/When/Then” form, which will be familiar to all agile team members, technical or not. Cucumber test steps then need to be implemented, and this is done by writing code to drive interactions with the app.
In theory, these frameworks allow a single test suite to run on both iOS and Android. This sounds very appealing, however in practice you are likely to find that differences in your UI layout between platforms will mean this is unrealistic.
These frameworks also have several moving parts to build, configure and maintain, such as web servers and custom app builds, making it a difficult process to set the tests up for developers, testers and on your continuous integration server.
Finally, tests on wrapper frameworks are notoriously flaky and slow to run, with arbitrary “waits” required throughout your test code to allow the app to catch up with the test tools.
Native Speaker
Native test tools on iOS and Android have really come of age in their latest generations. Appium or Calabash may have been a fine choice in the early days of mobile development, when native options were less attractive. But XCUITest for iOS and Espresso for Android are robust, well-supported and flexible.
Tests run right inside Xcode and Android Studio alongside your existing unit tests, and can be run easily on devices or emulators, by developers or on your Jenkins server. Native tests also tend to be faster to run than wrapper frameworks.
Finally, they offer greater scope for flexibility as we will see in the next section.
Inject Those Dependencies
So much for test tooling. Your next consideration should be ensuring your app’s dependencies can be stubbed and mocked easily. Many apps rely on a high level of interaction with a server, which is fine (but slow) if you are testing general read-only scenarios.
However, consider the situation when you are testing server error conditions. Or perhaps your app has a user registration and login flow. How can your app create a fresh user in an automated test every time? You are likely to want to run your tests on many devices simultaneously.
My mantra here is: remember that you’re testing your application, not your server.
The solution is to stub your app’s network data: specifying within your test code exactly what response you want to network calls made by your app. On iOS, we have had great success with our XCUITests using the technique described in this blog.
Over on Android, Espresso’s flexibility means a wide range of options is available. When testing projects which use our cross-platform app framework, Kirin, code to stub the network on Android automated tests can be shared easily with unit tests created for the app’s core logic.
Get Testing!
So, you’ve selected your test technology, and you’re writing your tests to be flexible and independent of any external conditions. Great! Now, full steam ahead and get your tests in place to cover all your high-impact app scenarios.
There are many more bells and whistles you can add to get even more value from your tests (cloud testing services, for example). And while it’s important to make the right choices, it’s equally important not to become paralyzed by choice; the sooner you start, the sooner you’ll reap the benefits.
This content is made possible by a guest author, or sponsor; it is not written by and does not necessarily reflect the views of App Developer Magazine's editorial staff.
Become a subscriber of App Developer Magazine for just $5.99 a month and take advantage of all these perks.
MEMBERS GET ACCESS TO
- - Exclusive content from leaders in the industry
- - Q&A articles from industry leaders
- - Tips and tricks from the most successful developers weekly
- - Monthly issues, including all 90+ back-issues since 2012
- - Event discounts and early-bird signups
- - Gain insight from top achievers in the app store
- - Learn what tools to use, what SDK's to use, and more
Subscribe here
