How mobile app testing has changed

Technologies are rapidly evolving and ways of testing them are changing too. But one thing remains true - product quality must be high. Therefore, it is important to have a clear vision on how you will guarantee that your product is working as planned. Depending on the project, you can build your own testing team, hire professional consultants or do a combination of both. But you should always remind yourself - Think Quality First. That's why we sat down with Andrejs Frisfelds, co-founder of TestDevLab, to learn more about how developers can skip the headache of testing and still ensure a consistent, high-quality product.

ADM: How has mobile testing changed in years?

Frisfelds: I would not say that testing has become harder or easier than it was 10 years ago. Back then we could test an application by just running it on one device model (as binaries were even built for one particular device model with Symbian or with Brew platforms) and rather quickly go through the critical use cases manually. Now we have to consider more device models, screen resolutions, OEM's, network conditions etc. However, now we can also automate majority of the critical cases and conditions as we have great tooling support for automation, professional crowd testing, beta testing or even UI/UX testing.

ADM: What are biggest challenges in this field today?

Frisfelds: Nowadays it is hard to pick the right amount of tests to execute and the right number and the correct device models to test on. It's also difficult to understand what are the priorities for issues and how likely real users will face them. From tooling and technology perspective biggest challenge is choosing the best framework for automation as well as how to distribute the application to our crowd or beta testers, how will we collect all the data and which data is actually meaningful. Today tooling allows you to collect so much information that it can become difficult to sift through it all.

ADM: What is the difference between testing apps on real devices and emulators?

Frisfelds: Testing on emulators - either cloud (Amazon, Genymotion etc.) or local, can speed up verification in development phase as developers do not need to purchase a lot of real devices. Automation can also be triggered and test execution is faster than on real devices. Emulators are also good for customer support teams if there are some reported UI glitches. More engaged support representatives can check UI themselves by using emulators without involving the  technical team to validate if indeed there is some global issue or user has some local/specific issue. In regards to real devices - applications requiring networking (even if it can be emulated), hardware acceleration and any other hardware support, can reliably be tested only on real devices. Besides hardware, many mobile operators are deploying custom OS versions on devices which means that you need a real device with this customized OS to test your application's behavior on. Furthermore, a real device, even if the test device has a lot of applications and specific setups, is not a “sterile” environment, which most probably will represent majority of your user base. Emulators provide an ideal environment, but it is a hard environment in which to tackle end user issues and try to achieve the level of their experiences on your end.

ADM: Which platform (Android or iOS) is harder from testing perspective?

Frisfelds: Both have their features that sometimes “help” in testing, debugging and development of applications. Everybody knows that if you talk about Android, you talk about many screen resolutions, OEM's, OS versions etc. Basically if you want to validate your application, it is not enough to just take one or two devices with the latest OS version and test it. On the other hand, Android is quite open as a platform that for testing/debugging purposes allows you to dig quite deep and understand the nature of the issues and you can figure out workarounds for majority of cases. iOS, however, is not so diverse in terms of OS and screen variety, because mainly the latest two iOS versions are supported and majority of iPhone users are using the newest or the second newest device model. However, there are issues you can not solve in your application as the environment is quite closed and if the platform does not allow you to, for example, pick up native dialer app, you are not left with many options (at least legal ones).

ADM: Which tools do you use for testing?

Frisfelds: Since our company specializes in testing and test automation, we use a variety of tools.

If our clients are already using certain solutions that works well, we do not push them to use something else. Instead we adapt tooling they are using currently. For example, for mobile UI automation we use Appium, Robotium and Calaba.sh frameworks. For backend test automation and load testing we use jMeter, Apimation.com, Postman as well as develop custom solutions. For professional crowd testing you can use our test service Test48.com as well as some other players in this field with a bit different approach such as Testbirds, Applause etc.

ADM: How can Test48.com help developers and businesses?

Frisfelds: We take off the burden of hiring professional testers from our clients. We position ourselves as a “Test team in the cloud”. Our main goal is to point to as many issues in your applications as possible in just 48 hours. This allows our customers to submit release candidates or work-in-progress applications to Test48.com. In a relatively short time we test the application and provide a report with issues and potential improvements. If we compare the results you get from our service to hiring a professional in-house test team or standard crowd test solution, Test48 can save a lot of time and resources.

ADM: What makes Test48 different from other crowd testing platforms?

Frisfelds: We do it in 48 hours (client can see the countdown on the dashboard) and focus on the project to provide feasible and rich reporting. All our testers are experienced ISTQB Certified professionals that will dig into your application to find bugs and issues. Bug and improvement reports are professionally formatted with all the necessary attachments and information for the development team to be able to reproduce, tackle and fix the issues.

ADM: Why should one confide testing to someone else than your own development team?

Frisfelds: When a developer tests their own code they do it so to prove that it works. If you give your solution for someone else to test they will do it so to prove that it does NOT work. Additionally, an external team does not know or care about your internal policies, personnel or the sacrifices you've made on quality; they will point out issues. Your team has to make a decision on the received facts, issues and potential improvements.

ADM: Is it possible to automate all your product's feature tests?

Frisfelds: It is. A more suitable question is - do you have all the necessary resources to maintain all that automation you have set up and developed to test all your features? Probably not. Automation requires deep understanding on how far you will go with feature/code coverage of your automation either for UI or integration/unit tests. There is no ideal decision or only one right answer. You have to find a solution that works in your case and commit to it.

ADM: Why demand for testing is growing so rapidly?

Frisfelds: There are several reasons. For example, one of our clients needed their Android application's release candidate tested before publishing it on Google Play store. Even though the client does their own integration and unit testing and they have generally good testers, we still found 25 total bugs of which 2 were critical and 10 were major issues. If they had published the application without us testing it first they would have had to:

- Fix the 2 critical issues, which would require another round of testing and releasing a hotfix and pulling two developers to work on it outside next release development;

- Fix the 10 major issues, which would require additional two developers working on them for four days and testers validating fixes and releases;

- Deal with growing user churn and users not upgrading to the newest version, which would hinder company’s reliability to deploy a stable product.

This extra work could have cost them a lot of money in engineering hours as well as their reputation as a good quality product. However, our service cost was $350 USD and, besides the bug list, they received a few quite interesting points on improvements.

This example highlights that companies despite their current development stage (startup, mature company etc.) realize the importance of getting information about issues early enough to fix them and save money and reputation in the long run.

Another reason is that testing services are becoming more available and anyone can get a reliable, good quality service very fast because SAAS/PAAS markets are growing in the testing sector as well. If a company/product team is not interested in the service, they can develop their own test solutions by using many different open source as well as paid platforms and solutions and make sure they reach acceptable quality levels of their application/solution before launching to market.

About Andrejs

Before co-founding TestDevLab he worked in quality assurance field at Skype for Mobile thin client deployment in co-operation with mobile operators and then consultancy for Libon product built by Orange S.A. Had pleasure and honor to teach Software Testing and Cybersecurity class for Master's degree students at Ventspils University College.