A poor tester blames his tools

We've all heard that an artist is only as good as his tools. But does that ring true for app testing? The shear number of application testing products may seem overwhelming at first, but finding the most efficient environment for your app can be the difference between success and failure.

Asaf Saar is the Director of Product Management at Sauce Labs, a company that develops automated testing solutions for mobile and web. We recently had a discussion with Asaf about the importance of app testing within the correct environment.

ADM: Which came first: emulators, simulators or real devices?

Saar: Our initial offering in 2008 was a cloud-based Selenium grid for automated cross-browser testing across browsers on Windows and Mac OS. Interest in mobile had reached a fever pitch by 2012, so we focused our attention there and released our first Android emulators so that developers could perform automated testing at scale. Then, with the release of Appium in late 2012, developers had access to technology to easily automate mobile testing for native and hybrid apps. Sauce was quick to adopt Appium and provided support in early 2013. Today, we offer over 150 Android emulators and iOS simulators on our cloud along with over 1000 physical devices.

Emulators and simulators were developed by Google and Apple so that developers could easily test apps on their respective platforms. Before emulators and simulators were available, developers could only test on the actual phones they had in their possession - so it was difficult/impossible to scale testing.

ADM: Can you provide a brief description of each?

Saar: An emulator is designed to take the place of a physical mobile device and duplicates every aspect of the original device’s hardware and software behavior.

While a simulator sets up a similar safe environment to the original real-life device’s OS, it doesn’t attempt to simulate the real device’s hardware. So what you will see is the OS and the interface of the device you want to use, but you won’t experience all the problems the hardware might cause. Some apps may run a little differently, which is why final testing on real iOS devices is so important.

ADM: When did Sauce first identify the need for a healthy mix of all environments?

Saar: With emulators and simulators you can’t test hardware specifics like a camera, memory or SMS. You ultimately need to test on real devices at the end of your test cycle to ensure optimal performance.

ADM: How are your competitors approaching these environments? Are they prioritizing one method above all others?

Saar: No, most are just focusing on real devices - but they are more expensive.

ADM: What is the purpose of each - emulators, simulators and real devices - and in what scenario does each best perform?

Saar: Emulators and simulators are best when you need massive concurrency, as you can spin up an unlimited amount of virtual instances. They also allow you to reduce build times as they are immediately available. And, of course, they are much cheaper and allow you to save costs.

Real devices are best when you need a variety of device types for panel, display and compatibility testing purposes. They also allow you to replicate an issue to match exact model as reported by your users. You have no choice but test on real devices if you need to test hardware dependencies like CPU, memory, display, GPS and carrier network-based testing like real phone calls and text messages

ADM: Why are they needed to achieve automated and continuous testing?

Saar: They each fit a different scenario and are useful at a different points in the development process. A mix of all three is the only possible way to achieve automated and continuous testing that is both effective and cost sensitive.