How to avoid flaky automated testing results
|Richard Harris in Application Testing Tuesday, November 28, 2017|
How developers can avoid flaking and flawed automated testing results with Sauce Labs' new Extended Debugging feature.
End-to-end tests measure readiness for deployment by ensuring that the application’s integrity is behaving as expected and systems communicate from start to finish. So, it’s no wonder that even a minor hiccup could throw a wrench in such a finely-tuned procedure.
Lubos Parobek is the VP of Product at Sauce Labs, a company that develops automated testing solutions for mobile and web. We recently had a discussion with Lubos on how its latest Extended Debugging platform helps to fight against failed and flaky tests.
ADM: What is Extended Debugging from Sauce Labs? Is this an extension of an existing service?
ADM: How does Extended Debugging differ from other tools on the market? Firefox (Firebug) and Google (Chrome Developer Tools) seem to offer similar benefits.
Parobek: Our closest competition (right now) is BrowserMob proxy. Conversely, Firebug and Chrome Developer Tools are native-browser extensions. This meaning they require developers to manually reproduce the issue and then deploy for information on what’s going on under the hood. With Extended Debugging, enabled customers can see the root cause of their failed/flaky automated tests without having to replicate the issue manually.
ADM: Is Extended Debugging effective in rectifying flaky tests?
Parobek: Yes, Extended Debugging is effective in rectifying flaky tests. In most cases, networking is the root cause of test flakiness. HAR files allow developers to pinpoint slow loading/failed requests without having to leave the Sauce Labs “Test Details” page.
ADM: From an operational standpoint, what effect do flaky tests have on the development lifecycle?
Parobek: Flaky tests introduce unnecessarily debugged cycles into the release process and force developers to spend their time on non-functional issues - slowing down the product lifecycle. As a result, this lowers organizations’ confidence in functional testing; thus development teams are less likely to write full end-to-end tests.
ADM: What are some of the most common app failures that you’ve seen? How can those be avoided thanks to Extended Debugging?
Parobek: Most common errors, include slow-loading pages and failed rest API endpoints. Both can be easily accessed via HAR files.
ADM: What role does Extended Debugging play in continuous delivery?
Parobek: Armed with the ability to effectively pinpoint the source of flaky tests, developers can spend less time debugging, and efficiently remediate issues.
ADM: What are the top-five things developers should keep in mind when investigation test failures?
Parobek: Whenever a testing issue happens, it’s important to ask a few simple questions that will help resolve things in a more economical fashion: Can you rule out a timing issue? How can a real problem with the System Under Test (SUT) explain the results? How reliable has this test been in the past? Is the problem obvious based on Stack Trace/Screenshot? Can I reproduce the problem manually? Those are just a few of the questions that see that developers test first - and they usually lead to a speedy resolution.
Write and run code every step of the way, using Android Studio to create apps that integrate with other apps, download and display pictures from the web, play sounds, and more. Each chapter and app has been designed and tested to provide the knowledge and experience you need to get started in Android development.
How to create a profitable, sustainable business developing and marketing mobile apps.