How to avoid flaky automated testing results

Flaky tests are the bane of any tester’s existence, forcing them to spend precious time on non-functional issues. This slows down the product lifecycle and lowers an organization’s confidence in functional testing, which also decreases the likelihood of development teams writing end-to-end tests.

End-to-end tests measure readiness for deployment by ensuring that the application’s integrity is behaving as expected and systems communicate from start to finish. So, it’s no wonder that even a minor hiccup could throw a wrench in such a finely-tuned procedure.

Lubos Parobek is the VP of Product at Sauce Labs, a company that develops automated testing solutions for mobile and web. We recently had a discussion with Lubos on how its latest Extended Debugging platform helps to fight against failed and flaky tests.

ADM: What is Extended Debugging from Sauce Labs? Is this an extension of an existing service?

Parobek: Extended Debugging is a new feature on Sauce Labs’ platform, allowing customers to capture data that is useful in understanding the root cause of test failures and flakiness. Served as Browser Console logs, these act as a resource for JavaScript errors that occur during the execution of a test; HAR files point at networking issues associated.

ADM: How does Extended Debugging differ from other tools on the market? Firefox (Firebug) and Google (Chrome Developer Tools) seem to offer similar benefits.

Parobek: Our closest competition (right now) is BrowserMob proxy. Conversely, Firebug and Chrome Developer Tools are native-browser extensions. This meaning they require developers to manually reproduce the issue and then deploy for information on what’s going on under the hood. With Extended Debugging, enabled customers can see the root cause of their failed/flaky automated tests without having to replicate the issue manually.

ADM: Is Extended Debugging effective in rectifying flaky tests?

Parobek: Yes, Extended Debugging is effective in rectifying flaky tests. In most cases, networking is the root cause of test flakiness. HAR files allow developers to pinpoint slow loading/failed requests without having to leave the Sauce Labs “Test Details” page.

ADM: From an operational standpoint, what effect do flaky tests have on the development lifecycle?

Parobek: Flaky tests introduce unnecessarily debugged cycles into the release process and force developers to spend their time on non-functional issues - slowing down the product lifecycle. As a result, this lowers organizations’ confidence in functional testing; thus development teams are less likely to write full end-to-end tests.

ADM: What are some of the most common app failures that you’ve seen? How can those be avoided thanks to Extended Debugging?

Parobek: Most common errors, include slow-loading pages and failed rest API endpoints. Both can be easily accessed via HAR files.

ADM: What role does Extended Debugging play in continuous delivery?

Parobek: Armed with the ability to effectively pinpoint the source of flaky tests, developers can spend less time debugging, and efficiently remediate issues.

ADM: What are the top-five things developers should keep in mind when investigation test failures?

Parobek: Whenever a testing issue happens, it’s important to ask a few simple questions that will help resolve things in a more economical fashion: Can you rule out a timing issue? How can a real problem with the System Under Test (SUT) explain the results? How reliable has this test been in the past? Is the problem obvious based on Stack Trace/Screenshot? Can I reproduce the problem manually? Those are just a few of the questions that see that developers test first - and they usually lead to a speedy resolution.