Impact of Visual AI on test automation

Applitools, the provider of Visual AI-powered test automation, has announced an industry report that reveals the impact of Visual AI on the performance of the most popular open-source test frameworks - Cypress, Selenium, and WebdriverIO. Data was sourced from 288 quality engineers across 101 countries, spending an average of 11 hours during November 2019. This amounts to a combined 80 work weeks, or well over a year and a half of quality engineering hours, making it the largest industry study of its kind to date.

To download the "Impact of Visual AI on Test Automation Report" visit: (https://applitools.com/impact-of-visual-ai/) 

"The Impact of Visual AI on Test Automation Report" provides important learning including:

• 5.8x Faster Test Creation: When using existing open-source code-based frameworks in isolation, it took testers seven hours to create test code in contrast to just 1.2 hours using Visual AI. As a result, testers are able to greatly expand test coverage while simultaneously releasing faster than ever before.
• 3.8x Increase in Test Code Stability: Participants used 32 locators and labels on average to provide test coverage using code-based frameworks alone. This number dropped to just seven with Visual AI. This increased stability greatly reduces test flakiness, broken CI/CD builds, and time spent on maintenance. Better test stability leads to efficiency gains along whatever dimension the quality engineering team values most - either more coverage, more time to manage quality or even faster test cycles in support of CI/CD.
• 45% More Efficient at Catching Bugs: Participants caught 19 out of 20 bugs (95%) using Visual AI compared to just 13 (65%) when using code-based frameworks in isolation. When test builds are faster, provide more coverage, and are more stable and easily maintained, testers catch more bugs pre-production while still delivering more features, faster than previously thought possible.
• Visual AI Helped Testers Succeed in Their Work When Used in Conjunction with Open Source Frameworks: All 288 testers that participated in the study achieved success rates of 79% when using their preferred code-based framework vs. 88% with Visual AI. This is a nine-point jump in their performance after just a single hour of Visual AI training. Eric Davidson, Software Engineer from Chick-Fil-A comments, "This was an incredible opportunity to experience the power of Visual AI. Applitools has created a tool that easily integrates with existing frameworks and quickly augments functional test suites..."

In addition to the standard metrics of tester efficiency, the study yielded a new measure called Test Code Efficiency. Similar to the concept of "code efficiency" that software developers look for in their work, test code efficiency measures both the number of lines of test code a quality engineer can write per hour and the amount of coverage each one of those lines provides. Visual AI, which captures screenshots and images of an entire page in a single line of code and detects meaningful differences with 99.999% accurate computer vision, is empirically proven to be 5.9x More Test Code Efficient than open-source code-based approaches in isolation. As a result of these efficiency gains, engineering teams are able to deliver software faster with improved quality.

"Visual AI is a tremendous efficiency multiplier for automated testing. It improves the productivity of developers and test automation engineers more than 5X, delivers huge time and money savings in the app delivery cycle, removes the testing bottleneck in the CI/CD process, and helps deliver software applications with unprecedented visual and functional quality on any computer and mobile device. Testers need to look at Visual AI as a superpower. A virtual assistant that can access every screen and page in minutes, inspect it for changes and interpret every one of them the same way the human eye and brain would, but do so without ever getting tired or making any mistakes. We believe this capability is a major part of the future of test automation," said Gil Sever, CEO of Applitools.

"According to Gartner’s Critical Capabilities for Software Test Automation (December 17, 2019), "61% of [its 2019 Software Quality Tools and Practices Survey] respondents said that AI/ML features would be very valuable in software testing tools. Improved defect detection (48%), reduction in test maintenance cost (42%) and improved test coverage (41%) were seen as the top benefits expected. (multiple answers were allowed)." "The Impact of Visual AI on Test Automation Report" provides empirical evidence that Applitools Visual AI delivers exactly what the market expects from the addition of AI/ML into automated testing.

Data was sourced via the "Global Visual AI Rockstar Hackathon" which challenged software testers to create test suites for five real-world functional testing use-cases using their preferred code-based approach, including Selenium, Cypress, or WebdriverIO, and then repeat the process after adding Applitools Visual AI. This approach provided an unbiased, highly controlled environment from 288 quality engineers from 101 different countries during the 30 days ending November 30, 2019. 

"Completing the hackathon was a keystone achievement in my career! I learned more by participating in this hackathon than any other automation instruction I’ve taken in the past number of years. I’m now 100 percent convinced that Visual AI testing is an essential tool for efficiently validating web and mobile software applications," said Tracy Mazelin, QA Engineer at Paylocity.

"This was the most interesting and useful event of the year in the field of test automation. This allows you to take a look at test automation from a different point of view and gives an opportunity to radically improve your existing approaches," said Viktar Silakou, Lead QA Automation Engineer at Globant S.A.

"The Impact of Visual AI on Test Automation report drives home the advantages that AI can have on teams’ ability to deliver software on time and on budget. 288 engineers developed nearly 3,000 automated tests, and spent over 1.5 years of combined quality engineering effort to show that Visual AI is easy to integrate and adopt. When integrated with your Selenium, Cypress.io, or WebdriverIO test frameworks, Applitools Visual AI enables 5.8X faster test creation, is 5.9X more test code efficient, improves test code stability 3.8X, and ultimately provides superior test coverage leading to a 45% increase in early stage bug detection. Visual AI that enables tests to be written faster and with less complexity allows developers to focus on feature development. Additionally, the improved stability of tests results in faster builds and less time troubleshooting erroneous failures," said James Lamberti, CMO of Appltiools.

About Applitools

Applitools enables Visual AI-powered test automation to help teams release high-quality web and mobile apps faster and more efficiently.

Applitools Visual AI modernizes important test automation use cases -- Functional Testing, Visual Testing, Web and Mobile UI/UX Testing, Cross Browser Testing, Responsive Web Design Testifng, Cross-Device Testing, PDF Testing, Accessibility Testing, and Compliance Testing -- to transform the way organizations deliver innovation at the speed of CI/CD at a significantly lower Total Cost of Ownership (TCO).

Hundreds of companies from verticals such as Software, Banking, Insurance, Retail, Pharmaceuticals, and Publishing -- including 50 of the Fortune 100 -- use Applitools to deliver the best possible digital experiences to millions of customers on any device and browser, and across every screen size and operating system.

Applitools is headquartered in San Mateo, California, with an R&D center in Tel Aviv, Israel. For more information, please visit applitools.com.