Sonatype implements Applitools to ensure app quality

Sonatype was in search of a solution to prevent visual bugs across the variety of operating systems and browsers supported by the Nexus platform.

After implementing Applitools, the engineering team can solely focus on delivering value, while Applitools uncovers countless unexpected changes before code ever leaves development.

Moving forward, Sonatype will deploy a shared component library which will represent the future of their UI. This new deployment will also include Applitools to deliver component level testing, further diminishing visual risk and catching bugs even earlier in the app-dev process.

About Sonatype

Sonatype is the leader in software supply chain automation technology with more than 400 employees, over 1,200 enterprise customers, and is trusted by more than 10 million software developers.  Sonatype’s Nexus platform enables DevOps teams and developers to automatically integrate security at every stage of the modern development pipeline by combining in-depth component intelligence with real-time remediation guidance.

The Situation

Sonatype first bought Applitools in 2017 to address a core problem affecting web applications: visual drift due to code changes with unaccounted UI dependencies. In developing the Nexus Lifecycle product, Sonatype learned that small code changes could change the application visually in unanticipated ways.

Nexus runs on developer systems that include Windows, Mac, and Linux. Their customers would run various versions of browsers – from most stable to most recent. The big driver for Sonatype involved ensuring Internet Explorer 8 support even as development moved beyond IE 8 to more recent browser versions.

Multiple development teams worked on the product. Teams would reuse each other’s code. Small changes in markup or function used by one team might impact the code of another. Sonatype used a range of functional automation tools, but visual issues went undetected. Understanding the limitations of functional tests to catch visual defects, developers had to devote engineering time to uncover these failing cases manually.

Jamie Whitehouse and everyone on the development team spent time each release working to uncover and address these undetected failures. Often, this work occurred as spot checks of the 1,000+ pages of the application during development. In reality, this work, and the inherent risk of unintended changes, slowed the delivery of the product to market.

Looking For A Solution

Jamie and his team sought an approach for highlighting these differences. Key capabilities he needed:

• Automated – a solution that could capture and highlight visual differences without manual intervention
• Accurate – low incidence of false positives and missed errors. Sonatype needed element-level accuracy, and a way to ignore pixel-level differences that plague pixel diff tools.
• Integrated – easy to integrate with existing tests.
• Complete – In evaluating tools, Jamie wanted to avoid tools that would require his team’s incremental development work to use. Tooling work would cost development hours without delivering value to Sonatype customers.
• Supported – Sonaytpe would pay for support instead of using their own engineers to support the solution.

Jamie’s team investigated the available solutions. They wrote off pixel-diff comparators because the team’s experience indicated a high degree of false positives from pixel diffs. Also, the Sonatype use case differed greatly from graphic design use cases, where pixel match matters. They also wrote off open source tools, which put engineering resources at risk to develop and maintain a solution that met Sonatype’s needs.

Given the constraints, Jamie says, Applitools was the obvious choice to try. Applitools provided AI-based approach that allowed it to identify and compare visual elements instead of comparing pixels. It also provided a range of match comparisons, including layout comparisons, content checks, and even exact pixel comparison (if needed). Jamie’s team saw the promise of automated visual validation without false positives.

During their initial deployment, Applitools met their needs. It integrated with their existing test automation, delivered accurate results, and Applitools provided great support.

Results

After two years of using Applitools, Sonatype has reaped some serious rewards.

While they continue to justify the cost of Applitools based on the engineering effort saved, the real benefit involves the complete elimination of visual drift in their applications. Sonatype now knows, with certainty, the effect of changes to rendering and markup across reused components – simply through running their test automation.  Developers can deliver code changes to existing code and know that Applitools can properly catch all the changes – including unintended changes – early in the development process.

If Sonatype engineers make a change in their margins across a number of pages, all the differences show up as highlights in Applitools. Features in Applitools like Auto-Maintenance make visual validation a time saver. Auto-Maintenance lets engineers quickly accept identical changes across a number of pages – leaving only the unanticipated differences. As Jamie says Applitools takes the guesswork out of testing the rendered pages.

“With unit tests, you check them in once and then run them forever. Once they’re running, you expect them to stay green and check them only when they’re red. Now, imagine being able to do the same thing with UI tests. You write them once and only check them when they’re red. Imagine how much more productive you would be if you didn’t have to worry about missing visual changes?”

With Applitools deployed, Jamie has his engineering team focused on delivering value in Nexus Lifecycle. Applitools has helped uncover countless unexpected changes during the development phase, as opposed to waiting for final testing. The true value of using Applitools – development free from visual risk – is hard to quantify. But, when calculating the speed of delivering products to market, Jamie knows he’s much faster with Applitools.

Where Next?

To reduce its UI risk further, the Sonatype team is deploying a shared components library. Their library will form the basis of the future UI, with components reused across their Nexus Lifecycle application. The component team evaluated Applitools alongside several solutions specifically geared toward component testing. They concluded that Applitools would test their entire component library with the same accuracy and workflow they rely on in their existing end-to-end tests.

Sonatype plans to use Applitools to test the entire library in action across all browsers and viewports. Their goal is to deliver visual validation even earlier in the development phase and diminish their visual risk in end-to-end tests.