DevOps Continuous Testing Requires Speed with Relevance

Continuous Testing (CT) for DevOps environments requires a sophisticated view of test automation to accelerate the testing aspects of the delivery pipeline. Getting thorough testing, at speed, is tricky. How can thorough testing be accomplished in quicker and quicker cycles? A solution blueprint and known good “best practices” are needed to guide a DevOps test automation solution.

A DevOps automated testing solution is more than running tests fast. It requires a carefully thought out approach. Problems found by automated testing must be handled in real time and resolved continuously. Many test results produced by automated testing cannot practically be handled in quick cycles using tradition automated testing methods. 

Rapidly accumulated automated test results pile up and cause release delays even though many of the failures may be not important. A sophisticated view of what is comprehensive testing becomes essential. And the results also must be handled in a more sophisticated way. 

The word “Relevant” is critical to DevOps automated test solutions.

- Only run the most relevant tests;

- and report only the relevant results; 

- to the relevant consumers;

- at the relevant time.

Mastering these following Five CT best practices will ensure fast and relevant DevOps test automation.

Before developers integrate their changes into the integration or trunk branch, due diligence to prevent integration failures is especially important. Consider assigning a CT architect to produce guidelines for automated test practices, train developers and install automated test peer reviews. 

If all software functions are designed to be testable at the system level, automated test cases and evident test results can be available at the time code changes are committed to the trunk branch. Metrics and thresholds can determine whether code changes may be accepted. For example, complex code changes to high risk and error-prone code areas may require more automated tests to be committed together with the code changes.  

When the total number of tests is either too many to be executed in the CI cycle times, or the coverage is too low to be relevant the test suites can be divided into specific cycles and triggered intelligently depending on the code change and delivery pipeline conditions. Ideally the test suites can be created from a pool of tests dynamically in accordance with the relevance of the code changes.

Many existing test tools are not CT-ready and cannot accomplish best practices DevOps.  Restful APIs allow CT tools to integrate easily with everything else and each other. CT-ready tools enable test activities to be cached and pipelined so that tests are not waiting for resources. CT-Ready test tools may be run virtualized for horizontal scaling yet optimized for execution on physical servers when time-critical critical performance testing is required.

Elastic Lab-as-a-Service orchestration tools support physical, virtual and hybrid test resources and their interconnections equally and the entire lab is operated in a self-service fashion. This includes the ability to inventory, catalogue, invoke, pre-load, implement and re-configure systems and networked test topologies in concert with pipe-lined test schedules.

Faster integration cycles require faster CT. Accelerating CT processes can be accomplished by using powerful computing resources, designing the tests to focus on one verdict and get to the highest priority verdict quickly and pre-load and pre-configure the next test while the current is running. 

Run as many test agents in parallel as you can afford and set thresholds for failures and immediately release resources when triggered. As the scale of CT increases it becomes essential to dynamically configure test schedules. A preferred method is to assign test cases according to changed source code test results because it works at the system level, is statistically relevant and can be used to speed up results analysis also.

Use intelligent analytics and thresholds to speed up results analysis. CT analytics is more than just reporting pass/fail results. Test analytics are both descriptive and predictive – they automatically keep track of test results trends correlated to software changes for each case and assign priorities, diagnostics and probable solutions rather than simply listing the pass/fail results. 

The metrics strategy sets thresholds for each test phase that determine whether  failures  are significant enough to revert the changes or it is necessary to stop the DevOps cycles long enough to fix the problems.

Consider the importance of best practices:

- Does your team have intimate knowledge of CT best practices?

- Can you afford the cost of not getting it right the first time?

- Can your organization afford the delays that inevitably occur when they get interrupted by higher priority product escalations?

- Can you afford the ongoing cost of tools integrations with new tools and maintenance and enhancements that will surely occur in the future?

Many organizations have not fully recognized the strategic nature of getting CT right in order for DevOps to accomplish its primary success goals of improving innovation, quality, time-to-market, and ROI. There is much more to CT than just running some automated tests after a build. 

A deep and broad understanding of CT best practices and a good DevOps automated testing solution blueprint that balances test speed and relevance is vital to putting together a successful DevOps