An Introduction to Performance Testing in a Continuous Delivery Environment

At Orasi, we believe this situation necessitates that both developers and testers update their approaches to testing, and especially performance testing, incorporating it more early, more often and at a greater level of automation. In this article, we’ll discuss how performance testing under a continuous delivery model enables organizations to address and surmount the challenges of current users’ extreme expectations.

The approach for performance testing historically has been to address it late in the development cycle, after functional testing between user acceptance testing and go-live. This method finds performance problems late in the SDLC --and often inadequate time to remediate all the issues discovered. Unit and integration testing are both essentially useless in discovering performance problems with complex, multithreaded applications such as modern web and mobile apps.

Furthermore, for apps that perform multiple, integrated high-level functions, discovering performance problems after thousands of lines of code have been written can negate much of the value of the entire project. At the minimum, resolution becomes a more complicated process. In the worst case scenario, perfectly good code ends up being picked apart, only for the developer to discover that the performance issue is with a third-party component or service. 

A far better approach is to initiate performance testing of complex and/or service-based applications as soon as possible. For companies not engaged in continuous delivery, incorporating it into each iteration can substantially reduce the odds of this happening. However, for organizations using a continuous delivery model and running a continuous integration server, it makes even more sense to automate performance testing and integrate it into the everyday build process.

Developers can link the performance tests to run within minutes after checking-in code and conduct performance testing at that point. Performance and stress testing in such frequent cycles enables developers to identify and resolve interoperability problems or bottlenecks between application and component behaviors when they are much less expensive to fix. With every layer of functionality and services added, performance testing immediately after code check in can highlight the specific problems that have been introduced with that code.

Despite the advantages of this approach, it can have challenges, and it sometimes requires special approaches. A few of the factors to consider include:

When I explain performance testing in a continuous delivery environment and point out its advantages, most developers, testers and stakeholders have an “A ha!” moment. Yet, enterprises continue with their hidebound methods of performance testing for even the most complex, service-laden applications. Invariably, when an organization doesn’t performance test early and adequately, conspicuous performance issues may still be present in the final build.

We have seen cases where teams end up working through significant performance issues in the production environment. That outcome is completely unacceptable if the goal is to achieve even a reasonable level of user adoption and satisfaction. By treating performance testing as part of the continuous integration process and not a standalone exercise late in the game, developing companies can largely avoid this nightmare, reduce costs, expedite delivery and provide users with a better experience.