Increasingly for all applications, but especially for mobile apps, performance is king. We have entered an era of near-zero user tolerance for poor or slow performing applications. Complicating matters, applications are becoming increasingly complex, and most rely upon multiple third-party components and services. In many cases, these services are interdependent upon one another, as well.
At
Orasi, we believe this situation necessitates that both developers and testers update their approaches to testing, and especially performance testing, incorporating it more early, more often and at a greater level of automation. In this article, we’ll discuss how performance testing under a continuous delivery model enables organizations to address and surmount the challenges of current users’ extreme expectations.
From There to Here
The approach for performance testing historically has been to address it late in the development cycle, after functional testing between user acceptance testing and go-live. This method finds performance problems late in the SDLC --and often inadequate time to remediate all the issues discovered. Unit and integration testing are both essentially useless in discovering performance problems with complex, multithreaded applications such as modern web and mobile apps.
Furthermore, for apps that perform multiple, integrated high-level functions, discovering performance problems after thousands of lines of code have been written can negate much of the value of the entire project. At the minimum, resolution becomes a more complicated process. In the worst case scenario, perfectly good code ends up being picked apart, only for the developer to discover that the performance issue is with a third-party component or service.
A far better approach is to initiate performance testing of complex and/or service-based applications as soon as possible. For companies not engaged in continuous delivery, incorporating it into each iteration can substantially reduce the odds of this happening. However, for organizations using a continuous delivery model and running a continuous integration server, it makes even more sense to automate performance testing and integrate it into the everyday build process.
Developers can link the performance tests to run within minutes after checking-in code and conduct performance testing at that point. Performance and stress testing in such frequent cycles enables developers to identify and resolve interoperability problems or bottlenecks between application and component behaviors when they are much less expensive to fix. With every layer of functionality and services added, performance testing immediately after code check in can highlight the specific problems that have been introduced with that code.
Considerations for Performance Testing Under Continuous Delivery
Despite the advantages of this approach, it can have challenges, and it sometimes requires special approaches. A few of the factors to consider include:
- If the activity being tested involves a large amount of data or is a service from a third party that does not offer 24/7 availability, the test will likely have to be simulated rather than live, in order to support this model. Developers can develop a simulation stub that can stand in for the real service, building in any latency that must be accounted for in app performance testing.
- Any long-running activity or service (more than a few minutes) is probably not an ideal candidate for continuous performance testing.
- Companies using service virtualization will need to examine their approach or not be able to incorporate performance testing into continuous integration. Virtualization tools often require too much external interaction, making it more difficult for a continuous integration server to run its build. Virtualization servers that are integrated with Jenkins and other CI tools are a better candidate to integrate.
Obvious but not Evident
When I explain performance testing in a continuous delivery environment and point out its advantages, most developers, testers and stakeholders have an “A ha!” moment. Yet, enterprises continue with their hidebound methods of performance testing for even the most complex, service-laden applications. Invariably, when an organization doesn’t performance test early and adequately, conspicuous performance issues may still be present in the final build.
We have seen cases where teams end up working through significant performance issues in the production environment. That outcome is completely unacceptable if the goal is to achieve even a reasonable level of user adoption and satisfaction. By treating performance testing as part of the continuous integration process and not a standalone exercise late in the game, developing companies can largely avoid this nightmare, reduce costs, expedite delivery and provide users with a better experience.