Continuous Integration Has Now Blurred the Lines between Development and Functional Testing

The traditional development lifecycle practice was to “huck” code over the fence from stage to stage and huck it back with our bug lists. 

Continuous Integration has now blurred the line between development and functional testing as the process becomes more automated and iterative.  With mobile app testing becoming a clear bottleneck to rapid delivery, customers need an automated mobile regression testing solution to eliminate wasted manual testing time. 

SOASTA originally integrated mobile functional testing with its TouchTest product via a Jenkins plugin.  In fact, Jenkins’ creator, Kohsuke Kawaguchi, aided the process by helping with the architecture of the plugin to fully automate all mobile testing tasks.

"Automation is central to continuous integration, and mobile testing has hard and unique challenges, such as automated deployment of apps to real devices and automated executions of tests. I'm very happy that SOASTA engaged the Jenkins community to make their services easily accessible to our users." - Kohsuke Kawaguchi / Creator of Jenkins & CTO, CloudBees 

Continuous Delivery has been defined as a process whereby code is continuously ready to deploy.  But when it comes to performance testing, it’s still common to rely on a single load test wedged in just before deployment or, often, simply hucking the code over the fence to the production environment and hoping for positive results.  These methods practically guarantee some nasty surprises.  

True Continuous Delivery requires addressing performance as a first class citizen and treating it more as a feature than an afterthought.  This requires an approach that applies a “shift left” strategy to enable continuous performance testing.

For SOASTA users, the good news is that the same Jenkins plugin used for mobile test automation enables use of CloudTest to integrate and automate performance testing into the iterative testing process.  As developers check in code, a light load test can be automatically run with CloudTest and the results of those tests are populated back into Jenkins to be reviewed daily by the team.  This is a proven approach to introduce performance baseline trends and results into the regular regression testing process.

Continuous Deployment represents that line between staging and production where there is confidence that deployed code will not wreak havoc on user and business success.  It’s also critical to have visibility into how the code is affecting the real user.  Has the change in flow affected the customer conversion rate, and how? And, not tomorrow, or even an hour from now, but as it’s happening.

When performance is aligned with business metrics, it’s best to track it with real users and with details to direct how to respond. Sometimes poor performance is within the system or app within your control, and sometimes it’s a third party CDN, but it is always your problem. When 1/10th of a second can affect your conversion rate, real dollars are at stake.

SOASTA’s mPulse lets teams measure the experience of real users.  The data gathered then feeds the cycle of continuous deployment with validation of use cases, geographic distribution, and information about user patience and reactions to poor performance.  Customers have determined that significant business improvements are realized at 1/10th of second increments, which lets users easily prioritize and continuously improve the site’s performance.

Continuous Integration, Continuous Delivery and Continuous Deployment should all work together to deliver exceptional customer experience. The right testing partner gives you the tools to make that possible through every critical phase.