Testing is Like the Rodney Dangerfield of DevOps It Gets No Respect

When Joan Wrabetz, CTO of QualiSystems, reached out to discuss why “Test is like the Rodney Dangerfield of DevOps – it gets no respect”, I thought, “Ok, I’ll listen to that.” So here you go:

ADM: So, you mentioned to me that “Test is like the Rodney Dangerfield of DevOps – it gets no respect”, but you also mention that in a recent Gartner survey, over 50% of respondents listed test automation as a top enabler of DevOps success. What does this mean?

Wrabetz: This is really illustrative of the growing recognition that DevOps cannot succeed if only one part of the DevOps continuum is automated.

If DevOps is a continuum, it starts with planning and development and ends with deployment into operations, and test is right in the middle of every DevOps cycle.

ADM: So, what has been the focus of DevOps and what needs to be employed for it really to pay off?

Wrabetz: Generally, DevOps projects tend to start in only one part of the continuum. 

Much of the focus to date in DevOps has been on either application build automation, or deployment of apps into operations. But for DevOps to pay off, there needs to be continuous automation from development all the way through to operations – and that means automating test.

ADM: Does Test ever drive DevOps?

Wrabetz: Yes. Test organizations have been focused on automation already for 10 years, so often they are the driver for DevOps. When test is the focus of DevOps, it is called Continuous Integration. 

However, it is also important to work to continuously expand automation outside of test. When test teams take the lead, they need to find ways to push automation back into development and forward into staging and production. In fact, the problem of test automation is particularly difficult and this means that the team that implements test automation is in a great position to lead DevOps initiatives beyond test.

ADM: Why is automating the various testing steps (regression, performance, stress, GUI, QA, Interoperability and security testing) so difficult?  

Wrabetz: Well, it can be broken down to five key components that must be automated successfully.

ADM: OK. So, I’m guessing the first is probably the toughest to automate? Yes?

Wrabetz: Yes, you’ve got it. The infrastructure - It starts with one very important characteristic that is critical to testing which is often not automated – the ability to automatically mimic the production infrastructure in which the software (or hardware) being tested is expected to run. 

ADM: How does this support test?

Wrabetz: Having the ability to give each tester (or set of automated tests) their own personal replica of the target production infrastructure – aka a “sandbox” – is critical to ensuring that the software coming out of test can be automatically and continuously deployed into production. Automating tests is not nearly as difficult as automating the underlying infrastructure in which those tests run.  

ADM: So what is new, or can be overlooked in automating testing?

Wrabetz: Often not considered in automating testing is the requirement to mimic the workloads that would be common in production.  This includes the network traffic, other application workloads, and network security profiles. These activities need to be automated within the sandbox to create a more realistic environment for testing.

ADM: What’s key in automating the tests themselves?

Wrabetz: Of course, the tests themselves also need to be automated and there are a variety of tools to assist in this process. As software gets more complex, this continues to be a challenge, with mobile testing replacing GUI testing as the most challenging type of testing to automate.

ADM: So, developers see the results, but how do they get buy in from the higher ups?

Wrabetz: In order to provide continuous automation from development through testing to delivery, it is important to see and analyze test results automatically.

ADM: What’s the key enabler of all of this?

Wrabetz: Automation into and out of testing needs to be enabled. This means that sandboxes and test automation suites need to be API-driven so that they can be started by a previous tool in the DevOps toolchain and can also initiate a tool that follows testing in the toolchain.

ADM: I’ve heard you mention sandboxes a lot throughout this conversation. Can you expand on the subject?

Wrabetz: Sandboxes are self-contained infrastructure environments that can be configured to look exactly like the final target deployment environment, but can be created and run anywhere. For example, developers can create a sandbox that looks like the production environment – from network and hardware to OS versions and software to cloud APIs. They do their development in that sandbox for a short period of time and when they are done they tear down the sandbox.  You can think of a sandbox as an “uber container” for the infrastructure.

ADM: How does this apply to testers?

Wrabetz: Testers can do the same thing, and in addition, they can run a bunch of tests with the sandbox configured to look like their internal IT environment and then automatically re-configure the sandbox on the fly to look like the external cloud environment and run more tests. This allows them to test all of the possible environments that the application could run in without disrupting the actual production infrastructure. 

ADM: Technically, what is a sandbox? 

Wrabetz: Intuitively we know that a sandbox is a protected space where you have complete control and others are allowed in only if you invite them. You can bring your own toys into the sandbox and make anything you want in the sand. Just stomp it out if you don’t like it and start over. 

Technically, sandboxes follow these same rules. A number of vendors are now providing sandbox solutions (some are called “Environment as a Service”) that have a simple interface for replicating any target infrastructure environment and configuring it with as much control as you want. 

ADM: What are the key attributes of a sandbox?

Wrabetz: They allow you to bring applications, tools, tests and automated processes into that sandbox. They provide protections so that others cannot mess with any infrastructure that you are currently using in your sandbox. 

They also provide reservation and scheduling for many people so that whole teams of developers and/or testers can share physical and virtual infrastructure on-the-fly for hours, days or weeks at a time. Finally, a good sandbox solution can be triggered from the outside (for example, from a DevOps tool).  

In the world of hybrid clouds, applications need to be deployable on both on-premise infrastructure and on public cloud infrastructure. Sandboxes allow testers and developers to mimic both environments and define applications that can run successfully in both types of infrastructure.