Centralized or Siloed Considering two test automation approaches

Each approach has its own benefits and shortcomings worth considering:

- Siloed

In a siloed reference architecture, every aspect of the delivery chain is broken into isolated components. Benefits include easier scalability and overall fine tuning, as you can optimize individual components without impacting other aspects of the delivery chain. A siloed approach offers better isolation within a team, which may encourage improved focus.

However, the first challenge inherent in a siloed setup has to do with one of its key benefits. isolation. Segmentation creates barriers between teams, and naturally hinders communication as each team owns and cares for individual parts. For example, developers will care about everything up to static code analysis, but not after. And QA may not even have visibility into code analysis, unit testing, and exception monitoring.

A second challenge is having more points of failure. Each isolated component is another thing to manage. And let's face it, modern processes also introduce more overhead, along with more places where things can break.

The siloed approach relies very much on integrations, leading to what I consider to be the last main challenge. On the one hand, I’ll posit that these integrations aren’t always a bad thing, and modern development tools are very good at providing integration points. But without enough attention to integration, this type of delivery chain can cause extra manual effort as code moves from one stage to another. Since this is counter to the overall goal of automation in itself, it’s definitely an issue worth noting.

In short, a siloed approach does not fit with the “shift left” goals of modern development - it does not put the emphasis on functional testing like it should. This is especially true when you consider additional problems, including expansion of the delivery chain, and how late integration and functional testing can occur within it.

- Centralized

This second approach is essentially the opposite of a siloed approach. Here the delivery chain is segmented by environment, and the greatest amount of testing happens in the integration environment. Ideally, the integration environment is flexible enough to allow testing to happen as frequently as every developer commit. See the representation below, which I really like:

There are a few common misconceptions around the centralized approach. First is the definition of “environment,” which can go beyond a single set of servers. When we discuss an environment, especially in a containerized world, it’s important to note that it can include many sets of infrastructure, all with their own configurations. For functional testing browser grids, you in fact need integration environments. This supports many parallelized instances, enabling better test coverage and speed.

However, this depiction, typically called the “continuous integration server,” is not very realistic. Instead I think it should be referred to as the “integration process.” Integration represents an abstraction of infrastructure that houses processes, use test cases, and overall oversight. The test suite and use case scan undergo code analysis, unit testing, and functional testing all at the same time - in parallel. Depending on how optimized your test suite is, you can achieve this many times a day.

In turn, the delivery chain needs to support full-stack deployments so that existing integration servers are torn down and replaced with new ones in every release. The best way to achieve this is by leveraging a cloud-based testing and provisioning service to do this for you, rather than doing it yourself.

The integration process usually starts with a web hook in your source repo or are lease automation tool. Code is deployed to a set of servers upon every commit, and runs the test suite.

The first challenge of such an environment is deciding who should own it. In the most ideal situations, I’ve seen QA serve as the steward of the integration environment while everyone else on the team has access. In this case it’s up to QA to ensure that the test suites run, that testing follows the best strategies, and that the required functional test cloud, unit test infrastructure, and code analysis tools are procured and setup. This means QA teams need to be concerned with all aspects of quality, not just functional testing, as well as strategy and automation rather than just execution.

When deciding between a siloed or centralized approach, there is no one size fits all. I personally favor the centralized approach due to the value of integration. Either approach you decide upon can have as much to do with your company’s culture and history as much as technology.

When making the decision, realize that other factors can be at play - for example specific development stacks can impact the setup of your architecture. If you are unable to identify where you fit, it most likely means your team needs to take the time to consider your overall delivery chain, rather than focus entirely on application releases.