Finding bugs in code through better QA

Let’s face it, smartphone apps have raised the bar for all of us. Ever since Apple released the first iPhone a little over a decade ago, there’s been a huge emphasis on creating software that is clean, elegant and easy to use. Consumer apps and websites have created much higher expectations for all software, and these days even business applications are expected to delight their users and work perfectly every time. This must all be achieved in the context of faster, more frequent release cycles, raising the bar even further. 

This has important implications for QA, since eliminating code bugs and hard-to-use features are central to this improved user experience. Dev teams use a variety of metrics to track the effectiveness of their QA, and metrics are certainly important for tracking success and guiding investments in the future. But some of the popular metrics we use fall short; they can be counterproductive and don’t truly show us where to focus our efforts.

First I’ll take a brief look at some of the popular app metrics in use and highlight what I see as their shortcomings, then suggest some more effective ways to measure QA moving forward. 

Popular ways to find bugs in code using QA metrics include:

• Number of bugs found during QA. This looks at software quality at various stages of development and often influences whether to grow or shrink QA teams. The challenge is that this metric tends to pit two teams against each other. QA’s mission is to find as many bugs as possible, and dev teams often want to push back and argue that some of those weren’t bugs at all, and maybe the QA team doesn’t know what it’s talking about. This tension isn’t healthy. After a while, the goal for each side becomes making their metrics look good, which will never lead to lead to the best results.
   
• Core coverage. This metric looks at the proportion of code and workflows covered by QA tests, with a higher proportion naturally viewed as good. But this metric doesn’t measure the quality of coverage itself. You might be 80 percent or 90 percent covered, but if your tests aren’t capturing the right use cases or looking at all the right workflows, you could still be missing errors. That’s a big shortcoming for such a widely used metric.
   
• Number of bugs found in production. This is useful but it doesn’t always provide the complete picture. You’re depending on your users and teams to actually find and report the bugs that exist, and if they don’t do their part you’re none the wiser. It’s also a backwards-looking metric, meaning it doesn’t help you root out bugs before your code is in production.

When thinking about how to improve on these, it’s useful to think about what metrics should achieve. Businesses are investing more than a quarter of their IT budgets to QA because it’s so critical to retaining users, and without good metrics, you’re making these investment decisions in the dark. You need to know where to focus your efforts to get the biggest return on investment.

Therefore, your QA testing should be tied to how important a particular workflow is. Workflows that are critical to the business obviously deserve more attention than those that are not. If you’re an e-commerce company, for instance, the checkout process needs to be perfect, since it’s how you make money and where customers often abandon a purchase at the last minute.

Similarly, processes that are most frequently used by customers also deserve more attention. There’s little sense investing the same amount on QA for your most used feature versus a little-known process buried deep in a menu somewhere.

So how do we measure all that? A good way is to look at traffic and log-level data, which show the most popular flows through your app or website. A certain amount of qualitative judgment is required: if you add a brand new feature, it may not be the most used part of your app but you should make sure it works when people try it for the first time. But in general, let high usage be your guide, and be sure you’re testing a higher number of use cases and edge cases around that flow.

This provides a ranking from your most important to least important flows. With this in place, you can look at your test distribution to ensure you’re focusing investment on the right areas.

The common metrics discussed above do have value and I’m not suggesting we throw them out, but they should be considered within the larger business context and used properly. When it comes to a number of bugs found during QA, for instance, it may be helpful to map the results back to specific engineers or teams and work with those people more closely.

Software quality is more important than ever, and businesses are investing heavily to ensure the best experience for end users. But we need to think carefully about what we measure to ensure we get the biggest bang for our buck.