Let’s face it, smartphone apps have raised the bar for all of us. Ever since Apple released the first iPhone a little over a decade ago, there’s been a huge emphasis on creating software that is clean, elegant and easy to use. Consumer apps and websites have created much higher expectations for all software, and these days even business applications are expected to delight their users and work perfectly every time. This must all be achieved in the context of faster, more frequent release cycles, raising the bar even further.
This has important implications for QA, since eliminating code bugs and hard-to-use features are central to this improved user experience. Dev teams use a variety of metrics to track the effectiveness of their QA, and metrics are certainly important for tracking success and guiding investments in the future. But some of the popular metrics we use fall short; they can be counterproductive and don’t truly show us where to focus our efforts.
First I’ll take a brief look at some of the popular app metrics in use and highlight what I see as their shortcomings, then suggest some more effective ways to measure QA moving forward.
When thinking about how to improve on these, it’s useful to think about what metrics should achieve. Businesses are investing more than a quarter of their IT budgets to QA because it’s so critical to retaining users, and without good metrics, you’re making these investment decisions in the dark. You need to know where to focus your efforts to get the biggest return on investment.
Therefore, your QA testing should be tied to how important a particular workflow is. Workflows that are critical to the business obviously deserve more attention than those that are not. If you’re an e-commerce company, for instance, the checkout process needs to be perfect, since it’s how you make money and where customers often abandon a purchase at the last minute.
Similarly, processes that are most frequently used by customers also deserve more attention. There’s little sense investing the same amount on QA for your most used feature versus a little-known process buried deep in a menu somewhere.
So how do we measure all that? A good way is to look at traffic and log-level data, which show the most popular flows through your app or website. A certain amount of qualitative judgment is required: if you add a brand new feature, it may not be the most used part of your app but you should make sure it works when people try it for the first time. But in general, let high usage be your guide, and be sure you’re testing a higher number of use cases and edge cases around that flow.
This provides a ranking from your most important to least important flows. With this in place, you can look at your test distribution to ensure you’re focusing investment on the right areas.
The common metrics discussed above do have value and I’m not suggesting we throw them out, but they should be considered within the larger business context and used properly. When it comes to a number of bugs found during QA, for instance, it may be helpful to map the results back to specific engineers or teams and work with those people more closely.
Software quality is more important than ever, and businesses are investing heavily to ensure the best experience for end users. But we need to think carefully about what we measure to ensure we get the biggest bang for our buck.
Address:
3003 East Chestnut Expy
STE# 575
Springfield, Mo 65802
Phone: 1-844-277-3386
Fax:417-429-2935
E-Mail: contact@appdevelopermagazine.com