Embracing feature experimentation one step at a time

For many organizations, when it comes to product development and evolution, experimentation can be a loaded word. It can carry connotations of both guesswork, ‘gut-feel,’ and ad-hoc exploration as well as formal, math-heavy, dense work depending on your experience. But in a world of continuous delivery, where incremental updates to products are constant and can lead to a competitive advantage, quantifying the improvement - or lack thereof - is vital.

Some of the world’s most successful and disruptive companies - like Netflix or Facebook, for example - have taken their market-leading position precisely because they have developed a strategy and culture that embraces experimentation. For them, it has become a key way to test and evaluate new products or service features, while containing some of the potentially detrimental risks to customer experience. For many of the trailblazers, experimentation has been part of the company DNA since the beginning, but to shift the mindset and culture of an existing development team can be harder. Here are a few ways to skip the misconceptions and lead your team to the experimental waters.  

Don’t make it about the math

To take the guesswork out of feature experimentation, organizations need quantifiable and mathematically sound data with which to analyze the results of any changes to a product. The first step is usually to create a testable hypothesis and then record and analyze the data during testing. Gathering and interpreting the necessary complex statistical data that measures the KPIs of a particular feature roll-out can seem difficult and daunting if it has to be done manually. It centers on an understanding of calculus and, for many people, stirs memories of unfathomable equations from their high school math classes.

Fortunately, feature experimentation tools on the market today can ‘frame’ the hypotheses you want to test, analyze related statistics and then provide automated feedback regarding feature level telemetry in a way that’s easy to understand, relate to and act upon. The math is still there, and while it is great to encourage those who want to understand the back end calculations, product developers, engineers and marketing teams can just act on the presented data. Apps like Brilliant are a good place for those interested in diving deeper into the numbers to start to resurface their dormant math skills.

Make feature experimentation visible throughout the whole organization

One of the first things you have to understand and accept about feature experimentation is that even armed with reliable statistical data, they can and do fail. Making all feature experimentation visible across the organization - and getting the whole company, not just engineers but customer service, product, and marketing involved in early product testing - means that the whole company can experience the successes and the failures of feature experimentation. It helps everyone at the organization understand that, sometimes, a failed idea is an important step on the road to a product’s ultimate growth and success.

If you’re new to feature experimentation, take a look at areas where you know there are issues with your product. Perhaps based on your user research or user experience analytics tools you have identified a part of your app that frustrates users or where you are seeing users drop off. Take some time to develop some simple hypothesis around this that you can test. For example, “If we provide users with tooltips on how to complete submission forms in our app, we will increase form completion rates.”  

At this point, it’s worth considering the use of feature flags to roll out the product changes you need to test your hypothesis. That way you can test your idea amongst your own employees or a small customer group before rolling it out to a wider customer cross-section. Using feature flags also means that, in the event that your feature experiment does fail - causing lots of app crashes, errors or bugs for example - you can easily ‘dial-back' the update without having to manually alter coding, which is always a security and operational risk in its own right. However, there are experiments that simply don’t move the metrics in the way you predicted, this can help to evolve the hypothesis for the next iteration of the experiment or provide different, positive results. Using feature flags alongside feature experimentation is, possibly, one of the best ways to keep product development running at the speed required in modern business, while also minimizing potential dips in customer satisfaction or service disruption.

All of this experimentation also provides the backdrop for a data-driven organization that makes all future decisions based on measurement and metrics. Instead of basing product decisions on gut feel, experimentation shows what works and what doesn’t in an undisputable, proven format.

Create a culture of experimentation

If you’ve followed the first two steps, then you’re already on your way to developing a culture of experimentation within your organization. Celebrating the losses as much as the wins with emails, meetings, and even cake can encourage everyone to understand that they can learn from failed experiments as much as successful ones, if not even more. This engrains the idea that experimentation is about trial and error and throwing out ideas and testing them to get answers, not always just the right answer.

It’s also important to make sure that experimentation is part of initial conversations during the hiring process and to be clear to candidates at all levels that they are expected to embrace the philosophy. Recruiting analytical people that are willing to share their learnings, to test things and, on occasion, to accept that they have been proven wrong is critical. Baking this into progression and promotion processes ensures that it becomes a lasting part of someone's career and their role at the company. Additionally, senior hires need to understand the risks associated with experimentation and are able to work in a positive way with their team when an idea has failed.

At its best, feature experimentation offers a way to engineering, product management, and marketing, helping create agile engineering and product teams that can accelerate the pace of product delivery and make data-driven decisions - without necessarily needing to interpret raw statistical data. This helps businesses achieve a key aim of accelerating time to value and time to market, while also going some way to mitigating a significant amount of risk and ultimately delivering better product outcomes.