Optimizely GA Release of Stats Engine Facilitates AB and Multivariate Testing

Optimizely has announced the general availability version of Stats Engine as part of its optimization platform. The Stats Engine was developed in collaboration with statisticians at Stanford University and was created with current approaches in the field of statistical analysis and the company’s own algorithms to allow customers to interpret and act on data.

 

Through A/B and multivariate testing, in addition to other tools, Optimizely delivers real-time data and statistics. The company says that most important advance offered through the Stats Engine is the ability to extract valid, actionable data at any point in time rather than having to wait for an experiment to reach a pre-determined sample size reducing mid-course statistical flukes that can seem like relevant results.

 

The statistical framework of Optimizely’s Stats Engine combines sequential hypothesis testing and multiple testing corrections, and applies them to real-time data sets.  In doing so, Optimizely says that it has reduced the built-in limitations of traditional statistics that can result in experiment error rates of more than 30 percent.

Stats Engine Highlights: 

 

- Sequential hypothesis testing: With classical statistics, teams are advised to look at the results of a running experiment just once – and only then after they’ve reached a pre-set sample size – due to the risk of mistaking statistical oddities for statistical relevance.  Stats Engine assumes that a test has an infinite rather than a fixed sample size. It generates an accurate level of confidence after every visitor, so marketing or product management teams can check results frequently and make decisions as soon as results are statistically significant which means that no additional analysis is necessary.

 

- False discovery rate control: Having multiple variations and goals in an experiment increases the likelihood of error, based on random chance alone and the challenge of multiple comparisons. Stats Engine accounts for this by adjusting its calculations based on the total number of variations and goals in an experiment. This results in a highly accurate confidence calculation, regardless of the number of goals and variations tested at once.

 

Optimizely has created a Stats Engine landing page to provide more information about using the platform.