Online Experiments and Company Success

This piece writes on the summary and a short reflection on the article “The Surprising Power of Online Experiments” by Harvard Business School.

Bing’s experiments showed that slightly darker blues and greens in titles and a slightly light black in captions improved the users’ experience. When rolled out to all users, the color changes boosted revenue by more than $10 million annually.

Key summaries and ideas of the article:

1. All leading technology and platform companies leverage on controlled experiments (over 10,000 annually) engaging millions of users to test the market. This “experiment with everything” approach yielded surprisingly large payoffs.
2. Though the business world glorifies big, disruptive ideas, in reality most progress is achieved by implementing hundreds or thousands of minor improvements. Opening links in new tabs (instead of in the current tab) has increased “clicks per user” by 5%, in a 12-million users experiment.
3. Online experiments helped Microsoft Bing to understand the trade-off of investment intime of search vs. optimality of search. When improving the relevance of search results slows the software’s response time, experimentation is a good way to judge which is better.
4. Stay positive with insignificant experimental results. At Google and Microsoft, only about 10% to 30% of experiments generate positive results.
5. A lack of infrastructure will keep the marginal costs of testing high and could make seniors managers reluctant to call for more experimentation. There are several recommendations on the organisation of experimentation personnel, i.e. (1) centralised model(a team of data scientists serving the entire company), (2) decentralised model(distributing data scientists throughout different business units), and (3) center-of-excellence model(a hybrid of (1) and (2)).
6. Keys to manage online experiments: (1) coming up with an overall evaluation criterion (OEC) as KPIto align the expectations on executives and data analysts and (2) beware of low-quality data (conduct A/A tests!!), and finally, (3) avoid assumptions about causality.

At a time when the web is vital to almost all businesses, rigorous online experiments should be standard operating procedures. … Any company that has at least a few thousands daily active users can conduct these tests.

Some reflection and discussion:

1. This is one of the best HBR pieces that I have read, in a sense that it gave me brand new perspectives, and the current practice of online experiments among technology firms. I believe everyone of us should have spotted some minor changes to the Google/Facebook interface you were using at some point in time. While we may also acknowledge that it was an online experiment, at least I do not know that it could be one of 10,000 experiments conducted annually.
2. It is definitely surprising to know the colour change on Bing has boosted their marketing revenue by $10 million per annum, especially with the fact that this incremental is grounded by actual consumer preference and experimentation, instead of theoretical psychology.
3. Facing big data, or just a tremendous dataset (e.g. with > 1 million rows of records), we often assume the data is “big enough” to ensure randomness. It is tempting to draw causal conclusion, estimate and infer effects on other situation from the training data — which is likely not the case. A/A testing is very useful to ensure correct interpretation and knowledge can be learnt from data.
4. Maximizing the value and exploiting the power of online experiments is much easier said than done. Only when companies are up to some economies of scale could they afford a dedicated data scientist team for robust, controlled online experiments. It requires different types of expertise in experimental design, statistical analysis and proper validation.
5. For many readers who do not specialise in Statistics, it is high time for you to review more on the concept of online experiments. Even if you are not working with a company that tracks web traffic, you must also benefit from learning the ways to design experiments and gaining insights from statistical analysis.