ROI Calculation on Online Controlled Experiment

As online services such as e-commerce and mobile applications keeps growing, the need of optimizing the user experience does as well. By conducting Online Controlled Experiments, companies can get an insight to which features, design choices and implementations that users enjoy the most. This projects explores two areas in relation to Return on Investment (ROI) calculations for Online Controlled Experiments. First, challenges and pitfalls with calculating a reliable ROI metric are described as well as references to research working to mitigate these challenges. The challenges presented are all related to how to accurately measure the effect between two candidate variants which sums up the main challenges with ROI calculation. Secondly, a model based on expected return on investment is constructed and explored in order to investigate whether the model can help to optimize test parameters for a two sample t-test in the setting of Online Controlled Experiments. The results of the model analysis shows that the model has limited practical use since it maximizes the ROI - quotient without taking into account to magnitude of potential revenue increase as well as potential cost.What color should this button on our website be? Should we rearrange the current layout on our e-commerce site? Should our app allow login credential from social apps? These are all questions Online Controlled Experiments can help to answer. As the number of online services such as e-commerce and online advertising keeps growing, along with proliferation of data generated by user engagement of these services, opportunities to conduct controlled experiments to empirically test different versions of the services increases. Tests conducted usually entails but are not limited to; design variation such as color and layout changes to the inclusion or exclusion of certain features, functions or technical implementations. Today, online controlled experiments are used to great extent across the industry to improve online services. Technology giants such as Amazon, Facebook and Google conduct more than 10.000 experiments annually and has reported large gains from the experiments. For instance, in 2008 Microsoft conducted a test where users would be redirected to a new window/tab when clicking on the Hotmail link on the MSN home page instead of staying in the same window. Initial tests confined to 900.000 UK users showed a user engagement increase measured by number of clicks made on the MSN home page by 8.9%. Later in 2010, the same test was conducted on the US market with 2.7 million users where the same metric showed a 5% increase. Till this day, this simple technique is still widely in use by major websites such as Facebook and Twitter. In order to conduct these online controlled experiments, a company must acquire both infrastructure and competence which obviously incurs costs. For the company to deem these costs financially viable, some kind of return on investment (ROI) calculation is needed. Today, there is no standardized way of conducing ROI calculations and this is due to challenges in measuring the actual improvements from different versions – we can’t accurately calculate what we can’t accurately measure. Think of it this way; you have a company that sells Apples. What is your return on investment? Well, if you don’t know how many apples you sold, that task suddenly becomes quite hard. In order to measure potential improvements between variants, we use something called statistical tests. There are a lot of different kind of statistical test one can use to measure the differences but all of them involves some kind of uncertainty. In simplified terms, it is due to this uncertainty the challenge of accurate measurement arises and we talk about a few of those cases in this project. To adjust for these uncertainties, a test practitioner can adjust some test parameters. We also investigate with a model whether these test parameters can be chosen based on some financial metric. While it is shown that the model can’t effectively be used to choose test parameters, it can be used to check whether some chosen test parameters makes sense to use