Volodymyr Kazantsev - Beyond t tests - How to Conclude an Online Experiment Using Python

PyData Dallas 2015
"A/B testing and control-group testing are very well-known techniques to learn about the market and consumer preferences. In reality, however, lots of companies make incorrect conclusions about their experiments, due to the lack of statistical knowledge. Moreover, there is surprisingly little material available about types of statistical tests that are appropriate in online setting. In this talk, I will try to establish a conceptual framework that we will use to analyse different types of experiments, starting with a simple “conversion” testing using null-hypothesis tests and move to more advanced topics, finishing with actual %% uplift measurement of heavily skewed data, such as payments data and long-term customer lifetime value (CLV), using Bayesian Credible Intervals. Agenda (Draft): 1. types of tests in online setting: product A/B, exclusion groups for marketing activities 2. goals of a test: yes/no vs. %% Uplift. Why do we care about the measuring the uplift? 3. brief overview of relevant inferential statistics (and how to do all that in python): central limit theorem, confidence intervals, t-test 4. We will apply those technique to real-life like data in ipython notebook 5. I will introduce and apply two alternative techniques that are actually used in production to reject null-hypothesis when dealing with heavily-skewed data 6. How to measure the uplift of CLV" 00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.