A/B Testing Mistakes to Avoid in Your Data Science Interview: Tips and Tricks!

In this video, I'm going to talk about a few mistakes that people often make when interpreting A/B testing results. You may often encounter these questions during an interview for data scientist positions. Want to know what these mistakes are and how to solve these questions correctly? Stay tuned!

Choosing the Right Metrics for A/B Testing

A/B Testing "Real-Life" Example: A Step by Step Walkthrough

Data Scientists Must Know: A/B Testing Fundamentals
🟢Get all my free data science interview resources

// Comment
Got any questions? Something to add?
Write a comment below to chat.

// Let's connect on LinkedIn:

====================
Contents of this video:
====================
0:00 Overview
1:26 Data Scientists' Roles
1:48 Data Peeking
3:40 Multiple Testing Problem
8:49 Lack of Statistical Power

[thegreatlazydazz]

sublime video, emma. I get the feeling you will explain every useful thing in the Hippo book, thus saving us all the bother of reading it.

[Gavindsa]

Emma your videos are amazing! Keep doing what you do. Looking forward to more such content

[janeli]

Hi Emma, thanks for keeping producing such high quality videos. Do you mind explain how to deal with multiple test problems in other scenarios that you mentioned, much as sliced data into multiple segments?

[snowguo]

Thank you! this is very helpful! Im done with this video. All the content are noted!

[kylechen]

thx for sharing Emma, i found these very helpful and actually thats what i encountered in real life work. mkt ran some wrong a/b testing without actually understanding it then asked me to analyze the result. maybe i should share your channel to them lol

[alexandragrishechko]

Your videos are super helpful, thank you!

[arunvaibhav]

You're really great!!! Please upload Facebook Data Scientist interview experience. Thanks!

[Max-yvlw]

Hi Emma! I really like your videos. They are very insightful and helped me a lot when I was preparing my DS interviews:)

[zhuyanshu]

Can you clarify: multi hypothesis problem arise by testing a segment? Control vs. only Web segment? Or multiple treatment group such as :Control vs. Web vs. IOS?

[PikaChu-thnb]

Great video. Very insightful. I would be interested to hear your thoughts on A/B tests set up to ensure that changes do not break anything.
For example if an e-commerce website begins to place ads on the website, they want to make sure that adding ads onto the website does not cause people to buy less. What would be a good way to think about tests like this?

[xueyingding]

Thanks for your videos! For using tiered sig. Lvls to check multiple metrics, how can we calculate the sample size? Should we calculate based on the most important metrics or should we calculate for each metric and choose the largest needed sample size?

[武恃名]

Cannot find the material on p119 in the hippo book, the whole chapter talks about ethics.

[jasdeepsinghgrover]

Good explanation but I think the last error is incorrectly handled.... Imagine you run an experiment and it is significant (you haven't checked the observed power yet), if you accept it then it is wrong but if you rerun it you just nearly doubled the p value. We should be only looking at the rerun or let the experiment have significant power (probably more than our threshold)

[Ayy]

Hi emma, this is very helpful! Thank you for making those videos, just a quick follow-up questions, I think testing multiple metrics in an A/B test is common; like usually we will pick one metric as the main metric and the couple other will just serve as support metrics; so if I just make whether or launch decision based on the significance of this one key metrics, it's fine, right?