The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

Ronny Kohavi, PhD, is a consultant, teacher, and leading expert on the art and science of A/B testing. Previously, Ronny was Vice President and Technical Fellow at Airbnb, Technical Fellow and corporate VP at Microsoft (where he led the Experimentation Platform team), and Director of Data Mining and Personalization at Amazon. He was also honored with a lifetime achievement award by the Experimentation Culture Awards in September 2020 and teaches a popular course on experimentation on Maven. In today’s podcast, we discuss:
• How to foster a culture of experimentation
• How to avoid common pitfalls and misconceptions when running experiments
• His most surprising experiment results
• The critical role of trust in running successful experiments
• When not to A/B test something
• Best practices for helping your tests run faster
• The future of experimentation

—

Where to find Ronny Kohavi:

Where to find Lenny:

In this episode, we cover:
(00:00) Ronny’s background
(04:29) How one A/B test helped Bing increase revenue by 12%
(09:00) What data says about opening new tabs
(10:34) Small effort, huge gains vs. incremental improvements 
(13:16) Typical fail rates
(15:28) UI resources
(16:53) Institutional learning and the importance of documentation and sharing results
(20:44) Testing incrementally and acting on high-risk, high-reward ideas
(22:38) A failed experiment at Bing on integration with social apps
(24:47) When not to A/B test something
(27:59) Overall evaluation criterion (OEC)
(32:41) Long-term experimentation vs. models
(36:29) The problem with redesigns
(39:31) How Ronny implemented testing at Microsoft
(42:54) The stats on redesigns 
(45:38) Testing at Airbnb
(48:06) Covid’s impact and why testing is more important during times of upheaval 
(50:06) Ronny’s book, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
(51:45) The importance of trust
(55:25) Sample ratio mismatch and other signs your experiment is flawed
(1:00:44) Twyman’s law
(1:02:14) P-value
(1:06:27) Getting started running experiments
(1:07:43) How to shift the culture in an org to push for more testing
(1:10:18) Building platforms
(1:12:25) How to improve speed when running experiments
(1:14:09) Lightning round

Referenced:

Lenny may be an investor in the companies discussed.

[sriramvenkitachalam]

I can't believe this channel is free!

[rudman]

"be ready to fail" -> one of my main takeaways from

truly enjoyed this episode (can't find a place to rate on spotify so came over to youtube to like it!)

[sovlog]

BIG THX, Lenny and Ronny! Amazing content!

[couponbote]

So did I understand well that if a small company has only a few thousand website visitors A/B testing doesn't make much sense? (minute 27:00)

[abhisheku]

Thanks you so much for make this video. I want to share your video to my friend.

[antoniomarraffa]

What about pre-post tests, to avoid really testing everything?

[bartoszzmarzy]

Damn, that was so eye opening! And it’s so pleasant to listen to Ronny! Super meaty episode, thanks for it🙌

[olhalukianova]

The book was translated to russian, what a great achievement 🤮🤮🤮

Though, I like this episode. Thank you!

[bootsandladders]

I like this guy! @Ronny please reach out if you want to be apart of our very lean startup :)

[zhefeng]

all good! only shows $100 off instead of $500