Experimentation vs AB Testing

Summary:

In this Whiteboard Wednesday session, the concept of high-impact experimentation is explored, and why it is superior to standard testing. The discussion focuses on optimizing for marginal improvements with high certainty versus aiming for a 10x change in business with higher risk. It's noted that A-B testing is not the ideal method for achieving significant growth. The importance of understanding context and incentives when choosing between high-certainty and high-risk experiments is emphasized. Additionally, the session highlights the need to identify the 'Ohio' in experiment design, where funds are directed to test uncertain ideas. The session concludes by distinguishing between bad bets and good experiments.

Chapters:

00:00 The Power of High Impact Experimentation
00:59 Context and Incentives in A-B Testing
03:25 Pursuing High-Risk Moonshots
04:22 Avoiding Bad Bets with Marginal Improvements