2024-10-17 Quick Introduction To A/B Testing - Lightning Lesson

Topics covered
- The pros/cons for A/B Testing.
When should you run A/B tests and why?
- Necessary ingredients.
How many users do you need (power)? What are you optimizing for (the OEC)?
- Humbling statistics.
Most experiments fail to improve the metrics they were designed to improve. Plan to fail fast and pivot often
- Twyman's Law.
That A/B test that improved purchases by 20% (e.g., by rounding the buttons) is bogus. Be skeptical.

YouTube muted Otis Redding's song at 23:43-23:59 (copyrighted). You can listen to it here: