Mastering Recommendation Systems Evaluation: An A/B Testing Approach with Insights from the Industry

This talk aims to provide attendees with a practical understanding of A/B testing in the evaluation of recommendation systems, including unique insights from industry practices and specific tricks that enhance effectiveness.

My report includes next steps:
- Introduction to recommendation systems, their ubiquity, and the imperative for evaluation, including industry examples.
- Techniques for designing effective hypotheses for A/B tests, focusing on recommendation systems.
- Choosing pertinent metrics for robust evaluation of recommendation systems with industry examples.
- Conducting A/B tests: industry best practices, common pitfalls, and strategies for mitigation, reinforced by real-world cases.
- Accurate interpretation of A/B testing results and management of statistical biases, with insights from the field.
By the end of the talk, attendees will have a comprehensive understanding of how to apply A/B testing effectively to recommendation systems, select relevant metrics, interpret results accurately, and navigate common challenges, backed by industry best practices and practical examples.

Bio:
Ildar Safilo
Experienced manager in MLE/DS/SE/DA, I possess extensive expertise in machine learning, analytics, and software engineering. I excel at leading teams to create groundbreaking businesses and delivering innovative solutions for real-world business cases across various industries, including IT, banking, telecommunications, marketplaces, game development, shops, Travel-tech and streaming platforms.
Expert in building recommendation and ranking systems, as well as personalization automation with machine learning, and advanced A/B testing.
Co-author and lecturer of a popular online course on recommender system development with over 1000 students.
Co-author an open-source Python library called RecTools, specifically designed for building recommender systems. The library is hosted on GitHub at RecTools and has received widespread recognition and adoption in the industry.
Graduate with a Master’s degree in Mathematics and Computer Science and over 6 years of experience in data science.

===

PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.

PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.

00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.

[kseniialisitsina]

Amazing presentation. Always a challenge to compare models in offline setup.

[sergeykosyy]

Good explanation.
Maybe you can provide some concrete examples in the future (what purpose do you have, initial state, what challenges do you have and how do you achieve results).
I understood about nda, commerce secrets etc. But maybe it will be possible in the future.