QuantUniversity Guest Lecture series: The Disagreement Problem in Explainable Machine Learning

Abstract : As various post hoc explanation methods are increasingly being leveraged to explain complex models in high-stakes settings, it becomes critical to develop a deeper understanding of if and when the explanations output by these methods disagree with each other, and how such disagreements are resolved in practice. However, there is little to no research that provides answers to these critical questions. In our work, we introduce and study the disagreement problem in explainable machine learning. More specifically, we formalize the notion of disagreement between explanations, analyze how often such disagreements occur in practice, and how do practitioners resolve these disagreements. To this end, we first conduct interviews with data scientists to understand what constitutes disagreement between explanations generated by different methods for the same model prediction, and introduce a novel quantitative framework to formalize this understanding. We then leverage this framework to carry out a rigorous empirical analysis with four real-world datasets, six state-of-the-art post hoc explanation methods, and eight different predictive models, to measure the extent of disagreement between the explanations generated by various popular explanation methods. In addition, we carry out an online user study with data scientists to understand how they resolve the aforementioned disagreements. Our results indicate that state-of-the-art explanation methods often disagree in terms of the explanations they output. Our findings also underscore the importance of developing principled evaluation metrics that enable practitioners to effectively compare explanations.

Satya is a PhD student at Harvard University working on trustworthy aspects of machine learning with Prof. Hima Lakkaraju and Prof. Finale Doshi-Velez. Before starting his PhD, he worked at Amazon Alexa on several responsible AI initiatives. He has published in the areas of algorithmic fairness, differential privacy, and machine learning explainability at prominent ML conferences such as ACL, FAccT, EACL, CHI, and ICLR. Satya completed his masters from Carnegie Mellon University (Pittsburgh), where he worked on building smart conversational agents.

[harry_rotter]

Thank you very much for this video, I need to make a presentation about this exact paper and I wasn´t really sure what the paper was about. But now I have full insight.