[CW Paper-Club] Introduction to Conformal Prediction & Distribution-Free Uncertainty Quantification

In this week's session, Fábio Oliveira presents the paper "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification" published in 2021 by Anastasios N. Angelopoulos and Stephen Bates.

This paper introduces conformal prediction as a user-friendly approach for quantifying uncertainty in black-box machine learning models, crucial in high-risk applications like medical diagnostics. The method provides distribution-free, statistically rigorous prediction sets for any pre-trained model, ensuring a user-specified probability of containing the ground truth. The paper offers a hands-on introduction with practical theory, examples, and Python code, making it applicable to various machine learning tasks.

If you're interested in joining our team at CloudWalk, please check out our job openings on LinkedIn! Don't forget to check out the paper, which is available at the link provided below.

Recording Date: November 11 2023
00:00 Paper presentation

CW Paper-Club Sessions is a weekly meeting presented by CloudWalk's R&D team members, where they dive into interesting research papers, providing valuable insights into the latest developments in technology and research.

Full Paper Abstract:
Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction (a.k.a. conformal inference) is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 90%. It is easy-to-understand, easy-to-use, and general, applying naturally to problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on. This hands-on introduction is aimed to provide the reader a working understanding of conformal prediction and related distribution-free uncertainty quantification techniques with one self-contained document. We lead the reader through practical theory for and examples of conformal prediction and describe its extensions to complex machine learning tasks involving structured outputs, distribution shift, time-series, outliers, models that abstain, and more. Throughout, there are many explanatory illustrations, examples, and code samples in Python.