Holistic Health Metrics of ML Based Product

💻 Abstract: 
Metrics Holistic Health Metrics of ML Based Product. The common objective for ML-based products is to always be sure that the whole system produces correct results. Depending on a domain, this problem could be less or more critical. For example, in the finance and health industries, errors could be very pricey. Building a holistic metrics system that covers all blocks of the application will help to avoid errors, react fast when they occur, and reduce manual work.
In my presentation, I will describe WHERE and WHEN to collect those metrics (data level, model level, post-model level, business level; after training, regularly on prod, live alerting on product run). Also, I will mention important aspects that you should pay attention to making the metrics easy accessible; taking time and effort for new metrics tool to be adapted by your colleges; not hesitating to alter metrics to fulfill your needs
The model's accuracy is not enough. Learn how to cover your product with metrics so you can finally sleep better.

🔊 Speaker bio:
Lina Palianytsia- Lead Machine Learning Engineer of GlobalLogic
Machine Learning Engineer with a focus on quality and stability of ML based products.
Major in Math and Finance.

Timestamps:

0:00 Intro
0:33 Introduction of the speaker
0:48 Contents
1:19 Definition of Holistic Health Metrics
3:20 Metrics tool is a Dashboard
5:06 Metrics are like a safety net
6:06 There are two questions to face when you decide that you want to make this metrics tool
6:24 Options when to measure metrics
7:29 Options where to measure metrics
9:58 12 Decisions to make in metrics design
12:24 Use thresholds to notify you
14:32 Recommendations

❓ Q&A ❓

17:13 What are some examples of more nuanced metrics you might have seen in projects?
17:50 What is your favorite visualization package tool?
18:17 Have you used/tried "probability of model prediction" for detecting model health/deviance?
19:01 How do you log metrics in ML flow?
20:12 Is it good or bad to consolidate different health metrics into one overall aggregate health metric?
21:02 Can you extrapolate on the Bermuda Triangle in one of the slides when monitoring metrics for multiple models?

24:20 Closing remarks