Model Performance Monitoring and Why You Need it Yesterday // Amit Paka // MLOps Coffee Sessions #42

MLOps.community

Coffee Sessions #42 with Amit Paka of Fiddler AI, Model Performance Monitoring.

//Abstract
Machine Learning accelerates business growth but is prone to performance degradation due to its high reliance on data. Moreover, MLOps is often fragmented in many organizations, causing frictions to debug models in production. With new rules from the EU that focus on trust and transparency, it’s becoming more important to keep track of model performance. But how? We propose a new framework, a centralized ML Model Performance Management powered by Explainable AI. Learn more about how you can stay compliant while maximizing your model performance at all times with explainability and continuous monitoring.
//Bio
Amit is the co-founder and CPO of Fiddler, a Machine Learning Monitoring company that empowers companies to efficiently monitor and troubleshoot ML models with Explainable AI. Prior to founding Fiddler, Paka led the shopping apps product team at Samsung. Paka founded Parable, the Creative Photo Network, now part of the Samsung family. He also led PayPal's consumer in-store mobile payments launching innovations like hardware beacon payments and has developed successful startup products particularly in online advertising - paid search, a contextual, ad exchange, and display advertising. Paka has passions for actualizing new concepts, building great teams, and pushing the envelope, and aims to leverage these skills to help define how AI can be fair, ethical, and responsible.
--------------- ✌️Connect With Us ✌️ -------------
Join our slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register
Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/
Connect with Amit on LinkedIn: https://www.linkedin.com/in/amitpaka/
Timestamps:
[00:00] Thank you to Fiddler AI!
[00:46] Introduction to Amit Paka
[05:04] Amit's background in tech
[09:55] EU Regulation
[12:39] "The goal that the EU seems to be going for is they want to go for helping build human-centric and responsible AI."
[13:28] 4 AI Categories:
1. Unacceptable risk applications
2. High-risk applications
3. Limited risk applications
4. Minimal risk applications
[14:58] Deep dive into High-risk applications
[17:28] Digital Services Act (DSA) and Digital Marketing Act (DMA)
[19:02] Military
[19:33] "They don't know what they don't know and they probably wanted the door open."
[21:13] US on JIC Team - transparency and increasing trustworthiness on AI
[23:06] Diversity of industries and Explainability
[24:22] "The urgent need for Explainability comes from verticals that are facing the problems today on the ground and cannot run their business." [30:09] Model Performance Management (MPM)
[34:05] "When your model is facing issues, you now have to root-cause it within life."
[35:40] Control Theory
[36:10] "Control Theory means that you do not just measure it but you can influence it so you can actually keep it."
[38:14] Abstraction into being useful
[43:23] "You can train a model that accurately represents the reality."
[44:00] Data scientist doing ML Flow
[49:55] Amit's favorite surprise!
[53:04] Banking and Insurance adoption of ML
[55:48] Advise ML Scientists and Data Scientists in terms of Explainable AI
[58:25] "Models are incredibly hard to debug. You're just training a model for high accuracy but you don't know how that accuracy is distributed."
[59:49] Linking of EU Regulation and MPM