AI Today Podcast #133: Challenges of Managing ML Models: Interview with Luke Marsden, Dotscience
Topics covered
Popular Clips
Episode Highlights
Deployment
Deploying machine learning models from development to production presents significant challenges. explains that many companies struggle to transition models from a data scientist's laptop to a production environment, often hitting a 'brick wall' due to the complexity involved 1. This complexity is exacerbated by the need for DevOps engineers to understand machine learning frameworks and tools, which are not as mature as traditional software development tools 2.
The biggest problem that, like I said, was a bit of a surprise when we discovered it, was that a lot of companies just struggle to get machine learning models into production at all.
---
Additionally, the process involves multiple inputs like constantly changing data, hyperparameters, and metrics, making the deployment loop far more intricate than traditional software 2.
Monitoring
Monitoring machine learning models in production is another complex task. Unlike traditional software, where metrics like request latency and error rates are sufficient, machine learning models can produce erroneous predictions even if these metrics appear normal 3. illustrates this with an example of autonomous vehicles misclassifying road signs due to changes in environmental conditions like snow 4.
If the number of stop signs that you're actually seeing drops below a threshold of the number that you're expecting to see, then you really want to page a human.
---
This necessitates advanced statistical techniques and alert systems to monitor the model's behavior effectively 4.
Version Control
Version control in machine learning adds another layer of complexity. Traditional version control systems like GitHub are not designed to handle the iterative nature of model development, where data and parameters are constantly changing 5. points out that data scientists often fail to record every change, leading to issues with reproducibility and traceability 5.
Humans are bad at remembering to record every single thing they do.
---
This lack of comprehensive version control can cause significant problems when models are deployed into production and later need to be audited or reproduced 5.