Operationalizing Machine Learning: Interview with Shreya Shankar

Early Validation

In the realm of machine learning, early validation is crucial for successful deployment. Shreya Shankar emphasizes that validating models as early as possible can prevent the costly mistake of deploying ineffective models. She notes, "You don't want to push a bad model to production. You don't want to wait until your final stage of A/B testing in order to find out that something is not going to work well" 1. This proactive approach not only enhances the velocity of deployment but also ensures that models are robust and reliable before reaching production 2.

   

Reproducibility

Reproducibility in machine learning is a complex challenge that requires a nuanced approach. Shreya suggests that exact reproducibility is often unattainable due to variables like infrastructure and data changes. She questions, "If I have the artifact but I can't reproduce that artifact, does that help me?" 3. Instead, she advocates for a focus on improving reproducibility incrementally, which can enhance team functionality and governance 4.

   

Tooling Layers

Understanding the layers of ML tooling is essential for optimizing workflows. Shreya outlines a four-layer framework, from the frequently changed run layer to the foundational infrastructure layer. She explains, "It's the easiest to do in the run layer. It is much harder to kick something out of the infrastructure layer and replace that" 5. This layered approach helps in identifying where changes can be made with minimal disruption, guiding ML engineers in tool selection and integration 6.