SDS 435: Scaling Up Machine Learning — with Erica Greene

Feature Drift

Feature drift poses significant challenges in machine learning models at Etsy, as explained by Erica Greene. She shares a notable incident where a feature's distribution changed drastically, impacting model performance and highlighting the need for robust monitoring systems 1. Jon Krohn emphasizes the importance of having monitoring in place to detect such drifts and prevent disruptions 2.

Silent changes in feature distributions keep me up at night. We now have much more monitoring, but there's still a long way to go.

--- Erica Greene

Implementing effective monitoring and alerting systems is crucial to managing these changes and maintaining model accuracy.

   

Strategic Prototyping

Strategic prototyping is essential for selecting and developing machine learning initiatives at Etsy. Erica Greene describes a process where her team brainstorms potential projects and evaluates them based on risk and feasibility 3. This approach involves creating concise proposals to determine the viability of each idea, ensuring that resources are allocated effectively.

We try to be a little bit more strategic about it, making sure that there's a good basis for that.

--- Erica Greene

Jon Krohn appreciates this method, noting its importance in grounding work in practical business realities 4.