Sean Taylor — Business Decision Problems

Model Evaluation

Sean Taylor emphasizes the complexity of model comparison and evaluation in machine learning. He notes that while the space of models is rapidly expanding, determining if a new model is superior to an existing one remains a nuanced challenge. Sean highlights the importance of considering various factors beyond traditional metrics like AUC and precision-recall curves, such as cost, stability, and interpretability 1.

We focus so much effort on training models, getting features on all crazy architectures. The space of models that we can consider is increasing rapidly, but we still are bottlenecked on, like, is this model better than the one that we already had?

--- Sean Taylor

He suggests exploring posterior predictive checks and off-policy evaluation as methods to improve model assessment, ensuring they are fit for deployment 2.

   

Deployment Challenges

Deploying machine learning models into production presents significant challenges, particularly in managing training data and ensuring model reliability. Sean Taylor points out that preparing datasets for model consumption is often a slow process, despite technological advancements like feature stores 3. He also discusses the importance of maintaining trust in models once deployed, as errors can have substantial consequences, such as disrupting Lyft's marketplace 3.

It's like a really big downside risk to losing reliability. So getting to the point where we trust the decisions and that we can.

--- Sean Taylor

Sean advocates for simpler solutions, like tree-based models, which have historically provided reliable results at Lyft. He acknowledges the potential of neural networks but stresses the need for gradual integration to ensure consistent performance 4.