665: How to be socially impactful and financially successful in your data career — with Josh Wills

Data Quality

Maintaining high data quality is crucial for the success of machine learning projects. Josh Wills emphasizes the need for immediate detection of data quality issues, advocating for real-time monitoring to prevent catastrophic failures 1. He highlights the challenge of integrating diverse perspectives within organizations, where different groups may disagree on quality metrics 2. This disagreement can lead to what he describes as the "infinite loop of sadness," where progress is hindered by conflicting priorities.

Simple analysis on top of high-quality data usually wins the day.

--- Josh Wills

Jon Krohn and Josh discuss the importance of establishing schemas and quality tests throughout the data pipeline to ensure data integrity and reliability 1.

   

ML Iteration

Iterative improvement is essential in machine learning, where models must constantly evolve to remain effective. Josh Wills discusses the importance of continual iteration, noting that without it, the effort in building models is not justified 3. He contrasts traditional A/B testing with more dynamic approaches like contextual bandits, which allow for faster adaptation to new data 3.

Debugging is harder, monitoring is harder, everything is harder than it is under regulators.

--- Josh Wills

Josh also highlights the complexities of debugging and monitoring in machine learning systems, especially when dealing with unpredictable elements like those introduced by chat GPT 4.