LIVE FROM TWIMLcon! Overcoming the Barriers to Deep Learning in Production with Andrew Ng - #304

Complexity

Andrew Ng discusses the distinction between essential and accidental complexities in machine learning, emphasizing the importance of addressing these challenges effectively. He highlights how modern machine learning, particularly deep learning, has simplified workflows by removing many of the essential complexities that previously existed in software engineering 1. Ng notes that while tools have improved, the core challenge remains in clearly defining the problem and sourcing the right data.

The heart of what makes a machine learning problem difficult is still thinking through clearly what is the problem you want to solve and where to get the data.

--- Andrew Ng

He also points out the ongoing evolution of machine learning workflows, comparing it to the development of version control systems, and stresses the need for better tools for data editing and versioning 2.

   

Robustness

Ng addresses the challenges of robustness and generalization in machine learning models, particularly when models are deployed in varied data environments. He explains that while academic research often overlooks these issues, they are critical in real-world applications where models must adapt to new data distributions 3. Ng shares that his team employs a rigorous process called FMEA analysis to anticipate and mitigate risks in machine learning deployments.

We like going through a very rigorous exercise where we, you know, pre mortem or think through all the risks and have a team to discuss and debate.

--- Andrew Ng

This approach helps in planning for unexpected scenarios and ensuring models perform reliably in production settings 4.

   

Experimentation

Ng highlights the importance of rapid experimentation in machine learning to foster iterative improvement and innovation. He describes a unique workflow used by some of his teams, involving daily sprints that allow for quick error analysis and experimentation 5. This method contrasts with traditional agile development cycles and is particularly effective for machine learning tasks that require frequent adjustments.

In machine learning, it turns out a lot of workflow building a machine learning system, it feels more like debugging than development.

--- Andrew Ng

Ng also mentions the need for better tools to support these rapid cycles, especially in terms of data versioning and collaboration 2.