Peter Skomoroch — Product Management for AI

Uncertainty

Navigating the uncertainty inherent in machine learning projects is a significant challenge for product managers. Peter Skomoroch explains that unlike traditional software, which is deterministic, machine learning is probabilistic, leading to unpredictable outcomes 1. This unpredictability stems from various factors, including algorithmic randomness and differing data sets, making it difficult to plan and commit to deadlines. Skomoroch highlights the importance of benchmarks, as seen in projects like the Netflix Prize, to manage this uncertainty effectively 2.

   

Benchmarks

Constructing precise benchmarks is crucial for the success of AI projects. Peter Skomoroch emphasizes that many teams fail because they skip the hard work of creating clear benchmarks, which are essential for evaluating model performance 3. Without these benchmarks, teams struggle to debug issues and measure progress accurately. Skomoroch notes that benchmarks provide clear objectives and help teams avoid the pitfalls of relying solely on proxy metrics like A/B testing 2.

   

Planning

AI project planning differs significantly from traditional software planning due to its inherent uncertainties. Peter Skomoroch explains that most machine learning projects resemble R&D efforts rather than straightforward software development 4. Effective planning involves aligning projects with business goals and ensuring they have a measurable impact. Skomoroch points out that AI projects often lack clear business outcomes, making it essential to integrate uncertainty management into the planning process 1.