What metrics should we use?

Determining the right metrics to use depends heavily on the context and goals of your project. Here are some insights from different experts:

1. Model-Specific Metrics: Jennifer Prendki discusses the importance of having tailored metrics for each model. For example, in retail, metrics like add-to-carts or revenue are used for A/B tests, but it's essential to have specific metrics for different phases like understanding and guiding the customer 1.

2. Business Health Metrics: Yaniv Bernstein emphasizes the need for high-level Key Performance Indicators (KPIs) in startups. Metrics such as revenue growth, customer conversion, and churn rates are crucial. Chris Saad adds that selecting the right North Star metric, such as transactions in a marketplace, can significantly impact business success 2.

3. Product Metrics: Shreyas Doshi breaks down product metrics into categories like health metrics, usage metrics, adoption metrics, satisfaction metrics, ecosystem metrics, and outcome metrics. He stresses that adoption metrics should not be the only focus; understanding what leads to adoption (usage metrics) is also crucial 3.

   • 

     1

     Metrics for Model Performance

     00:25:32
   • 

     2

     Metrics for Startups

     00:06:49
     
   • 

     3

     Understanding Product Metrics

     00:06:54
   • 

     4

     Metrics and Manipulation

     00:42:44
     
   • 

     5

     Reevaluating Metrics

     00:25:12
   • 

     6

     Metrics and Monitoring

     00:23:17
     

4. Avoiding Metric Manipulation: Gurwinder Bhogal highlights the risk of Goodhart's Law, where focusing on a single metric can lead to it being gamed. It's important to ensure metrics reflect genuine progress rather than just being targets that can be manipulated 4.

5. Empirical Evaluation: Sameer Singh points out that while metrics are useful, their scope and what they actually measure need to be clear. Metrics should not be the sole way of evaluating performance, and it's important to understand what you're measuring and if it's the right thing 5.

6. Trends Over Absolute Numbers: Bernhard Merckle and Eberhard Wolff discuss how the trend of metrics over time can be more insightful than absolute numbers. Absolute metrics can help identify outliers, but trends can indicate the evolution of software quality and other long-term changes 6.

Key Takeaways:

• Tailor metrics to the specific project and phase.
• Use high-level KPIs to measure overall business health.
• Consider multiple categories of product metrics.
• Be cautious of metric manipulation and ensure metrics reflect genuine progress.
• Use trends over time to gain deeper insights.

These approaches ensure a comprehensive and effective measurement strategy, aligning metrics with overall business goals and model performance.