Can We Trust Scientific Discoveries Made Using Machine Learning? with Genevera Allen - TWiML...

Terminology

The discussion highlights the lack of adequate terminology to address reproducibility challenges in machine learning. Genevera Allen emphasizes the need for a robust framework to understand and communicate these issues effectively. She notes, "We don't even have terms to necessarily talk about this yet. So how can we expect practitioners and users to understand how to appropriately apply these techniques in these situations?" 1. This gap in terminology hinders the ability to measure and improve the reproducibility of data-driven discoveries 2.

   

Scientific Examples

Examples from cancer genomics illustrate the reproducibility issues in scientific research. Genevera Allen shares that while some clustering techniques have successfully identified breast cancer subtypes, similar methods have failed to produce consistent results in other cancer types 3. She explains, "One paper will come out and they'll make a big splash... and then another paper comes out maybe two years later on another cohort of patients that has a different clustering" 3. This inconsistency highlights the challenge of ensuring scientific discoveries are truly reproducible across different datasets and studies 4.