635: The Perils of Manually Labeling Data for Machine Learning Models — with Shayan Mohanty

Bias Types

Bias in machine learning is multifaceted, encompassing both technical and societal dimensions. Shayan Mohanty explains that while bias can be a model parameter, it can also lead to degenerative bias when outdated stereotypes influence data labeling 1. This type of bias skews models away from reality, highlighting the need for explicit bias checks. Jon Krohn adds that bias isn't inherently negative, as it can be used to adjust model outputs beneficially 2.

   

Degenerative Bias

Addressing degenerative bias is crucial in machine learning, especially when hand labeling data. Shayan notes that hand labeling can introduce biases, is slow to adapt to data drift, and incurs high costs 3. His company, Watchful, offers a solution by automating labeling through heuristics, reducing the reliance on manual processes. Jon emphasizes the importance of streamlining data labeling to improve efficiency and accuracy 4.

   

Labeling Challenges

The challenges of hand labeling data extend beyond bias, affecting reproducibility and efficiency. Shayan highlights the non-deterministic nature of manual labeling, which complicates data pipelines and model training 5. Automating weak supervision is a promising approach, as it uses ensembling techniques to generate labels from diverse signals. This method accelerates the labeling process and enhances model robustness 6.