Solon discusses the inherent unreliability of training data, particularly in predictive policing and hiring practices. He highlights how biased historical decisions can distort model predictions, leading to perpetuated prejudices. The conversation emphasizes the need for innovative solutions to address these underlying data issues rather than merely tweaking model performance.