Omer discusses the challenges of overfitting in large datasets and the effectiveness of dropout techniques. He highlights the evolution of model vulnerabilities, noting that while finding systematic ways to fool models is becoming more difficult, the pursuit of new methods remains an intriguing area of research. The conversation underscores the importance of pre-training in improving model robustness against simple tricks that previously led to failures.
