Published May 26, 2020

114 - Behavioral Testing of NLP Models, with Marco Tulio Ribeiro

Join Marco Tulio Ribeiro as he delves into the inadequacies of NLP model evaluations, advocating for behavioral testing to bridge the gap between user expectations and current model performances, while addressing distributional shifts and real-world challenges in sentiment analysis.

Episode Highlights

Topics covered

Episode Highlights

Related Episodes

119 - Social NLP, with Diyi Yang
Answers 383 questions
35 - Replicability Analysis for Natural Language Processing, with Roi Reichart
Answers 383 questions
117 - Interpreting NLP Model Predictions, with Sameer Singh
Answers 383 questions
40 - On the State of the Art of Evaluation in Neural Language Models, with Gábor Melis
Answers 383 questions
81 - BlackboxNLP, with Afra Alishahi and Tal Linzen
Answers 383 questions
128 - Dynamic Benchmarking, with Douwe Kiela
Answers 383 questions
44 - Truly Low Resource NLP, with Anders Søgaard
Answers 383 questions
109 - What Does Your Model Know About Language, with Ellie Pavlick
Answers 383 questions
130 - Linking human cognitive patterns to NLP Models, with Lisa Beinborn
Answers 383 questions
115 - AllenNLP, interviewing Matt Gardner
Answers 383 questions
93 - NLP/ML for clinical data, with Alistair Johnson
Answers 383 questions
108 - Data-To-Text Generation, with Verena Rieser and Ondřej Dušek
Answers 383 questions
86 - NLP for Evidence-based Medicine, with Byron Wallace
Answers 383 questions
106 - Ethical Considerations In NLP Research, with Emily Bender
Answers 383 questions
92 - Computational Humanities, with David Bamman
Answers 383 questions

Dexa/NLP Highlights

114 - Behavioral Testing of NLP Models, with Marco Tulio Ribeiro

Topics covered

Popular Clips

Testing NLP Models

Real-World Generalization

User-Builder Disconnect

Model Limitations

Testing NLP Models

Testing Data Discrepancies

Testing Sentiment Analysis

Understanding Failure Rates

Testing NLP Models

Sentiment Analysis Failures

Testing NLP Models

Expert vs. Non-Expert

Testing NLP Models

Episode Highlights

User and Developer Roles

Real-world Generalization

Model Evaluation Issues

Behavioral Testing

Related Episodes

119 - Social NLP, with Diyi Yang

35 - Replicability Analysis for Natural Language Processing, with Roi Reichart

117 - Interpreting NLP Model Predictions, with Sameer Singh

40 - On the State of the Art of Evaluation in Neural Language Models, with Gábor Melis

81 - BlackboxNLP, with Afra Alishahi and Tal Linzen

128 - Dynamic Benchmarking, with Douwe Kiela

44 - Truly Low Resource NLP, with Anders Søgaard

109 - What Does Your Model Know About Language, with Ellie Pavlick

130 - Linking human cognitive patterns to NLP Models, with Lisa Beinborn

115 - AllenNLP, interviewing Matt Gardner

93 - NLP/ML for clinical data, with Alistair Johnson

108 - Data-To-Text Generation, with Verena Rieser and Ondřej Dušek

86 - NLP for Evidence-based Medicine, with Byron Wallace

106 - Ethical Considerations In NLP Research, with Emily Bender

92 - Computational Humanities, with David Bamman

114 - Behavioral Testing of NLP Models, with Marco Tulio Ribeiro

Topics covered

Popular Clips

Episode Highlights

User and Developer RolesMarco Tulio Ribeiro explores the disconnect between user expectations and builder implementations in NLP models. He highlights the role of non-experts in testing, advocating for a decentralized approach to evaluation.

User and Developer Roles

Real-world GeneralizationMarco Tulio Ribeiro discusses the challenges NLP models face in generalizing from training data to real-world environments. He emphasizes the importance of handling distributional shifts to maintain model performance.

Real-world Generalization

Model Evaluation Issues

Behavioral TestingMarco Tulio Ribeiro explores behavioral testing for NLP models, a framework inspired by software engineering. He discusses its significance in evaluating model behavior beyond traditional accuracy metrics, ensuring readiness for real-world applications.

Behavioral Testing

Related Episodes