Published Aug 27, 2018

67 - GLUE: A Multi-Task Benchmark and Analysis Platform, with Sam Bowman

Sam Bowman delves into the GLUE benchmark, a pioneering framework for evaluating natural language understanding models, exploring its impact on model architecture, task selection, and the pivotal role of diagnostic datasets in assessing linguistic capabilities.

Episode Highlights

Topics covered

Episode Highlights

Related Episodes

40 - On the State of the Art of Evaluation in Neural Language Models, with Gábor Melis
Answers 383 questions
79 - The glass ceiling in NLP, with Natalie Schluter
Answers 383 questions
128 - Dynamic Benchmarking, with Douwe Kiela
Answers 383 questions
22 - Deep Multitask Learning for Semantic Dependency Parsing, with Noah Smith
Answers 383 questions
39 - Organizing the SemEval task on scientific information extraction, with Isabelle Augenstein
Answers 383 questions
44 - Truly Low Resource NLP, with Anders Søgaard
Answers 383 questions
92 - Computational Humanities, with David Bamman
Answers 383 questions
63 - Neural Lattice Language Models, with Jacob Buckman
Answers 383 questions
52 - Sequence-to-Sequence Learning as Beam-Search Optimization, with Sam Wiseman
Answers 383 questions
81 - BlackboxNLP, with Afra Alishahi and Tal Linzen
Answers 383 questions
115 - AllenNLP, interviewing Matt Gardner
Answers 383 questions
102 - Biomedical NLP research at the National Institute of Health with Dina Demner-Fushman
Answers 383 questions
64 - Neural Network Models for Sentence Pair Tasks, with Wuwei Lan and Wei Xu
Answers 383 questions
114 - Behavioral Testing of NLP Models, with Marco Tulio Ribeiro
Answers 383 questions
106 - Ethical Considerations In NLP Research, with Emily Bender
Answers 383 questions

Dexa/NLP Highlights

67 - GLUE: A Multi-Task Benchmark and Analysis Platform, with Sam Bowman

Topics covered

Popular Clips

Understanding Textual Entailment

Diagnostic Dataset Insights

Model Evaluation Insights

Model Phenomena Categories

Multitask Model Insights

Language Understanding Tasks

Bridging Generalization Gaps

Pre-training Insights

Generalization in NLP

Diagnostic Task Insights

Leaderboards in NLP

Sentence Vectors Debate

Evaluating Sentence Models

Episode Highlights

Diagnostic Dataset Design

GLUE Benchmarking

Model Architecture

Related Episodes

40 - On the State of the Art of Evaluation in Neural Language Models, with Gábor Melis

79 - The glass ceiling in NLP, with Natalie Schluter

128 - Dynamic Benchmarking, with Douwe Kiela

22 - Deep Multitask Learning for Semantic Dependency Parsing, with Noah Smith

39 - Organizing the SemEval task on scientific information extraction, with Isabelle Augenstein

44 - Truly Low Resource NLP, with Anders Søgaard

92 - Computational Humanities, with David Bamman

63 - Neural Lattice Language Models, with Jacob Buckman

52 - Sequence-to-Sequence Learning as Beam-Search Optimization, with Sam Wiseman

81 - BlackboxNLP, with Afra Alishahi and Tal Linzen

115 - AllenNLP, interviewing Matt Gardner

102 - Biomedical NLP research at the National Institute of Health with Dina Demner-Fushman

64 - Neural Network Models for Sentence Pair Tasks, with Wuwei Lan and Wei Xu

114 - Behavioral Testing of NLP Models, with Marco Tulio Ribeiro

106 - Ethical Considerations In NLP Research, with Emily Bender

67 - GLUE: A Multi-Task Benchmark and Analysis Platform, with Sam Bowman

Topics covered

Popular Clips

Episode Highlights

Diagnostic Dataset DesignThe GLUE diagnostic dataset offers a comprehensive framework for evaluating natural language understanding by focusing on specific linguistic phenomena. Sam Bowman discusses the dataset's construction and its role in providing insights into model capabilities.

Diagnostic Dataset Design

GLUE BenchmarkingSam Bowman introduces GLUE, a benchmark designed to evaluate natural language understanding models. He discusses its development, task selection, and the role of leaderboards in NLP research.

GLUE Benchmarking

Model ArchitectureSam Bowman provides insights into the GLUE platform's approach to evaluating model architectures and multitask learning. He discusses the significance of pre-training in enhancing model performance and the potential for future improvements in generalization.

Model Architecture

Related Episodes