Published Jan 29, 2019

80 - Leaderboards and Science, with Siva Reddy

Siva Reddy delves into the dual nature of NLP leaderboards, exploring their ability to democratize research while also risking overfitting and transparency issues. Alongside hosts Waleed Ammar and Matt Gardner, he proposes solutions to enhance their integrity and fairness, such as using diverse test sets and automation.

Episode Highlights

Topics covered

Episode Highlights

Related Episodes

119 - Social NLP, with Diyi Yang
Answers 383 questions
114 - Behavioral Testing of NLP Models, with Marco Tulio Ribeiro
Answers 383 questions
128 - Dynamic Benchmarking, with Douwe Kiela
Answers 383 questions
142 - Science Of Science, with Kyle Lo
Answers 383 questions
131 - Opportunities and Barriers between HCI and NLP, with Nanna Inie and Leon Derczynski
Answers 383 questions
117 - Interpreting NLP Model Predictions, with Sameer Singh
Answers 383 questions
43 - Reinforced Video Captioning with Entailment Rewards, with Ramakanth and Mohit
Answers 383 questions
44 - Truly Low Resource NLP, with Anders Søgaard
Answers 383 questions
113 - Managing Industry Research Teams, with Fernando Pereira
Answers 383 questions
37 - On Statistical Significance, Training Variance, and Why Reporting Score Distributions Matters
Answers 383 questions
67 - GLUE: A Multi-Task Benchmark and Analysis Platform, with Sam Bowman
Answers 383 questions
81 - BlackboxNLP, with Afra Alishahi and Tal Linzen
Answers 383 questions
93 - NLP/ML for clinical data, with Alistair Johnson
Answers 383 questions
102 - Biomedical NLP research at the National Institute of Health with Dina Demner-Fushman
Answers 383 questions
79 - The glass ceiling in NLP, with Natalie Schluter
Answers 383 questions

Dexa/NLP Highlights

80 - Leaderboards and Science, with Siva Reddy

Topics covered

Popular Clips

Leaderboard Pitfalls

NLP Challenges

Research Direction Insights

NLP Leaderboards Discussion

Addressing Leaderboard Challenges

Welcoming Leaderboards

Test Set Integrity

Hidden Test Set Challenges

Hidden Test Set Debate

Fair Comparison Dilemma

Leaderboards Impact

Leaderboard Impact

Leaderboard Performance

Episode Highlights

Proposed Solutions

Leaderboard Benefits

Challenges with Leaderboards

Related Episodes

119 - Social NLP, with Diyi Yang

114 - Behavioral Testing of NLP Models, with Marco Tulio Ribeiro

128 - Dynamic Benchmarking, with Douwe Kiela

142 - Science Of Science, with Kyle Lo

131 - Opportunities and Barriers between HCI and NLP, with Nanna Inie and Leon Derczynski

117 - Interpreting NLP Model Predictions, with Sameer Singh

43 - Reinforced Video Captioning with Entailment Rewards, with Ramakanth and Mohit

44 - Truly Low Resource NLP, with Anders Søgaard

113 - Managing Industry Research Teams, with Fernando Pereira

37 - On Statistical Significance, Training Variance, and Why Reporting Score Distributions Matters

67 - GLUE: A Multi-Task Benchmark and Analysis Platform, with Sam Bowman

81 - BlackboxNLP, with Afra Alishahi and Tal Linzen

93 - NLP/ML for clinical data, with Alistair Johnson

102 - Biomedical NLP research at the National Institute of Health with Dina Demner-Fushman

79 - The glass ceiling in NLP, with Natalie Schluter

80 - Leaderboards and Science, with Siva Reddy

Topics covered

Popular Clips

Episode Highlights

Proposed SolutionsSiva Reddy explores methods to mitigate overfitting and improve transparency in NLP leaderboards. He discusses the importance of diverse test sets and automation to enhance research integrity and fairness.

Proposed Solutions

Leaderboard Benefits

Challenges with Leaderboards

Related Episodes