Hidden Test Set Debate

Matt and Siva discuss the necessity of a hidden test set in NLP research, highlighting the balance between trust in researchers and the need for unbiased evaluation. They delve into the challenges of statistical significance testing without access to test data, proposing alternative methods for comparison on dev sets.