12 - Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

Supervised vs. Unsupervised

The discussion highlights the ongoing debate between supervised and unsupervised methods for feature representation in natural language processing. Waleed Ammar points out that while their SNLI-trained model performs slightly better than skip-thought vectors, it still falls short of surpassing supervised baselines 1. This raises questions about the practical utility of unsupervised methods when labeled data is available. Matt Gardner suggests exploring hybrid approaches, such as combining unsupervised and supervised representations, to enhance model performance 1.

It's an interesting open question how you can take representations learned in some unsupervised fashion and modify them in, or incorporate them into some downstream model.

--- Waleed Ammar

The conversation underscores the complexity of choosing the right approach for feature representation in NLP tasks.

   

Feature Limitations

Feature representation models face significant limitations, particularly when encoding sentences into single vectors. Waleed Ammar argues that this approach is too restrictive, especially when considering sequence-to-sequence models with attention mechanisms 1. Matt Gardner and Ammar discuss the potential of using attention over words to enhance sentence representations, questioning whether single vectors are the best method for downstream tasks 2.

Most models that you see nowadays have this kind of attention because single vectors are too limiting.

--- Waleed Ammar

The dialogue reflects the ongoing exploration of more flexible and effective feature representation techniques in NLP.