Pre-Training Tasks
The discussion delves into five pre-training tasks categorized into language, vision, and cross-modality. Hao explains the importance of distinguishing between feature regression and label prediction, emphasizing how each task captures different levels of information. The conversation also touches on the model's learning dynamics, comparing feature regression to autoencoding rather than model distillation, highlighting the nuanced approach to understanding object features.In this clip
From this podcast
NLP Highlights
107 - Multi-Modal Transformers, with Hao Tan and Mohit Bansal
Related Questions