Building General Purpose Pretraining Objectives
Alex discusses the challenges of creating baseline algorithms for transformer networks and how they developed pretraining objectives based on contrastive learning models. They found that these objectives helped improve performance, but not consistently across all domains. The goal is to continue developing more general purpose pretraining objectives for better results.In this clip
From this podcast
The Gradient
Alex Tamkin on Self-Supervised Learning and Large Language Models
Related Questions