Transformer Design Insights
Nathan, Michael, and Tri discuss the unique design choices in transformers, highlighting the importance of attention and MLP interleaving. They delve into the challenge of quadratic scaling in attention cost and explore how transformers overcome the limitations of RNNs in processing text efficiently.In this clip
From this podcast
Interconnects Audio
Interviewing Tri Dao and Michael Poli of Together AI on the future of LLM architectures
Related Questions