Nathan and Tri discuss the future of AI architectures, debating the staying power of transformers and the potential for new ideas like state space and linear attention to enhance performance. They question if attention will remain dominant and ponder the impact of larger players entering the space in 2024.
