Scaling Research Challenges
Irwan discusses the complexities of translating research ideas into practice, particularly when moving from academic to Google-scale environments. He highlights the significant cost and difficulty in predicting performance at smaller scales, emphasizing the challenges faced when testing new architectures in vastly different settings.In this clip
From this podcast
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
Mixture-of-Experts and Trends in Large-Scale Language Modeling with Irwan Bello - #569
Related Questions
Why is scaling it up challenging?
Is it true that when the cost of trying new technologies goes to zero, deciding what to build becomes the bottleneck, and that knowing which model output is merely plausible and which is actually good is not a commodity skill, making taste in technology a defining advantage?