Shubho discusses the challenges of scaling complex attentional models, particularly the memory limitations encountered when increasing layer sizes. He emphasizes the need for innovative solutions, such as graph partitioning across multiple GPUs and the intricate process of rewriting programs to optimize performance. The conversation highlights the artisan-like approach required to tackle these issues, reminiscent of the early days of programming before the advent of compilers.
