Challenges in Network Adaptability

Sara discusses the challenges of proposing new ideas in network architecture and the limitations imposed by current hardware designs. She highlights the need for adaptability in network training and the difficulties in growing dynamic networks due to hardware constraints. The conversation also touches on the potential of structured sparsity as a solution to these limitations.