Jonathan and others discuss the importance of over parameterization in network training, the concept of lottery tickets, and the significance of resetting learning rates in the rewinding algorithm. They explore the trade-offs in network exploration and exploitation, emphasizing the need for better methodologies to uncover lottery tickets early in the training process.
