Zachary Lipton discusses the limitations of current research on initialization in machine learning models. He questions the focus on mutual information and emphasizes the need to understand the right attributes of an initialization that contribute to the efficacy of a downstream model. Lipton warns against drawing definitive conclusions from probing work and highlights the importance of exploring curious properties of representations.
