797: Deep Learning Classics and Trends — with Dr. Rosanne Liu

Intrinsic Dimensions

Dr. Rosanne Liu discusses the concept of intrinsic dimensions in deep learning, a foundational idea that has influenced modern techniques like LoRA. She explains how intrinsic dimensions help measure the difficulty of a task when combined with a network, offering a scientific approach to understanding task complexity 1. This concept has been pivotal in developing parameter-efficient fine-tuning methods, allowing large language models to be trained more effectively without overfitting 2. Liu reflects on the impact of her work, noting that it has inspired significant advancements in the field 3.

The dimension of training, usually people think of it as the number of parameters, because however many parameters you have, you can move alongside each parameter.

--- Rosanne Liu

Her insights into intrinsic dimensions continue to shape the landscape of machine learning research.

   

Training Dynamics

In exploring training dynamics, Rosanne Liu emphasizes the complexity of understanding neural networks' internal representations. She argues that as models become more powerful, they become harder to interpret, akin to understanding complex human emotions compared to simpler animal behaviors 4. This complexity necessitates a balance between curiosity-driven and goal-driven research, allowing for both innovative exploration and structured advancements 5. Liu also highlights the importance of fostering research growth through practice and community support, particularly for those without access to traditional labs 6.

Understanding is like, you push forward, you can always understand more, because there's nothing in this world we can confidently say that we understand 100%.

--- Rosanne Liu

Her work underscores the ongoing challenge of interpreting and improving AI models.

   

LLM Evaluations

Evaluating large language models (LLMs) presents unique challenges, as Rosanne Liu explains through her involvement with the Big Bench project. This initiative aims to crowdsource benchmark tests, allowing the community to contribute tasks that assess model capabilities comprehensively 7. However, Liu notes the difficulties in maintaining neutrality and preventing models from overfitting to specific evaluations 8. The dynamic nature of evaluation metrics requires constant innovation to ensure they remain relevant and unbiased 9.

It's possible to organize over 100 institutions and 400 authors.

--- Rosanne Liu

Her insights highlight the importance of transparent and adaptive evaluation processes in AI research.