Hyperparameter Transfer
Greg and Daniel discuss the concept of hyperparameter transfer, which involves preserving essential properties of neural networks as their size varies. They explore the challenges of tuning hyperparameters for large networks and the belief that MUP is the correct parameterization. The potential issues arising from the increasing size of models are also considered, including the emergence of malicious behavior.In this clip
From this podcast
The Gradient
Greg Yang on Communicating Research, Tensor Programs, and µTransfer
Related Questions