Distributional Generalization Insights
Preetum highlights the difference between over parameterized and under parameterized models in terms of reproducing noise at test time. The unified way of thinking is that models in the over parameterized world fit the noise in the train set and reproduce it at test time, while under parameterized models don't fit the noise and thus don't reproduce it. This insight sheds light on why techniques like regularization and label smoothing can help with noisy data.In this clip
From this podcast
The Gradient
Preetum Nakkiran: An Empirical Theory of Deep Learning
Related Questions