#032- Simon Kornblith / GoogleAI - SimCLR and Paper Haul!

Loss Function Effects

The impact of loss functions on neural networks is profound, particularly in the penultimate layers. Simon Kornblith explains that while the first two-thirds of a network learn general features, the last third is heavily influenced by the loss function, affecting the penultimate layer's representation 1. This distinction is crucial for tasks like transfer learning, where specialized representations may not transfer well to new tasks 2.

The last third of the network is setting up the penultimate layer representation in a way that is good for your loss function.

--- Simon Kornblith

Understanding these dynamics can guide the development of more versatile neural networks.

   

Transfer Learning

Transfer learning benefits from understanding how loss functions shape neural network representations. Simon Kornblith notes that loss functions yielding high accuracy on tasks like ImageNet often result in representations that transfer poorly to other tasks 3. This is because such functions create highly specialized class separations in the penultimate layer, limiting versatility 4.

If you use loss functions that give you higher accuracy on ImageNet, you tend to learn representations that transfer substantially worse.

--- Simon Kornblith

Exploring these insights can lead to more effective transfer learning strategies.