#032- Simon Kornblith / GoogleAI - SimCLR and Paper Haul!

Similarity

Simon Kornblith introduces a novel similarity measure for neural networks, addressing the challenge of comparing representations from networks trained with different initializations. He explains that traditional methods struggle due to misalignment of neurons across networks. Instead, Simon's approach involves creating matrices that measure the similarity of examples within a network, bypassing the alignment issue.

We get centered kernel alignment, which gives us a similarity score between zero and one.

--- Simon Kornblith

This method, known as centered kernel alignment, normalizes the dot product to ensure scaling invariance, providing a robust similarity index 1.

   

Evolution

Simon discusses the evolution of neural network representations, highlighting the advantages of machine learning over neuroscience in manipulating and analyzing systems. In machine learning, researchers can alter network architectures and observe changes in representations, a flexibility not possible in biological systems.

In a neural network, we can record all the neurons, which is extremely challenging in a biological organism.

--- Simon Kornblith

This ability allows for deeper insights into how representations evolve across layers and architectures 2.

   

Pathology

The concept of 'blockiness' in neural networks is explored, indicating a halt in representational evolution and potential over-parameterization. Simon Kornblith's centered kernel alignment reveals that when layers exhibit self-similarity, they may not be learning new information.

The presence of this block structure is an indicator of the halting of evolution and a strong indicator of over-parameterization.

--- Tim Scarfe

This phenomenon suggests that deeper or wider networks may not necessarily improve accuracy, as they might have already fit the data to their capacity 3.