Neel Nanda - Mechanistic Interpretability (Sparse Autoencoders)

Feature Learning

Sparse Autoencoders (SAEs) play a crucial role in feature learning, with their performance improving as they are trained on more data. Neel Nanda highlights that the frequency of a feature significantly impacts its learning, with more common features being more likely to be captured by the model 1. However, the process can be probabilistic, especially for niche features, leading to variability in feature learning across different model widths 1.

SAEs get better when trained on more data. As a general rule, sometimes they'll kind of saturate.

--- Neel Nanda

This variability suggests that practitioners should experiment with different SAE widths to capture the desired features effectively 2.

   

Interpretability

Understanding the interpretability of features in neural networks is a complex task. Neel Nanda notes that larger Sparse Autoencoders (SAEs) often face training stability issues, with many features remaining inactive or "dead" 3. This suggests that while larger models may hold more concepts, they also require careful management to ensure meaningful feature activation.

I want to know what's going on in the model. What is the computation happening, what are the variables?

--- Neel Nanda

The goal is to understand the computation within the model, which involves identifying the variables and their interactions, rather than merely increasing the number of features 4.