Detecting Overfitting
Timothy reveals a novel way to detect overfitting in large language models using n-gram statistics without relying on a holdout set. He explains how transformers can become overly reliant on long contexts, leading to memorization rather than robust predictions. By evaluating performance on shorter contexts, a U-shaped curve indicative of overfitting emerges, providing a new perspective on training dynamics in neural networks.In this clip
From this podcast
Machine Learning Street Talk (MLST)
Is ChatGPT an N-gram model on steroids?
Related Questions