Lex Fridman avatar

Dexa/Lex Fridman

Learn more

Infinite Context Models

Aman discusses the potential of infinite context windows in language models and the challenges of making them truly effective. He highlights the importance of retrieval systems in enhancing model performance and explores the intriguing idea of training models specifically to understand particular code bases, such as VS Code. The conversation raises questions about the future of model architecture, particularly whether to integrate retrieval processes or keep them separate for optimal results.