Jonas discusses a paradigm where memory is dynamic and tied to underlying computations, emphasizing the importance of model expectations in determining information gain. He highlights the limitations of current retrieval augmented generation systems, particularly their reliance on nearest neighbor searches, which can lead to redundant information. The debate on whether to use context or gradient steps for data ingestion remains an open question in the field.
