Gary Marcus' keynote at AGI-24

Scaling Issues

Gary Marcus argues that the scaling of AI models has hit a plateau. He explains that while initial scaling efforts yielded significant improvements, recent data shows diminishing returns. Marcus highlights the limitations of current approaches, including the lack of sufficient data and architectural issues 1. He humorously critiques the persistent belief in the scaling hypothesis, comparing it to drawing an owl by simply adding more data and compute 1.

In my entire career, I've never seen an experiment fail so many times at such great cost as the experiment on the scaling hypothesis that you could create AGI simply by adding more data and compute. It doesn't work. Can we try something different already?

--- Gary Marcus

Marcus also revisits his earlier discussions on foundation models, emphasizing that many of the issues he raised years ago remain unresolved 2.

   

Economic Risks

The economic landscape of AI development is fraught with challenges. Gary Marcus and Yannic Kilcher discuss the valuation bubble surrounding large language models (LLMs) and generative AI companies 3. Marcus points out that while there are practical applications for LLMs, the economic expectations are often inflated, leading to potential financial risks 4.

There is a bubble around generative AI. It is going to have an impact and there are some practical real-world applications. We're both trying to guess the balance of that and neither of us really know.

--- Gary Marcus

The conversation underscores the uncertainty in predicting the economic viability of AI technologies and the potential for an AI winter due to overhyped promises and investor burnout 4.