Super Data Science: ML & AI Podcast with Jon Krohn avatar

Dexa/Super Data Science: ML & AI Podcast with Jon Krohn

Learn more
Published May 5, 2023

676: The Chinchilla Scaling Laws — with Jon Krohn (@JonKrohnLearns)

Jon Krohn delves into the transformative Chinchilla Scaling Laws from Google DeepMind, highlighting how smaller, data-rich models can surpass larger ones, and introduces Cerebras GPT as a game-changing, open-source language model. The episode focuses on the efficiency and adaptability of small language models, particularly for edge devices and specific domains, demonstrating the potential of these scaling laws in revolutionizing AI training.
Episode Highlights
Super Data Science: ML & AI Podcast with Jon Krohn logo

Topics covered

Popular Clips

Episode Highlights

  • Launch

    Cerebras GPT has emerged as a significant player in the landscape of open-source language models. highlights the release of Cerebras GPT, which includes seven models ranging from 111 million to 13 billion parameters, all available on Hugging Face. These models are designed to be fine-tuned for proprietary tasks, offering a commercial use-friendly Apache 2.0 license.

    The key thing here that's new about the Cerebras GPT release is that the models that they released follow these chinchilla scaling laws.

    ---

    This flexibility allows developers to tailor the models for specific natural language generation tasks, potentially reaching capabilities akin to ChatGPT 1.

       

    Comparison

    Cerebras GPT stands out by adhering to the Chinchilla Scaling Laws, setting it apart from other models like Lama and Alpaca. explains that these models are comparable in size, with the largest Cerebras model fitting on a single large GPU, similar to Lama and Alpaca. This makes them ideal for fine-tuning to achieve performance levels close to ChatGPT.

    So that's really key, especially when you compare with other well known, relatively small LLMs today, single gpu LLMs like llama and Alpaca.

    ---

    The open-source nature and commercial-friendly licensing of Cerebras GPT provide a competitive edge in the rapidly evolving AI landscape 1.

Related Episodes