Super Data Science: ML & AI Podcast with Jon Krohn avatar

Dexa/Super Data Science: ML & AI Podcast with Jon Krohn

Learn more
Published Mar 19, 2024

767: Open-Source LLM Libraries and Techniques — with Dr. Sebastian Raschka

Dr. Sebastian Raschka delves into the future of large language models (LLMs), sharing insights on open-source projects like PyTorch Lightning and litGPT that streamline LLM implementation. He introduces cutting-edge training techniques and architectures poised to revolutionize the field, including efficient methods like Lora, Dora, and multi-query attention.
Episode Highlights
Super Data Science: ML & AI Podcast with Jon Krohn logo

Topics covered

Popular Clips

Episode Highlights

  • Efficiency

    and explore the pressing challenges in making large language models (LLMs) more efficient and practical. Sebastian highlights the potential of specialized architectures, like the mixture of experts, to reduce inference costs without compromising performance 1. He suggests that while transformers are currently dominant, alternatives like the Mamba architecture could offer more efficient solutions for specific applications 2.

    We don't really know how to make them better. We had lstms, we had grus, but what's the next thing?

    ---

    Sebastian emphasizes the need for ongoing research to explore these alternatives and optimize LLMs for diverse tasks 1.

       

    Architectures

    The conversation shifts to new model architectures that could potentially replace transformers in LLMs. Sebastian discusses the Mamba and Hyena models, highlighting their unique state space structures and efficiency in handling specific tasks 3. He notes that while these models show promise, their scalability remains an open question 3.

    Mamba is, I think, the hot topic. So hyena was last year, it was like a structured state space model, where mamba is, I think a selective state space model.

    ---

    Sebastian expresses optimism about the potential of these architectures to offer viable alternatives to transformers, particularly for specialized applications 4.

       

    Innovations

    Sebastian delves into recent innovations in LLM training methods, emphasizing the rapid pace of development in this field. He shares his excitement about techniques like LoRA, which enable efficient fine-tuning of large models, making them more accessible and cost-effective 5. Sebastian also highlights the importance of studying various training choices, such as learning rate schedules, to optimize model performance 6.

    It's really almost its own research field. From fine tuning to pre training.

    ---

    These innovations are crucial for unlocking the full potential of LLMs and ensuring they remain at the forefront of AI research and application 6.

Related Episodes