Gradient Dissent - A Machine Learning Podcast avatar

Dexa/Gradient Dissent - A Machine Learning Podcast

Learn more
Published May 9, 2024

Shaping AI Benchmarks with Together AI Co-Founder Percy Liang

AI expert Percy Liang delves into the intricacies of benchmarking language models through the HELM framework, explores the synergy between AI and music, and underscores the significance of transparency and community in open-source AI development.
Episode Highlights
Gradient Dissent - A Machine Learning Podcast logo

Topics covered

Popular Clips

Episode Highlights

  • Transparency

    Percy Liang emphasizes the need for transparency in both open and closed AI models. He argues that while open models offer more transparency, they still fall short due to the lack of available training data, which complicates the interpretation of evaluation results 1. Percy also discusses the centralization of power in AI development and the need for a more participatory process in model training and evaluation 2.

    There's been a lot of movement in the open model ecosystem. There's a lot of interest, which is great. I would say that the TED talk actually is broader than just having open models.

    --- Percy Liang

    He highlights the importance of decentralizing power and ensuring that different values are considered in AI alignment.

       

    Community

    Community involvement is crucial for the development of open-source AI models. Percy Liang praises projects like Eleuther and the Big Science Project for their community-driven approach to AI development 2. He also introduces Together AI, a platform he co-founded to support open innovation and make AI development more accessible 3.

    The goal of Together is to build a platform for developers who want to do AI of any sort.

    --- Percy Liang

    Together AI aims to replicate the success of open-source software in the AI domain by providing tools for model fine-tuning, serving, and benchmarking.

       

    Challenges

    Open-source AI development faces several challenges, including legal issues, resource constraints, and the centralization of power. Percy Liang points out that while open models allow for customization and plurality, subtle biases can still be embedded in the original models 2. He also stresses the need for better attribution systems to credit creators whose data is used in training these models 1.

    If we could set up some better attribution or economic system where there's actually credit that goes all the way back to the creators, then maybe we can mitigate some of these concerns.

    --- Percy Liang

    Addressing these challenges is essential for creating a more equitable and transparent AI ecosystem.

Related Episodes