How EleutherAI Trains and Releases LLMs: Interview with Stella Biderman

Origins

EleutherAI emerged as a pioneering force in open-source AI research during the pandemic, starting as a Discord server for AI enthusiasts and researchers. Stella Biderman explains that it quickly became a community hub for discussing AI technologies like GPT-3, which were not widely accessible at the time 1. The organization is unique as a nonprofit focused on training and releasing large language models, a space typically dominated by major tech companies 2. Over time, EleutherAI has evolved from a volunteer-driven initiative to a structured nonprofit, thanks to significant support from donors like Stability AI and Hugging Face 3.

   

Challenges

Training large language models posed significant challenges for EleutherAI, especially in its early days. Stella highlights the initial skepticism about training billion-parameter models outside major tech firms, which they overcame through determination and resourcefulness 4. The organization faced financial and computational hurdles, relying on donated compute power and creating their own datasets, like "The Pile," to train models effectively 5. Despite these obstacles, EleutherAI managed to train a 20 billion parameter model, demonstrating the potential of open-source AI research 6.