Inference Compute Challenges
Eiman discusses the significant shift in focus from training to inference compute, highlighting the complexities of running large models like llama 3.1405 b, which requires an extensive GPU setup. He emphasizes the challenges of scaling inference and the often underestimated effort needed to establish a production-ready system. The conversation also introduces Protopia's stained glass transform solution, which aims to address privacy concerns while tackling industry-wide efficiency and security issues.In this clip
From this podcast
Super Data Science: ML & AI Podcast with Jon Krohn
SDS 843: Safe, Fast and Efficient AI — with Protopia’s Dr. Eiman Ebrahimi
Related Questions