Practical AI avatar

Dexa/Practical AI

Learn more
Published Feb 14, 2023

Serverless GPUs

Eric Dundeman delves into the transformative potential of serverless GPUs in AI, addressing challenges like cold boot times and infrastructure setup, while highlighting how these technologies enable rapid innovation and personalized AI models for users.
Episode Highlights
Practical AI logo

Topics covered

Popular Clips

Episode Highlights

  • Workload Suitability

    Serverless platforms offer unique advantages and challenges for AI workloads. explains that while serverless GPUs are ideal for dynamic, unpredictable traffic, they may not suit steady, continuous workloads due to cold boot issues and similar costs to traditional setups 1. He notes that many users prefer custom workloads, with 80% deploying from custom repositories, indicating a need for unique APIs and application logic 2.

    Ideally we make serverless so good that you don't need to think about that.

    ---

    Serverless platforms are evolving to minimize these limitations, aiming to simplify the deployment process and enhance flexibility.

       

    Deployment Processes

    Deploying AI models in a serverless framework requires specific skills and tools. outlines that users typically need Python expertise, familiarity with frameworks like Hugging Face, and some Docker knowledge to effectively deploy models on Banana 3. The deployment process involves using one-click templates to quickly set up models, followed by customization and iteration within the Banana framework 4.

    We have a CI pipeline built in. So when you select that repo, we build the model, we deploy it.

    ---

    This streamlined process allows users to focus on fine-tuning and optimizing their models for specific use cases.

       

    Workflow Insights

    Serverless workflows offer flexibility but require careful consideration of workload patterns. shares insights on managing cold starts, explaining that Banana allows users to configure idle times and auto-scaling to balance cost and performance 5. Many users prioritize cost over latency, opting for machines to shut down and incur cold starts rather than maintaining constant readiness.

    The faster your cold starts are, the more willing users are to take those cold starts because it's less impactful on their inferences.

    ---

    This approach helps users optimize their serverless workflows according to their specific needs and constraints.

Related Episodes