847: AI Engineering 101 — with Ed Donner

Open vs Closed

The choice between open and closed source AI models involves several considerations. Ed Donner highlights that open source models are beneficial when dealing with proprietary or sensitive data, as they allow for local data processing without third-party involvement 1. He also notes that open source models can be cost-effective for high-volume inference tasks, although closed source models might be cheaper for smaller scale operations 1. Jon Krohn adds that navigating open source options can be overwhelming due to the plethora of available models, but resources like Hugging Face's leaderboard can aid in making informed decisions 2.

There are a lot of known limitations. They can be gamed. There's plenty of examples of contamination that's happened that people are sort of overfit to benchmarks but still they give you a decent indication of what you're working with.

--- Ed Donner

Understanding these trade-offs is crucial for AI engineers when selecting the right model for their needs.

   

Prototyping

Prototyping with expensive models is a strategic approach in AI engineering. Jon Krohn suggests starting with robust models like GPT-4 for initial prototyping, as they provide a strong foundation for testing and iteration 3. Ed Donner emphasizes the importance of building a baseline model before selecting an LLM, which helps in understanding the problem space and setting performance benchmarks 3. This empirical approach allows for informed decision-making and refinement of models based on real-world performance metrics 4.

It's good to start logistic regression.

--- Jon Krohn

Such a methodical process ensures that the chosen model aligns with business objectives and technical requirements.

   

Empirical Selection

Empirical model selection is a cornerstone of effective AI engineering. Ed Donner describes this process as involving trial and error, where benchmarks guide the selection of models from various providers like Microsoft, Google, and Meta 5. He stresses the importance of creating comprehensive test sets, possibly using synthetic data, to evaluate model performance across anticipated use cases 6. This approach ensures that models are not only technically sound but also robust against adversarial challenges.

Trial and error, try a few things out, see what results you get.

--- Ed Donner

By leveraging empirical methods, AI engineers can optimize model selection to meet specific business needs and user expectations.