Evaluation Challenges
Nathan discusses the complexities of evaluating models, particularly the impact of custom prompts and setups on performance scores. Riley agrees, noting that while following vendor recommendations is essential, the differences between optimized and default prompts can significantly affect outcomes. They also explore the evolving landscape of model training, suggesting that familiarity with various chat templates may reduce the importance of strict prompting guidelines.In this clip
From this podcast
Interconnects Audio
Interviewing Riley Goodside on the science of prompting
Related Questions