Exploring the balance between model size and performance, Sebastian highlights the potential of a 2 billion parameter model as a new research standard. He compares it to Microsoft's Phi model, noting that smaller models can facilitate quicker iterations without sacrificing capability. The discussion also touches on architectural similarities and innovations, such as the use of multi-query attention and a unique activation function called geglue, suggesting exciting possibilities for future experiments.
