Scaling LLMs and Accelerating Adoption: Interview with Aidan Gomez

Transformer Basics

Aidan Gomez, CEO and co-founder of Cohere, reflects on his role in developing the transformer model. He emphasizes the simplicity and scalability of transformers compared to previous models like LSTMs and RNNs. The architecture, which stacks attention blocks on top of feed-forward layers, has proven to be both efficient and scalable, allowing for significant advancements in machine learning 1.

Transformers are really just like an attention block stacked on top of a multilayer perception or feed forward layer, and then a bunch of those stacked on top of each other.

--- Aidan Gomez

Gomez also highlights the inspiration behind the attention mechanism, noting its fundamental role in human intelligence and its intuitive appeal in computational models 2.

   

SSM Potential

State-space models (SSMs) are emerging as promising alternatives to transformers. Gomez explains that SSMs aim to strike a balance between the fully autoregressive nature of transformers and the state-dependent nature of LSTMs and RNNs. They offer scalability and the potential for an infinite context window, which could overcome some limitations of transformers 3.

SSMs are trying to find this middle ground where you have some window within which you can do lookup, but for everything that's outside of that, you can rely on an internal memory that you can read and write from.

--- Aidan Gomez

Gomez believes that the sequence length constraint of transformers might push the field towards adopting SSMs, especially as multimodal applications become more prevalent 4.

   

Compute Efficiency

Saturating compute is crucial for the success of machine learning architectures. Gomez explains that an architecture must efficiently utilize computational resources, primarily through large matrix multiplications, to scale effectively. Transformers excel in this aspect, but Gomez suggests that other architectures could also achieve similar performance if they can saturate compute 5.

You really want your entire architecture to just look like big map moles, because you can.

--- Aidan Gomez

The interplay between software and hardware optimizations has locked the field into using transformers, but Gomez believes there are many potential architectures that could perform equally well 5.

   

Model Adaptability

Large models offer significant flexibility and adaptability for various use cases. Gomez discusses Cohere's approach of providing both general-purpose models and specialized endpoints for tasks like summarization and classification. This dual strategy allows users to leverage the broad capabilities of large models while also benefiting from targeted solutions 6.

The general purpose nature of the technology is a huge piece of the value prop that you can just go to one model and you can get it to do tons of different tasks for you at the same time.

--- Aidan Gomez

Fine-tuning remains a valuable tool for optimizing model performance, but Gomez notes that making models adaptable through prompting and other methods can often achieve the desired results without the need for extensive customization 7.