Ep 2: Databricks CTO Matei Zaharia on scaling and orchestrating large language models

Scaling Challenges

Scaling large language models (LLMs) presents significant challenges, particularly in balancing cost and performance. Matei Zaharia highlights the importance of both modeling and systems expertise in scaling LLMs, noting that while systems engineers must think critically, scaling can often be achieved by simply adding more machines and data 1. He also points out that there is potential to reduce costs through advancements in model architecture and training processes, such as using larger or cleaner datasets and model compression techniques 2.

The systems, people building the system have to think. But if you built some model architecture, some training process like SGD or whatever, that seems to do well, and you can throw in ten x more machines at it or like ten x more data and it does even better. That's fantastic.

--- Matei Zaharia

These strategies could make today's models more practical and cost-effective for widespread use.

   

Applications

The integration of LLMs into platforms like Databricks aims to enhance accessibility and functionality. Matei Zaharia discusses potential applications, such as enabling natural language queries in place of traditional coding languages like Python or SQL, and improving search capabilities over datasets and documentation 3. He also mentions the development of complex applications using language models, emphasizing the importance of programmatic control to ensure accuracy and reliability 4.

We want to use language model for things it's uniquely good at, like approximate matching or things like that. But as much as possible, we want to retain, like, programmatic control of what the system does.

--- Matei Zaharia

These integrations aim to make data tools more user-friendly and efficient, expanding their use to non-experts.

   

Future Architectures

Future architectures for LLMs may involve combining them with traditional systems to improve information retrieval. Matei Zaharia suggests that knowledge-intensive applications could benefit from a hybrid approach, where traditional search indexes store knowledge and LLMs generate responses 5. He also explores the idea of decoupling semantic understanding from factual knowledge within LLMs, which could enhance their adaptability and accuracy over time 6.

We want something that's 99% reliable. Like, you know, what's the failure probability of each stage that we want to get it to work?

--- Matei Zaharia

This approach could lead to more robust and versatile AI systems.