Ep 2: Databricks CTO Matei Zaharia on scaling and orchestrating large language models

Founding Story

The founding story of Databricks is a testament to the power of academic research transforming into a successful enterprise. Matei Zaharia, while pursuing his PhD at UC Berkeley, was inspired by the potential of large-scale data-intensive computing, which was gaining traction with tech giants like Google and Yahoo. He envisioned democratizing this technology for broader use, leading to the creation of Apache Spark, an open-source project that quickly gained traction 1. This vision laid the groundwork for Databricks, which now serves over 7,000 organizations, including 40% of the Fortune 500 2.

It's perfect for machine learning, it's perfect for unsupervised learning, you know, when deep learning became popular and so on.

--- Matei Zaharia

Matei's journey from a research project to a billion-dollar company highlights the impact of innovative thinking and strategic execution.

   

Cloud-Only Strategy

Databricks' decision to adopt a cloud-only strategy was a pivotal move that significantly contributed to its market success. Matei Zaharia explains that this approach allowed the company to focus on new workloads and migrations, rather than displacing established systems 3. The cloud offered a fast feedback loop and minimized software version divergence, enabling Databricks to deliver a superior product experience. This strategic choice positioned Databricks as a leading option for enterprises moving to the cloud, capitalizing on the shift initiated by major cloud providers like Amazon and Microsoft 2.

We thought, okay, fast feedback loop, which is needed to develop a great product, plus this opportunity to go after greenfield applications or migrations.

--- Matei Zaharia

The cloud-only strategy not only differentiated Databricks but also aligned with the evolving technological landscape.