Episode 413: Spencer Kimball on CockroachDB

Open Source

The inception of CockroachDB stemmed from the need for a resilient, scalable, and open-source database that could handle data locality, similar to Google's advanced databases. Spencer Kimball explains that the idea was to create a database that could be used by many projects, offering capabilities like resilience and data locality management 1. The open-source model was crucial in this development, as it allowed for a more accessible and community-driven approach to software acquisition, contrasting with the cumbersome enterprise software procurement processes of the past 2.

The way that you had to get all of your software that you had to use to build whatever new application or service was to go through this unwieldy and terrible process.

--- Spencer Kimball

This shift towards open source has significantly influenced the software development landscape, making tools more accessible to developers worldwide.

   

Postgres

Aligning CockroachDB with Postgres was a strategic decision to leverage the widespread adoption and robust ecosystem of Postgres. Spencer Kimball highlights that creating a unique SQL dialect was unnecessary when Postgres offered a well-established, error-free platform 3. This compatibility not only eases integration with existing tools but also capitalizes on the familiarity many developers have with Postgres, making CockroachDB more accessible and appealing.

We opted for trying to get reasonably compatible with Postgres.

--- Spencer Kimball

The decision reflects a broader industry trend where SQL's resurgence has reaffirmed its importance in database systems, despite the initial popularity of NoSQL models 4.

   

Open Source Challenges

Open source models face challenges in the evolving landscape of infrastructure as a service. Spencer Kimball notes that while open source offers freedom and flexibility, the operational overhead can be significant, prompting a shift towards services like AWS that simplify deployment and management 5. However, even these services can be costly and complex, especially for small-scale projects, highlighting the need for truly serverless databases that eliminate the need for manual provisioning 6.

The average developer out there, especially one that's in a smaller company, but even ones that are in big companies on teams that are doing new projects, they're looking not so much at databases they have to run themselves anymore.

--- Spencer Kimball

This evolution underscores the pressure on open source models to adapt and innovate in response to changing consumption patterns.