Episode 433: Jay Kreps on ksqlDB

Scalability

KSQLDB's dynamic scalability is a standout feature, allowing it to adapt to changing workloads by adding nodes. Jay Kreps explains that the system can process hundreds of thousands of records per second on a single node, but its true power lies in its ability to scale horizontally. "You could add if you want more, then you just add more machines and you can do that dynamically as it runs," he notes, highlighting its capacity to handle millions of records per second 1. This flexibility is achieved through a persistent key-value interface, which is pluggable within Kafka streams, allowing users to choose from various libraries like RocksDB 2.

   

Performance

Performance benchmarks for KSQLDB reveal its capability to handle large volumes of streaming data efficiently. Eberhard Wolff inquires about its performance compared to relational databases, to which Jay responds by emphasizing KSQLDB's design for stream processing 3. The choice of RocksDB as the underlying storage engine is pivotal, offering high performance and flexibility despite its complexity. "The con is it has about a million tuning knobs," Jay admits, but assures that it can be optimized for performance 4.