Episode 469: Dhruba Borthakur on Embedding Real-time Analytics in Applications

Batch vs. Real-Time

The transition from batch to real-time analytics marks a significant evolution in data processing. Dhruba Borthakur explains that traditional batch systems like Hadoop were designed for storing and querying massive datasets, focusing on cost-effective storage rather than speed 1. Real-time systems, however, prioritize low latency and the ability to process fresh data quickly, which is crucial for timely decision-making 2.

The focus was not on leveraging your most recent data set. The focus was not on low latency queries.

--- Dhruba Borthakur

This shift requires new architectures that can handle the velocity, volume, and variety of modern big data 2.

   

ALT Architecture

The Aggregator Leaf Tailor (ALT) architecture offers a novel approach to real-time analytics. Dhruba Borthakur highlights its efficiency in handling low-latency queries and large datasets by separating the components for data writing, storage, and querying 3. This separation contrasts with systems like Apache Druid and Kudu, where shared resources can lead to bottlenecks during high data influx or query loads 4.

The beauty of the ALT architecture is that it is focused on serving low latency queries on large datasets.

--- Dhruba Borthakur

By disaggregating these processes, ALT ensures that neither data ingestion nor query performance is compromised, even under varying workloads 4.

   

Indexing Innovations

Innovations in indexing, such as converged indexing, are pivotal for real-time data processing. Dhruba Borthakur describes converged indexing as a system that automatically builds multiple types of indexes, like columnar, inverted, and row indexes, to optimize query performance 5. This approach reduces the cost and complexity of indexing, enabling fast queries across all data fields without manual intervention 6.

The system automatically builds these indexes and then based on the query, it tries to use the right index for the right purpose.

--- Dhruba Borthakur

Such innovations make it economically feasible to handle large streaming databases efficiently, enhancing both speed and scalability 6.