Episode 199: Michael Stonebraker on Current Developments in Databases

Hadoop Stack

Michael Stonebraker provides a detailed explanation of the Hadoop stack, emphasizing its three-tier architecture. At its core is HDFS, a file system, followed by an open-source implementation of MapReduce, and topped with SQL-like tools such as Hive and Pig. Stonebraker notes that over 95% of Facebook's workload on Hadoop is processed through Hive, highlighting the stack's reliance on SQL for data processing 1.

Hadoop is really a three level stack. At the bottom level is hdfs, which is a file system. On top of that is an open source implementation of Mapreduce, which is what Hadoop has historically been called.

--- Michael Stonebraker

This segmentation of the database market underscores the importance of understanding Hadoop's components and their roles in managing data processing tasks 2.

   

Performance Issues

Stonebraker critiques Hadoop's performance, particularly when using Hive on top of the MapReduce layer. He argues that this configuration results in significantly slower performance compared to parallel database systems, often by a factor of 50 to 100 3.

If you run Hive on top of the Mapreduce layer, on top of htfs, and you're specifying queries in Hive, or I, let's just call it SQL, then that stack runs wildly slower than any parallel database system.

--- Michael Stonebraker

He advises against using the traditional Hadoop stack for most applications, suggesting alternatives like Impala for better performance 4.