Episode 436: Apache Samza with Yi Pan

Modular Design

Yi Pan, the lead maintainer of Apache Samza, explains the modular architecture of Samza, emphasizing its open nature for integration with various systems. This design choice ensures maximum plugability and modularity, allowing Samza to operate with different execution environments like Yarn and Kubernetes 1. Yi highlights that Samza was built to address the need for real-time data processing, contrasting with traditional batch processing systems like Hadoop, which are optimized for throughput rather than latency 2.

Samza is a stream processing platform built at LinkedIn to process events as they come in, generating results to serve online requests in minutes or seconds.

--- Yi Pan

This capability is crucial for applications requiring dynamic user interactions and quick analytical results.

   

Performance

Samza's performance capabilities are highlighted by its advanced auto-scaling features, which address the challenges of fluctuating traffic and differing hardware environments. Yi Pan notes that these features significantly reduce the operational overhead for users, especially those unfamiliar with the physical system's operational details 3. At LinkedIn, Samza processes millions of events per second, demonstrating its ability to handle large-scale data streams efficiently 4.

We deployed the auto-scaling system with our large-scale operational pipeline at LinkedIn, achieving great performance and operational cost savings.

--- Yi Pan

These advancements make Samza a preferred choice for handling mission-critical jobs with minimal downtime.

   

Execution Layers

Samza's interaction with execution layers like Yarn and Kubernetes is a key aspect of its architecture. Yi Pan explains that while Samza was initially developed alongside Kafka, it remains a pluggable service that can integrate with various systems, including Event Hub and Kinesis 5. This flexibility extends to its execution layer, where Samza can operate independently of Yarn, using alternatives like Zookeeper for coordination 6.

Samza leverages Yarn for resource scheduling but can also use Zookeeper-based coordination for standalone implementations.

--- Yi Pan

This adaptability ensures that Samza can be tailored to fit different organizational needs and technological environments.