SE-Radio Episode 276: Björn Rabenstein on Site Reliability Engineering

Reliability Hierarchy

Mikey Dickerson's hierarchy of reliability, akin to Maslow's hierarchy of needs, is foundational in setting up reliable systems. Björn Rabenstein explains that monitoring forms the base of this hierarchy, essential for understanding system operations and enabling subsequent layers like incident response and postmortem analysis 1. Each layer builds upon the previous, ensuring a robust and reliable system. Rabenstein emphasizes the importance of this structure, noting, "Without monitoring, nothing else works" 2. Capacity planning, a higher-level function, becomes feasible only after establishing these foundational layers 3.

   

Monitoring

Monitoring is crucial for system reliability, with tools like Prometheus playing a key role. Björn Rabenstein discusses the importance of symptom-based alerting, which focuses on user experience rather than just system failures 4. Google's four golden signals—latency, traffic, errors, and saturation—are critical metrics for monitoring system health 4. He highlights, "If your monitoring system is only able to tell you that the machine doesn't ping anymore, you will not be able to set something up like that" 4. Understanding these signals helps in effective capacity planning and redundancy management 5.

   

On-Call Strategies

Effective on-call strategies are vital for managing workload and ensuring system reliability. Björn Rabenstein describes the Google approach, where SRE teams handle first-level alerts, escalating to developers only when necessary 6. At Soundcloud, a different strategy was adopted, integrating SRE practices across the engineering team, making everyone a "little SRE" 7. He notes, "SREs are not pager monkeys," emphasizing the balance between operational duties and development work 8. This approach fosters a culture where developers are also responsible for the systems they build 7.