Site Reliability Engineering - Embracing Risk

Setting SLOs

Setting Service Level Objectives (SLOs) is crucial in Site Reliability Engineering (SRE) practices. Johnny Underwood explains that SLOs help define the acceptable level of service reliability, such as a 99.99% uptime target, which translates to a maximum of 52.56 minutes of downtime annually 1. This approach removes politics by setting clear expectations and using third-party monitoring to measure actual uptime. Michael Outlaw adds that Google focuses on unplanned downtime and uses successful request rates instead of uptime as a metric due to their globally distributed services 2.

Just because the cloud service has the green check mark doesn't mean that your business is operating well.

--- Joe Zack

This highlights the importance of understanding the nuances of service metrics and their impact on business operations.

   

Error Budgeting

Error budgeting is a strategic approach in SRE that balances reliability with the rollout of new features. Michael Outlaw describes how error budgets are formed based on the service's SLO, allowing for a defined amount of unreliability within a quarter 3. This method reduces political tension between SRE and product development teams by providing a clear framework for managing service reliability. Johnny Underwood notes that if the error budget is nearly depleted, development teams focus on testing and hardening their products instead of releasing new features 4.

Hope is not a strategy.

--- Johnny Underwood

This philosophy encourages teams to be proactive in managing reliability and feature development.