SE-Radio Episode 284: John Allspaw on System Failures: Preventing, Responding, and Learning From

Production Testing

Testing in production is a strategy that involves running tests on live systems to ensure realistic reliability evaluation. John Allspaw explains that this approach, similar to Netflix's Chaos Monkey, involves purposefully injecting faults into production environments to test system resilience 1. This method encourages engineers to anticipate and prepare for real-world failures, fostering a culture of proactive problem-solving. Allspaw highlights the importance of making it safe for teams to experiment and learn from these tests, as it builds anticipation skills and confidence in handling unexpected issues 2.

If you're not comfortable enough to do it in production, it just means you're not done yet.

--- John Allspaw

This mindset shift allows organizations to better prepare for and manage potential system failures.

   

Infrastructure Challenges

Infrastructure testing presents unique challenges due to the complexity of interactions between different system components. Robert Blumen notes that while there are many resources for testing code, the domain of infrastructure testing is less advanced 3. Allspaw emphasizes the importance of gathering diverse perspectives to understand potential failure points, as unexpected interactions can lead to system issues 4. He also mentions the value of writing tests, even if they cannot capture every possible scenario, as they provide learning opportunities and insights into system behavior.

There are always things that we don't even know to test for.

--- John Allspaw

This underscores the need for continuous learning and adaptation in testing practices.