SE Radio 632: Goran Petrovic on Mutation Testing at Google

Developer Resistance

Developer resistance to mutation testing at Google initially stemmed from the overwhelming number of mutants generated, which could be daunting for engineers. Goran Petrovic recalls his early experiences with the tool, where the sheer volume of mutants made it impractical to review them all, leading to fatigue and disinterest 1. He emphasizes the importance of presenting information carefully to avoid overwhelming developers, as they might disable the tool if it becomes too cumbersome 2.

If they don't like your tool in two days, they will just disable it for them and then you lost a user.

--- Goran Petrovic

To address these concerns, Google implemented strategies to filter and reduce the number of mutants, ensuring that developers could efficiently manage their workload.

   

Scalability Solutions

Scalability was a significant challenge for mutation testing at Google, given the vast number of tests and code changes. Goran Petrovic explains that aggressive suppression heuristics were employed to filter out unnecessary mutants, focusing only on lines of code that had changed in a pull request 2. This approach reduced the number of mutants by over 90%, making the process feasible even with Google's extensive infrastructure 3.

If you filter out most of the mutants, we call it aggressive suppression heuristics.

--- Goran Petrovic

By leveraging caching and efficient compilation strategies, Google managed to maintain scalability without compromising the effectiveness of mutation testing.

   

Tool Optimization

Optimizing mutation testing tools involved identifying and eliminating unproductive mutants that did not contribute to meaningful test improvements. Goran Petrovic describes the process of encoding heuristics to filter out such mutants, which initially comprised a large portion of the output 4. This refinement increased the percentage of useful mutants to around 90%, significantly enhancing the tool's utility for developers 5.

We have to prune this somehow. And the first step was just to figure out what these unproductive mutants were.

--- Goran Petrovic

With these optimizations, mutation testing became a more valuable resource, encouraging developers to write better tests and improve code quality.