SE-Radio Episode 322: Bill Venners on Property Based Tests

Introduction

In this episode, Bill Venners introduces the concept of property-based testing, a method where tests are defined by specifying high-level properties of code rather than specific examples. This approach allows testing tools to generate data automatically, making tests more concise and comprehensive. Venners explains that property-based tests are essentially functions that assert certain behaviors, akin to the design by contract methodology in Eiffel, which uses preconditions and postconditions to define expected outcomes 1. He notes, "It's a function from the input type that you generate that actually the tool generates to a true or false essentially" 2.

   

Applications

Property-based testing is particularly useful in scenarios where a reference implementation exists, allowing for easy comparison of outputs. Bill Venners highlights that while property-based tests can replace many unit tests, they may not be suitable for integration tests due to their complexity and the time required to run multiple iterations 3. He shares an experience where property-based testing identified an edge condition bug after years of testing, underscoring its potential value despite initial doubts 4. "Every single one of those property-based tests would try a hundred different ways to make it fail," he recalls, emphasizing the thoroughness of this approach.

   

Edge Testing

Edge condition testing is a significant advantage of property-based testing, as these tests often explore scenarios developers might overlook. Bill Venners explains that tools like ScalaCheck prioritize edge conditions, such as maximum integer values, to ensure comprehensive coverage 5. He mentions that while property-based testing is effective for pure functions, it can be challenging to apply to impure functions with side effects 6. "It's just kind of selected for these basic types," he says, highlighting the method's focus on edge cases.