SE Radio 589: Zac Hatfield-Dodds on Property-Based Testing in Python

Ghostwriting

Hypothesis offers a powerful feature known as ghostwriting, which automates the generation of test cases for Python functions. Zac Hatfield-Dodds explains that this tool can inspect modules like NumPy and write test cases for hundreds of functions by analyzing their names and arguments 1. This process involves type inference and custom logic to generate suitable test inputs, even without type annotations.

We have this strategy which will generate instances of a type. And so we'll create that strategy, and then we'll look at its representation, which kind of tells you how it was defined, and we can just substitute that into the source code.

--- Zac Hatfield-Dodds

Compared to other tools like GitHub Copilot, ghostwriting in Hypothesis is more adept at understanding code semantics and finding rare edge cases 2.

   

Fuzz Testing

Fuzz testing in Hypothesis is a robust method for uncovering bugs by generating random inputs and observing their effects on code. Zac highlights that this technique is particularly effective for finding defects that require multiple conditions to be met 3. Unlike traditional fuzzers like AFL, Hypothesis's fuzzing mode is tailored for Python and integrates deeply with test suites, allowing for more efficient input mutations.

We say, say we have 1000 test functions, we don't need to run each of them on a separate cpu core. We actually run each of them for a few seconds at a time and measure how much progress we're making.

--- Zac Hatfield-Dodds

This approach maximizes bug discovery rates by dynamically optimizing test execution 4.

   

Explain Mode

Explain Mode in Hypothesis enhances debugging by providing insights into which parts of the code are executed during tests. Zac describes how this feature can pinpoint the exact lines of code responsible for test failures, aiding developers in understanding and resolving issues 5. This mode leverages coverage data to highlight code segments related to both passing and failing inputs.

Coverage is the idea that we can observe which lines of code were executed by our tests.

--- Zac Hatfield-Dodds

Additionally, Hypothesis mitigates common issues in randomized testing, such as flakiness, by saving test inputs in a local database for consistent re-testing 6.