AI code that facilitates good science

Best Practices

Joel Grus emphasizes the importance of writing reproducible research code, which is crucial for ensuring scientific integrity. He suggests writing unit tests even for research code, as they help verify that models function as intended. Grus explains that separating library code from experiment code simplifies running experiments and avoids confusion over code versions.

If your model is not doing what you think it's doing, I mean, that's bad science out of the gate.

--- Joel Grus

Additionally, he highlights the need for clear instructions and proper dependency management to facilitate reproducibility 1 2.

   

Testing

Testing AI models presents unique challenges due to their inherent non-determinism, but Joel Grus offers strategies to address this. He suggests focusing on invariants that don't depend on randomness, such as ensuring models run without crashing and produce outputs of the correct shape. Grus also recommends using small datasets to verify that models can learn perfectly, providing confidence in their functionality.

You can come up with these tests in a way that give you some confidence that the model is doing what it's supposed to do.

--- Joel Grus

By structuring tests to account for randomness, researchers can ensure robust model performance 3 2.