Outro

A bit of demystification

In property-based tests, unlike example-based tests (oracle), we do not specify any particular values: only the type of the input data and the property to be tested. The framework (Hypothesis/RapidCheck) will take care of actually executing the test. Magic you would say to me.

Almost magical. There is still a lot of smoke in most cases. If we take our function, we suspect that we cannot test the whole set of floats… Under the hood, there is therefore a random number generator, which stops at some point given (configurable, but not infinite). So basically I could have generated a set of random integers, written my traditional oracle test, and put a good old for loop in front of it:

import sys
import numpy as np

def add_numbers(a: int, b: int) -> int:
    return a + b

n_iteration = 1000
data_in = np.random.randint(-sys.maxsize - 1, sys.maxsize, (n_iteration, 2))
for a, b in data_in:
    assert(add_numbers(a, b) == add_numbers(b, a))

Of course, this is possible for simple cases, and there are fortunately a large number of cases where property-based tests do not really have simple equivalents.

Finally, what should I choose for my tests?

Oracle tests or property tests? Our answer is: both! If you are looking for some motivations for using property-based testing, here are a few that are quite relevant:

Property-based tests are generally more generic than Oracle tests.
They expose a more concise description of the requirements and expected behavior of our code than oracles tests.
As a result, a property-based test can replace a large number of oracle tests.
By delegating input generation to specialized tools, property-based testing will be more efficient in finding implementation problems, particular untreated values…
Property-based tests require more thinking effort than oracle tests, and therefore they allow for better introspection.
As a result, code design is often better.

But it is not always easy or relevant to find properties, so we can still rely on oracle tests. It is the combination of both that will make your code even more robust :-) To go further, we invite you to read this quite interesting blog post (based on F# - but the idea is the same).

To finish, and to meditate: programming is not just a matter of lines of code, it is above all conceiving a design that makes it possible to fulfill objectives.

Other forms of randomized testing

As you could see in this chapter, randomizing your tests is a powerful technique to make sure that you do not forget about algorithmic edge cases. Other variations of the concept that you may want to learn about include…

Fuzz testing aka fuzzing: This is a close cousin of property-based testing which is commonly used in security-critical code, such as complex file format parsers that operate on untrusted data from the Internet. In fuzzing, a specialized test harness constantly feeds your code with randomly mutated versions of a correct input, where mutations are informed by real-time tracking of changes in source code coverage. The goal of the test harness is to make your code crash. If a crash does happen, it indicates that your code is not yet fully robust in the face of unexpected inputs, such as those that could be generated by an attacker.
Mutating testing: A good test suite should catch involuntary code changes that change program results. Mutation testing verifies this by flipping the idea of property-based testing around and randomly modifying the program under test, making sure that these program changes cause the test suite to fail. If the test suite does not fail, it suggests that there are some aspects of your program that are not yet well covered by the test suite, as random changes in program outputs are not detected by the test suite.