Pytest against a wide range of data with Python hypothesis

The Hypothesis Python pytest library allows you to run your python tests against a wild range of data matching a set of hypothesis. In other words, your test function is provided with data matching the setup specifications and runs your Code Under Test (CUT) against it.

It is a nice way to automatically discover edge cases in your code without you even having to think about it.

Let’s go through an example. Let’s say you want to test the following function:

def divide_list_elements(my_list, denominator):
    return [item/denominator for item in my_list]

python> divide_list_elements([2, 4, 6], 2)
[1.0, 2.0, 3.0]

If you are like me, you would have then implemented your test strategy manually, grouped under a class because it is neat:

import unittest

class TestDivideListElements(unittest.TestCase):

    def test_divide_list_elements_one_element(self):
        result = divide_list_elements([42], 2)
        assert result == [21.0]

    def test_divide_list_elements_no_element(self):
        result = divide_list_elements([], 4)
        assert result == []

zsh> poetry run pytest tests/test_hypothesis.py::TestDivideListElements
collected 2 items

tests/test_hypothesis.py::TestDivideListElements::test_divide_list_elements_no_element PASSED
tests/test_hypothesis.py::TestDivideListElements::test_divide_list_elements_one_element PASSED

======================= 2 passed in 0.13s =======================

Well, all good right? We could have stopped there.

Now, let’s say, instead of manually defining your inputs, you let the hypothesis library managing this for you:

from hypothesis import given
from hypothesis import strategies as st

@given(st.lists(st.integers()), st.integers())
def test_divide_list_elements(input_list, input_denominator):
    result = divide_list_elements(input_list, input_denominator)
    expected = list(map(lambda x: x/input_denominator, input_list))
    assert result == expected

Running the test leaves you with an unexpected outcome:

zsh> poetry run pytest tests/test_hypothesis.py
>   return [item/denominator for item in my_list]
E   ZeroDivisionError: division by zero
E   Falsifying example: test_divide_list_elements(
E       input_list=[0],
E       input_denominator=0,
E   )

tests/test_hypothesis.py:17: ZeroDivisionError

You have obviously forgot to check about the division by 0…

Here is what is so beautiful about hypothesis: it can discovers for you edge cases you have forgotten about.

Let’s (1) redact our function:

def divide_list_elements(my_list: list, denominator: int) -> list:
    assert denominator != 0
    return [item/denominator for item in my_list]

(2) change the tests and (3) add the faulty test-case into our testing suit:

import pytest
import unittest
from hypothesis import given, example
from hypothesis import strategies as st


@given(st.lists(st.integers()), st.integers())
@example(input_list=[42], input_denominator=0)
def test_divide_list_elements(input_list, input_denominator):
    if input_denominator == 0:
        with pytest.raises(AssertionError) as exc_info:
            divide_list_elements(input_list, input_denominator)
            expected = "assert 0 != 0"
            assert expected == str(exc_info.value)
    else:
        result = divide_list_elements(input_list, input_denominator)
        expected = list(map(lambda x: x/input_denominator, input_list))
        assert result == expected

(4) run the tests again:

zsh> poetry run pytest -s tests/test_hypothesis.py::test_divide_list_elements
collected 1 item

tests/test_hypothesis.py::test_divide_list_elements PASSED

========================= 1 passed in 0.28s =====================

Notes:

• The assert denominator != 0 statement ensures our function is given correct preconditions (referring to The Pragmatic Programmer, design by contracts and crash early! “Dead Programs Tell No Lies: A dead program does a lot less damage than a crippled one.“)

• The @example(input_list=[42], input_denominator=0) statement is using the example decorator, which ensures a specific example is always tested. Here we want to make sure this edge case we missed is always checked.

• The with pytest.raises(AssertionError) ensures that whatever is in the next block of code should raise an AssertionError exception. If not exception is raised, the test fails.

To learn more about parametrization: Factorize your pytest functions using the parameterized fixture.