pythontojob
Log inSign up
Chapter 06

parametrize — one test, many cases

Replacing copy-pasted tests and loops inside a test with @pytest.mark.parametrize. Test IDs, ids= and pytest.param, stacking decorators, values and errors in one table, and the common mistakes.

45 minPython 3.12
  1. 1Encounter
  2. 2Understand
  3. 3Worked
  4. 4Predict
  5. 5Apply
  6. 6Stretch

The problem we are solving

Here is a small function that turns an exam score into a grade, grading.py:

python
def grade(score):
    if not 0 <= score <= 100:
        raise ValueError(f"score must be between 0 and 100, got {score}")
    if score >= 80:
        return "A"
    if score >= 60:
        return "B"
    if score >= 40:
        return "C"
    return "F"
text
$ python3 -c "from grading import grade; print(grade(95), grade(79), grade(40), grade(0))"
A B C F

Testing it properly means checking many scores, especially the edges: 80 and 79, 60 and 59, 40 and 39. The first way anyone writes that is to copy a test and change two values:

python
from grading import grade


def test_grade_95():
    assert grade(95) == "A"


def test_grade_79():
    assert grade(79) == "B"


def test_grade_40():
    assert grade(40) == "C"
text
$ pytest -q test_copy.py
...                                                                      [100%]
3 passed in 0.01s

It works, and it does not scale. Three functions say one thing — "this score gives this grade" — and only two values differ between them. Ten more cases means ten more copies to keep in step.

So the next idea is a list of cases and a loop inside one test. Two of the expected grades below are wrong on purpose — 79 should be B, and 40 should be C:

python
from grading import grade


def test_grades():
    cases = [(95, "A"), (80, "A"), (79, "C"), (40, "D"), (0, "F")]
    for score, expected in cases:
        assert grade(score) == expected
text
$ pytest -q --tb=short test_loop.py
F                                                                        [100%]
=================================== FAILURES ===================================
_________________________________ test_grades __________________________________
test_loop.py:7: in test_grades
    assert grade(score) == expected
E   AssertionError: assert 'B' == 'C'
E     
E     - C
E     + B
=========================== short test summary info ============================
FAILED test_loop.py::test_grades - AssertionError: assert 'B' == 'C'
1 failed in 0.01s

That report is worse than it looks. There were two wrong cases, and you heard about one: the first failing assert ends the test, so the case for 40 never ran. The report does not say which score failed — you have to work out that 'B' == 'C' came from 79. And "1 failed" counts five cases as one fact. Fix the first, run again, and only then meet the second.

What you want is the brevity of the loop with the reporting of the copies: one function, many separate tests.

By the end of this chapter you can

  • Run one test function over many cases with @pytest.mark.parametrize
  • Pass several values per case and read the test IDs pytest generates
  • Name cases with ids= and with pytest.param(..., id=...)
  • Mark a single case xfail or skip without touching the others
  • Combine two parametrize decorators into every pairing of their values
  • Mix "returns a value" and "raises an error" cases in one table
  • Recognise the common parametrize mistakes from their error messages

Prerequisites: comparing floats with `pytest.approx`.

Before you write the test

Parametrize is a way of writing down a table. So the work before any test code is to build that table, and to decide whether a table is the right shape at all.

1. State the contract. grade(score) promises: a whole score from 0 to 100 goes in; one of "A", "B", "C", "F" comes out; anything outside 0–100 raises ValueError whose message names the score. It has no side effects — no files, no printing — so there is nothing to set up or clean up. That is what makes it a good fit: every case is "these inputs, that result".

2. Check your setup. You need the virtual environment active, pytest installed, and grading.py importable from the test file — in these examples it sits next to the tests in the same folder. Nothing else: parametrize is built into pytest, and contextlib is in the standard library.

3. Plan the cases. Boundaries matter most, because that is where >= versus > mistakes live. Each rule gets the value on each side of its line:

| Case | Input | Expected | |---|---|---| | top of the range | 100 | "A" | | lowest A | 80 | "A" | | highest B | 79 | "B" | | lowest B | 60 | "B" | | highest C | 59 | "C" | | lowest C | 40 | "C" | | highest F | 39 | "F" | | bottom of the range | 0 | "F" | | just below the range | -1 | ValueError | | just above the range | 101 | ValueError |

Each row becomes one case. The "Case" column is not decoration — it becomes the test ID later, so a failure tells you which rule broke.

4. Decide what not to test. Not every score from 0 to 100: 95 and 85 test the same rule as 80, and add run time without adding information. One value per side of each boundary is enough. Do not test Python's own comparison operators, and do not put two different behaviours under one test just because they fit in a table — if the rows need different assertions, they belong in different tests.

The rest of the chapter turns this table into code, one tool at a time.

One argument, many values

The decorator takes the name of an argument and a list of values for it. Pytest creates one test per value and passes the value in under that name:

python
import pytest

from grading import grade


@pytest.mark.parametrize("score", [80, 95, 100])
def test_top_grade(score):
    assert grade(score) == "A"
text
$ pytest -v test_one.py
collecting ... collected 3 items

test_one.py::test_top_grade[80] PASSED                                   [ 33%]
test_one.py::test_top_grade[95] PASSED                                   [ 66%]
test_one.py::test_top_grade[100] PASSED                                  [100%]

============================== 3 passed in 0.01s ===============================

One function, three tests, each with its own line and its own name. The part in square brackets is the test ID, built from the value. score is not a fixture and has no default — the name in the decorator string simply has to match a parameter of the function.

Several arguments per case

Most cases need an input and an expected result. Name both, separated by a comma, and give each case as a tuple in the same order. Here is the loop again, with the same two wrong expectations:

python
import pytest

from grading import grade


@pytest.mark.parametrize(
    "score,expected",
    [(95, "A"), (80, "A"), (79, "C"), (40, "D"), (0, "F")],
)
def test_grade(score, expected):
    assert grade(score) == expected
text
$ pytest -q --tb=short test_param.py
..FF.                                                                    [100%]
=================================== FAILURES ===================================
_______________________________ test_grade[79-C] _______________________________
test_param.py:11: in test_grade
    assert grade(score) == expected
E   AssertionError: assert 'B' == 'C'
E     
E     - C
E     + B
_______________________________ test_grade[40-D] _______________________________
test_param.py:11: in test_grade
    assert grade(score) == expected
E   AssertionError: assert 'C' == 'D'
E     
E     - D
E     + C
=========================== short test summary info ============================
FAILED test_param.py::test_grade[79-C] - AssertionError: assert 'B' == 'C'
FAILED test_param.py::test_grade[40-D] - AssertionError: assert 'C' == 'D'
2 failed, 3 passed in 0.01s

Compare this with the loop. Every case ran. Both failures are reported, each under an ID — test_grade[79-C] — that names the case. The count is honest: 2 failed, 3 passed. (With the default long traceback, pytest also prints the arguments of each failing case, score = 79, expected = 'C'.)

The names can be written as one string, "score,expected" or "score, expected" — spaces after commas are allowed — or as a list of strings, ["score", "expected"]. Both produce exactly the same tests. The string is the common form; the list is handy when the names are built by code.

How the IDs are made

For numbers, strings, booleans and None, pytest puts the value itself in the ID and joins the values of a case with -. For anything else — a list, a dict, an object — it does not try. It uses the argument name plus the case's position:

python
import pytest


@pytest.mark.parametrize(
    ["value", "expected"],
    [
        (1.5, True),
        ("hello world", None),
        ([1, 2], {"a": 1}),
    ],
)
def test_show_ids(value, expected):
    pass
text
$ pytest -v test_ids_auto.py
collecting ... collected 3 items

test_ids_auto.py::test_show_ids[1.5-True] PASSED                         [ 33%]
test_ids_auto.py::test_show_ids[hello world-None] PASSED                 [ 66%]
test_ids_auto.py::test_show_ids[value2-expected2] PASSED                 [100%]

============================== 3 passed in 0.01s ===============================

value2-expected2 means "value and expected of case number 2, counting from zero". It is unique, and it tells you nothing. When such a case fails, you count down the list to find it. That is the reason to name cases yourself.

Naming the cases: ids=

Pass ids= a list of strings, one per case, in the same order:

python
import pytest

from grading import grade


@pytest.mark.parametrize(
    "score,expected",
    [(80, "A"), (79, "B"), (40, "C"), (39, "F")],
    ids=["lowest-A", "highest-B", "lowest-C", "highest-F"],
)
def test_grade_boundaries(score, expected):
    assert grade(score) == expected
text
$ pytest -v test_ids_list.py
collecting ... collected 4 items

test_ids_list.py::test_grade_boundaries[lowest-A] PASSED                 [ 25%]
test_ids_list.py::test_grade_boundaries[highest-B] PASSED                [ 50%]
test_ids_list.py::test_grade_boundaries[lowest-C] PASSED                 [ 75%]
test_ids_list.py::test_grade_boundaries[highest-F] PASSED                [100%]

============================== 4 passed in 0.01s ===============================

Now the output says why each case exists. The weakness is that the names live in a second list, away from their cases: insert a case in the middle, forget its name, and every name after it is off by one.

ids= also accepts a function. Pytest calls it once for each value, not each case, and joins the results with -:

python
import pytest

from grading import grade


def describe(value):
    if isinstance(value, int):
        return f"score={value}"
    return f"grade={value}"


@pytest.mark.parametrize("score,expected", [(80, "A"), (79, "B")], ids=describe)
def test_grade_named(score, expected):
    assert grade(score) == expected
text
$ pytest -v test_ids_func.py
collecting ... collected 2 items

test_ids_func.py::test_grade_named[score=80-grade=A] PASSED              [ 50%]
test_ids_func.py::test_grade_named[score=79-grade=B] PASSED              [100%]

============================== 2 passed in 0.01s ===============================

A function suits values whose automatic ID is useless, such as objects. If it returns None for a value, pytest falls back to the automatic ID for that value.

One case at a time: pytest.param

pytest.param wraps a single case and lets you attach things to it: an id= that sits right next to its values, and marks= that apply to that case only.

python
import pytest

from grading import grade


@pytest.mark.parametrize(
    "score,expected",
    [
        pytest.param(100, "A", id="perfect"),
        pytest.param(60, "B", id="lowest-B"),
        pytest.param(59.5, "B", id="round-up", marks=pytest.mark.xfail(reason="no rounding yet")),
        pytest.param(0, "F", id="zero", marks=pytest.mark.skip(reason="example of skip")),
    ],
)
def test_grade_cases(score, expected):
    assert grade(score) == expected
text
$ pytest -v test_param_obj.py
collecting ... collected 4 items

test_param_obj.py::test_grade_cases[perfect] PASSED                      [ 25%]
test_param_obj.py::test_grade_cases[lowest-B] PASSED                     [ 50%]
test_param_obj.py::test_grade_cases[round-up] XFAIL (no rounding yet)    [ 75%]
test_param_obj.py::test_grade_cases[zero] SKIPPED (example of skip)      [100%]

=================== 2 passed, 1 skipped, 1 xfailed in 0.01s ====================

The round-up case records a decision not yet made — should 59.5 round up to a B? — without failing the run, and the other cases are untouched. (xfail and skip get their own chapter later; for now, know that a mark can go on one case.)

Because the ID is a real name, one case can be run on its own: pytest "test_param_obj.py::test_grade_cases[perfect]". Quote it — the shell treats square brackets specially.

Two decorators: every combination

Stack two parametrize decorators and pytest runs the test for every pairing of their values — the cartesian product. Two currencies times three amounts is six tests:

python
import pytest


@pytest.mark.parametrize("currency", ["USD", "EUR"])
@pytest.mark.parametrize("amount", [0, 10, 999])
def test_format(amount, currency):
    text = f"{amount} {currency}"
    assert text.endswith(currency)
text
$ pytest -v test_stack.py
collecting ... collected 6 items

test_stack.py::test_format[0-USD] PASSED                                 [ 16%]
test_stack.py::test_format[0-EUR] PASSED                                 [ 33%]
test_stack.py::test_format[10-USD] PASSED                                [ 50%]
test_stack.py::test_format[10-EUR] PASSED                                [ 66%]
test_stack.py::test_format[999-USD] PASSED                               [ 83%]
test_stack.py::test_format[999-EUR] PASSED                               [100%]

============================== 6 passed in 0.01s ===============================

The decorator nearest the function supplies the first part of the ID. Stack only when the inputs really are independent — every amount should work with every currency. If only certain pairs make sense, list those pairs as tuples in one decorator. And watch the multiplication: three decorators of ten values each is a thousand tests.

Values and errors in one table

Sometimes the cases split into "returns this" and "raises that". You can make the expectation itself a parameter, and enter it with with:

python
from contextlib import nullcontext

import pytest

from grading import grade


@pytest.mark.parametrize(
    "score,expectation",
    [
        (50, nullcontext("C")),
        (100, nullcontext("A")),
        (-1, pytest.raises(ValueError)),
        (101, pytest.raises(ValueError, match="got 101")),
    ],
)
def test_grade_or_error(score, expectation):
    with expectation as expected:
        assert grade(score) == expected
text
$ pytest -v test_raises.py
collecting ... collected 4 items

test_raises.py::test_grade_or_error[50-expectation0] PASSED              [ 25%]
test_raises.py::test_grade_or_error[100-expectation1] PASSED             [ 50%]
test_raises.py::test_grade_or_error[-1-expectation2] PASSED              [ 75%]
test_raises.py::test_grade_or_error[101-expectation3] PASSED             [100%]

============================== 4 passed in 0.01s ===============================

contextlib.nullcontext is a context manager that does nothing: with nullcontext("C") as expected just sets expected = "C", so the normal cases run the assert as usual. In the error cases pytest.raises is the context manager: grade raises inside it, the assert never finishes, and the test passes because the right error arrived. If no error arrives you get the familiar DID NOT RAISE.

This is tidy for a short table. If the test body starts needing if statements to tell the two kinds apart, split it into two tests. Note the IDs, too: expectation0 is the object rule again. The complete example fixes that with pytest.param.

Parametrizing a class

Put the decorator on a test class and every test method in it receives the parameter:

python
import pytest

from grading import grade


@pytest.mark.parametrize("score", [80, 100])
class TestTopBand:
    def test_is_a(self, score):
        assert grade(score) == "A"

    def test_is_a_string(self, score):
        assert isinstance(grade(score), str)
text
$ pytest -v test_class.py
collecting ... collected 4 items

test_class.py::TestTopBand::test_is_a[80] PASSED                         [ 25%]
test_class.py::TestTopBand::test_is_a[100] PASSED                        [ 50%]
test_class.py::TestTopBand::test_is_a_string[80] PASSED                  [ 75%]
test_class.py::TestTopBand::test_is_a_string[100] PASSED                 [100%]

============================== 4 passed in 0.01s ===============================

Two methods times two values: four tests. Every method in the class must accept score, or collection fails.

A complete example

shipping.py:

python
import math

RATES = {"local": 50, "national": 120}


def shipping_cost(weight_kg, zone):
    if zone not in RATES:
        raise KeyError(f"unknown zone: {zone}")
    if weight_kg <= 0:
        raise ValueError("weight must be positive")
    # The first kilogram is in the base rate; each started kilogram after it costs 20.
    return RATES[zone] + (math.ceil(weight_kg) - 1) * 20
text
$ python3 -c "from shipping import shipping_cost; print(shipping_cost(0.5, 'local'), shipping_cost(1.2, 'local'), shipping_cost(3, 'national'))"
50 70 160

test_shipping.py:

python
from contextlib import nullcontext

import pytest

from shipping import shipping_cost


@pytest.mark.parametrize(
    "weight_kg,zone,expected",
    [
        pytest.param(0.5, "local", 50, id="light-local"),
        pytest.param(1, "local", 50, id="exactly-1kg"),
        pytest.param(1.2, "local", 70, id="part-kg-rounds-up"),
        pytest.param(3, "national", 160, id="heavy-national"),
    ],
)
def test_cost(weight_kg, zone, expected):
    assert shipping_cost(weight_kg, zone) == expected


@pytest.mark.parametrize("zone", ["local", "national"])
@pytest.mark.parametrize("weight_kg", [0.1, 0.5, 1])
def test_first_kg_is_flat(weight_kg, zone):
    assert shipping_cost(weight_kg, zone) == shipping_cost(1, zone)


@pytest.mark.parametrize(
    "weight_kg,zone,expectation",
    [
        pytest.param(2, "local", nullcontext(70), id="ok"),
        pytest.param(0, "local", pytest.raises(ValueError), id="zero-weight"),
        pytest.param(-1, "local", pytest.raises(ValueError), id="negative"),
        pytest.param(1, "moon", pytest.raises(KeyError, match="moon"), id="bad-zone"),
    ],
)
def test_cost_or_error(weight_kg, zone, expectation):
    with expectation as expected:
        assert shipping_cost(weight_kg, zone) == expected
text
$ pytest -v test_shipping.py
collecting ... collected 14 items

test_shipping.py::test_cost[light-local] PASSED                          [  7%]
test_shipping.py::test_cost[exactly-1kg] PASSED                          [ 14%]
test_shipping.py::test_cost[part-kg-rounds-up] PASSED                    [ 21%]
test_shipping.py::test_cost[heavy-national] PASSED                       [ 28%]
test_shipping.py::test_first_kg_is_flat[0.1-local] PASSED                [ 35%]
test_shipping.py::test_first_kg_is_flat[0.1-national] PASSED             [ 42%]
test_shipping.py::test_first_kg_is_flat[0.5-local] PASSED                [ 50%]
test_shipping.py::test_first_kg_is_flat[0.5-national] PASSED             [ 57%]
test_shipping.py::test_first_kg_is_flat[1-local] PASSED                  [ 64%]
test_shipping.py::test_first_kg_is_flat[1-national] PASSED               [ 71%]
test_shipping.py::test_cost_or_error[ok] PASSED                          [ 78%]
test_shipping.py::test_cost_or_error[zero-weight] PASSED                 [ 85%]
test_shipping.py::test_cost_or_error[negative] PASSED                    [ 92%]
test_shipping.py::test_cost_or_error[bad-zone] PASSED                    [100%]

============================== 14 passed in 0.01s ==============================

Three functions, fourteen tests, and every line says what it checked. Three things are worth noticing.

First, the IDs in test_cost give the reason for each case — part-kg-rounds-up names the rule being protected, which 1.2-local-70 would not.

Second, test_first_kg_is_flat stacks because weight and zone really are independent. It also compares the function with itself rather than with a number, so it survives a change of rates.

Third, pytest.param(..., id=...) turns expectation2 into negative. The table of values and errors now reads like a specification.

When it breaks

function uses no argument 'scores' A name in the decorator string matches no parameter of the function — here "scores,expected" against def test_grade(score, expected). It fails at collection, before any test runs:

python
import pytest

from grading import grade


@pytest.mark.parametrize("scores,expected", [(80, "A"), (40, "C")])
def test_grade(score, expected):
    assert grade(score) == expected
text
$ pytest -q test_mismatch.py

==================================== ERRORS ====================================
______________________ ERROR collecting test_mismatch.py _______________________
In test_mismatch.py::test_grade: function uses no argument 'scores'
=========================== short test summary info ============================
ERROR test_mismatch.py - Failed: In test_mismatch.py::test_grade: function us...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.01s

fixture 'expected' not found The opposite: the function has a parameter the decorator never names, so pytest goes looking for a fixture of that name and finds none. Add the name to the decorator string.

the number of names (2) ... must be equal to the number of values (1) One case has the wrong number of values — say (40,) in a list for "score,expected":

text
$ pytest -q test_count.py

==================================== ERRORS ====================================
________________________ ERROR collecting test_count.py ________________________
test_count.py::test_grade: in "parametrize" the number of names (2):
  ['score', 'expected']
must be equal to the number of values (1):
  (40,)

IDs like [score0] and a TypeError mentioning a tuple With one argument name, each value in the list is passed whole. Write tuples there out of habit and the function receives a tuple:

python
import pytest

from grading import grade


@pytest.mark.parametrize("score", [(80,), (95,)])
def test_top_grade(score):
    assert grade(score) == "A"
text
$ pytest -v --tb=no test_tuples.py
collecting ... collected 2 items

test_tuples.py::test_top_grade[score0] FAILED                            [ 50%]
test_tuples.py::test_top_grade[score1] FAILED                            [100%]

=========================== short test summary info ============================
FAILED test_tuples.py::test_top_grade[score0] - TypeError: '<=' not supported...
FAILED test_tuples.py::test_top_grade[score1] - TypeError: '<=' not supported...
============================== 2 failed in 0.01s ===============================

The IDs are the clue: a plain number would have given [80], so [score0] says the value was something else. For one argument, write [80, 95].

A case passes alone and fails alongside others Parameter values are created once, when the module is collected, and the same object is handed to every test that receives it. If a test changes a mutable value — a list, a dict — the next test sees the change:

python
import pytest


@pytest.mark.parametrize("cart", [[]])
class TestCart:
    def test_add_item(self, cart):
        cart.append("pen")
        assert cart == ["pen"]

    def test_starts_empty(self, cart):
        assert cart == []
text
$ pytest -q --tb=short test_shared.py
.F                                                                       [100%]
=================================== FAILURES ===================================
______________________ TestCart.test_starts_empty[cart0] _______________________
test_shared.py:11: in test_starts_empty
    assert cart == []
E   AssertionError: assert ['pen'] == []
E     
E     Left contains one more item: 'pen'
E     Use -v to get more diff
=========================== short test summary info ============================
FAILED test_shared.py::TestCart::test_starts_empty[cart0] - AssertionError: a...
1 failed, 1 passed in 0.01s

Pass something immutable, such as the tuple (), and build a fresh object inside each test with cart = list(items). Then each test starts from its own empty list and both pass.