Chapter 04

Testing exceptions — pytest.raises

Errors are behaviour, so they need tests too. pytest.raises, DID NOT RAISE, subclass matching, why match= is a regex, inspecting the caught error with excinfo, and why only the last line belongs inside the with block.

45 minPython 3.12
  1. 1Encounter
  2. 2Understand
  3. 3Worked
  4. 4Predict
  5. 5Apply
  6. 6Stretch

The problem we are solving

Here is a small function that guards a balance. It has two refusals built in: an amount that is not positive, and an amount larger than what is there.

wallet.py:

python
class InsufficientFunds(Exception):
    def __init__(self, balance: float, amount: float):
        self.balance = balance
        self.amount = amount
        super().__init__(f"cannot withdraw {amount:.2f}; balance is {balance:.2f}")


def withdraw(balance: float, amount: float) -> float:
    if amount <= 0:
        raise ValueError(f"amount must be positive (got {amount})")
    if amount > balance:
        raise InsufficientFunds(balance, amount)
    return balance - amount

Those two raise lines are behaviour, exactly as much as the return is. A caller relies on withdraw(100.0, -5.0) refusing, and if that refusal disappears one day, money moves the wrong way. So it deserves a test. The first test most people write looks like this:

python
from wallet import withdraw


def test_negative_amount_is_rejected():
    try:
        withdraw(100.0, -5.0)
    except ValueError:
        pass

Now suppose someone deletes the if amount <= 0 check from withdraw. Run pytest -q:

text
.                                                                        [100%]
1 passed in 0.01s

The check is gone and the test still passes. With no exception, the try block simply finishes, except is skipped, and the function ends without a failed assert. The test only ever checked that if a ValueError happened, it would be ignored. It never said the error must happen.

This chapter is about saying that properly — and about the handful of ways a test of an error can pass when it should not.

By the end of this chapter you can

  • Test that code raises with pytest.raises and read DID NOT RAISE
  • Explain why pytest.raises(Exception) is almost always too broad
  • Check the message with match=, knowing it is a regular expression
  • Inspect the caught exception through excinfo
  • Keep only the raising line inside the with block, and say why
  • Recognise an ExceptionGroup and check what is inside it

Prerequisites: Discovery and running tests.


Before you write the test

A test of an error is only as good as the decision behind it. Before typing pytest.raises, settle three things on paper.

1. What exactly is promised? Read withdraw as a contract, not as code:

  • Input: a balance and an amount. Output: the new balance, balance - amount.
  • Refusal one: an amount that is zero or negative raises ValueError, and the message names the amount.
  • Refusal two: an amount larger than the balance raises InsufficientFunds, which carries balance and amount as attributes.
  • Side effects: none. withdraw changes nothing outside itself, so "nothing changed after a refusal" is automatic here — it will not be in the bank example at the end.

Each refusal is a separate promise, with three parts you can check: which class, what message, and what state is left behind.

2. What has to be in place? Nothing beyond the last chapter. pytest.raises is part of pytest, and re is in the standard library. You need the virtual environment active, pytest installed, and wallet.py importable from the test file — in this chapter it sits in the same folder as test_wallet.py. Check the first two once:

text
$ python -m pytest --version
pytest 9.1.1

Then look at the code's real behaviour before you write a single claim about it. A short probe answers questions a guess cannot — is 0.0 returned, or refused? what does the message look like?

python
from wallet import InsufficientFunds, withdraw

print(withdraw(100.0, 30.0))
print(withdraw(100.0, 100.0))
try:
    withdraw(100.0, 100.01)
except InsufficientFunds as error:
    print(repr(error))
text
70.0
0.0
InsufficientFunds('cannot withdraw 100.01; balance is 100.00')

3. Which cases? Errors live at boundaries, so plan around them — the ordinary case, the edges of each refusal on both sides, and the clearly invalid:

| Case | Input | Expected | |---|---|---| | Ordinary withdrawal | withdraw(100.0, 30.0) | returns 70.0 | | Whole balance (boundary, allowed) | withdraw(100.0, 100.0) | returns 0.0, no error | | Just over (boundary, refused) | withdraw(100.0, 100.01) | InsufficientFunds, amount == 100.01 | | Zero (boundary, refused) | withdraw(100.0, 0) | ValueError | | Negative | withdraw(100.0, -5.0) | ValueError, message contains (got -5.0) | | Clear overdraft | withdraw(100.0, 150.0) | InsufficientFunds, balance == 100.0, amount == 150.0 |

The two "allowed" rows matter as much as the refusals. A check written as >= instead of > refuses the whole-balance case, and only a test that expects no error there will notice.

What not to test. Python's own behaviour: that calling withdraw(100.0) with a missing argument raises TypeError is Python's promise, not yours. The full wording of every message, unless the wording itself is the contract — a fragment that carries the value is enough. And anything the function never promised, such as what happens when you pass the string "ten". Testing those adds tests that break on harmless changes and protect nothing.

The rest of the chapter turns this table into tests, one tool at a time.


pytest.raises — "this must raise"

pytest.raises is a context manager. Everything inside the with block is expected to raise the named exception:

python
import pytest

from wallet import withdraw


def test_negative_amount_is_rejected():
    with pytest.raises(ValueError):
        withdraw(100.0, -5.0)


def test_zero_is_rejected():
    with pytest.raises(ValueError):
        withdraw(100.0, 0)
text
..                                                                       [100%]
2 passed in 0.01s

If a ValueError comes out of the block, pytest.raises catches it and the test carries on after the block. That is the half try/except already had. The other half is what happens when nothing comes out. Delete the if amount <= 0 check again and run the same file:

text
FF                                                                       [100%]
=================================== FAILURES ===================================
_______________________ test_negative_amount_is_rejected _______________________

    def test_negative_amount_is_rejected():
>       with pytest.raises(ValueError):
E       Failed: DID NOT RAISE ValueError

test_wallet.py:7: Failed
____________________________ test_zero_is_rejected _____________________________

    def test_zero_is_rejected():
>       with pytest.raises(ValueError):
E       Failed: DID NOT RAISE ValueError

test_wallet.py:12: Failed
=========================== short test summary info ============================
FAILED test_wallet.py::test_negative_amount_is_rejected - Failed: DID NOT RAI...
FAILED test_wallet.py::test_zero_is_rejected - Failed: DID NOT RAISE ValueError
2 failed in 0.01s

DID NOT RAISE ValueError is the sentence the try/except version could never say. That is all pytest.raises is: catch the expected error, and fail loudly when it does not arrive.

Which exceptions count as a match

pytest.raises(X) matches exactly the way except X: does: X itself or any subclass of it. Python's exceptions form a family tree, and you can see a branch of it:

python
print(KeyError.__mro__)
print(issubclass(KeyError, LookupError))
print(issubclass(IndexError, LookupError))
text
(<class 'KeyError'>, <class 'LookupError'>, <class 'Exception'>, <class 'BaseException'>, <class 'object'>)
True
True

So a KeyError is a LookupError. That is useful, and it is also the trap. Here are two tests that both pass:

python
import pytest

from wallet import withdraw


def test_missing_price_is_a_lookup_error():
    prices = {"pen": 15.0}
    with pytest.raises(LookupError):
        prices["eraser"]


def test_too_broad():
    with pytest.raises(Exception):
        withdraw(100.0)  # forgot the amount
text
..                                                                       [100%]
2 passed in 0.01s

The first is fine — catching the parent on purpose. The second is a test that proves nothing. withdraw(100.0) never reached the function body: Python raised a TypeError for the missing argument, and since every ordinary exception is a subclass of Exception, pytest.raises(Exception) happily accepted it. A typo, a wrong import, an AttributeError — all of them would make this test green.

The rule: name the most specific exception the code promises to raise. The wider the class, the more unrelated failures it hides.

The other direction is strict. An exception that is not the named class or a subclass is not caught; it escapes the block and fails the test as an ordinary error. (pytest -q --tb=short trims each traceback to one line per frame.)

python
import pytest

from wallet import withdraw


def test_overdraft_is_rejected():
    with pytest.raises(ValueError):
        withdraw(100.0, 150.0)
text
F                                                                        [100%]
=================================== FAILURES ===================================
__________________________ test_overdraft_is_rejected __________________________
test_wallet.py:8: in test_overdraft_is_rejected
    withdraw(100.0, 150.0)
wallet.py:12: in withdraw
    raise InsufficientFunds(balance, amount)
E   wallet.InsufficientFunds: cannot withdraw 150.00; balance is 100.00
=========================== short test summary info ============================
FAILED test_wallet.py::test_overdraft_is_rejected - wallet.InsufficientFunds:...
1 failed in 0.01s

Not DID NOT RAISE — something was raised, just not what you named, and pytest shows you where it came from.

match= — checking the message

The type says what kind of refusal it was. The message says why, and often carries the offending value. match= checks it:

python
import pytest

from wallet import withdraw


def test_message_says_positive():
    with pytest.raises(ValueError, match="must be positive"):
        withdraw(100.0, -5.0)


def test_whole_message():
    with pytest.raises(ValueError, match="amount must be positive (got -5.0)"):
        withdraw(100.0, -5.0)
text
.F                                                                       [100%]
=================================== FAILURES ===================================
______________________________ test_whole_message ______________________________

    def test_whole_message():
>       with pytest.raises(ValueError, match="amount must be positive (got -5.0)"):
E       AssertionError: Regex pattern did not match.
E         Expected regex: 'amount must be positive (got -5.0)'
E         Actual message: 'amount must be positive (got -5.0)'
E        Did you mean to `re.escape()` the regex?

test_wallet.py:12: AssertionError
=========================== short test summary info ============================
FAILED test_wallet.py::test_whole_message - AssertionError: Regex pattern did...
1 failed, 1 passed in 0.01s

The expected text and the actual message are identical, and yet it failed. The reason is in the first line: match= is not a plain string. It is a regular expression, and pytest runs re.search(pattern, str(exception)). Two consequences:

  • re.search looks anywhere in the message. "must be positive" matched in the middle. To pin the start or the end, use ^ and $.
  • Some characters mean something. In a regex, ( ) make a group and match no parenthesis at all; . matches any character; +, *, ?, [, $ all have jobs too.

The . case is the quieter one, because it fails in the passing direction:

python
import re

print(re.search("100.00", "balance is 100.00"))
print(re.search("100.00", "balance is 100500"))
print(re.search(re.escape("100.00"), "balance is 100500"))
text
<re.Match object; span=(11, 17), match='100.00'>
<re.Match object; span=(11, 17), match='100500'>
None

"100.00" happily matched 100500. re.escape turns every special character into a literal one, so when you mean "this exact text", pass the text through it — exactly as pytest's hint suggested:

python
import re

import pytest

from wallet import withdraw


def test_whole_message():
    message = "amount must be positive (got -5.0)"
    with pytest.raises(ValueError, match=re.escape(message)):
        withdraw(100.0, -5.0)
text
.                                                                        [100%]
1 passed in 0.01s

How much of the message to match is a judgement call. A short, distinctive fragment — the value, the key word — survives harmless rewording; the whole sentence breaks the moment someone fixes a typo.

excinfo — the exception you caught

with pytest.raises(...) as excinfo gives you a handle on what was caught. Look inside it once with pytest -q -s (-s lets print through):

python
import pytest

from wallet import InsufficientFunds, withdraw


def test_look_inside():
    with pytest.raises(InsufficientFunds) as excinfo:
        withdraw(100.0, 150.0)

    print()
    print("type  :", excinfo.type)
    print("value :", repr(excinfo.value))
    print("str   :", str(excinfo.value))
    print("fields:", excinfo.value.balance, excinfo.value.amount)
text
type  : <class 'wallet.InsufficientFunds'>
value : InsufficientFunds('cannot withdraw 150.00; balance is 100.00')
str   : cannot withdraw 150.00; balance is 100.00
fields: 100.0 150.0
.
1 passed in 0.01s
  • excinfo.type is the class that was actually raised — useful when you caught a parent and want to know which child came.
  • excinfo.value is the exception object itself, with every attribute it carries.
  • str(excinfo.value) is the message — the same string match= searches.

The prints were only for looking. In a real test you assert:

python
import pytest

from wallet import InsufficientFunds, withdraw


def test_overdraft_reports_both_numbers():
    with pytest.raises(InsufficientFunds) as excinfo:
        withdraw(100.0, 150.0)

    assert excinfo.value.balance == 100.0
    assert excinfo.value.amount == 150.0
    assert str(excinfo.value) == "cannot withdraw 150.00; balance is 100.00"
text
.                                                                        [100%]
1 passed in 0.01s

When an exception carries data as attributes, asserting on the attributes is sturdier than parsing the message. Notice the asserts sit after the with block, not inside it. That is not style; the next section is why.

Only the raising line goes inside

The moment an exception is raised, Python leaves the block. Every line after the raising one is skipped — including asserts:

python
import pytest

from wallet import InsufficientFunds, withdraw


def test_failed_withdrawal_keeps_balance():
    balance = 100.0
    with pytest.raises(InsufficientFunds):
        balance = withdraw(balance, 150.0)
        assert balance == 999.0  # never runs


def test_text_amount_is_rejected():
    with pytest.raises(ValueError):
        amount = float("ten")  # raises ValueError itself
        withdraw(100.0, amount)
text
..                                                                       [100%]
2 passed in 0.01s

Both tests are green, and both are wrong. In the first, assert balance == 999.0 is a false claim that never ran. In the second, float("ten") raised a ValueError on the setup line, so withdraw was never even called — the test passes whether withdraw checks anything or not.

The rule: setup before the block, one raising call inside it, checks after it.

python
import pytest

from wallet import InsufficientFunds, withdraw


def test_failed_withdrawal_keeps_balance():
    balance = 100.0
    with pytest.raises(InsufficientFunds):
        balance = withdraw(balance, 150.0)

    assert balance == 100.0
text
.                                                                        [100%]
1 passed in 0.01s

Now the assert runs, and it checks something real: a refused withdrawal left balance untouched, because the assignment never happened.

More than one acceptable exception

pytest.raises also takes a tuple, again just like except. Any of the listed classes will do, and excinfo.type tells you which one came:

python
import pytest

from wallet import InsufficientFunds, withdraw


def test_bad_withdrawal_is_refused():
    with pytest.raises((ValueError, InsufficientFunds)) as excinfo:
        withdraw(100.0, 150.0)

    print()
    print(excinfo.type.__name__)
text
InsufficientFunds
.
1 passed in 0.01s

Use this when the contract really is "one of these" — for instance, code that may legitimately raise either depending on the platform. For your own functions, one precise class is nearly always the better promise.

Testing that something does not raise

There is no pytest.does_not_raise() to reach for, and you do not need one. Call the code. If it raises, the test fails with that exception and its traceback:

python
from wallet import withdraw


def test_whole_balance_can_be_withdrawn():
    assert withdraw(100.0, 100.0) == 0.0
text
.                                                                        [100%]
1 passed in 0.01s

Change amount > balance to amount >= balance in withdraw — an off-by-one many people write — and pytest -q --tb=short reports:

text
F                                                                        [100%]
=================================== FAILURES ===================================
_____________________ test_whole_balance_can_be_withdrawn ______________________
test_wallet.py:5: in test_whole_balance_can_be_withdrawn
    assert withdraw(100.0, 100.0) == 0.0
           ^^^^^^^^^^^^^^^^^^^^^^
wallet.py:12: in withdraw
    raise InsufficientFunds(balance, amount)
E   wallet.InsufficientFunds: cannot withdraw 100.00; balance is 100.00
=========================== short test summary info ============================
FAILED test_wallet.py::test_whole_balance_can_be_withdrawn - wallet.Insuffici...
1 failed in 0.01s

An unexpected exception is already a failure. Wrapping a call in try/except: pytest.fail() only throws away that traceback.

Several errors at once: ExceptionGroup

Since Python 3.11, code can raise several exceptions together as an ExceptionGroup. Add a batch version to wallet.py that tries every withdrawal and reports all the failures together:

python
def withdraw_all(balance: float, amounts: list[float]) -> float:
    errors = []
    for amount in amounts:
        try:
            balance = withdraw(balance, amount)
        except (ValueError, InsufficientFunds) as error:
            errors.append(error)
    if errors:
        raise ExceptionGroup(f"{len(errors)} withdrawals failed", errors)
    return balance

A group is its own exception class. pytest.raises(ValueError) does not look inside it. To check the contents, catch the group and ask excinfo.group_contains, or describe the whole group with pytest.RaisesGroup:

python
import pytest

from wallet import InsufficientFunds, withdraw_all


def test_plain_raises_does_not_look_inside():
    with pytest.raises(ValueError):
        withdraw_all(100.0, [-5.0, 30.0, 500.0])


def test_group_contains():
    with pytest.raises(ExceptionGroup) as excinfo:
        withdraw_all(100.0, [-5.0, 30.0, 500.0])

    assert excinfo.group_contains(ValueError, match="must be positive")
    assert excinfo.group_contains(InsufficientFunds)
    assert not excinfo.group_contains(KeyError)


def test_raises_group():
    with pytest.RaisesGroup(ValueError, InsufficientFunds):
        withdraw_all(100.0, [-5.0, 30.0, 500.0])

Run with pytest -q --tb=no to see only the outcome:

text
F..                                                                      [100%]
=========================== short test summary info ============================
FAILED test_wallet.py::test_plain_raises_does_not_look_inside - ExceptionGrou...
1 failed, 2 passed in 0.01s

The first test fails: the group escaped pytest.raises(ValueError) untouched. group_contains answers "is there one of these anywhere inside?" and accepts match= too. pytest.RaisesGroup is stricter: the group must hold exactly the listed exceptions — no more, no fewer, in any order. Both arrived in pytest 8 and are there in pytest 9. You will not need them often; you will need them the day a library hands you a group.


A complete example

A small bank with three refusals: an unknown account, a non-positive amount, and an overdraft. The amount check works exactly like withdraw's, so the tests below spend their effort on the other two. AccountNotFound deliberately subclasses LookupError, so callers can catch it the way they would catch a missing dictionary key.

bank.py:

python
class AccountNotFound(LookupError):
    def __init__(self, account_id: str):
        self.account_id = account_id
        super().__init__(f"no account with id {account_id!r}")


class InsufficientFunds(Exception):
    def __init__(self, balance: float, amount: float):
        self.balance = balance
        self.amount = amount
        super().__init__(f"cannot withdraw {amount:.2f}; balance is {balance:.2f}")


class Bank:
    def __init__(self) -> None:
        self._balances: dict[str, float] = {}

    def open(self, account_id: str, deposit: float) -> None:
        self._balances[account_id] = deposit

    def balance(self, account_id: str) -> float:
        try:
            return self._balances[account_id]
        except KeyError:
            raise AccountNotFound(account_id) from None

    def transfer(self, source: str, target: str, amount: float) -> None:
        if amount <= 0:
            raise ValueError(f"amount must be positive (got {amount})")
        # Look both accounts up first, so nothing moves if either is missing.
        available = self.balance(source)
        self.balance(target)
        if amount > available:
            raise InsufficientFunds(available, amount)
        self._balances[source] -= amount
        self._balances[target] += amount

test_bank.py:

python
import pytest

from bank import AccountNotFound, Bank, InsufficientFunds


def make_bank() -> Bank:
    bank = Bank()
    bank.open("alice", 100.0)
    bank.open("bob", 20.0)
    return bank


def test_transfer_moves_money():
    bank = make_bank()

    bank.transfer("alice", "bob", 30.0)  # must not raise

    assert bank.balance("alice") == 70.0
    assert bank.balance("bob") == 50.0


def test_unknown_account_is_a_lookup_error():
    bank = make_bank()

    with pytest.raises(LookupError) as excinfo:
        bank.balance("carol")

    assert excinfo.type is AccountNotFound
    assert excinfo.value.account_id == "carol"


def test_unknown_target_moves_nothing():
    bank = make_bank()

    with pytest.raises(AccountNotFound, match="'carol'"):
        bank.transfer("alice", "carol", 30.0)

    assert bank.balance("alice") == 100.0


def test_overdraft_reports_both_numbers():
    bank = make_bank()

    with pytest.raises(InsufficientFunds) as excinfo:
        bank.transfer("bob", "alice", 25.0)

    assert excinfo.value.balance == 20.0
    assert excinfo.value.amount == 25.0
    assert bank.balance("bob") == 20.0

pytest -v:

text
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.1.1, pluggy-1.6.0
rootdir: /home/you/bank
collecting ... collected 4 items

test_bank.py::test_transfer_moves_money PASSED                           [ 25%]
test_bank.py::test_unknown_account_is_a_lookup_error PASSED              [ 50%]
test_bank.py::test_unknown_target_moves_nothing PASSED                   [ 75%]
test_bank.py::test_overdraft_reports_both_numbers PASSED                 [100%]

============================== 4 passed in 0.01s ===============================

Three things are worth noticing.

First, every test keeps its with block to one call, and checks state after it. test_unknown_target_moves_nothing is the valuable one: if transfer took the money from Alice before discovering that Carol does not exist, it would still raise AccountNotFound — and the assert after the block would catch the half-finished transfer with assert 70.0 == 100.0.

Second, test_unknown_account_is_a_lookup_error catches the parent class on purpose, because that is the promise bank.py makes, and then uses excinfo.type to pin down which child arrived.

Third, match="'carol'" checks that the message names the account that was asked for — the one detail a person reading the error needs most.


When it breaks

Failed: DID NOT RAISE ValueError The block finished without raising. Either the code really lost its check (the test did its job), or the call inside the block is not the one you think — wrong arguments that happen to be valid, or a function that returns an error value instead of raising.

AssertionError: Regex pattern did not match. followed by Did you mean to re.escape() the regex? The message is right but contains regex characters, usually (, ), ., [ or +. Wrap the text in re.escape(...). Without the hint, the message really is different: compare the Expected regex and Actual message lines character by character.

Failed: Invalid regex pattern provided to 'match': missing ), unterminated subpattern at position 0 The pattern is not even a valid regex — here, an opening ( with no closing one. Same fix: re.escape.

The test fails with some other exception, such as wallet.InsufficientFunds: cannot withdraw 150.00; balance is 100.00 An exception was raised, but not the one named or a subclass of it, so pytest.raises let it through. Decide which one the code is supposed to raise, and fix either the test or the code.

AssertionError: .value can only be used after the context manager exits You touched excinfo.value inside the with block. Move the line below the block.

TypeError: Expected a BaseException type, but got 'str' You wrote pytest.raises("ValueError"). Pass the class itself, without quotes.

FAILED ... - ExceptionGroup... from a pytest.raises(ValueError) block The code raised an ExceptionGroup that contains a ValueError. A group is not a ValueError; use pytest.raises(ExceptionGroup) with excinfo.group_contains(ValueError), or pytest.RaisesGroup(ValueError).

The test passes, but you are sure it should not Look for lines after the raising one inside the with block — they never run — and for pytest.raises(Exception), which accepts nearly anything. Then break the code on purpose and confirm the test goes red.