FreeCampus Python

Keep Small Examples Honest with Doctest

Run and diagnose deterministic transcript examples with doctest, apply comparison options carefully, and move complex behavior to clearer pytest tests.
python-foundations documentation-publishing doctest executable-examples
Open in Colab
  • Level: Beginner
  • Estimated time: 3–4 hours
  • You will learn: Make small public examples executable, read their exact diffs, control unstable text honestly, and recognize when pytest communicates the contract better.
  • Practice in: A local module and terminal, with Colab or JupyterLab for isolated experiments

Open this lesson in Google Colab

An example makes a promise twice: the code should be valid for the public interface, and the displayed result should match what a reader will observe. Doctest can turn a small Python transcript into an executable comparison. It is valuable when the transcript is already good documentation; it is not a reason to squeeze every behavior into a docstring.

This lesson asks:

  1. How does doctest distinguish input, continuation, output, and prose?
  2. Which command proves that examples were discovered rather than silently skipped?
  3. How do you read a doctest diff without immediately changing expected output?
  4. When do ellipsis or whitespace options express a real tolerance, and when do they hide a broken promise?
  5. Which examples belong in pytest instead?

1. A transcript has exact roles

Add a small deterministic example to the public function:

from dataclasses import dataclass


@dataclass(frozen=True)
class ExhibitScore:
    """Store one exhibit ranking result."""

    name: str
    score: int


def rank_exhibit(name: str, votes: int, minutes_open: int) -> ExhibitScore:
    """Return the score for one named exhibit.

    >>> rank_exhibit("Moon Dial", votes=4, minutes_open=75)
    ExhibitScore(name='Moon Dial', score=44)
    >>> rank_exhibit("Sun Clock", votes=1, minutes_open=29)
    ExhibitScore(name='Sun Clock', score=10)
    """
    if votes < 0 or minutes_open < 0:
        raise ValueError("votes and minutes_open must be non-negative")
    return ExhibitScore(name, votes * 10 + minutes_open // 30 * 2)

Read each line by role:

  • >>> begins an input statement as it would appear in the interactive interpreter.
  • ... continues an input statement; it is not decorative indentation.
  • Lines without a prompt after an input are expected text output or representation.
  • Prose outside the transcript remains documentation and is not compared as output.

Doctest uses repr() for a value left as an expression. That is why strings have quotes and the dataclass shows its constructor-like representation. print() would compare the displayed string instead.

A multi-line input uses the continuation prompt:

def ranked_names(records: list[ExhibitScore]) -> list[str]:
    """Return names ordered by descending score, then by name.

    >>> records = [
    ...     ExhibitScore("Moon Dial", 44),
    ...     ExhibitScore("Sun Clock", 10),
    ... ]
    >>> ranked_names(records)
    ['Moon Dial', 'Sun Clock']
    """
    ordered = sorted(records, key=lambda item: (-item.score, item.name))
    return [item.name for item in ordered]

The continuation indentation must be valid Python after the prompt is removed. Keep setup short enough that the reader can still see the public behavior.

Doctest discovers a transcript, executes its input, renders the observed value, and compares text.

flowchart LR
  A[Docstring or text file] --> B[Discover examples]
  B --> C[Execute input]
  C --> D[Capture output or repr]
  D --> E{Expected text matches?}
  E -->|Yes| F[Passing example]
  E -->|No| G[Failure diff]

2. Prove that the examples were collected

From the project root, run a module directly:

PYTHONPATH=src python -m doctest -v src/museum_quest/ranking.py

A successful summary for the two rank_exhibit examples includes:

2 tests in ranking.rank_exhibit
2 tests in 8 items.
2 passed and 0 failed.
Test passed.

Exact item counts can change as the module gains documented objects. The important evidence is the intended number of tests and zero failures.

Without -v, a successful run can produce no output. Silence is only useful if you already know collection occurred. Use verbose output while authoring or assert the result through the library API:

import doctest
import museum_quest.ranking

result = doctest.testmod(museum_quest.ranking, verbose=False)
assert result.attempted == 2
assert result.failed == 0

Checking both fields matters. failed == 0 also describes a module with zero discovered examples. The attempted count protects against a misspelled prompt, wrong module, or uncollected file.

Pytest can collect docstrings along with ordinary tests:

pytest --doctest-modules src tests -q

Or commit the policy in the sample project’s configuration:

[tool.pytest.ini_options]
pythonpath = ["src"]
addopts = "--doctest-modules"

Do not enable collection and assume every prose-looking prompt is suitable. Run and review the suite, then keep the configuration versioned with the project.

Checkpoint: collect real attempts

3. Read the diff before repairing anything

Change the expected Moon Dial score to 46 while leaving the implementation and inputs unchanged. A failure resembles:

Failed example:
    rank_exhibit("Moon Dial", votes=4, minutes_open=75)
Expected:
    ExhibitScore(name='Moon Dial', score=46)
Got:
    ExhibitScore(name='Moon Dial', score=44)

Use all four pieces:

  1. Input: votes 4 and open time 75.
  2. Expected: score 46.
  3. Got: score 44.
  4. Policy: four votes add 40; two complete 30-minute periods add 4.

The observed 44 matches the documented policy. Repair the stale expected output to 44. If product requirements actually changed to count partial periods, then change the contract, implementation, normal tests, doctest, explanation, and release communication together. Do not pick whichever edit makes one command green fastest.

Exceptions are observable results

Document an anticipated invalid input with the exception’s final line:

def rank_exhibit(name: str, votes: int, minutes_open: int) -> ExhibitScore:
    """Return the score for one named exhibit.

    >>> rank_exhibit("Moon Dial", votes=-1, minutes_open=75)
    Traceback (most recent call last):
    ...
    ValueError: votes and minutes_open must be non-negative
    """
    if votes < 0 or minutes_open < 0:
        raise ValueError("votes and minutes_open must be non-negative")
    return ExhibitScore(name, votes * 10 + minutes_open // 30 * 2)

Doctest treats the traceback header and final exception detail specially. The middle ... stands for stack frames. The exception type and message remain reader-visible contract. If only incidental detail varies, use the narrowest supported comparison rather than replacing the entire expected failure with an ellipsis.

Blank output needs an explicit marker

A literal blank line ends expected output. When a printed value includes a blank line, doctest uses <BLANKLINE>:

def score_card(result: ExhibitScore) -> str:
    """Return a two-part score card.

    >>> print(score_card(ExhibitScore("Moon Dial", 44)))
    Moon Dial
    <BLANKLINE>
    Score: 44
    """
    return f"{result.name}\n\nScore: {result.score}"

Use this only when the blank line helps the example. Large formatting snapshots are often clearer in an ordinary test with a named expected string.

4. Use comparison options as precise contracts

Doctest normally compares text exactly. Options can describe harmless variation, but they can also make an example meaningless.

Normalize layout when layout is not the promise

def score_words() -> str:
    """Return the score column headings.

    >>> print(score_words())  # doctest: +NORMALIZE_WHITESPACE
    Exhibit    Votes    Score
    """
    return "Exhibit\tVotes\tScore"

NORMALIZE_WHITESPACE accepts equivalent runs of whitespace. This is reasonable when the example teaches the three headings, not exact tab layout. It would be wrong for a fixed-width export whose spacing is the public format.

Ellipsis should cover one unstable fragment

def describe_session(session_id: str) -> str:
    """Return a short session description.

    >>> describe_session("museum-42")  # doctest: +ELLIPSIS
    'session museum-42: ... ready'
    """
    return f"session {session_id}: temporary workspace ready"

The stable session ID and final state remain checked. This is too broad:

'...'

It accepts almost any non-empty representation and teaches the reader nothing. Prefer redesigning the example so unstable timestamps, paths, addresses, random values, or unordered output are not central.

Sort unordered results for the example

Instead of tolerating arbitrary set order:

def exhibit_tags() -> set[str]:
    """Return the available exhibit tags.

    >>> sorted(exhibit_tags())
    ['astronomy', 'clockwork', 'light']
    """
    return {"light", "clockwork", "astronomy"}

The public meaning is membership, so sorting only the displayed example creates a stable teaching view without changing the returned type.

Checkpoint: choose a truthful comparison

5. Use a text file when the narrative is larger than one object

A tutorial-like transcript can live in docs/examples.txt:

Rank two exhibits and compare their public results.

    >>> from museum_quest import rank_exhibit
    >>> moon = rank_exhibit("Moon Dial", 4, 75)
    >>> sun = rank_exhibit("Sun Clock", 1, 29)
    >>> moon.score > sun.score
    True

Run it directly:

PYTHONPATH=src python -m doctest -v docs/examples.txt

A text file can connect several public objects without making one function’s docstring carry a long tutorial. It still needs small, deterministic setup and clear prose.

Do not confuse the source with the published QMD site automatically. If Quarto renders a different code block than the tested text file, those are two copies that can drift. Lesson 5 asks which source owns the example and how the rendered page receives it.

6. Choose pytest when structure communicates better

Doctest is a poor fit when an example needs:

  • a temporary directory or several files;
  • complex setup/cleanup;
  • repeated boundary inputs;
  • approximate numeric comparisons;
  • structured assertions over many fields;
  • test doubles or injected dependencies;
  • intentionally unstable output; or
  • a failure message clearer than a long textual diff.

This boundary table belongs in pytest:

import pytest

from museum_quest import ExhibitScore, rank_exhibit


@pytest.mark.parametrize(
    ("minutes_open", "expected_score"),
    [(0, 40), (29, 40), (30, 42), (59, 42), (60, 44)],
)
def test_complete_period_boundaries(
    minutes_open: int,
    expected_score: int,
) -> None:
    assert rank_exhibit("Moon Dial", 4, minutes_open) == ExhibitScore(
        "Moon Dial",
        expected_score,
    )

A docstring can keep one ordinary example and perhaps one important failure. The parametrized test protects the detailed period boundaries without overwhelming a reader who only needs the public shape.

Avoid hidden state between examples

This transcript depends on earlier mutation:

def add_vote(scores: list[int]) -> int:
    """Append one vote and return the count.

    >>> values = []
    >>> add_vote(values)
    1
    >>> add_vote(values)
    2
    """
    scores.append(1)
    return len(scores)

The sequence is valid as one doctest, but copying the second call alone produces a different result. If independence matters to understanding, repeat the setup or use a function example that creates no hidden evolving state.

Notebook state creates a similar risk. Restart and run from the top. A locally passing cell after an old import does not prove the committed module or docstring works.

7. Diagnose discovery, comparison, and design failures separately

Use the earliest category supported by evidence:

Evidence Likely category Next action
0 tests in 7 items Discovery Inspect prompts, module path, and collection command
NameError on a public name Setup/import Check installation, imports, and example context
Expected/Got differ in one value Contract or stale output Trace inputs and intended behavior before editing
Same facts, different spacing Text representation Decide whether layout is public; use a narrow option if not
Timestamp/address changes Nondeterminism Remove unstable value or expose a stable public view
Ten setup lines obscure one assertion Wrong test form Move structured case to pytest

Checkpoint: decide between doctest and pytest

8. Lab: run a mixed doctest clinic

Create src/museum_quest/examples.py containing six documented objects:

  1. a stable rank_exhibit ordinary example with stale score 46;
  2. a negative-votes example with the wrong exception message;
  3. a tag-set example that assumes set order;
  4. a session description whose expectation is only '...';
  5. a score-card example missing <BLANKLINE>; and
  6. a five-row time-boundary transcript that would be clearer in pytest.

Work one failure at a time:

PYTHONPATH=src python -m doctest -v src/museum_quest/examples.py

For each failure, record:

Object Attempted input Expected Got Contract decision Smallest repair

Then:

  • assert the intended attempted count and zero failures with testmod;
  • move the boundary table to a parametrized pytest test;
  • keep one readable ordinary case and one useful failure in public docs;
  • run pytest with doctest collection from a clean process;
  • explain why each option directive is narrower than changing everything to ellipsis.
Hint: classify before changing text

Mark each issue as discovery, setup, intended-behavior mismatch, unstable representation, or unsuitable test structure. The category narrows the repair.

Reveal representative stable examples 
def exhibit_tags() -> set[str]:
    """Return the available exhibit tags.

    >>> sorted(exhibit_tags())
    ['astronomy', 'clockwork', 'light']
    """
    return {"light", "clockwork", "astronomy"}


def score_card(result: ExhibitScore) -> str:
    """Return a two-part score card.

    >>> print(score_card(ExhibitScore("Moon Dial", 44)))
    Moon Dial
    <BLANKLINE>
    Score: 44
    """
    return f"{result.name}\n\nScore: {result.score}"

The time-boundary cases remain more readable in the parametrized pytest example from section 6.

Key points

  • Doctest is strongest when a small deterministic transcript is already useful documentation.
  • Check attempted and failed counts so an empty collection cannot masquerade as success.
  • Read input, expected, got, and intended contract before editing a failure.
  • Use whitespace, ellipsis, and exception options only for narrow variation that is not part of the public promise.
  • Prefer stable public views over timestamps, addresses, random values, or unordered representations.
  • Move complex setup and boundary tables to pytest when structured assertions communicate better.

References

Next: Review Documentation Like Code

Back to top