FreeCampus Python

Preserve Bugs and Explore General Rules

Turn minimal failures into named regressions, complete an honest red-green-refactor cycle, and use Hypothesis strategies and shrinking to test invariants beyond a few remembered examples.
python-foundations testing-python-programs regression-testing tdd property-based-testing hypothesis
Open in Colab
  • Level: Python Foundations
  • Estimated time: 4–5 hours
  • You will learn: Preserve a discovered failure, work through red–green–refactor without changing the contract mid-cycle, and use generated examples and shrinking to investigate a general rule.
  • Practice in: A local pytest project with Hypothesis installed; notebook users can run direct examples after installing the same dependency

Open this lesson in Google Colab

A hand-picked suite can be thoughtful and still miss a strange input. A user may type a tab where you expected a space, combine Unicode letters, repeat a separator, or provide a number near a boundary you did not remember. Once such a failure is found, a named regression prevents its quiet return. A property test can then explore the broader rule that the one example violated.

This lesson connects three practices:

They are complementary. TDD does not discover an unclear contract for you. Generated examples do not replace readable ordinary examples. A regression test does not prove all related inputs. Use each for the question it answers.

As you work, answer:

1. Reduce a failure before preserving it

Meteor Watch normalizes a station callsign for matching:

def normalize_callsign(text):
    """Trim outside whitespace, collapse inside whitespace, and uppercase."""
    return " ".join(text.strip().split(" ")).upper()

It works for one ordinary string:

assert normalize_callsign(" ridge seven ") == "RIDGE SEVEN"

But a user report contains a tab:

assert normalize_callsign("ridge\tseven") == "RIDGE SEVEN"

The result is "RIDGE\tSEVEN" because split(" ") recognizes only the literal space separator. The original report may have contained several records and a large traceback. Reduce it while preserving the failure:

  1. remove unrelated records;
  2. remove unrelated fields;
  3. shorten the station text;
  4. replace multiple whitespace forms one at a time; and
  5. stop when removing the tab makes the failure disappear.

"a\tb" is a smaller counterexample to the same rule:

def test_tab_between_callsign_words_is_collapsed_regression():
    assert normalize_callsign("a\tb") == "A B"

Run this test against the buggy implementation. It must fail because a tab remains, not because the package import or fixture setup is broken. Then repair the implementation with text.split(), whose no-argument form splits on runs of Unicode whitespace:

def normalize_callsign(text):
    """Trim outside whitespace, collapse inside whitespace, and uppercase."""
    return " ".join(text.split()).upper()

Keep the regression after the repair. Its descriptive name records why a tab matters even if a future refactor changes the implementation again.

Record enough history, not an incident report in the test

The name and a short comment may be enough:

def test_tab_between_callsign_words_is_collapsed_regression():
    # Tabs appeared in station exports from older field terminals.
    assert normalize_callsign("a\tb") == "A B"

Issue links can live in a comment when the history is useful and stable. Do not paste a long ticket into the test. The executable input, expected result, and reason should remain understandable if the link disappears.

2. Make red fail for the intended missing behavior

TDD is often summarized as:

  1. red: write a small test for one missing or changed behavior;
  2. green: write the smallest clear production change that satisfies it; and
  3. refactor: improve structure while all tests remain green.

The color is not enough. A test that is red because meteor_watch cannot import does not demonstrate missing behavior. Read the report and confirm the expected assertion or exception is responsible.

Meteor Watch needs a new "critical" result when wind is at least 110 kph. The current contract has only green, amber, and red. Start with the changed requirement table:

Wind Visibility Expected
109.9 10 red
110 10 critical

Write the boundary test first:

def test_wind_one_hundred_ten_begins_critical_alert():
    assert alert_level(110, 10) == "critical"

Run only that node. The useful red evidence is assert 'red' == 'critical'. If it passes before production changes, either behavior already exists or the test is not reaching the intended code.

Checkpoint: preserve a meaningful failure

3. Write the smallest clear green change

Add the critical branch before red:

def alert_level(wind_kph, visibility_km):
    if wind_kph < 0:
        raise ValueError("wind must be non-negative")
    if visibility_km < 0:
        raise ValueError("visibility must be non-negative")
    if wind_kph >= 110:
        return "critical"
    if wind_kph >= 70 or visibility_km < 1:
        return "red"
    if wind_kph >= 40 or visibility_km < 5:
        return "amber"
    return "green"

Run the new node, then the existing boundary table. The new test may be green while an old invariant test still allows only {"green", "amber", "red"}. That old failure is valuable: the public vocabulary changed and its test must be updated deliberately.

Do not implement speculative categories, configuration, or notification rules that no current contract asks for. “Smallest” does not mean cryptic; it means no unrequested design expansion.

4. Refactor only while the behavior stays green

The branch ordering is readable, but duplicated validation can move into a helper if several functions use it:

def validate_measurements(wind_kph, visibility_km):
    if wind_kph < 0:
        raise ValueError("wind must be non-negative")
    if visibility_km < 0:
        raise ValueError("visibility must be non-negative")


def alert_level(wind_kph, visibility_km):
    validate_measurements(wind_kph, visibility_km)
    if wind_kph >= 110:
        return "critical"
    if wind_kph >= 70 or visibility_km < 1:
        return "red"
    if wind_kph >= 40 or visibility_km < 5:
        return "amber"
    return "green"

Run the complete suite after the extraction. Refactoring means changing structure without intentionally changing public behavior. If you add another category during this step, you have mixed a feature with the refactor and made failures harder to interpret.

Each TDD phase has different evidence. Refactor returns to green after a structural change; it does not quietly add a second requirement.

flowchart LR
  A["One explicit requirement"] --> B["Red for the intended reason"]
  B --> C["Small clear implementation"]
  C --> D["Green focused and full suite"]
  D --> E["Behavior-preserving refactor"]
  E -->|"stay green"| D
  D --> F["Choose next requirement"]

When TDD helps—and when to explore first

TDD is useful when a small behavior can be stated before implementation:

  • a parser accepts a documented record;
  • a boundary changes at a known value;
  • a defect has a reproducible input;
  • a new function has a clear contract; or
  • an adapter must translate a known status.

Exploration may come first when the problem, library, or user need is unclear. Use a notebook or disposable spike to learn. Then discard or clean the spike, write the contract learned from it, and add durable tests. Pretending an uncertain experiment was known in advance turns TDD into ceremony.

5. State properties that cover more than remembered examples

Examples say what should happen for specific inputs. A property says what relationship should hold across a range of inputs.

Useful property shapes include:

  • idempotence: normalizing twice equals normalizing once;
  • round trip: encoding then decoding returns the original supported value;
  • bounds: a risk score always remains within its promised interval;
  • monotonicity: increasing wind while holding visibility fixed cannot reduce severity;
  • invariance: adding outside whitespace does not change a normalized callsign; and
  • model agreement: a production result equals a smaller independently trusted model.

For callsigns:

def test_normalize_callsign_is_idempotent_for_one_example():
    once = normalize_callsign("  Ridge\tSeven ")
    twice = normalize_callsign(once)

    assert twice == once

That is a property-shaped example. It communicates the invariant but explores only one string. Hypothesis can generate many strings from a strategy.

6. Install Hypothesis and generate valid text

Install Hypothesis in the practice environment:

python -m pip install hypothesis
python -c "import hypothesis; print(hypothesis.__version__)"

The repository declares Hypothesis as a development dependency, so a Poetry development installation includes it.

Import given and strategies, conventionally named st:

from hypothesis import given
from hypothesis import strategies as st


@given(st.text())
def test_normalize_callsign_is_idempotent(text):
    once = normalize_callsign(text)
    assert normalize_callsign(once) == once

For each generated text, Hypothesis calls the test and checks the assertion. The exact number of attempted examples is configurable and may include database replays or targeted cases; do not assert that a run always uses one fixed list.

The empty string is valid under st.text(). Decide whether it belongs to the normalizer contract. If empty callsigns should be rejected, test a validator with a strategy that represents its accepted domain rather than hiding all empty failures.

Constrain the strategy to the contract

If a station callsign accepts letters, digits, spaces, tabs, and hyphens with a maximum raw length of 40:

from hypothesis import strategies as st


callsign_text = st.text(
    alphabet=st.characters(
        categories=("L", "N"),
        include_characters=" \t-",
    ),
    min_size=1,
    max_size=40,
)

Use it:

from hypothesis import given


@given(callsign_text)
def test_normalization_is_idempotent_for_callsign_text(text):
    normalized = normalize_callsign(text)
    assert normalize_callsign(normalized) == normalized

Constraining generation directly is usually clearer and faster than generating arbitrary huge strings and rejecting most with .filter(...) or assume(...). Use assume when a relationship between generated values is genuinely hard to encode, and watch for health-check evidence that too many examples are discarded.

Checkpoint: choose properties and strategies

7. Read the counterexample and the shrink

Restore the buggy split(" ") normalizer and run the idempotence property. Idempotence may still pass because preserving a tab twice is stable. This is an important lesson: a true but weak property can miss the defect.

Add an invariant from the contract: normalized output contains no whitespace other than single spaces between non-space tokens.

from hypothesis import given


@given(callsign_text)
def test_normalized_callsign_has_only_single_spaces(text):
    normalized = normalize_callsign(text)
    assert "\t" not in normalized
    assert "  " not in normalized
    assert normalized == normalized.strip()

Hypothesis may report a falsifying example such as text='0\t0'. It tries to shrink a failure: find a simpler value that still makes the assertion false. A small counterexample helps reveal the rule and reduces debugging noise.

Representative output:

E   AssertionError: assert '\t' not in '0\t0'
E     '\t' is contained here:
E       0   0
E   Falsifying example: test_normalized_callsign_has_only_single_spaces(
E       text='0\t0',
E   )

Do not assume the first shown value is the only possible failure. Read the assertion, strategy domain, and minimized example. Reproduce it as a direct call, confirm split(" ") is responsible, and repair with split().

Generation searches the described domain. Shrinking keeps the failure while removing irrelevant complexity, producing a more useful debugging input.

flowchart LR
  A["Callsign strategy"] --> B["Generated examples"]
  B --> C{"Property holds?"}
  C -->|"yes"| B
  C -->|"no"| D["Failing example"]
  D --> E["Shrink while failure remains"]
  E --> F["Minimal counterexample"]
  F --> G["Diagnosis and regression"]

Preserve important generated failures

Hypothesis maintains an examples database that normally replays failures. Still add a named ordinary regression when the value represents a meaningful bug or communicates a boundary:

def test_tab_between_callsign_words_is_collapsed_regression():
    assert normalize_callsign("0\t0") == "0 0"

The property continues exploring variations. The named regression explains the specific obligation without depending on database state.

8. Build numeric strategies around real boundaries

Properties can complement exact alert examples. Define severity ordering:

SEVERITY = {
    "green": 0,
    "amber": 1,
    "red": 2,
    "critical": 3,
}

Increasing wind should not reduce severity while visibility is fixed:

from hypothesis import given
from hypothesis import strategies as st


@given(
    lower=st.floats(
        min_value=0,
        max_value=200,
        allow_nan=False,
        allow_infinity=False,
    ),
    increase=st.floats(
        min_value=0,
        max_value=200,
        allow_nan=False,
        allow_infinity=False,
    ),
    visibility=st.floats(
        min_value=0,
        max_value=20,
        allow_nan=False,
        allow_infinity=False,
    ),
)
def test_more_wind_never_reduces_alert_severity(
    lower,
    increase,
    visibility,
):
    higher = lower + increase
    lower_severity = SEVERITY[alert_level(lower, visibility)]
    higher_severity = SEVERITY[alert_level(higher, visibility)]

    assert higher_severity >= lower_severity

The strategy excludes NaN and infinity because the current numeric contract accepts finite non-negative measurements. If production should reject non-finite floats, add explicit examples and a separate property for that validation contract.

Properties can be wrong. If increasing wind legitimately changes another mode, the monotonic claim may overconstrain behavior. Review the property like any expected value.

9. Reproduce generated failures without hard-coding a random seed

Hypothesis reports a counterexample and saves it in its database. Rerun the focused node first. During investigation:

  1. copy the minimal value into a direct function call;
  2. confirm the property fails for the documented reason;
  3. simplify production or the property one change at a time;
  4. retain a named regression if the example communicates a real defect; and
  5. rerun the property and full suite.

Hypothesis offers settings, seeds, and explicit @example values, but do not pin a seed merely to turn property-based exploration into one fixed random list. Use @example(...) when a known case should always accompany generated cases; use a normal regression test when a descriptive node is more helpful.

Checkpoint: interpret shrinking and combine evidence

10. Shrink the strange callsign

Build a final lab around normalize_callsign and alert_level:

  1. begin with the buggy split(" ") normalizer;
  2. keep ordinary examples for trimming, casing, repeated spaces, and hyphens;
  3. add an idempotence property and explain why it does not expose every whitespace bug;
  4. define a bounded callsign strategy including letters, digits, spaces, tabs, and hyphens;
  5. add a property that normalized output has no tabs, no doubled spaces, and no outside whitespace;
  6. retain Hypothesis’s minimal counterexample and reduce it as a direct call;
  7. add a named tab regression, verify it is red, and fix production with split();
  8. add the critical wind requirement through a separate red–green–refactor cycle;
  9. add the monotonic-severity property for finite non-negative measurements; and
  10. run ordinary examples, regressions, properties, and the complete project from a clean process.

Your evidence log should distinguish the regression failure, TDD boundary failure, and property counterexample. Do not describe one as proof of the others.

Hint 1: make the whitespace property stricter than idempotence Assert that "\t" and " " are absent and that normalized == normalized.strip(). The buggy result can be stable under a second call while still containing forbidden whitespace.
Hint 2: constrain generation at the strategy Use st.text with an alphabet containing letters, digits, " ", "\t", and "-", plus min_size=1 and max_size=40. Avoid generating arbitrary objects and discarding nearly all of them.
Hint 3: keep the two red phases separate Finish and verify the normalization regression before starting the new critical-alert requirement. A refactor happens with the existing suite green; the next feature begins with a new intentional red node.
Show a complete normalization property group 
from hypothesis import given
from hypothesis import strategies as st

from meteor_watch.callsigns import normalize_callsign


callsign_text = st.text(
    alphabet=st.characters(
        categories=("L", "N"),
        include_characters=" \t-",
    ),
    min_size=1,
    max_size=40,
)


def test_callsign_trims_collapses_and_uppercases():
    assert normalize_callsign("  ridge   seven ") == "RIDGE SEVEN"


def test_tab_between_callsign_words_is_collapsed_regression():
    assert normalize_callsign("0\t0") == "0 0"


@given(callsign_text)
def test_normalization_is_idempotent(text):
    once = normalize_callsign(text)
    assert normalize_callsign(once) == once


@given(callsign_text)
def test_normalized_callsign_uses_single_spaces(text):
    normalized = normalize_callsign(text)
    assert "\t" not in normalized
    assert "  " not in normalized
    assert normalized == normalized.strip()

The repaired implementation is intentionally small:

def normalize_callsign(text):
    """Trim, collapse whitespace, and uppercase a station callsign."""
    return " ".join(text.split()).upper()
Add the separate critical threshold and monotonic-severity work from the lab contract before declaring the lesson complete.

Key points

  • Reduce a real failure to the smallest input that preserves its cause.
  • Keep meaningful failures as descriptive regression tests after the repair.
  • TDD red must fail for the intended missing behavior, not a broken environment.
  • Green is the smallest clear implementation of the current requirement; refactor changes structure while the complete suite remains green.
  • Explore first when the problem is unclear, then turn what you learned into a contract and durable tests.
  • Properties express relationships such as idempotence, round trips, bounds, and monotonicity across generated examples.
  • Strategies should describe the supported input domain directly.
  • Shrinking searches for a simpler value that still fails.
  • A passing weak property can miss a defect; review the property as carefully as an expected example.
  • Keep communicative examples, named regressions, and broader properties together when each supplies distinct evidence.

Continue learning

Back to top