FreeCampus Python

Producing Values One at a Time

Trace iterable and iterator state, build finite generators that pause and resume, and compose lazy pipelines whose consumers can stop early.
python-foundations functions-call-behavior iterators generators
Open in Colab
  • Level: Python Foundations · Unit 5
  • Estimated time: 4–5.5 hours
  • You will learn: Distinguish reusable iterables from stateful iterators, predict partial consumption and exhaustion, and write finite generator functions, expressions, delegation, and lazy pipelines.
  • Practice in: Google Colab, JupyterLab, or a local editor

Open this lesson in Google Colab

1. Sometimes the caller needs only the next useful reading

Suppose a remote sensor collected many values, but an alert needs only the first three readings above a threshold. Building every transformed result first can do unnecessary work. An iterator lets the consumer ask for one value at a time and stop early.

This lesson separates several related terms:

  • an iterable can provide an iterator;
  • an iterator remembers its current consumption position;
  • a generator function defines paused production with yield;
  • a generator object is the stateful iterator returned by calling that function; and
  • a generator expression creates a compact lazy transformation.

The important question is not merely “is this lazy?” It is: which object owns the current position, when does code run, and what happens after exhaustion?

2. A list is iterable; iter creates an iterator

readings = [12, 18, 25]
cursor = iter(readings)

assert iter(cursor) is cursor
assert next(cursor) == 12
assert next(cursor) == 18
assert next(cursor) == 25

The list stores all three values and can create iterators. cursor is one iterator with a current position. Each next(cursor) returns the next item and advances that position.

A for loop performs this protocol for you: it gets an iterator, requests next values, and stops normally when the iterator signals that no value remains.

3. Exhaustion is persistent

After the third item, another plain next(cursor) raises StopIteration. Keep that failing call commented in the normal lesson run:

# next(cursor)

next also accepts a default result:

assert next(cursor, "no reading") == "no reading"
assert next(cursor, "still empty") == "still empty"

Exhaustion does not rewind the iterator. Once consumed, it stays consumed. A for loop catches the stop signal internally; ordinary application code rarely calls StopIteration handling directly.

4. Collections restart; iterator objects continue

Two loops over the list each start from a newly created iterator:

readings = [12, 18, 25]
assert [value for value in readings] == [12, 18, 25]
assert [value for value in readings] == [12, 18, 25]

Two loops over the same iterator share its progress:

cursor = iter(readings)
first = next(cursor)
remaining = [value for value in cursor]
after_exhaustion = [value for value in cursor]

assert first == 12
assert remaining == [18, 25]
assert after_exhaustion == []

Create independent positions by calling iter on the collection twice:

left = iter(readings)
right = iter(readings)

assert next(left) == 12
assert next(left) == 18
assert next(right) == 12

The list is shared source data, but each iterator remembers its own position.

Checkpoint: iterator state

5. Calling a generator function does not run its body yet

A function containing yield is a generator function:

def countdown(start):
    print("generator body started")
    while start > 0:
        yield start
        start -= 1


numbers = countdown(3)

The assignment creates a generator object. It prints nothing yet. The first request begins execution:

assert next(numbers) == 3

At that point Python:

  1. creates local start = 3 for this generator call;
  2. prints the start message;
  3. reaches yield start;
  4. produces 3; and
  5. suspends the frame before start -= 1.

The call and first execution are separate moments. This timing is one of the most important differences between an ordinary function call and a generator function call.

6. yield pauses without discarding local state

The next request resumes after the earlier yield:

assert next(numbers) == 2
assert next(numbers) == 1
assert next(numbers, "done") == "done"

Before yielding 2, the resumed body executes start -= 1, changing the remembered local value from 3 to 2. A later request repeats that sequence.

One generator object keeps its suspended frame between requests.

stateDiagram-v2
  [*] --> Created
  Created --> Running: first next request
  Running --> Suspended: yield one value
  Suspended --> Running: next request resumes
  Running --> Suspended: yield another value
  Running --> Exhausted: return or body ends
  Exhausted --> [*]

A normal function loses its ordinary frame after returning. A generator frame is deliberately retained while suspended, including local values and the next instruction position.

7. return ends production; yield produces a stream item

def limited_steps(limit):
    step = 1
    while step <= limit:
        yield step
        step += 1
    return


assert list(limited_steps(3)) == [1, 2, 3]
assert list(limited_steps(0)) == []

Each yield contributes an item visible to ordinary iteration. Reaching return or the end signals exhaustion. Python generators can attach a value to StopIteration, but ordinary for and list(...) consumption does not treat that value as another yielded item. Foundations code should return only to stop unless a specialized protocol explicitly needs more.

Make finite boundaries clear. This course avoids an unbounded generator unless the consumer has an equally visible limit.

Checkpoint: generator timing

8. yield from delegates to another iterable

Without delegation, a generator can relay items with a nested loop:

def flatten_groups(groups):
    for group in groups:
        for item in group:
            yield item

yield from expresses the relay directly:

def flatten_groups(groups):
    for group in groups:
        yield from group


groups = [["red", "blue"], [], ["silver"]]
assert list(flatten_groups(groups)) == ["red", "blue", "silver"]

When a group is empty, delegation produces no values and continues. In the recursive leaf_names preview, yield from leaf_names(child) delegates to a child generator rather than materializing its entire result first.

9. Generator expressions are lazy comprehensions

Square brackets create a list immediately. Parentheses create a generator expression:

readings = [12, 18, 25, 30]

eager = [value * 2 for value in readings]
lazy = (value * 2 for value in readings)

assert eager == [24, 36, 50, 60]
assert next(lazy) == 24
assert list(lazy) == [36, 50, 60]

The partial next consumed the first lazy result. Converting the same generator to a list continues from its current position; it does not repeat 24.

Use a list when the complete result will be reused, indexed, or measured. Use an iterator when one-pass incremental consumption is the desired contract.

10. Consumers can stop a lazy source early

any and all short-circuit. They stop as soon as the answer is known:

def observed(values):
    for value in values:
        print("checked", value)
        yield value


has_large = any(value > 20 for value in observed([4, 25, 100]))
assert has_large is True

The generator does not need to check 100 because 25 > 20 already makes any(...) true. Similar early stopping happens with next and bounded tools such as itertools.islice.

Lazy execution changes timing: errors and side effects inside production occur when a consumer requests the relevant value, not necessarily when the generator object is created. Prefer generators that yield data without surprising effects; the print above exists only to reveal timing.

11. Build a pipeline from small lazy stages

def above_threshold(values, threshold):
    for value in values:
        if value >= threshold:
            yield value


def to_fahrenheit(celsius_values):
    for value in celsius_values:
        yield value * 9 / 5 + 32


source = [12, 18, 25, 30]
usable = above_threshold(source, 18)
converted = to_fahrenheit(usable)

assert next(converted) == 64.4
assert list(converted) == [77.0, 86.0]

The first request flows through both stages only until one output is available. The pipeline resumes from its shared positions later. A stage should document that its input is consumed; passing the same iterator elsewhere means those consumers share progress.

12. Shared consumers can surprise each other

source = iter([10, 20, 30, 40])

assert next(source) == 10
assert list(source) == [20, 30, 40]
assert next(source, None) is None

If two functions receive source, whichever consumes first determines what the other sees. To repeat work, keep a restartable collection or call a generator factory again:

def reading_source():
    yield from [10, 20, 30, 40]


assert list(reading_source()) == [10, 20, 30, 40]
assert list(reading_source()) == [10, 20, 30, 40]

Each function call returns a new generator object with new state.

Checkpoint: pipelines and exhaustion

13. Lab: stream usable sensor readings

Build a finite source and two lazy stages:

def sensor_readings(values):
    """Yield each supplied sensor value in order."""
    raise NotImplementedError


def valid_readings(values, *, minimum=0, maximum=100):
    """Yield values inside the inclusive supported range."""
    raise NotImplementedError


def changes(values):
    """Yield each difference from the previous value."""
    raise NotImplementedError

Use progressive consumption checks:

source_values = [-5, 10, 14, 200, 20, 20]
stream = sensor_readings(source_values)

assert iter(stream) is stream
assert next(stream) == -5
assert next(stream) == 10
assert list(stream) == [14, 200, 20, 20]
assert next(stream, "exhausted") == "exhausted"

fresh = sensor_readings(source_values)
usable = valid_readings(fresh, minimum=0, maximum=100)
assert next(usable) == 10
assert list(usable) == [14, 20, 20]
assert list(usable) == []

pipeline = changes(valid_readings(sensor_readings(source_values)))
assert next(pipeline) == 4
assert list(pipeline) == [6, 0]
assert source_values == [-5, 10, 14, 200, 20, 20]
assert list(valid_readings([], minimum=0, maximum=100)) == []

changes needs one previous valid value before it can yield a difference. An empty or one-item input therefore yields no differences. Do not build hidden intermediate lists in these functions.

Hint: keep only the state each stage needs

The source yields each input directly. The validator yields only values inside both boundaries. In changes, obtain the first item with next(iterator, sentinel), then loop over the remaining items, yield current - previous, and update previous.

Show one complete solution after attempting the lab 
def sensor_readings(values):
    """Yield each supplied sensor value in order."""
    yield from values


def valid_readings(values, *, minimum=0, maximum=100):
    """Yield values inside the inclusive supported range."""
    for value in values:
        if minimum <= value <= maximum:
            yield value


def changes(values):
    """Yield each difference from the previous value."""
    iterator = iter(values)
    missing = object()
    previous = next(iterator, missing)
    if previous is missing:
        return

    for current in iterator:
        yield current - previous
        previous = current

After all assertions pass, predict list(changes([7])) and list(changes([7, 3, 9])) before running them. Explain why the source list stays unchanged while iterator positions advance.

14. Explain the state owner

For each object in the lab, state:

  1. when its generator body first runs;
  2. which local values remain during suspension;
  3. which consumer requests the next value;
  4. whether it can restart; and
  5. what proves exhaustion.

Restart the notebook and run the full cell sequence once. Stateful examples are easy to misread when earlier exploratory calls already consumed values.

Key points

TipKey points
  • An iterable can provide iterators; an iterator owns a one-pass consumption position.
  • Collections commonly create fresh iterators, while reusing one iterator continues from its current position.
  • Calling a generator function creates a generator object without running its body.
  • The first request starts the body; yield produces a value and preserves the suspended frame for a later request.
  • Reaching return or the end exhausts a generator permanently.
  • Generator expressions and pipelines perform work as consumers request it, so short-circuit consumers can stop early.
  • Use a list for reusable materialized results and a generator when one-pass incremental production is the intended contract.

References

Back to top