Generators and Iterators for Big Data

When a dataset is too big to fit in memory, or when you only need values one at a time, generators are the answer. They produce values lazily, on demand, instead of building a giant list up front. This is one of the most practical advanced-Python skills for data and AI work, where files routinely have millions of rows.

What You'll Learn

• What "lazy evaluation" means and why it matters
• How to write a generator with yield
• The difference between an iterable and an iterator
• How to chain generators into a memory-light pipeline

The Problem: Loading Everything at Once

Reading a huge file into a list loads every row into memory at the same time:

def read_all(path):
    rows = []
    with open(path) as f:
        for line in f:
            rows.append(line.strip())
    return rows           # the entire file now lives in memory

If the file is 5 GB, this crashes. A generator processes one line at a time and never holds more than one in memory.

Writing a Generator with yield

Replace return with yield and the function becomes a generator. Each yield hands back one value and pauses until the next value is requested.

def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line.strip()

for line in read_lines("big.log"):
    process(line)         # one line in memory at a time

Calling read_lines(...) does not run the body immediately. It returns a generator object. The code runs a little at a time, each time the loop asks for the next value. That is lazy evaluation.

Follow One Loop, Step by Step

Lazy execution is easier to trust once you can trace it. For the loop above:

1. read_lines("big.log") runs none of the function body. It only builds a generator object that remembers where to start.
2. The for loop asks the generator for its first value. Now the body starts running: the file opens, the first line is read, and yield hands it out. The function then freezes right there, file still open, variables intact.
3. process(line) runs on that first line.
4. The loop asks for the next value. The function wakes up exactly where it froze, reads line two, yields it, and freezes again.
5. When the file runs out, the generator signals that it is finished, the with block closes the file, and the loop ends.

Steps 2 and 4 are the magic. A normal function forgets everything when it returns; a generator pauses and remembers. That pause is why only one line ever needs to exist in memory.

Under the hood, the loop is calling next() on the generator each time. You can do it by hand, which makes the pausing visible:

gen = read_lines("big.log")
first = next(gen)    # runs the body up to the first yield, then pauses
second = next(gen)   # resumes exactly where it paused, yields line two

Iterable vs Iterator

This distinction clears up a lot of confusion:

A key gotcha: a generator is consumed once. After you loop over it, it is empty. If you need the values twice, either re-create the generator or collect them into a list with list(gen).

Building a Pipeline

The real power is chaining generators. Each stage stays lazy, so even a multi-step transform over a massive file uses almost no memory.

def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line.strip()

def only_errors(lines):
    for line in lines:
        if "ERROR" in line:
            yield line

def to_upper(lines):
    for line in lines:
        yield line.upper()

# Nothing runs until we consume the final loop:
pipeline = to_upper(only_errors(read_lines("app.log")))
for line in pipeline:
    print(line)

To see why the pipeline stays lazy, follow one line through it. The final for loop asks to_upper for a value. to_upper needs a line to work on, so it asks only_errors, which asks read_lines, which reads one line from the file. If that line contains "ERROR" it passes through, gets uppercased, and reaches your print; if not, only_errors quietly asks for the next line until one qualifies. Then the whole chain repeats. The three stages take turns on a single line at a time, and at no point does a full list of lines exist anywhere.

Generator Expressions Recap

You met these last lesson. The parenthesized form is just an inline generator:

squares = (x * x for x in range(1_000_000))   # lazy, no big list
total = sum(squares)

Use the def ... yield form when the logic needs multiple lines or its own name; use the expression form for short one-liners.

Try It

Run this. The countdown generator yields values lazily. Then add a second generator that doubles each value before printing.

Key Takeaways

• Generators produce values lazily with yield, holding one value at a time instead of a whole list.
• Calling a generator function returns a generator object; the body runs only as values are requested.
• An iterable can be looped repeatedly; an iterator (including a generator) is consumed once.
• Chaining generators builds memory-light data pipelines over huge files.
• Use def ... yield for multi-line logic and a generator expression (...) for short cases.