LATEST 🚀 Welcome to Upgradix by Shrash Studio – Your destination for free technology learning and career growth.
🕘 Updated:
Python

Python - Iterators & Generators

Iterators and generators allow Python programs to process millions of items one-by-one without loading the entire dataset into RAM, preventing memory crashes.

Python Iterators & Generators – Efficient Memory Management for Large Datasets

Quick Answer: Iterators and generators allow Python programs to process millions of items one-by-one without loading the entire dataset into RAM, preventing memory crashes.

Imagine Priya manages a wholesale grocery distribution center in Bengaluru processing 10 million transaction records every single day.

Loading all 10 million orders into a standard Python list at once consumes gigabytes of memory and crashes the server.

How would you process this massive file line-by-line without running out of RAM?

1. Why Do We Need Iterators and Generators?

The problem with loading lists into memory

Standard lists store all their items in computer memory at the same time.

Example 1: Creating a large list in memory

This approach allocates memory for all items instantly, risking memory exhaustion.

# Loading 10 million numbers into a list
large_list = [x for x in range(10_000_000)]
print("List created in memory")

Expected output

List created in memory

This code runs on small ranges, but with massive data it causes problems.

We need a lazy evaluation mechanism that yields items on demand.

The solution: iterators and generators

Iterator = an object that lets us traverse a container and yields items one at a time.

Generator = a special function that uses the yield keyword to return items lazily.

Memory Comparison: Lists vs Generators in Priya's Warehouse
How lazy evaluation saves system RAM
Data Source (10M Records) Raw transaction log file
Lazy Iteration
Record 1 Loaded & Processed
Record 2 Loaded & Processed
Next Item... Yielded on demand
Source Data Active Step Single Item in RAM

Figure 1: Generators process items one-by-one, keeping memory usage flat.

2. Building Custom Iterators with __iter__ and __next__

Method = a block of code inside a class that performs an action.

To build a custom iterator in Python, a class must implement two special dunder methods: __iter__() and __next__().

Example 2: A custom transaction iterator

This code defines a custom iterator that counts up to a given limit.

class TransactionBatch:
    def __init__(self, max_id):
        self.current = 1
        self.max_id = max_id

    def __iter__(self):
        return self

    def __next__(self):
        if self.current > self.max_id:
            raise StopIteration
        val = $"Txn-{self.current}"
        self.current += 1
        return val

if __name__ == "__main__":
    batch = TransactionBatch(3)
    for txn in batch:
        print(txn)

Expected output

Txn-1
Txn-2
Txn-3

Notice how StopIteration signals the end of the iteration cycle.

3. Writing Memory-Efficient Generators Using yield

Keyword = a reserved word in Python with special meaning.

The yield keyword pauses function execution, saves its state, and returns a value to the caller.

Example 3: A generator function for transaction logs

This generator yields items one at a time without storing them in a list.

def generate_transactions(count):
    for i in range(1, count + 1):
        yield $"Order-{i}"

if __name__ == "__main__":
    gen = generate_transactions(3)
    print(next(gen))
    print(next(gen))
    print(next(gen))

Expected output

Order-1
Order-2
Order-3

Each call to next() resumes the generator right after the yield statement.

4. Generator Expressions vs List Comprehensions

Generator expression = a compact generator written using parentheses instead of square brackets.

Example 4: Comparing list comprehension and generator expression

Parentheses create a lazy generator, whereas square brackets create an eager list.

# Eager list comprehension (allocates memory instantly)
list_comp = [x * 2 for x in range(5)]

# Lazy generator expression (evaluates on demand)
gen_expr = (x * 2 for x in range(5))

print("List:", list_comp)
print("Generator Object:", gen_expr)
print("Generator Value:", next(gen_expr))

Expected output

List: [0, 2, 4, 6, 8]
Generator Object:  at 0x...>
Generator Value: 0
Comparing lists and generators for dataset processing
Feature Lists Generators
Memory Consumption High (stores all items in RAM) Minimal (stores one item at a time)
Evaluation Eager (computed immediately) Lazy (computed on demand)
Reusability Can be iterated multiple times Exhausted after one iteration

The important terms to remember

Design tip: Use generator expressions when processing large data streams or files to keep memory usage constant.

5. Common Mistakes to Avoid

1. Attempting to reuse an exhausted generator

Iterating over a generator a second time yields no items because its state is already consumed.

Better approach: Recreate the generator function or store items in a list if multiple passes are required.

6. Interview Questions & Answers

Question 1: What is the difference between return and yield in Python?

What the interviewer expects: A clear explanation of state persistence and function termination.

Strong answer: "A return statement terminates a function completely and sends back a single result. A yield statement pauses the function, saves its local state, and returns an intermediate value, allowing execution to resume later."

What interviewers expect at different levels

7. Practice Exercises

Beginner: Fibonacci generator

Write a generator function that yields the first 10 Fibonacci numbers.

Expected result: Prints Fibonacci numbers sequentially without storing a massive list.

Intermediate: File line reader

Create a generator that reads a large text file line-by-line, stripping whitespace.

Hint: Use with open(...) as f: combined with a for line in f: loop.

8. Frequently Asked Questions (FAQ)

Can you find the length of a generator using len()?

No. Generators do not support len() because their total item count is not computed in advance.

Are generators faster than lists?

Generators save memory, but execution speed depends on whether items need to be recalculated or reused.

9. Next Steps

Now that you have mastered iterators and generators, you are ready to explore decorators, closures, and error handling in the next article.

  1. Instant memory allocation consumes all available RAM.
  2. Garbage collection struggles with large monolithic objects.
  3. Processing cannot start until the entire list is built in memory.
    • Iterator = an object implementing __iter__() and __next__().
    • Generator = a function containing the yield keyword.
    • Lazy Evaluation = computing values only when requested.
    • Junior developer: Knows how to use generator functions with yield.
    • Senior developer: Understands iterator protocols, memory efficiency, and generator exhaustion.
    • Technical architect: Designs streaming data pipelines for high-throughput enterprise systems.

Comments & Ratings (0)

Log in or create a student account to comment and rate this article.

Be the first to comment on this article.