Python Iterators & Generators – Efficient Memory Management for Large Datasets
Quick Answer: Iterators and generators allow Python programs to process millions of items one-by-one without loading the entire dataset into RAM, preventing memory crashes.
Imagine Priya manages a wholesale grocery distribution center in Bengaluru processing 10 million transaction records every single day.
Loading all 10 million orders into a standard Python list at once consumes gigabytes of memory and crashes the server.
How would you process this massive file line-by-line without running out of RAM?
1. Why Do We Need Iterators and Generators?
The problem with loading lists into memory
Standard lists store all their items in computer memory at the same time.
Example 1: Creating a large list in memory
This approach allocates memory for all items instantly, risking memory exhaustion.
# Loading 10 million numbers into a list
large_list = [x for x in range (10_000_000 )]
print("List created in memory")
Expected output
List created in memory
This code runs on small ranges, but with massive data it causes problems.
We need a lazy evaluation mechanism that yields items on demand.
The solution: iterators and generators
Iterator = an object that lets us traverse a container and yields items one at a time.
Generator = a special function that uses the yield keyword to return items lazily.
Figure 1: Generators process items one-by-one, keeping memory usage flat.
2. Building Custom Iterators with __iter__ and __next__
Method = a block of code inside a class that performs an action.
To build a custom iterator in Python, a class must implement two special dunder methods: __iter__() and __next__().
Example 2: A custom transaction iterator
This code defines a custom iterator that counts up to a given limit.
class TransactionBatch :
def __init__ (self , max_id):
self .current = 1
self .max_id = max_id
def __iter__ (self ):
return self
def __next__ (self ):
if self .current > self .max_id:
raise StopIteration
val = $"Txn-{self.current}"
self .current += 1
return val
if __name__ == "__main__":
batch = TransactionBatch (3 )
for txn in batch:
print(txn)
Expected output
Txn-1 Txn-2 Txn-3
Notice how StopIteration signals the end of the iteration cycle.
3. Writing Memory-Efficient Generators Using yield
Keyword = a reserved word in Python with special meaning.
The yield keyword pauses function execution, saves its state, and returns a value to the caller.
Example 3: A generator function for transaction logs
This generator yields items one at a time without storing them in a list.
def generate_transactions (count):
for i in range (1 , count + 1 ):
yield $"Order-{i}"
if __name__ == "__main__":
gen = generate_transactions (3 )
print(next (gen))
print(next (gen))
print(next (gen))
Expected output
Order-1 Order-2 Order-3
Each call to next() resumes the generator right after the yield statement.
4. Generator Expressions vs List Comprehensions
Generator expression = a compact generator written using parentheses instead of square brackets.
Example 4: Comparing list comprehension and generator expression
Parentheses create a lazy generator, whereas square brackets create an eager list.
# Eager list comprehension (allocates memory instantly)
list_comp = [x * 2 for x in range (5 )]
# Lazy generator expression (evaluates on demand)
gen_expr = (x * 2 for x in range (5 ))
print("List:", list_comp)
print("Generator Object:", gen_expr)
print("Generator Value:", next (gen_expr))
Expected output
List: [0, 2, 4, 6, 8] Generator Object:at 0x...> Generator Value: 0
| Feature | Lists | Generators |
|---|---|---|
| Memory Consumption | High (stores all items in RAM) | Minimal (stores one item at a time) |
| Evaluation | Eager (computed immediately) | Lazy (computed on demand) |
| Reusability | Can be iterated multiple times | Exhausted after one iteration |
The important terms to remember
Design tip: Use generator expressions when processing large data streams or files to keep memory usage constant.
5. Common Mistakes to Avoid
1. Attempting to reuse an exhausted generator
Iterating over a generator a second time yields no items because its state is already consumed.
Better approach: Recreate the generator function or store items in a list if multiple passes are required.
6. Interview Questions & Answers
Question 1: What is the difference between return and yield in Python?
What the interviewer expects: A clear explanation of state persistence and function termination.
Strong answer: "A return statement terminates a function completely and sends back a single result. A yield statement pauses the function, saves its local state, and returns an intermediate value, allowing execution to resume later."
What interviewers expect at different levels
7. Practice Exercises
Beginner: Fibonacci generator
Write a generator function that yields the first 10 Fibonacci numbers.
Expected result: Prints Fibonacci numbers sequentially without storing a massive list.
Intermediate: File line reader
Create a generator that reads a large text file line-by-line, stripping whitespace.
Hint: Use with open(...) as f: combined with a for line in f: loop.
8. Frequently Asked Questions (FAQ)
Can you find the length of a generator using len()?
No. Generators do not support len() because their total item count is not computed in advance.
Are generators faster than lists?
Generators save memory, but execution speed depends on whether items need to be recalculated or reused.
9. Next Steps
Now that you have mastered iterators and generators, you are ready to explore decorators, closures, and error handling in the next article.
- Instant memory allocation consumes all available RAM.
- Garbage collection struggles with large monolithic objects.
- Processing cannot start until the entire list is built in memory.
- Iterator = an object implementing
__iter__()and__next__(). - Generator = a function containing the
yieldkeyword. - Lazy Evaluation = computing values only when requested.
- Junior developer: Knows how to use generator functions with
yield. - Senior developer: Understands iterator protocols, memory efficiency, and generator exhaustion.
- Technical architect: Designs streaming data pipelines for high-throughput enterprise systems.
- Iterator = an object implementing