Python Notes
Master Python iterators with __iter__, __next__, custom iterator classes, infinite iterators, and real-world use cases with detailed examples and Hindi explanations.
Introduction
Python iterators are a fundamental concept that unlocks Python's "lazy evaluation" power. Whenever you use a for loop, Python uses the iterator protocol internally. Understanding iterators helps you write memory-efficient and performant code.
An iterator is an object that implements two methods:
__iter__()— returns the iterator object itself__next__()— returns the next value, raisesStopIterationwhen exhausted
An iterable is any object that implements __iter__() (lists, tuples, strings, dicts, sets).
2. Iterable vs Iterator
True False True True [1, 2, 3] [] [1, 2, 3]
True False True True
3. Custom Iterator Class
# Simple counter iterator
class CountUp:
"""Iterator that counts from 1 to n"""
def __init__(self, start, end):
self.current = start
self.end = end
def __iter__(self):
return self # self return karo
def __next__(self):
if self.current > self.end:
raise StopIteration
value = self.current
self.current += 1
return value
# Usage
counter = CountUp(1, 5)
for num in counter:
print(num, end=" ")
# Output: 1 2 3 4 5
# Can also use next() directly
counter2 = CountUp(10, 12)
print(next(counter2)) # 10
print(next(counter2)) # 11
print(next(counter2)) # 121 2 3 4 5 10 11 12
📝 Hindi Explanation
To create a Custom Iterator, you need to define both__iter__()and__next__()methods in a class. Returnselfin__iter__(), and in__next__()return the next value or raiseStopIterationwhen done.
4. Iterator with State
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34] 0 1 1 2 3 5 8 13
[2, 4, 6, 8, 10]
5. Infinite Iterator
[5, 8, 11, 14, 17] [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
📝 Hindi Explanation
An Infinite Iterator is one that never raisesStopIteration. Useitertools.islice()oritertools.takewhile()with such iterators to limit the output, otherwise you'll get an infinite loop.
6. Resettable Iterator
[0, 1, 2, 3, 4] [] [0, 1, 2, 3, 4]
7. Separating Iterable and Iterator
[1, 2, 3] [1, 2, 3] 1 1 2
📝 Hindi Explanation
Best Practice: Keep Iterable and Iterator separate. If__iter__()returnsself, the object can only be iterated once. If it returns a fresh iterator each time, multiple independent iterations are possible.
8. Built-in Iterator Tools
[1, 2, 3, 4, 5, 6] Alice: 95 Bob: 87 Charlie: 92 1. apple 2. banana 3. cherry [1, 4, 9, 16, 25] [2, 4]
9. Iterator vs Generator
[1, 4, 9, 16, 25] [1, 4, 9, 16, 25] Iterator size: 48 bytes Generator size: 104 bytes
10. Real-World: File Line Iterator
class FileLineIterator:
"""Read a large file in a memory-efficient way"""
def __init__(self, filepath, encoding='utf-8'):
self.filepath = filepath
self.encoding = encoding
self._file = None
def __iter__(self):
self._file = open(self.filepath, 'r', encoding=self.encoding)
return self
def __next__(self):
line = self._file.readline()
if line == '':
self._file.close()
raise StopIteration
return line.rstrip('\n')
def __del__(self):
if self._file and not self._file.closed:
self._file.close()
# Usage (assuming data.txt exists)
# for line in FileLineIterator("data.txt"):
# process(line)
#
# Memory efficient: only one line loaded at a time!
# Better: Context manager version
class SafeFileIterator:
def __init__(self, filepath):
self.filepath = filepath
self._file = None
def __enter__(self):
self._file = open(self.filepath, 'r')
return self
def __exit__(self, *args):
if self._file:
self._file.close()
def __iter__(self):
return self
def __next__(self):
line = self._file.readline()
if not line:
raise StopIteration
return line.strip()
# with SafeFileIterator("data.txt") as f:
# for line in f:
# print(line)📝 Hindi Explanation
Real-world use case: When processing large files, loading the entire file into memory is expensive. A custom FileIterator reads only one line at a time, keeping memory usage constant regardless of file size.
11. Chaining Custom Iterators
[12, 14, 16, 18, 20]
12. Best Practices
1 None ['red', 'green', 'blue', 'red', 'green', 'blue'] [1, 3, 6, 10, 15] [1, 3, 5]
13. Common Pitfalls
Summary
| Concept | Key Point |
|---|---|
| Iterable | Has __iter__(), returns iterator |
| Iterator | Has __iter__() + __next__() |
| StopIteration | Signals end of iteration |
| iter() | Gets iterator from iterable |
| next() | Gets next value from iterator |
| Infinite Iterator | Never raises StopIteration |
| Lazy Evaluation | Values computed on demand |
When to Use Iterators vs Generators?
- Iterators: When you need complex state management, multiple methods, or class hierarchy
- Generators: When logic is simple and sequential — much less code
10 [20, 30]
Remember: Every generator is an iterator, but not every iterator is a generator. Generators are the Pythonic way to create iterators!
⚠️ Common Mistakes
❌ Mistake 1: Using an iterator multiple times without reset
15 0
Fix: Create a new iterator or store values in a list first. A consumed iterator won't produce values again.
❌ Mistake 2: Not returning self in __iter__()
Fix: Always define__iter__()to returnself. Without__iter__(), the object won't be considered iterable.
❌ Mistake 3: StopIteration raise karna bhoolna
Fix: Always include a termination condition and raise StopIteration when the data is exhausted.❌ Mistake 4: Sharing mutable state inside an Iterator
Fix: Use instance variables (self.index), not class variables. Otherwise multiple iterators will interfere with each other.❌ Mistake 5: next() without default on empty iterator
empty = iter([])
# ❌ Yeh crash karega!
# value = next(empty) # StopIteration exception!
# ✅ Default value do
value = next(empty, "No data")
print(value)No data
Fix: Usenext(iterator, default)when you're not sure if data exists. This safely returnsNoneor a default value.
❌ Mistake 6: File iterator mein resource cleanup bhoolna
class LeakyFileIterator:
def __init__(self, path):
self.file = open(path, 'r') # ❌ File kabhi close nahi hogi!
def __iter__(self):
return self
def __next__(self):
line = self.file.readline()
if not line:
raise StopIteration # ❌ File still open!
return line.strip()Fix: Callself.file.close()before raisingStopIteration, or implement a context manager (__enter__/__exit__).
❌ Mistake 7: Iterable modify karna while iterating
[1, 3, 5]
Fix: Iterate over a copy using a list comprehension: for item in data[:]:✅ Key Takeaways
- 🔑 Iterator Protocol =
__iter__()+__next__()— these two methods make any object an iterator - 🔑 Iterable ≠ Iterator — A List is an iterable (can be iterated repeatedly), but a list_iterator is an iterator (consumed after one pass)
- 🔑 Lazy Evaluation — Iterators are memory efficient because values are generated on-demand, not all loaded into memory at once
- 🔑 StopIteration — This exception signals the end of iteration;
forloop catches it automatically - 🔑
next(iterator, default)— Providing a default value is safe practice to avoid exceptions - 🔑 Separate Iterable & Iterator — In production code, keep iterable and iterator classes separate so multiple independent iterations are possible
- 🔑 Infinite Iterators — Use with
itertools.islice()ortakewhile(), otherwise you'll get an infinite loop - 🔑 Generator > Iterator — For simple cases, use generator functions (less code, same performance); for complex state management, use class-based iterators
- 🔑 Resource Cleanup — Always include cleanup logic in file/network iterators (
try/finally, context managers) - 🔑 Built-in Tools — The
itertoolsmodule has powerful iterator utilities:chain(),cycle(),islice(),accumulate(),takewhile()
❓ FAQ
Q1: What is the difference between an Iterator and an Iterable?
Answer: An Iterable is an object with an __iter__() method (list, tuple, string, dict). An Iterator is an object with both __iter__() AND __next__() methods. Every Iterator is an Iterable, but not every Iterable is an Iterator.
Q2: How do you reuse an iterator once it's exhausted?
Answer: An exhausted iterator won't produce values again. Options are: (1) Create a new iterator with iter(iterable), (2) Keep Iterable and Iterator classes separate so __iter__() always returns a fresh iterator, (3) Convert to a list first if multiple passes are needed.
Q3: How does a for loop work internally?
Answer: for item in iterable: internally does this: first calls iter(iterable) to get an iterator, then repeatedly calls next(iterator) until StopIteration is raised. The StopIteration is caught automatically and the loop ends.
Q4: When should you use a Generator vs a class-based Iterator?
Answer: Use a Generator when the logic is simple and sequential — the yield keyword gets the job done with very little code. Use a class-based Iterator when you need complex state management, multiple public methods, or reset/rewind capability.
Q5: Are iterators thread-safe?
Answer: No, by default Python iterators are not thread-safe. If multiple threads call next() on the same iterator, race conditions can occur. For thread safety, wrap with threading.Lock() or use queue.Queue.
Q6: itertools.chain() vs + operator — what's the difference?
Answer: The + operator (like list1 + list2) creates a new list in memory. itertools.chain(list1, list2) returns a lazy iterator that yields items from both without copying. Chain is far more memory efficient for large data.
Q7: How do you safely use infinite iterators?
Answer: Always use a limiting mechanism with infinite iterators: itertools.islice(iterator, n) to take the first n items, itertools.takewhile(condition, iterator) to take while a condition is true, or a manual break in the loop.
Q8: When should you use the Iterator pattern in real projects?
Answer: (1) When processing large files line-by-line, (2) When fetching database results in batches, (3) When handling network streams/API pagination, (4) When generating infinite sequences (IDs, timestamps).
🎯 Interview Questions
Q1: What is the Iterator Protocol in Python? Explain with an example.
Answer: The Iterator Protocol is a Python convention requiring two methods: __iter__() which returns the iterator object, and __next__() which returns the next value or raises StopIteration. Any object implementing both is an iterator.
Q2: What is the fundamental difference between an Iterable and an Iterator? Explain with code.
Answer: An Iterable only has __iter__() (e.g., list, dict) — it can be iterated repeatedly because a fresh iterator is returned each time. An Iterator has both __iter__() and __next__() — it maintains position state and is consumed after one pass.
Q3: Create a custom iterator class that generates prime numbers up to n.
Answer: Store the limit in __init__, return self in __iter__, and in __next__ find and return the next prime. Use trial division for prime checking, and raise StopIteration when the limit is exceeded.
Q4: What are the two forms of the iter() function?
Answer: (1) iter(iterable) — creates an iterator from an iterable by calling __iter__(). (2) iter(callable, sentinel) — repeatedly calls the callable and yields results until the sentinel value is returned (then raises StopIteration).
Q5: When is it necessary to manually handle the StopIteration exception?
Answer: When you call next() directly (outside a for loop), you must handle StopIteration. The for loop handles it automatically. Best practice: use next(iter, default) to avoid exception handling altogether.
Q6: How does the Iterator pattern achieve memory efficiency? Compare with a List.
Answer: A List stores all elements in memory at once — 1 million items = 1 million objects in RAM. An Iterator uses lazy evaluation — only one value exists in memory at a time. For 1 million items: List ≈ 8MB, Iterator ≈ 120 bytes.
Q7: What are the drawbacks of keeping Iterable and Iterator in the same class?
Answer: If __iter__() returns self, the object can only be iterated once. Multiple for loops on the same object won't give expected results because the iterator's state is shared. Solution: Return a new iterator instance from __iter__().
Q8: Name 5 important functions from the itertools module and their use cases.
Answer: (1) chain(*iterables) — concatenate multiple iterables without memory copy. (2) islice(iter, start, stop) — slice from an iterator (works with infinite iterators). (3) cycle(iterable) — repeat infinitely. (4) accumulate(iter, func) — running totals. (5) takewhile(pred, iter) — take while predicate is true.
Q9: Can the __iter__ method in Python also be a generator function? Explain.
Answer: Yes! You can use yield inside the __iter__() method, which makes it automatically return a generator. This approach eliminates the need to write __next__ separately and keeps the code concise.
Q10: Real-world scenario: Database cursor as iterator — kaise implement karoge?
Answer: A database cursor uses the iterator pattern — fetchone() returns one row (like __next__), and when rows are exhausted, it returns None or raises an exception. This prevents loading millions of rows into memory at once.
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Python Iterators - Advanced Guide.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Python Master Course topic.
Search Terms
python-master-course, python master course, python, master, course, advanced, iterators, python iterators - advanced guide
Related Python Master Course Topics