Python Notes
Deep dive into Python generators, yield keyword, generator expressions, send(), throw(), close() methods, pipelines, and coroutines with Hindi explanations.
Introduction
Generators are a powerful Python feature that implements lazy evaluation. Using the yield keyword, you can create a function that generates values one at a time instead of storing them all in memory at once.
Key Benefits:
- Memory efficient (values computed on demand)
- Can represent infinite sequences
- Clean, readable code
- Supports coroutine patterns
2. Generator State & Multiple yield
def multi_yield_demo():
print("Step 1: Before first yield")
yield "First value"
print("Step 2: Between yields")
yield "Second value"
print("Step 3: Before last yield")
yield "Third value"
print("Step 4: Generator finished")
gen = multi_yield_demo()
value1 = next(gen)
print(f"Got: {value1}")
# Output:
# Step 1: Before first yield
# Got: First value
value2 = next(gen)
print(f"Got: {value2}")
# Output:
# Step 2: Between yields
# Got: Second value
value3 = next(gen)
print(f"Got: {value3}")
# Output:
# Step 3: Before last yield
# Got: Third value
try:
next(gen)
except StopIteration:
print("Generator exhausted!")
# Output:
# Step 4: Generator finished
# Generator exhausted!3. Infinite Generator
4. Generator Expressions
📝 Hindi Explanation
Generator Expression is exactly like a list comprehension, but uses()instead of[]. It is lazy — values are computed only when demanded. Perfect for one-pass operations on large data.
5. send() Method — Two-way Communication
# Values can be sent to a generator using send()
def accumulator():
"""Receive values and maintain a running total"""
total = 0
while True:
value = yield total # yield returns current total, then receives value
if value is None:
break
total += value
acc = accumulator()
print(next(acc)) # 0 — generator start karo (priming)
print(acc.send(10)) # 10
print(acc.send(20)) # 30
print(acc.send(5)) # 35
print(acc.send(15)) # 50# More realistic: echo generator
def echo_generator():
"""Jo bhejo wahi wapas do"""
received = None
while True:
received = yield received
echo = echo_generator()
next(echo) # Prime the generator
print(echo.send("Hello")) # Hello
print(echo.send("World")) # World
print(echo.send(42)) # 42
# Generator-based coroutine for data processing
def running_average():
"""Real-time average calculator"""
count = 0
total = 0.0
average = None
while True:
value = yield average
if value is None:
return
count += 1
total += value
average = total / count
avg = running_average()
next(avg) # Prime
print(avg.send(10)) # 10.0
print(avg.send(20)) # 15.0
print(avg.send(30)) # 20.0
print(avg.send(40)) # 25.06. throw() and close() Methods
# throw() — generator mein exception bhejo
def safe_generator():
try:
while True:
try:
value = yield
print(f"Processing: {value}")
except ValueError as e:
print(f"ValueError caught inside generator: {e}")
yield "error handled"
except GeneratorExit:
print("Generator closing...")
gen = safe_generator()
next(gen) # Prime
gen.send(42) # Processing: 42
next(gen) # continue
# throw() se exception bhejo
result = gen.throw(ValueError, "invalid data")
print(f"Throw result: {result}") # error handled
# close() — generator band karo
gen.close() # GeneratorExit raise karta hai# try/finally in generator — cleanup guaranteed
def resource_generator():
"""Acquire resource and perform cleanup"""
print("Acquiring resource...")
resource = "database_connection"
try:
for i in range(5):
yield f"Data {i} from {resource}"
finally:
print(f"Releasing resource: {resource}")
gen = resource_generator()
print(next(gen)) # Data 0 from database_connection
print(next(gen)) # Data 1 from database_connection
gen.close() # Triggers finally — resource released
# Output: Releasing resource: database_connection📝 Hindi Explanation
throw() injects an exception into the generator. The generator can catch it or let it propagate. close() sends aGeneratorExitexception and terminates the generator — its cleanup code infinallywill still run.
7. yield from — Sub-generators
# yield from — nested generator delegation
def inner_gen(start, end):
for i in range(start, end):
yield i
def outer_gen():
print("Starting outer generator")
yield from inner_gen(1, 4) # Delegate to inner
print("Middle of outer generator")
yield from inner_gen(10, 13) # Delegate again
print("End of outer generator")
for val in outer_gen():
print(val)
# Output:
# Starting outer generator
# 1
# 2
# 3
# Middle of outer generator
# 10
# 11
# 12
# End of outer generator8. Generator Pipelines
📝 Hindi Explanation
Generator Pipeline is a powerful pattern where multiple generators are chained together. Each generator is a step — data enters from one end and comes out transformed at the other. No intermediate lists are created, keeping memory constant.
9. Generator-based Context Manager (contextlib)
from contextlib import contextmanager
@contextmanager
def timer_context():
"""Execution time measure karo"""
import time
start = time.time()
try:
yield # with block yahan execute hoga
finally:
end = time.time()
print(f"Elapsed: {end - start:.4f}s")
with timer_context():
total = sum(range(1000000))
print(f"Sum: {total}")
# Output:
# Sum: 499999500000
# Elapsed: 0.0452s
@contextmanager
def database_connection(db_path):
"""Temporary database connection"""
import sqlite3
conn = sqlite3.connect(db_path)
try:
print(f"Connected to {db_path}")
yield conn
except Exception as e:
conn.rollback()
raise
finally:
conn.close()
print(f"Disconnected from {db_path}")
# with database_connection(":memory:") as conn:
# cursor = conn.cursor()
# cursor.execute("CREATE TABLE test (id INTEGER)")10. Async Generators (Python 3.6+)
11. Performance Comparison
12. Best Practices & Common Pitfalls
Summary
| Method | Purpose | Returns |
|---|---|---|
yield value | Pause and return value | value to caller |
yield from iterable | Delegate to sub-generator | sub-generator values |
send(value) | Send value into generator | next yielded value |
throw(exc) | Inject exception | next yielded value |
close() | Close generator | None |
# Quick Reference
def generator_template():
# Setup / acquire resources
resource = "setup"
try:
while True:
# Receive input via send()
received = yield "output_value"
if received is None:
break
# Process received value
process(received)
except GeneratorExit:
pass # handle close()
finally:
pass # cleanup always runs
def process(x): passGenerators are Python's built-in coroutines! They are the foundation for asyncio and modern async Python programming.⚠️ Common Mistakes
Mistake 1: Forgetting to Prime the Generator 🚫
def my_coroutine():
value = yield
yield value * 2
gen = my_coroutine()
# ❌ Wrong - bina prime kiye send()
# gen.send(5) # TypeError: can't send non-None value to a just-started generator
# ✅ Correct - call next() or send(None) first
gen = my_coroutine()
next(gen) # Prime karo
print(gen.send(5)) # Ab kaam karega10
Mistake 2: Reusing an Exhausted Generator 🔄
gen = (x**2 for x in range(5))
print(list(gen)) # First time kaam karega
print(list(gen)) # Second time empty list![0, 1, 4, 9, 16] []
Fix: Recreate the generator or create a factory function:
def square_gen():
return (x**2 for x in range(5))
print(list(square_gen())) # Fresh generator har baar
print(list(square_gen())) # Works again![0, 1, 4, 9, 16] [0, 1, 4, 9, 16]
Mistake 3: Expecting a return Value from a Generator ❌
def bad_gen():
for i in range(3):
yield i
return "done" # Yeh directly accessible nahi hai!
gen = bad_gen()
for val in gen:
print(val)
# "done" won't be printed! It's in StopIteration.value0 1 2
Fix: Use yield from which captures the return value.Mistake 4: Accessing a Generator with len() or Indexing 📏
10 0
Mistake 5: Sharing a Generator Between Multiple Functions 🤝
def processor1(gen):
for item in gen:
if item > 3:
return item
def processor2(gen):
return sum(gen)
gen = (x for x in range(10))
result1 = processor1(gen) # Yeh gen ko partially consume karega
result2 = processor2(gen) # Yeh sirf remaining values process karega!
print(f"First > 3: {result1}")
print(f"Sum of remaining: {result2}")First > 3: 4 Sum of remaining: 35
Fix: Create a separate generator for each function, or use itertools.tee().Mistake 6: Putting Side Effects Inside a Generator ⚡
# ❌ Side effects in generators are risky
count = 0
def counting_gen(n):
global count
for i in range(n):
count += 1 # Side effect!
yield i
gen = counting_gen(5)
# If gen is never consumed, count will never be updated
print(f"Before: count = {count}")
list(gen) # Ab consume hua
print(f"After: count = {count}")Before: count = 0 After: count = 5
Mistake 7: Not Handling StopIteration 🛑
def limited_gen():
yield 1
yield 2
gen = limited_gen()
print(next(gen))
print(next(gen))
# ❌ Third next() will raise StopIteration
try:
print(next(gen))
except StopIteration:
print("Generator khatam ho gaya! Default value use karo: next(gen, 'default')")1 2 Generator khatam ho gaya! Default value use karo: next(gen, 'default')
✅ Key Takeaways
- 🔁 Generators use lazy evaluation — values are computed only when needed, making them memory efficient.
- 📦
yieldpauses the function and preserves state. The function resumes from that point whennext()is called. - 🧮 Generator Expressions
(x for x in iterable)are the memory-efficient version of list comprehensions — best for large data. - 📨
send()method turns a generator into a two-way communication channel — values can be both sent and received. - 🔗
yield fromelegantly delegates to sub-generators — perfect for flattening nested generators. - 🏗️ Generator Pipelines are ideal for streaming data processing — each stage is a generator, no intermediate lists.
- ⚡ Generators are one-shot — once exhausted, they cannot be reused. Use factory functions or
itertools.tee(). - 🔒
contextlib.contextmanagerdecorator turns generators into context managers — making resource management withwithstatements easy. - 🌐 Async Generators (
async def+yield) support asynchronous iteration — used withasync for. - 🎯 Generators are the foundation of
asyncio— Python's modern async programming model is based on generators.
❓ FAQ
Q1: What is the difference between a Generator and an Iterator?
Answer: An Iterator is a protocol (an object that implements __iter__() and __next__()). A Generator is a special type of iterator created using the yield keyword. Generators automatically implement the iterator protocol with less code.
Q2: When should you use a Generator vs a List?
Answer: Use a Generator when: (1) The data is too large to fit in memory, (2) You need infinite sequences, (3) You're doing pipeline processing. Use a List when: (1) You need random access/indexing, (2) You need to iterate multiple times, (3) You need to know the length.
Q3: What is the difference between yield and return?
Answer: return terminates the function and gives one value. yield pauses the function, gives a value, and resumes from that point on the next call. A function can yield multiple values over multiple calls.
Q4: What is the practical use of the send() method?
Answer: send() lets you feed values to a generator at runtime — useful for the coroutine pattern. Real examples: calculating running averages, implementing state machines, and building interactive pipelines.
Q5: How do you debug a Generator?
Answer: (1) Inspect the current frame with gen.gi_frame, (2) Check the source file with gen.gi_code.co_filename, (3) Check sub-generator delegation with gen.gi_yieldfrom, (4) Use list(gen) to see all values (caution: exhausts it!).
Q6: Are Generators thread-safe?
Answer: No! Generators are not thread-safe. If multiple threads call next() on the same generator, race conditions can occur. For thread safety, wrap with threading.Lock().
Q7: What is the difference between yield from and a simple loop with yield?
Answer: yield from doesn't just forward values — it also propagates send(), throw(), and close() to the inner generator. Plus, the inner generator's return value is captured as the result of the yield from expression.
Q8: What is the difference between a Generator Expression and a Lambda?
Answer: A generator expression returns an iterator that lazily produces values. A lambda is an anonymous function that evaluates a single expression and returns the result immediately.
🎯 Interview Questions
Q1: What is a Generator in Python and how is it different from a regular function?
Answer: A Generator is a special function that uses the yield keyword. A regular function terminates with return giving one value, while a generator pauses at each yield, preserves state, and can produce multiple values over multiple calls.
Q2: Explain the internal mechanism of a Generator — how does Python manage it internally?
Answer: Python maintains a frame object (gi_frame) for each generator that stores local variables, instruction pointer, and stack state. When yield is hit, the frame is frozen. On the next next() call, execution resumes from the stored instruction pointer.
Q3: What is the use case of yield from? Explain with a simple example.
Answer: yield from delegates to a sub-generator. It transparently passes send(), throw(), and close() to the inner generator, and the inner generator's return value becomes the value of the yield from expression.
Q4: What is the Generator Pipeline pattern? Give a real-world example.
Answer: In a generator pipeline, multiple generators form a chain — each generator consumes data from the previous one and yields transformed data. This processes millions of records without storing intermediate results in memory.
Q5: How does exception handling work inside a Generator?
Answer: (1) try/except inside a generator works normally. (2) throw(exc_type) injects an exception from outside — the generator can catch it at the yield point and continue. (3) close() injects GeneratorExit — if not caught, the generator terminates.
Q6: What is the difference between a Generator and a List in terms of memory efficiency? Quantify it.
Answer: A list stores all values in memory at once — [x for x in range(1_000_000)] consumes ~8MB. A generator holds only one value at a time — (x for x in range(1_000_000)) uses only ~120 bytes. That's a ~70,000x memory difference!
Q7: How does the send() method work? Explain the Coroutine pattern.
Answer: send(value) resumes the generator and sets the value of the yield expression. Flow: (1) Prime the generator (next(gen) or gen.send(None)), (2) send(value) resumes and the yielded expression receives the value, (3) Generator yields the next result.
Q8: What is an Async Generator? How is it different from a regular Generator?
Answer: An Async Generator uses yield with async def. It works with async for loops. Key differences: (1) await expressions can be used inside, (2) Iteration itself is async (__anext__ returns a coroutine), (3) Both async for and async comprehensions support it.
Q9: How is the itertools module used with Generators? Name important functions.
Answer: itertools provides generator-based utility functions: (1) islice(gen, n) — take n values from a generator, (2) chain(*gens) — concatenate multiple generators, (3) tee(gen, n) — duplicate a generator (use carefully — memory!), (4) accumulate() — running totals, (5) takewhile(pred, gen) — take while condition is true.
Q10: How does a Generator-based Context Manager (@contextmanager) work?
Answer: The @contextmanager decorator turns a generator function into a context manager. Pattern: (1) Code before yield is __enter__ — resource acquisition. (2) The yielded value is passed to as. (3) Code after yield is __exit__ — resource cleanup. The finally block ensures cleanup even on exception.
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Advanced Python Generators.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Python Master Course topic.
Search Terms
python-master-course, python master course, python, master, course, advanced, generators, advanced python generators
Related Python Master Course Topics