Python Notes
Deep dive into Python async/await, asyncio event loop, coroutines, tasks, async generators, aiohttp, and real-world asynchronous patterns with Hindi explanations.
Introduction
Async programming allows Python to perform concurrent I/O operations in a single thread. When one operation is waiting (network response, disk read), another operation continues executing.
Analogy: A chef who puts one dish in the oven and preps another — one task at a time, but efficiently.
2. Event Loop
import asyncio
# Event loop kya hai?
"""
Event Loop asyncio ka heart hai:
1. Coroutines register karta hai
2. I/O events monitor karta hai
3. Ready coroutines ko run karta hai
4. Callbacks schedule karta hai
"""
# Getting the event loop
async def show_loop_info():
loop = asyncio.get_event_loop()
print(f"Loop type: {type(loop).__name__}")
print(f"Is running: {loop.is_running()}")
asyncio.run(show_loop_info())
# Manual event loop (older style — avoid in new code)
# loop = asyncio.new_event_loop()
# loop.run_until_complete(show_loop_info())
# loop.close()
# asyncio.run() is the preferred way (Python 3.7+)
async def main():
await asyncio.sleep(0) # Yield control once
print("Main coroutine")
asyncio.run(main())Loop type: _UnixSelectorEventLoop Is running: True Main coroutine
3. Tasks — Concurrent Execution
import asyncio
import time
async def fetch_data(source, delay):
print(f"Fetching from {source}...")
await asyncio.sleep(delay) # Simulate network delay
data = f"data_from_{source}"
print(f"Got data from {source}")
return data
async def main():
# asyncio.create_task() — run in background
start = time.time()
# Create tasks (start immediately)
task1 = asyncio.create_task(fetch_data("API_1", 2))
task2 = asyncio.create_task(fetch_data("API_2", 1))
task3 = asyncio.create_task(fetch_data("API_3", 1.5))
# Wait for all tasks
results = await asyncio.gather(task1, task2, task3)
print(f"\nAll results: {results}")
print(f"Total time: {time.time()-start:.2f}s") # ~2s, not 4.5s!
asyncio.run(main())Fetching from API_1... Fetching from API_2... Fetching from API_3... Got data from API_2 Got data from API_3 Got data from API_1 All results: ['data_from_API_1', 'data_from_API_2', 'data_from_API_3'] Total time: 2.00s
4. asyncio.gather() vs asyncio.wait()
gather results: ['A done', 'B done', 'C done']
With exceptions: ['A done', ValueError('B failed!'), 'C done']
First completed: 1 tasks
Result: Y done5. Async Generators
Received: {'id': 0, 'value': 42}
Received: {'id': 1, 'value': 73}
Received: {'id': 2, 'value': 15}
Received: {'id': 3, 'value': 88}
Received: {'id': 4, 'value': 56}
Average: 54.8
Collected: [{'id': 0, 'value': 31}, {'id': 1, 'value': 67}, {'id': 2, 'value': 44}, {'id': 3, 'value': 92}, {'id': 4, 'value': 19}]📝 Hindi Explanation
Async Generator uses bothasync defandyield. Iterate it withasync foror[x async for x in gen]async comprehension. It's perfect for real-time data streams (websockets, live feeds, streaming APIs).
6. Async Context Managers
Connecting to database asynchronously... Query result: Result of: SELECT * FROM users Closing database connection... [Database query] 0.2003s
7. Async Synchronization Primitives
Lock-protected value: 5000 Task-0: Started Task-1: Started Task-2: Started Task-0: Done Task-1: Done Task-2: Done Task-3: Started Task-4: Started Task-5: Started ... Produced: item-0 [C0] Consumed: item-0
8. Real-World: Async HTTP Requests
Successful: 10, Failed: 0 Fetched 10 URLs in 0.50s
📝 Hindi Explanation
aiohttp is an asyncio-based HTTP client library.async with aiohttp.ClientSession() as session:manages the connection pool. Withasyncio.gather(), all requests are sent in parallel — 10 URLs take 0.5s (the time for 1 URL), not 10s!
9. Error Handling in Async Code
Success: [2, 4, 8, 10, 14, 16]
Errors: [(0, ValueError('Failed for 0')), (3, ValueError('Failed for 3')), (6, ValueError('Failed for 6')), (9, ValueError('Failed for 9'))]
Operation timed out!
Caught task exception: Task failed!10. Cancellation
import asyncio
async def long_running():
try:
print("Long running task started")
await asyncio.sleep(10)
print("Long running task done")
except asyncio.CancelledError:
print("Task was cancelled!")
# Cleanup here
raise # Always re-raise CancelledError!
async def main():
task = asyncio.create_task(long_running())
await asyncio.sleep(1)
# Cancel the task
task.cancel()
try:
await task
except asyncio.CancelledError:
print("Main: Task cancelled confirmed")
asyncio.run(main())Long running task started Task was cancelled! Main: Task cancelled confirmed
11. Best Practices
import asyncio
# ✅ Use asyncio.run() as entry point
async def main():
pass
asyncio.run(main())
# ✅ Use asyncio.create_task() for concurrent background tasks
async def concurrent_demo():
task1 = asyncio.create_task(asyncio.sleep(1))
task2 = asyncio.create_task(asyncio.sleep(1))
await asyncio.gather(task1, task2)
# ✅ Handle CancelledError properly
async def cancellable():
try:
await asyncio.sleep(10)
except asyncio.CancelledError:
# Cleanup
raise # MUST re-raise!
# ✅ Use asyncio.wait_for() for timeouts
async def with_timeout():
try:
result = await asyncio.wait_for(asyncio.sleep(5), timeout=2)
except asyncio.TimeoutError:
pass
# ❌ Don't mix sync blocking with async
# async def bad():
# time.sleep(5) # Blocks entire event loop!
# ✅ Use asyncio.sleep(0) to yield control
async def good():
for i in range(1000):
if i % 100 == 0:
await asyncio.sleep(0) # Yield to other coroutines
# ❌ Don't call async functions without await
# result = async_function() # Returns coroutine object, not result!
# ✅ Always await async functions
async def correct():
result = await some_async_function()
async def some_async_function():
return 42Summary
| Concept | Tool | Use Case |
|---|---|---|
| Run coroutine | asyncio.run() | Entry point |
| Concurrent tasks | asyncio.gather() | Multiple coroutines |
| Background task | asyncio.create_task() | Fire and forget |
| Timeout | asyncio.wait_for() | Deadline control |
| Rate limiting | asyncio.Semaphore() | Limit concurrency |
| Communication | asyncio.Queue() | Producer-consumer |
| Mutual exclusion | asyncio.Lock() | Shared state |
asyncio is perfect for: Web servers (FastAPI, aiohttp), web scraping, API clients, real-time applications, and any I/O-heavy workloads.
Processed 20 items
⚠️ Common Mistakes
1. ❌ Using time.sleep() in async code
import asyncio, time
# ❌ WRONG — blocks the entire event loop!
async def bad_sleep():
time.sleep(5) # No other coroutine will run for 5 seconds!
# ✅ CORRECT — non-blocking sleep
async def good_sleep():
await asyncio.sleep(5) # Event loop doosre tasks run karegatime.sleep()is synchronous — it blocks the entire event loop. Always useawait asyncio.sleep()in async code.
2. ❌ Forgetting await before coroutine calls
async def fetch_data():
return "data"
# ❌ WRONG — returns coroutine object, not result!
async def bad():
result = fetch_data() # RuntimeWarning: coroutine was never awaited
print(result) # <coroutine object fetch_data at 0x...>
# ✅ CORRECT
async def good():
result = await fetch_data()
print(result) # "data"Using await is mandatory — without await you only get a coroutine object, not the actual result!3. ❌ Creating tasks but never awaiting them
# ❌ WRONG — task might not complete before program exits
async def bad():
asyncio.create_task(some_work()) # Fire and forget — may get garbage collected!
# ✅ CORRECT — always keep a reference and await
async def good():
task = asyncio.create_task(some_work())
await task # Ya gather() use karoAfter creating a task, you must await it or keep a reference to it, otherwise the task may be silently cancelled.
4. ❌ Not handling CancelledError properly
# ❌ WRONG — swallowing CancelledError breaks cancellation
async def bad():
try:
await asyncio.sleep(10)
except asyncio.CancelledError:
print("Cancelled") # Error swallowed — task won't actually cancel!
# ✅ CORRECT — always re-raise CancelledError
async def good():
try:
await asyncio.sleep(10)
except asyncio.CancelledError:
print("Cleaning up...")
raise # MUST re-raise!Catching and re-raising CancelledError is essential. If you swallow it, the cancellation chain will break.5. ❌ Running CPU-bound work in async functions
# ❌ WRONG — blocks the event loop
async def bad():
result = sum(range(10_000_000)) # CPU-intensive, no await points!
# ✅ CORRECT — offload to thread pool
async def good():
loop = asyncio.get_event_loop()
result = await loop.run_in_executor(None, sum, range(10_000_000))Don't perform CPU-heavy work (calculations, image processing) directly in async functions. Offload to a thread pool using run_in_executor().6. ❌ Mixing asyncio.run() inside an already running event loop
# ❌ WRONG — RuntimeError: This event loop is already running
async def bad():
asyncio.run(some_coroutine()) # Can't nest asyncio.run()!
# ✅ CORRECT — just await directly
async def good():
await some_coroutine()You cannot callasyncio.run()when an event loop is already running. Useawaitdirectly instead.
7. ❌ Not using return_exceptions=True with asyncio.gather()
# ❌ RISKY — one failure cancels everything
async def risky():
results = await asyncio.gather(task1(), task2(), task3()) # task2 fails = all fail!
# ✅ SAFE — handle failures gracefully
async def safe():
results = await asyncio.gather(task1(), task2(), task3(), return_exceptions=True)
for r in results:
if isinstance(r, Exception):
print(f"Error: {r}")Use return_exceptions=True when you need all results — a single failure won't cancel the remaining tasks.✅ Key Takeaways
- 🔹
async defdefines a coroutine function — unlike a normal function, calling it returns a coroutine object that executes when youawaitit. - 🔹
awaitpauses execution (without blocking!) — the event loop can run other coroutines while waiting. - 🔹
asyncio.run()is the standard entry point (Python 3.7+) — it creates the event loop, runs the coroutine, and handles cleanup. - 🔹
asyncio.gather()runs multiple coroutines concurrently and returns results in an ordered list — perfect for parallel I/O. - 🔹
asyncio.create_task()schedules a coroutine to run in the background — the result is available when you await it. - 🔹 Async is NOT parallel — it's cooperative multitasking in a single thread. For CPU-bound work, use
multiprocessingorrun_in_executor(). - 🔹
asyncio.Semaphoreis used for rate limiting — essential for limiting API calls and DB connections. - 🔹
asyncio.Queueimplements the producer-consumer pattern — best for async pipelines and worker pools. - 🔹 Error handling: use
return_exceptions=Truewith gather() — one failed task won't cancel the others. - 🔹 Always handle
CancelledError— perform cleanup and re-raise it, don't swallow it!
❓ FAQ
Q1: What is the difference between asyncio and threading?
A: asyncio performs cooperative multitasking in a single thread — coroutines voluntarily yield control at await points. Threading is OS-level preemptive multitasking where the OS switches threads. asyncio is better for I/O-bound work (less overhead, no GIL issues), while threading is useful for CPU-bound or legacy blocking code.
Q2: Does async code automatically run faster?
A: No! Async only provides speed benefits for I/O-bound operations (network calls, file reads, DB queries) where there is wait time. For CPU-bound work (calculations, image processing), there's no benefit — use multiprocessing for that. The advantage of async is that other work can happen during wait time.
Q3: What is the difference between asyncio.gather() and asyncio.wait()?
A: gather() is simpler — all results come in an ordered list. wait() gives more control — you can set return conditions like FIRST_COMPLETED, FIRST_EXCEPTION, or ALL_COMPLETED. With wait() you can get partial results and cancel pending tasks.
Q4: Can I use an existing synchronous library in async code?
A: Yes! Use loop.run_in_executor() — it runs the blocking function in a thread pool without blocking the event loop:
import asyncio
import requests # Sync library
async def fetch():
loop = asyncio.get_event_loop()
response = await loop.run_in_executor(None, requests.get, "https://api.example.com")
return response.json()Q5: How does FastAPI use async internally?
A: FastAPI runs on uvicorn (ASGI server) which runs an asyncio event loop. Each request is a coroutine — async def endpoints run directly on the event loop, while regular def endpoints run in a thread pool. That's why you shouldn't make blocking calls in FastAPI's async def endpoints.
Q6: async for kab use karna chahiye?
A: When data is arriving asynchronously one item at a time from a stream — websocket messages, database cursor results, file chunks streaming, or Server-Sent Events. A regular for loop doesn't work with async iterators.
Q7: What is the difference between asyncio.Queue and queue.Queue?
A: asyncio.Queue is async/await compatible — await queue.get() doesn't block the event loop. queue.Queue is for threading — queue.get() is blocking. Don't mix them — always use asyncio.Queue in async code!
Q8: How do you debug async code in production?
A: (1) Enable asyncio.get_event_loop().set_debug(True) — it detects slow callbacks and unawaited coroutines. (2) Set the PYTHONASYNCIODEBUG=1 environment variable. (3) Use the aiodebug library. (4) Track task IDs with structured logging (loguru/structlog).
🎯 Interview Questions
Q1: What is the difference between concurrency and parallelism in Python's asyncio?
A: Concurrency means multiple tasks make progress by interleaving execution on a single thread — asyncio achieves this via cooperative multitasking at await points. Parallelism means multiple tasks execute simultaneously on different CPU cores (multiprocessing). asyncio is concurrent but NOT parallel — it excels at I/O-bound workloads where tasks spend most time waiting.
Q2: Explain the asyncio event loop and its lifecycle.
A: The event loop is the central execution mechanism: (1) It maintains a queue of ready coroutines. (2) Runs a coroutine until it hits an await. (3) Registers the awaited I/O operation with the OS (epoll/kqueue/IOCP). (4) Picks the next ready coroutine. (5) When I/O completes, marks the waiting coroutine as ready. Lifecycle: asyncio.run() creates the loop → runs the main coroutine → closes the loop. Only one event loop runs per thread.
Q3: What happens when you call an async def function without await?
A: It returns a coroutine object without executing the function body. Python will emit RuntimeWarning: coroutine 'func_name' was never awaited. The coroutine is then garbage collected without execution. This is a common bug — always await coroutines or wrap them in asyncio.create_task().
Q4: How does asyncio.gather() handle exceptions? What's return_exceptions?
A: By default, if any coroutine raises an exception, gather() cancels all remaining tasks and propagates the first exception. With return_exceptions=True, exceptions are returned as values in the results list alongside successful results — no task is cancelled, and you can handle errors individually.
Q5: How would you limit concurrent async operations (e.g., API rate limiting)?
A: Use asyncio.Semaphore:
This is essential for respecting API rate limits and preventing resource exhaustion.
Q6: What is the difference between asyncio.create_task() and directly awaiting a coroutine?
A: await coroutine() runs sequentially — the current coroutine waits for it to complete before continuing. asyncio.create_task(coroutine()) schedules it for concurrent execution — it starts running in the background immediately, and you can await the task later or gather multiple tasks. Use create_task() when you want concurrency.
Q7: How do you handle timeouts in async code?
A: Use asyncio.wait_for(coroutine, timeout=seconds). It raises asyncio.TimeoutError if the coroutine doesn't complete within the specified time. For more control, use asyncio.timeout() (Python 3.11+) context manager. Always wrap timeout-sensitive operations in try/except to handle graceful fallback.
Q8: Can you mix synchronous and asynchronous code? How?
A: Yes — use loop.run_in_executor() to run blocking sync code in a thread pool without blocking the event loop. For the reverse (calling async from sync), use asyncio.run() at the top level. Never call blocking functions directly inside async functions — it starves the event loop and defeats the purpose of async.
Q9: What are async context managers and when do you use them?
A: Async context managers implement __aenter__ and __aexit__ (both async). Use them when setup/teardown involves I/O — database connections, HTTP sessions, file handles, locks. Example: async with aiohttp.ClientSession() as session: ensures the connection pool is properly opened and closed asynchronously.
Q10: Design a producer-consumer system using asyncio. What components do you need?
A: Components: (1) asyncio.Queue as the buffer between producers and consumers. (2) Producer coroutines that await queue.put(item). (3) Consumer coroutines that await queue.get() and call queue.task_done(). (4) await queue.join() to wait until all items are processed. (5) Graceful shutdown via cancellation or sentinel values. Use maxsize to apply backpressure on fast producers.
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Python Asynchronous Programming with asyncio.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Python Master Course topic.
Search Terms
python-master-course, python master course, python, master, course, advanced, asynchronous, programming
Related Python Master Course Topics