Python Notes

Hub History of Python - Complete Guide 2026 Python Applications - Complete Guide 2026 Python Features - Complete Guide 2026 What is Python? - Complete Guide 2026 Why Learn Python? - Complete Guide 2026 Your First Python Program – Hello World & Beyond 2026 How to Install Python on Windows, Mac & Linux – Complete Guide 2026 Install VS Code for Python – Complete Setup Guide with Extensions 2026 Python Environment Setup – Virtual Environments, pip & Project Structure 2026 Python IDLE – Complete Guide for Beginners 2026 Python Comments Python Data Types Python Input and Output Python Operators Python Syntax Python Strings Basics Python Type Casting Python Variables Break, Continue and Pass in Python Elif Statement in Python For Loop in Python If-Else Statement in Python If Statement in Python Nested If in Python Pattern Programs in Python While Loop in Python Decorators in Python Function Arguments in Python Functions Basics in Python Generators in Python Lambda Functions in Python Recursive Functions in Python Return Statement in Python Scope of Variables in Python Abstraction in Python Classes and Objects in Python Constructors in Python Encapsulation in Python Inheritance in Python Magic Methods (Dunder Methods) in Python Method Overriding in Python Polymorphism in Python Python Comprehensions – List, Dict, Set & Generator Python Dictionaries – Key-Value Store Python Dictionary Methods – Complete Reference Python List Methods – All Built-in Methods Python Lists – Complete Guide Python Set Methods – Complete Reference Python Sets – Unordered Unique Collections Python Tuples – Immutable Sequences Python Asynchronous Programming with asyncio Python Context Managers Advanced Python Decorators Advanced Python Generators Python Iterators - Advanced Guide Python Memory Management Python Multiprocessing Python Multithreading CRUD Operations in Python Databases MySQL with Python (mysql-connector-python)ORM Introduction with SQLAlchemy PostgreSQL with Python (psycopg2)SQLite3 with Python Introduction to Django Django Models and ORM Django REST API with DRF Django Views Introduction to Flask Flask Routing Flask Templates with Jinja2 Email Automation with Python PDF Automation with Python Selenium Web Automation with Python Task Scheduler with Python Web Scraping with Python Data Analysis with Python Data Cleaning with Python Data Visualization Project Data Visualization with Matplotlib NumPy Introduction Pandas Introduction Statistical Visualization with Seaborn Classification Algorithms Clustering Algorithms Introduction to Machine Learning Machine Learning Project: Churn Prediction Model Evaluation and Selection Regression Algorithms Save pipeline (includes preprocessing + model)AI Assistant Project Calculator Project Chatbot Project Expense Tracker Project Face Detection Project Password Generator Project Weather App Project Python Coding Interview Questions Python OOP Interview Questions and Answers Python Master Cheatsheet Python Interview Questions and Answers Python Best Practices Common Python Errors and Solutions Python Debugging Techniques Python Learning Resources

Python Multiprocessing

Master Python multiprocessing with Process class, Pool, shared memory, inter-process communication, pipes, queues, and comparison with multithreading.

Introduction

Multiprocessing provides true parallel execution in Python. Each process uses its own memory space and runs independently of the GIL (Global Interpreter Lock). This makes it ideal for CPU-bound tasks.

Multithreading vs Multiprocessing:

Threads: Shared memory, GIL limited, I/O-bound
Processes: Separate memory, true parallelism, CPU-bound

2. Process Class — Full Control

Example

import multiprocessing import os import time class ComputeProcess(multiprocessing.Process): """Custom process class""" def __init__(self, task_id, data, result_dict): super().__init__() self.task_id = task_id self.data = data self.result_dict = result_dict def run(self): """Override run() for process logic""" pid = os.getpid() print(f"Process {self.task_id} (PID: {pid}) starting...") # CPU-bound computation result = sum(x**2 for x in range(self.data)) self.result_dict[self.task_id] = result print(f"Process {self.task_id} done: {result}") if __name__ == "__main__": # Manager for shared dict across processes with multiprocessing.Manager() as manager: results = manager.dict() processes = [ ComputeProcess(i, 100_000 * (i+1), results) for i in range(4) ] for p in processes: p.start() for p in processes: p.join() print("\nResults:") for task_id, result in sorted(results.items()): print(f" Task {task_id}: {result:,}")

Output

Process 0 (PID: 23401) starting...
Process 1 (PID: 23402) starting...
Process 2 (PID: 23403) starting...
Process 3 (PID: 23404) starting...
Process 0 done: 333328333350000
Process 1 done: 2666626666700000
Process 2 done: 8999925000050000
Process 3 done: 21333173333400000

Results:
  Task 0: 333,328,333,350,000
  Task 1: 2,666,626,666,700,000
  Task 2: 8,999,925,000,050,000
  Task 3: 21,333,173,333,400,000

3. Process Pool — map() and starmap()

Example

import multiprocessing import time def compute_square(n): """CPU-bound function""" return n * n def compute_power(base, exponent): """Multiple arguments""" return base ** exponent if __name__ == "__main__": numbers = list(range(1, 11)) # Pool with number of CPUs cpu_count = multiprocessing.cpu_count() print(f"CPUs available: {cpu_count}") with multiprocessing.Pool(processes=cpu_count) as pool: # map() — single argument squares = pool.map(compute_square, numbers) print(f"Squares: {squares}") # Output: Squares: [1, 4, 9, 16, 25, 36, 49, 64, 81, 100] # starmap() — multiple arguments pairs = [(2, i) for i in range(1, 11)] powers = pool.starmap(compute_power, pairs) print(f"Powers of 2: {powers}") # Output: Powers of 2: [2, 4, 8, 16, 32, 64, 128, 256, 512, 1024] # map_async() — non-blocking async_result = pool.map_async(compute_square, range(5)) # Do other work here... result = async_result.get(timeout=30) print(f"Async squares: {result}") # Output: Async squares: [0, 1, 4, 9, 16] # imap() — iterator (memory efficient) for result in pool.imap(compute_square, numbers): print(result, end=" ") print()

Output

CPUs available: 8
Squares: [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
Powers of 2: [2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]
Async squares: [0, 1, 4, 9, 16]
1 4 9 16 25 36 49 64 81 100

📝 Hindi Explanation

Pool automatically creates and manages processes. map() takes a list of inputs, applies the function to each, and returns a results list in order. This is the simplest way to parallelize batch work.

4. Shared Memory — Value and Array

Example

import multiprocessing import time def increment_counter(counter, lock, n): """Shared counter increment karo""" for _ in range(n): with lock: counter.value += 1 def fill_array(arr, start, end, value): """Shared array fill karo""" for i in range(start, end): arr[i] = value if __name__ == "__main__": # Shared integer value counter = multiprocessing.Value('i', 0) # 'i' = signed integer lock = multiprocessing.Lock() processes = [ multiprocessing.Process(target=increment_counter, args=(counter, lock, 1000)) for _ in range(5) ] for p in processes: p.start() for p in processes: p.join() print(f"Final counter: {counter.value}") # Always 5000 # Shared array arr = multiprocessing.Array('d', [0.0] * 10) # 'd' = double processes = [ multiprocessing.Process(target=fill_array, args=(arr, i*2, i*2+2, float(i))) for i in range(5) ] for p in processes: p.start() for p in processes: p.join() print(f"Shared array: {list(arr)}") # Output: Shared array: [0.0, 0.0, 1.0, 1.0, 2.0, 2.0, 3.0, 3.0, 4.0, 4.0]

Output

Final counter: 5000
Shared array: [0.0, 0.0, 1.0, 1.0, 2.0, 2.0, 3.0, 3.0, 4.0, 4.0]

5. Inter-Process Communication — Pipe

Example

import multiprocessing def sender(conn, messages): """Messages send karo""" for msg in messages: print(f"Sending: {msg}") conn.send(msg) conn.send(None) # Sentinel conn.close() def receiver(conn): """Messages receive karo""" results = [] while True: msg = conn.recv() if msg is None: break print(f"Received: {msg}") results.append(msg) conn.close() return results if __name__ == "__main__": # Pipe create karo — two-way communication parent_conn, child_conn = multiprocessing.Pipe() messages = ["Hello", "World", "Python", "Multiprocessing"] p_sender = multiprocessing.Process(target=sender, args=(child_conn, messages)) p_receiver = multiprocessing.Process(target=receiver, args=(parent_conn,)) p_sender.start() p_receiver.start() p_sender.join() p_receiver.join() print("All messages transferred!") # Two-way communication def worker_two_way(conn): while True: msg = conn.recv() if msg == "quit": break response = f"Processed: {msg.upper()}" conn.send(response) conn.close() if __name__ == "__main__": parent, child = multiprocessing.Pipe(duplex=True) p = multiprocessing.Process(target=worker_two_way, args=(child,)) p.start() for msg in ["hello", "world", "python"]: parent.send(msg) response = parent.recv() print(f"Request: {msg} -> Response: {response}") parent.send("quit") p.join()

Output

Sending: Hello
Sending: World
Sending: Python
Sending: Multiprocessing
Received: Hello
Received: World
Received: Python
Received: Multiprocessing
All messages transferred!
Request: hello -> Response: Processed: HELLO
Request: world -> Response: Processed: WORLD
Request: python -> Response: Processed: PYTHON

📝 Hindi Explanation

Pipe is a direct communication channel between two processes. Pipe() returns a pair of connection objects — one for the parent and one for the child. For more than 2 processes, use Queue instead.

6. Inter-Process Communication — Queue

Example

import multiprocessing import time import random def producer(queue, num_items): """Queue mein items daalo""" for i in range(num_items): item = f"data-{i}" queue.put(item) print(f"Produced: {item}") time.sleep(random.uniform(0.01, 0.1)) # Sentinel values — workers stop karo for _ in range(3): queue.put(None) print("Producer done") def consumer(queue, worker_id, result_list): """Take items from queue and process them""" results = [] while True: item = queue.get() if item is None: print(f"Consumer-{worker_id} stopping") break # Process result = item.upper() results.append(result) print(f"Consumer-{worker_id}: {item} -> {result}") time.sleep(random.uniform(0.05, 0.15)) result_list.extend(results) if __name__ == "__main__": queue = multiprocessing.Queue(maxsize=5) # Buffer size 5 with multiprocessing.Manager() as manager: all_results = manager.list() prod = multiprocessing.Process(target=producer, args=(queue, 10)) consumers = [ multiprocessing.Process(target=consumer, args=(queue, i, all_results)) for i in range(3) ] prod.start() for c in consumers: c.start() prod.join() for c in consumers: c.join() print(f"\nTotal results: {len(all_results)}")

Output

Produced: data-0
Produced: data-1
Consumer-0: data-0 -> DATA-0
Consumer-1: data-1 -> DATA-1
...
Total results: 10

7. Manager — Shared State

Example

import multiprocessing def worker_with_manager(shared_dict, shared_list, key, value, n): """Shared Manager objects update karo""" shared_dict[key] = value * n shared_list.append(f"{key}:{value}") if __name__ == "__main__": with multiprocessing.Manager() as manager: # Manager objects — proxy-based shared state shared_dict = manager.dict() shared_list = manager.list() shared_value = manager.Value('i', 0) processes = [ multiprocessing.Process( target=worker_with_manager, args=(shared_dict, shared_list, f"key{i}", i*10, 2) ) for i in range(5) ] for p in processes: p.start() for p in processes: p.join() print(f"Shared dict: {dict(shared_dict)}") print(f"Shared list: {list(shared_list)}")

Output

Shared dict: {'key0': 0, 'key1': 20, 'key2': 40, 'key3': 60, 'key4': 80}
Shared list: ['key0:0', 'key1:10', 'key2:20', 'key3:30', 'key4:40']

8. ProcessPoolExecutor (concurrent.futures)

Example

from concurrent.futures import ProcessPoolExecutor, as_completed import math import time def is_prime(n): """CPU-bound: primality test""" if n < 2: return False if n == 2: return True if n % 2 == 0: return False for i in range(3, int(math.sqrt(n)) + 1, 2): if n % i == 0: return False return True def count_primes_in_range(start, end): """Count primes in a range""" return sum(1 for n in range(start, end) if is_prime(n)) if __name__ == "__main__": # Split work into chunks ranges = [(i*100000, (i+1)*100000) for i in range(8)] # Sequential start = time.time() seq_results = [count_primes_in_range(s, e) for s, e in ranges] seq_time = time.time() - start print(f"Sequential: {sum(seq_results)} primes in {seq_time:.2f}s") # Parallel with ProcessPoolExecutor start = time.time() with ProcessPoolExecutor() as executor: futures = {executor.submit(count_primes_in_range, s, e): (s, e) for s, e in ranges} par_results = [] for future in as_completed(futures): par_results.append(future.result()) par_time = time.time() - start print(f"Parallel: {sum(par_results)} primes in {par_time:.2f}s") print(f"Speedup: {seq_time/par_time:.2f}x")

Output

Sequential: 54321 primes in 3.45s
Parallel: 54321 primes in 0.92s
Speedup: 3.75x

📝 Hindi Explanation

ProcessPoolExecutor is the high-level interface for multiprocessing. Use it like a Thread Pool, but with processes. Provides true speedup in CPU-bound tasks because each process has its own GIL.

9. Multiprocessing vs Multithreading Comparison

Example

import multiprocessing import threading import time def cpu_bound(n): """Pure CPU computation""" return sum(i*i for i in range(n)) def measure_time(approach, func, args_list): start = time.time() if approach == "sequential": results = [func(*args) for args in args_list] elif approach == "threading": threads = [threading.Thread(target=func, args=args) for args in args_list] for t in threads: t.start() for t in threads: t.join() results = [] elif approach == "multiprocessing": with multiprocessing.Pool() as pool: results = pool.starmap(func, args_list) elapsed = time.time() - start return elapsed if __name__ == "__main__": tasks = [(500_000,)] * 8 t1 = measure_time("sequential", cpu_bound, tasks) t2 = measure_time("threading", cpu_bound, tasks) t3 = measure_time("multiprocessing", cpu_bound, tasks) print(f"Sequential: {t1:.3f}s") print(f"Threading: {t2:.3f}s (GIL limits benefit)") print(f"Multiprocessing: {t3:.3f}s (true parallelism!)") print(f"Speedup: {t1/t3:.2f}x")

Output

Sequential:      2.456s
Threading:       2.398s (GIL limits benefit)
Multiprocessing: 0.678s (true parallelism!)
Speedup:         3.62x

10. Best Practices

Example

import multiprocessing # ✅ Always use if __name__ == "__main__" if __name__ == "__main__": p = multiprocessing.Process(target=lambda: None) p.start() p.join() # ✅ Use Pool for batch CPU work with multiprocessing.Pool() as pool: results = pool.map(str.upper, ["hello", "world"]) # ✅ Use Manager for complex shared state with multiprocessing.Manager() as m: shared = m.dict() # ✅ Use Queue for producer-consumer q = multiprocessing.Queue() # ✅ Terminate stuck processes p = multiprocessing.Process(target=lambda: None) p.start() p.join(timeout=5) if p.is_alive(): p.terminate() p.join() # ❌ Don't share mutable objects directly (use Manager or Value/Array) # shared_list = [] # Not accessible from child process! # ❌ Don't create too many processes (overhead) # Optimal: num_processes = os.cpu_count() import os optimal = os.cpu_count() print(f"Optimal process count: {optimal}")

Output

Optimal process count: 8

Summary

Tool	Best For
`Process`	Manual control over single processes
`Pool.map()`	Batch CPU work, uniform tasks
`Pool.starmap()`	Multiple arguments per task
`Queue`	Producer-consumer, task queues
`Pipe`	Direct parent-child communication
`Value`/`Array`	Simple shared primitives
`Manager`	Complex shared data structures
`ProcessPoolExecutor`	High-level API, futures

Key Rule: Use multiprocessing for CPU-bound (calculations, image processing, ML training), and threading for I/O-bound (network requests, file I/O, database queries).

⚠️ Common Mistakes

1. ❌ `if name == "main"` Guard Missing

Forgetting this on Windows is the most common mistake. Without it, infinite process spawning can occur (fork bomb effect) and the system will hang.

python example

# ❌ WRONG — Windows pe crash karega
import multiprocessing

def worker():
    print("Working...")

p = multiprocessing.Process(target=worker)
p.start()  # RuntimeError on Windows!

# ✅ CORRECT
if __name__ == "__main__":
    p = multiprocessing.Process(target=worker)
    p.start()
    p.join()

Memory is not shared between processes. If you directly modify a list or dict, the changes won't be visible to other processes.

Example

# ❌ WRONG — child process changes are not visible to parent results = [] def worker(results): results.append("done") # This won't reflect in parent! # ✅ CORRECT — Manager use karo with multiprocessing.Manager() as manager: results = manager.list() p = multiprocessing.Process(target=worker, args=(results,)) p.start() p.join() print(results) # ['done'] ✅

3. ❌ Lock Use Na Karna with Shared Memory

Modifying a shared Value or Array without a Lock creates race conditions. The counter's value will be unpredictable.

python example

# ❌ WRONG — Race condition
counter = multiprocessing.Value('i', 0)
def bad_increment(counter):
    for _ in range(1000):
        counter.value += 1  # NOT atomic!

# ✅ CORRECT — Lock use karo
lock = multiprocessing.Lock()
def good_increment(counter, lock):
    for _ in range(1000):
        with lock:
            counter.value += 1

4. ❌ Too Many Processes Create Karna

Creating more processes than CPU cores DECREASES performance due to context switching overhead. The optimal count is os.cpu_count().

python example

# ❌ WRONG — 100 processes on 8-core machine
pool = multiprocessing.Pool(processes=100)  # Overhead!

# ✅ CORRECT
pool = multiprocessing.Pool(processes=os.cpu_count())

5. ❌ Lambda ya Local Functions Pass Karna

In multiprocessing, functions must be pickled to be sent to child processes. Lambda and nested functions cannot be pickled.

python example

# ❌ WRONG — PicklingError
p = multiprocessing.Process(target=lambda: print("hi"))

# ✅ CORRECT — Top-level named function use karo
def greet():
    print("hi")

p = multiprocessing.Process(target=greet)

6. ❌ join() Call Na Karna

Without join(), the main process won't wait for the child to finish. Orphan processes will be created and resources will leak.

python example

# ❌ WRONG — process orphan ho sakta hai
p = multiprocessing.Process(target=worker)
p.start()
# Program exits, process still running!

# ✅ CORRECT
p.start()
p.join()  # Wait for completion

7. ❌ Large Objects as Arguments Pass Karna

Arguments are sent to processes via pickle/serialization. If you pass a 100MB DataFrame, a copy is made for each process — massive memory waste!

Example

# ❌ WRONG — huge data copied to each process pool.map(process_chunk, [huge_dataframe] * 4) # ✅ CORRECT — pass file path, load in process pool.map(process_file_chunk, ["data_part1.csv", "data_part2.csv", ...])

✅ Key Takeaways

🔑 Multiprocessing = True Parallelism — Each process has its own GIL and memory space, so CPU-bound tasks get real speedup.

🔑 if __name__ == "__main__" — This guard is MANDATORY on Windows. Best practice is to use it everywhere regardless of OS.

🔑 Pool.map() for Batch Work — When you need to run the same function on multiple inputs, Pool is the cleanest and most efficient approach.

🔑 Shared Memory needs Lock — ALWAYS use a Lock when working with Value and Array, otherwise race conditions will occur.

🔑 Manager for Complex Data — Use multiprocessing.Manager() to share complex objects like Dict, List, and Namespace between processes.

🔑 Queue for Producer-Consumer — Queue is the best option for safe communication between multiple producers and consumers.

🔑 ProcessPoolExecutor = Modern API — Part of the concurrent.futures module, offering a simple and clean interface with Future objects.

🔑 Optimal Processes = CPU Count — Creating more processes than os.cpu_count() is counterproductive.

🔑 Pickling Constraint — Everything sent to child processes (functions, arguments) must be picklable. Lambda, closures, and complex objects will fail.

🔑 CPU-bound → Multiprocessing, I/O-bound → Threading — This is the golden rule. The wrong choice yields no performance benefit.

❓ FAQ

Q1: What is the difference between Multiprocessing and Multithreading?

Answer: In multiprocessing, each task runs in its own separate process with its own memory space and GIL. In multithreading, all threads share the same process memory. Multiprocessing gives true parallelism for CPU-bound work; threading is better for I/O-bound tasks.

Q2: How many processes should you put in a Pool?

Answer: Generally os.cpu_count() is optimal. If tasks involve some I/O, try cpu_count() + 1 or cpu_count() * 2. But more isn't always better — excess processes increase context switching overhead.

Q3: Can variables be shared directly between processes?

Answer: No! Each process has its own memory copy. To share data, you must explicitly use multiprocessing.Value, multiprocessing.Array, Manager objects, or shared_memory (Python 3.8+).

Q4: "PicklingError" kyun aata hai?

Answer: Multiprocessing sends arguments and functions to child processes by pickling (serializing) them. Lambda functions, nested functions, and some complex objects can't be pickled, hence they can't be used with multiprocessing.

Q5: What is the difference between `fork` and `spawn` start methods?

Answer: fork (Linux default) — creates an exact copy of the parent process, fast but unsafe with threads. spawn (Windows default) — starts a fresh Python interpreter and imports the module, slower but safer and more portable.

Q6: Is exception handling different in multiprocessing?

Answer: Yes! If an exception occurs in a child process, it doesn't automatically propagate to the parent. Pool.map() and ProcessPoolExecutor capture exceptions and re-raise them when you access results. For raw Process, you need explicit error handling.

Q7: Memory consumption is too high — what should I do?

Answer: Each process has its own memory copy. Solutions: (1) Pass large data as file paths; load within the process. (2) Use Pool.imap() or imap_unordered() for streaming results. (3) Use shared_memory for zero-copy access. (4) Reduce the number of processes.

Q8: What is a multiprocessing daemon process?

Answer: Setting p.daemon = True makes the process a daemon. Daemon processes terminate automatically when the main process exits. They're useful for background tasks that don't need explicit cleanup. But daemon processes cannot spawn child processes of their own.

🎯 Interview Questions

Q1: What is the GIL in Python and how does multiprocessing bypass it?

Answer: GIL (Global Interpreter Lock) is a mutex in the Python interpreter that ensures only one thread executes Python bytecode at a time. This prevents true thread-level parallelism for CPU-bound work. Multiprocessing bypasses it by creating separate processes, each with its own interpreter and GIL.

Q2: `Pool.map()` vs `Pool.apply_async()` — when to use which?

Answer: Pool.map() is a blocking call — results are only available after all tasks complete. Best when you have uniform tasks and need all results together. Pool.apply_async() returns immediately with an AsyncResult — best when tasks have varying durations and you want to process results as they arrive.

Q3: How can deadlock occur in multiprocessing? Give an example.

Answer: Deadlock occurs when two or more processes wait for each other's resources:

python example

lock1 = multiprocessing.Lock()
lock2 = multiprocessing.Lock()

def process_a():
    lock1.acquire()
    time.sleep(0.1)
    lock2.acquire()  # Waits for lock2, held by process_b

def process_b():
    lock2.acquire()
    time.sleep(0.1)
    lock1.acquire()  # Waits for lock1, held by process_a
# DEADLOCK! Both will keep waiting for each other.

Solution: Always acquire locks in same order across all processes, ya lock.acquire(timeout=5) use karo.

Q4: `multiprocessing.Queue` vs `multiprocessing.Pipe` — when to use which?

Answer: Pipe is only for 2 endpoints (parent ↔ child). It's fast because it's a direct connection. Queue supports multiple producers and consumers, is thread/process safe, but slightly slower. Use Pipe for simple bidirectional communication; Queue for complex architectures.

Q5: Are `multiprocessing.Value` and `multiprocessing.Array` thread-safe?

Answer: Individual operations (read/write) are atomic, but compound operations (read-modify-write like counter.value += 1) are NOT atomic. This is why you always need a Lock for compound operations on shared state.

Q6: What is Python 3.8+'s `multiprocessing.shared_memory` and when should you use it?

Answer: shared_memory.SharedMemory creates a named shared memory block that multiple processes can access directly without pickling/copying. Use for large NumPy arrays or binary data where serialization overhead is unacceptable.

Example

from multiprocessing import shared_memory import numpy as np # Process 1: Create shm = shared_memory.SharedMemory(create=True, size=1000, name="my_data") arr = np.ndarray((100,), dtype=np.float64, buffer=shm.buf) arr[:] = range(100) # Process 2: Access (zero-copy!) existing_shm = shared_memory.SharedMemory(name="my_data") arr2 = np.ndarray((100,), dtype=np.float64, buffer=existing_shm.buf) print(arr2[0]) # 0.0

Q7: What happens if a worker crashes in a process pool?

Answer: If using Pool.map() and a worker raises an exception, it's re-raised in the main process. The Pool keeps the remaining workers running. With apply_async(), the exception is stored in the AsyncResult — access it via .get(). Dead workers are replaced automatically.

Q8: How do you handle logging in multiprocessing?

Answer: Writing to the same log file from multiple processes is unsafe (file corruption). Solutions: (1) QueueHandler + QueueListener — all processes send log records to a queue, one listener writes them. (2) Separate log files per process. (3) logging.handlers.SocketHandler for centralized logging.

Q9: What is the role of the `chunksize` parameter in `Pool.map()`?

Answer: chunksize determines how many items are given to a worker in a single batch. By default, Pool divides items equally. Small chunks increase IPC overhead but improve load balancing. Large chunks reduce overhead but may cause uneven work distribution. Rule of thumb: len(data) / (pool_size * 4).

Q10: When should you use multiprocessing in real-world projects — give 3 practical examples.

Answer:

Image Processing Pipeline — Resizing/compressing 1000 images. Each image is independent and CPU-intensive → Pool.map() is perfect.
Data Science / ML — Feature engineering on a large dataset. Split DataFrame into chunks, process in parallel, merge results.
Web Scraping + Processing — After downloading pages (threading), parsing HTML and NLP analysis (CPU-bound) → use multiprocessing for the parsing step.

Example

# Real example: Parallel image processing from concurrent.futures import ProcessPoolExecutor from PIL import Image import os def resize_image(filepath): img = Image.open(filepath) img = img.resize((800, 600)) output = filepath.replace("originals/", "resized/") img.save(output) return output if __name__ == "__main__": images = [f"originals/{f}" for f in os.listdir("originals/")] with ProcessPoolExecutor() as executor: results = list(executor.map(resize_image, images)) print(f"Resized {len(results)} images!")

Output

Resized 150 images!

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Python Multiprocessing.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Python Master Course topic.

Search Terms

python-master-course, python master course, python, master, course, advanced, multiprocessing, python multiprocessing

Continue learning this concept

Advanced PythonPython Asynchronous Programming with asyncioDeep dive into Python async/await, asyncio event loop, coroutines, tasks, async generators, aiohttp, and real-world asynchronous patterns with Hindi explanations.Advanced PythonPython Context ManagersComprehensive guide to Python context managers using with statement, __enter__ and __exit__ methods, contextlib module, nested context managers and custom resource management.Advanced PythonAdvanced Python DecoratorsMaster advanced Python decorators including decorator factories, class decorators, stacking decorators, functools.wraps, parameterized decorators, and real-world patterns.Advanced PythonAdvanced Python GeneratorsDeep dive into Python generators, yield keyword, generator expressions, send(), throw(), close() methods, pipelines, and coroutines with Hindi explanations.Advanced PythonPython Iterators - Advanced GuideMaster Python iterators with __iter__, __next__, custom iterator classes, infinite iterators, and real-world use cases with detailed examples and Hindi explanations.Advanced PythonPython Memory ManagementComprehensive guide to Python memory management, reference counting, garbage collection, memory profiling, optimization techniques, and best practices with Hindi explanations.

PreviousPython Memory Management NextPython Multithreading

Python Notes

Python Multiprocessing

Master Python multiprocessing with Process class, Pool, shared memory, inter-process communication, pipes, queues, and comparison with multithreading.

Introduction

Multithreading vs Multiprocessing:

Threads: Shared memory, GIL limited, I/O-bound
Processes: Separate memory, true parallelism, CPU-bound

2. Process Class — Full Control

Example

Output

Process 0 (PID: 23401) starting...
Process 1 (PID: 23402) starting...
Process 2 (PID: 23403) starting...
Process 3 (PID: 23404) starting...
Process 0 done: 333328333350000
Process 1 done: 2666626666700000
Process 2 done: 8999925000050000
Process 3 done: 21333173333400000

Results:
  Task 0: 333,328,333,350,000
  Task 1: 2,666,626,666,700,000
  Task 2: 8,999,925,000,050,000
  Task 3: 21,333,173,333,400,000

3. Process Pool — map() and starmap()

Example

Output

CPUs available: 8
Squares: [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
Powers of 2: [2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]
Async squares: [0, 1, 4, 9, 16]
1 4 9 16 25 36 49 64 81 100

📝 Hindi Explanation

Pool automatically creates and manages processes. map() takes a list of inputs, applies the function to each, and returns a results list in order. This is the simplest way to parallelize batch work.

4. Shared Memory — Value and Array

Example

Output

Final counter: 5000
Shared array: [0.0, 0.0, 1.0, 1.0, 2.0, 2.0, 3.0, 3.0, 4.0, 4.0]

5. Inter-Process Communication — Pipe

Example

Output

Sending: Hello
Sending: World
Sending: Python
Sending: Multiprocessing
Received: Hello
Received: World
Received: Python
Received: Multiprocessing
All messages transferred!
Request: hello -> Response: Processed: HELLO
Request: world -> Response: Processed: WORLD
Request: python -> Response: Processed: PYTHON

📝 Hindi Explanation

Pipe is a direct communication channel between two processes. Pipe() returns a pair of connection objects — one for the parent and one for the child. For more than 2 processes, use Queue instead.

6. Inter-Process Communication — Queue

Example

Output

Produced: data-0
Produced: data-1
Consumer-0: data-0 -> DATA-0
Consumer-1: data-1 -> DATA-1
...
Total results: 10

7. Manager — Shared State

Example

Output

Shared dict: {'key0': 0, 'key1': 20, 'key2': 40, 'key3': 60, 'key4': 80}
Shared list: ['key0:0', 'key1:10', 'key2:20', 'key3:30', 'key4:40']

8. ProcessPoolExecutor (concurrent.futures)

Example

Output

Sequential: 54321 primes in 3.45s
Parallel: 54321 primes in 0.92s
Speedup: 3.75x

📝 Hindi Explanation

ProcessPoolExecutor is the high-level interface for multiprocessing. Use it like a Thread Pool, but with processes. Provides true speedup in CPU-bound tasks because each process has its own GIL.

9. Multiprocessing vs Multithreading Comparison

Example

Output

Sequential:      2.456s
Threading:       2.398s (GIL limits benefit)
Multiprocessing: 0.678s (true parallelism!)
Speedup:         3.62x

10. Best Practices

Example

Output

Optimal process count: 8

Summary

Tool	Best For
`Process`	Manual control over single processes
`Pool.map()`	Batch CPU work, uniform tasks
`Pool.starmap()`	Multiple arguments per task
`Queue`	Producer-consumer, task queues
`Pipe`	Direct parent-child communication
`Value`/`Array`	Simple shared primitives
`Manager`	Complex shared data structures
`ProcessPoolExecutor`	High-level API, futures

Key Rule: Use multiprocessing for CPU-bound (calculations, image processing, ML training), and threading for I/O-bound (network requests, file I/O, database queries).

⚠️ Common Mistakes

1. ❌ `if name == "main"` Guard Missing

Forgetting this on Windows is the most common mistake. Without it, infinite process spawning can occur (fork bomb effect) and the system will hang.

python example

# ❌ WRONG — Windows pe crash karega
import multiprocessing

def worker():
    print("Working...")

p = multiprocessing.Process(target=worker)
p.start()  # RuntimeError on Windows!

# ✅ CORRECT
if __name__ == "__main__":
    p = multiprocessing.Process(target=worker)
    p.start()
    p.join()

Memory is not shared between processes. If you directly modify a list or dict, the changes won't be visible to other processes.

Example

3. ❌ Lock Use Na Karna with Shared Memory

Modifying a shared Value or Array without a Lock creates race conditions. The counter's value will be unpredictable.

python example

# ❌ WRONG — Race condition
counter = multiprocessing.Value('i', 0)
def bad_increment(counter):
    for _ in range(1000):
        counter.value += 1  # NOT atomic!

# ✅ CORRECT — Lock use karo
lock = multiprocessing.Lock()
def good_increment(counter, lock):
    for _ in range(1000):
        with lock:
            counter.value += 1

4. ❌ Too Many Processes Create Karna

Creating more processes than CPU cores DECREASES performance due to context switching overhead. The optimal count is os.cpu_count().

python example

# ❌ WRONG — 100 processes on 8-core machine
pool = multiprocessing.Pool(processes=100)  # Overhead!

# ✅ CORRECT
pool = multiprocessing.Pool(processes=os.cpu_count())

5. ❌ Lambda ya Local Functions Pass Karna

In multiprocessing, functions must be pickled to be sent to child processes. Lambda and nested functions cannot be pickled.

python example

# ❌ WRONG — PicklingError
p = multiprocessing.Process(target=lambda: print("hi"))

# ✅ CORRECT — Top-level named function use karo
def greet():
    print("hi")

p = multiprocessing.Process(target=greet)

6. ❌ join() Call Na Karna

Without join(), the main process won't wait for the child to finish. Orphan processes will be created and resources will leak.

python example

# ❌ WRONG — process orphan ho sakta hai
p = multiprocessing.Process(target=worker)
p.start()
# Program exits, process still running!

# ✅ CORRECT
p.start()
p.join()  # Wait for completion

7. ❌ Large Objects as Arguments Pass Karna

Arguments are sent to processes via pickle/serialization. If you pass a 100MB DataFrame, a copy is made for each process — massive memory waste!

Example

✅ Key Takeaways

🔑 Multiprocessing = True Parallelism — Each process has its own GIL and memory space, so CPU-bound tasks get real speedup.

🔑 if __name__ == "__main__" — This guard is MANDATORY on Windows. Best practice is to use it everywhere regardless of OS.

🔑 Pool.map() for Batch Work — When you need to run the same function on multiple inputs, Pool is the cleanest and most efficient approach.

🔑 Shared Memory needs Lock — ALWAYS use a Lock when working with Value and Array, otherwise race conditions will occur.

🔑 Manager for Complex Data — Use multiprocessing.Manager() to share complex objects like Dict, List, and Namespace between processes.

🔑 Queue for Producer-Consumer — Queue is the best option for safe communication between multiple producers and consumers.

🔑 ProcessPoolExecutor = Modern API — Part of the concurrent.futures module, offering a simple and clean interface with Future objects.

🔑 Optimal Processes = CPU Count — Creating more processes than os.cpu_count() is counterproductive.

🔑 Pickling Constraint — Everything sent to child processes (functions, arguments) must be picklable. Lambda, closures, and complex objects will fail.

🔑 CPU-bound → Multiprocessing, I/O-bound → Threading — This is the golden rule. The wrong choice yields no performance benefit.

❓ FAQ

Q1: What is the difference between Multiprocessing and Multithreading?

Q2: How many processes should you put in a Pool?

Q3: Can variables be shared directly between processes?

Answer: No! Each process has its own memory copy. To share data, you must explicitly use multiprocessing.Value, multiprocessing.Array, Manager objects, or shared_memory (Python 3.8+).

Q4: "PicklingError" kyun aata hai?

Q5: What is the difference between `fork` and `spawn` start methods?

Q6: Is exception handling different in multiprocessing?

Q7: Memory consumption is too high — what should I do?

Q8: What is a multiprocessing daemon process?

🎯 Interview Questions

Q1: What is the GIL in Python and how does multiprocessing bypass it?

Q2: `Pool.map()` vs `Pool.apply_async()` — when to use which?

Q3: How can deadlock occur in multiprocessing? Give an example.

Answer: Deadlock occurs when two or more processes wait for each other's resources:

python example

lock1 = multiprocessing.Lock()
lock2 = multiprocessing.Lock()

def process_a():
    lock1.acquire()
    time.sleep(0.1)
    lock2.acquire()  # Waits for lock2, held by process_b

def process_b():
    lock2.acquire()
    time.sleep(0.1)
    lock1.acquire()  # Waits for lock1, held by process_a
# DEADLOCK! Both will keep waiting for each other.

Solution: Always acquire locks in same order across all processes, ya lock.acquire(timeout=5) use karo.

Q4: `multiprocessing.Queue` vs `multiprocessing.Pipe` — when to use which?

Q5: Are `multiprocessing.Value` and `multiprocessing.Array` thread-safe?

Q6: What is Python 3.8+'s `multiprocessing.shared_memory` and when should you use it?

Example

Q7: What happens if a worker crashes in a process pool?

Q8: How do you handle logging in multiprocessing?

Q9: What is the role of the `chunksize` parameter in `Pool.map()`?

Q10: When should you use multiprocessing in real-world projects — give 3 practical examples.

Answer:

Image Processing Pipeline — Resizing/compressing 1000 images. Each image is independent and CPU-intensive → Pool.map() is perfect.
Data Science / ML — Feature engineering on a large dataset. Split DataFrame into chunks, process in parallel, merge results.
Web Scraping + Processing — After downloading pages (threading), parsing HTML and NLP analysis (CPU-bound) → use multiprocessing for the parsing step.

Example

Output

Resized 150 images!

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Python Multiprocessing.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Python Master Course topic.

Search Terms

python-master-course, python master course, python, master, course, advanced, multiprocessing, python multiprocessing

Continue learning this concept

PreviousPython Memory Management NextPython Multithreading

Introduction

2. Process Class — Full Control

3. Process Pool — map() and starmap()

📝 Hindi Explanation

4. Shared Memory — Value and Array

5. Inter-Process Communication — Pipe

📝 Hindi Explanation

6. Inter-Process Communication — Queue

7. Manager — Shared State

8. ProcessPoolExecutor (concurrent.futures)

📝 Hindi Explanation

9. Multiprocessing vs Multithreading Comparison

10. Best Practices

Summary

⚠️ Common Mistakes

1. ❌ if __name__ == "__main__" Guard Missing

2. ❌ Mutable Objects Directly Share Karna

3. ❌ Lock Use Na Karna with Shared Memory

4. ❌ Too Many Processes Create Karna

5. ❌ Lambda ya Local Functions Pass Karna

6. ❌ join() Call Na Karna

7. ❌ Large Objects as Arguments Pass Karna

✅ Key Takeaways

❓ FAQ

Q1: What is the difference between Multiprocessing and Multithreading?

Q2: How many processes should you put in a Pool?

Q3: Can variables be shared directly between processes?

Q4: "PicklingError" kyun aata hai?

Q5: What is the difference between fork and spawn start methods?

Q6: Is exception handling different in multiprocessing?

Q7: Memory consumption is too high — what should I do?

Q8: What is a multiprocessing daemon process?

🎯 Interview Questions

Q1: What is the GIL in Python and how does multiprocessing bypass it?

Q2: Pool.map() vs Pool.apply_async() — when to use which?

Q3: How can deadlock occur in multiprocessing? Give an example.

Q4: multiprocessing.Queue vs multiprocessing.Pipe — when to use which?

Q5: Are multiprocessing.Value and multiprocessing.Array thread-safe?

Q6: What is Python 3.8+'s multiprocessing.shared_memory and when should you use it?

Q7: What happens if a worker crashes in a process pool?

Q8: How do you handle logging in multiprocessing?

Q9: What is the role of the chunksize parameter in Pool.map()?

Q10: When should you use multiprocessing in real-world projects — give 3 practical examples.

Continue learning this concept

Introduction

2. Process Class — Full Control

3. Process Pool — map() and starmap()

📝 Hindi Explanation

4. Shared Memory — Value and Array

5. Inter-Process Communication — Pipe

📝 Hindi Explanation

6. Inter-Process Communication — Queue

7. Manager — Shared State

8. ProcessPoolExecutor (concurrent.futures)

📝 Hindi Explanation

9. Multiprocessing vs Multithreading Comparison

10. Best Practices

Summary

⚠️ Common Mistakes

1. ❌ if __name__ == "__main__" Guard Missing

2. ❌ Mutable Objects Directly Share Karna

3. ❌ Lock Use Na Karna with Shared Memory

4. ❌ Too Many Processes Create Karna

5. ❌ Lambda ya Local Functions Pass Karna

6. ❌ join() Call Na Karna

7. ❌ Large Objects as Arguments Pass Karna

✅ Key Takeaways

❓ FAQ

Q1: What is the difference between Multiprocessing and Multithreading?

Q2: How many processes should you put in a Pool?

Q3: Can variables be shared directly between processes?

Q4: "PicklingError" kyun aata hai?

Q5: What is the difference between fork and spawn start methods?

Q6: Is exception handling different in multiprocessing?

Q7: Memory consumption is too high — what should I do?

Q8: What is a multiprocessing daemon process?

🎯 Interview Questions

Q1: What is the GIL in Python and how does multiprocessing bypass it?

Q2: Pool.map() vs Pool.apply_async() — when to use which?

Q3: How can deadlock occur in multiprocessing? Give an example.

1. ❌ `if name == "main"` Guard Missing

Q5: What is the difference between `fork` and `spawn` start methods?

Q2: `Pool.map()` vs `Pool.apply_async()` — when to use which?

Q4: `multiprocessing.Queue` vs `multiprocessing.Pipe` — when to use which?

Q5: Are `multiprocessing.Value` and `multiprocessing.Array` thread-safe?

Q6: What is Python 3.8+'s `multiprocessing.shared_memory` and when should you use it?

Q9: What is the role of the `chunksize` parameter in `Pool.map()`?

1. ❌ `if name == "main"` Guard Missing

Q5: What is the difference between `fork` and `spawn` start methods?

Q2: `Pool.map()` vs `Pool.apply_async()` — when to use which?

Q4: `multiprocessing.Queue` vs `multiprocessing.Pipe` — when to use which?

Q5: Are `multiprocessing.Value` and `multiprocessing.Array` thread-safe?

Q6: What is Python 3.8+'s `multiprocessing.shared_memory` and when should you use it?

Q9: What is the role of the `chunksize` parameter in `Pool.map()`?