Python Notes

Hub History of Python - Complete Guide 2026 Python Applications - Complete Guide 2026 Python Features - Complete Guide 2026 What is Python? - Complete Guide 2026 Why Learn Python? - Complete Guide 2026 Your First Python Program – Hello World & Beyond 2026 How to Install Python on Windows, Mac & Linux – Complete Guide 2026 Install VS Code for Python – Complete Setup Guide with Extensions 2026 Python Environment Setup – Virtual Environments, pip & Project Structure 2026 Python IDLE – Complete Guide for Beginners 2026 Python Comments Python Data Types Python Input and Output Python Operators Python Syntax Python Strings Basics Python Type Casting Python Variables Break, Continue and Pass in Python Elif Statement in Python For Loop in Python If-Else Statement in Python If Statement in Python Nested If in Python Pattern Programs in Python While Loop in Python Decorators in Python Function Arguments in Python Functions Basics in Python Generators in Python Lambda Functions in Python Recursive Functions in Python Return Statement in Python Scope of Variables in Python Abstraction in Python Classes and Objects in Python Constructors in Python Encapsulation in Python Inheritance in Python Magic Methods (Dunder Methods) in Python Method Overriding in Python Polymorphism in Python Python Comprehensions – List, Dict, Set & Generator Python Dictionaries – Key-Value Store Python Dictionary Methods – Complete Reference Python List Methods – All Built-in Methods Python Lists – Complete Guide Python Set Methods – Complete Reference Python Sets – Unordered Unique Collections Python Tuples – Immutable Sequences Python Asynchronous Programming with asyncio Python Context Managers Advanced Python Decorators Advanced Python Generators Python Iterators - Advanced Guide Python Memory Management Python Multiprocessing Python Multithreading CRUD Operations in Python Databases MySQL with Python (mysql-connector-python)ORM Introduction with SQLAlchemy PostgreSQL with Python (psycopg2)SQLite3 with Python Introduction to Django Django Models and ORM Django REST API with DRF Django Views Introduction to Flask Flask Routing Flask Templates with Jinja2 Email Automation with Python PDF Automation with Python Selenium Web Automation with Python Task Scheduler with Python Web Scraping with Python Data Analysis with Python Data Cleaning with Python Data Visualization Project Data Visualization with Matplotlib NumPy Introduction Pandas Introduction Statistical Visualization with Seaborn Classification Algorithms Clustering Algorithms Introduction to Machine Learning Machine Learning Project: Churn Prediction Model Evaluation and Selection Regression Algorithms Save pipeline (includes preprocessing + model)AI Assistant Project Calculator Project Chatbot Project Expense Tracker Project Face Detection Project Password Generator Project Weather App Project Python Coding Interview Questions Python OOP Interview Questions and Answers Python Master Cheatsheet Python Interview Questions and Answers Python Best Practices Common Python Errors and Solutions Python Debugging Techniques Python Learning Resources

Web Scraping with Python

Complete guide to web scraping with Python using requests, BeautifulSoup4, lxml, handling pagination, scraping APIs, and best practices with Hindi explanations.

Introduction

Web Scraping is automatically extracting data from websites. Combining requests (HTTP requests) and BeautifulSoup4 (HTML parsing) in Python creates a powerful scraping toolkit.

bash example

pip install requests beautifulsoup4 lxml

2. BeautifulSoup4 — HTML Parsing

Example

from bs4 import BeautifulSoup # Sample HTML html = """ <html> <head><title>Python Tutorial</title></head> <body> <div class="container"> <h1 id="main-title">Learn Python Web Scraping</h1> <p class="intro">Python se web scraping seekho.</p> <ul class="topic-list"> <li class="topic" data-id="1">BeautifulSoup</li> <li class="topic" data-id="2">Requests</li> <li class="topic" data-id="3">Selenium</li> </ul> <a href="https://example.com/python" class="link">Python Docs</a> <a href="https://example.com/bs4" class="link">BS4 Docs</a> </div> </body> </html> """ # Parse HTML soup = BeautifulSoup(html, "lxml") # lxml is faster # Find elements title = soup.find("h1") print(f"Title text: {title.text}") print(f"Title id: {title.get('id')}") # find() — first match intro = soup.find("p", class_="intro") print(f"Intro: {intro.text}") # find_all() — all matches topics = soup.find_all("li", class_="topic") for topic in topics: print(f" Topic {topic['data-id']}: {topic.text}") # CSS Selectors with select() all_links = soup.select("a.link") for link in all_links: print(f"Link: {link.text} -> {link['href']}") # Select single element first_link = soup.select_one("a.link") print(f"First link: {first_link.text}")

3. Navigating HTML Tree

Example

from bs4 import BeautifulSoup html = """ <div class="post"> <header> <h2 class="title">Python Tutorial</h2> <span class="author">WoHoTech</span> <time datetime="2026-06-12">June 12, 2026</time> </header> <section class="content"> <p>First paragraph with <strong>bold text</strong>.</p> <p>Second paragraph with <a href="/link">a link</a>.</p> </section> <footer> <span class="tags">python, tutorial, scraping</span> </footer> </div> """ soup = BeautifulSoup(html, "lxml") post = soup.find("div", class_="post") # Parent/Children navigation header = post.find("header") print(f"Parent: {header.parent.name}") # div # Direct children for child in header.children: if child.name: # Skip text nodes print(f"Child: {child.name} — {child.text.strip()}") # All descendants all_text = post.get_text(separator=" ", strip=True) print(f"All text: {all_text[:100]}") # next_sibling / previous_sibling section = soup.find("section") prev = section.previous_sibling next_el = section.next_sibling # Attrs time_elem = soup.find("time") print(f"datetime attr: {time_elem['datetime']}") print(f"All attrs: {time_elem.attrs}") # Nested search bold_text = soup.find("p").find("strong").text print(f"Bold text: {bold_text}")

4. CSS Selectors

Example

from bs4 import BeautifulSoup html = """ <table class="data-table" id="products"> <thead> <tr> <th>Name</th><th>Price</th><th>Stock</th> </tr> </thead> <tbody> <tr class="product-row" data-id="1"> <td class="name">Laptop</td> <td class="price">₹75,000</td> <td class="stock in-stock">In Stock</td> </tr> <tr class="product-row" data-id="2"> <td class="name">Phone</td> <td class="price">₹25,000</td> <td class="stock out-of-stock">Out of Stock</td> </tr> <tr class="product-row" data-id="3"> <td class="name">Tablet</td> <td class="price">₹35,000</td> <td class="stock in-stock">In Stock</td> </tr> </tbody> </table> """ soup = BeautifulSoup(html, "lxml") # CSS selector examples print("=== CSS Selectors ===") # By class prices = soup.select("td.price") print(f"All prices: {[p.text for p in prices]}") # By id table = soup.select_one("#products") print(f"Table id: {table.get('id')}") # Descendant selector names = soup.select("table#products tbody .name") print(f"Names: {[n.text for n in names]}") # Attribute selector in_stock = soup.select("[class~=in-stock]") # class contains 'in-stock' print(f"In stock items: {len(in_stock)}") # nth-child first_row = soup.select("tbody tr:nth-child(1)") print(f"First row: {first_row[0].find('td', class_='name').text}") # data attribute row2 = soup.select("[data-id='2']") print(f"Product ID 2: {row2[0].find('td', class_='name').text}") # Multiple classes in_stock_items = soup.select("td.stock.in-stock") print(f"In stock count: {len(in_stock_items)}") # Scrape full table products = [] rows = soup.select("tbody tr.product-row") for row in rows: products.append({ "id": row.get("data-id"), "name": row.select_one(".name").text, "price": row.select_one(".price").text, "status": row.select_one(".stock").text, }) print("\nProducts table:") for p in products: print(f" {p}")

📝 Hindi Explanation

CSS Selectors are the most powerful way to find HTML elements: - tag — element type se - .class — class se - #id — ID se - [attr='val'] — attribute se - parent > child — direct child - a b — descendant (any level) - :nth-child(n) — position se

5. Real-World Scraping

Example

import requests from bs4 import BeautifulSoup import time import json class WebScraper: """Reusable web scraper class""" def __init__(self, base_url, delay=1.0): self.base_url = base_url self.delay = delay self.session = requests.Session() self.session.headers.update({ "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36", }) def fetch(self, url, params=None): """URL fetch karo with error handling""" try: response = self.session.get(url, params=params, timeout=10) response.raise_for_status() time.sleep(self.delay) # Rate limiting return response except requests.RequestException as e: print(f"Error fetching {url}: {e}") return None def parse_html(self, html_content): """HTML parse karo""" return BeautifulSoup(html_content, "lxml") def extract_links(self, soup, base_url=None): """Extract all links from the page""" links = [] for a_tag in soup.find_all("a", href=True): href = a_tag["href"] text = a_tag.text.strip() # Convert relative URLs to absolute if href.startswith("http"): full_url = href elif href.startswith("/"): full_url = f"{base_url or self.base_url}{href}" else: continue links.append({"text": text, "url": full_url}) return links def extract_table(self, soup, table_selector="table"): """Convert HTML table to list of dicts""" table = soup.select_one(table_selector) if not table: return [] headers = [th.text.strip() for th in table.select("thead th")] if not headers: headers = [th.text.strip() for th in table.select("tr:first-child th, tr:first-child td")] rows = [] for tr in table.select("tbody tr, tr:not(:first-child)"): cells = [td.text.strip() for td in tr.select("td")] if cells and len(cells) == len(headers): rows.append(dict(zip(headers, cells))) return rows # Simulated scraping (using httpbin for demo) scraper = WebScraper("https://httpbin.org") # Test with httpbin response = scraper.fetch("https://httpbin.org/html") if response: soup = scraper.parse_html(response.text) heading = soup.find("h1") if heading: print(f"Heading: {heading.text}")

6. Handling Pagination

Example

import requests from bs4 import BeautifulSoup import time def scrape_with_pagination(base_url, max_pages=5): """Multiple pages scrape karo""" all_items = [] page = 1 while page <= max_pages: print(f"Scraping page {page}...") # Build URL with page parameter url = f"{base_url}?page={page}" try: response = requests.get(url, timeout=10) response.raise_for_status() except requests.RequestException: print(f"Failed to fetch page {page}") break soup = BeautifulSoup(response.text, "lxml") # Extract items from current page items = soup.select(".item") # Adjust selector if not items: print(f"No items on page {page} — stopping") break for item in items: all_items.append({ "title": item.select_one(".title").text.strip(), "url": item.select_one("a")["href"], }) # Check for next page button next_btn = soup.select_one(".next-page") if not next_btn or "disabled" in next_btn.get("class", []): print("Last page reached") break page += 1 time.sleep(1) # Be respectful! return all_items # URL-based pagination def scrape_all_pages(base_url, page_param="page"): """Generic pagination handler""" results = [] page = 1 while True: response = requests.get(base_url, params={page_param: page}) data = response.json() # If API if not data.get("items"): break results.extend(data["items"]) if not data.get("has_next"): break page += 1 time.sleep(0.5) return results

7. Scraping JSON APIs

Example

import requests import json def scrape_json_api(api_url, params=None, auth_token=None): """Scrape data from REST API""" headers = {"Accept": "application/json"} if auth_token: headers["Authorization"] = f"Bearer {auth_token}" response = requests.get(api_url, params=params, headers=headers) if response.status_code == 200: return response.json() else: print(f"API error: {response.status_code} — {response.text}") return None # Public APIs example # JSONPlaceholder (free test API) posts = scrape_json_api("https://jsonplaceholder.typicode.com/posts", params={"_limit": 5}) if posts: for post in posts: print(f"Post {post['id']}: {post['title'][:50]}") # Handling nested JSON user = scrape_json_api("https://jsonplaceholder.typicode.com/users/1") if user: print(f"\nUser: {user['name']}") print(f"Email: {user['email']}") print(f"City: {user['address']['city']}") print(f"Company: {user['company']['name']}")

8. Scraping with Sessions and Cookies

Example

import requests # Session maintain karo (login and scrape) session = requests.Session() session.headers.update({ "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36", }) def login_and_scrape(login_url, username, password, target_url): """Login karke protected pages scrape karo""" # CSRF token get karo login_page = session.get(login_url) soup = BeautifulSoup(login_page.text, "lxml") csrf_token = soup.find("input", {"name": "_token"}) # Login karo login_data = { "username": username, "password": password, "_token": csrf_token["value"] if csrf_token else None, } response = session.post(login_url, data=login_data) if response.ok: # Logged in! Now scrape protected_page = session.get(target_url) return BeautifulSoup(protected_page.text, "lxml") return None # Cookies handling session.cookies.update({"session_id": "abc123"}) print(f"Cookies: {dict(session.cookies)}")

9. Best Practices and Ethics

Example

import requests from urllib.robotparser import RobotFileParser import time def check_robots_txt(url, user_agent="*"): """Check karo ki scraping allowed hai""" from urllib.parse import urlparse parsed = urlparse(url) robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt" rp = RobotFileParser() rp.set_url(robots_url) try: rp.read() return rp.can_fetch(user_agent, url) except: return True # Assume allowed if robots.txt not accessible # Check before scraping url = "https://example.com/data" if check_robots_txt(url): print("Scraping allowed!") else: print("Scraping NOT allowed — respect robots.txt!") # Rate limiting decorator import functools def rate_limited(calls_per_second=1): """Rate limit the requests""" min_interval = 1.0 / calls_per_second last_call = [0.0] def decorator(func): @functools.wraps(func) def wrapper(*args, **kwargs): elapsed = time.time() - last_call[0] if elapsed < min_interval: time.sleep(min_interval - elapsed) last_call[0] = time.time() return func(*args, **kwargs) return wrapper return decorator @rate_limited(calls_per_second=0.5) # Max 1 request per 2 seconds def fetch_page(url): return requests.get(url)

Summary

Tool	Use Case
`requests`	HTTP requests (GET/POST)
`BeautifulSoup4`	HTML parsing, element extraction
`lxml`	Fast HTML/XML parser (BS4 backend)
`.find()`	First matching element
`.find_all()`	All matching elements
`.select()`	CSS selector
`.select_one()`	Single CSS selector match
`session.get()`	Session with persistent headers/cookies

Ethics: Always check robots.txt, add delays between requests, don't overload servers, cache results, and use official APIs when available. Scraping is a tool — use it responsibly!

Output Examples 📋

Section 1 Output — HTTP Requests

Output

Status code: 200
Content type: application/json
Response size: 498 bytes
{"headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...", "Accept": "text/html,application/xhtml+xml..."}}
URL with params: https://httpbin.org/get?q=python+scraping&lang=en&page=1

Section 2 Output — BeautifulSoup4 HTML Parsing

Output

Title text: Learn Python Web Scraping
Title id: main-title
Intro: Python se web scraping seekho.
  Topic 1: BeautifulSoup
  Topic 2: Requests
  Topic 3: Selenium
Link: Python Docs -> https://example.com/python
Link: BS4 Docs -> https://example.com/bs4
First link: Python Docs

Section 3 Output — Navigating HTML Tree

Output

Parent: div
Child: h2 — Python Tutorial
Child: span — WoHoTech
Child: time — June 12, 2026
All text: Python Tutorial WoHoTech June 12, 2026 First paragraph with bold text. Second paragraph with a link. python, tutorial, scraping
datetime attr: 2026-06-12
All attrs: {'datetime': '2026-06-12'}
Bold text: bold text

Section 4 Output — CSS Selectors

Output

=== CSS Selectors ===
All prices: ['₹75,000', '₹25,000', '₹35,000']
Table id: products
Names: ['Laptop', 'Phone', 'Tablet']
In stock items: 2
First row: Laptop
Product ID 2: Phone
In stock count: 2

Products table:
  {'id': '1', 'name': 'Laptop', 'price': '₹75,000', 'status': 'In Stock'}
  {'id': '2', 'name': 'Phone', 'price': '₹25,000', 'status': 'Out of Stock'}
  {'id': '3', 'name': 'Tablet', 'price': '₹35,000', 'status': 'In Stock'}

Section 5 Output — Real-World Scraping

Output

Heading: Herman Melville - Moby Dick

Section 6 Output — Handling Pagination

Output

Scraping page 1...
Scraping page 2...
Scraping page 3...
No items on page 3 — stopping

Section 7 Output — JSON APIs

Output

Post 1: sunt aut facere repellat provident occaecati exc
Post 2: qui est esse
Post 3: ea molestias quasi exercitationem repellat qui i
Post 4: eum et est occaecati
Post 5: nesciunt quas odio

User: Leanne Graham
Email: Sincere@april.biz
City: Gwenborough
Company: Romaguera-Crona

Section 8 Output — Sessions and Cookies

Output

Cookies: {'session_id': 'abc123'}

Section 9 Output — Best Practices

Output

Scraping allowed!

⚠️ Common Mistakes

❌ Mistake 1: User-Agent Header Na Bhejana

python example

# ❌ Wrong — many sites block requests without User-Agent
response = requests.get("https://example.com")

# ✅ Right — always send proper User-Agent
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
response = requests.get("https://example.com", headers=headers)

🔑 Tip: Without a User-Agent, many websites will return 403 Forbidden or empty responses.

❌ Mistake 2: Error Handling Skip Karna

python example

# ❌ Wrong — will crash if site is down!
response = requests.get("https://example.com")
soup = BeautifulSoup(response.text, "lxml")

# ✅ Right — handle errors gracefully
try:
    response = requests.get("https://example.com", timeout=10)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "lxml")
except requests.RequestException as e:
    print(f"Error: {e}")

🔑 Tip: Network requests can always fail — always use try/except and set a timeout.

❌ Mistake 3: Rate Limiting Na Karna

python example

# ❌ Wrong — server pe attack jaisa lagega
for url in urls:
    response = requests.get(url)

# ✅ Right — requests ke beech delay rakho
import time
for url in urls:
    response = requests.get(url)
    time.sleep(1)  # 1 second ka gap

🔑 Tip: Without delays, the server will be overloaded and your IP will be banned.

❌ Mistake 4: Hardcoded Selectors Use Karna

python example

# ❌ Wrong — ek selector change hone pe pure code break
title = soup.select_one("body > div:nth-child(3) > div > h2")

# ✅ Right — robust selectors use karo
title = soup.select_one("h2.article-title")

🔑 Tip: Use class-based or ID-based selectors, not position-based ones. Websites change their HTML structure frequently.

❌ Mistake 5: `.text` vs `.string` Ka Confusion

python example

# ❌ Wrong — .string returns None if element has children
content = soup.find("div").string  # None if nested elements

# ✅ Right — .text ya .get_text() use karo
content = soup.find("div").get_text(strip=True)

🔑 Tip: .string only works when an element contains a single NavigableString. .get_text() is more reliable.

❌ Mistake 6: robots.txt Check Na Karna

python example

# ❌ Wrong — blindly scraping without permission check
data = requests.get("https://example.com/private-data")

# ✅ Right — check robots.txt first
from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
if rp.can_fetch("*", "https://example.com/private-data"):
    data = requests.get("https://example.com/private-data")

🔑 Tip: To avoid legal issues, always respect robots.txt. Some websites can take legal action against scraping.

❌ Mistake 7: Not Validating Data

python example

# ❌ Wrong — element None ho sakta hai
price = soup.find("span", class_="price").text  # AttributeError!

# ✅ Right — check for None first
price_elem = soup.find("span", class_="price")
price = price_elem.text if price_elem else "N/A"

🔑 Tip: Website structure can change or elements may be missing — always check for None.

✅ Key Takeaways

🔹 requests + BeautifulSoup4 = Python's most popular web scraping combo — requests handles HTTP, BeautifulSoup4 parses HTML
🔹 Always set a User-Agent header — many websites block requests or return incomplete data without one
🔹 find() vs find_all() — find() returns the first match (single element), find_all() returns a list (multiple elements)
🔹 CSS Selectors (select()) are more powerful for complex queries — handles class, id, attributes, and nested elements
🔹 Rate Limiting is essential — use time.sleep() between requests. A 1-2 second gap is respectful scraping
🔹 Use Session when cookies/login need to be maintained — requests.Session() persists headers and cookies across requests
🔹 Always check robots.txt before starting to scrape — it shows the website owner's permissions/restrictions
🔹 Error handling (try/except) is mandatory — network failures, timeouts, 404s — anything can happen at runtime
🔹 Use lxml parser for speed — it's ~10x faster than the default html.parser on large HTML documents
🔹 Prefer JSON APIs over HTML scraping — if the website has an API available, use it directly; it's more stable and faster

❓ FAQ

Q1: Web Scraping legal hai ya illegal? 🤔

Answer: Web scraping itself is not illegal, BUT — violating a website's Terms of Service, scraping copyrighted content, or collecting personal data without consent can have legal consequences. Always check robots.txt and ToS.

Q2: BeautifulSoup vs Scrapy — kaunsa use karein? 🛠️

Answer: BeautifulSoup is best for simple scripts — when you need to scrape one or two pages or do quick data extraction. Scrapy is a full-featured framework for large-scale projects — built-in pipeline, middleware, concurrent requests, data export.

Q3: Website has blocked my IP — now what? 🚫

Answer: When IP-blocked: (1) Increase rate limiting (more delay), (2) Use rotating proxies, (3) Implement User-Agent rotation, (4) Use sessions/cookies properly, (5) Consider switching to a headless browser. Prevention: always add delays and don't hammer the server.

Q4: How to scrape JavaScript-rendered pages? ⚡

Answer: BeautifulSoup can only parse static HTML. For JS-rendered content: (1) Selenium — full browser automation, (2) Playwright — modern alternative, faster than Selenium, (3) requests-html — lightweight with JS rendering support, (4) Check if the site has a REST API returning JSON directly.

Q5: How to prevent infinite loops when handling pagination? 🔄

Answer: (1) Set a max_pages limit, (2) Check if the next page button exists or is disabled, (3) Track already-visited URLs, (4) Check for empty response/no new data, (5) Compare current page content with previous — if identical, stop.

Q6: Where should you store scraped data? 💾

Answer: Depends on data size and use-case: (1) CSV/JSON files — for small data and quick analysis, (2) SQLite — for medium data and local storage, (3) PostgreSQL/MongoDB — for large-scale production with querying needs, (4) Cloud storage (S3) — for raw data archival.

Q7: `lxml` vs `html.parser` — kaunsa parser better hai? 🏎️

Answer: lxml is significantly faster (C-based) and handles broken HTML leniently. html.parser is Python built-in — no extra installation required, good for simple tasks. For production scraping with large pages, always use lxml.

Q8: How to keep API keys and credentials secure in code? 🔐

Answer: (1) Use environment variables — os.environ.get("API_KEY"), (2) Use .env file + python-dotenv library, (3) Never commit credentials to code or version control, (4) Use secret managers (AWS Secrets Manager, HashiCorp Vault) in production.

🎯 Interview Questions

Q1: What is Web Scraping and what are its key components?

Answer: Web Scraping is an automated technique for programmatically extracting data from websites. Key components: (1) HTTP Client (requests) — sends requests and receives responses, (2) HTML Parser (BeautifulSoup/lxml) — parses HTML structure, (3) Data Extraction — locates and pulls specific data, (4) Storage — saves to file/database.

Q2: What is the difference between `find()` and `find_all()`?

Answer: find() returns the first matching element (Tag object or None), while find_all() returns all matching elements as a list (empty list if none found). find() is essentially find_all(limit=1)[0]. Use find() for unique elements (IDs) and find_all() for collections (list items, table rows).

Q3: What are the advantages of using requests.Session()?

Answer: (1) Cookies persist across requests — login state is maintained, (2) Headers are reused — no need to set them on every request, (3) Connection pooling — TCP connections are reused for performance, (4) Simplified auth — set once, apply everywhere.

Q4: What is the fundamental difference between Scrapy and BeautifulSoup?

Answer: BeautifulSoup is only an HTML/XML parser — it doesn't make HTTP requests or store data by itself. Scrapy is a complete web scraping framework — built-in request scheduling, data pipelines, middleware, caching, rate limiting, concurrent requests, and export formats.

Q5: What are anti-scraping mechanisms and how do you deal with them?

Answer: Common anti-scraping techniques: (1) IP blocking — solution: rotating proxies, (2) CAPTCHA — solution: CAPTCHA solving services or avoid triggering them, (3) JavaScript rendering — solution: headless browsers, (4) Rate limiting — solution: delays and request throttling, (5) Honeypot traps — solution: check link visibility before following.

Q6: What is the difference between XPath and CSS Selectors?

Answer: CSS Selectors: simpler syntax, familiar to web developers, forward-only navigation (parent to child). XPath: more powerful, bidirectional navigation (can go from child to parent), supports text-based selection. CSS is sufficient for 90% of cases; use XPath when you need parent traversal or complex text conditions.

Q7: What is the best approach for error handling in Web Scraping?

Answer: Multi-layer error handling: (1) Network level — try/except requests.RequestException, set timeout, (2) HTTP level — response.raise_for_status() to catch 4xx/5xx, (3) Parse level — check for None before accessing attributes, (4) Data level — validate extracted data types and formats. Add retries with exponential backoff for transient errors.

Q8: What is robots.txt and what is its importance in scraping?

Answer: robots.txt is a standard file at a website's root (e.g., https://example.com/robots.txt) that tells web crawlers which paths are allowed or disallowed for scraping. Respecting it is ethical practice and can protect you from legal issues. Use urllib.robotparser to programmatically check permissions.

Q9: How do you design a data pipeline for a large-scale scraping project?

Answer: Production pipeline: (1) URL Queue — store URLs in Redis/RabbitMQ, (2) Fetcher — concurrent HTTP requests with rate limiting, (3) Parser — extract structured data from HTML, (4) Validator — check data quality and completeness, (5) Storage — write to database/file with deduplication, (6) Monitor — track success rates, errors, and performance metrics.

Q10: What are the best tools for scraping dynamic websites (SPAs)?

Answer: For Single Page Applications (React, Angular, Vue): (1) Selenium — most popular, full browser automation, but slow, (2) Playwright — Microsoft's modern alternative, supports Chromium/Firefox/WebKit, faster than Selenium, (3) Puppeteer — Google's Node.js solution, Chrome-only but very fast, (4) Splash — lightweight headless browser as a service (with Scrapy), (5) requests-html — simple JS rendering for basic SPAs.

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Web Scraping with Python.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Python Master Course topic.

Search Terms

python-master-course, python master course, python, master, course, automation, web, scraping

Continue learning this concept

AutomationEmail Automation with PythonComplete guide to email automation with Python using smtplib, email library, sending HTML emails, attachments, reading emails with IMAP, and scheduling email automation.AutomationPDF Automation with PythonMaster PDF automation with Python using PyPDF2, reportlab, pdfplumber for reading, writing, merging, splitting PDFs, extracting text/tables, and generating PDF reports.AutomationSelenium Web Automation with PythonMaster Selenium WebDriver with Python for browser automation, form filling, clicking, screenshots, handling dynamic content, waits, and headless browsing with Hindi explanations.AutomationTask Scheduler with PythonComplete guide to task scheduling in Python using schedule library, APScheduler, cron jobs, threading timers, and building automated workflows with Hindi explanations.Web DevelopmentIntroduction to DjangoLearn Django, Python Web DevelopmentDjango Models and ORMMaster Django models and the ORM. Learn to define database models, field types, relationships (ForeignKey, ManyToMany, OneToOne), migrations, QuerySets, and advanced database queries in Django.

PreviousTask Scheduler with Python NextData Analysis with Python

Python Notes

Web Scraping with Python

Complete guide to web scraping with Python using requests, BeautifulSoup4, lxml, handling pagination, scraping APIs, and best practices with Hindi explanations.

Introduction

Web Scraping is automatically extracting data from websites. Combining requests (HTTP requests) and BeautifulSoup4 (HTML parsing) in Python creates a powerful scraping toolkit.

bash example

pip install requests beautifulsoup4 lxml

2. BeautifulSoup4 — HTML Parsing

Example

3. Navigating HTML Tree

Example

4. CSS Selectors

Example

📝 Hindi Explanation

CSS Selectors are the most powerful way to find HTML elements: - tag — element type se - .class — class se - #id — ID se - [attr='val'] — attribute se - parent > child — direct child - a b — descendant (any level) - :nth-child(n) — position se

5. Real-World Scraping

Example

6. Handling Pagination

Example

7. Scraping JSON APIs

Example

8. Scraping with Sessions and Cookies

Example

9. Best Practices and Ethics

Example

Summary

Tool	Use Case
`requests`	HTTP requests (GET/POST)
`BeautifulSoup4`	HTML parsing, element extraction
`lxml`	Fast HTML/XML parser (BS4 backend)
`.find()`	First matching element
`.find_all()`	All matching elements
`.select()`	CSS selector
`.select_one()`	Single CSS selector match
`session.get()`	Session with persistent headers/cookies

Ethics: Always check robots.txt, add delays between requests, don't overload servers, cache results, and use official APIs when available. Scraping is a tool — use it responsibly!

Output Examples 📋

Section 1 Output — HTTP Requests

Output

Status code: 200
Content type: application/json
Response size: 498 bytes
{"headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...", "Accept": "text/html,application/xhtml+xml..."}}
URL with params: https://httpbin.org/get?q=python+scraping&lang=en&page=1

Section 2 Output — BeautifulSoup4 HTML Parsing

Output

Title text: Learn Python Web Scraping
Title id: main-title
Intro: Python se web scraping seekho.
  Topic 1: BeautifulSoup
  Topic 2: Requests
  Topic 3: Selenium
Link: Python Docs -> https://example.com/python
Link: BS4 Docs -> https://example.com/bs4
First link: Python Docs

Section 3 Output — Navigating HTML Tree

Output

Parent: div
Child: h2 — Python Tutorial
Child: span — WoHoTech
Child: time — June 12, 2026
All text: Python Tutorial WoHoTech June 12, 2026 First paragraph with bold text. Second paragraph with a link. python, tutorial, scraping
datetime attr: 2026-06-12
All attrs: {'datetime': '2026-06-12'}
Bold text: bold text

Section 4 Output — CSS Selectors

Output

=== CSS Selectors ===
All prices: ['₹75,000', '₹25,000', '₹35,000']
Table id: products
Names: ['Laptop', 'Phone', 'Tablet']
In stock items: 2
First row: Laptop
Product ID 2: Phone
In stock count: 2

Products table:
  {'id': '1', 'name': 'Laptop', 'price': '₹75,000', 'status': 'In Stock'}
  {'id': '2', 'name': 'Phone', 'price': '₹25,000', 'status': 'Out of Stock'}
  {'id': '3', 'name': 'Tablet', 'price': '₹35,000', 'status': 'In Stock'}

Section 5 Output — Real-World Scraping

Output

Heading: Herman Melville - Moby Dick

Section 6 Output — Handling Pagination

Output

Scraping page 1...
Scraping page 2...
Scraping page 3...
No items on page 3 — stopping

Section 7 Output — JSON APIs

Output

Post 1: sunt aut facere repellat provident occaecati exc
Post 2: qui est esse
Post 3: ea molestias quasi exercitationem repellat qui i
Post 4: eum et est occaecati
Post 5: nesciunt quas odio

User: Leanne Graham
Email: Sincere@april.biz
City: Gwenborough
Company: Romaguera-Crona

Section 8 Output — Sessions and Cookies

Output

Cookies: {'session_id': 'abc123'}

Section 9 Output — Best Practices

Output

Scraping allowed!

⚠️ Common Mistakes

❌ Mistake 1: User-Agent Header Na Bhejana

python example

# ❌ Wrong — many sites block requests without User-Agent
response = requests.get("https://example.com")

# ✅ Right — always send proper User-Agent
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
response = requests.get("https://example.com", headers=headers)

🔑 Tip: Without a User-Agent, many websites will return 403 Forbidden or empty responses.

❌ Mistake 2: Error Handling Skip Karna

python example

# ❌ Wrong — will crash if site is down!
response = requests.get("https://example.com")
soup = BeautifulSoup(response.text, "lxml")

# ✅ Right — handle errors gracefully
try:
    response = requests.get("https://example.com", timeout=10)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "lxml")
except requests.RequestException as e:
    print(f"Error: {e}")

🔑 Tip: Network requests can always fail — always use try/except and set a timeout.

❌ Mistake 3: Rate Limiting Na Karna

python example

# ❌ Wrong — server pe attack jaisa lagega
for url in urls:
    response = requests.get(url)

# ✅ Right — requests ke beech delay rakho
import time
for url in urls:
    response = requests.get(url)
    time.sleep(1)  # 1 second ka gap

🔑 Tip: Without delays, the server will be overloaded and your IP will be banned.

❌ Mistake 4: Hardcoded Selectors Use Karna

python example

# ❌ Wrong — ek selector change hone pe pure code break
title = soup.select_one("body > div:nth-child(3) > div > h2")

# ✅ Right — robust selectors use karo
title = soup.select_one("h2.article-title")

🔑 Tip: Use class-based or ID-based selectors, not position-based ones. Websites change their HTML structure frequently.

❌ Mistake 5: `.text` vs `.string` Ka Confusion

python example

# ❌ Wrong — .string returns None if element has children
content = soup.find("div").string  # None if nested elements

# ✅ Right — .text ya .get_text() use karo
content = soup.find("div").get_text(strip=True)

🔑 Tip: .string only works when an element contains a single NavigableString. .get_text() is more reliable.

❌ Mistake 6: robots.txt Check Na Karna

python example

# ❌ Wrong — blindly scraping without permission check
data = requests.get("https://example.com/private-data")

# ✅ Right — check robots.txt first
from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
if rp.can_fetch("*", "https://example.com/private-data"):
    data = requests.get("https://example.com/private-data")

🔑 Tip: To avoid legal issues, always respect robots.txt. Some websites can take legal action against scraping.

❌ Mistake 7: Not Validating Data

python example

# ❌ Wrong — element None ho sakta hai
price = soup.find("span", class_="price").text  # AttributeError!

# ✅ Right — check for None first
price_elem = soup.find("span", class_="price")
price = price_elem.text if price_elem else "N/A"

🔑 Tip: Website structure can change or elements may be missing — always check for None.

✅ Key Takeaways

🔹 requests + BeautifulSoup4 = Python's most popular web scraping combo — requests handles HTTP, BeautifulSoup4 parses HTML
🔹 Always set a User-Agent header — many websites block requests or return incomplete data without one
🔹 find() vs find_all() — find() returns the first match (single element), find_all() returns a list (multiple elements)
🔹 CSS Selectors (select()) are more powerful for complex queries — handles class, id, attributes, and nested elements
🔹 Rate Limiting is essential — use time.sleep() between requests. A 1-2 second gap is respectful scraping
🔹 Use Session when cookies/login need to be maintained — requests.Session() persists headers and cookies across requests
🔹 Always check robots.txt before starting to scrape — it shows the website owner's permissions/restrictions
🔹 Error handling (try/except) is mandatory — network failures, timeouts, 404s — anything can happen at runtime
🔹 Use lxml parser for speed — it's ~10x faster than the default html.parser on large HTML documents
🔹 Prefer JSON APIs over HTML scraping — if the website has an API available, use it directly; it's more stable and faster

❓ FAQ

Q1: Web Scraping legal hai ya illegal? 🤔

Q2: BeautifulSoup vs Scrapy — kaunsa use karein? 🛠️

Q3: Website has blocked my IP — now what? 🚫

Q4: How to scrape JavaScript-rendered pages? ⚡

Q5: How to prevent infinite loops when handling pagination? 🔄

Q6: Where should you store scraped data? 💾

Q7: `lxml` vs `html.parser` — kaunsa parser better hai? 🏎️

Q8: How to keep API keys and credentials secure in code? 🔐

🎯 Interview Questions

Q1: What is Web Scraping and what are its key components?

Q2: What is the difference between `find()` and `find_all()`?

Q3: What are the advantages of using requests.Session()?

Q4: What is the fundamental difference between Scrapy and BeautifulSoup?

Q5: What are anti-scraping mechanisms and how do you deal with them?

Q6: What is the difference between XPath and CSS Selectors?

Q7: What is the best approach for error handling in Web Scraping?

Q8: What is robots.txt and what is its importance in scraping?

Q9: How do you design a data pipeline for a large-scale scraping project?

Q10: What are the best tools for scraping dynamic websites (SPAs)?

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Web Scraping with Python.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Python Master Course topic.

Search Terms

python-master-course, python master course, python, master, course, automation, web, scraping

Continue learning this concept

PreviousTask Scheduler with Python NextData Analysis with Python

Introduction

2. BeautifulSoup4 — HTML Parsing

3. Navigating HTML Tree

4. CSS Selectors

📝 Hindi Explanation

5. Real-World Scraping

6. Handling Pagination

7. Scraping JSON APIs

8. Scraping with Sessions and Cookies

9. Best Practices and Ethics

Summary

Output Examples 📋

Section 1 Output — HTTP Requests

Section 2 Output — BeautifulSoup4 HTML Parsing

Section 3 Output — Navigating HTML Tree

Section 4 Output — CSS Selectors

Section 5 Output — Real-World Scraping

Section 6 Output — Handling Pagination

Section 7 Output — JSON APIs

Section 8 Output — Sessions and Cookies

Section 9 Output — Best Practices

⚠️ Common Mistakes

❌ Mistake 1: User-Agent Header Na Bhejana

❌ Mistake 2: Error Handling Skip Karna

❌ Mistake 3: Rate Limiting Na Karna

❌ Mistake 4: Hardcoded Selectors Use Karna

❌ Mistake 5: .text vs .string Ka Confusion

❌ Mistake 6: robots.txt Check Na Karna

❌ Mistake 7: Not Validating Data

✅ Key Takeaways

❓ FAQ

Q1: Web Scraping legal hai ya illegal? 🤔

Q2: BeautifulSoup vs Scrapy — kaunsa use karein? 🛠️

Q3: Website has blocked my IP — now what? 🚫

Q4: How to scrape JavaScript-rendered pages? ⚡

Q5: How to prevent infinite loops when handling pagination? 🔄

Q6: Where should you store scraped data? 💾

Q7: lxml vs html.parser — kaunsa parser better hai? 🏎️

Q8: How to keep API keys and credentials secure in code? 🔐

🎯 Interview Questions

Q1: What is Web Scraping and what are its key components?

Q2: What is the difference between find() and find_all()?

Q3: What are the advantages of using requests.Session()?

Q4: What is the fundamental difference between Scrapy and BeautifulSoup?

Q5: What are anti-scraping mechanisms and how do you deal with them?

Q6: What is the difference between XPath and CSS Selectors?

Q7: What is the best approach for error handling in Web Scraping?

Q8: What is robots.txt and what is its importance in scraping?

Q9: How do you design a data pipeline for a large-scale scraping project?

Q10: What are the best tools for scraping dynamic websites (SPAs)?

Continue learning this concept

Introduction

2. BeautifulSoup4 — HTML Parsing

3. Navigating HTML Tree

4. CSS Selectors

📝 Hindi Explanation

5. Real-World Scraping

6. Handling Pagination

7. Scraping JSON APIs

8. Scraping with Sessions and Cookies

9. Best Practices and Ethics

Summary

Output Examples 📋

Section 1 Output — HTTP Requests

Section 2 Output — BeautifulSoup4 HTML Parsing

Section 3 Output — Navigating HTML Tree

Section 4 Output — CSS Selectors

Section 5 Output — Real-World Scraping

Section 6 Output — Handling Pagination

Section 7 Output — JSON APIs

Section 8 Output — Sessions and Cookies

Section 9 Output — Best Practices

⚠️ Common Mistakes

❌ Mistake 1: User-Agent Header Na Bhejana

❌ Mistake 2: Error Handling Skip Karna

❌ Mistake 3: Rate Limiting Na Karna

❌ Mistake 4: Hardcoded Selectors Use Karna

❌ Mistake 5: .text vs .string Ka Confusion

❌ Mistake 6: robots.txt Check Na Karna

❌ Mistake 7: Not Validating Data

❌ Mistake 5: `.text` vs `.string` Ka Confusion

Q7: `lxml` vs `html.parser` — kaunsa parser better hai? 🏎️

Q2: What is the difference between `find()` and `find_all()`?

❌ Mistake 5: `.text` vs `.string` Ka Confusion

Q7: `lxml` vs `html.parser` — kaunsa parser better hai? 🏎️

Q2: What is the difference between `find()` and `find_all()`?