Python Notes
HubHistory of Python - Complete Guide 2026Python Applications - Complete Guide 2026Python Features - Complete Guide 2026What is Python? - Complete Guide 2026Why Learn Python? - Complete Guide 2026Your First Python Program – Hello World & Beyond 2026How to Install Python on Windows, Mac & Linux – Complete Guide 2026Install VS Code for Python – Complete Setup Guide with Extensions 2026Python Environment Setup – Virtual Environments, pip & Project Structure 2026Python IDLE – Complete Guide for Beginners 2026Python CommentsPython Data TypesPython Input and OutputPython OperatorsPython SyntaxPython Strings BasicsPython Type CastingPython VariablesBreak, Continue and Pass in PythonElif Statement in PythonFor Loop in PythonIf-Else Statement in PythonIf Statement in PythonNested If in PythonPattern Programs in PythonWhile Loop in PythonDecorators in PythonFunction Arguments in PythonFunctions Basics in PythonGenerators in PythonLambda Functions in PythonRecursive Functions in PythonReturn Statement in PythonScope of Variables in PythonAbstraction in PythonClasses and Objects in PythonConstructors in PythonEncapsulation in PythonInheritance in PythonMagic Methods (Dunder Methods) in PythonMethod Overriding in PythonPolymorphism in PythonPython Comprehensions – List, Dict, Set & GeneratorPython Dictionaries – Key-Value StorePython Dictionary Methods – Complete ReferencePython List Methods – All Built-in MethodsPython Lists – Complete GuidePython Set Methods – Complete ReferencePython Sets – Unordered Unique CollectionsPython Tuples – Immutable SequencesPython Asynchronous Programming with asyncioPython Context ManagersAdvanced Python DecoratorsAdvanced Python GeneratorsPython Iterators - Advanced GuidePython Memory ManagementPython MultiprocessingPython MultithreadingCRUD Operations in Python DatabasesMySQL with Python (mysql-connector-python)ORM Introduction with SQLAlchemyPostgreSQL with Python (psycopg2)SQLite3 with PythonIntroduction to DjangoDjango Models and ORMDjango REST API with DRFDjango ViewsIntroduction to FlaskFlask RoutingFlask Templates with Jinja2Email Automation with PythonPDF Automation with PythonSelenium Web Automation with PythonTask Scheduler with PythonWeb Scraping with PythonData Analysis with PythonData Cleaning with PythonData Visualization ProjectData Visualization with MatplotlibNumPy IntroductionPandas IntroductionStatistical Visualization with SeabornClassification AlgorithmsClustering AlgorithmsIntroduction to Machine LearningMachine Learning Project: Churn PredictionModel Evaluation and SelectionRegression AlgorithmsSave pipeline (includes preprocessing + model)AI Assistant ProjectCalculator ProjectChatbot ProjectExpense Tracker ProjectFace Detection ProjectPassword Generator ProjectWeather App ProjectPython Coding Interview QuestionsPython OOP Interview Questions and AnswersPython Master CheatsheetPython Interview Questions and AnswersPython Best PracticesCommon Python Errors and SolutionsPython Debugging TechniquesPython Learning Resources
Master data cleaning techniques using Python, Pandas, and NumPy. Learn to handle missing values, duplicates, outliers, data type conversions, string cleaning, and building a complete data cleaning pipeline.
Real-world data is messy. Data cleaning (also called data wrangling or data munging) is the process of detecting and correcting corrupt, inaccurate, or irrelevant records. It typically takes 60-80% of a data scientist's time.
Setup
bash example
pip install pandas numpy scipypython example
import pandas as pd
import numpy as np
from scipy import stats
import reLoading a Messy Dataset
Example
import pandas as pd
import numpy as np
# Create a messy dataset for practice
messy_data = {
'Name': [' Alice Johnson ', 'BOB SMITH', 'carol white', None, 'DAVE brown', 'Eve', 'alice johnson'],
'Age': [25, 30, -5, 28, 150, np.nan, 25],
'Email': ['alice@email.com', 'bob.smith@', 'carol@email.com', 'dave@email.com', '', 'eve@email.com', 'alice@email.com'],
'Phone': ['123-456-7890', '(123) 456-7890', '1234567890', None, '+1-123-456-7890', '123.456.7890', '123-456-7890'],
'Salary': ['$85,000', '$72,000', '95000', '$68,000.00', 'seventy-eight thousand', '$78,000', '$85,000'],
'Hire_Date': ['2023-01-15', '15/01/2020', 'January 10, 2019', None, '2022-03-20', '2024-06-01', '2023-01-15'],
'Department': ['Engineering', 'marketing', 'ENGINEERING', 'HR', 'hr', 'Marketing', 'Engineering'],
'Score': [92.5, 85.0, 97.0, 78.0, np.nan, 88.0, 92.5],
}
df = pd.DataFrame(messy_data)
print(f"Shape: {df.shape}")
print(f"Data types:\n{df.dtypes}")
print(df)
Step 1: Initial Exploration
Example
def explore_dataframe(df):
"""Initial exploration of a DataFrame."""
print("=" * 60)
print("DATAFRAME OVERVIEW")
print("=" * 60)
print(f"Shape: {df.shape[0]} rows × {df.shape[1]} columns")
print(f"\nData Types:\n{df.dtypes}")
print(f"\nMissing Values:\n{df.isnull().sum()}")
print(f"\nMissing % :\n{(df.isnull().mean() * 100).round(2)}")
print(f"\nDuplicate rows: {df.duplicated().sum()}")
print(f"\nNumeric Summary:\n{df.describe()}")
print(f"\nObject Columns Sample:")
for col in df.select_dtypes(include='object').columns:
print(f" {col}: {df[col].nunique()} unique → {df[col].value_counts().head(3).to_dict()}")
explore_dataframe(df)
Step 2: Remove Duplicates
Example
import pandas as pd
# Check for duplicates
print(f"Duplicate rows: {df.duplicated().sum()}")
# View duplicate rows
print(df[df.duplicated(keep=False)]) # Show all copies of duplicates
# Duplicate based on specific columns
print(df[df.duplicated(subset=['Name', 'Email'], keep=False)])
# Drop duplicates
df_clean = df.drop_duplicates()
df_clean = df.drop_duplicates(subset=['Email'], keep='first') # Keep first occurrence
df_clean = df.drop_duplicates(subset=['Name'], keep='last') # Keep last occurrence
print(f"After dedup: {df_clean.shape}")
Step 3: Handle Missing Values
Example
import pandas as pd
import numpy as np
# Identify missing values
print(df.isnull().sum())
print(df.isnull().mean() * 100) # Percentage missing
# Visualize missing data pattern
for col in df.columns:
missing = df[col].isnull().sum()
if missing > 0:
print(f" {col}: {missing} missing ({missing/len(df)*100:.1f}%)")
# Strategy 1: Drop rows with too many missing values
threshold = 0.5 # Drop rows with >50% missing
df_clean = df.dropna(thresh=int(df.shape[1] * threshold))
# Strategy 2: Drop columns with too many missing values
col_threshold = 0.3 # Drop columns with >30% missing
df_clean = df.dropna(axis=1, thresh=int(df.shape[0] * (1 - col_threshold)))
# Strategy 3: Fill with appropriate values
def smart_fill(df):
df_filled = df.copy()
for col in df_filled.columns:
if df_filled[col].dtype in ['float64', 'int64']:
# Fill numeric with median (robust to outliers)
df_filled[col] = df_filled[col].fillna(df_filled[col].median())
elif df_filled[col].dtype == 'object':
# Fill categorical with mode
mode = df_filled[col].mode()
if len(mode) > 0:
df_filled[col] = df_filled[col].fillna(mode[0])
return df_filled
df_filled = smart_fill(df)
# Strategy 4: Forward/backward fill (time series)
df_filled = df.fillna(method='ffill')
df_filled = df.fillna(method='bfill')
# Strategy 5: Interpolation
df['Score'] = df['Score'].interpolate(method='linear')
Step 4: Clean String Data
Example
import pandas as pd
import re
# Strip whitespace
df['Name'] = df['Name'].str.strip()
# Normalize case
df['Name'] = df['Name'].str.title() # Title Case
df['Department'] = df['Department'].str.lower() # lowercase
df['Email'] = df['Email'].str.lower()
# Remove special characters
df['Name'] = df['Name'].str.replace(r'[^a-zA-Z\s]', '', regex=True)
# Standardize phone numbers
def clean_phone(phone):
if pd.isna(phone) or phone == '':
return None
digits = re.sub(r'[^\d]', '', str(phone)) # Keep only digits
if len(digits) == 11 and digits.startswith('1'):
digits = digits[1:] # Remove country code
if len(digits) == 10:
return f"{digits[:3]}-{digits[3:6]}-{digits[6:]}"
return None
df['Phone_clean'] = df['Phone'].apply(clean_phone)
print(df[['Phone', 'Phone_clean']])
# Validate email
def is_valid_email(email):
if pd.isna(email) or email == '':
return False
pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
return bool(re.match(pattern, str(email)))
df['Email_valid'] = df['Email'].apply(is_valid_email)
df['Email'] = df['Email'].where(df['Email_valid'], other=None)
# String contains / regex extraction
df['Domain'] = df['Email'].str.extract(r'@([^.]+)', expand=False)
df['Name_parts'] = df['Name'].str.split(' ')
df['First_name'] = df['Name'].str.split(' ').str[0]
df['Last_name'] = df['Name'].str.split(' ').str[-1]
Step 5: Fix Data Types
Example
import pandas as pd
import numpy as np
# Convert salary strings to numeric
def parse_salary(s):
if pd.isna(s):
return np.nan
s = str(s).strip()
s = re.sub(r'[\$,]', '', s) # Remove $ and commas
try:
return float(s)
except ValueError:
return np.nan
df['Salary_clean'] = df['Salary'].apply(parse_salary)
print(df[['Salary', 'Salary_clean']])
# Convert to appropriate numeric types
df['Age'] = pd.to_numeric(df['Age'], errors='coerce')
df['Score'] = pd.to_numeric(df['Score'], errors='coerce')
# Integer conversion (when no missing values)
df['Age'] = df['Age'].astype('Int64') # Nullable integer type
# Parse dates from multiple formats
def parse_date(date_str):
if pd.isna(date_str):
return pd.NaT
formats = [
'%Y-%m-%d', '%d/%m/%Y', '%m/%d/%Y',
'%B %d, %Y', '%b %d, %Y', '%d-%m-%Y'
]
for fmt in formats:
try:
return pd.to_datetime(date_str, format=fmt)
except (ValueError, TypeError):
continue
# Try pandas auto-parse as fallback
try:
return pd.to_datetime(date_str, infer_datetime_format=True)
except:
return pd.NaT
df['Hire_Date_clean'] = df['Hire_Date'].apply(parse_date)
# Extract date components
df['Hire_Year'] = df['Hire_Date_clean'].dt.year
df['Hire_Month'] = df['Hire_Date_clean'].dt.month
df['Tenure_Days'] = (pd.Timestamp('2026-06-12') - df['Hire_Date_clean']).dt.days
# Categorical type (saves memory for low-cardinality columns)
df['Department'] = df['Department'].astype('category')
print(f"Memory saved: category uses {df['Department'].memory_usage()} bytes")
Step 6: Handle Outliers
Example
import pandas as pd
import numpy as np
from scipy import stats
df['Age'] = pd.to_numeric(df['Age'], errors='coerce')
df['Score'] = pd.to_numeric(df['Score'], errors='coerce')
# Method 1: Domain knowledge rules
df.loc[df['Age'] < 18, 'Age'] = np.nan # Age can't be < 18
df.loc[df['Age'] > 100, 'Age'] = np.nan # Age can't be > 100
# Method 2: IQR (Interquartile Range)
def remove_outliers_iqr(series, multiplier=1.5):
Q1 = series.quantile(0.25)
Q3 = series.quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - multiplier * IQR
upper = Q3 + multiplier * IQR
return series.between(lower, upper), lower, upper
valid_mask, low, high = remove_outliers_iqr(df['Score'].dropna())
print(f"Score outlier bounds: [{low:.1f}, {high:.1f}]")
df.loc[~df['Score'].between(low, high), 'Score'] = np.nan
# Method 3: Z-score
def detect_outliers_zscore(series, threshold=3):
z_scores = np.abs(stats.zscore(series.dropna()))
return z_scores > threshold
df_numeric = df[['Score']].dropna()
outliers = detect_outliers_zscore(df_numeric['Score'])
print(f"Z-score outliers: {outliers.sum()}")
# Method 4: Cap outliers (Winsorization) instead of removing
def winsorize(series, lower_pct=0.05, upper_pct=0.95):
lower = series.quantile(lower_pct)
upper = series.quantile(upper_pct)
return series.clip(lower, upper)
df['Score_capped'] = winsorize(df['Score'].dropna())
Step 7: Standardize Categorical Values
Example
import pandas as pd
# Standardize department names
dept_mapping = {
'engineering': 'Engineering',
'eng': 'Engineering',
'eng.': 'Engineering',
'marketing': 'Marketing',
'mkt': 'Marketing',
'human resources': 'HR',
'hr': 'HR',
'h.r.': 'HR',
}
df['Department_std'] = df['Department'].str.lower().str.strip().map(dept_mapping)
print(df[['Department', 'Department_std']])
# Handle unmapped values
df['Department_std'] = df['Department_std'].fillna('Other')
# Using fuzzy matching for approximate string matching
# pip install thefuzz
# from thefuzz import process
# def standardize_fuzzy(value, choices, threshold=80):
# match, score = process.extractOne(value, choices)
# return match if score >= threshold else 'Unknown'
# Standard departments
DEPARTMENTS = ['Engineering', 'Marketing', 'HR', 'Finance', 'Operations']
# df['Dept_fuzzy'] = df['Department'].apply(lambda x: standardize_fuzzy(x, DEPARTMENTS))
Step 8: Feature Engineering
Example
import pandas as pd
import numpy as np
# Create derived features
df['Salary_clean'] = pd.to_numeric(df['Salary'].str.replace(r'[\$,]', '', regex=True), errors='coerce')
# Salary per year of experience (if we have Years column)
# df['Salary_per_year'] = df['Salary_clean'] / df['Years']
# Age groups
bins = [0, 25, 35, 50, 100]
labels = ['Young', 'Mid', 'Senior', 'Executive']
df['Age_group'] = pd.cut(df['Age'], bins=bins, labels=labels)
# Salary quartiles
df['Salary_quartile'] = pd.qcut(df['Salary_clean'].dropna(), q=4, labels=['Q1', 'Q2', 'Q3', 'Q4'])
# One-hot encoding
dept_dummies = pd.get_dummies(df['Department_std'], prefix='dept')
df = pd.concat([df, dept_dummies], axis=1)
print(df.columns.tolist())
Complete Cleaning Pipeline
Example
import pandas as pd
import numpy as np
import re
def clean_dataset(filepath):
"""Complete data cleaning pipeline."""
print("Loading data...")
df = pd.read_csv(filepath)
print(f" Initial shape: {df.shape}")
# Step 1: Remove exact duplicates
df = df.drop_duplicates()
print(f" After dedup: {df.shape}")
# Step 2: Standardize column names
df.columns = (
df.columns
.str.strip()
.str.lower()
.str.replace(' ', '_')
.str.replace(r'[^a-z0-9_]', '', regex=True)
)
# Step 3: Strip whitespace from all string columns
str_cols = df.select_dtypes(include='object').columns
for col in str_cols:
df[col] = df[col].str.strip()
# Step 4: Replace empty strings with NaN
df = df.replace('', np.nan)
df = df.replace('N/A', np.nan)
df = df.replace('None', np.nan)
# Step 5: Report missing values
missing = df.isnull().sum()
print(f" Missing values:\n{missing[missing > 0]}")
# Step 6: Type inference and conversion
for col in df.columns:
if df[col].dtype == 'object':
# Try numeric conversion
try:
df[col] = pd.to_numeric(df[col])
except (ValueError, TypeError):
pass
# Try date conversion
if 'date' in col.lower():
df[col] = pd.to_datetime(df[col], errors='coerce')
print(f" Final shape: {df.shape}")
return df
# For demo, create and clean a CSV
sample_df = pd.DataFrame({
'Name': [' Alice ', 'Bob', 'Bob', 'Carol'],
'Age': ['25', '30', '30', ''],
'Salary': ['$85,000', '$72,000', '$72,000', 'N/A'],
})
sample_df.to_csv('/tmp/messy_data.csv', index=False)
cleaned = clean_dataset('/tmp/messy_data.csv')
print(cleaned)
Data Quality Report
Example
import pandas as pd
import numpy as np
def data_quality_report(df, name='Dataset'):
report = {
'dataset': name,
'rows': len(df),
'columns': len(df.columns),
'total_cells': df.size,
'missing_cells': df.isnull().sum().sum(),
'missing_pct': df.isnull().mean().mean() * 100,
'duplicate_rows': df.duplicated().sum(),
'column_report': {}
}
for col in df.columns:
col_info = {
'dtype': str(df[col].dtype),
'missing': int(df[col].isnull().sum()),
'missing_pct': round(df[col].isnull().mean() * 100, 2),
'unique': int(df[col].nunique()),
'unique_pct': round(df[col].nunique() / len(df) * 100, 2),
}
if df[col].dtype in ['float64', 'int64']:
col_info.update({
'mean': round(df[col].mean(), 2),
'std': round(df[col].std(), 2),
'min': df[col].min(),
'max': df[col].max(),
})
report['column_report'][col] = col_info
return report
# Generate report
df = pd.DataFrame({'a': [1, 2, None, 4], 'b': ['x', 'y', 'x', None]})
report = data_quality_report(df, 'Sample Data')
print(f"Quality Score: {100 - report['missing_pct']:.1f}%")
for col, info in report['column_report'].items():
print(f" {col}: {info['missing_pct']}% missing, {info['unique']} unique values")
Summary
In this lesson, you learned:
- ✅ Initial data exploration and profiling
- ✅ Removing duplicate records
- ✅ Handling missing values (drop, fill, interpolate)
- ✅ Cleaning string data (strip, case, regex, validation)
- ✅ Type conversion (numeric, dates, categorical)
- ✅ Detecting and handling outliers (IQR, Z-score, Winsorization)
- ✅ Standardizing categorical values
- ✅ Feature engineering from raw data
- ✅ Building a complete cleaning pipeline
⚠️ Common Mistakes
1. ❌ Dropping All Rows with Missing Values Blindly
Example
# ❌ Galat - Ye bahut zyada data kho sakta hai
df_clean = df.dropna() # Agar har row me kuch missing hai toh sab chala jayega!
# ✅ Sahi - Threshold set karo ya specific columns handle karo
df_clean = df.dropna(thresh=int(df.shape[1] * 0.7)) # 70% non-null chahiye
🚨 Tip: dropna() bina threshold ke use mat karo — aapka dataset almost empty ho sakta hai!2. ❌ Using Mean Instead of Median for Skewed Data
Example
# ❌ Galat - Mean outliers se affect hota hai
df['Salary'].fillna(df['Salary'].mean()) # Outliers se mean biased ho jayega
# ✅ Sahi - Median robust hota hai outliers ke against
df['Salary'].fillna(df['Salary'].median())
📊 Yaad rakho: Jab data skewed ho (salary, income), toh median use karo. Mean sirf normally distributed data ke liye theek hai.
3. ❌ Not Handling Data Types Before Analysis
Example
# ❌ Galat - String column pe arithmetic nahi hogi
df['Age'].mean() # Error ya wrong result agar Age string hai
# ✅ Sahi - Pehle type convert karo
df['Age'] = pd.to_numeric(df['Age'], errors='coerce')
df['Age'].mean() # Ab correct answer aayega
🔢 errors='coerce' use karo — invalid values NaN ban jayengi instead of error.
4. ❌ Forgetting to Normalize String Case Before Comparison
Example
# ❌ Galat - 'Engineering' aur 'ENGINEERING' alag count honge
df['Department'].value_counts() # 3 alag entries dikhega same department ke liye
# ✅ Sahi - Pehle case normalize karo
df['Department'] = df['Department'].str.lower().str.strip()
df['Department'].value_counts() # Ab correct count aayega
5. ❌ Removing Outliers Without Domain Knowledge
Example
# ❌ Galat - Blindly remove based on statistics
df = df[np.abs(stats.zscore(df['Score'])) < 2] # Valid high scores bhi hatt jayenge!
# ✅ Sahi - Domain knowledge ke saath rules define karo
# Score 0-100 range me hona chahiye (business rule)
df.loc[df['Score'] > 100, 'Score'] = np.nan
df.loc[df['Score'] < 0, 'Score'] = np.nan
🎯 Best practice: Pehle domain experts se pucho ki valid range kya hai, phir outlier rules banao.
6. ❌ Modifying Original DataFrame Without Copy
Example
# ❌ Galat - Original data lose ho jayega
df['Name'] = df['Name'].str.upper() # Ab original values nahi mil sakti
# ✅ Sahi - Copy banao ya new column me store karo
df_clean = df.copy()
df_clean['Name_cleaned'] = df_clean['Name'].str.upper()
💾 Hamesha .copy() use karo jab original data preserve karna ho!7. ❌ Not Validating After Cleaning
Example
# ❌ Galat - Clean kiya lekin verify nahi kiya
df['Email'] = df['Email'].str.lower()
# ... aur assume kar liya sab theek hai
# ✅ Sahi - Cleaning ke baad always validate karo
df['Email'] = df['Email'].str.lower()
assert df['Email'].str.contains(r'^[\w.]+@[\w]+\.[\w]+$', na=True).all(), "Invalid emails found!"
print(f"Validation passed: {df['Email'].notna().sum()} valid emails")
✅ Key Takeaways
- 📊 Data cleaning me 60-80% time lagta hai — Ye data science ka sabse important step hai, isko skip mat karo!
- 🔍 Pehle explore, phir clean —
df.info(),df.describe(),df.isnull().sum()se shuru karo. Bina samjhe cleaning karna dangerous hai.
- 🗑️ Duplicates intelligently handle karo —
drop_duplicates(subset=[...])use karo specific columns ke basis pe, blindly all columns pe mat check karo.
- 🕳️ Missing values ke liye sahi strategy choose karo — Numeric ke liye median, categorical ke liye mode, time-series ke liye interpolation ya ffill/bfill.
- 📝 String cleaning essential hai —
.str.strip(),.str.lower(), regex se normalize karo. Inconsistent strings se wrong grouping hoti hai.
- 🔢 Data types sahi rakho —
pd.to_numeric(errors='coerce')aurpd.to_datetime(errors='coerce')se safe conversion karo.
- 📈 Outliers ke liye multiple methods jaano — IQR method (general purpose), Z-score (normally distributed data), domain rules (best approach), aur Winsorization (cap karna).
- 🏷️ Categorical values standardize karo — Mapping dictionaries use karo ya fuzzy matching. 'HR', 'hr', 'H.R.' sab same hona chahiye.
- 🔧 Reusable pipeline banao — Functions me cleaning logic wrap karo taaki har baar repeat na karna pade.
df.pipe()method chain ke liye best hai.
- ✅ Cleaning ke baad Data Quality Report generate karo — Missing %, unique values, data types verify karo. Ye final validation step hai jo confidence deta hai.
❓ FAQ
Q1: Data cleaning aur data preprocessing me kya difference hai?
Answer: Data cleaning ka focus hai errors, inconsistencies, aur missing values fix karna — jaise duplicates hatana, wrong formats correct karna. Data preprocessing broader hai — isme cleaning ke saath scaling, encoding, feature selection, aur transformation bhi aata hai. Cleaning preprocessing ka ek subset hai. Socho: cleaning = fixing problems, preprocessing = preparing for ML models. 🧹➡️🤖
Q2: Missing values ko kab drop karna chahiye aur kab fill?
Answer: Drop karo jab:
- Column me 70%+ values missing hain (column drop karo)
- Row me bahut zyada fields missing hain
- Dataset bahut bada hai aur missing rows negligible hain
Fill karo jab:
- Data limited hai aur har row important hai
- Missing pattern random hai (MCAR - Missing Completely At Random)
- Numeric data me median/mean logical hai
📌 Rule of thumb: 5% se kam missing → drop. 5-30% → fill with median/mode. 30%+ → column drop ya advanced imputation (KNN, MICE).
Q3: IQR method aur Z-score me kab kya use kare?
Answer:
- IQR Method → Jab data normally distributed nahi hai (skewed data). Ye robust hai outliers ke against.
Q1 - 1.5*IQRseQ3 + 1.5*IQRrange use hoti hai. - Z-score → Jab data approximately normally distributed hai. Z > 3 ya Z < -3 ko outlier maante hain.
- Domain Rules → Sabse best! Age 0-120, Salary > 0, Score 0-100 — business logic se define karo. 🎯
Q4: errors='coerce' kya karta hai pd.to_numeric() me?
Answer: errors='coerce' invalid values ko NaN bana deta hai instead of error throw karna. Ye safe conversion hai:
'raise'(default) → Error aayega invalid value pe'coerce'→ Invalid → NaN (recommended for cleaning)'ignore'→ Column waise hi return hoga (not useful)
python example
pd.to_numeric('hello', errors='coerce') # Returns NaN ✅
pd.to_numeric('hello', errors='raise') # Raises ValueError ❌Q5: Large dataset (millions of rows) ko efficiently kaise clean kare?
Answer: Large datasets ke liye:
- Chunked reading:
pd.read_csv('file.csv', chunksize=100000)se chunks me process karo - Dtypes specify karo:
pd.read_csv(..., dtype={'col': 'int32'})se memory bachao - Category type: Low-cardinality columns ko
astype('category')karo — 90% memory save! - Vectorized operations:
.apply()avoid karo,.strmethods aur numpy operations use karo - Dask/Vaex: Pandas se bade data ke liye Dask ya Vaex library use karo
⚡ Pro tip: df.info(memory_usage='deep') se actual memory usage check karo.Q6: Cleaning pipeline ko reproducible kaise banaye?
Answer: Best practices for reproducible cleaning:
- Functions me wrap karo — Har step ko function banao
- Pipeline pattern use karo —
df.pipe(func1).pipe(func2).pipe(func3) - Config file rakho — Mappings, thresholds sab ek YAML/JSON me
- Logging add karo — Har step ke baad shape, missing count print karo
- Version control — Git me code + data transformations track karo
- Tests likho —
assertstatements se output validate karo
Q7: Regular expressions (regex) data cleaning me kitna important hai?
Answer: Regex extremely important hai data cleaning me! Use cases:
- Phone numbers standardize karna:
re.sub(r'[^\d]', '', phone) - Email validation:
r'^[\w.]+@[\w]+\.\w+$' - Currency symbols hatana:
str.replace(r'[$,]', '', regex=True) - Whitespace normalize karna:
str.replace(r'\s+', ' ', regex=True)
Pandas me .str.contains(), .str.extract(), .str.replace() sab regex support karte hain. Basic regex seekh lo — cleaning ka 50% kaam easy ho jayega! 🎯
Q8: Data cleaning ke baad quality kaise measure kare?
Answer: Data Quality ke 6 dimensions check karo:
- Completeness — Kitne % values present hain (
1 - df.isnull().mean()) - Uniqueness — Duplicates nahi hone chahiye (
df.duplicated().sum() == 0) - Validity — Values valid range me hain (domain rules pass)
- Consistency — Same entity ka same representation ('HR' vs 'hr' nahi hona chahiye)
- Accuracy — Values real-world se match karte hain
- Timeliness — Data up-to-date hai
📊 Quality Score formula: (total_cells - issues) / total_cells * 100🎯 Interview Questions
Q1: What is data cleaning and why is it important in data science?
Answer: Data cleaning (data wrangling/munging) is the process of identifying and correcting errors, inconsistencies, and inaccuracies in a dataset. It's important because:
- Garbage In, Garbage Out — ML models dirty data se wrong predictions denge
- 60-80% time data scientists ka cleaning me jaata hai
- Business decisions depend on data quality — wrong data = wrong decisions
- Model performance directly depend karta hai clean data pe
Steps include: handling missing values, removing duplicates, fixing data types, standardizing formats, and handling outliers.
Q2: Explain different strategies to handle missing values with their pros and cons.
Answer:
| Strategy | When to Use | Pros | Cons |
|---|---|---|---|
| Drop rows | <5% missing, large dataset | Simple, no bias introduced | Data loss |
| Drop columns | >70% missing in column | Removes noisy features | Loses information |
| Mean/Median fill | Numeric, random missing | Preserves distribution center | Reduces variance |
| Mode fill | Categorical data | Simple for categories | Over-represents mode |
| Forward/Back fill | Time series data | Maintains temporal patterns | Assumes continuity |
| Interpolation | Ordered numeric data | Smooth estimates | Assumes linearity |
| KNN Imputation | Complex patterns | Uses similar rows | Computationally expensive |
| MICE | Multiple columns missing | Statistical rigor | Complex, slow |
Best practice: Analyze missing pattern first (MCAR, MAR, MNAR), then choose strategy accordingly.
Q3: How do you detect and handle outliers? Explain IQR and Z-score methods.
Answer: IQR Method:
Example
Q1 = df['col'].quantile(0.25)
Q3 = df['col'].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
outliers = df[(df['col'] < lower_bound) | (df['col'] > upper_bound)]
- Works on any distribution, robust to extreme values.
Z-score Method:
Example
from scipy import stats
z_scores = np.abs(stats.zscore(df['col'].dropna()))
outliers = df[z_scores > 3]
- Assumes normal distribution. Z > 3 means 99.7% se bahar hai.
Handling options: Remove, cap (Winsorize), transform (log), or flag as separate feature.
Q4: What is the difference between dropna(), fillna(), and interpolate() in Pandas?
Answer:
dropna()— Rows/columns remove karta hai jahan NaN hai.threshparameter se control karo ki minimum kitne non-null chahiye.fillna()— NaN ko specified value se replace karta hai (scalar, dict, method like ffill/bfill).interpolate()— Missing values ko neighboring values se estimate karta hai. Methods: linear, polynomial, spline. Time-series ke liye ideal.
python example
df.dropna(thresh=3) # Keep rows with at least 3 non-null values
df.fillna({'age': 30, 'city': 'Unknown'}) # Column-specific fill
df.interpolate(method='linear') # Linear interpolationQ5: How would you handle inconsistent categorical data (e.g., 'HR', 'hr', 'Human Resources')?
Answer: Multi-step approach:
- Case normalization:
df['col'].str.lower().str.strip() - Mapping dictionary: Explicit mapping for known variations
- Fuzzy matching:
thefuzzlibrary for approximate string matching - Regex patterns: Group similar patterns together
Example
mapping = {'hr': 'HR', 'human resources': 'HR', 'h.r.': 'HR'}
df['dept'] = df['dept'].str.lower().str.strip().map(mapping).fillna('Other')
Interview me batao ki production me ek canonical mapping table maintain karte hain jo new variations handle karti hai.
Q6: Explain the concept of data pipeline for cleaning. How do you make it reproducible?
Answer: A data cleaning pipeline is a sequence of transformation steps applied in order:
python example
def pipeline(df):
return (df
.pipe(remove_duplicates)
.pipe(handle_missing)
.pipe(fix_types)
.pipe(standardize_strings)
.pipe(handle_outliers)
.pipe(validate_output)
)Reproducibility ensure karne ke liye:
- Functions me wrap karo (no manual steps)
- Configuration externalize karo (thresholds, mappings)
- Logging add karo (shape changes, missing counts track karo)
- Unit tests likho (expected output verify karo)
- Version control use karo (Git)
- Idempotent banao (multiple times run karne pe same result aaye)
Q7: What is Winsorization and when would you use it instead of removing outliers?
Answer: Winsorization caps extreme values at specified percentiles instead of removing them:
Example
lower = df['col'].quantile(0.05) # 5th percentile
upper = df['col'].quantile(0.95) # 95th percentile
df['col_winsorized'] = df['col'].clip(lower, upper)
Use Winsorization when:
- Dataset chhota hai aur rows lose nahi kar sakte
- Outliers real values hain but extreme (e.g., CEO salary)
- Model ko extreme values se protect karna hai but information retain karni hai
- Statistical tests ke liye robust estimates chahiye
Don't use when: Outliers are data entry errors (remove them instead).
Q8: How do you handle mixed data types in a column?
Answer: Mixed types common problem hai (e.g., column me '25', 25, 'twenty-five' sab hain):
Example
# Step 1: Identify mixed types
df['col'].apply(type).value_counts()
# Step 2: Coerce to target type
df['col'] = pd.to_numeric(df['col'], errors='coerce') # Non-numeric → NaN
# Step 3: Handle text numbers separately
word_to_num = {'one': 1, 'two': 2, 'twenty-five': 25}
df['col'] = df['col'].replace(word_to_num)
df['col'] = pd.to_numeric(df['col'], errors='coerce')
Interview tip: Batao ki real-world me pehle df[col].apply(type).value_counts() se problem identify karte hain, phir custom parser likhte hain.
Q9: What's the difference between df.replace() and df.fillna()? When to use which?
Answer:
fillna()— Sirf NaN/None values ko replace karta haireplace()— Koi bhi value ko kisi bhi value se replace kar sakta hai (including NaN)
python example
# fillna: Only replaces NaN
df.fillna(0) # All NaN → 0
df.fillna({'A': 0, 'B': 'N/A'}) # Column-specific
# replace: Any value → any value
df.replace('', np.nan) # Empty strings → NaN
df.replace({'N/A': np.nan, 'None': np.nan, '-': np.nan}) # Multiple replacements
df.replace(r'^\s*$', np.nan, regex=True) # Regex-based replacementBest practice: Pehle replace() se sentinel values (empty strings, 'N/A', 'None') ko NaN banao, phir fillna() se handle karo.
Q10: How would you validate data quality after cleaning? What metrics would you track?
Answer: Post-cleaning validation checklist:
Example
def validate_cleaned_data(df):
checks = {
'no_duplicates': df.duplicated().sum() == 0,
'no_missing_critical': df[['id', 'name']].isnull().sum().sum() == 0,
'valid_age_range': df['age'].between(0, 120).all(),
'valid_email': df['email'].str.contains(r'@.*\.', na=True).all(),
'correct_types': df['salary'].dtype == 'float64',
'no_empty_strings': not (df.select_dtypes('object') == '').any().any(),
}
for check, passed in checks.items():
print(f"{'✅' if passed else '❌'} {check}: {'PASS' if passed else 'FAIL'}")
return all(checks.values())
Metrics to track:
- Completeness score:
(1 - missing_ratio) * 100 - Uniqueness: Duplicate count = 0
- Validity: All values within domain rules
- Consistency: Standardized categories, uniform formats
- Row retention rate: Cleaning ke baad kitna data bacha
*Next Lesson: Data Analysis →*
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Data Cleaning with Python.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Python Master Course topic.
Search Terms
python-master-course, python master course, python, master, course, data, science, cleaning
Related Python Master Course Topics
Continue learning this concept
Data ScienceData Analysis with PythonLearn exploratory data analysis (EDA) with Python, Pandas, and NumPy. Master descriptive statistics, correlation analysis, group comparisons, time series basics, and building insight-driven data reports.Data ScienceData Visualization ProjectBuild a complete data visualization project analyzing real-world COVID-19 and economic data. Step-by-step tutorial covering data loading, cleaning, EDA, and creating a professional multi-panel visualization report.Data ScienceData Visualization with MatplotlibMaster Matplotlib, PythonData ScienceNumPy IntroductionLearn NumPy, the foundational Python library for numerical computing. Master arrays, array operations, broadcasting, indexing, reshaping, linear algebra, and random number generation with practical examples.Data SciencePandas IntroductionLearn Pandas, the powerful Python data manipulation library. Master DataFrames, Series, data loading, selection, filtering, groupby, merging, and essential data operations for data analysis.Data ScienceStatistical Visualization with SeabornLearn Seaborn for beautiful statistical data visualization. Master distribution plots, categorical plots, regression plots, heatmaps, pair plots, and FacetGrid with practical data science examples.