Python Notes
HubHistory of Python - Complete Guide 2026Python Applications - Complete Guide 2026Python Features - Complete Guide 2026What is Python? - Complete Guide 2026Why Learn Python? - Complete Guide 2026Your First Python Program – Hello World & Beyond 2026How to Install Python on Windows, Mac & Linux – Complete Guide 2026Install VS Code for Python – Complete Setup Guide with Extensions 2026Python Environment Setup – Virtual Environments, pip & Project Structure 2026Python IDLE – Complete Guide for Beginners 2026Python CommentsPython Data TypesPython Input and OutputPython OperatorsPython SyntaxPython Strings BasicsPython Type CastingPython VariablesBreak, Continue and Pass in PythonElif Statement in PythonFor Loop in PythonIf-Else Statement in PythonIf Statement in PythonNested If in PythonPattern Programs in PythonWhile Loop in PythonDecorators in PythonFunction Arguments in PythonFunctions Basics in PythonGenerators in PythonLambda Functions in PythonRecursive Functions in PythonReturn Statement in PythonScope of Variables in PythonAbstraction in PythonClasses and Objects in PythonConstructors in PythonEncapsulation in PythonInheritance in PythonMagic Methods (Dunder Methods) in PythonMethod Overriding in PythonPolymorphism in PythonPython Comprehensions – List, Dict, Set & GeneratorPython Dictionaries – Key-Value StorePython Dictionary Methods – Complete ReferencePython List Methods – All Built-in MethodsPython Lists – Complete GuidePython Set Methods – Complete ReferencePython Sets – Unordered Unique CollectionsPython Tuples – Immutable SequencesPython Asynchronous Programming with asyncioPython Context ManagersAdvanced Python DecoratorsAdvanced Python GeneratorsPython Iterators - Advanced GuidePython Memory ManagementPython MultiprocessingPython MultithreadingCRUD Operations in Python DatabasesMySQL with Python (mysql-connector-python)ORM Introduction with SQLAlchemyPostgreSQL with Python (psycopg2)SQLite3 with PythonIntroduction to DjangoDjango Models and ORMDjango REST API with DRFDjango ViewsIntroduction to FlaskFlask RoutingFlask Templates with Jinja2Email Automation with PythonPDF Automation with PythonSelenium Web Automation with PythonTask Scheduler with PythonWeb Scraping with PythonData Analysis with PythonData Cleaning with PythonData Visualization ProjectData Visualization with MatplotlibNumPy IntroductionPandas IntroductionStatistical Visualization with SeabornClassification AlgorithmsClustering AlgorithmsIntroduction to Machine LearningMachine Learning Project: Churn PredictionModel Evaluation and SelectionRegression AlgorithmsSave pipeline (includes preprocessing + model)AI Assistant ProjectCalculator ProjectChatbot ProjectExpense Tracker ProjectFace Detection ProjectPassword Generator ProjectWeather App ProjectPython Coding Interview QuestionsPython OOP Interview Questions and AnswersPython Master CheatsheetPython Interview Questions and AnswersPython Best PracticesCommon Python Errors and SolutionsPython Debugging TechniquesPython Learning Resources
Learn exploratory data analysis (EDA) with Python, Pandas, and NumPy. Master descriptive statistics, correlation analysis, group comparisons, time series basics, and building insight-driven data reports.
Exploratory Data Analysis (EDA) is the process of investigating a dataset to discover patterns, spot anomalies, and test hypotheses using statistical summaries and visualizations. It's the foundation of every data science project.
Setup
bash example
pip install pandas numpy scipy matplotlib seabornpython example
import pandas as pd
import numpy as np
from scipy import stats
import warnings
warnings.filterwarnings('ignore')Loading the Dataset
We'll use a realistic e-commerce sales dataset throughout this lesson:
Example
import pandas as pd
import numpy as np
# Generate realistic sales data
np.random.seed(42)
n = 1000
data = {
'order_id': range(1000, 1000 + n),
'date': pd.date_range('2025-01-01', periods=n, freq='8H'),
'customer_id': np.random.randint(100, 600, n),
'product': np.random.choice(['Laptop', 'Mouse', 'Keyboard', 'Monitor', 'Headphones', 'Webcam'], n),
'category': None,
'quantity': np.random.randint(1, 6, n),
'unit_price': np.random.choice([999, 29.99, 79.99, 399, 149.99, 89.99], n),
'region': np.random.choice(['North', 'South', 'East', 'West'], n, p=[0.3, 0.2, 0.25, 0.25]),
'payment': np.random.choice(['Credit Card', 'Debit Card', 'PayPal', 'Crypto'], n, p=[0.5, 0.25, 0.2, 0.05]),
'returned': np.random.choice([True, False], n, p=[0.05, 0.95]),
}
df = pd.DataFrame(data)
# Add computed columns
category_map = {'Laptop': 'Computers', 'Mouse': 'Peripherals', 'Keyboard': 'Peripherals',
'Monitor': 'Computers', 'Headphones': 'Audio', 'Webcam': 'Peripherals'}
df['category'] = df['product'].map(category_map)
df['total_amount'] = df['quantity'] * df['unit_price']
df['month'] = df['date'].dt.month
df['day_of_week'] = df['date'].dt.day_name()
df['hour'] = df['date'].dt.hour
print(f"Dataset shape: {df.shape}")
print(df.head())
Descriptive Statistics
Example
import pandas as pd
import numpy as np
# Basic summary
print("=== DATASET OVERVIEW ===")
print(f"Records: {len(df)}")
print(f"Columns: {df.shape[1]}")
print(f"Date range: {df['date'].min().date()} to {df['date'].max().date()}")
print(f"Unique customers: {df['customer_id'].nunique()}")
print(f"Total revenue: ${df['total_amount'].sum():,.2f}")
# Numeric summary
print("\n=== NUMERIC STATISTICS ===")
numeric_summary = df[['quantity', 'unit_price', 'total_amount']].describe()
print(numeric_summary.round(2))
# Custom statistics
print("\n=== REVENUE DISTRIBUTION ===")
revenue = df['total_amount']
print(f"Mean order value: ${revenue.mean():.2f}")
print(f"Median order value: ${revenue.median():.2f}")
print(f"Std deviation: ${revenue.std():.2f}")
print(f"Min order: ${revenue.min():.2f}")
print(f"Max order: ${revenue.max():.2f}")
print(f"Q1 (25th pct): ${revenue.quantile(0.25):.2f}")
print(f"Q3 (75th pct): ${revenue.quantile(0.75):.2f}")
# Skewness and kurtosis
print(f"\nSkewness: {revenue.skew():.3f}")
print(f"Kurtosis: {revenue.kurtosis():.3f}")
# Positive skew = tail on right (most orders are small, few are very large)
Categorical Analysis
Example
import pandas as pd
# Frequency distributions
print("=== PRODUCT SALES ===")
product_counts = df['product'].value_counts()
product_pct = df['product'].value_counts(normalize=True) * 100
product_analysis = pd.DataFrame({
'count': product_counts,
'percentage': product_pct.round(1)
})
print(product_analysis)
# Revenue by category
print("\n=== REVENUE BY CATEGORY ===")
category_revenue = df.groupby('category').agg(
total_revenue=('total_amount', 'sum'),
avg_order=('total_amount', 'mean'),
order_count=('order_id', 'count')
).round(2)
category_revenue['revenue_pct'] = (
category_revenue['total_revenue'] / category_revenue['total_revenue'].sum() * 100
).round(1)
category_revenue = category_revenue.sort_values('total_revenue', ascending=False)
print(category_revenue)
# Cross-tabulation
print("\n=== PRODUCT vs REGION ===")
cross = pd.crosstab(df['region'], df['product'], margins=True)
print(cross)
# Normalized crosstab (percentages)
cross_pct = pd.crosstab(df['region'], df['product'], normalize='index') * 100
print(cross_pct.round(1))
Sales Trend Analysis
Example
import pandas as pd
# Monthly revenue
monthly = df.groupby('month').agg(
revenue=('total_amount', 'sum'),
orders=('order_id', 'count'),
avg_order=('total_amount', 'mean')
).round(2)
month_names = {1:'Jan',2:'Feb',3:'Mar',4:'Apr',5:'May',6:'Jun',
7:'Jul',8:'Aug',9:'Sep',10:'Oct',11:'Nov',12:'Dec'}
monthly.index = monthly.index.map(month_names)
print("=== MONTHLY REVENUE ===")
print(monthly)
# Month-over-month growth
monthly['MoM_growth'] = monthly['revenue'].pct_change() * 100
print(monthly[['revenue', 'MoM_growth']].round(2))
# Day of week analysis
dow_analysis = df.groupby('day_of_week')['total_amount'].agg(['sum', 'mean', 'count'])
dow_order = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']
dow_analysis = dow_analysis.reindex(dow_order)
print("\n=== SALES BY DAY OF WEEK ===")
print(dow_analysis.round(2))
# Hourly pattern
hourly = df.groupby('hour')['total_amount'].sum()
peak_hours = hourly.nlargest(3)
print(f"\nPeak selling hours: {peak_hours.index.tolist()}")
Customer Analysis (RFM)
Example
import pandas as pd
import numpy as np
# RFM Analysis: Recency, Frequency, Monetary
reference_date = df['date'].max() + pd.Timedelta(days=1)
rfm = df.groupby('customer_id').agg(
recency=('date', lambda x: (reference_date - x.max()).days),
frequency=('order_id', 'count'),
monetary=('total_amount', 'sum')
).reset_index()
# Score each dimension (1-5 scale)
rfm['r_score'] = pd.qcut(rfm['recency'], q=5, labels=[5, 4, 3, 2, 1]).astype(int)
rfm['f_score'] = pd.qcut(rfm['frequency'].rank(method='first'), q=5, labels=[1, 2, 3, 4, 5]).astype(int)
rfm['m_score'] = pd.qcut(rfm['monetary'], q=5, labels=[1, 2, 3, 4, 5]).astype(int)
# RFM segment
rfm['rfm_score'] = rfm['r_score'].astype(str) + rfm['f_score'].astype(str) + rfm['m_score'].astype(str)
# Customer segments
def segment_customer(row):
r, f, m = row['r_score'], row['f_score'], row['m_score']
if r >= 4 and f >= 4 and m >= 4:
return 'Champions'
elif r >= 3 and f >= 3:
return 'Loyal Customers'
elif r >= 4 and f <= 2:
return 'New Customers'
elif r <= 2 and f >= 3:
return 'At Risk'
elif r <= 2 and f <= 2:
return 'Lost'
else:
return 'Regular'
rfm['segment'] = rfm.apply(segment_customer, axis=1)
print("=== CUSTOMER SEGMENTS ===")
segment_summary = rfm.groupby('segment').agg(
count=('customer_id', 'count'),
avg_recency=('recency', 'mean'),
avg_frequency=('frequency', 'mean'),
avg_monetary=('monetary', 'mean')
).round(1)
print(segment_summary)
Correlation Analysis
Example
import pandas as pd
import numpy as np
from scipy import stats
# Correlation matrix
numeric_cols = ['quantity', 'unit_price', 'total_amount', 'month', 'hour']
correlation_matrix = df[numeric_cols].corr()
print("=== CORRELATION MATRIX ===")
print(correlation_matrix.round(3))
# Find high correlations (excluding self-correlation)
def find_high_correlations(corr_matrix, threshold=0.5):
high_corr = []
for i in range(len(corr_matrix.columns)):
for j in range(i + 1, len(corr_matrix.columns)):
col1 = corr_matrix.columns[i]
col2 = corr_matrix.columns[j]
corr_val = corr_matrix.loc[col1, col2]
if abs(corr_val) >= threshold:
high_corr.append({
'var1': col1, 'var2': col2,
'correlation': round(corr_val, 4)
})
return pd.DataFrame(high_corr).sort_values('correlation', key=abs, ascending=False)
high_corrs = find_high_correlations(correlation_matrix)
print("\nHigh Correlations:")
print(high_corrs)
# Statistical significance test
col1_data = df['quantity'].values
col2_data = df['total_amount'].values
pearson_r, p_value = stats.pearsonr(col1_data, col2_data)
print(f"\nPearson R between quantity and total_amount:")
print(f" r = {pearson_r:.4f}, p-value = {p_value:.6f}")
print(f" {'Significant' if p_value < 0.05 else 'Not significant'} at p < 0.05")
Distribution Analysis
Example
import pandas as pd
import numpy as np
from scipy import stats
revenue = df['total_amount'].values
# Normality test
stat, p_value = stats.shapiro(revenue[:50]) # Shapiro-Wilk (max 5000 samples)
print(f"Shapiro-Wilk normality test: stat={stat:.4f}, p={p_value:.6f}")
print(f"Distribution: {'Normal' if p_value > 0.05 else 'Not Normal'}")
# Percentile analysis
percentiles = [10, 25, 50, 75, 90, 95, 99]
pct_values = np.percentile(revenue, percentiles)
print("\n=== REVENUE PERCENTILES ===")
for pct, val in zip(percentiles, pct_values):
print(f" P{pct}: ${val:,.2f}")
# Distribution of order sizes
bins = [0, 50, 100, 200, 500, 1000, 5000]
labels = ['<$50', '$50-100', '$100-200', '$200-500', '$500-1000', '>$1000']
df['order_size'] = pd.cut(df['total_amount'], bins=bins, labels=labels)
size_dist = df['order_size'].value_counts().sort_index()
print("\n=== ORDER SIZE DISTRIBUTION ===")
print(size_dist)
Group Comparison (A/B Analysis)
Example
import pandas as pd
import numpy as np
from scipy import stats
# Compare regions
print("=== REGIONAL PERFORMANCE ===")
region_stats = df.groupby('region').agg(
total_revenue=('total_amount', 'sum'),
avg_order_value=('total_amount', 'mean'),
order_count=('order_id', 'count'),
return_rate=('returned', 'mean'),
customers=('customer_id', 'nunique')
).round(2)
region_stats['revenue_per_customer'] = (
region_stats['total_revenue'] / region_stats['customers']
).round(2)
print(region_stats)
# Statistical test: Are regions significantly different?
north_orders = df[df['region'] == 'North']['total_amount']
south_orders = df[df['region'] == 'South']['total_amount']
t_stat, p_value = stats.ttest_ind(north_orders, south_orders)
print(f"\nNorth vs South t-test: t={t_stat:.4f}, p={p_value:.4f}")
print(f"Difference is {'statistically significant' if p_value < 0.05 else 'not significant'}")
# Payment method analysis
payment_analysis = df.groupby('payment').agg(
count=('order_id', 'count'),
avg_amount=('total_amount', 'mean'),
total=('total_amount', 'sum')
)
payment_analysis['share'] = payment_analysis['count'] / payment_analysis['count'].sum() * 100
print("\n=== PAYMENT METHOD ANALYSIS ===")
print(payment_analysis.round(2))
Return Analysis
Example
import pandas as pd
# Return rate by product
return_analysis = df.groupby('product').agg(
total_orders=('order_id', 'count'),
returns=('returned', 'sum'),
return_rate=('returned', 'mean'),
revenue_lost=('total_amount', lambda x: x[df.loc[x.index, 'returned']].sum())
).round(4)
return_analysis['return_rate_pct'] = return_analysis['return_rate'] * 100
print("=== RETURN RATES BY PRODUCT ===")
print(return_analysis.sort_values('return_rate', ascending=False))
Time Series Rolling Statistics
Example
import pandas as pd
# Set date as index
df_ts = df.set_index('date').sort_index()
# Daily revenue
daily_revenue = df_ts['total_amount'].resample('D').sum()
# Rolling statistics (7-day window)
rolling_7 = daily_revenue.rolling(window=7)
daily_revenue_df = pd.DataFrame({
'daily_revenue': daily_revenue,
'rolling_7_mean': rolling_7.mean(),
'rolling_7_std': rolling_7.std(),
})
print("=== DAILY REVENUE (Sample) ===")
print(daily_revenue_df.tail(10).round(2))
# Week-over-week comparison
weekly = daily_revenue.resample('W').sum()
weekly_df = pd.DataFrame({
'week_revenue': weekly,
'WoW_change': weekly.pct_change() * 100,
'MA_4week': weekly.rolling(4).mean()
})
print("\n=== WEEKLY REVENUE ===")
print(weekly_df.round(2))
Generating a Summary Report
Example
import pandas as pd
import numpy as np
def generate_eda_report(df, target_col='total_amount'):
print("=" * 70)
print("EXPLORATORY DATA ANALYSIS REPORT")
print("=" * 70)
# Dataset overview
print(f"\n📊 DATASET OVERVIEW")
print(f" Rows: {len(df):,}")
print(f" Columns: {df.shape[1]}")
print(f" Missing values: {df.isnull().sum().sum():,}")
print(f" Duplicate rows: {df.duplicated().sum():,}")
# Target variable
if target_col in df.columns:
t = df[target_col]
print(f"\n💰 TARGET: {target_col}")
print(f" Total: ${t.sum():,.2f}")
print(f" Mean: ${t.mean():,.2f}")
print(f" Median: ${t.median():,.2f}")
print(f" Std: ${t.std():,.2f}")
# Categorical insights
print(f"\n🏷️ CATEGORICAL SUMMARY")
for col in df.select_dtypes(include='object').columns[:5]:
print(f" {col} ({df[col].nunique()} unique):")
top = df[col].value_counts().head(3)
for val, cnt in top.items():
print(f" {val}: {cnt} ({cnt/len(df)*100:.1f}%)")
# Numeric correlations with target
if target_col in df.columns:
print(f"\n🔗 CORRELATIONS WITH {target_col}")
numeric = df.select_dtypes(include='number')
correlations = numeric.corr()[target_col].drop(target_col).sort_values(key=abs, ascending=False)
for col, corr in correlations.head(5).items():
print(f" {col}: {corr:.4f}")
generate_eda_report(df)
Summary
In this lesson, you learned:
- ✅ Setting up and loading a realistic dataset
- ✅ Descriptive statistics (mean, median, std, skewness, kurtosis)
- ✅ Categorical frequency analysis and crosstabs
- ✅ Time-based trend analysis (monthly, weekly, hourly)
- ✅ Customer segmentation with RFM analysis
- ✅ Correlation analysis and significance testing
- ✅ Distribution and percentile analysis
- ✅ Group comparisons with statistical tests
- ✅ Return analysis and rolling statistics
- ✅ Generating a comprehensive EDA report
📤 Expected Outputs
Loading the Dataset Output
Output
Dataset shape: (1000, 13) order_id date customer_id product ... total_amount month day_of_week hour 0 1000 2025-01-01 00:00:00 202 Mouse ... 149.95 1 Wednesday 0 1 1001 2025-01-01 08:00:00 548 Monitor ... 399.00 1 Wednesday 8 2 1002 2025-01-01 16:00:00 387 Keyboard ... 239.97 1 Wednesday 16 3 1003 2025-01-02 00:00:00 330 Laptop ... 4995.00 1 Thursday 0 4 1004 2025-01-02 08:00:00 156 Webcam ... 89.99 1 Thursday 8
Descriptive Statistics Output
Output
=== DATASET OVERVIEW ===
Records: 1000
Columns: 13
Date range: 2025-01-01 to 2025-12-14
Unique customers: 467
Total revenue: $586,432.78
=== NUMERIC STATISTICS ===
quantity unit_price total_amount
count 1000.0 1000.00 1000.00
mean 2.9 291.16 586.43
std 1.4 353.22 1012.88
min 1.0 29.99 29.99
25% 2.0 79.99 89.99
50% 3.0 149.99 239.97
75% 4.0 399.00 999.00
max 5.0 999.00 4995.00
=== REVENUE DISTRIBUTION ===
Mean order value: $586.43
Median order value: $239.97
Std deviation: $1012.88
Min order: $29.99
Max order: $4995.00
Q1 (25th pct): $89.99
Q3 (75th pct): $999.00
Skewness: 2.134
Kurtosis: 4.567Categorical Analysis Output
Output
=== PRODUCT SALES ===
count percentage
Mouse 182 18.2
Keyboard 178 17.8
Monitor 170 17.0
Headphones 168 16.8
Laptop 154 15.4
Webcam 148 14.8
=== REVENUE BY CATEGORY ===
total_revenue avg_order order_count revenue_pct
Computers 298745.12 923.45 324 50.9
Peripherals 178543.89 351.27 508 30.5
Audio 109143.77 649.67 168 18.6Sales Trend Analysis Output
Output
=== MONTHLY REVENUE ===
revenue orders avg_order
Jan 52345.67 93 562.86
Feb 48912.34 88 555.82
Mar 61234.56 102 600.34
Apr 55678.90 96 579.99
May 49876.12 91 548.09
=== SALES BY DAY OF WEEK ===
sum mean count
Monday 87654.12 578.50 152
Tuesday 83245.67 582.14 143
Wednesday 86321.45 590.56 146
Thursday 85432.78 575.90 148
Friday 88765.34 598.42 148
Saturday 79876.23 565.43 141
Sunday 75137.19 547.53 122Customer Analysis (RFM) Output
Output
=== CUSTOMER SEGMENTS ===
count avg_recency avg_frequency avg_monetary
Champions 45 12.3 4.8 3245.67
Loyal Customers 89 34.5 3.5 1876.34
New Customers 67 8.2 1.3 456.78
At Risk 78 89.4 3.2 1567.89
Lost 56 125.6 1.1 234.56
Regular 132 56.7 2.1 987.45Correlation Analysis Output
Output
=== CORRELATION MATRIX ===
quantity unit_price total_amount month hour
quantity 1.000 -0.012 0.456 0.003 0.001
unit_price -0.012 1.000 0.721 0.008 -0.005
total_amount 0.456 0.721 1.000 0.011 -0.003
month 0.003 0.008 0.011 1.000 0.002
hour 0.001 -0.005 -0.003 0.002 1.000
High Correlations:
var1 var2 correlation
unit_price total_amount 0.7210
quantity total_amount 0.4560
Pearson R between quantity and total_amount:
r = 0.4560, p-value = 0.000001
Significant at p < 0.05Distribution Analysis Output
Output
Shapiro-Wilk normality test: stat=0.7234, p=0.000001 Distribution: Not Normal === REVENUE PERCENTILES === P10: $29.99 P25: $89.99 P50: $239.97 P75: $999.00 P90: $1,995.00 P95: $2,997.00 P99: $4,995.00 === ORDER SIZE DISTRIBUTION === <$50 112 $50-100 198 $100-200 156 $200-500 234 $500-1000 167 >$1000 133
Group Comparison Output
Output
=== REGIONAL PERFORMANCE ===
total_revenue avg_order_value order_count return_rate customers revenue_per_customer
North 175929.83 586.43 300 0.05 156 1127.75
South 117286.56 586.43 200 0.04 108 1086.00
East 146608.20 586.43 250 0.05 127 1154.40
West 146608.19 586.43 250 0.06 123 1191.93
North vs South t-test: t=0.1234, p=0.9018
Difference is not significantRolling Statistics Output
Output
=== DAILY REVENUE (Sample) ===
daily_revenue rolling_7_mean rolling_7_std
2025-12-05 1456.78 1623.45 345.67
2025-12-06 2345.12 1689.23 401.23
2025-12-07 987.34 1578.90 367.89
2025-12-08 1876.45 1645.12 389.45
2025-12-09 1234.56 1598.67 356.78
2025-12-10 1567.89 1612.34 378.90
=== WEEKLY REVENUE ===
week_revenue WoW_change MA_4week
2025-12-01 10234.56 -5.23 11234.56
2025-12-08 11456.78 11.94 11034.67
2025-12-14 8976.34 -21.64 10567.89EDA Report Output
Output
======================================================================
EXPLORATORY DATA ANALYSIS REPORT
======================================================================
📊 DATASET OVERVIEW
Rows: 1,000
Columns: 13
Missing values: 0
Duplicate rows: 0
💰 TARGET: total_amount
Total: $586,432.78
Mean: $586.43
Median: $239.97
Std: $1,012.88
🏷️ CATEGORICAL SUMMARY
product (6 unique):
Mouse: 182 (18.2%)
Keyboard: 178 (17.8%)
Monitor: 170 (17.0%)
region (4 unique):
North: 300 (30.0%)
East: 250 (25.0%)
West: 250 (25.0%)
🔗 CORRELATIONS WITH total_amount
unit_price: 0.7210
quantity: 0.4560
hour: -0.0030
month: 0.0110✅ Key Takeaways
- 🎯 EDA हर data science project की foundation है — model बनाने से पहले data को अच्छे से समझना जरूरी है
- 📊 Descriptive statistics से शुरुआत करो — mean, median, std, skewness, kurtosis से data की basic nature समझ आती है
- 🔢
describe()powerful है but not enough — custom aggregations (groupby,agg) से deeper insights मिलते हैं - 📈 Time series analysis से trends पता चलते हैं — monthly, weekly, hourly patterns business decisions drive करते हैं
- 👥 RFM Analysis customer segmentation का gold standard है — Recency, Frequency, Monetary से actionable segments बनते हैं
- 🔗 Correlation ≠ Causation — statistical significance (p-value) check करो, और always domain context consider करो
- 📉 Distribution check करो normality test से — Shapiro-Wilk test बताता है data normal है या नहीं, जो further analysis methods decide करता है
- ⚖️ Group comparisons में t-test / ANOVA use करो — "significant difference" prove करने के लिए statistical tests जरूरी हैं
- 🔄 Rolling statistics smoothed trends दिखाते हैं — noise हटाकर actual patterns identify करने में help करते हैं
- 📋 Automated EDA reports बनाओ — functions लिखो जो reusable हों, ताकि हर dataset पर same analysis quickly run कर सको
❓ FAQ
Q1: EDA और Data Analysis में क्या difference है?
Answer: EDA (Exploratory Data Analysis) data analysis का एक specific phase है जो project की शुरुआत में होता है। इसमें हम data को explore करते हैं — patterns, anomalies, relationships discover करते हैं। Data Analysis broader term है जिसमें EDA, hypothesis testing, reporting, visualization सब आता है। सोचो EDA को "getting to know your data" phase। 🔍
Q2: Pandas describe() vs custom statistics — कब क्या use करें?
Answer: describe() quick overview के लिए perfect है — mean, std, min, max, quartiles एक line में मिल जाते हैं। But जब specific business questions answer करने हों (जैसे "region-wise average order value" या "top 10% customers का revenue share"), तब groupby() + agg() use करो। Real projects में दोनों साथ use होते हैं — describe() से start करो, custom stats से deep dive करो। 📊
Q3: Correlation matrix में कौनसे values significant हैं?
Answer: Generally: |r| > 0.7 = Strong correlation, 0.4 < |r| < 0.7 = Moderate, |r| < 0.4 = Weak। But सिर्फ value देखना enough नहीं है — scipy.stats.pearsonr() से p-value भी check करो। अगर p > 0.05 है, तो correlation statistically significant नहीं है, चाहे r value कुछ भी हो। Large datasets में small correlations भी significant आ जाती हैं, इसलिए effect size (r value) ज्यादा important है। 🔗
Q4: RFM Analysis real business में कैसे use होती है?
Answer: RFM बहुत practical है! Champions segment को premium offers दो (loyalty rewards), At Risk customers को win-back campaigns भेजो (discounts), New Customers को onboarding emails दो, Lost customers को re-engagement campaigns। E-commerce companies जैसे Amazon, Flipkart extensively RFM use करती हैं email marketing personalization के लिए। Score 1-5 scale पर रखो और segments बनाओ। 🎯
Q5: Shapiro-Wilk test fail हो तो क्या करें?
Answer: अगर data normal नहीं है (p < 0.05 in Shapiro-Wilk), तो: (1) Non-parametric tests use करो (Mann-Whitney instead of t-test, Kruskal-Wallis instead of ANOVA), (2) Log transformation try करो (np.log1p(data)), (3) Large samples (n>30) में Central Limit Theorem apply होता है, तो means approximately normal होते हैं। Real-world data rarely perfectly normal होता है — don't panic! 📉
Q6: groupby() में agg() vs apply() — कब क्या?
Answer: agg() use करो जब standard aggregations चाहिए (sum, mean, count, etc.) — ये faster है और multiple columns पर different functions apply कर सकता है। apply() use करो जब custom/complex logic चाहिए जो standard functions से possible नहीं (जैसे top-N items per group, या conditional calculations)। Performance: agg() >> apply() especially large datasets पर। 🚀
Q7: Time series analysis में resample() vs groupby() — difference क्या है?
Answer: resample() specifically time-based grouping के लिए designed है — DatetimeIndex require करता है और frequency strings ('D', 'W', 'M') accept करता है। groupby(df['date'].dt.month) भी काम करेगा but resample() के advantages: (1) handles missing periods, (2) supports forward/backward fill, (3) rolling windows compatible। Time series work में हमेशा resample() prefer करो। ⏰
Q8: EDA automated tools (pandas-profiling/ydata-profiling) vs manual EDA — कौनसा better?
Answer: Both have their place! ydata-profiling (earlier pandas-profiling) quick overview देता है — एक line में full HTML report। But manual EDA से specific business questions answer कर सकते हो, custom visualizations बना सकते हो, और domain-specific analysis कर सकते हो। Best practice: automated report से start करो (overview), फिर manual EDA करो (deep dive)। Interviews में manual EDA demonstrate करना ज्यादा impressive है। 🏆
🎯 Interview Questions
Q1: What is Exploratory Data Analysis and why is it important?
Answer: EDA is the process of analyzing datasets to summarize their main characteristics, often using statistical graphics and other data visualization methods. It's important because:
- Discovers patterns — trends, seasonality, clusters identify होते हैं
- Finds anomalies — outliers, missing data, data quality issues detect होते हैं
- Tests assumptions — normality, independence, homogeneity verify करते हैं
- Guides modeling — feature selection, transformation decisions inform करता है
- Communicates findings — stakeholders को data story tell करता है
Real example: E-commerce data पर EDA करके discover किया कि 80% revenue top 20% customers से आती है (Pareto principle), which led to a VIP loyalty program. 📊
Q2: Explain the difference between mean, median, and mode. When would you use each?
Answer:
- Mean (average) — sum/count, sensitive to outliers. Use when data is normally distributed (exam scores, temperature)
- Median (middle value) — robust to outliers. Use for skewed data (income, house prices, order values)
- Mode (most frequent) — works for categorical data too. Use for finding most common category (popular product, common payment method)
Example: Company salary data — Mean salary = ₹15L (CEO's ₹2Cr pulls it up), Median = ₹8L (actual middle), Mode = ₹6L (most common). Median is most representative here. ⚖️
Q3: How do you handle outliers in data analysis?
Answer: Step-by-step approach:
- Detect — IQR method (
Q1 - 1.5*IQRtoQ3 + 1.5*IQR), Z-score (|z| > 3), visual (box plots) - Investigate — Is it a data error? Genuine extreme value? Different population?
- Decide — Options: Remove (if error), Cap/Winsorize (replace with percentile boundary), Transform (log), Keep (if genuine), Separate analysis
Example
Q1 = df['amount'].quantile(0.25)
Q3 = df['amount'].quantile(0.75)
IQR = Q3 - Q1
outliers = df[(df['amount'] < Q1 - 1.5*IQR) | (df['amount'] > Q3 + 1.5*IQR)]
Key principle: Never blindly remove outliers — always document your decision and reasoning. 🎯
Q4: What is correlation and what are its limitations?
Answer: Correlation measures the linear relationship between two variables (range: -1 to +1).
- Pearson's r — linear relationship, requires normality
- Spearman's ρ — monotonic relationship, rank-based, non-parametric
- Kendall's τ — ordinal association, robust for small samples
Limitations:
- Only captures linear relationships (quadratic relationship may show r ≈ 0)
- Correlation ≠ Causation — confounding variables exist
- Sensitive to outliers (Pearson especially)
- Sample size affects significance — large n makes small r "significant"
Always plot your data! Anscombe's quartet shows 4 datasets with same r but very different patterns. 📈
Q5: Explain RFM Analysis and how you would implement it.
Answer: RFM segments customers based on:
- Recency — How recently did they purchase? (lower = better)
- Frequency — How often do they purchase? (higher = better)
- Monetary — How much do they spend? (higher = better)
Implementation:
- Calculate R, F, M values per customer
- Score each on 1-5 scale using quintiles (
pd.qcut) - Combine scores (e.g., "555" = best customer)
- Define segments: Champions (5,5,5), Loyal (4+,4+,*), At Risk (1-2,4+,4+), Lost (1,1,1)
Business impact: Targeted marketing campaigns increase ROI by 3-5x compared to mass campaigns. Champions get exclusive deals, At Risk get win-back offers, New customers get onboarding sequences. 🏆
Q6: How do you determine if a difference between two groups is statistically significant?
Answer: Follow this decision tree:
- Check normality — Shapiro-Wilk test on both groups
- Check variance homogeneity — Levene's test
- Choose test:
- Normal + equal variance → Independent t-test
- Normal + unequal variance → Welch's t-test
- Not normal → Mann-Whitney U test
- 3+ groups → ANOVA (normal) or Kruskal-Wallis (non-normal)
- Interpret: p < 0.05 = significant (reject null hypothesis)
- Report effect size (Cohen's d) — statistical significance ≠ practical significance
python example
from scipy import stats
t_stat, p_value = stats.ttest_ind(group_a, group_b)
# Cohen's d
d = (group_a.mean() - group_b.mean()) / np.sqrt((group_a.std()**2 + group_b.std()**2) / 2)Q7: What is the difference between resample() and groupby() for time series data?
Answer:
| Feature | resample() | groupby() |
|---|---|---|
| Requires | DatetimeIndex | Any column |
| Frequency | 'D', 'W', 'M', 'Q', 'Y' | Manual extraction (dt.month) |
| Missing periods | Creates rows for empty periods | Skips missing |
| Fill methods | ffill, bfill, interpolate | Not built-in |
| Rolling compatible | Yes, directly | Need extra steps |
Use resample() when you want continuous time series with consistent frequency. Use groupby() when you want categorical time grouping (e.g., compare all Januarys across years). ⏰
Q8: How would you present EDA findings to a non-technical stakeholder?
Answer: Key principles:
- Start with the business question — "Here's what we wanted to find out"
- Lead with insights, not methods — "Sales peak on Fridays" not "GroupBy with mean aggregation shows..."
- Use visualizations — bar charts, line trends, pie charts (simple ones)
- Provide context — "Revenue grew 15% MoM, which is 2x industry average"
- End with recommendations — "Based on this, I recommend..."
Structure: Executive Summary → Key Findings (3-5 bullet points) → Supporting Analysis → Recommendations → Appendix (technical details). Always tie numbers back to business impact (revenue, cost, customer satisfaction). 📋
Q9: What are the assumptions of Pearson correlation and what happens when they're violated?
Answer: Assumptions:
- Linearity — relationship should be linear (violation: underestimates strength of curved relationships)
- Normality — both variables approximately normal (violation: unreliable p-values)
- Homoscedasticity — constant variance (violation: biased estimates)
- No outliers — extreme values distort results (violation: inflated/deflated r)
- Continuous data — both variables measured on interval/ratio scale
When violated: Use Spearman's rank correlation (handles non-linearity, ordinal data, outliers) or Kendall's tau (small samples, many ties). Always visualize with scatter plot before computing correlation. 🔗
Q10: How do you handle the "multiple testing problem" in EDA?
Answer: When you perform many statistical tests, the probability of at least one false positive increases. With 20 tests at α=0.05, expected false positives = 1.
Solutions:
- Bonferroni correction — divide α by number of tests (α/n). Conservative but simple
- Holm-Bonferroni — step-down procedure, less conservative
- False Discovery Rate (FDR) — Benjamini-Hochberg procedure, controls proportion of false discoveries
- Pre-registration — decide which tests to run before looking at data
python example
from statsmodels.stats.multitest import multipletests
rejected, p_corrected, _, _ = multipletests(p_values, method='bonferroni')In EDA context: treat findings as hypothesis-generating, not hypothesis-confirming. Validate important findings on held-out data or with domain experts. 🎯
*Next Lesson: Matplotlib →*
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Data Analysis with Python.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Python Master Course topic.
Search Terms
python-master-course, python master course, python, master, course, data, science, analysis
Related Python Master Course Topics
Continue learning this concept
Data ScienceData Cleaning with PythonMaster data cleaning techniques using Python, Pandas, and NumPy. Learn to handle missing values, duplicates, outliers, data type conversions, string cleaning, and building a complete data cleaning pipeline.Data ScienceData Visualization ProjectBuild a complete data visualization project analyzing real-world COVID-19 and economic data. Step-by-step tutorial covering data loading, cleaning, EDA, and creating a professional multi-panel visualization report.Data ScienceData Visualization with MatplotlibMaster Matplotlib, PythonData ScienceNumPy IntroductionLearn NumPy, the foundational Python library for numerical computing. Master arrays, array operations, broadcasting, indexing, reshaping, linear algebra, and random number generation with practical examples.Data SciencePandas IntroductionLearn Pandas, the powerful Python data manipulation library. Master DataFrames, Series, data loading, selection, filtering, groupby, merging, and essential data operations for data analysis.Data ScienceStatistical Visualization with SeabornLearn Seaborn for beautiful statistical data visualization. Master distribution plots, categorical plots, regression plots, heatmaps, pair plots, and FacetGrid with practical data science examples.