Python Notes
Build a complete data visualization project analyzing real-world COVID-19 and economic data. Step-by-step tutorial covering data loading, cleaning, EDA, and creating a professional multi-panel visualization report.
In this project, we'll build a comprehensive data visualization report for a fictional global e-commerce company. You'll apply all data analysis and visualization techniques from previous lessons to create a professional analytics report.
Project Overview
Goal: Analyze e-commerce sales data and create a 6-panel visualization dashboard.
Skills Used: Pandas, NumPy, Matplotlib, Seaborn, data cleaning, EDA, statistical analysis.
Output: A professional PNG/PDF visualization report and written data story.
Setup
pip install pandas numpy matplotlib seaborn scipyStep 1: Generate the Dataset
Step 2: Data Validation and Cleaning
Step 3: Compute Key Metrics
Step 4: Build the Dashboard
Step 5: Statistical Insights
Step 6: Export Report
Project Summary
Congratulations! You've built a complete data analysis and visualization project. Here's what you accomplished:
| Step | Task | Tools Used |
|---|---|---|
| 1 | Generate realistic dataset | NumPy, Pandas |
| 2 | Data validation & quality checks | Pandas |
| 3 | Compute KPIs and metrics | Pandas GroupBy, agg |
| 4 | Build 7-panel dashboard | Matplotlib, Seaborn |
| 5 | Statistical analysis | SciPy, statistical tests |
| 6 | Export reports | Pandas CSV export |
Key Skills Demonstrated
- ✅ Data generation and manipulation with Pandas/NumPy
- ✅ Multi-metric aggregation with
groupby().agg() - ✅ Multi-panel figures with
GridSpec - ✅ Mixing Matplotlib and Seaborn in one figure
- ✅ KPI callouts and annotation
- ✅ Statistical hypothesis testing
- ✅ Professional dashboard design
Extending This Project
- Add forecasting with Prophet or ARIMA
- Build an interactive dashboard with Plotly Dash
- Add geographic maps with GeoPandas and Folium
- Create an automated report with Jupyter nbconvert or PDFKit
- Connect to real data via APIs (Stripe, Shopify, etc.)
📤 Expected Outputs
Setup Output:
All libraries imported successfully!
Step 1 Output (Generate Dataset):
Dataset: 5,000 rows × 17 columns
Date range: 2025-01-01 to 2025-12-31
Total revenue: $2,847,653.21
order_id object
date datetime64[ns]
product object
category object
region object
customer_type object
quantity int64
unit_price float64
total_amount float64
discount_pct int64
returned bool
rating int64
discounted_amount float64
month int64
month_name object
day_of_week object
quarter int64
week int64
dtype: object
quantity unit_price total_amount discount_pct rating discounted_amount month quarter week
count 5000.0 5000.00 5000.00 5000.00 5000.0 5000.00 5000.0 5000.0 5000.0
mean 3.2 342.18 1095.47 4.75 4.11 1042.35 7.2 2.8 29.4
std 5.1 389.42 2187.93 6.12 0.98 2102.67 3.4 1.1 15.2
min 1.0 25.52 25.52 0.00 1.00 21.69 1.0 1.0 1.0
max 24.0 1348.91 26897.50 20.00 5.00 26897.50 12.0 4.0 53.0Step 2 Output (Data Validation):
=== DATA QUALITY REPORT === ✅ No missing values ✅ Duplicate orders: 0 ✅ Negative amounts: 0 ✅ Negative quantities: 0 📅 Date range: 364 days 📊 Records by category: Accessories 1850 Electronics 1750 Audio 700 Office 700 Data validation complete!
Step 3 Output (Key Metrics):
Total Revenue: $2,847,653.21
Total Orders: 5,000
Average Order Value: $569.53
Top 5 Products:
revenue units avg_rating
product
Laptop Pro 892456.32 1023 4.12
Laptop Air 587234.18 832 4.08
4K Monitor 312456.78 712 4.15
Standing Desk 287654.23 456 4.21
Ergonomic Chair 198765.43 389 4.18Step 4 Output (Dashboard):
Dashboard saved to ecommerce_dashboard.png!
Step 5 Output (Statistical Insights):
============================================================ KEY BUSINESS INSIGHTS ============================================================ 📈 REVENUE TRENDS Best month: Nov ($412,345) Worst month: Jan ($125,678) Jan → Dec growth: 187.3% 🏷️ CATEGORY INSIGHTS Dominant category: Electronics (52.3% revenue) Highest return rate: Accessories (4.8%) 👥 CUSTOMER INSIGHTS Highest AOV: Enterprise ($2,847.32) Most orders: Retail (3,000 orders) 📊 STATISTICAL TEST: Retail vs Enterprise order values t-statistic: -18.4532, p-value: 0.000000 Result: Significantly different 🎄 SEASONALITY Q4 share of annual revenue: 38.2% ============================================================
Step 6 Output (Export):
✅ CSV exports complete! - monthly_summary.csv - category_summary.csv - top_products.csv - regional_summary.csv - data_sample.csv
⚠️ Common Mistakes
1. 🚫 plt.show() ke baad figure save karna
Galat: plt.show() call karne ke baad savefig() use karna — isse blank image save hoti hai!
# ❌ Wrong
plt.show()
fig.savefig('dashboard.png') # Blank file!
# ✅ Correct
fig.savefig('dashboard.png', dpi=150, bbox_inches='tight')
plt.show()2. 🚫 Heatmap ke liye data pivot na karna
Galat: Seaborn heatmap ko directly raw data pass karna — pehle .pivot() ya .pivot_table() use karo.
# ❌ Wrong - raw grouped data
sns.heatmap(grouped_data)
# ✅ Correct - pivoted data
pivot = df.pivot_table(values='revenue', index='day', columns='month', aggfunc='sum')
sns.heatmap(pivot, annot=True, fmt='.0f')3. 🚫 Color palette mein insufficient colors
Galat: 5 categories hain lekin sirf 3 colors define kiye — graph inconsistent dikhega.
4. 🚫 GridSpec spacing set na karna
Galat: Multi-panel dashboard mein hspace aur wspace tune na karna — panels overlap ho jaate hain.
# ❌ Wrong - no spacing
gs = gridspec.GridSpec(3, 3)
# ✅ Correct - proper spacing
gs = gridspec.GridSpec(3, 3, hspace=0.45, wspace=0.35, top=0.92, bottom=0.05)5. 🚫 Large dataset mein loop se plotting
Galat: Har row ke liye individual plot call — extremely slow! Vectorized approach use karo.
6. 🚫 Figure close na karna (Memory leak)
Galat: Multiple figures create karke plt.close() na call karna — memory leak hoti hai.
7. 🚫 twinx() ke saath legend handle na karna
Galat: Dual-axis chart mein dono axes ki legends separate dikhti hain ya missing hoti hain.
# ❌ Wrong - separate/missing legends
ax1.legend()
ax2.legend()
# ✅ Correct - combined legend
lines1, labels1 = ax1.get_legend_handles_labels()
lines2, labels2 = ax2.get_legend_handles_labels()
ax1.legend(lines1 + lines2, labels1 + labels2, loc='upper left')✅ Key Takeaways
- 📊 Dashboard Design Pattern:
GridSpecuse karke multi-panel layouts create karo — yeh professional reports ke liye industry standard hai - 🧹 Data Validation First: Visualization se pehle ALWAYS data quality check karo — missing values, duplicates, negative amounts detect karo
- 📈 KPI Summary Row: Dashboard ke top par KPI boxes lagao — executives ko quick snapshot chahiye bina details mein jaaye
- 🎨 Color Consistency: Puri report mein consistent color palette use karo —
cat_colorsaurregion_colorsdefine karke reuse karo - 🔢 Aggregation Before Plotting: Raw data directly plot mat karo — pehle
groupby().agg()se summarize karo, phir visualize karo - 📉 Statistical Validation: Insights ko support karne ke liye statistical tests (t-test, correlation) use karo — "looks different" enough nahi hai
- 💾 Save Before Show:
fig.savefig()ALWAYSplt.show()se pehle call karo — otherwise blank image save hogi - 🗺️ Heatmaps for 2D patterns: Jab do categorical variables ka interaction dikhana ho (day × month), heatmap best choice hai
- 📁 Modular Code Structure: Data generation, validation, metrics, visualization, aur export ko alag functions mein rakho — reusability badhti hai
- 🔄 Reproducibility:
np.random.seed()set karo taaki same results baar baar aa sakein — team collaboration mein essential hai
❓ FAQ
Q1: Matplotlib aur Seaborn ko ek hi figure mein mix kar sakte hain? 🤔
Ans: Haan, bilkul! Seaborn internally Matplotlib use karta hai. Aap fig.add_subplot() se axes create karke kisi mein sns.heatmap(ax=ax7) aur kisi mein ax1.plot() use kar sakte ho. Yeh project exactly yehi approach use karta hai — Panel 7 mein Seaborn heatmap hai aur baaki panels Matplotlib se bane hain.
Q2: Dashboard ko PDF mein kaise save karein PNG ki jagah? 📄
Ans: Sirf file extension change karo: fig.savefig('dashboard.pdf', dpi=150, bbox_inches='tight'). Matplotlib automatically format detect kar leta hai extension se. PDF vector format hai isliye zooming ke baad bhi sharp rahega — presentations ke liye prefer karo.
Q3: Real-time data ke saath yeh dashboard kaise use karein? 🔄
Ans: generate_ecommerce_data() function ki jagah actual data source connect karo — CSV file (pd.read_csv()), database (pd.read_sql()), ya API (requests + pd.DataFrame()). Baaki pipeline (metrics, dashboard) same rahegi. Production mein Plotly Dash ya Streamlit use karo for interactive dashboards.
Q4: gridspec.GridSpec aur plt.subplots() mein kya farak hai? 🧩
Ans: plt.subplots(3,3) uniform grid banata hai — sab panels same size ke hote hain. GridSpec flexible hai — ek panel ko 2 columns span kara sakte ho (jaise Panel 1 revenue trend wide hai: gs[0, :2]). Complex dashboards ke liye hamesha GridSpec prefer karo.
Q5: Itna bada dataset generate karna kyun zaroori hai? Chhota dataset se kaam nahi chalega? 📊
Ans: 5000 rows realistic patterns dikhane ke liye minimum hai — seasonal trends, statistical tests, aur category-wise breakdowns meaningful results ke liye sufficient data chahiye. 100 rows mein seasonal pattern clearly nahi dikhega aur t-test unreliable hoga. Real-world projects mein lakhs/crores rows hoti hain.
Q6: bbox_inches='tight' kya karta hai savefig mein? 📐
Ans: Yeh extra whitespace remove karta hai figure ke around. Without it, saved image mein bahut saara blank space aa sakta hai, especially jab annotations ya legends figure ke bahar extend hoti hain. Always use karo for clean exports.
Q7: Heatmap mein NaN values aaye toh kya karein? 🔥
Ans: pivot_table mein fill_value=0 use karo ya sns.heatmap(data, mask=data.isnull()) se NaN cells ko grey/blank chhod do. Project mein humne unstack(fill_value=0) use kiya hai — agar koi day-month combination mein data nahi hai toh 0 show hoga.
Q8: Customer type analysis mein dual axis (twinx) kab use karna chahiye? 📈📉
Ans: Jab do metrics same x-axis share karein lekin unki scale bahut alag ho — jaise orders (thousands mein) aur average order value (dollars mein). Agar same scale hai toh grouped bar chart better hai. Twinx overuse mat karo — confusing ho sakta hai readers ke liye.
🎯 Interview Questions
Q1: Explain how you would design a multi-panel analytics dashboard from scratch.
Answer: Main approach step-by-step hota hai: (1) Identify KPIs — stakeholders se top 5-7 metrics finalize karo, (2) Choose chart types — line chart for trends, bar for comparisons, pie for proportions, heatmap for 2D patterns, (3) Layout design — GridSpec use karo taaki important panels ko zyada space mile (e.g., revenue trend wide), (4) Color consistency — ek color dictionary define karo aur poori report mein use karo, (5) KPI summary row — top par high-level numbers with formatting (e.g., $2.85M), (6) Statistical validation — har insight ko test se back karo, (7) Export — savefig() with dpi=150 aur bbox_inches='tight'.
Q2: What's the difference between groupby().agg() and groupby().apply()? When would you use each?
Answer: agg() multiple aggregation functions ek saath apply karta hai efficiently — sum, mean, count, etc. Yeh optimized hai aur fast hai. apply() custom function leta hai jo puri group DataFrame par operate karta hai — zyada flexible but slower. Use agg() jab standard aggregations chahiye (90% cases). Use apply() jab complex logic ho jaise conditional calculations, multiple columns ka interaction, ya custom statistical computations jo built-in functions se na ho sakein.
Q3: How do you handle overlapping labels in a complex visualization?
Answer: Multiple techniques hain: (1) rotation=45 ya rotation=90 for tick labels, (2) plt.tight_layout() ya bbox_inches='tight' for overall spacing, (3) textcoords='offset points' with xytext for annotation positioning, (4) fontsize reduce karo crowded areas mein, (5) Horizontal bar charts (barh) use karo jab category names lamba hon, (6) GridSpec mein hspace aur wspace increase karo. Is project mein humne Q4 annotations sirf specific months par lagaye — sab par nahi — taaki clutter avoid ho.
Q4: Explain the statistical significance test used in this project. When would you choose a different test?
Answer: Humne independent samples t-test (scipy.stats.ttest_ind) use kiya Retail vs Enterprise order values compare karne ke liye. Yeh test assume karta hai: (1) independent samples, (2) approximately normal distribution, (3) similar variance. Different test choose karein: (a) Mann-Whitney U — jab data normal na ho, (b) ANOVA — jab 3+ groups compare karne hon, (c) Chi-square — categorical variables ke liye, (d) Paired t-test — same subjects ka before/after comparison. P-value < 0.05 matlab difference statistically significant hai.
Q5: How would you make this dashboard interactive for production use?
Answer: Static Matplotlib dashboard ko interactive banane ke liye: (1) Plotly Dash — Python mein full web app, callbacks se interactivity, (2) Streamlit — rapid prototyping, st.selectbox() se filters, (3) Panel/Bokeh — complex layouts ke liye, (4) Tableau/Power BI — non-technical stakeholders ke liye. Architecture: Data pipeline (Airflow/Prefect) → Database (PostgreSQL) → Caching (Redis) → Dashboard framework → Deployment (Docker + cloud). Humara current code easily adapt hota hai — compute_kpis() function same rahega, sirf visualization layer change hogi.
Q6: What are best practices for choosing color palettes in data visualization?
Answer: Key principles: (1) Sequential palette (light→dark) quantitative data ke liye — like heatmap mein YlOrRd, (2) Categorical palette distinct groups ke liye — Set2, tab10, (3) Diverging palette jab center meaningful ho (e.g., profit/loss), (4) Colorblind-safe — sns.color_palette('colorblind') ya Viridis, (5) Max 7-8 distinct colors — usse zyada mein labeling use karo, (6) Consistent meaning — red = danger, green = success across all panels. Is project mein humne cat_colors define kiya aur consistently use kiya.
Q7: How do you ensure reproducibility in a data analysis project?
Answer: Reproducibility ke liye: (1) Random seed — np.random.seed(42) har stochastic operation se pehle, (2) Environment — requirements.txt ya environment.yml with pinned versions, (3) Version control — Git se code track karo, (4) Data versioning — DVC ya fixed snapshots, (5) Modular functions — har step independent function mein (jaise humara generate_ecommerce_data(), compute_kpis(), build_dashboard()), (6) Documentation — docstrings aur comments, (7) Automated pipeline — Makefile ya scripts se one-command execution.
Q8: Explain the concept of "data storytelling" and how this dashboard implements it.
Answer: Data storytelling = data + visuals + narrative combined for impact. Is dashboard mein: (1) Hook — KPI row immediately attention grab karta hai ($2.85M revenue), (2) Context — Monthly trend shows seasonal pattern (WHY revenue varies), (3) Detail — Category, region, product panels give granular insights, (4) Pattern — Heatmap reveals day×month interaction patterns, (5) Action — Statistical tests validate ki Enterprise significantly different hai from Retail. Flow: Overview → Trend → Breakdown → Pattern → Validation. Effective data story answer karti hai: "What happened?", "Why?", "So what?"
Q9: How would you handle a dataset with 10 million rows for this type of analysis?
Answer: Large data ke liye optimizations: (1) Chunked reading — pd.read_csv(chunksize=100000), (2) Efficient dtypes — category dtype for categorical columns (80% memory save), (3) Pre-aggregation — Database mein SQL GROUP BY karo, raw data Python mein mat laao, (4) Sampling — EDA ke liye 1% stratified sample use karo, final report mein full data, (5) Dask/Vaex — out-of-memory computation, (6) Parquet format — CSV ki jagah columnar format (10x faster reads), (7) Database backend — DuckDB ya PostgreSQL for aggregations. Visualization ke liye pre-aggregated data hi plot karo — 10M points plot karna meaningless hai.
Q10: What metrics would you add to this dashboard for a real e-commerce company?
Answer: Additional KPIs: (1) Customer Lifetime Value (CLV) — cohort analysis se, (2) Conversion Rate — visitors → orders funnel, (3) Cart Abandonment Rate, (4) Customer Acquisition Cost (CAC) — marketing spend / new customers, (5) Net Promoter Score (NPS) — rating data se derive, (6) Inventory Turnover — stock efficiency, (7) Churn Rate — monthly active customers track karo, (8) Revenue per Visit (RPV), (9) Refund Rate trend — monthly track for quality issues, (10) Cohort Retention — month-over-month repeat purchase rate. Priority depends on business stage — growth stage mein CAC/CLV ratio matter karta hai, mature stage mein retention aur NPS.
*Next Section: Machine Learning →*
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Data Visualization Project.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Python Master Course topic.
Search Terms
python-master-course, python master course, python, master, course, data, science, visualization
Related Python Master Course Topics