Machine Learning Project: Churn Prediction

In this end-to-end project, you'll build a production-ready machine learning system to predict which customers are likely to cancel their subscription. This is one of the most common and valuable ML applications in business.

Project Setup

Step 1: Generate Dataset

Step 2: Exploratory Data Analysis

Step 3: EDA Visualization

Step 4: Feature Engineering

Step 5: Build ML Pipeline

Step 6: Train and Compare Models

Step 7: Hyperparameter Tuning

Step 8: Final Evaluation and Report

Step 9: Prediction Function (Production Ready)

Project Summary

Final Model Performance: ~0.88 AUC, ~0.72 F1 score

Extensions to Explore

1. SHAP values — explain individual predictions
2. Threshold optimization — optimize precision-recall tradeoff for business goals
3. Online learning — update model with new data monthly
4. A/B testing — test model effectiveness on real customers
5. REST API — serve predictions via Flask/FastAPI

📤 Expected Outputs

Step 1 Output: Dataset Generation

Dataset shape: (5000, 16)
Churn rate: 26.8%

Churn distribution:
0    3660
1    1340
Name: churn, dtype: int64

   customer_id  tenure_months  age  gender    contract_type    payment_method  monthly_charges  num_services  tech_support_calls  late_payments  paperless_billing internet_service  senior_citizen  total_charges  charges_per_service  churn
0   CUST-00001             52   45    Male       One Year      Bank Transfer         67.45             4                   1              0                  1      Fiber Optic               0        3507.40                16.86      0
1   CUST-00002              8   28  Female  Month-to-Month  Electronic Check         95.20             3                   2              3                  1              DSL               0         761.60                31.73      1
2   CUST-00003             34   62    Male       Two Year      Credit Card          42.80             6                   0              0                  0               No               0        1455.20                 7.13      0

Step 2 Output: EDA Results

============================================================
EXPLORATORY DATA ANALYSIS
============================================================

📊 Overview: 5,000 customers, 16 features
Churn rate: 26.8%  (1,340 churned)

✅ No missing values

📋 Churn Rate by Contract:
                  Churn_Rate%  Churned  Total
contract_type                                
Month-to-Month         38.2      1051   2750
One Year               14.5       181   1250
Two Year                10.8      108   1000

📋 Churn Rate by Tenure:
tenure_group
0-1 yr     35.2
1-2 yr     28.1
2-4 yr     22.4
4+ yr      16.3

Step 4 Output: Feature Engineering

Features after engineering: 26
New features: ['charges_x_late', 'calls_x_charges', 'is_month_to_month', 'is_electronic_check', 'is_fiber', 'high_monthly_charges', 'new_customer', 'long_term_customer', 'has_issues', 'tenure_bracket']

Step 5 Output: Pipeline Setup

Numeric features (20): ['tenure_months', 'age', 'monthly_charges', 'num_services', 'tech_support_calls']...
Categorical features (3): ['gender', 'contract_type', 'payment_method']

Train: (4000, 26), Test: (1000, 26)
Train churn rate: 26.8%
Test churn rate:  26.8%

Step 6 Output: Model Comparison

=== MODEL TRAINING ===

📊 Logistic Regression
   CV AUC: 0.8215 ± 0.0134
   Test AUC: 0.8189, F1: 0.6421

📊 Random Forest
   CV AUC: 0.8654 ± 0.0098
   Test AUC: 0.8701, F1: 0.6932

📊 XGBoost
   CV AUC: 0.8812 ± 0.0087
   Test AUC: 0.8845, F1: 0.7156

🏆 Best Model: XGBoost (AUC = 0.8845)

Step 7 Output: Hyperparameter Tuning

Fitting 3 folds for each of 25 candidates, totalling 75 fits

Best hyperparameters:
  clf__colsample_bytree: 0.8
  clf__learning_rate: 0.05
  clf__max_depth: 5
  clf__min_child_weight: 3
  clf__n_estimators: 287
  clf__scale_pos_weight: 2.73
  clf__subsample: 0.9
Best CV AUC: 0.8891
Final Test AUC: 0.8923
Final Test F1:  0.7248

Step 8 Output: Final Evaluation

============================================================
FINAL MODEL EVALUATION REPORT
============================================================

              precision    recall  f1-score   support

    Retained       0.92      0.89      0.90       732
     Churned       0.72      0.78      0.75       268

    accuracy                           0.86      1000
   macro avg       0.82      0.83      0.83      1000
weighted avg       0.87      0.86      0.86      1000

📈 BUSINESS IMPACT
  Total customers evaluated: 1,000
  Actual churners: 268 (26.8%)
  Customers flagged for intervention: 289
  Correctly identified churners: 209
  Missed churners (FN): 59
  False alarms (FP): 80

  Assuming avg customer value = $500/month:
  Potential monthly revenue saved: $104,500

✅ Model saved: churn_model.joblib
✅ Metadata saved: churn_model_metadata.json

Step 9 Output: Prediction Function

🔮 CHURN PREDICTION
  Churn Probability:    78.3%
  Prediction:           Churn
  Risk Level:           High
  Recommended Action:   Urgent retention offer

⚠️ Common Mistakes

1. Data Leakage करना 🚫 — Test data का information training time पर use करना (जैसे पूरा dataset scale करके फिर split करना)। Always split FIRST, फिर fit_transform only on training data!

1. Imbalanced Data ignore करना — Churn dataset naturally imbalanced होता है (70-80% retained)। बिना class_weight='balanced' या scale_pos_weight के model majority class ही predict करेगा।

1. Feature Engineering skip करना — Raw features से directly model train करना। Interaction features (charges_x_late), binary flags (is_month_to_month) model performance significantly improve करते हैं।

1. Only Accuracy देखना ❌ — Imbalanced datasets पर accuracy misleading होती है। 90% accuracy मतलब model ने सब "Not Churn" predict किया! AUC-ROC, F1-score, Precision-Recall देखो।

1. Cross-Validation नहीं करना — Single train-test split से model performance vary करती है। StratifiedKFold use करो ताकि हर fold में churn ratio same रहे।

1. Hyperparameter Tuning भूलना — Default parameters rarely optimal होते हैं। RandomizedSearchCV या GridSearchCV से model को tune करो — 3-5% improvement common है।

1. Business Context ignore करना 💼 — Model metrics अच्छे हैं but business impact calculate नहीं किया। False Negatives (missed churners) vs False Positives (unnecessary intervention) का cost different होता है — threshold accordingly adjust करो।

✅ Key Takeaways

• 🎯 End-to-end ML workflow follow करो: Data → EDA → Feature Engineering → Model → Evaluation → Deployment — कोई step skip मत करो

• 📊 EDA सबसे important step है — Data समझे बिना model बनाना "andhe mein teer maarna" है। Churn rate by segment, distributions, correlations analyze करो

• 🔧 Feature Engineering = Performance Boost — Domain knowledge से meaningful features create करो (is_month_to_month, has_issues, new_customer) — raw data से better signal मिलता है

• 🏗️ Pipelines use करो — ColumnTransformer + Pipeline ensure करते हैं कि preprocessing steps train और test दोनों पर consistent apply हों, data leakage prevent होती है

• ⚖️ Imbalanced data handle करो — class_weight='balanced', scale_pos_weight, SMOTE, या threshold tuning — imbalance ignore करने से model useless बनता है

• 📈 Multiple models compare करो — एक ही model पर depend मत हो। Logistic Regression (baseline), Random Forest, XGBoost compare करके best select करो

• 🔍 Hyperparameter tuning significant है — RandomizedSearchCV efficient way है optimal parameters find करने का, typically 3-5% improvement देता है

• 💰 Business metrics calculate करो — AUC अच्छा है but "कितना revenue save होगा?" वो stakeholders को समझ आता है। Model metrics को business value में translate करो

• 💾 Model saving essential है — joblib.dump() से model save करो, metadata (features, performance, date) भी save करो — reproducibility और deployment दोनों के लिए

• 🚀 Production-ready prediction function बनाओ — जो single customer data accept करे, feature engineering करे, और actionable output दे (risk level, recommended action)

❓ FAQ

Q1: Churn prediction project में कौन सा model सबसे अच्छा perform करता है?

Generally XGBoost या Gradient Boosting best perform करते हैं churn prediction में because वो non-linear relationships, feature interactions, और imbalanced data efficiently handle करते हैं। But always start with Logistic Regression as baseline — कभी-कभी well-engineered features के साथ simpler models भी competitive होते हैं। Model choice data pe depend करता है!

Q2: Data Leakage कैसे avoid करें?

Golden rule: Train-test split FIRST, फिर बाकी सब। Pipeline use करो जो fit_transform() only training data पर करे और transform() test data पर। कभी भी test data की statistics (mean, std) training time पर use मत करो। total_charges जैसे features जो future info contain करते हैं, उन्हें carefully handle करो।

Q3: Imbalanced dataset handle करने के कौन-कौन से तरीके हैं?

Multiple approaches हैं: (1) class_weight='balanced' parameter, (2) XGBoost में scale_pos_weight, (3) SMOTE/ADASYN oversampling, (4) Undersampling majority class, (5) Threshold tuning (default 0.5 से 0.3-0.4 पर लाना), (6) Focal Loss use करना। Best approach depends on dataset size और business requirements।

Q4: Feature Engineering में कौन-कौन सी techniques useful हैं?

Churn projects में: (1) Interaction features — charges × late_payments, (2) Binary flags — is_month_to_month, new_customer, (3) Binning — tenure brackets, (4) Ratio features — charges_per_service, (5) Aggregation — customer lifetime value, (6) Time-based — trend in usage last 3 months। Domain knowledge सबसे valuable है!

Q5: Model को production में deploy कैसे करें?

Steps: (1) joblib.dump() से model + preprocessor save करो, (2) FastAPI/Flask से REST endpoint create करो, (3) Docker container बनाओ, (4) AWS SageMaker / GCP AI Platform पर deploy करो, (5) Monitoring setup करो (data drift, model performance), (6) A/B testing framework implement करो। Start simple with Flask API, फिर gradually scale करो।

Q6: Cross-validation में StratifiedKFold क्यों use करते हैं regular KFold नहीं?

Churn dataset imbalanced होता है (26% churn vs 74% retained)। Regular KFold randomly split करता है — किसी fold में 10% churn हो सकता है, किसी में 40%। StratifiedKFold ensure करता है कि हर fold में same churn ratio maintain हो, जिससे model evaluation reliable और consistent होती है।

Q7: AUC-ROC vs F1-Score — कब कौन सा metric use करें?

AUC-ROC use करो जब आपको overall model ranking ability assess करनी हो — across all thresholds कैसा perform करता है। F1-Score use करो जब fixed threshold पर precision-recall balance important हो। Business scenario: "Top 20% risky customers find करो" → AUC better; "Each customer को retain/churn classify करो" → F1 better। Ideally दोनों report करो!

Q8: Model explain कैसे करें — कौन सा feature important है?

Multiple ways: (1) Feature Importance — model.feature_importances_ (tree-based models), (2) SHAP values — individual prediction explain करते हैं (global + local), (3) Permutation Importance — model-agnostic approach, (4) Partial Dependence Plots — feature vs prediction relationship visualize करते हैं। Stakeholders को SHAP summary plots सबसे ज़्यादा convince करते हैं।

🎯 Interview Questions

Q1: Walk me through the complete ML pipeline you'd build for a churn prediction project.

Answer: Complete pipeline includes: (1) Data Collection & Understanding — gather customer data, understand business context, define target variable; (2) EDA — analyze churn rate, distributions, correlations, missing values; (3) Feature Engineering — create interaction features, binary flags, aggregations from domain knowledge; (4) Preprocessing — handle missing values, encode categoricals, scale numerics using ColumnTransformer + Pipeline; (5) Model Selection — train baseline (Logistic Regression) + advanced models (Random Forest, XGBoost), compare via cross-validation; (6) Hyperparameter Tuning — RandomizedSearchCV/GridSearchCV with stratified CV; (7) Evaluation — AUC-ROC, F1, classification report, business metrics; (8) Deployment — save model with joblib, create prediction API, setup monitoring।

Q2: How do you handle class imbalance in a churn dataset? What's the tradeoff?

Answer: Techniques: (1) Algorithm-level — class_weight='balanced' (sklearn), scale_pos_weight (XGBoost); (2) Data-level — SMOTE oversampling, random undersampling, ADASYN; (3) Threshold tuning — lower decision threshold from 0.5 to 0.3; (4) Ensemble methods — BalancedRandomForest, EasyEnsemble; (5) Cost-sensitive learning — assign higher misclassification cost to minority class। Tradeoff: Addressing imbalance increases recall (catch more churners) but typically decreases precision (more false alarms)। Business context decides optimal balance — cost of missing a churner vs cost of unnecessary retention offers।

Q3: Explain the difference between data leakage and overfitting. How do you prevent each?

Answer: Data Leakage = training model on information that won't be available at prediction time (e.g., using future data, fitting scaler on test set)। Prevention: split first, use Pipelines, carefully audit temporal features। Overfitting = model memorizes training data noise, performs poorly on unseen data। Prevention: regularization (L1/L2), max_depth limits, min_samples_leaf, cross-validation, early stopping, dropout। Key difference: leakage gives falsely HIGH metrics even on test set; overfitting shows gap between train and test performance।

Q4: Why did you choose XGBoost over Random Forest for churn prediction? When might Random Forest be better?

Answer: XGBoost advantages: (1) Sequential boosting corrects previous errors → better accuracy, (2) Built-in regularization (L1/L2), (3) scale_pos_weight for imbalance, (4) Handles missing values natively, (5) Generally 2-5% higher AUC on tabular data। Random Forest better when: (1) Dataset is very small (less prone to overfitting), (2) Less tuning time available (more robust with defaults), (3) Interpretability is priority (simpler feature importance), (4) Training speed matters (parallel trees vs sequential)। In production, XGBoost's marginal accuracy gain may not justify complexity — depends on business value of that 2-3% improvement।

Q5: How would you explain model predictions to non-technical stakeholders?

Answer: Strategy: (1) SHAP Waterfall plots — show how each feature pushes prediction towards churn/retain for specific customers; (2) Business language — "This customer has 78% churn risk because: Month-to-Month contract (+20%), 4 tech support calls (+12%), only 3 months tenure (+10%)"; (3) Summary statistics — "Model identifies 80% of churners in top 30% risk score"; (4) Lift charts — "Targeting top 20% risky customers gives 3.5x more churners than random"; (5) Dollar impact — "Model saves estimated $104,500/month in retained revenue"। Always translate ML metrics into business outcomes।

Q6: What metrics would you use to monitor this model in production?

Answer: Three categories: (1) Model Performance — track AUC, F1, precision, recall weekly on new labeled data (when customers actually churn); (2) Data Drift — monitor input feature distributions (PSI/KS-test), detect when data generating process changes (new pricing plan, COVID impact); (3) Prediction Drift — average predicted probability shifting, prediction volume changes; (4) Business Metrics — actual churn rate vs predicted, intervention success rate, revenue saved; (5) System Health — latency, throughput, error rates। Set alerts: retrain if AUC drops >5% or PSI > 0.2 on any key feature।

Q7: How would you decide the optimal threshold for churn classification?

Answer: Default 0.5 threshold is rarely optimal for business। Approach: (1) Plot Precision-Recall curve — see tradeoff at each threshold; (2) Calculate business cost at each threshold — cost of false negative (lost customer = $500/month lifetime) vs false positive (unnecessary retention offer = $50); (3) Optimize for business objective — if FN cost >> FP cost, lower threshold to catch more churners; (4) Use Youden's J-statistic (sensitivity + specificity - 1) for balanced tradeoff; (5) Segment-based thresholds — high-value customers get lower threshold (more aggressive retention)। Typical churn projects use 0.3-0.4 threshold।

Q8: What's the role of feature engineering in this project? Give examples of features that significantly improved performance.

Answer: Feature engineering contributed ~5-8% AUC improvement in this project। Key engineered features: (1) is_month_to_month — strongest predictor, month-to-month contract = highest churn; (2) new_customer (tenure < 12 months) — early tenure customers churn 2x more; (3) charges_x_late — interaction between high charges AND late payments = compounding risk; (4) has_issues — combined tech support + late payments flag; (5) tenure_bracket — non-linear tenure effect captured via binning। Philosophy: domain knowledge > automated feature selection। Talk to business stakeholders — they know WHY customers leave।

Q9: How would you handle a scenario where model performance degrades in production?

Answer: Systematic debugging: (1) Check data quality — missing values increased? New categories in categorical features? Schema changes? (2) Detect data drift — compare recent feature distributions vs training data using PSI/KS-test; (3) Check for concept drift — customer behavior changed (e.g., competitor launched new plan); (4) Retraining strategy — sliding window (last 6 months data) or expanding window; (5) A/B test — compare retrained model vs current in production; (6) Root cause analysis — which segment degraded most? (7) Escalation plan — fallback to simpler rule-based system if model completely fails। Prevention: automated monitoring + scheduled retraining pipeline।

Q10: Compare Logistic Regression vs Tree-based models for churn. When is Logistic Regression sufficient?

Answer: Logistic Regression strengths: (1) Highly interpretable — coefficients directly show feature impact, (2) Fast training and prediction, (3) Works well with good feature engineering, (4) Less prone to overfitting on small datasets, (5) Provides calibrated probabilities naturally। When sufficient: Small datasets (<1000 samples), regulatory requirements (explainability needed), real-time serving with strict latency, features are already well-engineered and relationships are roughly linear। Tree-based better when: Large datasets, complex non-linear interactions, many features, can afford some interpretability loss for 3-5% accuracy gain। In practice: always train Logistic Regression as baseline — if it achieves 0.82 AUC vs XGBoost's 0.85, the simpler model might win on deployment complexity and maintainability。

*Next Section: Projects →*