Python Notes
Master ML model evaluation — cross-validation strategies, evaluation metrics for regression and classification, overfitting detection, hyperparameter tuning, model selection best practices, and model persistence with joblib.
Proper model evaluation is critical to building reliable ML systems. This lesson covers comprehensive evaluation strategies, metrics, overfitting detection, and model persistence.
Setup
pip install scikit-learn pandas numpy matplotlib seaborn joblibimport numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import (
train_test_split, cross_val_score, cross_validate,
KFold, StratifiedKFold, RepeatedStratifiedKFold,
learning_curve, validation_curve, GridSearchCV
)
from sklearn.metrics import *
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_breast_cancer, load_boston
import warnings
warnings.filterwarnings('ignore')Cross-Validation Strategies
Classification Metrics Deep Dive
ROC and Precision-Recall Curves
Regression Metrics Deep Dive
from sklearn.metrics import (
mean_squared_error, mean_absolute_error, r2_score,
mean_absolute_percentage_error, explained_variance_score,
max_error, mean_squared_log_error
)
import numpy as np
# Simulate regression predictions
np.random.seed(42)
y_true_reg = np.random.lognormal(10, 0.5, 500) # House prices
y_pred_reg = y_true_reg * np.random.uniform(0.8, 1.2, 500)
rmse = np.sqrt(mean_squared_error(y_true_reg, y_pred_reg))
mae = mean_absolute_error(y_true_reg, y_pred_reg)
r2 = r2_score(y_true_reg, y_pred_reg)
mape = mean_absolute_percentage_error(y_true_reg, y_pred_reg) * 100
ev = explained_variance_score(y_true_reg, y_pred_reg)
max_err = max_error(y_true_reg, y_pred_reg)
print("=== REGRESSION METRICS ===\n")
print(f"MSE: {mean_squared_error(y_true_reg, y_pred_reg):>15,.0f}")
print(f"RMSE: {rmse:>15,.0f} ← same units as target")
print(f"MAE: {mae:>15,.0f} ← robust to outliers")
print(f"MAPE: {mape:>14.2f}% ← percentage error")
print(f"R²: {r2:>15.4f} ← 1.0 = perfect, <0 = worse than mean")
print(f"Expl. Variance: {ev:>8.4f}")
print(f"Max Error: {max_err:>12,.0f}")Learning Curves
Validation Curves (Hyperparameter Effect)
Nested Cross-Validation
Model Persistence
Model Comparison Framework
Summary
In this lesson, you learned:
- ✅ Cross-validation strategies (K-Fold, Stratified, Repeated, LOO)
- ✅ Deep dive into classification metrics (accuracy, precision, recall, F1, AUC, MCC)
- ✅ ROC curves and Precision-Recall curves
- ✅ Regression metrics (RMSE, MAE, MAPE, R², explained variance)
- ✅ Learning curves to diagnose overfitting/underfitting
- ✅ Validation curves to understand hyperparameter effects
- ✅ Nested cross-validation to avoid optimism bias
- ✅ Model persistence with joblib
- ✅ Framework for comparing multiple models
📤 Code Output Examples
Cross-Validation Strategies Output:
K-Fold (5): 0.9614 ± 0.0146 Stratified KFold: 0.9649 ± 0.0129 Repeated StratKFold: 0.9625 ± 0.0152 LOO CV (50 samples): 0.9400 Shuffle Split: 0.9632 ± 0.0112
Classification Metrics Output:
=== CLASSIFICATION METRICS ===
Accuracy: 0.9040
Balanced Accuracy: 0.8375
Precision: 0.7021
Recall: 0.6600
F1 Score: 0.6804
ROC AUC: 0.9118
Avg Precision: 0.7256
MCC: 0.6143
Cohen's Kappa: 0.5987
Log Loss: 0.3412
precision recall f1-score support
0 0.94 0.97 0.95 400
1 0.70 0.66 0.68 100
accuracy 0.90 500
macro avg 0.82 0.81 0.82 500
weighted avg 0.90 0.90 0.90 500
Confusion Matrix Breakdown:
True Positives (TP): 66
True Negatives (TN): 386
False Positives (FP): 14 ← Type I Error
False Negatives (FN): 34 ← Type II Error
Specificity (TNR): 0.9650
Sensitivity (TPR/Recall): 0.6600Regression Metrics Output:
=== REGRESSION METRICS === MSE: 2,345,678,901 RMSE: 48,432 ← same units as target MAE: 23,156 ← robust to outliers MAPE: 10.42% ← percentage error R²: 0.8921 ← 1.0 = perfect, <0 = worse than mean Expl. Variance: 0.8934 Max Error: 187,654
Validation Curve Output:
Optimal max_depth: 10 (CV Accuracy: 0.9614)
Nested Cross-Validation Output:
Non-nested CV score (optimistic): 0.9789 Nested CV score (honest): 0.9719 ± 0.0138 Optimism bias: 0.0070
Model Persistence Output:
Train accuracy: 1.0000 Test accuracy: 0.9649 Model saved to cancer_classifier.joblib Loaded model accuracy: 0.9649 (should match 0.9649) Predictions on 5 samples: malignant (confidence: 0.990) benign (confidence: 0.870) malignant (confidence: 0.960) malignant (confidence: 0.985) benign (confidence: 0.920) Metadata saved to model_metadata.json
Model Comparison Framework Output:
=== MODEL COMPARISON (10-Fold CV) ===
Model Train Acc Val Acc F1 AUC
Logistic Regression 0.9912 0.9719 ± 0.0156 0.9640 0.9953
Decision Tree 0.9716 0.9333 ± 0.0284 0.9190 0.9298
Random Forest 1.0000 0.9614 ± 0.0195 0.9520 0.9932
Gradient Boosting 1.0000 0.9649 ± 0.0173 0.9564 0.9941
SVM (RBF) 0.9930 0.9789 ± 0.0138 0.9726 0.9967
KNN (k=5) 0.9751 0.9561 ± 0.0211 0.9451 0.9891
Naive Bayes 0.9418 0.9386 ± 0.0247 0.9214 0.9912⚠️ Common Mistakes
❌ Mistake 1: Accuracy पर blindly trust करना imbalanced datasets में
# ❌ WRONG — 95% accuracy but model predicts only majority class
# Imbalanced: 95% class-0, 5% class-1
y_pred_bad = np.zeros(1000) # Always predict 0
print(f"Accuracy: {accuracy_score(y_true, y_pred_bad):.2f}") # 0.95 — misleading!
# ✅ RIGHT — F1, AUC, Balanced Accuracy use करो
print(f"Balanced Accuracy: {balanced_accuracy_score(y_true, y_pred_bad):.2f}") # 0.50
print(f"F1 Score: {f1_score(y_true, y_pred_bad):.2f}") # 0.00❌ Mistake 2: Test data पर hyperparameter tuning करना (Data Leakage)
# ❌ WRONG — test set use किया tuning के लिए, ab evaluation biased है
grid_search.fit(X_test, y_test) # NEVER DO THIS!
# ✅ RIGHT — Train/Validation split या Nested CV use करो
grid_search.fit(X_train, y_train) # Only train data for tuning
final_score = grid_search.score(X_test, y_test) # Test only at the end❌ Mistake 3: Single train-test split पर depend करना
एक बार split करके result report करना unreliable है — different random seeds पर scores vary करते हैं। Always use cross-validation for stable estimates! 🔁
❌ Mistake 4: Feature scaling बिना cross-validation pipeline में करना
# ❌ WRONG — पूरा data scale किया फिर split किया = data leakage!
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X) # Test info leaked into training!
scores = cross_val_score(model, X_scaled, y, cv=5)
# ✅ RIGHT — Pipeline use करो, scaling fold के अंदर होगी
from sklearn.pipeline import make_pipeline
pipeline = make_pipeline(StandardScaler(), RandomForestClassifier())
scores = cross_val_score(pipeline, X, y, cv=5)❌ Mistake 5: Overfitting को ignore करना
अगर training accuracy 100% है लेकिन test accuracy 75% है, तो model ने data memorize किया है — generalize नहीं कर रहा। Learning curves plot करो और regularization / pruning try करो! 📉
❌ Mistake 6: Wrong metric choose करना problem के लिए
| Problem Type | ❌ Wrong Choice | ✅ Better Choice |
|---|---|---|
| Medical diagnosis | Accuracy | Recall (miss करना costly है) |
| Spam filtering | Recall only | Precision (false positive annoying) |
| Imbalanced classes | Accuracy | F1, AUC-PR, MCC |
| Regression with outliers | MSE/RMSE | MAE, Huber Loss |
❌ Mistake 7: Model save करते time metadata save न करना
बिना metadata के — 6 months बाद पता नहीं चलेगा कि model किस data पर, किन features से, और कौन से hyperparameters से train हुआ था। Always save version info, metrics, and feature list! 📋
✅ Key Takeaways
- 🔄 Cross-Validation is non-negotiable — Single split unreliable है, Stratified K-Fold (k=5 या 10) standard practice है
- 📊 Metric selection depends on business problem — Medical = Recall, Spam = Precision, Imbalanced = F1/AUC
- 📈 Learning Curves diagnose problems visually — Gap = Overfitting, Both low = Underfitting, Converging = Good fit
- 🧪 Nested CV gives honest performance estimates — Inner loop tunes, outer loop evaluates, no optimism bias
- ⚖️ Precision-Recall curve > ROC curve for imbalanced data — ROC can look good even when model is bad on minority class
- 🔧 Validation curves help find optimal hyperparameters — Sweet spot जहाँ training और validation दोनों अच्छे हों
- 💾 Model persistence = joblib + metadata — सिर्फ model save करना enough नहीं, context (features, metrics, date) भी save करो
- 🏗️ Pipeline ensures no data leakage — Preprocessing steps cross-validation के अंदर होनी चाहिए
- 🔬 MCC is the most balanced metric — Imbalanced datasets पर accuracy, F1 से ज़्यादा reliable single number
- 📋 Compare multiple models systematically — Same CV strategy, same splits, same metrics — then decide best model
❓ FAQ
Q1: Cross-validation में k की value कितनी रखनी चाहिए? 🤔
Answer: Most cases में k=5 या k=10 best practice है। Small datasets (< 100 samples) के लिए Leave-One-Out (LOO) use करो। k=5 faster है, k=10 slightly better estimate देता है but computationally expensive है। Rule of thumb: dataset जितना small, k उतना large रखो। Very large datasets (>100K) में k=3 भी sufficient है।
Q2: ROC AUC और Precision-Recall AUC में कब कौन use करें?
Answer: ROC AUC — balanced datasets या जब True Negatives matter करते हों (e.g., "is this email NOT spam?")। PR AUC — imbalanced datasets जहाँ positive class rare हो (fraud detection, disease diagnosis)। Imbalanced data पर ROC AUC misleadingly high हो सकता है, PR AUC ज़्यादा honest picture देता है।
Q3: Overfitting detect करने का सबसे reliable तरीका क्या है?
Answer: Learning Curves सबसे informative हैं! Train score बहुत high (≈1.0) और validation score significantly lower है तो overfitting है। Additionally: (1) Train vs Test accuracy gap check करो, (2) Cross-validation variance high है तो model unstable है, (3) Model complexity बढ़ाने पर validation score decrease हो तो overfitting start हो रही है।
Q4: GridSearchCV vs RandomizedSearchCV कब use करें?
Answer: GridSearchCV — जब hyperparameter space small हो (< 100 combinations) और computation budget ज्यादा हो। RandomizedSearchCV — Large search spaces (100+ combinations), continuous parameters, या limited time budget में। Research shows कि RandomizedSearch 60 iterations में usually GridSearch जितना अच्छा result देता है but fraction of time में।
Q5: Model save करने के लिए joblib vs pickle — क्या difference है?
Answer: joblib recommended है scikit-learn models के लिए क्योंकि: (1) Large numpy arrays efficiently compress करता है, (2) compress=3 option size reduce करता है, (3) Scikit-learn docs officially recommend करते हैं। pickle general Python objects के लिए ठीक है but large arrays पर slow है। Note: दोनों में security risk है — untrusted sources से model load मत करो!
Q6: Nested Cross-Validation कब ज़रूरी है?
Answer: जब आप hyperparameter tuning + honest performance estimate दोनों चाहते हो। अगर same CV folds पर tune और evaluate करोगे तो "optimism bias" आएगा (score inflated दिखेगा)। Research papers, model comparison studies, और production deployment decisions में Nested CV ज़रूरी है। Simple exploratory analysis में regular CV + holdout test set sufficient है।
Q7: F1 Score कब misleading हो सकता है?
Answer: F1 Score misleading है जब: (1) True Negatives matter करते हों (F1 TN ignore करता है), (2) Classes extremely imbalanced हों (Weighted F1 better), (3) आपको precision और recall को equally weight नहीं करना — तब F-beta score use करो (beta=2 recall favor, beta=0.5 precision favor)। ऐसे cases में MCC (Matthews Correlation Coefficient) ज्यादा reliable है।
Q8: Production में model update/retrain कब करना चाहिए?
Answer: Model retrain करो जब: (1) Data drift — input distribution change हो जाए, (2) Concept drift — relationship between features और target change हो, (3) Performance degradation — monitoring metrics below threshold गिरें, (4) Significant new data available हो। Best practice: scheduled retraining (weekly/monthly) + drift detection alerts setup करो।
🎯 Interview Questions
Q1: Explain the bias-variance tradeoff. How do you detect and handle overfitting vs underfitting?
Answer: Bias-Variance Tradeoff ML models की fundamental challenge है। High Bias (Underfitting): Model बहुत simple है — training और test दोनों पर poor performance। Learning curve पर दोनों scores low converge करते हैं। Fix: more complex model, more features, less regularization। High Variance (Overfitting): Model training data memorize कर लेता है। Training score high, test score significantly lower। Learning curve पर large gap। Fix: more data, regularization, feature selection, pruning, ensemble methods। Sweet spot: model complex enough to capture patterns but not so complex that it memorizes noise।
Q2: Why is accuracy a bad metric for imbalanced datasets? What alternatives would you use?
Answer: Imbalanced dataset (e.g., 99% negative, 1% positive) में एक dummy model जो always "negative" predict करे — 99% accuracy देगा! But positive class पर completely useless है। Better alternatives: (1) Precision — predicted positives में से कितने actually positive, (2) Recall — actual positives में से कितने correctly identified, (3) F1 Score — precision-recall harmonic mean, (4) AUC-PR — imbalanced data पर ROC से बेहतर, (5) MCC — all 4 confusion matrix values consider करता है, -1 to +1 range, (6) Balanced Accuracy — class-weighted accuracy।
Q3: Explain Nested Cross-Validation. When and why is it necessary?
Answer: Nested CV two-level cross-validation structure है: Inner loop (3-5 folds) hyperparameter tuning करता है (GridSearch/RandomSearch), Outer loop (5-10 folds) model's generalization performance estimate करता है। Why necessary: अगर same data पर tune और evaluate करो → optimistic bias (performance overestimated by 1-5%)। When to use: (1) Final model selection और comparison, (2) Research paper submissions, (3) When reporting "expected performance in production"। Cost: Computationally expensive (inner_k × outer_k × param_combinations model fits)।
Q4: What is data leakage? Give examples and how to prevent it.
Answer: Data Leakage = information from test/future data accidentally enters training process, leading to unrealistically good performance that won't generalize। Examples: (1) Feature scaling on entire dataset before split (test distribution leaks into training), (2) Target encoding using all data, (3) Time-series data random split (future info leaks), (4) Duplicate samples across train/test, (5) Features that implicitly contain the target। Prevention: (1) Always use Pipeline — preprocessing inside CV folds, (2) Time-aware splits for temporal data, (3) Check for feature-target correlations suspiciously close to 1.0, (4) Think about "would I have this feature at prediction time?"
Q5: Compare ROC curve and Precision-Recall curve. When to use which?
Answer: ROC Curve (FPR vs TPR): (1) X-axis: False Positive Rate (FP/(FP+TN)), (2) Shows performance across ALL thresholds, (3) AUC = 0.5 random, AUC = 1.0 perfect, (4) Best for: balanced datasets, when TNs matter। PR Curve (Recall vs Precision): (1) Focuses only on positive class performance, (2) Baseline = class proportion (not 0.5), (3) Best for: imbalanced datasets, rare event detection। Key insight: ROC can be deceptively high on imbalanced data because TN count inflates FPR denominator — model looks good even with many FPs relative to TPs।
Q6: How do you choose between different evaluation metrics for a specific problem?
Answer: Metric selection depends on business cost of errors: (1) Medical screening → High Recall (missing cancer = death, false alarm = just more tests), (2) Spam filter → High Precision (false positive = user misses important email), (3) Fraud detection → F1 or AUC-PR (imbalanced + both errors costly), (4) Recommendation system → Precision@K, NDCG, (5) Regression (house prices) → MAE (interpretable) or MAPE (percentage), (6) Regression with outliers → MAE over RMSE (less sensitive to extreme errors)। Always ask: "Which error is more expensive in production?"
Q7: What is cross-validation? Compare K-Fold, Stratified K-Fold, and Leave-One-Out.
Answer: Cross-validation model performance estimate करने की technique है जो all data use करती है for both training and validation। K-Fold: Data को k equal parts में split — k-1 train, 1 test, rotate k times। Simple, works well for large balanced datasets। Stratified K-Fold: Same as K-Fold but each fold में class proportions preserved। Essential for classification, especially imbalanced। Leave-One-Out (LOO): k = n (each sample is test once)। Pros: maximum training data, no randomness। Cons: computationally expensive (n model fits), high variance estimate। Recommendation: Default = Stratified K-Fold (k=5 or 10), Small data = LOO, Noisy estimate = Repeated Stratified K-Fold।
Q8: How would you handle model selection in a production ML pipeline?
Answer: Production model selection systematic approach: (1) Define success metrics aligned with business KPIs, (2) Baseline model establish करो (simple model or rule-based), (3) Multiple candidates train करो with same preprocessing pipeline, (4) Nested CV for honest comparison, (5) Statistical significance test (paired t-test, McNemar's) to verify differences aren't random, (6) Beyond accuracy — consider inference time, model size, interpretability, maintainability, (7) Champion-challenger framework — new model must beat current production model on holdout set, (8) A/B testing in production for final validation, (9) Monitor continuously — set up drift detection and performance alerts।
Q9: Explain the difference between model evaluation and model validation.
Answer: Model Evaluation: Quantitative measurement of model performance using metrics (accuracy, F1, RMSE etc.) on held-out data. Answers "How well does the model perform?" Model Validation: Broader process ensuring model is suitable for its intended purpose. Includes: (1) Evaluation metrics, (2) Stability testing (does performance vary across different data slices?), (3) Fairness checks (bias across demographic groups), (4) Robustness testing (performance under noisy/adversarial inputs), (5) Business validation (does ML metric improvement translate to business value?), (6) A/B testing in production। Evaluation is a subset of validation।
Q10: You trained a model with 99% accuracy on the test set. What questions would you ask before deploying it?
Answer: High accuracy = red flag unless verified! Questions: (1) Is data imbalanced? — Check confusion matrix, F1, balanced accuracy, (2) Is there data leakage? — Features that encode the target? Future info in training?, (3) How was test set created? — Random split from same distribution? Truly unseen?, (4) What's the baseline? — If majority class = 99%, model learned nothing, (5) Is it stable? — Cross-validation variance? Different random seeds?, (6) Does it work on edge cases? — Test on adversarial/rare examples, (7) Train-test distribution match? — Same time period? Same data source?, (8) What's the production data distribution? — Will real data look like training data?, (9) What's the cost of wrong predictions? — Even 1% error could be catastrophic in some domains।
*Next Lesson: ML Project →*
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Model Evaluation and Selection.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Python Master Course topic.
Search Terms
python-master-course, python master course, python, master, course, machine, learning, model
Related Python Master Course Topics