Python Notes
Learn classification algorithms in Python — Logistic Regression, Decision Trees, Random Forest, SVM, KNN, Naive Bayes, and Gradient Boosting, with metrics, confusion matrices, ROC curves, and class imbalance handling.
Classification is a supervised learning task where we predict which category an input belongs to. This lesson covers major classification algorithms with practical examples.
Setup
pip install scikit-learn pandas numpy matplotlib seaborn xgboost imbalanced-learnimport numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (
accuracy_score, classification_report, confusion_matrix,
roc_auc_score, roc_curve, precision_recall_curve
)Dataset: Credit Card Fraud Detection
Evaluation Helper
Logistic Regression
Decision Tree Classifier
Random Forest Classifier
Support Vector Machine (SVM)
K-Nearest Neighbors (KNN)
Naive Bayes
Gradient Boosting and XGBoost
Handling Class Imbalance
Comparing All Models
Summary
In this lesson, you learned:
- ✅ Logistic Regression with L1/L2 regularization
- ✅ Decision Tree Classifier (visualization and feature importance)
- ✅ Random Forest Classifier (ensemble of trees)
- ✅ Support Vector Machine with kernel trick
- ✅ K-Nearest Neighbors and optimal k selection
- ✅ Naive Bayes for probabilistic classification
- ✅ Gradient Boosting and XGBoost
- ✅ Handling class imbalance (class_weight, SMOTE)
- ✅ Model comparison with accuracy, AUC, and F1
📤 Expected Outputs
Dataset Output:
Shape: (10000, 20) Class distribution: 0 9700 1 300 Name: fraud, dtype: int64 Fraud rate: 3.0% Training: (8000, 20), Testing: (2000, 20)
Logistic Regression Output:
==================================================
Model: Logistic Regression
==================================================
precision recall f1-score support
Legit 0.99 0.93 0.96 1940
Fraud 0.22 0.82 0.35 60
accuracy 0.93 2000
macro avg 0.61 0.87 0.65 2000
weighted avg 0.97 0.93 0.94 2000
ROC AUC: 0.9512
L2 (C=0.01): Accuracy=0.9340, AUC=0.9478
L2 (C=1.0): Accuracy=0.9295, AUC=0.9512
L2 (C=100): Accuracy=0.9290, AUC=0.9508
L1 (C=1.0): Accuracy=0.9310, AUC=0.9505Decision Tree Output:
==================================================
Model: Decision Tree
==================================================
precision recall f1-score support
Legit 0.99 0.97 0.98 1940
Fraud 0.34 0.67 0.45 60
accuracy 0.96 2000
macro avg 0.66 0.82 0.72 2000
weighted avg 0.97 0.96 0.96 2000
ROC AUC: 0.8872Random Forest Output:
==================================================
Model: Random Forest
==================================================
precision recall f1-score support
Legit 0.99 0.95 0.97 1940
Fraud 0.28 0.80 0.41 60
accuracy 0.95 2000
macro avg 0.63 0.87 0.69 2000
weighted avg 0.97 0.95 0.95 2000
ROC AUC: 0.9601
Top 10 important features:
feature_0 0.112
feature_3 0.098
feature_7 0.091
...SVM Output:
==================================================
Model: SVM (RBF)
==================================================
precision recall f1-score support
Legit 0.99 0.94 0.96 1940
Fraud 0.24 0.78 0.37 60
accuracy 0.94 2000
macro avg 0.62 0.86 0.67 2000
weighted avg 0.97 0.94 0.95 2000
ROC AUC: 0.9534
SVM linear: Accuracy=0.9325, AUC=0.9501
SVM rbf: Accuracy=0.9380, AUC=0.9534
SVM poly: Accuracy=0.9295, AUC=0.9412KNN Output:
==================================================
Model: KNN (k=5)
==================================================
precision recall f1-score support
Legit 0.98 0.98 0.98 1940
Fraud 0.28 0.30 0.29 60
accuracy 0.96 2000
macro avg 0.63 0.64 0.64 2000
weighted avg 0.96 0.96 0.96 2000
ROC AUC: 0.8245
Best k: 11 (AUC = 0.8612)Naive Bayes Output:
==================================================
Model: Gaussian Naive Bayes
==================================================
precision recall f1-score support
Legit 0.99 0.90 0.94 1940
Fraud 0.17 0.73 0.27 60
accuracy 0.89 2000
macro avg 0.58 0.82 0.61 2000
weighted avg 0.96 0.89 0.92 2000
ROC AUC: 0.9156
GNB tuned AUC: 0.9158XGBoost Output:
Gradient Boosting AUC: 0.9645 [0] validation_0-auc:0.92341 [50] validation_0-auc:0.96123 [100] validation_0-auc:0.97012 [150] validation_0-auc:0.97234 [199] validation_0-auc:0.97356 ================================================== Model: XGBoost ================================================== ROC AUC: 0.9736
Class Imbalance Output:
Class imbalance: {0: 7760, 1: 240}
After SMOTE: {0: 7760, 1: 7760}
SMOTE + RF AUC: 0.9589
SMOTE Pipeline AUC: 0.9501Model Comparison Output:
=== CLASSIFIER COMPARISON ===
Model Accuracy AUC F1
XGBoost 0.9685 0.9736 0.52
Random Forest 0.9500 0.9601 0.41
Gradient Boosting 0.9620 0.9645 0.48
SVM 0.9380 0.9534 0.37
Logistic Regression 0.9295 0.9512 0.35
Naive Bayes 0.8930 0.9156 0.27
KNN 0.9580 0.8245 0.29
Decision Tree 0.9605 0.8872 0.45⚠️ Common Mistakes
- Feature Scaling भूल जाना 🔥 — SVM, KNN, और Logistic Regression जैसे algorithms को scaling चाहिए। बिना
StandardScaler()के results बहुत गलत आ सकते हैं!
- Imbalanced data पर Accuracy देखना — अगर 97% data non-fraud है, तो model सब "non-fraud" predict करे तो भी 97% accuracy मिलेगी। हमेशा AUC, F1-score, और Precision-Recall देखो।
- SMOTE को test data पर apply करना ❌ — SMOTE सिर्फ training data पर use करो! Test data को original distribution में ही रहने दो, वरना evaluation misleading होगी।
- Hyperparameters tune नहीं करना — Default parameters से काम चला लेना beginners की सबसे बड़ी mistake है।
GridSearchCVयाRandomizedSearchCVहमेशा try करो।
- Decision Tree को बिना pruning के train करना 🌲 — बिना
max_depthयाmin_samples_leafset किए tree overfit कर जाएगा। Training accuracy 100% दिखेगी, testing पर disaster!
- Cross-validation skip करना — सिर्फ एक train-test split पर rely मत करो।
cross_val_score()use करो ताकि model की real performance पता चले।
- Data leakage allow करना 💀 — Scaling को pipeline के बाहर करना (test data की info training में leak हो जाती है)। हमेशा
make_pipeline()use करो!
✅ Key Takeaways
- 🎯 Classification supervised learning है जहाँ हम categories predict करते हैं — binary (Yes/No) या multiclass (Cat/Dog/Bird)।
- 📊 Logistic Regression simple और interpretable है — linear decision boundary बनाता है, L1/L2 regularization से overfitting control करता है।
- 🌲 Decision Trees easy to visualize हैं लेकिन easily overfit करते हैं —
max_depthऔरmin_samples_leafसे control करो।
- 🌳 Random Forest ensemble method है — multiple decision trees का average लेता है, overfitting reduce करता है, और feature importance देता है।
- ⚡ XGBoost/Gradient Boosting typically सबसे अच्छा perform करता है tabular data पर — sequential trees बनाता है जो previous errors correct करते हैं।
- 🧲 SVM high-dimensional data में excellent है — kernel trick से non-linear boundaries handle करता है, लेकिन large datasets पर slow है।
- 📏 KNN simple "lazy learner" है — कोई training नहीं, बस test time पर nearest neighbors देखता है। Scaling essential है!
- ⚖️ Class Imbalance real-world datasets में common है —
class_weight='balanced', SMOTE, या threshold tuning से handle करो।
- 📈 ROC-AUC imbalanced datasets के लिए best metric है — accuracy misleading हो सकती है, AUC model की discrimination power measure करता है।
- 🔧 Pipeline बनाओ हमेशा — preprocessing + model एक pipeline में रखो ताकि data leakage न हो और code clean रहे।
❓ FAQ
Q1: Logistic Regression और Linear Regression में क्या difference है? > Linear Regression continuous values predict करता है (जैसे price), जबकि Logistic Regression probability predict करता है (0 to 1 range) sigmoid function use करके। Output categories (0/1) में classify होता है।
Q2: Random Forest vs XGBoost — कौन सा better है? > XGBoost typically better perform करता है क्योंकि यह sequentially trees बनाता है (errors correct करता है), जबकि Random Forest parallel independent trees बनाता है। लेकिन XGBoost ज़्यादा sensitive है hyperparameters के लिए और overfit भी जल्दी करता है।
Q3: KNN में optimal k कैसे choose करें? > Cross-validation use करो different k values try करके। Generally odd number choose करो (ties avoid करने के लिए)। Small k = more noise sensitive, large k = smoother boundaries but may miss patterns। cross_val_score() से best k find करो।
Q4: SMOTE कब use करना चाहिए और कब नहीं? > SMOTE tab use करो जब minority class बहुत कम हो (< 10-15%)। लेकिन अगर data already somewhat balanced है (40-60% split), तो SMOTE unnecessary है। ध्यान रहे: SMOTE सिर्फ training data पर apply करो, test data पर कभी नहीं!
Q5: SVM इतना slow क्यों होता है large datasets पर? > SVM की time complexity O(n² to n³) है training में क्योंकि kernel matrix compute करनी पड़ती है। 10,000+ samples पर बहुत slow हो सकता है। Alternative: LinearSVC (linear kernel, faster) या SGDClassifier use करो large data के लिए।
Q6: Feature Scaling किन algorithms के लिए ज़रूरी है? > ज़रूरी: KNN (distance-based), SVM (margin-based), Logistic Regression (gradient descent), Neural Networks। Not needed: Decision Trees, Random Forest, XGBoost, Naive Bayes — ये split/probability-based हैं, scale independent हैं।
Q7: Precision vs Recall — कब किसे priority दें? > Precision priority जब False Positive costly हो (spam detection — legitimate email miss नहीं करना)। Recall priority जब False Negative dangerous हो (cancer detection — कोई patient miss नहीं होना चाहिए)। F1-score दोनों का balance है।
Q8: class_weight='balanced' internally क्या करता है? > यह minority class को higher weight देता है — formula: n_samples / (n_classes * n_samples_per_class)। इससे model minority class की mistakes को ज़्यादा penalize करता है, effectively SMOTE जैसा effect देता है बिना data duplicate किए।
🎯 Interview Questions
Q1: Explain the bias-variance tradeoff in the context of classification algorithms. > Answer: Bias-variance tradeoff classification में fundamental concept है। High bias (underfitting) — model too simple है, training और test दोनों पर poor performance (e.g., Linear model on non-linear data)। High variance (overfitting) — model training data को "memorize" कर लेता है, training पर excellent लेकिन test पर poor (e.g., deep Decision Tree without pruning)। Decision Tree = high variance, Logistic Regression = high bias। Random Forest variance reduce करता है (averaging), Boosting bias reduce करता है (sequential correction)।
Q2: How does the kernel trick work in SVM? > Answer: Kernel trick data को higher dimension में map करता है बिना actually compute किए। Example: 2D data जो linearly separable नहीं है, RBF kernel उसे infinite-dimensional space में map करता है जहाँ linear separation possible हो जाती है। Internally, SVM सिर्फ dot products compute करता है — kernel function K(x,y) directly higher-dimensional dot product return करती है। Common kernels: Linear (x·y), RBF (exp(-γ|x-y|²)), Polynomial ((x·y + c)^d)।
Q3: What is the ROC curve, and why is AUC a better metric than accuracy for imbalanced datasets? > Answer: ROC curve True Positive Rate (Recall) vs False Positive Rate plot करती है different thresholds पर। AUC (Area Under Curve) 0 to 1 range — 0.5 = random, 1.0 = perfect। Imbalanced data में accuracy misleading है — 99% legitimate transactions वाले dataset में सब "legitimate" predict करने पर 99% accuracy मिलती है! AUC threshold-independent है और model की discrimination ability measure करता है — कितना अच्छा separate कर रहा है positive vs negative classes।
Q4: Compare bagging (Random Forest) vs boosting (XGBoost). When would you use each? > Answer: Bagging (Random Forest): Parallel independent trees → average predictions → reduces variance. Robust to noise, less likely to overfit, fast training (parallelizable). Boosting (XGBoost): Sequential dependent trees → each corrects previous errors → reduces bias. Can overfit if not tuned, slower training (sequential).
Q5: How do you handle multiclass classification with algorithms that only support binary? > Answer: Two strategies: One-vs-Rest (OvR): N binary classifiers, each class vs all others। Class with highest probability wins। One-vs-One (OvO): N*(N-1)/2 classifiers, each pair of classes। Voting decides winner। Sklearn automatically handles this — LogisticRegression(multi_class='ovr') या SVC by default OvO use करता है। OvR simpler और faster है, OvO better for small datasets।
Q6: Explain the Naive Bayes assumption. When does it work well despite being "naive"? > Answer: Naive Bayes assumes all features are conditionally independent given the class label — यानी P(x1,x2|class) = P(x1|class) * P(x2|class)। Real world में यह assumption rarely true है, but still works well for: Text classification (bag-of-words, words somewhat independent), Spam detection, High-dimensional data (features ≫ samples)। Reason: classification only needs correct decision boundary, not accurate probabilities। Even with wrong probability estimates, the argmax (predicted class) can still be correct।
Q7: What is the curse of dimensionality, and how does it affect KNN? > Answer: High dimensions में distances lose meaning — सब points nearly equidistant हो जाते हैं। KNN purely distance-based है, so most affected। 20 features में "nearest neighbor" meaningful नहीं रहता। Solutions: Dimensionality reduction (PCA), Feature selection, Use Manhattan distance (L1, better in high-D), या switch to tree-based models (Random Forest) जो dimension-independent splits करते हैं।
Q8: How would you choose the right classification algorithm for a given problem? > Answer: Decision framework: (1) Data size: Small → SVM/KNN, Large → Random Forest/XGBoost/Logistic Regression। (2) Interpretability needed: Yes → Logistic Regression/Decision Tree, No → Ensemble/SVM। (3) Feature types: Text → Naive Bayes/Logistic Regression, Tabular → XGBoost/Random Forest। (4) Speed requirement: Fast inference → Logistic Regression, Fast training → Random Forest। (5) Imbalanced data: XGBoost (scale_pos_weight), Random Forest (class_weight)। Generally start with Logistic Regression (baseline) → Random Forest → XGBoost (best performance)।
Q9: What is threshold tuning in classification, and when should you do it? > Answer: Default threshold 0.5 है (probability > 0.5 → positive class), but optimal threshold problem-dependent है। Fraud detection में threshold lower करो (0.3) — ज़्यादा fraud catch करो, कुछ false positives acceptable। Medical diagnosis में threshold adjust करो based on cost of false negatives vs false positives। Method: Precision-Recall curve plot करो, F1-score maximize करने वाला threshold choose करो, या business cost function define करके optimal point find करो।
Q10: Explain cross-validation and why a single train-test split is not enough. > Answer: Single split result random split पर depend करता है — different split = different result। K-Fold CV: Data K parts में divide, K बार train (K-1 folds) + test (1 fold), average score = reliable estimate। Benefits: (1) Reduced variance in performance estimate, (2) Every data point gets tested, (3) Better for small datasets। Stratified K-Fold classification में ज़रूरी — ensures each fold has same class distribution। Typically K=5 या K=10 use करो। Rule: Model selection = CV score, final evaluation = held-out test set।
*Next Lesson: Clustering →*
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Classification Algorithms.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Python Master Course topic.
Search Terms
python-master-course, python master course, python, master, course, machine, learning, classification
Related Python Master Course Topics