Essential Machine Learning Interview Questions

Fundamental Concepts

Q1: What is the difference between supervised, unsupervised, and reinforcement learning?

A:

• Supervised Learning: Learn from labeled data (input-output pairs)
• Examples: classification, regression
• Used when: have labeled training data
• Common: Linear Regression, SVM, Random Forest

• Unsupervised Learning: Learn patterns from unlabeled data
• Examples: clustering, dimensionality reduction
• Used when: don't have labels or want to discover structure
• Common: K-Means, PCA, Hierarchical Clustering

• Reinforcement Learning: Agent learns by interacting with environment
• Examples: game playing, robotics
• Used when: can define reward signal
• Common: Q-Learning, Policy Gradient, DQN

Q2: Explain overfitting and underfitting

A: Overfitting occurs when model learns noise in training data rather than generalizable patterns. High training accuracy but low test accuracy. Caused by model complexity, small dataset, or insufficient regularization.

Solutions:

• Increase training data
• Regularization (L1, L2)
• Early stopping
• Cross-validation
• Reduce model complexity

Underfitting occurs when model is too simple to capture underlying patterns. Low accuracy on both training and test.

Solutions:

• Increase model complexity
• More features
• Reduce regularization
• More training iterations

Q3: What's the bias-variance tradeoff?

A: Total error = Bias² + Variance + Irreducible Error

• Bias: Model's tendency to learn wrong assumptions (systematic error)
• Variance: Model's sensitivity to small fluctuations in training data

Simple models → high bias, low variance (underfitting) Complex models → low bias, high variance (overfitting)

Goal: minimize total error, which requires balancing bias and variance.

Q4: When would you use L1 vs L2 regularization?

A:

• L1 (Lasso): Loss = MSE + λ * Σ|weights|
• Forces some weights exactly to zero
• Useful for feature selection
• Creates sparse models
• Better when you suspect many irrelevant features

• L2 (Ridge): Loss = MSE + λ * Σ(weights²)
• Shrinks all weights proportionally
• Keeps all features but with smaller weights
• More stable when features are correlated
• Better for general regularization

• Elastic Net: Combines both L1 and L2
• Balances feature selection with stability

Model Selection & Evaluation

Q5: How do you choose between different models?

A: Systematic approach:

1. Baseline First

• Start with simple model (linear regression, logistic regression)
• Measure performance as benchmark

1. Problem Type

• Classification problem? → Logistic Regression, SVM, Tree-based
• Regression problem? → Linear Regression, Gradient Boosting
• Clustering problem? → K-Means, DBSCAN, Hierarchical

1. Data Characteristics

• Large dataset? → Neural Networks, Gradient Boosting
• Small dataset? → Simple models with regularization
• High-dimensional? → Linear models, SVM
• Low-dimensional? → Can use complex models

1. Performance Requirements

• Speed critical? → Decision Trees, SVM
• Accuracy critical? → Ensemble methods, Neural Networks
• Interpretability critical? → Linear models, Decision Trees

1. Validation

• Use cross-validation on multiple models
• Compare on same metrics
• Select model with best CV performance

Q6: How do you evaluate a classification model?

A: Multiple metrics for different purposes:

• Accuracy: (TP+TN)/(total) - overall correctness, but misleading for imbalanced data
• Precision: TP/(TP+FP) - when false positives are costly
• Recall: TP/(TP+FN) - when false negatives are costly
• F1-Score: 2*(Precision*Recall)/(Precision+Recall) - balance precision and recall
• AUC-ROC: area under receiver operating characteristic curve - threshold-independent
• Confusion Matrix: detailed breakdown of predictions

Choose metrics based on problem:

• Medical diagnosis (false negatives bad) → prioritize recall
• Spam detection (false positives bad) → prioritize precision
• Balanced performance needed → F1-score
• Ranking problems → AUC-ROC

Q7: What's the difference between cross-validation and train-test split?

A:

• Train-Test Split: Single 70-30 or 80-20 split
• Pros: simple, fast
• Cons: high variance, depends on specific split

• Cross-Validation (K-Fold): Data split into k folds, train on k-1, test on 1
• Pros: uses all data, lower variance, more reliable
• Cons: slower (trains k times)
• Better for small datasets

• Stratified: Maintain class distribution in each fold (for classification)

• Time Series: Walk-forward validation (respect temporal ordering)

Best practice: use cross-validation for model selection, final evaluation on held-out test set.

Data Handling

Q8: How do you handle missing values?

A: Depends on missing data percentage and pattern:

1. <5% missing

• Delete rows (minimal data loss)

1. 5-20% missing

• Mean/median imputation (numerical)
• Mode imputation (categorical)
• KNN imputation (uses similar samples)
• Iterative imputation (MICE algorithm)

1. >20% missing

• Investigate root cause
• Consider if feature is worth keeping
• ML imputation methods
• Create "missing" indicator feature

1. Time series

• Forward fill: use previous value
• Backward fill: use next value
• Interpolation: fit curve through data

Q9: How do you handle categorical variables?

A:

• One-Hot Encoding
• Create binary column for each category
• Use for: low cardinality (<10 categories), tree-based models
• Pros: works with most algorithms
• Cons: curse of dimensionality with many categories

• Label Encoding
• Assign integer to each category (0, 1, 2, ...)
• Use for: tree-based models, ordinal variables
• Pros: space efficient
• Cons: implies ordering, misleading for linear models

• Target Encoding
• Replace category with mean target value
• Use for: high cardinality (100+ categories)
• Pros: information-rich, handles high cardinality
• Cons: risk of overfitting, leakage

• Hashing
• Hash category to fixed number of bins
• Use for: very high cardinality

Q10: How do you handle outliers?

A: Depends on data and domain:

1. Detection Methods

• IQR: Remove if x < Q1 - 1.5*IQR or x > Q3 + 1.5*IQR
• Z-score: Remove if |z| > 3 (99.7% confidence)
• Visualization: plots, histograms

1. Handling Strategies

• Remove: If truly erroneous data (< 1% of data)
• Keep: If legitimate extreme values (business metric peaks)
• Cap/Floor: Clip to percentile (95th, 99th)
• Separate model: Train different model for outliers
• Robust methods: Use algorithms less sensitive to outliers (median, quantile)

Feature Engineering

Q11: What is feature engineering and why is it important?

A: Feature engineering is the process of creating new features from raw data to improve model performance. Often the most impactful part of ML projects.

Why important:

• Raw data rarely optimal for algorithms
• Good features → simpler models
• Good features → better generalization
• Good features → reduced training time

Common techniques:

1. Mathematical transformations: log, sqrt, polynomials
2. Interactions: x1 * x2, x1 / x2
3. Binning: convert continuous to categorical
4. Encoding: categorical to numerical
5. Scaling: normalization, standardization
6. Domain knowledge: create meaningful features

Q12: How do you select features?

A: Multiple approaches:

1. Statistical Methods

• Correlation analysis: remove highly correlated features
• Chi-square test: for categorical features
• ANOVA: for numerical features vs categorical target

1. Model-Based

• Feature importance from tree-based models
• Permutation importance: shuffle feature, measure performance drop
• SHAP values: game theory approach to feature importance

1. Automated

• Recursive Feature Elimination (RFE)
• Lasso (L1 forces irrelevant features to zero)
• Backward selection: remove worst feature iteratively
• Forward selection: add best feature iteratively

1. Domain Knowledge

• Subject matter experts
• Business understanding
• Prior experience

Algorithms & Techniques

Q13: Explain how Random Forest works

A: Random Forest is ensemble method that builds multiple decision trees and combines predictions.

Process:

1. Bootstrap sampling: randomly sample data with replacement
2. Build trees: for each sample, build full decision tree

• At each split, randomly select subset of features
• This introduces randomness and prevents overfitting

1. Predictions:

• Classification: majority voting
• Regression: average predictions

Advantages:

• Handles non-linear relationships
• Feature importance built-in
• Robust to outliers
• Less prone to overfitting than single tree

Disadvantages:

• Less interpretable than single tree
• Slower inference than linear models
• Memory intensive with many trees

Q14: What is the difference between bagging and boosting?

A:

• Bagging (Bootstrap Aggregating):
• Train models independently on random subsets
• Combine by averaging (regression) or voting (classification)
• Example: Random Forest
• Reduces variance
• Parallel training possible

• Boosting:
• Train models sequentially
• Each new model focuses on samples previous models got wrong
• Combine by weighted average
• Examples: AdaBoost, Gradient Boosting, XGBoost
• Reduces bias and variance
• Sequential, can't parallelize

Q15: How does gradient boosting work?

A:

1. Start with weak learner (shallow tree)
2. Calculate residuals (actual - predicted)
3. Train new tree on residuals
4. Add to ensemble: predictions += learning_rate * new_tree_predictions
5. Repeat steps 2-4

Key parameters:

• n_estimators: number of trees (more = better, slower)
• learning_rate: shrinkage parameter (smaller = slower learning)
• max_depth: tree depth (shallower = less overfitting)
• min_samples_leaf: minimum samples in leaf node

Advantages:

• Often best predictive performance
• Handles mixed feature types
• Feature importance available
• Less sensitive to outliers

Production & Deployment

Q16: How do you monitor a deployed model?

A: Monitoring is crucial for production models:

1. Performance Monitoring

• Track prediction metrics on new data
• Set up alerts if performance drops >X%
• Compare against baseline

1. Data Drift Detection

• Monitor feature distributions
• Alert if input distribution changes
• Use statistical tests (Kolmogorov-Smirnov)

1. Model Drift

• Monitor prediction distribution
• Alert if output distribution changes
• May indicate concept drift in data

1. Operational Monitoring

• Latency: prediction time
• Throughput: predictions per second
• Error rate: crashes, exceptions
• Resource usage: CPU, memory

1. Trigger Retraining When

• Performance drops significantly
• Data drift detected
• New data collected (planned retraining)
• Business requirements change

Q17: How would you A/B test a new ML model?

A: Controlled experiment to compare models:

1. Setup

• Split users randomly
• Control group: existing model
• Treatment group: new model
• Ensure stratification (maintain distribution)

1. Duration

• Run long enough for statistical significance
• Minimum 1-2 weeks typically
• Depends on traffic volume

1. Metrics

• Primary metric: most important business metric
• Secondary metrics: other performance aspects
• Statistical significance: p-value < 0.05

1. Implementation

``python # Random assignment user_id_hash = hash(user_id) if user_id_hash % 100 < 50: use_control_model() else: use_treatment_model() ``

1. Analysis

• Calculate metric for each group
• Compute confidence intervals
• Test statistical significance
• Calculate business impact

Common Pitfalls

Q18: What are common ML mistakes?

A:

1. Data Leakage: Using information not available at prediction time

• Solution: careful feature engineering, temporal validation

1. Ignoring Class Imbalance: Using accuracy on imbalanced data

• Solution: use appropriate metrics (F1, AUC), stratified sampling

1. Not Validating Properly: Tuning hyperparameters on test set

• Solution: use validation set for tuning only

1. Poor Feature Engineering: Using raw features

• Solution: invest time in feature creation

1. Not Starting Simple: Immediately using complex models

• Solution: start with baselines

1. Ignoring Domain Knowledge: Treating ML as pure mathematics

• Solution: collaborate with domain experts

1. Over-engineering: Complex solutions when simple works

• Solution: prefer simplicity

Summary

Master these concepts to excel in ML interviews. Focus on:

1. Understanding tradeoffs (bias-variance, precision-recall)
2. Knowing when to use each algorithm
3. Proper validation and evaluation
4. Data handling and preprocessing
5. Production considerations

Practice explaining concepts clearly and be ready to dive deep into any area.