ML Notes
Complete overview of model evaluation strategies in machine learning including metrics for classification and regression, validation techniques, and comparison methods.
Building a model is only half the battle — evaluating it properly is equally important. Poor evaluation leads to deploying models that fail in production. This guide covers the essential evaluation strategies every ML practitioner needs.
Why Evaluation Matters
Without proper evaluation
┌────────────────────────────────────────────────┐
│ Train model → "99% accuracy!" → Deploy │
│ → Model fails in production → Users unhappy │
│ → Turns out training data was leaking labels │
└────────────────────────────────────────────────┘
With proper evaluation
┌────────────────────────────────────────────────┐
│ Train model → Cross-validate → Check metrics │
│ → Test on holdout → Validate assumptions │
│ → Monitor after deployment → Iterate │
└────────────────────────────────────────────────┘
Classification Metrics
from sklearn.metrics import (accuracy_score, precision_score, recall_score,
f1_score, classification_report, confusion_matrix)
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
data.data, data.target, test_size=0.2, random_state=42
)
model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print("=== Classification Metrics ===")
print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(f"Precision: {precision_score(y_test, y_pred):.4f}")
print(f"Recall: {recall_score(y_test, y_pred):.4f}")
print(f"F1 Score: {f1_score(y_test, y_pred):.4f}")
print("\nDetailed Report:")
print(classification_report(y_test, y_pred, target_names=data.target_names))Regression Metrics
from sklearn.metrics import (mean_squared_error, mean_absolute_error,
r2_score, mean_absolute_percentage_error)
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import GradientBoostingRegressor
import numpy as np
housing = fetch_california_housing()
X_train, X_test, y_train, y_test = train_test_split(
housing.data, housing.target, test_size=0.2, random_state=42
)
model = GradientBoostingRegressor(random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print("=== Regression Metrics ===")
print(f"MSE: {mean_squared_error(y_test, y_pred):.4f}")
print(f"RMSE: {np.sqrt(mean_squared_error(y_test, y_pred)):.4f}")
print(f"MAE: {mean_absolute_error(y_test, y_pred):.4f}")
print(f"R²: {r2_score(y_test, y_pred):.4f}")
print(f"MAPE: {mean_absolute_percentage_error(y_test, y_pred):.2%}")Metrics Comparison Table
| Metric | Type | Range | When to Use |
|---|---|---|---|
| Accuracy | Classification | [0, 1] | Balanced classes |
| Precision | Classification | [0, 1] | Cost of false positives is high |
| Recall | Classification | [0, 1] | Cost of false negatives is high |
| F1 Score | Classification | [0, 1] | Imbalanced classes |
| AUC-ROC | Classification | [0, 1] | Ranking quality |
| MSE | Regression | [0, ∞) | Penalize large errors |
| MAE | Regression | [0, ∞) | Robust to outliers |
| R² | Regression | (-∞, 1] | Proportion of variance explained |
Validation Strategies
from sklearn.model_selection import (cross_val_score, StratifiedKFold,
LeaveOneOut, TimeSeriesSplit)
# 1. K-Fold Cross Validation
scores = cross_val_score(model, X_train, y_train, cv=5, scoring='r2')
print(f"5-Fold CV: {scores.mean():.4f} (+/- {scores.std():.4f})")
# 2. Stratified K-Fold (maintains class proportions)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# 3. Time Series Split (for temporal data)
tss = TimeSeriesSplit(n_splits=5)Common Evaluation Pitfalls
- Data leakage: Information from test set leaks into training
- Using accuracy on imbalanced data: 99% accuracy with 99% majority class
- Not using proper cross-validation: Single train/test split is unreliable
- Ignoring business context: Best metric depends on the problem
- Overfitting to validation set: Too much hyperparameter tuning
Interview Questions
- When would you use F1 score instead of accuracy?
When classes are imbalanced. Accuracy can be misleading — a model that always predicts the majority class gets high accuracy but is useless.
- What is data leakage and how do you prevent it?
Data leakage occurs when information from the test set influences training. Prevent it by splitting data before any preprocessing, using pipelines, and being careful with time-series data.
- How do you choose between MSE and MAE for regression?
MSE penalizes large errors more heavily (squared term). Use MAE when outliers are expected and you want robustness. Use MSE when large errors are particularly costly.
- What is the advantage of cross-validation over a single train/test split?
CV provides a more reliable performance estimate by testing on multiple folds, reduces variance in the estimate, and uses all data for both training and validation.
- How do you evaluate a model after deployment?
Monitor prediction quality with live labels (when available), track prediction distribution for drift, set up A/B tests, and use shadow mode to compare new models against production.
Deep Dive: Core Concepts Explained
To truly master model evaluation overview, you need to understand not just the how but the why behind each step. The fundamental principle is that every technique in machine learning represents a specific assumption about the data. When that assumption holds in practice, the technique works well; when it is violated, performance degrades. This is why understanding the mathematical foundation matters — it tells you exactly when and why a method will succeed or fail.
Let us think about this from first principles. Every machine learning algorithm is essentially an optimization problem: find the parameters that minimize some measure of error on training data while generalizing to unseen data. The specific form of the error measure, the constraints on parameters, and the optimization procedure differ between algorithms, but this fundamental structure is universal. Once you internalize this perspective, learning new algorithms becomes much faster because you immediately ask: what is being optimized? What assumptions are being made? What are the failure modes?
Practitioners who understand these foundations can diagnose problems that others find mysterious. When a model underperforms, they can identify whether the issue is insufficient data, inappropriate model assumptions, poor optimization (not converging), or overfitting. Each diagnosis leads to a specific remedy, turning model development from trial-and-error into systematic engineering.
Practical Implementation Guide
When implementing model evaluation overview in real projects, follow this systematic approach. Start by establishing a simple baseline — often a trivial model like predicting the mean or most frequent class. This baseline tells you the minimum performance your sophisticated approach must beat to justify its complexity. Next, implement the standard version of the algorithm with default parameters. Evaluate it rigorously using cross-validation and appropriate metrics for your problem type.
Only after establishing this solid foundation should you begin optimization. Tune one hyperparameter at a time while holding others fixed, observing how each affects performance. Use grid search or randomized search for systematic exploration. Document every experiment with its parameters and results — this prevents repeating failed experiments and helps you build intuition about the parameter landscape.
For production deployment, consider computational constraints (training time, inference latency, memory requirements), interpretability requirements (can you explain predictions to stakeholders?), and maintenance burden (how often will the model need retraining?). Sometimes a simpler model that is easy to maintain and explain outweighs a marginally more accurate but complex alternative.
Common Mistakes and How to Avoid Them
Beginners working with model evaluation overview frequently make several avoidable mistakes. The most common is rushing to complex techniques without first understanding the data through exploratory analysis. Spend adequate time visualizing distributions, checking correlations, and understanding the domain before choosing an approach.
Another frequent error is evaluating on training data or improperly constructed test sets, leading to over-optimistic performance estimates that crumble in production. Always use proper cross-validation and maintain a truly held-out test set that you evaluate only once at the very end.
Overfitting is perhaps the most pervasive issue — models that perform brilliantly on training data but fail on new data. Signs include a large gap between training and validation performance. Remedies include more training data, stronger regularization, simpler models, data augmentation, and early stopping.
Finally, neglecting feature engineering in favor of trying increasingly complex algorithms is a common trap. In most practical scenarios, thoughtful feature engineering provides larger accuracy gains than switching from one algorithm to another. Invest time in understanding your features and creating informative new ones from domain knowledge.
Real-World Applications and Impact
The techniques covered in model evaluation overview have transformed numerous industries in recent years. In healthcare, they enable early disease detection from medical imaging and patient records, potentially saving millions of lives through earlier intervention. In finance, they power fraud detection systems processing millions of transactions per second, risk assessment models for lending decisions, and algorithmic trading strategies.
In technology companies, these methods drive recommendation systems (suggesting products, content, and connections), search ranking algorithms, natural language understanding in virtual assistants, and autonomous driving perception systems. In manufacturing, they enable predictive maintenance (detecting equipment failures before they occur), quality control automation, and supply chain optimization.
The key to successful real-world application is understanding that production ML systems require much more than just a good model. You need reliable data pipelines, monitoring for data and model drift, A/B testing frameworks to validate improvements, and graceful degradation when the model encounters out-of-distribution inputs. Building complete ML systems, not just models, is what creates business value.
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Model Evaluation Overview.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Machine Learning topic.
Search Terms
machine-learning, machine learning, machine, learning, fundamentals, model, evaluation, overview
Related Machine Learning Topics