ML Notes
HubAI vs ML vs Deep Learning — Machine LearningApplications of Machine LearningHistory of Machine LearningMachine Learning RoadmapWhat is Machine Learning?Calculus for Machine LearningData Preprocessing BasicsLinear Algebra for Machine LearningProbability Basics for Machine LearningPython for Machine LearningStatistics Basics for Machine LearningBias vs Variance TradeoffModel Evaluation OverviewOverfitting and UnderfittingReinforcement Learning Fundamentals — Machine LearningSupervised LearningTraining vs Testing DataTypes of Machine LearningUnsupervised LearningData Cleaning — Machine LearningData Collection for Machine LearningData Visualization for MLEncoding Categorical DataFeature EngineeringFeature ScalingHandling Missing ValuesTrain-Test SplitElastic Net RegressionLasso Regression (L1 Regularization)Multiple Linear RegressionPolynomial RegressionIntroduction to RegressionRegression ProjectsRidge Regression (L2 Regularization)Simple Linear RegressionIntroduction to ClassificationClassification ProjectsDecision TreesK-Nearest Neighbors (KNN)Logistic RegressionNaive Bayes ClassifierRandom Forest ClassificationSupport Vector Machine (SVM)XGBoost for ClassificationClustering in Machine Learning: Complete Guide to Unsupervised LearningClustering Projects: Real-World Applications and Implementation GuidesDBSCAN ClusteringGaussian Mixture Models (GMM): Probabilistic Clustering for Advanced Data AnalysisHierarchical ClusteringK-Means ClusteringDimensionality Reduction: Comprehensive Guide to Feature Reduction TechniquesFeature Selection Techniques: Choosing Informative Variables for ML ModelsLinear Discriminant Analysis (LDA): Supervised Dimensionality ReductionPrincipal Component Analysis (PCA)t-SNE: Non-Linear Dimensionality Reduction for Data VisualizationUMAP: Uniform Manifold Approximation and Projection for Modern Data VisualizationAdaBoost: Adaptive Boosting Algorithm Explained with ExamplesBagging: Bootstrap Aggregating for Reducing Model VarianceBoosting: Sequential Ensemble Learning for Bias and Variance ReductionEnsemble Learning: Complete Guide to Combining ML Models for Better PredictionsGradient Boosting: Building Powerful ML Models Through Sequential OptimizationRandom Forest: Complete Guide to Tree Ensembles and Feature ImportanceStacking: Advanced Ensemble Method Using Meta-Learners for Superior ML PredictionsActivation Functions in Neural NetworksBackpropagation: Complete Algorithm for Training Neural NetworksIntroduction to Deep Learning: Neural Networks, Architectures, and ApplicationsDeep Learning Projects: Hands-On Applications and Case StudiesNeural Networks Basics: Perceptron, Architecture, and TrainingPerceptron: Foundational Single-Neuron Classifier for Binary ClassificationPyTorch Basics: Building and Training Neural NetworksTensorFlow Basics: Building and Training Neural NetworksIntroduction to Natural Language Processing: Fundamentals and ApplicationsNLP Projects: Practical Applications from Sentiment Analysis to Machine TranslationStemming and Lemmatization: Reducing Words to Root Forms in NLPText Preprocessing: Essential Steps for NLP ApplicationsTF-IDF: Term Frequency-Inverse Document Frequency for Text Feature ExtractionTokenization in NLP: Breaking Text into Meaningful UnitsTransformers: The Architecture Behind Modern NLP ModelsWord Embeddings: Representing Words as Vectors for NLPIntroduction to Computer VisionComputer Vision ProjectsImage Classification: Building and Deploying CNN ModelsImage Processing BasicsImage Segmentation — Machine LearningObject DetectionOpenCV BasicsAgents and Environments in RLDeep Q-Networks (DQN)Policy Gradient Methods — Machine LearningQ-Learning AlgorithmReinforcement Learning: IntroductionReinforcement Learning ProjectsAccuracy, Precision, Recall: Classification Metrics Explained for ML InterviewConfusion Matrix: Complete Guide to Classification EvaluationCross-Validation: Comprehensive Guide to Robust Model EvaluationF1 Score: Balanced Classification Metric for Imbalanced DataGrid Search: Complete Guide to Hyperparameter Tuning in Machine LearningHyperparameter Tuning: Complete Guide to Optimizing Machine Learning ModelsModel Selection: Choosing the Best Algorithm for Your ML ProblemROC and AUC: Complete Guide to Classification Threshold EvaluationBuild an AI Assistant with LLMsBuild a Conversational ChatbotCustomer Churn PredictionFace Recognition System — Machine LearningHouse Price PredictionRecommendation System ProjectSentiment Analysis Project — Machine LearningSpam Email ClassifierML Case Study Questions for InterviewsML Coding Questions for Technical InterviewsMachine Learning Interview CheatsheetEssential Machine Learning Interview QuestionsStatistics for Machine Learning InterviewsDatasets for ML PracticeImportant ML FormulasMachine Learning GlossaryRecommended ML Books and CoursesCI/CD for Machine LearningDocker for Machine LearningFastAPI for Machine Learning APIsFlask API for Machine LearningMLOps: IntroductionModel Deployment StrategiesML Model Monitoring & Observability
Understanding the importance of splitting data into training, validation, and test sets with proper techniques, ratios, and common mistakes to avoid.
The way you split your data determines whether you can trust your model's performance estimates. Get this wrong, and you might deploy a model that works perfectly in development but fails catastrophically in production.
Why Split Data?
| │ If you test on training data | │ |
| │ Student studies EXACT exam questions | Gets 100% │ |
| │ But learned nothing | Fails real-world problems │ |
| │ If you test on unseen data | │ |
| │ Student studies concepts | Takes NEW exam │ |
The Three-Way Split
| Training Set | Validation | Test Set |
|---|---|---|
| (60-70%) | (15-20%) | (15-20%) |
| Used to train the | Used to tune | Used ONCE at the end |
| model parameters | hyperparams | for final evaluation |
Basic Train-Test Split
python example
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
import numpy as np
# Load data
iris = load_iris()
X, y = iris.data, iris.target
# Simple 80/20 split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Total samples: {len(X)}")
print(f"Training samples: {len(X_train)} ({len(X_train)/len(X):.0%})")
print(f"Testing samples: {len(X_test)} ({len(X_test)/len(X):.0%})")
# Check class distribution is maintained (stratified)
print(f"\nClass distribution:")
print(f" Original: {np.bincount(y) / len(y)}")
print(f" Training: {np.bincount(y_train) / len(y_train)}")
print(f" Testing: {np.bincount(y_test) / len(y_test)}")Three-Way Split for Hyperparameter Tuning
python example
# Method 1: Manual three-way split
X_temp, X_test, y_temp, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
X_train, X_val, y_train, y_val = train_test_split(
X_temp, y_temp, test_size=0.25, random_state=42, stratify=y_temp # 0.25 * 0.8 = 0.2
)
print(f"Training: {len(X_train)} samples ({len(X_train)/len(X):.0%})")
print(f"Validation: {len(X_val)} samples ({len(X_val)/len(X):.0%})")
print(f"Testing: {len(X_test)} samples ({len(X_test)/len(X):.0%})")Cross-Validation Instead of Fixed Validation Set
python example
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier
# Cross-validation uses all data more efficiently
model = RandomForestClassifier(random_state=42)
# 5-Fold Stratified CV
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X_train, y_train, cv=cv, scoring='accuracy')
print("Cross-Validation Results:")
for i, score in enumerate(scores, 1):
print(f" Fold {i}: {score:.4f}")
print(f" Mean: {scores.mean():.4f} (+/- {scores.std():.4f})")Special Cases
Time Series Data (NO random splitting!)
Example
from sklearn.model_selection import TimeSeriesSplit
# Time series must maintain temporal order
tss = TimeSeriesSplit(n_splits=5)
print("Time Series Split (maintains temporal order):")
print("=" * 50)
X_dummy = np.arange(100).reshape(-1, 1)
y_dummy = np.random.randint(0, 2, 100)
for i, (train_idx, test_idx) in enumerate(tss.split(X_dummy)):
print(f" Fold {i+1}: Train [{train_idx[0]}-{train_idx[-1]}], "
f"Test [{test_idx[0]}-{test_idx[-1]}]")
Recommended Split Ratios
| Dataset Size | Train | Validation | Test |
|---|---|---|---|
| < 1,000 | Use cross-validation | (built into CV) | 20% |
| 1,000 - 100,000 | 70% | 15% | 15% |
| 100,000 - 1M | 80% | 10% | 10% |
| > 1M | 90%+ | 5% | 5% |
Common Mistakes
python example
# WRONG: Scaling before splitting (data leakage!)
from sklearn.preprocessing import StandardScaler
# BAD - test data influences scaling
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X) # Uses ALL data
X_train_bad, X_test_bad = train_test_split(X_scaled, test_size=0.2)
# CORRECT - scale using only training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train) # Fit on train only
X_test_scaled = scaler.transform(X_test) # Transform test with train params
print("Correct: Scaler fitted only on training data")Interview Questions
- Why can't you use the test set for hyperparameter tuning?
The test set must simulate truly unseen data. If you tune hyperparameters on it, you're indirectly fitting to it, and your performance estimate becomes optimistic.
- What is stratified splitting and when is it important?
Stratified splitting maintains the class proportions from the original dataset in each split. It's critical for imbalanced datasets where random splitting might leave minority classes underrepresented.
- How do you split time series data?
Never randomly — always use temporal order. Train on past data, validate/test on future data. Use TimeSeriesSplit or walk-forward validation.
- What is data leakage through preprocessing?
If you fit a scaler (or any transform) on the entire dataset before splitting, test data statistics leak into training. Always fit transforms on training data only.
- When would you use Leave-One-Out cross-validation?
With very small datasets (< 50 samples) where you can't afford to hold out many samples. It's computationally expensive but gives almost unbiased estimates.
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Training vs Testing Data.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Machine Learning topic.
Search Terms
machine-learning, machine learning, machine, learning, fundamentals, training, testing, data
Related Machine Learning Topics
Continue learning this concept
Machine Learning FundamentalsBias vs Variance TradeoffMaster the bias-variance tradeoff in machine learning with intuitive explanations, mathematical formulations, visual examples, and practical strategies to balance model complexity.Machine Learning FundamentalsModel Evaluation OverviewComplete overview of model evaluation strategies in machine learning including metrics for classification and regression, validation techniques, and comparison methods.Machine Learning FundamentalsOverfitting and UnderfittingLearn to identify and prevent overfitting and underfitting in machine learning models with practical techniques, code examples, and regularization strategies.Machine Learning FundamentalsReinforcement Learning Fundamentals — Machine LearningIntroduction to reinforcement learning concepts including agents, environments, rewards, policies, and value functions with Python examples.Machine Learning FundamentalsSupervised LearningDeep dive into supervised learning covering classification, regression, common algorithms, training process, and practical Python implementations with sklearn.Machine Learning FundamentalsTypes of Machine LearningA complete guide to the three main types of machine learning: supervised, unsupervised, and reinforcement learning with examples, comparisons, and Python code.