ML Notes

Overfitting and Underfitting

Learn to identify and prevent overfitting and underfitting in machine learning models with practical techniques, code examples, and regularization strategies.

Every machine learning practitioner must master the art of building models that generalize well — models that perform on new, unseen data just as well as on training data. Overfitting and underfitting are the two enemies of generalization.

Visual Comparison

Underfitting Good Fit Overfitting (High Bias) (Balanced) (High Variance) y│ y│ . y│ . . │ ___________ │ . . │ ./ \. │ / │ . . │ / . . \ │ / . . . │ . ___ . │/. /\ /\ .\ │ / . . │ . ___/ \___ │ / \/ \/ │/ . . │./ . │./ └─────────────── x └─────────────── x └─────────────── x Straight line through Smooth curve that Wiggly curve that passes curved data (too simple) captures the trend through every point

What is Overfitting?

Overfitting occurs when a model learns not only the underlying patterns but also the noise and random fluctuations in the training data. It performs amazingly on training data but poorly on new data.

python example

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# Create a noisy dataset
X, y = make_classification(n_samples=200, n_features=20, 
                           n_informative=5, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Overfitting model - no constraints
overfit_model = DecisionTreeClassifier(random_state=42)  # No max_depth!
overfit_model.fit(X_train, y_train)

print("OVERFITTING EXAMPLE:")
print(f"  Training accuracy: {overfit_model.score(X_train, y_train):.4f}")
print(f"  Testing accuracy:  {overfit_model.score(X_test, y_test):.4f}")
print(f"  Gap: {overfit_model.score(X_train, y_train) - overfit_model.score(X_test, y_test):.4f}")
print("  → The model memorized training data but fails on new data!")

What is Underfitting?

Underfitting occurs when a model is too simple to capture the underlying patterns in the data. It performs poorly on both training and test data.

python example

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_moons

# Non-linearly separable data
X, y = make_moons(n_samples=200, noise=0.2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Underfitting model - linear model on non-linear data
underfit_model = LogisticRegression()
underfit_model.fit(X_train, y_train)

print("UNDERFITTING EXAMPLE:")
print(f"  Training accuracy: {underfit_model.score(X_train, y_train):.4f}")
print(f"  Testing accuracy:  {underfit_model.score(X_test, y_test):.4f}")
print("  → Both scores are low because the model is too simple!")

Techniques to Prevent Overfitting

1. Regularization

python example

from sklearn.linear_model import Ridge, Lasso, ElasticNet
from sklearn.datasets import make_regression
import numpy as np

X, y = make_regression(n_samples=100, n_features=50, n_informative=10, 
                       noise=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Compare regularized vs unregularized
from sklearn.linear_model import LinearRegression

models = {
    "No Regularization": LinearRegression(),
    "Ridge (L2)": Ridge(alpha=1.0),
    "Lasso (L1)": Lasso(alpha=1.0),
    "ElasticNet": ElasticNet(alpha=1.0, l1_ratio=0.5),
}

for name, model in models.items():
    model.fit(X_train, y_train)
    train_score = model.score(X_train, y_train)
    test_score = model.score(X_test, y_test)
    n_nonzero = np.sum(model.coef_ != 0) if hasattr(model, 'coef_') else 'N/A'
    print(f"{name:20s}: Train R²={train_score:.4f}, Test R²={test_score:.4f}, "
          f"Non-zero coefficients: {n_nonzero}")

2. Cross-Validation

Example

from sklearn.model_selection import cross_val_score from sklearn.tree import DecisionTreeClassifier X, y = make_classification(n_samples=200, n_features=20, random_state=42) # Test different complexities for depth in [2, 5, 10, 20, None]: model = DecisionTreeClassifier(max_depth=depth, random_state=42) scores = cross_val_score(model, X, y, cv=5) label = f"depth={depth}" if depth else "depth=unlimited" print(f" {label:20s}: CV accuracy = {scores.mean():.4f} (+/- {scores.std():.4f})")

3. Early Stopping (for iterative models)

python example

from sklearn.ensemble import GradientBoostingClassifier

# Early stopping prevents overfitting in boosting
model = GradientBoostingClassifier(
    n_estimators=500,
    validation_fraction=0.2,  # Hold out 20% for validation
    n_iter_no_change=10,      # Stop if no improvement for 10 rounds
    random_state=42
)
model.fit(X_train, y_train)
print(f"Stopped at {model.n_estimators_} estimators (out of 500 max)")
print(f"Test accuracy: {model.score(X_test, y_test):.4f}")

4. Data Augmentation and More Data

Example

# More training data is the most reliable way to reduce overfitting from sklearn.model_selection import learning_curve model = DecisionTreeClassifier(random_state=42) sizes, train_scores, val_scores = learning_curve( model, X, y, cv=5, train_sizes=[0.1, 0.2, 0.4, 0.6, 0.8, 1.0] ) print("Learning Curve (shows if more data helps):") for size, train, val in zip(sizes, train_scores.mean(axis=1), val_scores.mean(axis=1)): gap = train - val status = "↑ overfit" if gap > 0.1 else "✓ good" print(f" n={size:4d}: train={train:.3f}, val={val:.3f}, gap={gap:.3f} {status}")

Detection Checklist

Signal	Diagnosis	Action
Train high, test low	Overfitting	Regularize, simplify, more data
Both low	Underfitting	Add features, increase complexity
Both high, small gap	Good fit	Deploy!
Validation improves then drops	Overfitting	Early stopping

Interview Questions

How do you detect overfitting in practice?

Compare training and validation performance. A large gap indicates overfitting. Use learning curves and cross-validation to confirm.

Name 5 techniques to prevent overfitting.

Regularization (L1/L2), cross-validation, early stopping, dropout (neural nets), and reducing model complexity (fewer features, shallower trees).

Can a model overfit with very little data?

Yes! With few samples, complex models easily memorize the training set. This is why small datasets require simpler models or strong regularization.

What is the relationship between model complexity and generalization?

As complexity increases, training error decreases but test error eventually increases (U-shaped curve). The optimal complexity minimizes test error.

How does cross-validation help prevent overfitting?

CV tests the model on multiple held-out folds, giving a realistic estimate of performance on unseen data. It prevents you from selecting hyperparameters that only work on one specific test split.

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Overfitting and Underfitting.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Machine Learning topic.

Search Terms

machine-learning, machine learning, machine, learning, fundamentals, overfitting, and, underfitting

Continue learning this concept

Machine Learning FundamentalsBias vs Variance TradeoffMaster the bias-variance tradeoff in machine learning with intuitive explanations, mathematical formulations, visual examples, and practical strategies to balance model complexity.Machine Learning FundamentalsModel Evaluation OverviewComplete overview of model evaluation strategies in machine learning including metrics for classification and regression, validation techniques, and comparison methods.Machine Learning FundamentalsReinforcement Learning Fundamentals — Machine LearningIntroduction to reinforcement learning concepts including agents, environments, rewards, policies, and value functions with Python examples.Machine Learning FundamentalsSupervised LearningDeep dive into supervised learning covering classification, regression, common algorithms, training process, and practical Python implementations with sklearn.Machine Learning FundamentalsTraining vs Testing DataUnderstanding the importance of splitting data into training, validation, and test sets with proper techniques, ratios, and common mistakes to avoid.Machine Learning FundamentalsTypes of Machine LearningA complete guide to the three main types of machine learning: supervised, unsupervised, and reinforcement learning with examples, comparisons, and Python code.

PreviousModel Evaluation Overview NextReinforcement Learning Fundamentals — Machine Learning

ML Notes

Overfitting and Underfitting

Learn to identify and prevent overfitting and underfitting in machine learning models with practical techniques, code examples, and regularization strategies.

Visual Comparison

What is Overfitting?

Overfitting occurs when a model learns not only the underlying patterns but also the noise and random fluctuations in the training data. It performs amazingly on training data but poorly on new data.

python example

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# Create a noisy dataset
X, y = make_classification(n_samples=200, n_features=20, 
                           n_informative=5, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Overfitting model - no constraints
overfit_model = DecisionTreeClassifier(random_state=42)  # No max_depth!
overfit_model.fit(X_train, y_train)

print("OVERFITTING EXAMPLE:")
print(f"  Training accuracy: {overfit_model.score(X_train, y_train):.4f}")
print(f"  Testing accuracy:  {overfit_model.score(X_test, y_test):.4f}")
print(f"  Gap: {overfit_model.score(X_train, y_train) - overfit_model.score(X_test, y_test):.4f}")
print("  → The model memorized training data but fails on new data!")

What is Underfitting?

Underfitting occurs when a model is too simple to capture the underlying patterns in the data. It performs poorly on both training and test data.

python example

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_moons

# Non-linearly separable data
X, y = make_moons(n_samples=200, noise=0.2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Underfitting model - linear model on non-linear data
underfit_model = LogisticRegression()
underfit_model.fit(X_train, y_train)

print("UNDERFITTING EXAMPLE:")
print(f"  Training accuracy: {underfit_model.score(X_train, y_train):.4f}")
print(f"  Testing accuracy:  {underfit_model.score(X_test, y_test):.4f}")
print("  → Both scores are low because the model is too simple!")

Techniques to Prevent Overfitting

1. Regularization

python example

from sklearn.linear_model import Ridge, Lasso, ElasticNet
from sklearn.datasets import make_regression
import numpy as np

X, y = make_regression(n_samples=100, n_features=50, n_informative=10, 
                       noise=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Compare regularized vs unregularized
from sklearn.linear_model import LinearRegression

models = {
    "No Regularization": LinearRegression(),
    "Ridge (L2)": Ridge(alpha=1.0),
    "Lasso (L1)": Lasso(alpha=1.0),
    "ElasticNet": ElasticNet(alpha=1.0, l1_ratio=0.5),
}

for name, model in models.items():
    model.fit(X_train, y_train)
    train_score = model.score(X_train, y_train)
    test_score = model.score(X_test, y_test)
    n_nonzero = np.sum(model.coef_ != 0) if hasattr(model, 'coef_') else 'N/A'
    print(f"{name:20s}: Train R²={train_score:.4f}, Test R²={test_score:.4f}, "
          f"Non-zero coefficients: {n_nonzero}")

2. Cross-Validation

Example

3. Early Stopping (for iterative models)

python example

from sklearn.ensemble import GradientBoostingClassifier

# Early stopping prevents overfitting in boosting
model = GradientBoostingClassifier(
    n_estimators=500,
    validation_fraction=0.2,  # Hold out 20% for validation
    n_iter_no_change=10,      # Stop if no improvement for 10 rounds
    random_state=42
)
model.fit(X_train, y_train)
print(f"Stopped at {model.n_estimators_} estimators (out of 500 max)")
print(f"Test accuracy: {model.score(X_test, y_test):.4f}")

4. Data Augmentation and More Data

Example

Detection Checklist

Signal	Diagnosis	Action
Train high, test low	Overfitting	Regularize, simplify, more data
Both low	Underfitting	Add features, increase complexity
Both high, small gap	Good fit	Deploy!
Validation improves then drops	Overfitting	Early stopping

Interview Questions

How do you detect overfitting in practice?

Compare training and validation performance. A large gap indicates overfitting. Use learning curves and cross-validation to confirm.

Name 5 techniques to prevent overfitting.

Regularization (L1/L2), cross-validation, early stopping, dropout (neural nets), and reducing model complexity (fewer features, shallower trees).

Can a model overfit with very little data?

Yes! With few samples, complex models easily memorize the training set. This is why small datasets require simpler models or strong regularization.

What is the relationship between model complexity and generalization?

As complexity increases, training error decreases but test error eventually increases (U-shaped curve). The optimal complexity minimizes test error.

How does cross-validation help prevent overfitting?

CV tests the model on multiple held-out folds, giving a realistic estimate of performance on unseen data. It prevents you from selecting hyperparameters that only work on one specific test split.

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Overfitting and Underfitting.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Machine Learning topic.

Search Terms

machine-learning, machine learning, machine, learning, fundamentals, overfitting, and, underfitting

Continue learning this concept

PreviousModel Evaluation Overview NextReinforcement Learning Fundamentals — Machine Learning