ML Notes

Introduction to Regression

Complete introduction to regression analysis in machine learning covering types, applications, evaluation metrics, and when to choose regression over classification.

Regression is a supervised learning technique used to predict continuous numerical values. Unlike classification which predicts categories (spam/not spam), regression predicts quantities — house prices, temperatures, stock prices, or any measurable value.

Regression vs Classification

Input: Email text		Input: House features
Output: Spam or Not		Output: $425,000
(Discrete categories)		(Continuous number)

Types of Regression

Type	Formula	Use Case
Simple Linear	y = mx + b	One feature predicts target
Multiple Linear	y = w₁x₁ + w₂x₂ + ... + b	Multiple features
Polynomial	y = w₁x + w₂x² + w₃x³ + b	Non-linear relationships
Ridge (L2)	Linear + L2 penalty	Prevent overfitting
Lasso (L1)	Linear + L1 penalty	Feature selection
ElasticNet	Linear + L1 + L2	Best of both

Quick Comparison

python example

import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
from sklearn.datasets import fetch_california_housing

housing = fetch_california_housing()
X, y = housing.data, housing.target

models = {
    "Linear Regression": LinearRegression(),
    "Ridge (alpha=1)": Ridge(alpha=1.0),
    "Lasso (alpha=0.1)": Lasso(alpha=0.1),
}

print("Regression Model Comparison (California Housing):")
print(f"{'Model':<25} {'R² (CV Mean)':<15} {'R² (CV Std)'}")
print("-" * 55)
for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=5, scoring='r2')
    print(f"{name:<25} {scores.mean():<15.4f} {scores.std():.4f}")

Regression Evaluation Metrics

Example

from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score import numpy as np # Simulated predictions y_true = np.array([3.0, 5.0, 2.5, 7.0, 4.5]) y_pred = np.array([2.8, 5.2, 2.3, 6.8, 4.8]) print("Regression Metrics:") print(f" MSE: {mean_squared_error(y_true, y_pred):.4f}") print(f" RMSE: {np.sqrt(mean_squared_error(y_true, y_pred)):.4f}") print(f" MAE: {mean_absolute_error(y_true, y_pred):.4f}") print(f" R²: {r2_score(y_true, y_pred):.4f}") print(f"\nInterpretation:") print(f" RMSE: On average, predictions are off by ±{np.sqrt(mean_squared_error(y_true, y_pred)):.2f} units") print(f" R²: Model explains {r2_score(y_true, y_pred)*100:.1f}% of variance in target")

Real-World Regression Applications

Real estate: House/apartment price prediction
Finance: Stock price forecasting, revenue prediction
Healthcare: Patient recovery time, drug dosage
Marketing: Customer lifetime value, ad spend ROI
Energy: Power consumption forecasting
Manufacturing: Yield prediction, quality metrics

Interview Questions

What assumptions does linear regression make?

Linearity, independence of errors, homoscedasticity (constant variance), normal distribution of residuals, and no multicollinearity between features.

When would you use MAE vs MSE?

MSE penalizes large errors more (squared). Use MAE when outliers are expected and all errors should be treated equally. Use MSE when large errors are particularly costly.

What does R² = 0.85 mean?

The model explains 85% of the variance in the target variable. The remaining 15% is unexplained variation (noise or missing features).

Can R² be negative?

Yes! If the model is worse than predicting the mean for every sample. This means the model is actively harmful.

How do you handle non-linear relationships in regression?

Add polynomial features, use log/sqrt transforms, apply non-linear models (decision trees, SVR, neural networks), or use splines.

Deep Dive: Core Concepts Explained

To truly master regression, you need to understand not just the how but the why behind each step. The fundamental principle is that every technique in machine learning represents a specific assumption about the data. When that assumption holds in practice, the technique works well; when it is violated, performance degrades. This is why understanding the mathematical foundation matters — it tells you exactly when and why a method will succeed or fail.

Let us think about this from first principles. Every machine learning algorithm is essentially an optimization problem: find the parameters that minimize some measure of error on training data while generalizing to unseen data. The specific form of the error measure, the constraints on parameters, and the optimization procedure differ between algorithms, but this fundamental structure is universal. Once you internalize this perspective, learning new algorithms becomes much faster because you immediately ask: what is being optimized? What assumptions are being made? What are the failure modes?

Practitioners who understand these foundations can diagnose problems that others find mysterious. When a model underperforms, they can identify whether the issue is insufficient data, inappropriate model assumptions, poor optimization (not converging), or overfitting. Each diagnosis leads to a specific remedy, turning model development from trial-and-error into systematic engineering.

Practical Implementation Guide

When implementing regression in real projects, follow this systematic approach. Start by establishing a simple baseline — often a trivial model like predicting the mean or most frequent class. This baseline tells you the minimum performance your sophisticated approach must beat to justify its complexity. Next, implement the standard version of the algorithm with default parameters. Evaluate it rigorously using cross-validation and appropriate metrics for your problem type.

Only after establishing this solid foundation should you begin optimization. Tune one hyperparameter at a time while holding others fixed, observing how each affects performance. Use grid search or randomized search for systematic exploration. Document every experiment with its parameters and results — this prevents repeating failed experiments and helps you build intuition about the parameter landscape.

For production deployment, consider computational constraints (training time, inference latency, memory requirements), interpretability requirements (can you explain predictions to stakeholders?), and maintenance burden (how often will the model need retraining?). Sometimes a simpler model that is easy to maintain and explain outweighs a marginally more accurate but complex alternative.

Common Mistakes and How to Avoid Them

Beginners working with regression frequently make several avoidable mistakes. The most common is rushing to complex techniques without first understanding the data through exploratory analysis. Spend adequate time visualizing distributions, checking correlations, and understanding the domain before choosing an approach.

Another frequent error is evaluating on training data or improperly constructed test sets, leading to over-optimistic performance estimates that crumble in production. Always use proper cross-validation and maintain a truly held-out test set that you evaluate only once at the very end.

Overfitting is perhaps the most pervasive issue — models that perform brilliantly on training data but fail on new data. Signs include a large gap between training and validation performance. Remedies include more training data, stronger regularization, simpler models, data augmentation, and early stopping.

Finally, neglecting feature engineering in favor of trying increasingly complex algorithms is a common trap. In most practical scenarios, thoughtful feature engineering provides larger accuracy gains than switching from one algorithm to another. Invest time in understanding your features and creating informative new ones from domain knowledge.

Real-World Applications and Impact

The techniques covered in regression have transformed numerous industries in recent years. In healthcare, they enable early disease detection from medical imaging and patient records, potentially saving millions of lives through earlier intervention. In finance, they power fraud detection systems processing millions of transactions per second, risk assessment models for lending decisions, and algorithmic trading strategies.

In technology companies, these methods drive recommendation systems (suggesting products, content, and connections), search ranking algorithms, natural language understanding in virtual assistants, and autonomous driving perception systems. In manufacturing, they enable predictive maintenance (detecting equipment failures before they occur), quality control automation, and supply chain optimization.

The key to successful real-world application is understanding that production ML systems require much more than just a good model. You need reliable data pipelines, monitoring for data and model drift, A/B testing frameworks to validate improvements, and graceful degradation when the model encounters out-of-distribution inputs. Building complete ML systems, not just models, is what creates business value.

Building Intuition Through Examples

Let us ground regression with concrete examples that build intuition. Consider a simple analogy: predicting whether a student will pass an exam based on hours studied, attendance rate, and previous grades. A linear model might learn that each additional hour of study increases pass probability by 5 percent — simple, interpretable, but possibly wrong if the relationship is non-linear (diminishing returns after 30 hours, or a threshold effect where less than 10 hours almost guarantees failure regardless of other factors).

More complex models can capture these non-linear patterns but require more data and risk overfitting. The art of machine learning is choosing the right level of complexity for your data size and noise level. Too simple and you underfit (miss real patterns). Too complex and you overfit (hallucinate patterns from noise). This bias-variance tradeoff is the central tension in all of machine learning, and every technique we study offers a different way to navigate it.

When working through examples, always ask: what patterns is this model learning? Would those patterns generalize to new data from the same distribution? What if the distribution shifts (different students, different exam, different semester)? This critical thinking about generalization is what separates effective practitioners from those who produce impressive training metrics but disappointing production results.

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Introduction to Regression.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Machine Learning topic.

Search Terms

machine-learning, machine learning, machine, learning, regression, introduction, introduction to regression

Continue learning this concept

RegressionElastic Net RegressionLearn Elastic Net regression combining L1 and L2 regularization for handling correlated features and performing stable feature selection.RegressionLasso Regression (L1 Regularization)Learn Lasso regression for simultaneous feature selection and regularization including L1 penalty mechanics, sparsity, and comparison with Ridge.RegressionMultiple Linear RegressionComplete guide to multiple linear regression with multiple features, multicollinearity handling, feature significance testing, and practical Python examples.RegressionPolynomial RegressionLearn polynomial regression for modeling non-linear relationships including degree selection, overfitting prevention, and comparison with other non-linear methods.RegressionRegression ProjectsHands-on regression projects including house price prediction, salary estimation, and demand forecasting with complete Python implementations.RegressionRidge Regression (L2 Regularization)Master Ridge regression including L2 regularization theory, alpha tuning, comparison with OLS, and practical implementation for preventing overfitting.

PreviousPolynomial Regression NextRegression Projects