ML Notes
HubAI vs ML vs Deep Learning — Machine LearningApplications of Machine LearningHistory of Machine LearningMachine Learning RoadmapWhat is Machine Learning?Calculus for Machine LearningData Preprocessing BasicsLinear Algebra for Machine LearningProbability Basics for Machine LearningPython for Machine LearningStatistics Basics for Machine LearningBias vs Variance TradeoffModel Evaluation OverviewOverfitting and UnderfittingReinforcement Learning Fundamentals — Machine LearningSupervised LearningTraining vs Testing DataTypes of Machine LearningUnsupervised LearningData Cleaning — Machine LearningData Collection for Machine LearningData Visualization for MLEncoding Categorical DataFeature EngineeringFeature ScalingHandling Missing ValuesTrain-Test SplitElastic Net RegressionLasso Regression (L1 Regularization)Multiple Linear RegressionPolynomial RegressionIntroduction to RegressionRegression ProjectsRidge Regression (L2 Regularization)Simple Linear RegressionIntroduction to ClassificationClassification ProjectsDecision TreesK-Nearest Neighbors (KNN)Logistic RegressionNaive Bayes ClassifierRandom Forest ClassificationSupport Vector Machine (SVM)XGBoost for ClassificationClustering in Machine Learning: Complete Guide to Unsupervised LearningClustering Projects: Real-World Applications and Implementation GuidesDBSCAN ClusteringGaussian Mixture Models (GMM): Probabilistic Clustering for Advanced Data AnalysisHierarchical ClusteringK-Means ClusteringDimensionality Reduction: Comprehensive Guide to Feature Reduction TechniquesFeature Selection Techniques: Choosing Informative Variables for ML ModelsLinear Discriminant Analysis (LDA): Supervised Dimensionality ReductionPrincipal Component Analysis (PCA)t-SNE: Non-Linear Dimensionality Reduction for Data VisualizationUMAP: Uniform Manifold Approximation and Projection for Modern Data VisualizationAdaBoost: Adaptive Boosting Algorithm Explained with ExamplesBagging: Bootstrap Aggregating for Reducing Model VarianceBoosting: Sequential Ensemble Learning for Bias and Variance ReductionEnsemble Learning: Complete Guide to Combining ML Models for Better PredictionsGradient Boosting: Building Powerful ML Models Through Sequential OptimizationRandom Forest: Complete Guide to Tree Ensembles and Feature ImportanceStacking: Advanced Ensemble Method Using Meta-Learners for Superior ML PredictionsActivation Functions in Neural NetworksBackpropagation: Complete Algorithm for Training Neural NetworksIntroduction to Deep Learning: Neural Networks, Architectures, and ApplicationsDeep Learning Projects: Hands-On Applications and Case StudiesNeural Networks Basics: Perceptron, Architecture, and TrainingPerceptron: Foundational Single-Neuron Classifier for Binary ClassificationPyTorch Basics: Building and Training Neural NetworksTensorFlow Basics: Building and Training Neural NetworksIntroduction to Natural Language Processing: Fundamentals and ApplicationsNLP Projects: Practical Applications from Sentiment Analysis to Machine TranslationStemming and Lemmatization: Reducing Words to Root Forms in NLPText Preprocessing: Essential Steps for NLP ApplicationsTF-IDF: Term Frequency-Inverse Document Frequency for Text Feature ExtractionTokenization in NLP: Breaking Text into Meaningful UnitsTransformers: The Architecture Behind Modern NLP ModelsWord Embeddings: Representing Words as Vectors for NLPIntroduction to Computer VisionComputer Vision ProjectsImage Classification: Building and Deploying CNN ModelsImage Processing BasicsImage Segmentation — Machine LearningObject DetectionOpenCV BasicsAgents and Environments in RLDeep Q-Networks (DQN)Policy Gradient Methods — Machine LearningQ-Learning AlgorithmReinforcement Learning: IntroductionReinforcement Learning ProjectsAccuracy, Precision, Recall: Classification Metrics Explained for ML InterviewConfusion Matrix: Complete Guide to Classification EvaluationCross-Validation: Comprehensive Guide to Robust Model EvaluationF1 Score: Balanced Classification Metric for Imbalanced DataGrid Search: Complete Guide to Hyperparameter Tuning in Machine LearningHyperparameter Tuning: Complete Guide to Optimizing Machine Learning ModelsModel Selection: Choosing the Best Algorithm for Your ML ProblemROC and AUC: Complete Guide to Classification Threshold EvaluationBuild an AI Assistant with LLMsBuild a Conversational ChatbotCustomer Churn PredictionFace Recognition System — Machine LearningHouse Price PredictionRecommendation System ProjectSentiment Analysis Project — Machine LearningSpam Email ClassifierML Case Study Questions for InterviewsML Coding Questions for Technical InterviewsMachine Learning Interview CheatsheetEssential Machine Learning Interview QuestionsStatistics for Machine Learning InterviewsDatasets for ML PracticeImportant ML FormulasMachine Learning GlossaryRecommended ML Books and CoursesCI/CD for Machine LearningDocker for Machine LearningFastAPI for Machine Learning APIsFlask API for Machine LearningMLOps: IntroductionModel Deployment StrategiesML Model Monitoring & Observability
Master simple linear regression from mathematical derivation to Python implementation including gradient descent, normal equation, and assumptions validation.
Simple linear regression models the relationship between a single input feature (X) and a continuous target (y) using a straight line. It's the foundation of all regression techniques and a great starting point for understanding how models learn.
The Concept
Example
y = mx + b (from high school algebra)
y = w₁x + w₀ (ML notation)
ŷ = β₁x + β₀ (statistics notation)
Where:
y = predicted value (dependent variable)
x = input feature (independent variable)
w₁ = slope (weight/coefficient) - how much y changes per unit x
w₀ = intercept (bias) - y value when x = 0
Visualization
y (target)
│ /
│ / •
│ /• •
│ / • •
│ /•
│ /• •
│ / •
│ /•
│ /
│/______________ x (feature)
The line minimizes total distance to all points
Implementation from Scratch
Example
import numpy as np
class SimpleLinearRegression:
"""Linear regression using Ordinary Least Squares"""
def __init__(self):
self.slope = None
self.intercept = None
def fit(self, X, y):
"""Find best line using OLS formula"""
n = len(X)
x_mean = X.mean()
y_mean = y.mean()
# OLS formula: slope = Σ(xᵢ - x̄)(yᵢ - ȳ) / Σ(xᵢ - x̄)²
numerator = np.sum((X - x_mean) * (y - y_mean))
denominator = np.sum((X - x_mean) ** 2)
self.slope = numerator / denominator
self.intercept = y_mean - self.slope * x_mean
return self
def predict(self, X):
return self.slope * X + self.intercept
def score(self, X, y):
"""R² score"""
y_pred = self.predict(X)
ss_res = np.sum((y - y_pred) ** 2)
ss_tot = np.sum((y - y.mean()) ** 2)
return 1 - (ss_res / ss_tot)
# Example: predicting salary from years of experience
experience = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], dtype=float)
salary = np.array([30, 35, 40, 48, 55, 60, 68, 75, 82, 90], dtype=float) # in thousands
model = SimpleLinearRegression()
model.fit(experience, salary)
print(f"Equation: salary = {model.slope:.2f} × experience + {model.intercept:.2f}")
print(f"R² Score: {model.score(experience, salary):.4f}")
print(f"Prediction for 5.5 years: ${model.predict(5.5)*1000:,.0f}")
Using Scikit-Learn
Example
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np
# Reshape for sklearn (needs 2D array)
X = experience.reshape(-1, 1)
y = salary
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
model = LinearRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(f"Coefficient (slope): {model.coef_[0]:.4f}")
print(f"Intercept: {model.intercept_:.4f}")
print(f"R² Score: {r2_score(y_test, y_pred):.4f}")
print(f"RMSE: {np.sqrt(mean_squared_error(y_test, y_pred)):.4f}")
Gradient Descent Approach
Example
import numpy as np
def linear_regression_gradient_descent(X, y, lr=0.01, epochs=1000):
"""Learn parameters using gradient descent"""
n = len(X)
w = 0.0 # slope
b = 0.0 # intercept
history = []
for epoch in range(epochs):
# Predictions
y_pred = w * X + b
# Compute gradients
dw = -(2/n) * np.sum(X * (y - y_pred))
db = -(2/n) * np.sum(y - y_pred)
# Update parameters
w -= lr * dw
b -= lr * db
# Track loss
loss = np.mean((y - y_pred) ** 2)
history.append(loss)
return w, b, history
w, b, history = linear_regression_gradient_descent(experience, salary, lr=0.01, epochs=5000)
print(f"Gradient Descent Result:")
print(f" Slope: {w:.4f}, Intercept: {b:.4f}")
print(f" Final MSE: {history[-1]:.4f}")
print(f" Converged after {len(history)} iterations")
Checking Assumptions
python example
from sklearn.linear_model import LinearRegression
import numpy as np
model = LinearRegression()
model.fit(experience.reshape(-1, 1), salary)
residuals = salary - model.predict(experience.reshape(-1, 1))
print("Residual Analysis:")
print(f" Mean of residuals: {residuals.mean():.6f} (should be ~0)")
print(f" Std of residuals: {residuals.std():.4f}")
print(f" Normality (Shapiro-Wilk p-value): ", end="")
from scipy.stats import shapiro
_, p_value = shapiro(residuals)
print(f"{p_value:.4f} (> 0.05 means normal)")Interview Questions
- Derive the OLS formula for simple linear regression.
Minimize MSE = Σ(yᵢ - (wx+b))². Take partial derivatives with respect to w and b, set to zero. Solve the system: w = Σ(x-x̄)(y-ȳ)/Σ(x-x̄)², b = ȳ - w*x̄.
- What is the difference between OLS and gradient descent?
OLS gives the exact solution in one step (matrix inversion). Gradient descent iteratively approaches the solution. OLS is faster for small datasets; GD scales better to millions of features.
- What does R²=1 mean and is it always good?
Perfect fit — the model explains all variance. But it might indicate overfitting if you have many features relative to samples. Always check on test data.
- What if the relationship between X and y is non-linear?
Options: transform features (log, sqrt), use polynomial regression, use non-linear models, or check if a different feature would be linear.
- How do you interpret the coefficient in simple linear regression?
The coefficient means "for each 1-unit increase in X, y changes by [coefficient] units on average, holding other factors constant."
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Simple Linear Regression.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Machine Learning topic.
Search Terms
machine-learning, machine learning, machine, learning, regression, simple, linear, simple linear regression
Related Machine Learning Topics
Continue learning this concept
RegressionMultiple Linear RegressionComplete guide to multiple linear regression with multiple features, multicollinearity handling, feature significance testing, and practical Python examples.RegressionElastic Net RegressionLearn Elastic Net regression combining L1 and L2 regularization for handling correlated features and performing stable feature selection.RegressionLasso Regression (L1 Regularization)Learn Lasso regression for simultaneous feature selection and regularization including L1 penalty mechanics, sparsity, and comparison with Ridge.RegressionPolynomial RegressionLearn polynomial regression for modeling non-linear relationships including degree selection, overfitting prevention, and comparison with other non-linear methods.RegressionIntroduction to RegressionComplete introduction to regression analysis in machine learning covering types, applications, evaluation metrics, and when to choose regression over classification.RegressionRegression ProjectsHands-on regression projects including house price prediction, salary estimation, and demand forecasting with complete Python implementations.