ML Notes
HubAI vs ML vs Deep Learning — Machine LearningApplications of Machine LearningHistory of Machine LearningMachine Learning RoadmapWhat is Machine Learning?Calculus for Machine LearningData Preprocessing BasicsLinear Algebra for Machine LearningProbability Basics for Machine LearningPython for Machine LearningStatistics Basics for Machine LearningBias vs Variance TradeoffModel Evaluation OverviewOverfitting and UnderfittingReinforcement Learning Fundamentals — Machine LearningSupervised LearningTraining vs Testing DataTypes of Machine LearningUnsupervised LearningData Cleaning — Machine LearningData Collection for Machine LearningData Visualization for MLEncoding Categorical DataFeature EngineeringFeature ScalingHandling Missing ValuesTrain-Test SplitElastic Net RegressionLasso Regression (L1 Regularization)Multiple Linear RegressionPolynomial RegressionIntroduction to RegressionRegression ProjectsRidge Regression (L2 Regularization)Simple Linear RegressionIntroduction to ClassificationClassification ProjectsDecision TreesK-Nearest Neighbors (KNN)Logistic RegressionNaive Bayes ClassifierRandom Forest ClassificationSupport Vector Machine (SVM)XGBoost for ClassificationClustering in Machine Learning: Complete Guide to Unsupervised LearningClustering Projects: Real-World Applications and Implementation GuidesDBSCAN ClusteringGaussian Mixture Models (GMM): Probabilistic Clustering for Advanced Data AnalysisHierarchical ClusteringK-Means ClusteringDimensionality Reduction: Comprehensive Guide to Feature Reduction TechniquesFeature Selection Techniques: Choosing Informative Variables for ML ModelsLinear Discriminant Analysis (LDA): Supervised Dimensionality ReductionPrincipal Component Analysis (PCA)t-SNE: Non-Linear Dimensionality Reduction for Data VisualizationUMAP: Uniform Manifold Approximation and Projection for Modern Data VisualizationAdaBoost: Adaptive Boosting Algorithm Explained with ExamplesBagging: Bootstrap Aggregating for Reducing Model VarianceBoosting: Sequential Ensemble Learning for Bias and Variance ReductionEnsemble Learning: Complete Guide to Combining ML Models for Better PredictionsGradient Boosting: Building Powerful ML Models Through Sequential OptimizationRandom Forest: Complete Guide to Tree Ensembles and Feature ImportanceStacking: Advanced Ensemble Method Using Meta-Learners for Superior ML PredictionsActivation Functions in Neural NetworksBackpropagation: Complete Algorithm for Training Neural NetworksIntroduction to Deep Learning: Neural Networks, Architectures, and ApplicationsDeep Learning Projects: Hands-On Applications and Case StudiesNeural Networks Basics: Perceptron, Architecture, and TrainingPerceptron: Foundational Single-Neuron Classifier for Binary ClassificationPyTorch Basics: Building and Training Neural NetworksTensorFlow Basics: Building and Training Neural NetworksIntroduction to Natural Language Processing: Fundamentals and ApplicationsNLP Projects: Practical Applications from Sentiment Analysis to Machine TranslationStemming and Lemmatization: Reducing Words to Root Forms in NLPText Preprocessing: Essential Steps for NLP ApplicationsTF-IDF: Term Frequency-Inverse Document Frequency for Text Feature ExtractionTokenization in NLP: Breaking Text into Meaningful UnitsTransformers: The Architecture Behind Modern NLP ModelsWord Embeddings: Representing Words as Vectors for NLPIntroduction to Computer VisionComputer Vision ProjectsImage Classification: Building and Deploying CNN ModelsImage Processing BasicsImage Segmentation — Machine LearningObject DetectionOpenCV BasicsAgents and Environments in RLDeep Q-Networks (DQN)Policy Gradient Methods — Machine LearningQ-Learning AlgorithmReinforcement Learning: IntroductionReinforcement Learning ProjectsAccuracy, Precision, Recall: Classification Metrics Explained for ML InterviewConfusion Matrix: Complete Guide to Classification EvaluationCross-Validation: Comprehensive Guide to Robust Model EvaluationF1 Score: Balanced Classification Metric for Imbalanced DataGrid Search: Complete Guide to Hyperparameter Tuning in Machine LearningHyperparameter Tuning: Complete Guide to Optimizing Machine Learning ModelsModel Selection: Choosing the Best Algorithm for Your ML ProblemROC and AUC: Complete Guide to Classification Threshold EvaluationBuild an AI Assistant with LLMsBuild a Conversational ChatbotCustomer Churn PredictionFace Recognition System — Machine LearningHouse Price PredictionRecommendation System ProjectSentiment Analysis Project — Machine LearningSpam Email ClassifierML Case Study Questions for InterviewsML Coding Questions for Technical InterviewsMachine Learning Interview CheatsheetEssential Machine Learning Interview QuestionsStatistics for Machine Learning InterviewsDatasets for ML PracticeImportant ML FormulasMachine Learning GlossaryRecommended ML Books and CoursesCI/CD for Machine LearningDocker for Machine LearningFastAPI for Machine Learning APIsFlask API for Machine LearningMLOps: IntroductionModel Deployment StrategiesML Model Monitoring & Observability
Essential linear algebra concepts for ML including vectors, matrices, eigenvalues, and their applications in algorithms like PCA, SVD, and neural networks.
Linear algebra is the mathematical backbone of machine learning. Every dataset is a matrix, every model learns weight vectors, and transformations like PCA are eigenvalue decompositions. You don't need a PhD in math — but understanding these core concepts will make you a much better ML practitioner.
Why Linear Algebra Matters in ML
| ML Concept | Linear Algebra Operation |
| Dataset | Matrix (n × d) |
| Features | Vectors in d-dimensional space |
| Linear regression | Solving system of equations |
| PCA | Eigenvalue decomposition |
| Neural networks | Matrix multiplications + nonlinearities |
| Embeddings | Vectors in high-dimensional space |
Vectors
A vector is an ordered list of numbers representing a point or direction in space.
Example
import numpy as np
# Vectors represent data points
feature_vector = np.array([1.5, 2.3, 0.7, 4.1]) # One data sample
print(f"Feature vector: {feature_vector}")
print(f"Dimensions: {feature_vector.shape[0]}")
# Vector operations
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
# Dot product (used everywhere in ML)
dot = np.dot(a, b) # 1*4 + 2*5 + 3*6 = 32
print(f"Dot product: {dot}")
# Vector norm (magnitude/length)
norm = np.linalg.norm(a)
print(f"L2 Norm of a: {norm:.4f}")
# Unit vector (normalization)
unit_a = a / np.linalg.norm(a)
print(f"Unit vector: {unit_a.round(4)}")
print(f"Its norm: {np.linalg.norm(unit_a):.4f}")
# Cosine similarity (used in NLP, recommendations)
cosine_sim = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
print(f"Cosine similarity: {cosine_sim:.4f}")
Matrices
A matrix is a 2D array of numbers. In ML, your dataset IS a matrix.
Example
import numpy as np
# Dataset as a matrix: rows = samples, columns = features
X = np.array([
[1.0, 2.0, 3.0], # Sample 1
[4.0, 5.0, 6.0], # Sample 2
[7.0, 8.0, 9.0], # Sample 3
])
print(f"Dataset shape: {X.shape}") # (3 samples, 3 features)
# Matrix multiplication (core of neural networks)
W = np.array([[0.1, 0.2],
[0.3, 0.4],
[0.5, 0.6]])
output = X @ W # (3×3) @ (3×2) = (3×2)
print(f"Linear transformation: {X.shape} @ {W.shape} = {output.shape}")
# Transpose
print(f"X^T shape: {X.T.shape}")
# Inverse (used in solving linear regression analytically)
A = np.array([[2, 1], [5, 3]])
A_inv = np.linalg.inv(A)
print(f"A * A_inv = \n{(A @ A_inv).round(10)}") # Identity matrix
Eigenvalues and Eigenvectors
Eigenvectors are directions that don't change under a transformation — they only get scaled by the eigenvalue. This is the math behind PCA.
Example
import numpy as np
# Covariance matrix of data
np.random.seed(42)
X = np.random.randn(100, 2) @ np.array([[2, 1], [1, 3]])
# Compute covariance matrix
cov_matrix = np.cov(X.T)
print(f"Covariance matrix:\n{cov_matrix.round(3)}")
# Eigendecomposition (this IS what PCA does)
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)
print(f"\nEigenvalues: {eigenvalues.round(3)}")
print(f"Eigenvectors:\n{eigenvectors.round(3)}")
print(f"\nVariance explained:")
total = eigenvalues.sum()
for i, ev in enumerate(sorted(eigenvalues, reverse=True)):
print(f" PC{i+1}: {ev/total:.1%}")
Singular Value Decomposition (SVD)
SVD decomposes any matrix into three components. It's used in dimensionality reduction, recommendations, and image compression.
Example
import numpy as np
# SVD: A = U * Σ * V^T
A = np.array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9],
[10, 11, 12]])
U, S, Vt = np.linalg.svd(A, full_matrices=False)
print(f"U shape: {U.shape}") # Left singular vectors
print(f"S (singular values): {S.round(3)}") # Singular values
print(f"Vt shape: {Vt.shape}") # Right singular vectors
# Reconstruct with fewer components (dimensionality reduction)
k = 2 # Keep only top 2 components
A_approx = U[:, :k] @ np.diag(S[:k]) @ Vt[:k, :]
error = np.linalg.norm(A - A_approx)
print(f"\nReconstruction error with k={k}: {error:.4f}")
Linear Algebra in Neural Networks
Example
import numpy as np
# A neural network layer IS a matrix multiplication + bias + activation
def neural_layer(X, W, b, activation='relu'):
"""Single neural network layer: z = XW + b, a = activation(z)"""
z = X @ W + b
if activation == 'relu':
return np.maximum(0, z)
elif activation == 'sigmoid':
return 1 / (1 + np.exp(-z))
return z
# Example: 2-layer neural network
X = np.random.randn(10, 4) # 10 samples, 4 features
W1 = np.random.randn(4, 8) * 0.1 # Layer 1: 4 -> 8 neurons
b1 = np.zeros(8)
W2 = np.random.randn(8, 2) * 0.1 # Layer 2: 8 -> 2 outputs
b2 = np.zeros(2)
# Forward pass
hidden = neural_layer(X, W1, b1, 'relu') # (10,4) @ (4,8) = (10,8)
output = neural_layer(hidden, W2, b2, 'sigmoid') # (10,8) @ (8,2) = (10,2)
print(f"Input shape: {X.shape}")
print(f"Hidden shape: {hidden.shape}")
print(f"Output shape: {output.shape}")
Key Concepts Summary
| Concept | ML Application |
|---|---|
| Dot product | Similarity, projections, linear models |
| Matrix multiply | Neural network layers, transformations |
| Transpose | Gradient computation, covariance |
| Inverse | Analytical solutions (normal equation) |
| Eigendecomposition | PCA, spectral clustering |
| SVD | Dimensionality reduction, recommendations |
| Norms | Regularization (L1, L2), distance |
Interview Questions
- What is the geometric interpretation of a dot product?
The dot product measures the projection of one vector onto another. It equals |a|*|b|*cos(θ), so it's related to the angle between vectors.
- Why is matrix multiplication the core of neural networks?
Each layer computes z = Wx + b, which is a linear transformation that combines inputs using learned weights. Stacking these with nonlinearities creates powerful function approximators.
- What is PCA doing mathematically?
PCA finds the eigenvectors of the covariance matrix and projects data onto the top-k eigenvectors (directions of maximum variance).
- When would you use SVD vs eigendecomposition?
SVD works on any matrix (even non-square), eigendecomposition requires square matrices. SVD is more numerically stable and is preferred in practice.
- What is the rank of a matrix and why does it matter in ML?
Rank is the number of linearly independent rows/columns. Low-rank matrices indicate redundancy in features. Matrix factorization exploits low-rank structure for compression and recommendations.
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Linear Algebra for Machine Learning.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Machine Learning topic.
Search Terms
machine-learning, machine learning, machine, learning, prerequisites, linear, algebra, for
Related Machine Learning Topics
Continue learning this concept
PrerequisitesCalculus for Machine LearningEssential calculus concepts for ML including derivatives, gradient descent, chain rule, partial derivatives, and optimization with practical Python implementations.PrerequisitesPython for Machine LearningEssential Python programming concepts for machine learning including NumPy, Pandas, data structures, functions, and OOP with practical ML-focused examples.PrerequisitesData Preprocessing BasicsIntroduction to data preprocessing for machine learning covering the importance of clean data, common preprocessing steps, and an overview of the preprocessing pipeline.PrerequisitesProbability Basics for Machine LearningCore probability concepts for ML including Bayes theorem, conditional probability, joint distributions, and their applications in classification and generative models.PrerequisitesStatistics Basics for Machine LearningEssential statistics concepts for ML including descriptive statistics, probability distributions, hypothesis testing, and correlation with Python examples.Interview PreparationStatistics for Machine Learning InterviewsEssential statistics concepts for ML interviews: hypothesis testing, distributions, sampling, Bayesian thinking, and probability theory.