ML Notes
HubAI vs ML vs Deep Learning — Machine LearningApplications of Machine LearningHistory of Machine LearningMachine Learning RoadmapWhat is Machine Learning?Calculus for Machine LearningData Preprocessing BasicsLinear Algebra for Machine LearningProbability Basics for Machine LearningPython for Machine LearningStatistics Basics for Machine LearningBias vs Variance TradeoffModel Evaluation OverviewOverfitting and UnderfittingReinforcement Learning Fundamentals — Machine LearningSupervised LearningTraining vs Testing DataTypes of Machine LearningUnsupervised LearningData Cleaning — Machine LearningData Collection for Machine LearningData Visualization for MLEncoding Categorical DataFeature EngineeringFeature ScalingHandling Missing ValuesTrain-Test SplitElastic Net RegressionLasso Regression (L1 Regularization)Multiple Linear RegressionPolynomial RegressionIntroduction to RegressionRegression ProjectsRidge Regression (L2 Regularization)Simple Linear RegressionIntroduction to ClassificationClassification ProjectsDecision TreesK-Nearest Neighbors (KNN)Logistic RegressionNaive Bayes ClassifierRandom Forest ClassificationSupport Vector Machine (SVM)XGBoost for ClassificationClustering in Machine Learning: Complete Guide to Unsupervised LearningClustering Projects: Real-World Applications and Implementation GuidesDBSCAN ClusteringGaussian Mixture Models (GMM): Probabilistic Clustering for Advanced Data AnalysisHierarchical ClusteringK-Means ClusteringDimensionality Reduction: Comprehensive Guide to Feature Reduction TechniquesFeature Selection Techniques: Choosing Informative Variables for ML ModelsLinear Discriminant Analysis (LDA): Supervised Dimensionality ReductionPrincipal Component Analysis (PCA)t-SNE: Non-Linear Dimensionality Reduction for Data VisualizationUMAP: Uniform Manifold Approximation and Projection for Modern Data VisualizationAdaBoost: Adaptive Boosting Algorithm Explained with ExamplesBagging: Bootstrap Aggregating for Reducing Model VarianceBoosting: Sequential Ensemble Learning for Bias and Variance ReductionEnsemble Learning: Complete Guide to Combining ML Models for Better PredictionsGradient Boosting: Building Powerful ML Models Through Sequential OptimizationRandom Forest: Complete Guide to Tree Ensembles and Feature ImportanceStacking: Advanced Ensemble Method Using Meta-Learners for Superior ML PredictionsActivation Functions in Neural NetworksBackpropagation: Complete Algorithm for Training Neural NetworksIntroduction to Deep Learning: Neural Networks, Architectures, and ApplicationsDeep Learning Projects: Hands-On Applications and Case StudiesNeural Networks Basics: Perceptron, Architecture, and TrainingPerceptron: Foundational Single-Neuron Classifier for Binary ClassificationPyTorch Basics: Building and Training Neural NetworksTensorFlow Basics: Building and Training Neural NetworksIntroduction to Natural Language Processing: Fundamentals and ApplicationsNLP Projects: Practical Applications from Sentiment Analysis to Machine TranslationStemming and Lemmatization: Reducing Words to Root Forms in NLPText Preprocessing: Essential Steps for NLP ApplicationsTF-IDF: Term Frequency-Inverse Document Frequency for Text Feature ExtractionTokenization in NLP: Breaking Text into Meaningful UnitsTransformers: The Architecture Behind Modern NLP ModelsWord Embeddings: Representing Words as Vectors for NLPIntroduction to Computer VisionComputer Vision ProjectsImage Classification: Building and Deploying CNN ModelsImage Processing BasicsImage Segmentation — Machine LearningObject DetectionOpenCV BasicsAgents and Environments in RLDeep Q-Networks (DQN)Policy Gradient Methods — Machine LearningQ-Learning AlgorithmReinforcement Learning: IntroductionReinforcement Learning ProjectsAccuracy, Precision, Recall: Classification Metrics Explained for ML InterviewConfusion Matrix: Complete Guide to Classification EvaluationCross-Validation: Comprehensive Guide to Robust Model EvaluationF1 Score: Balanced Classification Metric for Imbalanced DataGrid Search: Complete Guide to Hyperparameter Tuning in Machine LearningHyperparameter Tuning: Complete Guide to Optimizing Machine Learning ModelsModel Selection: Choosing the Best Algorithm for Your ML ProblemROC and AUC: Complete Guide to Classification Threshold EvaluationBuild an AI Assistant with LLMsBuild a Conversational ChatbotCustomer Churn PredictionFace Recognition System — Machine LearningHouse Price PredictionRecommendation System ProjectSentiment Analysis Project — Machine LearningSpam Email ClassifierML Case Study Questions for InterviewsML Coding Questions for Technical InterviewsMachine Learning Interview CheatsheetEssential Machine Learning Interview QuestionsStatistics for Machine Learning InterviewsDatasets for ML PracticeImportant ML FormulasMachine Learning GlossaryRecommended ML Books and CoursesCI/CD for Machine LearningDocker for Machine LearningFastAPI for Machine Learning APIsFlask API for Machine LearningMLOps: IntroductionModel Deployment StrategiesML Model Monitoring & Observability
Core probability concepts for ML including Bayes theorem, conditional probability, joint distributions, and their applications in classification and generative models.
Probability theory is the language of uncertainty — and machine learning is fundamentally about making predictions under uncertainty. From Naive Bayes classifiers to Bayesian neural networks, probability is woven into the fabric of ML.
Core Probability Concepts
python example
import numpy as np
# Basic probability: P(event) = favorable outcomes / total outcomes
# Example: Rolling a die
total_outcomes = 6
p_even = 3 / total_outcomes # P(even) = P(2,4,6) = 3/6
print(f"P(even number on die): {p_even:.4f}")
# Complement: P(not A) = 1 - P(A)
p_odd = 1 - p_even
print(f"P(odd number): {p_odd:.4f}")
# Union: P(A or B) = P(A) + P(B) - P(A and B)
p_even_or_gt4 = 3/6 + 2/6 - 1/6 # Even: {2,4,6}, >4: {5,6}, Both: {6}
print(f"P(even OR >4): {p_even_or_gt4:.4f}")Conditional Probability
python example
# P(A|B) = P(A and B) / P(B)
# "Probability of A given that B happened"
# Example: Email classification
# P(spam) = 0.3, P(contains "free" | spam) = 0.8, P(contains "free" | not spam) = 0.1
p_spam = 0.3
p_free_given_spam = 0.8
p_free_given_not_spam = 0.1
# P(free) using total probability
p_free = p_free_given_spam * p_spam + p_free_given_not_spam * (1 - p_spam)
print(f"P(email contains 'free'): {p_free:.4f}")
# Bayes' Theorem: P(spam | free) = P(free | spam) * P(spam) / P(free)
p_spam_given_free = (p_free_given_spam * p_spam) / p_free
print(f"P(spam | contains 'free'): {p_spam_given_free:.4f}")
print("If an email contains 'free', there's a 77% chance it's spam!")Bayes' Theorem
Example
P(B|A) * P(A)
P(A|B) = ─────────────────
P(B)
Where:
P(A|B) = Posterior (what we want to know)
P(B|A) = Likelihood (data given hypothesis)
P(A) = Prior (initial belief)
P(B) = Evidence (normalizing constant)
Bayes in Action: Medical Diagnosis
python example
# Disease testing scenario
# P(disease) = 0.001 (rare disease)
# P(positive test | disease) = 0.99 (sensitivity)
# P(positive test | no disease) = 0.05 (false positive rate)
p_disease = 0.001
p_pos_given_disease = 0.99
p_pos_given_no_disease = 0.05
# P(positive)
p_positive = p_pos_given_disease * p_disease + p_pos_given_no_disease * (1 - p_disease)
# P(disease | positive test)
p_disease_given_pos = (p_pos_given_disease * p_disease) / p_positive
print("Medical Test Scenario:")
print(f" Disease prevalence: {p_disease:.3%}")
print(f" Test sensitivity: {p_pos_given_disease:.1%}")
print(f" False positive rate: {p_pos_given_no_disease:.1%}")
print(f"\n P(disease | positive test): {p_disease_given_pos:.2%}")
print(" Even with a 'positive' test, only ~2% chance of disease!")
print(" This is why rare disease screening needs confirmation tests.")Probability Distributions in ML
Example
from scipy import stats
import numpy as np
# Distributions used in ML models
print("=== Key Distributions for ML ===\n")
# 1. Bernoulli: Binary classification output
print("Bernoulli(p=0.7) - Binary classification:")
print(f" P(y=1) = 0.7, P(y=0) = 0.3\n")
# 2. Gaussian: Continuous features, noise modeling
print("Gaussian(μ=0, σ=1) - Feature distributions:")
samples = np.random.normal(0, 1, 10000)
print(f" 68% within 1σ: {np.mean(np.abs(samples) < 1):.1%}")
print(f" 95% within 2σ: {np.mean(np.abs(samples) < 2):.1%}\n")
# 3. Multinomial: Multi-class classification
print("Softmax output (multinomial):")
logits = np.array([2.0, 1.0, 0.5])
probabilities = np.exp(logits) / np.exp(logits).sum()
print(f" Classes: {probabilities.round(3)} (sum = {probabilities.sum():.1f})")
Maximum Likelihood Estimation
python example
import numpy as np
from scipy.optimize import minimize_scalar
# MLE: Find parameters that maximize P(data | parameters)
# Example: Estimating mean of normal distribution
np.random.seed(42)
data = np.random.normal(loc=5.0, scale=2.0, size=100)
# Log-likelihood for normal distribution
def neg_log_likelihood(mu, data=data, sigma=2.0):
n = len(data)
ll = -n/2 * np.log(2*np.pi*sigma**2) - np.sum((data - mu)**2) / (2*sigma**2)
return -ll # Negative because we minimize
result = minimize_scalar(neg_log_likelihood, bounds=(0, 10), method='bounded')
print(f"True mean: 5.0")
print(f"MLE estimate: {result.x:.4f}")
print(f"Sample mean: {data.mean():.4f}")
print("MLE for Gaussian mean = sample mean (they're the same!)")Joint and Marginal Probability
Example
import numpy as np
import pandas as pd
# Joint probability table
# Weather vs. Playing tennis
joint_prob = pd.DataFrame(
[[0.2, 0.1], # Sunny: P(play), P(not play)
[0.15, 0.05], # Cloudy
[0.05, 0.45]], # Rainy
columns=['Play', 'Not Play'],
index=['Sunny', 'Cloudy', 'Rainy']
)
print("Joint Probability Table:")
print(joint_prob)
print(f"\nSum of all probabilities: {joint_prob.values.sum():.2f}")
# Marginal probabilities (sum over one variable)
p_weather = joint_prob.sum(axis=1)
p_play = joint_prob.sum(axis=0)
print(f"\nMarginal P(weather):\n{p_weather}")
print(f"\nMarginal P(play):\n{p_play}")
# Conditional probability
p_play_given_sunny = joint_prob.loc['Sunny', 'Play'] / p_weather['Sunny']
print(f"\nP(Play | Sunny) = {p_play_given_sunny:.4f}")
Interview Questions
- Explain Bayes' theorem in simple terms.
Bayes' theorem updates our initial belief (prior) about something after seeing new evidence (data). It combines what we knew before with what the data tells us to form an updated belief (posterior).
- Why is the base rate important in probability?
Even a highly accurate test can give misleading results when the base rate is very low (rare events). This is the base rate fallacy — you must consider prevalence when interpreting test results.
- What is the difference between likelihood and probability?
Probability is P(data | parameters) — given known parameters, how likely is the data. Likelihood is the same function but viewed as a function of parameters for fixed data — what parameters best explain the data.
- How does Naive Bayes use probability for classification?
It applies Bayes' theorem assuming features are independent given the class. P(class | features) ∝ P(class) × ∏P(feature_i | class). Despite the naive assumption, it works surprisingly well.
- What is the difference between frequentist and Bayesian approaches?
Frequentists treat parameters as fixed and data as random (probability = long-run frequency). Bayesians treat parameters as random with distributions and update beliefs with data (probability = degree of belief).
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Probability Basics for Machine Learning.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Machine Learning topic.
Search Terms
machine-learning, machine learning, machine, learning, prerequisites, probability, basics, probability basics for machine learning
Related Machine Learning Topics
Continue learning this concept
PrerequisitesData Preprocessing BasicsIntroduction to data preprocessing for machine learning covering the importance of clean data, common preprocessing steps, and an overview of the preprocessing pipeline.PrerequisitesStatistics Basics for Machine LearningEssential statistics concepts for ML including descriptive statistics, probability distributions, hypothesis testing, and correlation with Python examples.PrerequisitesCalculus for Machine LearningEssential calculus concepts for ML including derivatives, gradient descent, chain rule, partial derivatives, and optimization with practical Python implementations.PrerequisitesLinear Algebra for Machine LearningEssential linear algebra concepts for ML including vectors, matrices, eigenvalues, and their applications in algorithms like PCA, SVD, and neural networks.PrerequisitesPython for Machine LearningEssential Python programming concepts for machine learning including NumPy, Pandas, data structures, functions, and OOP with practical ML-focused examples.Computer VisionImage Processing BasicsEssential image processing fundamentals covering pixel operations, filtering, edge detection, morphological operations, and color space transformations with OpenCV.