ML Notes

Statistics Basics for Machine Learning

Essential statistics concepts for ML including descriptive statistics, probability distributions, hypothesis testing, and correlation with Python examples.

Statistics provides the mathematical foundation for understanding data and making informed decisions about model performance. Every ML practitioner needs these concepts to properly analyze datasets, evaluate models, and communicate results.

Descriptive Statistics

Example

import numpy as np import pandas as pd # Sample dataset np.random.seed(42) data = np.random.normal(loc=50, scale=15, size=1000) # Measures of Central Tendency print("=== Central Tendency ===") print(f"Mean: {np.mean(data):.2f}") print(f"Median: {np.median(data):.2f}") from scipy import stats print(f"Mode: {stats.mode(data.round(0), keepdims=True).mode[0]:.0f}") # Measures of Spread print("\n=== Spread/Dispersion ===") print(f"Variance: {np.var(data):.2f}") print(f"Std Deviation: {np.std(data):.2f}") print(f"Range: {np.ptp(data):.2f}") print(f"IQR: {np.percentile(data, 75) - np.percentile(data, 25):.2f}") # Shape print("\n=== Shape ===") from scipy.stats import skew, kurtosis print(f"Skewness: {skew(data):.4f} (0 = symmetric)") print(f"Kurtosis: {kurtosis(data):.4f} (0 = normal)")

Probability Distributions

python example

import numpy as np
from scipy import stats

# Normal (Gaussian) Distribution - most common in ML
normal = stats.norm(loc=0, scale=1)
print("Normal Distribution (μ=0, σ=1):")
print(f"  P(X < 0):    {normal.cdf(0):.4f}")
print(f"  P(-1 < X < 1): {normal.cdf(1) - normal.cdf(-1):.4f}")  # ~68%
print(f"  P(-2 < X < 2): {normal.cdf(2) - normal.cdf(-2):.4f}")  # ~95%

# Bernoulli - binary outcomes (classification)
p_spam = 0.3
print(f"\nBernoulli(p=0.3): P(spam) = {p_spam}, P(not spam) = {1-p_spam}")

# Binomial - number of successes in n trials
binom = stats.binom(n=100, p=0.3)
print(f"\nBinomial(n=100, p=0.3):")
print(f"  Expected successes: {binom.mean():.1f}")
print(f"  P(exactly 30): {binom.pmf(30):.4f}")

# Poisson - rare events
poisson = stats.poisson(mu=5)
print(f"\nPoisson(λ=5): P(X=3) = {poisson.pmf(3):.4f}")

Correlation and Covariance

python example

import numpy as np
import pandas as pd

# Generate correlated features
np.random.seed(42)
n = 200
age = np.random.normal(35, 10, n)
income = age * 1500 + np.random.normal(0, 5000, n)  # Correlated
height = np.random.normal(170, 10, n)  # Not correlated

df = pd.DataFrame({'age': age, 'income': income, 'height': height})

# Correlation matrix
print("Correlation Matrix:")
print(df.corr().round(3))
print("\nInterpretation:")
print("  age-income: Strong positive (older → higher income)")
print("  age-height: Near zero (no relationship)")

# Pearson vs Spearman correlation
from scipy.stats import pearsonr, spearmanr
r_pearson, p_val = pearsonr(age, income)
r_spearman, _ = spearmanr(age, income)
print(f"\nPearson r: {r_pearson:.4f} (linear relationships)")
print(f"Spearman r: {r_spearman:.4f} (monotonic relationships)")

Hypothesis Testing

Example

from scipy import stats import numpy as np # A/B Test: Did our new ML model improve conversion rate? np.random.seed(42) group_a = np.random.binomial(1, 0.10, 1000) # Control: 10% conversion group_b = np.random.binomial(1, 0.12, 1000) # Treatment: 12% conversion # Two-proportion z-test from statsmodels.stats.proportion import proportions_ztest count = np.array([group_a.sum(), group_b.sum()]) nobs = np.array([len(group_a), len(group_b)]) z_stat, p_value = proportions_ztest(count, nobs) print("A/B Test Results:") print(f" Group A conversion: {group_a.mean():.1%}") print(f" Group B conversion: {group_b.mean():.1%}") print(f" Z-statistic: {z_stat:.4f}") print(f" P-value: {p_value:.4f}") print(f" Significant (α=0.05)? {'Yes' if p_value < 0.05 else 'No'}")

Central Limit Theorem

Example

import numpy as np # CLT: Sample means follow a normal distribution regardless of population np.random.seed(42) # Non-normal population (exponential) population = np.random.exponential(scale=2, size=100000) print(f"Population mean: {population.mean():.3f}, skewness: {skew(population):.3f}") # Take many sample means sample_means = [np.random.choice(population, size=30).mean() for _ in range(1000)] sample_means = np.array(sample_means) print(f"Sample means: mean={sample_means.mean():.3f}, std={sample_means.std():.3f}") print(f"Sample means skewness: {skew(sample_means):.3f} (much closer to 0!)") print("CLT: Sample means are approximately normal even from non-normal data")

Interview Questions

What is the difference between population and sample statistics?

Population statistics describe the entire group (μ, σ). Sample statistics (x̄, s) estimate population parameters from a subset. We use samples because measuring entire populations is usually impractical.

When would you use median instead of mean?

When data has outliers or is heavily skewed. Median is robust to extreme values (income data, house prices). Mean is sensitive to outliers.

What does a p-value actually mean?

The probability of observing results at least as extreme as the data, assuming the null hypothesis is true. It does NOT mean the probability that the hypothesis is true.

Why is the Central Limit Theorem important for ML?

It justifies using normal distribution assumptions for confidence intervals and hypothesis tests even when the underlying data isn't normal, as long as sample sizes are sufficient.

What is the difference between correlation and causation?

Correlation measures statistical association between variables. Causation means one variable directly influences another. Correlation does not imply causation — there may be confounders.

Practical Applications in Machine Learning

Understanding statistics isn't just academic — these concepts directly impact your ML workflow:

Feature Selection: Correlation analysis helps identify redundant features. If two features have correlation > 0.95, one is likely redundant. Statistical tests (chi-square for categorical, ANOVA for numerical vs categorical) quantify feature-target relationships.

Model Evaluation: Confidence intervals around accuracy scores tell you whether differences between models are statistically significant or just noise. A model with 85% ± 3% accuracy isn't meaningfully better than one with 84% ± 3%.

A/B Testing: When deploying ML models in production, hypothesis testing determines whether the new model genuinely outperforms the baseline. Without proper statistical testing, you might deploy a model that only appeared better due to random variation.

Outlier Detection: Z-scores and IQR methods use distributional properties to identify anomalous data points. Understanding when data is normally distributed (and when it isn't) determines which detection method is appropriate.

Common Statistical Misconceptions

Misconception 1: Large p-value means no effect exists. A large p-value means we failed to detect an effect with our sample size. The effect might exist but be too small to detect — this is a power issue, not proof of absence.

Misconception 2: Correlation of 0.7 is "strong." Context matters enormously. In physics, 0.7 might be weak. In social science, 0.7 is excellent. Always interpret correlation relative to your domain's standards.

Misconception 3: Normal distribution is required for ML. Many ML algorithms (decision trees, neural networks) make no distributional assumptions about features. Normality matters most for linear regression residuals, parametric statistical tests, and algorithms using Mahalanobis distance.

Misconception 4: More data always helps. While more data generally improves estimates, biased data doesn't become less biased with more samples. If your sampling method is flawed, additional data amplifies the bias rather than correcting it.

Quick Reference: When to Use Which Test

Scenario	Test
Compare two group means	t-test (independent or paired)
Compare 3+ group means	ANOVA
Test categorical associations	Chi-square
Correlation significance	Pearson's r test
Non-normal data comparison	Mann-Whitney U / Wilcoxon
Normality check	Shapiro-Wilk test

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Statistics Basics for Machine Learning.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Machine Learning topic.

Search Terms

machine-learning, machine learning, machine, learning, prerequisites, statistics, basics, statistics basics for machine learning

Continue learning this concept

PrerequisitesData Preprocessing BasicsIntroduction to data preprocessing for machine learning covering the importance of clean data, common preprocessing steps, and an overview of the preprocessing pipeline.PrerequisitesProbability Basics for Machine LearningCore probability concepts for ML including Bayes theorem, conditional probability, joint distributions, and their applications in classification and generative models.PrerequisitesCalculus for Machine LearningEssential calculus concepts for ML including derivatives, gradient descent, chain rule, partial derivatives, and optimization with practical Python implementations.PrerequisitesLinear Algebra for Machine LearningEssential linear algebra concepts for ML including vectors, matrices, eigenvalues, and their applications in algorithms like PCA, SVD, and neural networks.PrerequisitesPython for Machine LearningEssential Python programming concepts for machine learning including NumPy, Pandas, data structures, functions, and OOP with practical ML-focused examples.Computer VisionImage Processing BasicsEssential image processing fundamentals covering pixel operations, filtering, edge detection, morphological operations, and color space transformations with OpenCV.

PreviousPython for Machine Learning NextBias vs Variance Tradeoff

ML Notes

Statistics Basics for Machine Learning

Essential statistics concepts for ML including descriptive statistics, probability distributions, hypothesis testing, and correlation with Python examples.

Descriptive Statistics

Example

Probability Distributions

python example

import numpy as np
from scipy import stats

# Normal (Gaussian) Distribution - most common in ML
normal = stats.norm(loc=0, scale=1)
print("Normal Distribution (μ=0, σ=1):")
print(f"  P(X < 0):    {normal.cdf(0):.4f}")
print(f"  P(-1 < X < 1): {normal.cdf(1) - normal.cdf(-1):.4f}")  # ~68%
print(f"  P(-2 < X < 2): {normal.cdf(2) - normal.cdf(-2):.4f}")  # ~95%

# Bernoulli - binary outcomes (classification)
p_spam = 0.3
print(f"\nBernoulli(p=0.3): P(spam) = {p_spam}, P(not spam) = {1-p_spam}")

# Binomial - number of successes in n trials
binom = stats.binom(n=100, p=0.3)
print(f"\nBinomial(n=100, p=0.3):")
print(f"  Expected successes: {binom.mean():.1f}")
print(f"  P(exactly 30): {binom.pmf(30):.4f}")

# Poisson - rare events
poisson = stats.poisson(mu=5)
print(f"\nPoisson(λ=5): P(X=3) = {poisson.pmf(3):.4f}")

Correlation and Covariance

python example

import numpy as np
import pandas as pd

# Generate correlated features
np.random.seed(42)
n = 200
age = np.random.normal(35, 10, n)
income = age * 1500 + np.random.normal(0, 5000, n)  # Correlated
height = np.random.normal(170, 10, n)  # Not correlated

df = pd.DataFrame({'age': age, 'income': income, 'height': height})

# Correlation matrix
print("Correlation Matrix:")
print(df.corr().round(3))
print("\nInterpretation:")
print("  age-income: Strong positive (older → higher income)")
print("  age-height: Near zero (no relationship)")

# Pearson vs Spearman correlation
from scipy.stats import pearsonr, spearmanr
r_pearson, p_val = pearsonr(age, income)
r_spearman, _ = spearmanr(age, income)
print(f"\nPearson r: {r_pearson:.4f} (linear relationships)")
print(f"Spearman r: {r_spearman:.4f} (monotonic relationships)")

Hypothesis Testing

Example

Central Limit Theorem

Example

Interview Questions

What is the difference between population and sample statistics?

When would you use median instead of mean?

When data has outliers or is heavily skewed. Median is robust to extreme values (income data, house prices). Mean is sensitive to outliers.

What does a p-value actually mean?

The probability of observing results at least as extreme as the data, assuming the null hypothesis is true. It does NOT mean the probability that the hypothesis is true.

Why is the Central Limit Theorem important for ML?

It justifies using normal distribution assumptions for confidence intervals and hypothesis tests even when the underlying data isn't normal, as long as sample sizes are sufficient.

What is the difference between correlation and causation?

Correlation measures statistical association between variables. Causation means one variable directly influences another. Correlation does not imply causation — there may be confounders.

Practical Applications in Machine Learning

Understanding statistics isn't just academic — these concepts directly impact your ML workflow:

Common Statistical Misconceptions

Quick Reference: When to Use Which Test

Scenario	Test
Compare two group means	t-test (independent or paired)
Compare 3+ group means	ANOVA
Test categorical associations	Chi-square
Correlation significance	Pearson's r test
Non-normal data comparison	Mann-Whitney U / Wilcoxon
Normality check	Shapiro-Wilk test

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Statistics Basics for Machine Learning.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Machine Learning topic.

Search Terms

machine-learning, machine learning, machine, learning, prerequisites, statistics, basics, statistics basics for machine learning

Continue learning this concept

PreviousPython for Machine Learning NextBias vs Variance Tradeoff