Essential probability and statistics concepts for deep learning including distributions, Bayes theorem, maximum likelihood, and information theory.
Probability theory is not merely a prerequisite for deep learning — it is the language in which deep learning is written. When a classifier outputs [0.7, 0.2, 0.1], those are probabilities. When you minimize cross-entropy loss, you are doing maximum likelihood estimation. When you apply dropout, you are performing approximate Bayesian inference. When you train a VAE or diffusion model, you are learning probability distributions. Understanding this connection transforms deep learning from a bag of tricks into a coherent mathematical framework.
Why Probability in Deep Learning?
Let us map the connections explicitly:
- Output interpretation: Softmax converts raw logits into a valid probability distribution over classes
- Loss functions: Cross-entropy loss is the negative log-likelihood — minimizing it maximizes the probability of observing the training data
- Regularization: L2 weight decay is equivalent to placing a Gaussian prior on weights; dropout samples from a Bernoulli distribution
- Generative models: VAEs model latent variable distributions; GANs learn to sample from data distributions; diffusion models reverse a noise process
- Uncertainty: Bayesian neural networks estimate not just predictions but confidence intervals
- Initialization: Weights are drawn from carefully chosen distributions (Gaussian, uniform)
Probability Fundamentals
A probability distribution assigns a number between 0 and 1 to every possible outcome, with all probabilities summing (or integrating) to 1. This is exactly what softmax ensures for neural network outputs.
import numpy as np
# A discrete probability distribution must satisfy two properties:
# 1. Every probability is non-negative: P(x) >= 0
# 2. They sum to one: Σ P(x) = 1
# Softmax creates a valid probability distribution from ANY real-valued vector
def softmax(logits):
"""Convert arbitrary real numbers to a probability distribution."""
# Subtract max for numerical stability (doesn't change the result)
exp_logits = np.exp(logits - np.max(logits))
return exp_logits / exp_logits.sum()
# Example: a neural network outputs raw scores (logits)
logits = np.array([2.0, 1.0, 0.5, -1.0])
probs = softmax(logits)
print(f"Probabilities: {probs}") # [0.506, 0.186, 0.113, 0.025, ...]
print(f"Sum: {probs.sum():.6f}") # 1.000000
print(f"All non-negative: {(probs >= 0).all()}") # True
# Joint, marginal, and conditional probability
# P(A, B) = P(A|B) * P(B) = P(B|A) * P(A)
# This decomposition is fundamental to understanding generative models
Key Probability Distributions
Each distribution appears in specific contexts within deep learning. Knowing which distribution models what phenomenon is essential.
| Distribution | Type | Parameters | Deep Learning Application |
|---|
| Bernoulli | Discrete | p (probability) | Dropout masks, binary classification |
| Categorical | Discrete | p₁, p₂, ..., pₖ | Multi-class classification output |
| Gaussian (Normal) | Continuous | μ (mean), σ (std) | Weight initialization, VAE latent space |
| Uniform | Continuous | a (min), b (max) | Xavier initialization bounds |
import numpy as np
# --- Bernoulli Distribution ---
# Models binary outcomes: "on" with probability p, "off" with probability 1-p
# Used in dropout: each neuron is randomly "dropped" with probability p
def dropout(x, p=0.5, training=True):
"""Apply dropout during training.
Each element is zeroed with probability p.
Remaining elements are scaled by 1/(1-p) to maintain expected values.
"""
if not training:
return x # No dropout at test time
# Sample Bernoulli mask: 1 with prob (1-p), 0 with prob p
mask = np.random.binomial(1, 1 - p, size=x.shape)
return x * mask / (1 - p) # Inverted dropout scaling
# Demonstration
hidden_layer = np.ones(10) # All ones for clarity
print("Original:", hidden_layer)
print("After dropout (p=0.5):", dropout(hidden_layer, p=0.5))
# About half the values become 0, others are doubled
# --- Gaussian (Normal) Distribution ---
# The most important continuous distribution for deep learning
# Weight initialization draws from Gaussians with carefully chosen variance
# Standard normal: mean=0, std=1
z = np.random.randn(1000) # 1000 samples from N(0, 1)
# He initialization for ReLU networks: N(0, sqrt(2/fan_in))
fan_in = 512
weights = np.random.randn(fan_in, 256) * np.sqrt(2.0 / fan_in)
print(f"Weight mean: {weights.mean():.4f}") # ~0
print(f"Weight std: {weights.std():.4f}") # ~sqrt(2/512) ≈ 0.0625
# --- Gaussian in VAEs ---
# The encoder outputs μ and σ, defining a Gaussian distribution
# We sample z from N(μ, σ²) using the reparameterization trick:
mu = np.array([0.5, -0.3, 1.2]) # learned mean
log_var = np.array([-1.0, 0.5, -0.5]) # learned log-variance
sigma = np.exp(0.5 * log_var) # convert to std
epsilon = np.random.randn(3) # sample from N(0,1)
z = mu + sigma * epsilon # reparameterization trick
# This allows gradients to flow through the sampling operation
Bayes' Theorem
Bayes' theorem is the foundation of probabilistic reasoning — it tells us how to update our beliefs when we observe new evidence.
P(hypothesis | data) = P(data | hypothesis) × P(hypothesis) / P(data)
posterior = likelihood × prior / evidence
In deep learning, Bayesian thinking appears in several places:
# Connection 1: L2 regularization IS a Gaussian prior on weights
#
# Without regularization, we maximize likelihood: argmax P(data | weights)
# With L2 regularization, we maximize posterior: argmax P(weights | data)
# = argmax P(data | weights) × P(weights)
#
# If P(weights) = Gaussian(0, σ²), then:
# log P(weights) = -||w||² / (2σ²) + constant
# This is exactly the L2 penalty term!
#
# So: loss = -log P(data|w) - log P(w) = cross_entropy + λ * ||w||²
lambda_reg = 0.01 # Corresponds to prior variance σ² = 1/(2λ)
data_loss = 2.5 # Cross-entropy on training batch
l2_penalty = lambda_reg * np.sum(weights**2)
total_loss = data_loss + l2_penalty
# Connection 2: L1 regularization IS a Laplace prior (encourages sparsity)
# P(w) = Laplace(0, b) → log P(w) = -|w|/b
# This pushes weights to exactly zero, creating sparse networks
# Connection 3: Bayesian Neural Networks
# Instead of learning point estimates for weights, learn distributions:
# Each weight w ~ N(μ_w, σ_w²)
# This gives us uncertainty estimates for free
Maximum Likelihood Estimation (MLE)
Most loss functions in deep learning are derived from MLE. The idea: find parameters that maximize the probability of observing the training data.
# For classification: the likelihood of class label y given input x is P(y|x; θ)
# The negative log-likelihood for N samples is:
# NLL = -Σ log P(y_i | x_i; θ)
# This IS cross-entropy loss!
def cross_entropy_loss(predictions, targets):
"""Negative log-likelihood for classification.
predictions: (N, C) array of predicted probabilities
targets: (N,) array of true class indices
"""
N = len(targets)
# Select predicted probability for the correct class
correct_class_probs = predictions[np.arange(N), targets]
# Negative log-likelihood
nll = -np.log(correct_class_probs + 1e-9)
return np.mean(nll)
# Example
predictions = np.array([
[0.7, 0.2, 0.1], # model thinks class 0 (correct!)
[0.1, 0.8, 0.1], # model thinks class 1 (correct!)
[0.3, 0.4, 0.3], # model thinks class 1 (wrong! true is 2)
])
targets = np.array([0, 1, 2])
loss = cross_entropy_loss(predictions, targets)
print(f"Cross-entropy loss: {loss:.4f}") # Lower = better predictions
# For regression: assuming Gaussian noise, MLE gives MSE loss
# If y = f(x; θ) + ε, where ε ~ N(0, σ²), then:
# -log P(y | x; θ) = (y - f(x;θ))² / (2σ²) + constant
# Minimizing this = minimizing MSE!
def mse_loss(predictions, targets):
"""MSE loss — equivalent to MLE with Gaussian noise assumption."""
return np.mean((predictions - targets)**2)
Information theory provides the mathematical foundation for understanding what cross-entropy and KL divergence actually measure.
Entropy H(p) measures the average "surprise" or uncertainty in a distribution. A distribution concentrated on one class has low entropy (we are very certain); a uniform distribution has maximum entropy (maximum uncertainty).
# Entropy: H(p) = -Σ p(x) * log₂(p(x))
def entropy(probs):
"""Compute Shannon entropy in bits."""
# Filter out zeros to avoid log(0)
probs = probs[probs > 0]
return -np.sum(probs * np.log2(probs))
# High certainty = low entropy
certain = np.array([0.99, 0.005, 0.005])
print(f"Certain prediction entropy: {entropy(certain):.3f} bits") # ~0.09
# Maximum uncertainty = maximum entropy
uniform = np.array([1/3, 1/3, 1/3])
print(f"Uniform prediction entropy: {entropy(uniform):.3f} bits") # ~1.58
# KL Divergence: measures "distance" between two distributions
# D_KL(P || Q) = Σ P(x) * log(P(x) / Q(x))
# NOT symmetric: D_KL(P||Q) ≠ D_KL(Q||P)
def kl_divergence(p, q):
"""KL divergence from p to q."""
# Filter where p > 0 (convention: 0 * log(0/q) = 0)
mask = p > 0
return np.sum(p[mask] * np.log(p[mask] / (q[mask] + 1e-9)))
# Cross-entropy = Entropy + KL Divergence
# H(P, Q) = H(P) + D_KL(P || Q)
# Since H(P) is constant during training, minimizing cross-entropy
# is EQUIVALENT to minimizing KL divergence from P to Q
# i.e., making our model distribution Q match the true distribution P
true_dist = np.array([1.0, 0.0, 0.0]) # One-hot (true class is 0)
model_dist = np.array([0.7, 0.2, 0.1]) # Model's prediction
print(f"KL divergence: {kl_divergence(true_dist, model_dist):.4f}")
KL Divergence in VAEs: The VAE loss has two terms: reconstruction loss (how well can we reconstruct the input?) plus KL divergence between the learned latent distribution and a standard Gaussian. This KL term acts as a regularizer, ensuring the latent space is smooth and continuous.
# VAE KL divergence (closed form for two Gaussians):
# D_KL(N(μ, σ²) || N(0, 1)) = -0.5 * Σ(1 + log(σ²) - μ² - σ²)
def vae_kl_loss(mu, log_var):
"""KL divergence between learned Gaussian and standard normal."""
return -0.5 * np.sum(1 + log_var - mu**2 - np.exp(log_var))
mu = np.array([0.5, -0.3, 1.2])
log_var = np.array([-1.0, 0.5, -0.5])
kl = vae_kl_loss(mu, log_var)
print(f"VAE KL loss: {kl:.4f}")
Expectation and Variance
These summary statistics appear everywhere in deep learning — batch normalization computes running means and variances, loss functions compute expected values over batches, and uncertainty estimation relies on variance.
# Expectation (mean): E[X] = Σ x * P(x)
# For a dataset, this is simply the arithmetic mean
data = np.random.randn(1000)
print(f"Sample mean (≈ E[X]): {data.mean():.4f}")
# Variance: Var(X) = E[(X - E[X])²] = E[X²] - (E[X])²
# Measures spread/uncertainty
print(f"Sample variance: {data.var():.4f}")
# In batch normalization:
# For each feature: normalize using batch mean and variance
# x_normalized = (x - E[x]) / sqrt(Var(x) + ε)
batch = np.random.randn(32, 128) # 32 samples, 128 features
batch_mean = batch.mean(axis=0) # (128,) mean per feature
batch_var = batch.var(axis=0) # (128,) variance per feature
normalized = (batch - batch_mean) / np.sqrt(batch_var + 1e-5)
Interview Questions
- Why does minimizing cross-entropy train good classifiers?
Cross-entropy is the negative log-likelihood. Minimizing it is equivalent to maximum likelihood estimation — finding the model parameters that assign the highest probability to the correct labels in the training data. It is also equivalent to minimizing KL divergence between the true distribution and the model's predictions.
- How is dropout related to probability?
Dropout samples a Bernoulli mask each forward pass, randomly zeroing neurons with probability *p*. This can be interpreted as approximate Bayesian inference — at test time, using all neurons with scaled weights approximates the mean of an ensemble of exponentially many sub-networks.
- What is the connection between L2 regularization and Gaussian priors?
L2 regularization adds λ||w||² to the loss, which is mathematically equivalent to MAP (Maximum A Posteriori) estimation with a zero-mean Gaussian prior on weights: P(w) = N(0, 1/(2λ)). It expresses the prior belief that weights should be small.
- Explain KL divergence and its role in VAEs.
KL divergence D_KL(P||Q) measures how much distribution P differs from Q (in units of information). In VAEs, it regularizes the encoder's output distribution to stay close to a standard Gaussian N(0,1), ensuring the latent space is smooth, continuous, and easy to sample from during generation.
- Why is the reparameterization trick needed in VAEs?
Sampling z ~ N(μ, σ²) is a stochastic operation with no gradient. The trick rewrites it as z = μ + σ * ε where ε ~ N(0,1), making the randomness independent of the parameters. Gradients can now flow through μ and σ during backpropagation.