Important AI Formulas

Introduction

Mathematics is the language of artificial intelligence. Behind every neural network prediction, every search algorithm decision, and every probabilistic inference lies a mathematical formula that precisely defines the computation. This reference collects the most important formulas across AI subfields, explains what each one means intuitively, and shows when and how to apply them. Rather than memorizing these formulas in isolation, focus on understanding the reasoning behind each one.

Probability and Statistics Foundations

Bayes' Theorem

P(A\|B) = posterior	probability of A given evidence B
P(B\|A) = likelihood	probability of observing B if A is true
P(A) = prior	initial belief about A before seeing B
P(B) = evidence	total probability of observing B
A = "patient has disease" (prior: 1% prevalence	P(A) = 0.01)
B = "test is positive" (test accuracy	95%)

Information Theory

Entropy	H(X) = -Σ P(x_i) × log₂(P(x_i))
Fair coin	H = -0.5×log₂(0.5) - 0.5×log₂(0.5) = 1 bit
Biased coin (90/10)	H = -0.9×log₂(0.9) - 0.1×log₂(0.1) = 0.47 bits
Cross-Entropy	H(P,Q) = -Σ P(x_i) × log(Q(x_i))
KL Divergence	D_KL(P\|\|Q) = Σ P(x_i) × log(P(x_i)/Q(x_i))

Linear Algebra for AI

Dot Product	a·b = Σ a_i × b_i = \|a\|\|b\|cos(θ)
Used in	attention mechanisms, similarity, neural network layers
Matrix Multiplication	(AB)_ij = Σ_k A_ik × B_kj
Used in	every neural network forward pass
Cosine Similarity	cos(θ) = (a·b) / (\|a\| × \|b\|)
Range	[-1, 1], where 1 = identical direction
Used in	document similarity, recommendation systems, embedding comparison
Softmax	softmax(x_i) = e^(x_i) / Σ_j e^(x_j)
Used in	classification output layers, attention weights

Neural Network Formulas

Neuron output	y = f(Σ w_i × x_i + b) = f(W·x + b)
MSE (regression)	L = (1/n) Σ (ŷ_i - y_i)²
Cross-entropy (classification)	L = -(1/n) Σ [y_i log(ŷ_i) + (1-y_i)log(1-ŷ_i)]
Gradient Descent	θ_{t+1} = θ_t - η × ∂L/∂θ
Dropout (training)	y = x × Bernoulli(1-p) / (1-p)

Search Algorithm Formulas

A* Evaluation	f(n) = g(n) + h(n)
Optimal when h is admissible	h(n) ≤ h*(n) for all n
Manhattan Distance	h(n) = \|x₁-x₂\| + \|y₁-y₂\|
Euclidean Distance	h(n) = √((x₁-x₂)² + (y₁-y₂)²)
Best case	reduces branching factor from b to √b

Machine Learning Metrics

ROC-AUC	Area under the curve of TPR vs FPR at various thresholds
Random classifier	AUC = 0.5
Perfect classifier	AUC = 1.0

Reinforcement Learning Formulas

Value Function	V(s) = E[Σ γᵗ r_t \| s_0 = s]
Q-Function	Q(s,a) = E[Σ γᵗ r_t \| s_0 = s, a_0 = a]
TD Update	V(s) ← V(s) + α[r + γV(s') - V(s)]
Q-Learning	Q(s,a) ← Q(s,a) + α[r + γ max_a' Q(s',a') - Q(s,a)]
Policy Gradient	∇J(θ) = E[∇log π_θ(a\|s) × G_t]

Dimensionality Reduction

PCA	Find directions of maximum variance
Solve	Σv = λv (eigenvectors of covariance matrix)
Project	x_reduced = x × V_k (top k eigenvectors)
Variance explained	λ_i / Σλ_j
High-dim	p_ij = exp(-\|\|x_i-x_j\|\|²/2σ²) / Σ exp(...)
Low-dim	q_ij = (1 + \|\|y_i-y_j\|\|²)⁻¹ / Σ (1 + \|\|y_k-y_l\|\|²)⁻¹

Summary

These formulas represent the mathematical foundations of artificial intelligence. From Bayes' theorem (reasoning under uncertainty) to gradient descent (learning from data) to the Bellman equation (planning over time), each formula encapsulates a fundamental principle. Mastering these formulas means understanding not just the symbols, but the intuition behind why each computation is performed and how it contributes to intelligent behavior.

Introduction

Probability and Statistics Foundations

Bayes' Theorem

P(A\|B) = posterior	probability of A given evidence B
P(B\|A) = likelihood	probability of observing B if A is true
P(A) = prior	initial belief about A before seeing B
P(B) = evidence	total probability of observing B
A = "patient has disease" (prior: 1% prevalence	P(A) = 0.01)
B = "test is positive" (test accuracy	95%)

Information Theory

Entropy	H(X) = -Σ P(x_i) × log₂(P(x_i))
Fair coin	H = -0.5×log₂(0.5) - 0.5×log₂(0.5) = 1 bit
Biased coin (90/10)	H = -0.9×log₂(0.9) - 0.1×log₂(0.1) = 0.47 bits
Cross-Entropy	H(P,Q) = -Σ P(x_i) × log(Q(x_i))
KL Divergence	D_KL(P\|\|Q) = Σ P(x_i) × log(P(x_i)/Q(x_i))

Linear Algebra for AI

Dot Product	a·b = Σ a_i × b_i = \|a\|\|b\|cos(θ)
Used in	attention mechanisms, similarity, neural network layers
Matrix Multiplication	(AB)_ij = Σ_k A_ik × B_kj
Used in	every neural network forward pass
Cosine Similarity	cos(θ) = (a·b) / (\|a\| × \|b\|)
Range	[-1, 1], where 1 = identical direction
Used in	document similarity, recommendation systems, embedding comparison
Softmax	softmax(x_i) = e^(x_i) / Σ_j e^(x_j)
Used in	classification output layers, attention weights

Neural Network Formulas

Neuron output	y = f(Σ w_i × x_i + b) = f(W·x + b)
MSE (regression)	L = (1/n) Σ (ŷ_i - y_i)²
Cross-entropy (classification)	L = -(1/n) Σ [y_i log(ŷ_i) + (1-y_i)log(1-ŷ_i)]
Gradient Descent	θ_{t+1} = θ_t - η × ∂L/∂θ
Dropout (training)	y = x × Bernoulli(1-p) / (1-p)

Search Algorithm Formulas

A* Evaluation	f(n) = g(n) + h(n)
Optimal when h is admissible	h(n) ≤ h*(n) for all n
Manhattan Distance	h(n) = \|x₁-x₂\| + \|y₁-y₂\|
Euclidean Distance	h(n) = √((x₁-x₂)² + (y₁-y₂)²)
Best case	reduces branching factor from b to √b

Machine Learning Metrics

ROC-AUC	Area under the curve of TPR vs FPR at various thresholds
Random classifier	AUC = 0.5
Perfect classifier	AUC = 1.0

Reinforcement Learning Formulas

Value Function	V(s) = E[Σ γᵗ r_t \| s_0 = s]
Q-Function	Q(s,a) = E[Σ γᵗ r_t \| s_0 = s, a_0 = a]
TD Update	V(s) ← V(s) + α[r + γV(s') - V(s)]
Q-Learning	Q(s,a) ← Q(s,a) + α[r + γ max_a' Q(s',a') - Q(s,a)]
Policy Gradient	∇J(θ) = E[∇log π_θ(a\|s) × G_t]

Dimensionality Reduction

PCA	Find directions of maximum variance
Solve	Σv = λv (eigenvectors of covariance matrix)
Project	x_reduced = x × V_k (top k eigenvectors)
Variance explained	λ_i / Σλ_j
High-dim	p_ij = exp(-\|\|x_i-x_j\|\|²/2σ²) / Σ exp(...)
Low-dim	q_ij = (1 + \|\|y_i-y_j\|\|²)⁻¹ / Σ (1 + \|\|y_k-y_l\|\|²)⁻¹

Important AI Formulas

Introduction

Probability and Statistics Foundations

Bayes' Theorem

Information Theory

Linear Algebra for AI

Neural Network Formulas

Search Algorithm Formulas

Machine Learning Metrics

Reinforcement Learning Formulas

Dimensionality Reduction

Summary

Continue learning this concept

Important AI Formulas

Introduction

Probability and Statistics Foundations

Bayes' Theorem

Information Theory

Linear Algebra for AI

Neural Network Formulas

Search Algorithm Formulas

Machine Learning Metrics

Reinforcement Learning Formulas

Dimensionality Reduction

Summary

Continue learning this concept