AI Notes
Master backpropagation algorithm for neural networks. Learn chain rule, gradient computation, computational graphs. AI fundamentals 2024.
What is Backpropagation?
Backpropagation (backward propagation of errors) is the algorithm that makes neural network training possible. It computes the gradient of the loss function with respect to every weight in the network by applying the chain rule of calculus systematically from the output layer backward to the input layer.
Without backpropagation, there would be no practical way to determine how to adjust thousands or millions of weights to reduce prediction error. Before backpropagation's popularization by Rumelhart, Hinton, and Williams in 1986, neural networks were limited to single-layer perceptrons. Backpropagation unlocked multi-layer learning and triggered the modern deep learning revolution.
The Big Picture: Training Loop
| 1. FORWARD PASS: Input | compute predictions |
| 2. COMPUTE LOSS | Compare predictions to true labels |
| 3. BACKWARD PASS | Compute gradients (backpropagation) |
| 4. UPDATE WEIGHTS | w = w - lr × gradient |
Backpropagation is specifically Step 3—computing how much each weight contributed to the error.
Mathematical Foundation: The Chain Rule
The chain rule states that for composed functions:
Each layer contributes one term in this chain. Backpropagation organizes this computation efficiently by reusing intermediate gradients.
Worked Example: Complete 2-Layer Network
Setup
| Network: 2 inputs | 2 hidden (sigmoid) → 1 output (sigmoid) |
| Input | x = [0.5, 0.8] |
| True label | y = 1.0 |
| Learning rate | η = 0.5 |
| Hidden layer | W₁ = [[0.1, 0.3], b₁ = [0.1, 0.2] |
| Output layer | W₂ = [0.5, 0.6] b₂ = 0.3 |
Forward Pass
Hidden layer
z₁ = W₁ · x + b₁
z₁[0] = 0.1×0.5 + 0.3×0.8 + 0.1 = 0.05 + 0.24 + 0.1 = 0.39
z₁[1] = 0.2×0.5 + 0.4×0.8 + 0.2 = 0.10 + 0.32 + 0.2 = 0.62
h = sigmoid(z₁)
h[0] = σ(0.39) = 1/(1+e^(-0.39)) ≈ 0.596
h[1] = σ(0.62) = 1/(1+e^(-0.62)) ≈ 0.650
Output layer
z₂ = W₂ · h + b₂
z₂ = 0.5×0.596 + 0.6×0.650 + 0.3 = 0.298 + 0.390 + 0.3 = 0.988
ŷ = sigmoid(z₂) = σ(0.988) ≈ 0.729
Loss (MSE)
L = ½(y - ŷ)² = ½(1.0 - 0.729)² = ½(0.271)² ≈ 0.0367
Backward Pass
| Step 1 | Output gradient |
| Step 2 | Output weight gradients |
| Step 3 | Propagate to hidden layer |
| Step 4 | Hidden weight gradients |
Weight Update
General Backpropagation Algorithm
Computational Graph Perspective
Modern frameworks (PyTorch, TensorFlow) implement backpropagation via computational graphs:
x → [multiply W] → [add b] → [sigmoid] → [loss]
↓ ↓ ↓ ↓
∂/∂W ∂/∂b σ'(z) ∂L/∂ŷ
Reverse-mode autodiff:
Start from loss, propagate gradients backward
Each node knows its local gradient
Chain rule composes them automatically
This is why you never manually implement backpropagation in practice—the framework builds the graph during forward pass and automatically computes all gradients.
Common Issues and Solutions
Vanishing Gradients
| Problem | sigmoid derivative max = 0.25 |
| After 5 layers | gradient ≤ 0.25⁵ = 0.001 (nearly zero!) |
| - ReLU activation | derivative = 1 for positive inputs |
| - Residual connections | gradient shortcut paths |
| - Batch normalization | keeps gradients in healthy range |
Exploding Gradients
| Problem | gradients grow exponentially through layers |
| - Gradient clipping | if ||g|| > threshold, g = g × threshold/||g|| |
| - Weight initialization | Xavier/He initialization |
Backprop for Different Activations
| Sigmoid | σ'(z) = σ(z)(1 - σ(z)) max = 0.25 |
| Tanh | tanh'(z) = 1 - tanh²(z) max = 1.0 |
| ReLU | ReLU'(z) = 1 if z > 0, else 0 (no saturation!) |
| Softmax | Jacobian matrix (for output layer) |
Batch vs. Stochastic Backpropagation
Batch gradient descent
Compute gradient over ALL training examples
Update once per epoch. Stable but slow.
Stochastic (SGD)
Compute gradient on ONE example
Update immediately. Noisy but fast.
Mini-batch (standard practice)
Compute gradient on batch of 32-256 examples
Best of both: stable estimates + frequent updates
Enables GPU parallelism
Interview Questions
Q: Why can't we just use random search instead of backpropagation? A: A network with 1M parameters has a 1M-dimensional space. Random search in such high dimensions is astronomically unlikely to find good weights. Backpropagation provides the exact direction to improve—the gradient—making optimization tractable.
Q: What does the gradient ∂L/∂w actually tell us? A: It tells us the rate at which the loss changes when w changes by a tiny amount. A large positive gradient means increasing w increases loss (so we should decrease w). A gradient near zero means w has little effect on the current loss.
Q: Why do we need the forward pass stored during backprop? A: Local gradients depend on the intermediate values. For example, sigmoid'(z) = σ(z)(1-σ(z)) requires knowing σ(z), which was computed during forward pass. Without stored activations, we'd need to recompute them.
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Backpropagation - Learning Algorithm for Networks.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Artificial Intelligence topic.
Search Terms
artificial-intelligence, artificial intelligence, artificial, intelligence, neural, networks, backpropagation, backpropagation - learning algorithm for networks
Related Artificial Intelligence Topics