Complete guide to backpropagation algorithm in neural networks - the chain rule, gradient computation, weight updates, and implementation from scratch with PyTorch.
Backpropagation (backward propagation of errors) is the algorithm that makes training deep neural networks possible. It efficiently computes gradients of the loss function with respect to every weight in the network using the chain rule of calculus.
The Core Idea
After forward propagation produces a prediction, we compare it with the true label using a loss function. Backpropagation then flows the error backward through the network, computing how much each weight contributed to the error.
Forward Pass: x ──→ h1 ──→ h2 ──→ ŷ ──→ Loss
│
Backward Pass: ∂L/∂W1 ←── ∂L/∂h1 ←── ∂L/∂h2 ←── ∂L/∂ŷ ←──┘
The Chain Rule
The mathematical foundation of backpropagation is the chain rule:
If y = f(g(x)), then dy/dx = f'(g(x)) · g'(x)
For neural networks:
∂Loss/∂W[l] = ∂Loss/∂a[L] · ∂a[L]/∂z[L] · ∂z[L]/∂a[L-1] · ... · ∂z[l]/∂W[l]
Step-by-Step Derivation
Consider a 2-layer network with sigmoid activation and MSE loss:
| Layer 1 | z1 = W1·x + b1, a1 = σ(z1) |
| Layer 2 | z2 = W2·a1 + b2, ŷ = σ(z2) |
| Loss | L = (1/2)(y - ŷ)² |
Output Layer Gradients:
∂L/∂ŷ = -(y - ŷ)
∂ŷ/∂z2 = σ(z2)·(1 - σ(z2)) = ŷ·(1 - ŷ)
δ2 = ∂L/∂z2 = -(y - ŷ) · ŷ · (1 - ŷ)
∂L/∂W2 = δ2 · a1^T
∂L/∂b2 = δ2
Hidden Layer Gradients:
∂L/∂a1 = W2^T · δ2
∂a1/∂z1 = a1 · (1 - a1)
δ1 = (W2^T · δ2) ⊙ a1 · (1 - a1)
∂L/∂W1 = δ1 · x^T
∂L/∂b1 = δ1
Complete Implementation from Scratch
import numpy as np
class NeuralNetworkWithBackprop:
def __init__(self, layers):
"""layers: list of layer sizes e.g., [2, 4, 4, 1]"""
self.L = len(layers) - 1
self.weights = []
self.biases = []
for i in range(self.L):
w = np.random.randn(layers[i+1], layers[i]) * np.sqrt(2.0 / layers[i])
b = np.zeros((layers[i+1], 1))
self.weights.append(w)
self.biases.append(b)
def sigmoid(self, z):
return 1 / (1 + np.exp(-np.clip(z, -500, 500)))
def sigmoid_derivative(self, a):
return a * (1 - a)
def relu(self, z):
return np.maximum(0, z)
def relu_derivative(self, z):
return (z > 0).astype(float)
def forward(self, X):
self.activations = [X]
self.z_values = []
a = X
for i in range(self.L - 1):
z = self.weights[i] @ a + self.biases[i]
self.z_values.append(z)
a = self.relu(z)
self.activations.append(a)
z = self.weights[-1] @ a + self.biases[-1]
self.z_values.append(z)
a = self.sigmoid(z)
self.activations.append(a)
return a
def backward(self, y):
m = y.shape[1]
self.dW = [None] * self.L
self.db = [None] * self.L
# Output layer gradient
dA = -(y / self.activations[-1] - (1-y) / (1-self.activations[-1]))
dZ = dA * self.sigmoid_derivative(self.activations[-1])
self.dW[-1] = (1/m) * dZ @ self.activations[-2].T
self.db[-1] = (1/m) * np.sum(dZ, axis=1, keepdims=True)
# Hidden layers
for l in range(self.L - 2, -1, -1):
dA = self.weights[l+1].T @ dZ
dZ = dA * self.relu_derivative(self.z_values[l])
self.dW[l] = (1/m) * dZ @ self.activations[l].T
self.db[l] = (1/m) * np.sum(dZ, axis=1, keepdims=True)
def update_weights(self, learning_rate):
for i in range(self.L):
self.weights[i] -= learning_rate * self.dW[i]
self.biases[i] -= learning_rate * self.db[i]
def train(self, X, y, epochs=1000, lr=0.01):
losses = []
for epoch in range(epochs):
predictions = self.forward(X)
loss = -np.mean(y*np.log(predictions+1e-8) + (1-y)*np.log(1-predictions+1e-8))
losses.append(loss)
self.backward(y)
self.update_weights(lr)
if epoch % 200 == 0:
print(f"Epoch {epoch}, Loss: {loss:.4f}")
return losses
# Train on XOR problem
X = np.array([[0, 0, 1, 1], [0, 1, 0, 1]])
y = np.array([[0, 1, 1, 0]])
nn = NeuralNetworkWithBackprop([2, 8, 4, 1])
losses = nn.train(X, y, epochs=2000, lr=0.5)
predictions = nn.forward(X)
print(f"Final predictions: {predictions.round(2)}")
Backpropagation in PyTorch
PyTorch handles backpropagation automatically through its autograd system:
import torch
import torch.nn as nn
model = nn.Sequential(
nn.Linear(2, 8),
nn.ReLU(),
nn.Linear(8, 4),
nn.ReLU(),
nn.Linear(4, 1),
nn.Sigmoid()
)
criterion = nn.BCELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.5)
X = torch.tensor([[0,0],[0,1],[1,0],[1,1]], dtype=torch.float32)
y = torch.tensor([[0],[1],[1],[0]], dtype=torch.float32)
for epoch in range(2000):
predictions = model(X)
loss = criterion(predictions, y)
optimizer.zero_grad()
loss.backward() # Backpropagation happens here
optimizer.step()
if epoch % 400 == 0:
print(f"Epoch {epoch}, Loss: {loss.item():.4f}")
# Inspect gradients after a backward pass
predictions = model(X)
loss = criterion(predictions, y)
loss.backward()
for name, param in model.named_parameters():
if param.grad is not None:
print(f"{name}: grad norm = {param.grad.norm():.4f}")
Gradient Flow Visualization
∂L/∂W3 ∂L/∂W2 ∂L/∂W1
┌─────────────┬─────────────┬─────────────┐
│ │ │ │
↓ ↓ ↓ ↓
x ──→ [W1,ReLU] ──→ [W2,ReLU] ──→ [W3,σ] ──→ ŷ ──→ L
a1 a2 a3
δ1 ←────── δ2 ←────── δ3 ←────── ∂L/∂ŷ
Computational Complexity
| Operation | Time Complexity | Space Complexity |
|---|
| Forward Pass | O(n²) per layer | O(n) activations cached |
| Backward Pass | O(n²) per layer | O(n²) for gradients |
| Weight Update | O(n²) total | O(n²) for weight storage |
| Total per iteration | O(L·n²) | O(L·n²) |
Common Issues and Solutions
| Problem | Cause | Solution |
|---|
| Vanishing gradients | Sigmoid/tanh in deep nets | Use ReLU, ResNets |
| Exploding gradients | Large weight init | Gradient clipping, proper init |
| Slow convergence | Small learning rate | Adaptive optimizers (Adam) |
| Dead neurons | ReLU with negative inputs | Leaky ReLU, proper init |
Interview Questions
- Why is caching activations necessary during forward propagation?
Backpropagation needs the intermediate activations to compute gradients. Without caching, we would need to recompute forward passes for each layer.
- What is the vanishing gradient problem?
When using sigmoid/tanh in deep networks, gradients multiply through many layers. Since sigmoid derivative maxes at 0.25, gradients shrink exponentially, making early layers untrainable.
- How does gradient clipping prevent exploding gradients?
It scales down the gradient vector when its norm exceeds a threshold, preventing large weight updates while preserving gradient direction.
- What is the difference between backpropagation and gradient descent?
Backpropagation computes gradients using the chain rule. Gradient descent uses those gradients to update weights. They are complementary but distinct concepts.
- Can backpropagation get stuck in local minima?
In theory yes, but high-dimensional loss surfaces mostly have saddle points rather than true local minima. SGD with momentum helps escape these.