DL Notes

Hub Activation Function Comparison Guide Activation Functions Overview Leaky ReLU: Solving Dead Neurons ReLU Activation Function Sigmoid Activation Function Softmax Function for Multi-class Classification Tanh Activation Function Computer Vision Projects with Deep Learning Face Recognition with Deep Learning Faster R-CNN Object Detection Image Processing Fundamentals Image Segmentation — Deep Learning Object Detection Architectures U-Net Architecture YOLO Real-time Object Detection AlexNet CNN Architecture CNN Architecture Design Introduction to CNNs Convolution Operation Image Classification Project LeNet Architecture Pooling Layers ResNet and Residual Networks Transfer Learning in Deep Learning VGGNet Architecture GPU Acceleration for Deep Learning Keras Deep Learning Framework PyTorch Deep Learning Framework Setting Up Deep Learning Environment TensorFlow Framework Guide TensorFlow vs PyTorch Comparison Deploying Deep Learning Models to Cloud Deep Learning APIs and Services Docker For Deep Learning Fastapi For Deployment Flask For Deployment Model Deployment Model Monitoring Ai Agent Basics Autoencoders Diffusion Models Generative Adversarial Networks Gan Generative Ai Introduction Large Language Models Llms Prompt Engineering Variational Autoencoders Cnn Interview Questions Deep Learning Cheatsheet Deep Learning Interview Questions — Deep Learning Neural Network Questions Transformer Interview Questions AI vs ML vs Deep Learning — Deep Learning Applications of Deep Learning Deep Learning Roadmap History of Deep Learning What is Deep Learning?Batch Normalization Explained Dropout Hyperparameter Tuning Model Debugging Model Evaluation Metrics Regularization Methods Train Validation Test Split Chatbot Development Glove Nlp Projects Nlp With Deep Learning Sentiment Analysis Seq2Seq Models Word Embeddings Word2Vec Artificial Neural Networks (ANN)Backpropagation Biological vs Artificial Neurons Forward Propagation Gradient Descent Layers in Neural Networks Loss Functions in Deep Learning The Perceptron Adam Optimizer Batch Gradient Descent Learning Rate Scheduling Mini Batch Gradient Descent Momentum Optimization Basics Rmsprop Stochastic Gradient Descent Calculus for Deep Learning Data Preprocessing for Deep Learning Linear Algebra for Deep Learning NumPy Basics for Deep Learning Probability and Statistics for Deep Learning Python for Deep Learning Ai Chatbot Project Face Mask Detection Handwritten Digit Recognition Image Classification System Object Detection System Personal Ai Assistant Recommendation System — Deep Learning Sentiment Analysis System Bidirectional Rnn GRU Networks LSTM Networks Rnn Introduction Sequence Modeling Text Generation Project Vanishing Gradient Problem Datasets For Practice Deep Learning Glossary Important Formulas — Deep Learning Recommended Books And Papers Attention Mechanism Bert Model Encoder Decoder Architecture Gpt Models Positional Encoding Self Attention Transformer Projects Transformers Introduction

Backpropagation

Complete guide to backpropagation algorithm in neural networks - the chain rule, gradient computation, weight updates, and implementation from scratch with PyTorch.

Backpropagation (backward propagation of errors) is the algorithm that makes training deep neural networks possible. It efficiently computes gradients of the loss function with respect to every weight in the network using the chain rule of calculus.

The Core Idea

After forward propagation produces a prediction, we compare it with the true label using a loss function. Backpropagation then flows the error backward through the network, computing how much each weight contributed to the error.

Example

Forward Pass: x ──→ h1 ──→ h2 ──→ ŷ ──→ Loss │ Backward Pass: ∂L/∂W1 ←── ∂L/∂h1 ←── ∂L/∂h2 ←── ∂L/∂ŷ ←──┘

The Chain Rule

The mathematical foundation of backpropagation is the chain rule:

Example

If y = f(g(x)), then dy/dx = f'(g(x)) · g'(x) For neural networks: ∂Loss/∂W[l] = ∂Loss/∂a[L] · ∂a[L]/∂z[L] · ∂z[L]/∂a[L-1] · ... · ∂z[l]/∂W[l]

Step-by-Step Derivation

Consider a 2-layer network with sigmoid activation and MSE loss:

Layer 1	z1 = W1·x + b1, a1 = σ(z1)
Layer 2	z2 = W2·a1 + b2, ŷ = σ(z2)
Loss	L = (1/2)(y - ŷ)²

Output Layer Gradients:

∂L/∂ŷ = -(y - ŷ)

∂ŷ/∂z2 = σ(z2)·(1 - σ(z2)) = ŷ·(1 - ŷ)

δ2 = ∂L/∂z2 = -(y - ŷ) · ŷ · (1 - ŷ)

∂L/∂W2 = δ2 · a1^T

∂L/∂b2 = δ2

Hidden Layer Gradients:

Example

∂L/∂a1 = W2^T · δ2 ∂a1/∂z1 = a1 · (1 - a1) δ1 = (W2^T · δ2) ⊙ a1 · (1 - a1) ∂L/∂W1 = δ1 · x^T ∂L/∂b1 = δ1

Complete Implementation from Scratch

Example

import numpy as np class NeuralNetworkWithBackprop: def __init__(self, layers): """layers: list of layer sizes e.g., [2, 4, 4, 1]""" self.L = len(layers) - 1 self.weights = [] self.biases = [] for i in range(self.L): w = np.random.randn(layers[i+1], layers[i]) * np.sqrt(2.0 / layers[i]) b = np.zeros((layers[i+1], 1)) self.weights.append(w) self.biases.append(b) def sigmoid(self, z): return 1 / (1 + np.exp(-np.clip(z, -500, 500))) def sigmoid_derivative(self, a): return a * (1 - a) def relu(self, z): return np.maximum(0, z) def relu_derivative(self, z): return (z > 0).astype(float) def forward(self, X): self.activations = [X] self.z_values = [] a = X for i in range(self.L - 1): z = self.weights[i] @ a + self.biases[i] self.z_values.append(z) a = self.relu(z) self.activations.append(a) z = self.weights[-1] @ a + self.biases[-1] self.z_values.append(z) a = self.sigmoid(z) self.activations.append(a) return a def backward(self, y): m = y.shape[1] self.dW = [None] * self.L self.db = [None] * self.L # Output layer gradient dA = -(y / self.activations[-1] - (1-y) / (1-self.activations[-1])) dZ = dA * self.sigmoid_derivative(self.activations[-1]) self.dW[-1] = (1/m) * dZ @ self.activations[-2].T self.db[-1] = (1/m) * np.sum(dZ, axis=1, keepdims=True) # Hidden layers for l in range(self.L - 2, -1, -1): dA = self.weights[l+1].T @ dZ dZ = dA * self.relu_derivative(self.z_values[l]) self.dW[l] = (1/m) * dZ @ self.activations[l].T self.db[l] = (1/m) * np.sum(dZ, axis=1, keepdims=True) def update_weights(self, learning_rate): for i in range(self.L): self.weights[i] -= learning_rate * self.dW[i] self.biases[i] -= learning_rate * self.db[i] def train(self, X, y, epochs=1000, lr=0.01): losses = [] for epoch in range(epochs): predictions = self.forward(X) loss = -np.mean(y*np.log(predictions+1e-8) + (1-y)*np.log(1-predictions+1e-8)) losses.append(loss) self.backward(y) self.update_weights(lr) if epoch % 200 == 0: print(f"Epoch {epoch}, Loss: {loss:.4f}") return losses # Train on XOR problem X = np.array([[0, 0, 1, 1], [0, 1, 0, 1]]) y = np.array([[0, 1, 1, 0]]) nn = NeuralNetworkWithBackprop([2, 8, 4, 1]) losses = nn.train(X, y, epochs=2000, lr=0.5) predictions = nn.forward(X) print(f"Final predictions: {predictions.round(2)}")

Backpropagation in PyTorch

PyTorch handles backpropagation automatically through its autograd system:

Example

import torch import torch.nn as nn model = nn.Sequential( nn.Linear(2, 8), nn.ReLU(), nn.Linear(8, 4), nn.ReLU(), nn.Linear(4, 1), nn.Sigmoid() ) criterion = nn.BCELoss() optimizer = torch.optim.SGD(model.parameters(), lr=0.5) X = torch.tensor([[0,0],[0,1],[1,0],[1,1]], dtype=torch.float32) y = torch.tensor([[0],[1],[1],[0]], dtype=torch.float32) for epoch in range(2000): predictions = model(X) loss = criterion(predictions, y) optimizer.zero_grad() loss.backward() # Backpropagation happens here optimizer.step() if epoch % 400 == 0: print(f"Epoch {epoch}, Loss: {loss.item():.4f}") # Inspect gradients after a backward pass predictions = model(X) loss = criterion(predictions, y) loss.backward() for name, param in model.named_parameters(): if param.grad is not None: print(f"{name}: grad norm = {param.grad.norm():.4f}")

Gradient Flow Visualization

∂L/∂W3 ∂L/∂W2 ∂L/∂W1

┌─────────────┬─────────────┬─────────────┐

│ │ │ │

↓ ↓ ↓ ↓

x ──→ [W1,ReLU] ──→ [W2,ReLU] ──→ [W3,σ] ──→ ŷ ──→ L

a1 a2 a3

δ1 ←────── δ2 ←────── δ3 ←────── ∂L/∂ŷ

Computational Complexity

Operation	Time Complexity	Space Complexity
Forward Pass	O(n²) per layer	O(n) activations cached
Backward Pass	O(n²) per layer	O(n²) for gradients
Weight Update	O(n²) total	O(n²) for weight storage
Total per iteration	O(L·n²)	O(L·n²)

Common Issues and Solutions

Problem	Cause	Solution
Vanishing gradients	Sigmoid/tanh in deep nets	Use ReLU, ResNets
Exploding gradients	Large weight init	Gradient clipping, proper init
Slow convergence	Small learning rate	Adaptive optimizers (Adam)
Dead neurons	ReLU with negative inputs	Leaky ReLU, proper init

Interview Questions

Why is caching activations necessary during forward propagation?

Backpropagation needs the intermediate activations to compute gradients. Without caching, we would need to recompute forward passes for each layer.

What is the vanishing gradient problem?

When using sigmoid/tanh in deep networks, gradients multiply through many layers. Since sigmoid derivative maxes at 0.25, gradients shrink exponentially, making early layers untrainable.

How does gradient clipping prevent exploding gradients?

It scales down the gradient vector when its norm exceeds a threshold, preventing large weight updates while preserving gradient direction.

What is the difference between backpropagation and gradient descent?

Backpropagation computes gradients using the chain rule. Gradient descent uses those gradients to update weights. They are complementary but distinct concepts.

Can backpropagation get stuck in local minima?

In theory yes, but high-dimensional loss surfaces mostly have saddle points rather than true local minima. SGD with momentum helps escape these.

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Backpropagation.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Deep Learning topic.

Search Terms

deep-learning, deep learning, deep, learning, neural, network, fundamentals, backpropagation

Continue learning this concept

Neural Network FundamentalsArtificial Neural Networks (ANN)Complete guide to Artificial Neural Networks — understanding architecture, how information flows through layers, and building your first network from scratch.Neural Network FundamentalsBiological vs Artificial NeuronsUnderstanding the parallels and differences between biological neurons in the brain and artificial neurons in neural networks.Neural Network FundamentalsForward PropagationUnderstanding forward propagation — how neural networks compute predictions by passing inputs through layers of weights, biases, and activation functions.Neural Network FundamentalsGradient DescentUnderstanding gradient descent optimization algorithm - how neural networks learn by iteratively minimizing the loss function through parameter updates.Neural Network FundamentalsLayers in Neural NetworksUnderstanding different types of layers in neural networks - input, hidden, and output layers, dense connections, and how to design effective architectures.Neural Network FundamentalsLoss Functions in Deep LearningComprehensive guide to loss functions - MSE, Cross-Entropy, Hinge Loss, Focal Loss with mathematical formulations, PyTorch implementations, and selection criteria.

PreviousArtificial Neural Networks (ANN)NextBiological vs Artificial Neurons

DL Notes

Backpropagation

Complete guide to backpropagation algorithm in neural networks - the chain rule, gradient computation, weight updates, and implementation from scratch with PyTorch.

The Core Idea

Example

Forward Pass: x ──→ h1 ──→ h2 ──→ ŷ ──→ Loss │ Backward Pass: ∂L/∂W1 ←── ∂L/∂h1 ←── ∂L/∂h2 ←── ∂L/∂ŷ ←──┘

The Chain Rule

The mathematical foundation of backpropagation is the chain rule:

Example

If y = f(g(x)), then dy/dx = f'(g(x)) · g'(x) For neural networks: ∂Loss/∂W[l] = ∂Loss/∂a[L] · ∂a[L]/∂z[L] · ∂z[L]/∂a[L-1] · ... · ∂z[l]/∂W[l]

Step-by-Step Derivation

Consider a 2-layer network with sigmoid activation and MSE loss:

Layer 1	z1 = W1·x + b1, a1 = σ(z1)
Layer 2	z2 = W2·a1 + b2, ŷ = σ(z2)
Loss	L = (1/2)(y - ŷ)²

Output Layer Gradients:

∂L/∂ŷ = -(y - ŷ)

∂ŷ/∂z2 = σ(z2)·(1 - σ(z2)) = ŷ·(1 - ŷ)

δ2 = ∂L/∂z2 = -(y - ŷ) · ŷ · (1 - ŷ)

∂L/∂W2 = δ2 · a1^T

∂L/∂b2 = δ2

Hidden Layer Gradients:

Example

∂L/∂a1 = W2^T · δ2 ∂a1/∂z1 = a1 · (1 - a1) δ1 = (W2^T · δ2) ⊙ a1 · (1 - a1) ∂L/∂W1 = δ1 · x^T ∂L/∂b1 = δ1

Complete Implementation from Scratch

Example

Backpropagation in PyTorch

PyTorch handles backpropagation automatically through its autograd system:

Example

Gradient Flow Visualization

∂L/∂W3 ∂L/∂W2 ∂L/∂W1

┌─────────────┬─────────────┬─────────────┐

│ │ │ │

↓ ↓ ↓ ↓

x ──→ [W1,ReLU] ──→ [W2,ReLU] ──→ [W3,σ] ──→ ŷ ──→ L

a1 a2 a3

δ1 ←────── δ2 ←────── δ3 ←────── ∂L/∂ŷ

Computational Complexity

Operation	Time Complexity	Space Complexity
Forward Pass	O(n²) per layer	O(n) activations cached
Backward Pass	O(n²) per layer	O(n²) for gradients
Weight Update	O(n²) total	O(n²) for weight storage
Total per iteration	O(L·n²)	O(L·n²)

Common Issues and Solutions

Problem	Cause	Solution
Vanishing gradients	Sigmoid/tanh in deep nets	Use ReLU, ResNets
Exploding gradients	Large weight init	Gradient clipping, proper init
Slow convergence	Small learning rate	Adaptive optimizers (Adam)
Dead neurons	ReLU with negative inputs	Leaky ReLU, proper init

Interview Questions

Why is caching activations necessary during forward propagation?

Backpropagation needs the intermediate activations to compute gradients. Without caching, we would need to recompute forward passes for each layer.

What is the vanishing gradient problem?

When using sigmoid/tanh in deep networks, gradients multiply through many layers. Since sigmoid derivative maxes at 0.25, gradients shrink exponentially, making early layers untrainable.

How does gradient clipping prevent exploding gradients?

It scales down the gradient vector when its norm exceeds a threshold, preventing large weight updates while preserving gradient direction.

What is the difference between backpropagation and gradient descent?

Backpropagation computes gradients using the chain rule. Gradient descent uses those gradients to update weights. They are complementary but distinct concepts.

Can backpropagation get stuck in local minima?

In theory yes, but high-dimensional loss surfaces mostly have saddle points rather than true local minima. SGD with momentum helps escape these.

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Backpropagation.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Deep Learning topic.

Search Terms

deep-learning, deep learning, deep, learning, neural, network, fundamentals, backpropagation

Continue learning this concept

PreviousArtificial Neural Networks (ANN)NextBiological vs Artificial Neurons