AI Notes

Gradient Descent - Optimization Fundamentals

Master gradient descent for AI. Learn learning rate, batch size, stochastic vs mini-batch, convergence, optimization. BTech guide 2024.

The Optimization Problem

Training a neural network means finding weight values that minimize a loss function. The loss function measures how wrong the network's predictions are—lower loss means better predictions. With millions of parameters, we cannot try every possible combination. Instead, we use gradient descent: iteratively moving in the direction that reduces the loss most quickly.

Think of it as navigating a mountain landscape blindfolded. You cannot see the lowest valley, but you can feel the slope beneath your feet. By always stepping downhill (in the steepest direction), you eventually reach a low point. The gradient tells you the direction of steepest ascent, so you move in the opposite direction—steepest descent.

The Gradient Descent Update Rule

Example

θ_new = θ_old - η × ∇L(θ) Where: θ = parameters (weights and biases) η = learning rate (step size) ∇L(θ) = gradient of loss with respect to parameters The negative sign: gradient points UPHILL, we want DOWNHILL

Variants of Gradient Descent

Batch Gradient Descent

Computes the gradient using the entire training set:

Example

for each epoch: gradient = (1/N) × Σᵢ ∇L(θ; xᵢ, yᵢ) // average over all N examples θ = θ - η × gradient

Pros: Stable convergence, true gradient direction Cons: Very slow for large datasets (must process all data for one update), requires entire dataset in memory

Stochastic Gradient Descent (SGD)

Updates after each single example:

Example

for each epoch: shuffle training data for each example (xᵢ, yᵢ): gradient = ∇L(θ; xᵢ, yᵢ) // gradient from one example θ = θ - η × gradient

Pros: Fast updates, can escape local minima (noise helps), works with streaming data Cons: Very noisy updates, may never converge exactly, can't leverage GPU parallelism

Mini-Batch Gradient Descent (Standard Practice)

The practical middle ground—compute gradients on small batches:

Example

for each epoch: shuffle and split data into batches of size B (typically 32-256) for each mini-batch: gradient = (1/B) × Σⱼ ∇L(θ; xⱼ, yⱼ) θ = θ - η × gradient

Pros: Balances noise and stability, leverages GPU parallelism, standard in all modern frameworks

Worked Example: Gradient Descent on a Simple Function

Minimize f(x) = (x - 3)² + 1 (minimum at x = 3)

Starting point	x = 0
Learning rate	η = 0.2
Gradient	f'(x) = 2(x - 3)
Iteration 1	x = 0 - 0.2 × 2(0-3) = 0 + 1.2 = 1.2
Iteration 2	x = 1.2 - 0.2 × 2(1.2-3) = 1.2 + 0.72 = 1.92
Iteration 3	x = 1.92 - 0.2 × 2(1.92-3) = 1.92 + 0.432 = 2.352
Iteration 4	x = 2.352 - 0.2 × 2(2.352-3) = 2.352 + 0.259 = 2.611
Iteration 5	x = 2.611 - 0.2 × 2(2.611-3) = 2.611 + 0.156 = 2.767
Iteration 15	x ≈ 2.986 (very close to optimal x=3)

Each step gets smaller as we approach the minimum (gradient → 0 near optimum).

The Learning Rate: Critical Hyperparameter

Too large (η = 2.0)	Overshoots, diverges
x = 0	12 → -48 → ... (exploding!)
Too small (η = 0.001)	Converges but extremely slowly
x = 0	0.006 → 0.012 → ... (thousands of steps needed)
Just right (η = 0.1-0.5)	Steady convergence

Learning Rate Schedules

Step decay	Reduce lr by factor every N epochs
Cosine annealing	Smooth decay following cosine curve
Warmup + decay	Start small, increase, then decrease
One-cycle	Increase then decrease over training

Advanced Optimizers

Momentum

Accumulates velocity in consistent gradient directions:

Example

v_t = β × v_{t-1} + ∇L(θ) (β typically 0.9) θ = θ - η × v_t Intuition: A ball rolling downhill builds speed - Accelerates in consistent directions - Dampens oscillations in inconsistent directions - Helps escape shallow local minima

RMSprop

Adapts learning rate per-parameter based on gradient history:

Example

s_t = β × s_{t-1} + (1-β) × (∇L)² (running average of squared gradients) θ = θ - η × ∇L / (√s_t + ε) Parameters with large gradients get smaller effective lr Parameters with small gradients get larger effective lr Ε = 1e-8 prevents division by zero

Adam (Adaptive Moment Estimation)

Combines momentum and RMSprop—the default optimizer:

m_t = β₁ × m_{t-1} + (1-β₁) × ∇L (first moment	mean)
v_t = β₂ × v_{t-1} + (1-β₂) × (∇L)² (second moment	variance)
Default hyperparameters	β₁=0.9, β₂=0.999, ε=1e-8, η=0.001

Optimizer Comparison

Optimizer	Learning Rate	Momentum	Adaptive	Best For
SGD	Fixed	No	No	Simple problems
SGD+Momentum	Fixed	Yes	No	CNNs (with tuning)
RMSprop	Adaptive	No	Yes	RNNs
Adam	Adaptive	Yes	Yes	Default choice
AdamW	Adaptive	Yes	Yes	Transformers

Challenges in Gradient Descent

Local Minima vs. Saddle Points

Example

In high dimensions (millions of params): - True local minima are rare - Saddle points are everywhere (some directions go up, others down) - Momentum and adaptive methods escape saddle points well In practice, most "local minima" are actually quite good (loss values similar to global minimum)

The Loss Landscape

Example

Sharp minima: Small area, poor generalization Model memorizes training data Flat minima: Broad basin, good generalization Robust to parameter perturbation SGD noise tends to find FLAT minima (better!) Adam can converge to SHARP minima (sometimes worse generalization)

Gradient Clipping

Prevents exploding gradients (especially in RNNs):

Example

if ||∇L|| > max_norm: ∇L = ∇L × (max_norm / ||∇L||) Preserves gradient direction, limits magnitude

Practical Training Recipe

Example

1. Start with Adam optimizer, lr=0.001 2. Use cosine learning rate schedule with warmup 3. Train until validation loss plateaus 4. If not satisfied: try SGD+momentum with lr=0.01 (often generalizes better) 5. Use gradient clipping = 1.0 for RNNs/Transformers 6. Batch size: largest that fits in GPU memory (typically 32-256)

Interview Questions

Q: Why does SGD with noise sometimes outperform Adam? A: SGD's noise helps it explore the loss landscape more broadly, finding flatter (more generalizable) minima. Adam's adaptive rates can converge too quickly to sharp minima that overfit. For CNNs on image tasks, well-tuned SGD+momentum often achieves better final test accuracy.

Q: What happens if the learning rate is too high? A: The updates overshoot the minimum, and the loss oscillates or diverges. In the extreme case, weights grow to infinity (NaN loss). The critical learning rate is approximately 2/λ_max where λ_max is the largest eigenvalue of the Hessian.

Q: Why do we need bias correction in Adam? A: Initially, m and v are biased toward zero (initialized at zero, slowly accumulate). Without correction, early updates are too small. Dividing by (1 - β^t) compensates, making early steps properly scaled. As t grows, the correction approaches 1 and becomes negligible.

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Gradient Descent - Optimization Fundamentals.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Artificial Intelligence topic.

Search Terms

artificial-intelligence, artificial intelligence, artificial, intelligence, neural, networks, gradient, descent

Continue learning this concept

Neural NetworksActivation Functions - Non-linearity in NetworksMaster activation functions for neural networks. Learn ReLU, sigmoid, tanh, softmax, when to use each. AI fundamentals 2024.Neural NetworksArtificial Neural Networks (ANN) - Complete BTech GuideMaster artificial neural networks for AI. Learn ANN architecture, layers, activation functions, forward propagation, backpropagation algorithms. Technical interview questions 2024.Neural NetworksBackpropagation - Learning Algorithm for NetworksMaster backpropagation algorithm for neural networks. Learn chain rule, gradient computation, computational graphs. AI fundamentals 2024.Neural NetworksForward Propagation - Inference in NetworksMaster forward propagation for neural networks. Learn layer-by-layer computation, vectorization, implementation. AI fundamentals 2024.Neural NetworksNeural Network Training - Complete ProcessMaster neural network training. Learn epochs, batches, validation, early stopping, hyperparameter tuning. AI fundamentals 2024.Neural NetworksPerceptron Learning Algorithm - AI FundamentalsMaster perceptron algorithm for artificial intelligence. Learn perceptron learning rule, convergence theorem, limitations, implementation. BTech notes 2024.

PreviousForward Propagation - Inference in Networks NextNeural Network Training - Complete Process

AI Notes

Gradient Descent - Optimization Fundamentals

Master gradient descent for AI. Learn learning rate, batch size, stochastic vs mini-batch, convergence, optimization. BTech guide 2024.

The Optimization Problem

The Gradient Descent Update Rule

Example

Variants of Gradient Descent

Batch Gradient Descent

Computes the gradient using the entire training set:

Example

for each epoch: gradient = (1/N) × Σᵢ ∇L(θ; xᵢ, yᵢ) // average over all N examples θ = θ - η × gradient

Pros: Stable convergence, true gradient direction Cons: Very slow for large datasets (must process all data for one update), requires entire dataset in memory

Stochastic Gradient Descent (SGD)

Updates after each single example:

Example

for each epoch: shuffle training data for each example (xᵢ, yᵢ): gradient = ∇L(θ; xᵢ, yᵢ) // gradient from one example θ = θ - η × gradient

Pros: Fast updates, can escape local minima (noise helps), works with streaming data Cons: Very noisy updates, may never converge exactly, can't leverage GPU parallelism

Mini-Batch Gradient Descent (Standard Practice)

The practical middle ground—compute gradients on small batches:

Example

for each epoch: shuffle and split data into batches of size B (typically 32-256) for each mini-batch: gradient = (1/B) × Σⱼ ∇L(θ; xⱼ, yⱼ) θ = θ - η × gradient

Pros: Balances noise and stability, leverages GPU parallelism, standard in all modern frameworks

Worked Example: Gradient Descent on a Simple Function

Minimize f(x) = (x - 3)² + 1 (minimum at x = 3)

Starting point	x = 0
Learning rate	η = 0.2
Gradient	f'(x) = 2(x - 3)
Iteration 1	x = 0 - 0.2 × 2(0-3) = 0 + 1.2 = 1.2
Iteration 2	x = 1.2 - 0.2 × 2(1.2-3) = 1.2 + 0.72 = 1.92
Iteration 3	x = 1.92 - 0.2 × 2(1.92-3) = 1.92 + 0.432 = 2.352
Iteration 4	x = 2.352 - 0.2 × 2(2.352-3) = 2.352 + 0.259 = 2.611
Iteration 5	x = 2.611 - 0.2 × 2(2.611-3) = 2.611 + 0.156 = 2.767
Iteration 15	x ≈ 2.986 (very close to optimal x=3)

Each step gets smaller as we approach the minimum (gradient → 0 near optimum).

The Learning Rate: Critical Hyperparameter

Too large (η = 2.0)	Overshoots, diverges
x = 0	12 → -48 → ... (exploding!)
Too small (η = 0.001)	Converges but extremely slowly
x = 0	0.006 → 0.012 → ... (thousands of steps needed)
Just right (η = 0.1-0.5)	Steady convergence

Learning Rate Schedules

Step decay	Reduce lr by factor every N epochs
Cosine annealing	Smooth decay following cosine curve
Warmup + decay	Start small, increase, then decrease
One-cycle	Increase then decrease over training

Advanced Optimizers

Momentum

Accumulates velocity in consistent gradient directions:

Example

RMSprop

Adapts learning rate per-parameter based on gradient history:

Example

Adam (Adaptive Moment Estimation)

Combines momentum and RMSprop—the default optimizer:

m_t = β₁ × m_{t-1} + (1-β₁) × ∇L (first moment	mean)
v_t = β₂ × v_{t-1} + (1-β₂) × (∇L)² (second moment	variance)
Default hyperparameters	β₁=0.9, β₂=0.999, ε=1e-8, η=0.001

Optimizer Comparison

Optimizer	Learning Rate	Momentum	Adaptive	Best For
SGD	Fixed	No	No	Simple problems
SGD+Momentum	Fixed	Yes	No	CNNs (with tuning)
RMSprop	Adaptive	No	Yes	RNNs
Adam	Adaptive	Yes	Yes	Default choice
AdamW	Adaptive	Yes	Yes	Transformers

Challenges in Gradient Descent

Local Minima vs. Saddle Points

Example

The Loss Landscape

Example

Gradient Clipping

Prevents exploding gradients (especially in RNNs):

Example

if ||∇L|| > max_norm: ∇L = ∇L × (max_norm / ||∇L||) Preserves gradient direction, limits magnitude

Practical Training Recipe

Example

Interview Questions

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Gradient Descent - Optimization Fundamentals.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Artificial Intelligence topic.

Search Terms

artificial-intelligence, artificial intelligence, artificial, intelligence, neural, networks, gradient, descent

Continue learning this concept

PreviousForward Propagation - Inference in Networks NextNeural Network Training - Complete Process