DL Notes

Hub Activation Function Comparison Guide Activation Functions Overview Leaky ReLU: Solving Dead Neurons ReLU Activation Function Sigmoid Activation Function Softmax Function for Multi-class Classification Tanh Activation Function Computer Vision Projects with Deep Learning Face Recognition with Deep Learning Faster R-CNN Object Detection Image Processing Fundamentals Image Segmentation — Deep Learning Object Detection Architectures U-Net Architecture YOLO Real-time Object Detection AlexNet CNN Architecture CNN Architecture Design Introduction to CNNs Convolution Operation Image Classification Project LeNet Architecture Pooling Layers ResNet and Residual Networks Transfer Learning in Deep Learning VGGNet Architecture GPU Acceleration for Deep Learning Keras Deep Learning Framework PyTorch Deep Learning Framework Setting Up Deep Learning Environment TensorFlow Framework Guide TensorFlow vs PyTorch Comparison Deploying Deep Learning Models to Cloud Deep Learning APIs and Services Docker For Deep Learning Fastapi For Deployment Flask For Deployment Model Deployment Model Monitoring Ai Agent Basics Autoencoders Diffusion Models Generative Adversarial Networks Gan Generative Ai Introduction Large Language Models Llms Prompt Engineering Variational Autoencoders Cnn Interview Questions Deep Learning Cheatsheet Deep Learning Interview Questions — Deep Learning Neural Network Questions Transformer Interview Questions AI vs ML vs Deep Learning — Deep Learning Applications of Deep Learning Deep Learning Roadmap History of Deep Learning What is Deep Learning?Batch Normalization Explained Dropout Hyperparameter Tuning Model Debugging Model Evaluation Metrics Regularization Methods Train Validation Test Split Chatbot Development Glove Nlp Projects Nlp With Deep Learning Sentiment Analysis Seq2Seq Models Word Embeddings Word2Vec Artificial Neural Networks (ANN)Backpropagation Biological vs Artificial Neurons Forward Propagation Gradient Descent Layers in Neural Networks Loss Functions in Deep Learning The Perceptron Adam Optimizer Batch Gradient Descent Learning Rate Scheduling Mini Batch Gradient Descent Momentum Optimization Basics Rmsprop Stochastic Gradient Descent Calculus for Deep Learning Data Preprocessing for Deep Learning Linear Algebra for Deep Learning NumPy Basics for Deep Learning Probability and Statistics for Deep Learning Python for Deep Learning Ai Chatbot Project Face Mask Detection Handwritten Digit Recognition Image Classification System Object Detection System Personal Ai Assistant Recommendation System — Deep Learning Sentiment Analysis System Bidirectional Rnn GRU Networks LSTM Networks Rnn Introduction Sequence Modeling Text Generation Project Vanishing Gradient Problem Datasets For Practice Deep Learning Glossary Important Formulas — Deep Learning Recommended Books And Papers Attention Mechanism Bert Model Encoder Decoder Architecture Gpt Models Positional Encoding Self Attention Transformer Projects Transformers Introduction

Activation Function Comparison Guide

Comprehensive guide to activation function comparison guide in deep learning

Overview

Comprehensive guide to activation function comparison guide in deep learning This comprehensive guide explores the theoretical foundations, practical implementations, and real-world applications in deep learning.

Fundamental Concepts

Core Principles

The mathematical and computational foundations are rooted in:

Linear algebra: Matrix operations, eigenvalues, transformations
Calculus: Derivatives, chain rule, partial derivatives
Optimization: Gradient descent, convergence, stability
Numerical methods: Precision, efficiency, scalability

Understanding these principles enables you to:

Diagnose training issues quickly
Optimize model performance systematically
Make informed architecture decisions
Contribute to research effectively

Architecture Overview

Deep Learning Processing Pipeline: ┌──────────────────────────────────────┐ │ Input Data (images, text, sequences) │ ├──────────────────────────────────────┤ │ ↓ Preprocessing & Normalization │ ├──────────────────────────────────────┤ │ ↓ Feature Extraction (Layers) │ ├──────────────────────────────────────┤ │ ↓ Non-linear Transformations │ ├──────────────────────────────────────┤ │ ↓ Output Transformation │ ├──────────────────────────────────────┤ │ Final Output (predictions/embeddings)│ └──────────────────────────────────────┘

PyTorch Implementation

Model Definition

python example

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader

# Device setup
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Define neural network model
class DeepLearningModel(nn.Module):
    def __init__(self, input_size=784, hidden_size=512, num_classes=10):
        super(DeepLearningModel, self).__init__()
        
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.bn1 = nn.BatchNorm1d(hidden_size)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(0.3)
        
        self.fc2 = nn.Linear(hidden_size, 256)
        self.bn2 = nn.BatchNorm1d(256)
        
        self.fc3 = nn.Linear(256, 128)
        self.bn3 = nn.BatchNorm1d(128)
        
        self.fc4 = nn.Linear(128, num_classes)
    
    def forward(self, x):
        x = x.view(x.size(0), -1)
        x = self.relu(self.bn1(self.fc1(x)))
        x = self.dropout(x)
        x = self.relu(self.bn2(self.fc2(x)))
        x = self.dropout(x)
        x = self.relu(self.bn3(self.fc3(x)))
        x = self.fc4(x)
        return x

model = DeepLearningModel().to(device)

Training Loop

python example

def train_model(model, train_loader, val_loader, num_epochs=50):
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=0.001)
    scheduler = optim.lr_scheduler.ReduceLROnPlateau(
        optimizer, mode='min', factor=0.5, patience=5, verbose=True
    )
    
    best_val_loss = float('inf')
    
    for epoch in range(num_epochs):
        # Training phase
        model.train()
        train_loss = 0.0
        train_correct = 0
        train_total = 0
        
        for images, labels in train_loader:
            images, labels = images.to(device), labels.to(device)
            
            # Forward pass
            outputs = model(images)
            loss = criterion(outputs, labels)
            
            # Backward pass and optimization
            optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
            optimizer.step()
            
            # Statistics
            train_loss += loss.item()
            _, predicted = torch.max(outputs.data, 1)
            train_total += labels.size(0)
            train_correct += (predicted == labels).sum().item()
        
        # Validation phase
        model.eval()
        val_loss = 0.0
        val_correct = 0
        val_total = 0
        
        with torch.no_grad():
            for images, labels in val_loader:
                images, labels = images.to(device), labels.to(device)
                outputs = model(images)
                loss = criterion(outputs, labels)
                
                val_loss += loss.item()
                _, predicted = torch.max(outputs.data, 1)
                val_total += labels.size(0)
                val_correct += (predicted == labels).sum().item()
        
        avg_train_loss = train_loss / len(train_loader)
        avg_val_loss = val_loss / len(val_loader)
        train_acc = 100 * train_correct / train_total
        val_acc = 100 * val_correct / val_total
        
        print(f'Epoch {epoch+1}/{num_epochs}:')
        print(f'  Train Loss: {avg_train_loss:.4f}, Train Acc: {train_acc:.2f}%')
        print(f'  Val Loss: {avg_val_loss:.4f}, Val Acc: {val_acc:.2f}%')
        
        # Learning rate scheduling
        scheduler.step(avg_val_loss)
        
        # Save best model
        if avg_val_loss < best_val_loss:
            best_val_loss = avg_val_loss
            torch.save(model.state_dict(), 'best_model.pth')
    
    return model

TensorFlow/Keras Implementation

Model Creation

Example

import tensorflow as tf from tensorflow import keras from tensorflow.keras import layers # Build model using Sequential API model = keras.Sequential([ layers.Dense(512, activation='relu', input_shape=(784,)), layers.BatchNormalization(), layers.Dropout(0.3), layers.Dense(256, activation='relu'), layers.BatchNormalization(), layers.Dropout(0.3), layers.Dense(128, activation='relu'), layers.BatchNormalization(), layers.Dropout(0.2), layers.Dense(10, activation='softmax') ]) # Compile model model.compile( optimizer=keras.optimizers.Adam(learning_rate=0.001), loss='sparse_categorical_crossentropy', metrics=['accuracy'] ) # Summary model.summary()

Training with Callbacks

Example

# Define callbacks callbacks = [ keras.callbacks.EarlyStopping( monitor='val_loss', patience=10, restore_best_weights=True ), keras.callbacks.ReduceLROnPlateau( monitor='val_loss', factor=0.5, patience=5, min_lr=1e-6 ), keras.callbacks.ModelCheckpoint( 'best_model.h5', monitor='val_loss', save_best_only=True ) ] # Train model history = model.fit( X_train, y_train, batch_size=32, epochs=100, validation_split=0.2, callbacks=callbacks, verbose=1 ) # Evaluate on test set test_loss, test_acc = model.evaluate(X_test, y_test) print(f'Test Accuracy: {test_acc:.4f}')

Performance Comparison

Model Architecture Comparison

Model	Parameters	Speed (img/s)	Memory (MB)	Accuracy
ResNet-50	25.5M	750	4,200	76.1%
VGG-16	138M	120	8,500	71.3%
MobileNet	4.2M	3,000	1,200	70.2%
EfficientNet-B0	5.3M	800	1,800	77.1%
Inception-V3	27.2M	600	3,900	78.0%

Hyperparameter Impact

Parameter	Min	Max	Recommended	Impact
Learning Rate	1e-5	1e-1	1e-3	Very High
Batch Size	8	512	32-64	High
Dropout Rate	0.0	0.5	0.3	Medium
Weight Decay	1e-6	1e-2	1e-4	Medium
Momentum	0.0	0.99	0.9	Low

Real-World Applications

Application 1: Image Recognition

Use Case: Medical image analysis, autonomous vehicles
Architecture: CNN (ResNet, EfficientNet)
Performance: 94-99% accuracy depending on dataset
Key Challenge: Data annotation cost, domain adaptation

Application 2: Natural Language Processing

Use Case: Machine translation, sentiment analysis
Architecture: Transformers (BERT, GPT)
Performance: State-of-the-art on various benchmarks
Key Challenge: Context understanding, ambiguity resolution

Application 3: Time Series Forecasting

Use Case: Stock price prediction, weather forecasting
Architecture: LSTM, GRU, or Transformers
Performance: Depends on temporal complexity
Key Challenge: Capturing long-term dependencies

Advanced Techniques

Gradient Accumulation

python example

accumulation_steps = 4
optimizer.zero_grad()

for i, (images, labels) in enumerate(train_loader):
    outputs = model(images)
    loss = criterion(outputs, labels)
    loss = loss / accumulation_steps
    loss.backward()
    
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

Mixed Precision Training

python example

from torch.cuda.amp import autocast, GradScaler

scaler = GradScaler()
model = model.to(device)

for images, labels in train_loader:
    images, labels = images.to(device), labels.to(device)
    
    with autocast():
        outputs = model(images)
        loss = criterion(outputs, labels)
    
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()
    optimizer.zero_grad()

Troubleshooting Guide

Issue: Training Loss Not Decreasing

Symptoms: Loss stays constant or increases

Solutions:

Check data normalization: mean ~0, std ~1
Verify loss function is correct for task
Test on small batch to verify learning
Reduce learning rate systematically
Check for NaN values in activations
Verify labels are correct

Issue: Out of Memory

Solutions:

Reduce batch size (32 → 16)
Reduce model size or depth
Enable gradient checkpointing
Use mixed precision training
Implement gradient accumulation
Clear GPU cache: torch.cuda.empty_cache()

Issue: Overfitting

Symptoms: Train accuracy high, validation low

Solutions:

Increase dropout rate
Add L1/L2 regularization
Increase data augmentation
Use early stopping
Reduce model capacity
Collect more training data

Interview Questions & Answers

Q1: Explain the backpropagation algorithm

A: Backpropagation computes gradients efficiently using the chain rule:

Forward pass: compute output and activations
Compute loss at output
Backward pass: compute gradient of loss w.r.t. each weight

dL/dw = dL/da * da/dz * dz/dw

Update weights using gradient descent

Time complexity: O(n) same as forward pass Memory: stores activations for backward pass

Example

dL/dw_l = dL/da_l * da_l/dz_l * dz_l/dw_l

Q2: What's vanishing gradients and how to fix it?

A: In deep networks, gradients shrink exponentially going backward due to chain rule multiplying small numbers.

Causes: Sigmoid/Tanh derivative <= 0.25, many layers multiply small gradients

Solutions:

Use ReLU (derivative = 1 for positive)
Residual connections (skip layers)
Batch normalization (normalize activations)
Weight initialization (Xavier, He)
Gradient clipping (limit magnitude)

Q3: Explain batch normalization

A: Normalizes layer inputs to have mean 0, std 1:

Reduces internal covariate shift
Allows higher learning rates
Acts as regularization
Accelerates training

During training: use batch statistics During inference: use moving averages from training

Q4: How do you prevent overfitting?

Data augmentation: Generate variations of training data
Regularization: L1/L2 penalties on weights
Dropout: Randomly deactivate neurons during training
Early stopping: Monitor validation loss
Reduce capacity: Fewer parameters
Cross-validation: Multiple train/val splits

Q5: Difference between SGD, Momentum, and Adam?

SGD: Simple gradient descent, good generalization
Momentum: Accumulates gradients (faster convergence)
Adam: Adaptive learning rates per parameter (usually converges fastest)

Choose based on problem characteristics and computational budget.

Q6: How would you debug non-converging training?

Test on single batch: should overfit quickly
Check gradients: should be non-zero and finite
Verify loss: random predictions should give baseline
Visualize activations: should have reasonable range
Check data: verify shapes, normalization
Try smaller learning rate

Q7: Explain transfer learning

A: Reuse knowledge from pre-trained model on different task:

Feature extraction: Freeze backbone, train only head
Fine-tuning: Train all layers with low learning rate
Progressive unfreezing: Gradually unfreeze layers

Benefits: Fast convergence, works with limited data

Q8: What's the difference between training and inference?

Training: Compute gradients, use batch stats (dropout active)
Inference: No gradients, use moving averages (dropout inactive)

Critical: Call model.eval() before inference

Q9: How do you handle imbalanced datasets?

Weighted loss: Give more weight to minority class
Data augmentation: Oversample minority class
Different metrics: Use F1-score, AUC-ROC instead of accuracy
Threshold adjustment: Adjust decision boundary
Ensemble methods: Combine models

Q10: Explain attention mechanism

A: Allows model to focus on important parts:

Query, Key, Value projections
Attention weights: softmax(QK^T / sqrt(d_k))
Output: weighted sum of values
Enables long-range dependencies
Foundation of Transformers

*Last Updated: 2024 | Review Status: Current with latest practices*

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Activation Function Comparison Guide.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Deep Learning topic.

Search Terms

deep-learning, deep learning, deep, learning, activation, functions, function, comparison

Continue learning this concept

Activation FunctionsReLU Activation FunctionUnderstanding Rectified Linear Unit (ReLU) - the most popular activation function in deep learning, its variants, advantages, and the dying ReLU problem.Activation FunctionsSigmoid Activation FunctionDeep dive into the sigmoid activation function - mathematical properties, derivatives, use cases, limitations, and implementation in neural networks.Activation FunctionsSoftmax Function for Multi-class ClassificationComprehensive guide to softmax function for multi-class classification in deep learning Activation FunctionsTanh Activation FunctionComprehensive guide to tanh activation function in deep learning Activation FunctionsActivation Functions OverviewComplete overview of activation functions in deep learning - why they are needed, types, properties, and how to choose the right one for your neural network.Activation FunctionsLeaky ReLU: Solving Dead NeuronsComprehensive guide to leaky relu: solving dead neurons in deep learning

NextActivation Functions Overview

DL Notes

Activation Function Comparison Guide

Comprehensive guide to activation function comparison guide in deep learning

Overview

Fundamental Concepts

Core Principles

The mathematical and computational foundations are rooted in:

Linear algebra: Matrix operations, eigenvalues, transformations
Calculus: Derivatives, chain rule, partial derivatives
Optimization: Gradient descent, convergence, stability
Numerical methods: Precision, efficiency, scalability

Understanding these principles enables you to:

Diagnose training issues quickly
Optimize model performance systematically
Make informed architecture decisions
Contribute to research effectively

Architecture Overview

PyTorch Implementation

Model Definition

python example

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader

# Device setup
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Define neural network model
class DeepLearningModel(nn.Module):
    def __init__(self, input_size=784, hidden_size=512, num_classes=10):
        super(DeepLearningModel, self).__init__()
        
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.bn1 = nn.BatchNorm1d(hidden_size)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(0.3)
        
        self.fc2 = nn.Linear(hidden_size, 256)
        self.bn2 = nn.BatchNorm1d(256)
        
        self.fc3 = nn.Linear(256, 128)
        self.bn3 = nn.BatchNorm1d(128)
        
        self.fc4 = nn.Linear(128, num_classes)
    
    def forward(self, x):
        x = x.view(x.size(0), -1)
        x = self.relu(self.bn1(self.fc1(x)))
        x = self.dropout(x)
        x = self.relu(self.bn2(self.fc2(x)))
        x = self.dropout(x)
        x = self.relu(self.bn3(self.fc3(x)))
        x = self.fc4(x)
        return x

model = DeepLearningModel().to(device)

Training Loop

python example

def train_model(model, train_loader, val_loader, num_epochs=50):
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=0.001)
    scheduler = optim.lr_scheduler.ReduceLROnPlateau(
        optimizer, mode='min', factor=0.5, patience=5, verbose=True
    )
    
    best_val_loss = float('inf')
    
    for epoch in range(num_epochs):
        # Training phase
        model.train()
        train_loss = 0.0
        train_correct = 0
        train_total = 0
        
        for images, labels in train_loader:
            images, labels = images.to(device), labels.to(device)
            
            # Forward pass
            outputs = model(images)
            loss = criterion(outputs, labels)
            
            # Backward pass and optimization
            optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
            optimizer.step()
            
            # Statistics
            train_loss += loss.item()
            _, predicted = torch.max(outputs.data, 1)
            train_total += labels.size(0)
            train_correct += (predicted == labels).sum().item()
        
        # Validation phase
        model.eval()
        val_loss = 0.0
        val_correct = 0
        val_total = 0
        
        with torch.no_grad():
            for images, labels in val_loader:
                images, labels = images.to(device), labels.to(device)
                outputs = model(images)
                loss = criterion(outputs, labels)
                
                val_loss += loss.item()
                _, predicted = torch.max(outputs.data, 1)
                val_total += labels.size(0)
                val_correct += (predicted == labels).sum().item()
        
        avg_train_loss = train_loss / len(train_loader)
        avg_val_loss = val_loss / len(val_loader)
        train_acc = 100 * train_correct / train_total
        val_acc = 100 * val_correct / val_total
        
        print(f'Epoch {epoch+1}/{num_epochs}:')
        print(f'  Train Loss: {avg_train_loss:.4f}, Train Acc: {train_acc:.2f}%')
        print(f'  Val Loss: {avg_val_loss:.4f}, Val Acc: {val_acc:.2f}%')
        
        # Learning rate scheduling
        scheduler.step(avg_val_loss)
        
        # Save best model
        if avg_val_loss < best_val_loss:
            best_val_loss = avg_val_loss
            torch.save(model.state_dict(), 'best_model.pth')
    
    return model

TensorFlow/Keras Implementation

Model Creation

Example

Training with Callbacks

Example

Performance Comparison

Model Architecture Comparison

Model	Parameters	Speed (img/s)	Memory (MB)	Accuracy
ResNet-50	25.5M	750	4,200	76.1%
VGG-16	138M	120	8,500	71.3%
MobileNet	4.2M	3,000	1,200	70.2%
EfficientNet-B0	5.3M	800	1,800	77.1%
Inception-V3	27.2M	600	3,900	78.0%

Hyperparameter Impact

Parameter	Min	Max	Recommended	Impact
Learning Rate	1e-5	1e-1	1e-3	Very High
Batch Size	8	512	32-64	High
Dropout Rate	0.0	0.5	0.3	Medium
Weight Decay	1e-6	1e-2	1e-4	Medium
Momentum	0.0	0.99	0.9	Low

Real-World Applications

Application 1: Image Recognition

Use Case: Medical image analysis, autonomous vehicles
Architecture: CNN (ResNet, EfficientNet)
Performance: 94-99% accuracy depending on dataset
Key Challenge: Data annotation cost, domain adaptation

Application 2: Natural Language Processing

Use Case: Machine translation, sentiment analysis
Architecture: Transformers (BERT, GPT)
Performance: State-of-the-art on various benchmarks
Key Challenge: Context understanding, ambiguity resolution

Application 3: Time Series Forecasting

Use Case: Stock price prediction, weather forecasting
Architecture: LSTM, GRU, or Transformers
Performance: Depends on temporal complexity
Key Challenge: Capturing long-term dependencies

Advanced Techniques

Gradient Accumulation

python example

accumulation_steps = 4
optimizer.zero_grad()

for i, (images, labels) in enumerate(train_loader):
    outputs = model(images)
    loss = criterion(outputs, labels)
    loss = loss / accumulation_steps
    loss.backward()
    
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

Mixed Precision Training

python example

from torch.cuda.amp import autocast, GradScaler

scaler = GradScaler()
model = model.to(device)

for images, labels in train_loader:
    images, labels = images.to(device), labels.to(device)
    
    with autocast():
        outputs = model(images)
        loss = criterion(outputs, labels)
    
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()
    optimizer.zero_grad()

Troubleshooting Guide

Issue: Training Loss Not Decreasing

Symptoms: Loss stays constant or increases

Solutions:

Check data normalization: mean ~0, std ~1
Verify loss function is correct for task
Test on small batch to verify learning
Reduce learning rate systematically
Check for NaN values in activations
Verify labels are correct

Issue: Out of Memory

Solutions:

Reduce batch size (32 → 16)
Reduce model size or depth
Enable gradient checkpointing
Use mixed precision training
Implement gradient accumulation
Clear GPU cache: torch.cuda.empty_cache()

Issue: Overfitting

Symptoms: Train accuracy high, validation low

Solutions:

Increase dropout rate
Add L1/L2 regularization
Increase data augmentation
Use early stopping
Reduce model capacity
Collect more training data

Interview Questions & Answers

Q1: Explain the backpropagation algorithm

A: Backpropagation computes gradients efficiently using the chain rule:

Forward pass: compute output and activations
Compute loss at output
Backward pass: compute gradient of loss w.r.t. each weight

dL/dw = dL/da * da/dz * dz/dw

Update weights using gradient descent

Time complexity: O(n) same as forward pass Memory: stores activations for backward pass

Example

dL/dw_l = dL/da_l * da_l/dz_l * dz_l/dw_l

Q2: What's vanishing gradients and how to fix it?

A: In deep networks, gradients shrink exponentially going backward due to chain rule multiplying small numbers.

Causes: Sigmoid/Tanh derivative <= 0.25, many layers multiply small gradients

Solutions:

Use ReLU (derivative = 1 for positive)
Residual connections (skip layers)
Batch normalization (normalize activations)
Weight initialization (Xavier, He)
Gradient clipping (limit magnitude)

Q3: Explain batch normalization

A: Normalizes layer inputs to have mean 0, std 1:

Reduces internal covariate shift
Allows higher learning rates
Acts as regularization
Accelerates training

During training: use batch statistics During inference: use moving averages from training

Q4: How do you prevent overfitting?

Data augmentation: Generate variations of training data
Regularization: L1/L2 penalties on weights
Dropout: Randomly deactivate neurons during training
Early stopping: Monitor validation loss
Reduce capacity: Fewer parameters
Cross-validation: Multiple train/val splits

Q5: Difference between SGD, Momentum, and Adam?

SGD: Simple gradient descent, good generalization
Momentum: Accumulates gradients (faster convergence)
Adam: Adaptive learning rates per parameter (usually converges fastest)

Choose based on problem characteristics and computational budget.

Q6: How would you debug non-converging training?

Test on single batch: should overfit quickly
Check gradients: should be non-zero and finite
Verify loss: random predictions should give baseline
Visualize activations: should have reasonable range
Check data: verify shapes, normalization
Try smaller learning rate

Q7: Explain transfer learning

A: Reuse knowledge from pre-trained model on different task:

Feature extraction: Freeze backbone, train only head
Fine-tuning: Train all layers with low learning rate
Progressive unfreezing: Gradually unfreeze layers

Benefits: Fast convergence, works with limited data

Q8: What's the difference between training and inference?

Training: Compute gradients, use batch stats (dropout active)
Inference: No gradients, use moving averages (dropout inactive)

Critical: Call model.eval() before inference

Q9: How do you handle imbalanced datasets?

Weighted loss: Give more weight to minority class
Data augmentation: Oversample minority class
Different metrics: Use F1-score, AUC-ROC instead of accuracy
Threshold adjustment: Adjust decision boundary
Ensemble methods: Combine models

Q10: Explain attention mechanism

A: Allows model to focus on important parts:

Query, Key, Value projections
Attention weights: softmax(QK^T / sqrt(d_k))
Output: weighted sum of values
Enables long-range dependencies
Foundation of Transformers

*Last Updated: 2024 | Review Status: Current with latest practices*

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Activation Function Comparison Guide.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Deep Learning topic.

Search Terms

deep-learning, deep learning, deep, learning, activation, functions, function, comparison

Continue learning this concept

NextActivation Functions Overview