DL Notes

Hub Activation Function Comparison Guide Activation Functions Overview Leaky ReLU: Solving Dead Neurons ReLU Activation Function Sigmoid Activation Function Softmax Function for Multi-class Classification Tanh Activation Function Computer Vision Projects with Deep Learning Face Recognition with Deep Learning Faster R-CNN Object Detection Image Processing Fundamentals Image Segmentation — Deep Learning Object Detection Architectures U-Net Architecture YOLO Real-time Object Detection AlexNet CNN Architecture CNN Architecture Design Introduction to CNNs Convolution Operation Image Classification Project LeNet Architecture Pooling Layers ResNet and Residual Networks Transfer Learning in Deep Learning VGGNet Architecture GPU Acceleration for Deep Learning Keras Deep Learning Framework PyTorch Deep Learning Framework Setting Up Deep Learning Environment TensorFlow Framework Guide TensorFlow vs PyTorch Comparison Deploying Deep Learning Models to Cloud Deep Learning APIs and Services Docker For Deep Learning Fastapi For Deployment Flask For Deployment Model Deployment Model Monitoring Ai Agent Basics Autoencoders Diffusion Models Generative Adversarial Networks Gan Generative Ai Introduction Large Language Models Llms Prompt Engineering Variational Autoencoders Cnn Interview Questions Deep Learning Cheatsheet Deep Learning Interview Questions — Deep Learning Neural Network Questions Transformer Interview Questions AI vs ML vs Deep Learning — Deep Learning Applications of Deep Learning Deep Learning Roadmap History of Deep Learning What is Deep Learning?Batch Normalization Explained Dropout Hyperparameter Tuning Model Debugging Model Evaluation Metrics Regularization Methods Train Validation Test Split Chatbot Development Glove Nlp Projects Nlp With Deep Learning Sentiment Analysis Seq2Seq Models Word Embeddings Word2Vec Artificial Neural Networks (ANN)Backpropagation Biological vs Artificial Neurons Forward Propagation Gradient Descent Layers in Neural Networks Loss Functions in Deep Learning The Perceptron Adam Optimizer Batch Gradient Descent Learning Rate Scheduling Mini Batch Gradient Descent Momentum Optimization Basics Rmsprop Stochastic Gradient Descent Calculus for Deep Learning Data Preprocessing for Deep Learning Linear Algebra for Deep Learning NumPy Basics for Deep Learning Probability and Statistics for Deep Learning Python for Deep Learning Ai Chatbot Project Face Mask Detection Handwritten Digit Recognition Image Classification System Object Detection System Personal Ai Assistant Recommendation System — Deep Learning Sentiment Analysis System Bidirectional Rnn GRU Networks LSTM Networks Rnn Introduction Sequence Modeling Text Generation Project Vanishing Gradient Problem Datasets For Practice Deep Learning Glossary Important Formulas — Deep Learning Recommended Books And Papers Attention Mechanism Bert Model Encoder Decoder Architecture Gpt Models Positional Encoding Self Attention Transformer Projects Transformers Introduction

Sigmoid Activation Function

Deep dive into the sigmoid activation function - mathematical properties, derivatives, use cases, limitations, and implementation in neural networks.

The sigmoid function (also called the logistic function) maps any real number to a value between 0 and 1. It was historically the most popular activation function but has been largely replaced by ReLU in hidden layers.

Mathematical Definition

Example

sigma(x) = 1 / (1 + e^(-x)) Domain: (-infinity, +infinity) Range: (0, 1)

Properties

Example

1.0 ─────────────────────────......... | ../ | ./ 0.5 ─|─ ─ ─ ─ ─ ─ ─ ─/─ ─ ─ ─ ─ ─ ─ | ./ | ../ 0.0 ─.........───────────────────────── -6 -4 -2 0 2 4 6

Key properties:

Output bounded between 0 and 1 (probability interpretation)
Monotonically increasing
Differentiable everywhere
sigma(0) = 0.5
sigma(-x) = 1 - sigma(x)

Derivative of Sigmoid

Example

sigma'(x) = sigma(x) * (1 - sigma(x)) Maximum derivative: 0.25 (at x = 0) Approaches 0 as |x| grows large

python example

import numpy as np
import torch

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(x):
    s = sigmoid(x)
    return s * (1 - s)

# Demonstrate
x = np.linspace(-6, 6, 100)
print(f"sigmoid(0) = {sigmoid(0):.4f}")
print(f"sigmoid(5) = {sigmoid(5):.6f}")
print(f"sigmoid(-5) = {sigmoid(-5):.6f}")
print(f"Max derivative = {sigmoid_derivative(0):.4f}")
print(f"Derivative at x=5: {sigmoid_derivative(5):.6f}")

PyTorch Implementation

Example

import torch import torch.nn as nn # Using PyTorch sigmoid sigmoid_layer = nn.Sigmoid() x = torch.tensor([-3.0, -1.0, 0.0, 1.0, 3.0], requires_grad=True) output = sigmoid_layer(x) print(f"Input: {x.data}") print(f"Output: {output.data}") # Compute gradients output.sum().backward() print(f"Gradients: {x.grad}") # In a neural network (output layer for binary classification) class BinaryClassifier(nn.Module): def __init__(self, input_dim): super().__init__() self.network = nn.Sequential( nn.Linear(input_dim, 64), nn.ReLU(), nn.Linear(64, 32), nn.ReLU(), nn.Linear(32, 1), nn.Sigmoid() # Sigmoid on output for probability ) def forward(self, x): return self.network(x) model = BinaryClassifier(10) sample = torch.randn(5, 10) probabilities = model(sample) print(f"Predictions (probabilities): {probabilities.squeeze().detach()}")

The Vanishing Gradient Problem

Layer L gradient	sigma'(z) = max 0.25
Layer L-1 gradient	sigma'(z) * sigma'(z) = max 0.0625
Layer L-2 gradient	sigma'(z)^3 = max 0.015625
Layer 1 gradient	sigma'(z)^n --> nearly 0 for large n

When to Use Sigmoid

Use Case	Appropriate?	Reason
Output layer (binary classification)	Yes	Outputs probability [0,1]
Hidden layers (deep networks)	No	Vanishing gradients
Gates in LSTM/GRU	Yes	Need 0-1 gating values
Shallow networks (1-2 layers)	Acceptable	Less vanishing gradient risk
Multi-label output	Yes	Independent probabilities per class

Numerical Stability

python example

import numpy as np

# Numerically stable sigmoid
def stable_sigmoid(x):
    """Avoids overflow in exp(-x) for large positive x."""
    return np.where(
        x >= 0,
        1 / (1 + np.exp(-x)),
        np.exp(x) / (1 + np.exp(x))
    )

# Test with extreme values
print(f"sigmoid(100) = {stable_sigmoid(100)}")    # ~1.0
print(f"sigmoid(-100) = {stable_sigmoid(-100)}")  # ~0.0
print(f"sigmoid(1000) = {stable_sigmoid(1000)}")  # 1.0 (no overflow)

Sigmoid vs Other Activations

Property	Sigmoid	Tanh	ReLU
Range	(0, 1)	(-1, 1)	[0, inf)
Zero-centered	No	Yes	No
Max gradient	0.25	1.0	1.0
Vanishing gradient	Severe	Moderate	None (positive)
Computation cost	Expensive (exp)	Expensive (exp)	Cheap (max)
Dead neurons	No	No	Yes

Interview Questions

Why is sigmoid not used in hidden layers of deep networks?

Its maximum derivative is 0.25, causing gradients to shrink exponentially through layers (vanishing gradient problem), making deep networks nearly untrainable.

Why is sigmoid output not zero-centered and why does it matter?

Sigmoid outputs are always positive (0,1). This means gradients for weights are always all-positive or all-negative, causing zigzag updates in gradient descent.

What is the relationship between sigmoid and logistic regression?

Logistic regression uses sigmoid to convert linear predictions into probabilities. A single neuron with sigmoid activation is exactly logistic regression.

How does sigmoid relate to softmax?

Sigmoid is the special case of softmax with 2 classes. For binary: sigmoid(x) = softmax([x, 0])[0].

Why is BCEWithLogitsLoss preferred over applying sigmoid then BCELoss?

Combining log and sigmoid in one operation (log-sum-exp trick) avoids numerical instability from log(0) or log(1) edge cases.

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Sigmoid Activation Function.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Deep Learning topic.

Search Terms

deep-learning, deep learning, deep, learning, activation, functions, sigmoid, function

Continue learning this concept

Activation FunctionsActivation Function Comparison GuideComprehensive guide to activation function comparison guide in deep learning Activation FunctionsReLU Activation FunctionUnderstanding Rectified Linear Unit (ReLU) - the most popular activation function in deep learning, its variants, advantages, and the dying ReLU problem.Activation FunctionsSoftmax Function for Multi-class ClassificationComprehensive guide to softmax function for multi-class classification in deep learning Activation FunctionsTanh Activation FunctionComprehensive guide to tanh activation function in deep learning Activation FunctionsActivation Functions OverviewComplete overview of activation functions in deep learning - why they are needed, types, properties, and how to choose the right one for your neural network.Activation FunctionsLeaky ReLU: Solving Dead NeuronsComprehensive guide to leaky relu: solving dead neurons in deep learning

PreviousReLU Activation Function NextSoftmax Function for Multi-class Classification

DL Notes

Sigmoid Activation Function

Deep dive into the sigmoid activation function - mathematical properties, derivatives, use cases, limitations, and implementation in neural networks.

Mathematical Definition

Example

sigma(x) = 1 / (1 + e^(-x)) Domain: (-infinity, +infinity) Range: (0, 1)

Properties

Example

Key properties:

Output bounded between 0 and 1 (probability interpretation)
Monotonically increasing
Differentiable everywhere
sigma(0) = 0.5
sigma(-x) = 1 - sigma(x)

Derivative of Sigmoid

Example

sigma'(x) = sigma(x) * (1 - sigma(x)) Maximum derivative: 0.25 (at x = 0) Approaches 0 as |x| grows large

python example

import numpy as np
import torch

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(x):
    s = sigmoid(x)
    return s * (1 - s)

# Demonstrate
x = np.linspace(-6, 6, 100)
print(f"sigmoid(0) = {sigmoid(0):.4f}")
print(f"sigmoid(5) = {sigmoid(5):.6f}")
print(f"sigmoid(-5) = {sigmoid(-5):.6f}")
print(f"Max derivative = {sigmoid_derivative(0):.4f}")
print(f"Derivative at x=5: {sigmoid_derivative(5):.6f}")

PyTorch Implementation

Example

The Vanishing Gradient Problem

Layer L gradient	sigma'(z) = max 0.25
Layer L-1 gradient	sigma'(z) * sigma'(z) = max 0.0625
Layer L-2 gradient	sigma'(z)^3 = max 0.015625
Layer 1 gradient	sigma'(z)^n --> nearly 0 for large n

When to Use Sigmoid

Use Case	Appropriate?	Reason
Output layer (binary classification)	Yes	Outputs probability [0,1]
Hidden layers (deep networks)	No	Vanishing gradients
Gates in LSTM/GRU	Yes	Need 0-1 gating values
Shallow networks (1-2 layers)	Acceptable	Less vanishing gradient risk
Multi-label output	Yes	Independent probabilities per class

Numerical Stability

python example

import numpy as np

# Numerically stable sigmoid
def stable_sigmoid(x):
    """Avoids overflow in exp(-x) for large positive x."""
    return np.where(
        x >= 0,
        1 / (1 + np.exp(-x)),
        np.exp(x) / (1 + np.exp(x))
    )

# Test with extreme values
print(f"sigmoid(100) = {stable_sigmoid(100)}")    # ~1.0
print(f"sigmoid(-100) = {stable_sigmoid(-100)}")  # ~0.0
print(f"sigmoid(1000) = {stable_sigmoid(1000)}")  # 1.0 (no overflow)

Sigmoid vs Other Activations

Property	Sigmoid	Tanh	ReLU
Range	(0, 1)	(-1, 1)	[0, inf)
Zero-centered	No	Yes	No
Max gradient	0.25	1.0	1.0
Vanishing gradient	Severe	Moderate	None (positive)
Computation cost	Expensive (exp)	Expensive (exp)	Cheap (max)
Dead neurons	No	No	Yes

Interview Questions

Why is sigmoid not used in hidden layers of deep networks?

Its maximum derivative is 0.25, causing gradients to shrink exponentially through layers (vanishing gradient problem), making deep networks nearly untrainable.

Why is sigmoid output not zero-centered and why does it matter?

Sigmoid outputs are always positive (0,1). This means gradients for weights are always all-positive or all-negative, causing zigzag updates in gradient descent.

What is the relationship between sigmoid and logistic regression?

Logistic regression uses sigmoid to convert linear predictions into probabilities. A single neuron with sigmoid activation is exactly logistic regression.

How does sigmoid relate to softmax?

Sigmoid is the special case of softmax with 2 classes. For binary: sigmoid(x) = softmax([x, 0])[0].

Why is BCEWithLogitsLoss preferred over applying sigmoid then BCELoss?

Combining log and sigmoid in one operation (log-sum-exp trick) avoids numerical instability from log(0) or log(1) edge cases.

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Sigmoid Activation Function.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Deep Learning topic.

Search Terms

deep-learning, deep learning, deep, learning, activation, functions, sigmoid, function

Continue learning this concept

PreviousReLU Activation Function NextSoftmax Function for Multi-class Classification