Activation Functions - Non-linearity in Networks

Introduction

Activation functions are mathematical functions applied to the output of each neuron in a neural network. Without activation functions, a neural network would simply be a series of linear transformations, and no matter how many layers you stack, the result would still be a linear function. Activation functions introduce non-linearity, enabling networks to learn complex patterns like image recognition, language understanding, and game playing.

Think of it this way: if you combine multiple linear functions (y = ax + b) by composition, you always get another linear function. But real-world problems are rarely linear. The boundary between "cat" and "dog" in pixel space is a complex, curved surface. Activation functions give networks the power to learn these curved decision boundaries.

The Major Activation Functions

Sigmoid (Logistic Function)

Sigmoid squashes any input to a value between 0 and 1, making it useful for probability outputs. However, it has serious problems for hidden layers: the maximum gradient is only 0.25, causing gradients to shrink rapidly in deep networks (vanishing gradient problem). Also, outputs are not zero-centered, which slows down gradient descent.

Tanh (Hyperbolic Tangent)

Tanh is zero-centered (unlike sigmoid), which helps gradient descent converge faster. Its maximum gradient of 1.0 is better than sigmoid's 0.25. However, it still saturates for large inputs, causing vanishing gradients in very deep networks.

ReLU (Rectified Linear Unit)

ReLU revolutionized deep learning. Its gradient is exactly 1 for positive inputs, eliminating the vanishing gradient problem. It is also computationally trivial compared to computing exponentials. Most modern deep networks use ReLU or its variants in hidden layers.

Leaky ReLU

Leaky ReLU addresses the "dying ReLU" problem by allowing a small gradient for negative inputs.

Softmax (Multi-Class Output)

Softmax is used exclusively in the output layer for multi-class classification. It ensures outputs are positive and sum to 1, making them interpretable as class probabilities.

The Dead ReLU Problem

When a neuron's weights are updated such that the weighted sum is always negative, the ReLU output is always zero. Since the gradient is also zero for negative inputs, the weights never update again. The neuron is permanently "dead."

Choosing the Right Activation Function

Worked Example: How Activation Choice Affects Learning

Consider a 5-layer network with all sigmoid activations vs all ReLU:

Modern Activation Functions

GELU (Gaussian Error Linear Unit): Used in Transformers (BERT, GPT). Smoothly gates values based on their magnitude: GELU(x) = x × Φ(x) where Φ is the Gaussian CDF.

Swish: f(x) = x × sigmoid(x). Self-gated, smooth, found by neural architecture search at Google. Slightly outperforms ReLU in deep networks.

Mish: f(x) = x × tanh(softplus(x)). Similar benefits to Swish with slightly better performance in some vision tasks.

Summary

Activation functions are what give neural networks their power to learn complex, non-linear patterns. ReLU and its variants dominate hidden layers due to their computational efficiency and gradient-friendly properties. Sigmoid and softmax serve specific roles in output layers for classification tasks. Understanding why each activation function works (and fails) in different contexts is essential for designing effective neural network architectures.