Forward Propagation - Inference in Networks

Master forward propagation for neural networks. Learn layer-by-layer computation, vectorization, implementation. AI fundamentals 2024.

What is Forward Propagation?

Forward propagation is the process by which a neural network computes its output given an input. Data flows in one direction—from the input layer through hidden layers to the output layer—with each layer performing a linear transformation followed by a nonlinear activation function.

Understanding forward propagation is fundamental because it defines what a neural network actually computes. Every prediction, every classification, every generated token results from a forward pass. If you understand forward propagation completely, you understand what a neural network is mathematically: a composition of parameterized functions.

The Mathematics of a Single Layer

Each layer performs two operations:

1. Linear transformation: z = W · a + b

2. Nonlinear activation: a' = σ(z)

Where:

W = weight matrix (connections between layers)

a = activations from previous layer (or input)

b = bias vector

σ = activation function (ReLU, sigmoid, tanh, etc.)

z = pre-activation (weighted sum)

a' = post-activation (output of this layer)

Why Two Steps?

The linear transformation alone (stacking matrix multiplications) can only compute linear functions. No matter how many layers you stack, W₃·W₂·W₁·x = W_combined·x—still linear. The nonlinear activation σ breaks this limitation, enabling the network to approximate any continuous function (universal approximation theorem).

Complete Forward Pass: 3-Layer Network Example

Network Architecture

Input layer	3 neurons (features: x₁, x₂, x₃)
Hidden layer 1	4 neurons (ReLU activation)
Hidden layer 2	3 neurons (ReLU activation)
Output layer	2 neurons (Softmax for classification)

Dimensions

W₁	4×3 (4 hidden neurons, each receiving 3 inputs)
b₁	4×1
W₂	3×4 (3 neurons receiving from 4)
b₂	3×1
W₃	2×3 (2 output neurons from 3)
b₃	2×1

Numerical Walkthrough

Input	x = [1.0, 0.5, -0.3]
Note: z₁[2] was negative	ReLU kills it to 0 (neuron is "dead" for this input)
Prediction	Class 0 (56.8% confidence)

Activation Functions: When to Use Which

ReLU (Rectified Linear Unit)

Pros	No vanishing gradient for z > 0, computationally simple
Cons	"Dying ReLU" — neurons stuck at 0 if weights push z negative
Use	Default choice for hidden layers

Leaky ReLU

Example

LeakyReLU(z) = z if z > 0, else 0.01z Fixes dying ReLU by allowing small negative gradients

Sigmoid

σ(z) = 1 / (1 + e^(-z)) Output	(0, 1)
Pros	Outputs interpretable as probabilities
Cons	Vanishing gradient at extremes, outputs not zero-centered
Use	Binary classification output layer, gates in LSTM

Softmax (for output layer)

Code example

Softmax(zᵢ) = e^(zᵢ) / Σⱼ e^(zⱼ)

Converts logits to probability distribution (sums to 1)
Use: Multi-class classification output

Vectorized Implementation

In practice, forward propagation operates on batches for efficiency:

Example

def forward_pass(X, weights, biases): """ X: input batch (batch_size × features) Returns: output predictions """ A = X # first activation is the input for l in range(num_layers - 1): Z = A @ weights[l].T + biases[l] # linear transform A = relu(Z) # activation # Output layer (softmax for classification) Z_out = A @ weights[-1].T + biases[-1] output = softmax(Z_out) return output # Batch processing: 32 inputs simultaneously # X shape: (32, 3) — 32 examples, 3 features each # All matrix operations parallelized on GPU

Why Batched Processing?

Single input	32 separate matrix multiplications
Batch of 32	One large matrix multiplication
Single vector × matrix	~1% GPU utilization
Batch matrix × matrix	~90% GPU utilization
Speedup	50-100× for batch sizes of 32-256

The Role of Bias

Bias terms shift the activation function:

Without bias: z = Wx	activation always passes through origin
With bias: z = Wx + b	activation can shift left/right
No bias	ReLU(2x) activates only for x > 0
Bias -3	ReLU(2x - 3) activates only for x > 1.5

Forward Propagation in Different Architectures

Convolutional Neural Networks

Example

Instead of full matrix multiply: z = conv2d(input, kernel) + bias a = ReLU(z) a = max_pool(a) Same principle: linear op → nonlinearity → next layer

Recurrent Neural Networks

Example

Forward pass THROUGH TIME: h_t = tanh(W_hh · h_{t-1} + W_xh · x_t + b) Each timestep is one "layer" of forward prop Weights shared across timesteps

Computational Complexity

For a layer: n_in inputs	n_out outputs
Matrix multiply	O(n_in × n_out) multiplications
Bias addition	O(n_out)
Activation	O(n_out)
Example	[784, 256, 128, 10] network:

Interview Questions

Q: What happens during forward propagation if all weights are initialized to zero? A: Every neuron computes the same output (all zeros after multiplication). During backprop, all neurons receive identical gradients and learn identical weights. The network degenerates to a single effective neuron per layer. This is called the symmetry problem—broken by random initialization.

Q: Why do we need nonlinear activation functions? A: Without nonlinearities, a deep network collapses to a single linear transformation (matrix multiplication is associative). A 100-layer linear network equals one layer with W = W₁₀₀×...×W₁. Nonlinearities give each layer independent representational power.

Q: What is the output of forward propagation used for? A: Two purposes—(1) Making predictions at inference time (the final answer), and (2) Computing the loss during training, which triggers backpropagation to update weights. The stored intermediate activations are essential for efficient gradient computation.

What is Forward Propagation?

The Mathematics of a Single Layer

Each layer performs two operations:

1. Linear transformation: z = W · a + b

2. Nonlinear activation: a' = σ(z)

Where:

W = weight matrix (connections between layers)

a = activations from previous layer (or input)

b = bias vector

σ = activation function (ReLU, sigmoid, tanh, etc.)

z = pre-activation (weighted sum)

a' = post-activation (output of this layer)

Why Two Steps?

Complete Forward Pass: 3-Layer Network Example

Network Architecture

Input layer	3 neurons (features: x₁, x₂, x₃)
Hidden layer 1	4 neurons (ReLU activation)
Hidden layer 2	3 neurons (ReLU activation)
Output layer	2 neurons (Softmax for classification)

Dimensions

W₁	4×3 (4 hidden neurons, each receiving 3 inputs)
b₁	4×1
W₂	3×4 (3 neurons receiving from 4)
b₂	3×1
W₃	2×3 (2 output neurons from 3)
b₃	2×1

Numerical Walkthrough

Input	x = [1.0, 0.5, -0.3]
Note: z₁[2] was negative	ReLU kills it to 0 (neuron is "dead" for this input)
Prediction	Class 0 (56.8% confidence)

Activation Functions: When to Use Which

ReLU (Rectified Linear Unit)

Pros	No vanishing gradient for z > 0, computationally simple
Cons	"Dying ReLU" — neurons stuck at 0 if weights push z negative
Use	Default choice for hidden layers

Leaky ReLU

Example

LeakyReLU(z) = z if z > 0, else 0.01z Fixes dying ReLU by allowing small negative gradients

Sigmoid

σ(z) = 1 / (1 + e^(-z)) Output	(0, 1)
Pros	Outputs interpretable as probabilities
Cons	Vanishing gradient at extremes, outputs not zero-centered
Use	Binary classification output layer, gates in LSTM

Softmax (for output layer)

Code example

Softmax(zᵢ) = e^(zᵢ) / Σⱼ e^(zⱼ)

Converts logits to probability distribution (sums to 1)
Use: Multi-class classification output

Vectorized Implementation

In practice, forward propagation operates on batches for efficiency:

Example

Why Batched Processing?

Single input	32 separate matrix multiplications
Batch of 32	One large matrix multiplication
Single vector × matrix	~1% GPU utilization
Batch matrix × matrix	~90% GPU utilization
Speedup	50-100× for batch sizes of 32-256

The Role of Bias

Bias terms shift the activation function:

Without bias: z = Wx	activation always passes through origin
With bias: z = Wx + b	activation can shift left/right
No bias	ReLU(2x) activates only for x > 0
Bias -3	ReLU(2x - 3) activates only for x > 1.5

Forward Propagation in Different Architectures

Convolutional Neural Networks

Example

Instead of full matrix multiply: z = conv2d(input, kernel) + bias a = ReLU(z) a = max_pool(a) Same principle: linear op → nonlinearity → next layer

Recurrent Neural Networks

Example

Forward pass THROUGH TIME: h_t = tanh(W_hh · h_{t-1} + W_xh · x_t + b) Each timestep is one "layer" of forward prop Weights shared across timesteps

Computational Complexity

For a layer: n_in inputs	n_out outputs
Matrix multiply	O(n_in × n_out) multiplications
Bias addition	O(n_out)
Activation	O(n_out)
Example	[784, 256, 128, 10] network:

What is Forward Propagation?

The Mathematics of a Single Layer

Why Two Steps?

Complete Forward Pass: 3-Layer Network Example

Network Architecture

Dimensions

Numerical Walkthrough

Activation Functions: When to Use Which

ReLU (Rectified Linear Unit)

Leaky ReLU

Sigmoid

Softmax (for output layer)

Vectorized Implementation

Why Batched Processing?

The Role of Bias

Forward Propagation in Different Architectures

Convolutional Neural Networks

Recurrent Neural Networks

Computational Complexity

Interview Questions

Continue learning this concept

What is Forward Propagation?

The Mathematics of a Single Layer

Why Two Steps?

Complete Forward Pass: 3-Layer Network Example

Network Architecture

Dimensions

Numerical Walkthrough

Activation Functions: When to Use Which

ReLU (Rectified Linear Unit)

Leaky ReLU

Sigmoid

Softmax (for output layer)

Vectorized Implementation

Why Batched Processing?

The Role of Bias

Forward Propagation in Different Architectures

Convolutional Neural Networks

Recurrent Neural Networks

Computational Complexity

Interview Questions

Continue learning this concept