DL Notes
HubActivation Function Comparison GuideActivation Functions OverviewLeaky ReLU: Solving Dead NeuronsReLU Activation FunctionSigmoid Activation FunctionSoftmax Function for Multi-class ClassificationTanh Activation FunctionComputer Vision Projects with Deep LearningFace Recognition with Deep LearningFaster R-CNN Object DetectionImage Processing FundamentalsImage Segmentation — Deep LearningObject Detection ArchitecturesU-Net ArchitectureYOLO Real-time Object DetectionAlexNet CNN ArchitectureCNN Architecture DesignIntroduction to CNNsConvolution OperationImage Classification ProjectLeNet ArchitecturePooling LayersResNet and Residual NetworksTransfer Learning in Deep LearningVGGNet ArchitectureGPU Acceleration for Deep LearningKeras Deep Learning FrameworkPyTorch Deep Learning FrameworkSetting Up Deep Learning EnvironmentTensorFlow Framework GuideTensorFlow vs PyTorch ComparisonDeploying Deep Learning Models to CloudDeep Learning APIs and ServicesDocker For Deep LearningFastapi For DeploymentFlask For DeploymentModel DeploymentModel MonitoringAi Agent BasicsAutoencodersDiffusion ModelsGenerative Adversarial Networks GanGenerative Ai IntroductionLarge Language Models LlmsPrompt EngineeringVariational AutoencodersCnn Interview QuestionsDeep Learning CheatsheetDeep Learning Interview Questions — Deep LearningNeural Network QuestionsTransformer Interview QuestionsAI vs ML vs Deep Learning — Deep LearningApplications of Deep LearningDeep Learning RoadmapHistory of Deep LearningWhat is Deep Learning?Batch Normalization ExplainedDropoutHyperparameter TuningModel DebuggingModel Evaluation MetricsRegularization MethodsTrain Validation Test SplitChatbot DevelopmentGloveNlp ProjectsNlp With Deep LearningSentiment AnalysisSeq2Seq ModelsWord EmbeddingsWord2VecArtificial Neural Networks (ANN)BackpropagationBiological vs Artificial NeuronsForward PropagationGradient DescentLayers in Neural NetworksLoss Functions in Deep LearningThe PerceptronAdam OptimizerBatch Gradient DescentLearning Rate SchedulingMini Batch Gradient DescentMomentumOptimization BasicsRmspropStochastic Gradient DescentCalculus for Deep LearningData Preprocessing for Deep LearningLinear Algebra for Deep LearningNumPy Basics for Deep LearningProbability and Statistics for Deep LearningPython for Deep LearningAi Chatbot ProjectFace Mask DetectionHandwritten Digit RecognitionImage Classification SystemObject Detection SystemPersonal Ai AssistantRecommendation System — Deep LearningSentiment Analysis SystemBidirectional RnnGRU NetworksLSTM NetworksRnn IntroductionSequence ModelingText Generation ProjectVanishing Gradient ProblemDatasets For PracticeDeep Learning GlossaryImportant Formulas — Deep LearningRecommended Books And PapersAttention MechanismBert ModelEncoder Decoder ArchitectureGpt ModelsPositional EncodingSelf AttentionTransformer ProjectsTransformers Introduction
Complete guide to Artificial Neural Networks — understanding architecture, how information flows through layers, and building your first network from scratch.
An Artificial Neural Network is a computational model inspired by the human brain — a collection of interconnected nodes (neurons) organized in layers that learn to map inputs to outputs by adjusting connection strengths (weights). It's the fundamental building block of all deep learning.
Architecture Overview
Every feedforward neural network has three types of layers:
Input Layer Hidden Layers Output Layer
(features) (learned repr.) (predictions)
x₁ ─────┐ ┌─── h₁ ───┐ ┌─── h₁ ───┐
├───┤ ├───┤ ├──── ŷ₁
x₂ ─────┤ ├─── h₂ ───┤ ├─── h₂ ───┤
├───┤ ├───┤ ├──── ŷ₂
x₃ ─────┤ ├─── h₃ ───┤ ├─── h₃ ───┤
├───┤ ├───┤ ├──── ŷ₃
x₄ ─────┘ └─── h₄ ───┘ └─── h₄ ───┘
Layer 0 Layer 1 Layer 2 Layer 3
- Input layer: Receives raw features (pixel values, word embeddings, numerical data)
- Hidden layers: Transform representations through learned weights + non-linear activations
- Output layer: Produces final predictions (class probabilities, regression values)
How a Single Neuron Works
Each neuron performs three operations:
- Weighted sum: Multiply each input by its weight and sum
- Add bias: Shift the sum by a learnable bias term
- Activation: Apply a non-linear function
$$z = \sum_{i=1}^{n} w_i x_i + b = \mathbf{w}^T\mathbf{x} + b$$
$$a = \sigma(z)$$
Where σ is an activation function (ReLU, sigmoid, tanh).
python example
import numpy as np
def neuron(x, w, b, activation='relu'):
"""Single neuron computation"""
z = np.dot(w, x) + b # Linear transformation
if activation == 'relu':
return max(0, z)
elif activation == 'sigmoid':
return 1 / (1 + np.exp(-z))
elif activation == 'tanh':
return np.tanh(z)Why Multiple Layers?
A single neuron can only learn linear decision boundaries. Stacking layers with non-linear activations allows the network to learn arbitrarily complex functions.
Universal Approximation Theorem: A neural network with at least one hidden layer and a non-linear activation can approximate any continuous function to arbitrary precision (given enough neurons).
However, depth is more efficient than width — a deep network with fewer total neurons can represent functions that a shallow network would need exponentially many neurons to learn.
Building an ANN from Scratch
Example
import numpy as np
class ArtificialNeuralNetwork:
def __init__(self, layer_sizes):
"""
layer_sizes: list like [784, 128, 64, 10]
Creates a network with len(layer_sizes)-1 weight matrices
"""
self.weights = []
self.biases = []
for i in range(len(layer_sizes) - 1):
# He initialization for ReLU
W = np.random.randn(layer_sizes[i], layer_sizes[i+1]) * np.sqrt(2.0 / layer_sizes[i])
b = np.zeros(layer_sizes[i+1])
self.weights.append(W)
self.biases.append(b)
def relu(self, z):
return np.maximum(0, z)
def relu_derivative(self, z):
return (z > 0).astype(float)
def softmax(self, z):
exp_z = np.exp(z - z.max(axis=1, keepdims=True))
return exp_z / exp_z.sum(axis=1, keepdims=True)
def forward(self, X):
"""Forward pass through all layers"""
self.activations = [X]
self.z_values = []
current = X
for i in range(len(self.weights) - 1):
z = current @ self.weights[i] + self.biases[i]
self.z_values.append(z)
current = self.relu(z)
self.activations.append(current)
# Output layer with softmax
z = current @ self.weights[-1] + self.biases[-1]
self.z_values.append(z)
output = self.softmax(z)
self.activations.append(output)
return output
def compute_loss(self, predictions, targets):
"""Cross-entropy loss"""
m = targets.shape[0]
log_probs = -np.log(predictions[range(m), targets] + 1e-8)
return log_probs.mean()
def backward(self, targets, learning_rate=0.001):
"""Backpropagation"""
m = targets.shape[0]
# Output layer gradient
dz = self.activations[-1].copy()
dz[range(m), targets] -= 1
dz /= m
# Propagate backwards
for i in range(len(self.weights) - 1, -1, -1):
dw = self.activations[i].T @ dz
db = dz.sum(axis=0)
if i > 0:
da = dz @ self.weights[i].T
dz = da * self.relu_derivative(self.z_values[i-1])
# Update parameters
self.weights[i] -= learning_rate * dw
self.biases[i] -= learning_rate * db
# Create and train a network
net = ArtificialNeuralNetwork([784, 256, 128, 10])
# Training loop
for epoch in range(100):
predictions = net.forward(X_train)
loss = net.compute_loss(predictions, y_train)
net.backward(y_train, learning_rate=0.01)
if epoch % 10 == 0:
print(f"Epoch {epoch}, Loss: {loss:.4f}")
Building with PyTorch
Example
import torch
import torch.nn as nn
import torch.optim as optim
class SimpleANN(nn.Module):
def __init__(self, input_size, hidden_sizes, output_size):
super().__init__()
layers = []
prev_size = input_size
for hidden_size in hidden_sizes:
layers.append(nn.Linear(prev_size, hidden_size))
layers.append(nn.ReLU())
layers.append(nn.BatchNorm1d(hidden_size))
layers.append(nn.Dropout(0.3))
prev_size = hidden_size
layers.append(nn.Linear(prev_size, output_size))
self.network = nn.Sequential(*layers)
def forward(self, x):
return self.network(x)
# Instantiate
model = SimpleANN(input_size=784, hidden_sizes=[512, 256, 128], output_size=10)
optimizer = optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()
# Training
for epoch in range(50):
model.train()
outputs = model(X_batch)
loss = criterion(outputs, y_batch)
optimizer.zero_grad()
loss.backward()
optimizer.step()
Design Decisions
How Many Hidden Layers?
- 1 layer: Simple problems, tabular data
- 2-3 layers: Most practical applications
- 5+ layers: Complex patterns, large datasets (use skip connections)
How Many Neurons Per Layer?
- Start with a "funnel" shape: wide early, narrow later (e.g., 512→256→128)
- Too few = underfitting, too many = overfitting and slow training
- Rule of thumb: between input size and output size
Which Activation Function?
- Hidden layers: ReLU (default), LeakyReLU if dying ReLU is a problem
- Output layer: Softmax (multi-class), Sigmoid (binary), Linear (regression)
Common Pitfalls
| Problem | Symptom | Solution |
|---|---|---|
| Vanishing gradients | Deep layers don't learn | ReLU, skip connections, BatchNorm |
| Exploding gradients | Loss becomes NaN | Gradient clipping, proper initialization |
| Overfitting | Train acc high, val acc low | Dropout, regularization, more data |
| Underfitting | Both accuracies low | More layers/neurons, lower regularization |
Key Takeaways
- ANNs learn by adjusting weights through forward propagation → loss computation → backpropagation
- Non-linear activations are essential — without them, stacking layers is pointless (linear of linear = linear)
- Deeper networks are more efficient than wider ones for complex functions
- Weight initialization, batch normalization, and proper activation functions are crucial for training stability
- Start simple (2-3 layers) and only add complexity if the model underfits
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Artificial Neural Networks (ANN).
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Deep Learning topic.
Search Terms
deep-learning, deep learning, deep, learning, neural, network, fundamentals, artificial
Related Deep Learning Topics
Continue learning this concept
Neural Network FundamentalsBiological vs Artificial NeuronsUnderstanding the parallels and differences between biological neurons in the brain and artificial neurons in neural networks.Neural Network FundamentalsLayers in Neural NetworksUnderstanding different types of layers in neural networks - input, hidden, and output layers, dense connections, and how to design effective architectures.Neural Network FundamentalsBackpropagationComplete guide to backpropagation algorithm in neural networks - the chain rule, gradient computation, weight updates, and implementation from scratch with PyTorch.Neural Network FundamentalsForward PropagationUnderstanding forward propagation — how neural networks compute predictions by passing inputs through layers of weights, biases, and activation functions.Neural Network FundamentalsGradient DescentUnderstanding gradient descent optimization algorithm - how neural networks learn by iteratively minimizing the loss function through parameter updates.Neural Network FundamentalsLoss Functions in Deep LearningComprehensive guide to loss functions - MSE, Cross-Entropy, Hinge Loss, Focal Loss with mathematical formulations, PyTorch implementations, and selection criteria.