DL Notes
Understanding the parallels and differences between biological neurons in the brain and artificial neurons in neural networks.
Deep learning was inspired by the brain, but modern neural networks have diverged significantly from biology. Understanding both helps you appreciate the inspiration and recognize where the analogy breaks down.
The Biological Neuron
Your brain contains roughly 86 billion neurons, each connected to thousands of others through approximately 100 trillion synapses.
How Biological Neurons Work
- Dendrites receive chemical signals (neurotransmitters) from other neurons
- Soma integrates incoming signals — if the total exceeds a threshold, the neuron "fires"
- Axon transmits the electrical spike (action potential) to other neurons
- Synapses release neurotransmitters to signal downstream neurons
Key properties:
- All-or-nothing: A neuron either fires completely or not at all
- Temporal coding: Timing and frequency of spikes carry information
- Plasticity: Synapse strength changes based on activity (Hebbian learning: "neurons that fire together wire together")
The Artificial Neuron
Artificial Neuron (Perceptron model):
x₁ ──w₁──┐
│
x₂ ──w₂──┼──→ [Σ + b] ──→ [activation σ] ──→ output a
│
x₃ ──w₃──┘
Inputs Weights Summation Non-linearity Output
Mathematical Formulation
$$z = \sum_{i=1}^{n} w_i x_i + b = \mathbf{w}^T \mathbf{x} + b$$
$$a = \sigma(z)$$
import numpy as np
class ArtificialNeuron:
def __init__(self, n_inputs):
self.weights = np.random.randn(n_inputs) * 0.01
self.bias = 0.0
def forward(self, x):
z = np.dot(self.weights, x) + self.bias
return self.activation(z)
def activation(self, z):
# ReLU (most common in modern networks)
return max(0, z)Side-by-Side Comparison
| Feature | Biological Neuron | Artificial Neuron |
|---|---|---|
| Inputs | Dendrites (chemical signals) | Weighted inputs (numbers) |
| Processing | Soma (electrochemical integration) | Weighted sum + bias |
| Threshold | Action potential threshold (~-55mV) | Activation function |
| Output | Spike trains (frequency coding) | Single continuous value |
| Learning | Synaptic plasticity (LTP/LTD) | Gradient descent (backprop) |
| Connections | ~7,000 synapses per neuron | Fully connected or sparse |
| Speed | ~1-100 Hz firing rate | Nanoseconds (GPU) |
| Energy | ~20 watts for entire brain | Kilowatts for large models |
| Parallelism | Massively parallel (86B neurons) | Limited by hardware |
Where the Analogy Holds
Weight = Synapse Strength
Both systems have adjustable connection strengths. Stronger synapses (or larger weights) mean an input has more influence on the output.
Activation Threshold
Both biological neurons (action potential threshold) and artificial neurons (activation functions) implement non-linear decision boundaries.
Learning Through Experience
Both systems improve through exposure to data — biological via Hebbian plasticity, artificial via gradient descent.
# Hebbian Learning (biological inspiration)
# "Neurons that fire together, wire together"
def hebbian_update(pre_activation, post_activation, weight, lr=0.01):
delta_w = lr * pre_activation * post_activation
return weight + delta_w
# Gradient Descent (what we actually use)
# Minimize error by following gradients
def gradient_update(weight, gradient, lr=0.01):
return weight - lr * gradientWhere the Analogy Breaks Down
1. No Backpropagation in the Brain
The brain doesn't compute gradients backward through layers. How biological learning works at the network level remains an open question (though theories like "predictive coding" and "feedback alignment" exist).
2. Biological Neurons Are Temporal
Real neurons communicate through spike timing, not static values. A single biological neuron is computationally richer than an artificial one.
3. Brain Architecture Is Not Feedforward
The brain has massive feedback connections, lateral inhibition, neuromodulation, and recurrent loops. It's far more complex than a simple layer-by-layer feedforward network.
4. Energy Efficiency
The human brain operates on about 20 watts and handles vision, language, planning, motor control simultaneously. GPT-4 training consumed an estimated 50+ gigawatt-hours.
5. Local Learning Rules
Biological synapses only "know" about their immediate pre and post-synaptic activity. They don't have access to a global loss function computed at the output layer.
Modern Bridges Between Biology and AI
import torch
import torch.nn as nn
# Spiking Neural Networks - more biologically realistic
# Instead of continuous values, neurons emit discrete spikes
class SpikingNeuron:
def __init__(self, threshold=1.0, decay=0.9):
self.threshold = threshold
self.decay = decay
self.membrane_potential = 0.0
def step(self, input_current):
# Leaky integrate-and-fire model
self.membrane_potential = self.decay * self.membrane_potential + input_current
if self.membrane_potential >= self.threshold:
self.membrane_potential = 0.0 # Reset after spike
return 1.0 # Spike!
return 0.0 # No spike
# Self-attention (Transformer) - somewhat analogous to
# selective attention in the brain
attention = nn.MultiheadAttention(embed_dim=512, num_heads=8)
# The brain also attends selectively to relevant informationBrain-Inspired Innovations in Deep Learning
| Brain Feature | DL Innovation | Impact |
|---|---|---|
| Sparse coding | Dropout, sparse activations | Regularization |
| Sleep/consolidation | Replay buffers in RL | Better memory |
| Attention/focus | Attention mechanisms | Transformers |
| Hierarchical processing | Deep architectures | Feature learning |
| Lateral inhibition | Batch normalization | Stable training |
| Neurogenesis | Dynamic network growth | NAS, pruning |
Key Takeaways
- Artificial neurons are a vast simplification of biological neurons — inspired by, not imitating
- The key shared principle: weighted inputs → threshold → output with adjustable connections
- Modern deep learning has diverged far from neuroscience — backpropagation, batch processing, and gradient descent have no clear biological counterpart
- The brain is still far more energy-efficient and capable of few-shot learning
- Brain-inspired ideas continue to influence AI (attention, sparse coding, memory replay) even though the mechanisms differ
- Understanding the differences prevents you from over-interpreting biological metaphors in AI
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Biological vs Artificial Neurons.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Deep Learning topic.
Search Terms
deep-learning, deep learning, deep, learning, neural, network, fundamentals, biological
Related Deep Learning Topics