AI Notes

Deep Q-Networks - DQN

Master DQN for RL. Experience replay, double DQN. RL 2024.

From Q-Learning to Deep Q-Networks

Q-learning maintains a table Q(s,a) storing the expected return for each state-action pair. This works perfectly for small, discrete environments (grid worlds with dozens of states). But real-world problems have enormous or continuous state spaces—an Atari game screen has 210×160×3 = 100,800 pixels, creating more possible states than atoms in the universe. No table can store Q-values for every pixel configuration.

Deep Q-Networks (DQN), introduced by DeepMind in 2013, solved this by replacing the Q-table with a neural network that approximates Q(s,a). The network takes a state as input and outputs Q-values for all possible actions. This function approximation enables RL to tackle complex visual tasks—DQN achieved superhuman performance on many Atari games directly from pixel inputs.

DQN Architecture

Input	State (e.g., 84×84×4 stacked grayscale frames)
Conv Layer 1	32 filters, 8×8, stride 4, ReLU
Conv Layer 2	64 filters, 4×4, stride 2, ReLU
Conv Layer 3	64 filters, 3×3, stride 1, ReLU
Flatten	Dense Layer: 512 units, ReLU
Output	Q-values for each action [Q(s,left), Q(s,right), Q(s,fire), ...]
4 stacked frames	captures motion/velocity information

Key Innovations

1. Experience Replay

Problem	Sequential RL data is highly correlated
Solution	Store transitions in a replay buffer, sample randomly
Buffer stores	(state, action, reward, next_state, done)
Capacity	1,000,000 transitions
Training	Sample random mini-batch of 32 transitions

2. Target Network

Problem	Q-network updates change the target Q-values simultaneously
"Chasing a moving target"	instability, divergence
Solution	Maintain a separate target network Q_target
Q_target is frozen	stable targets → stable learning
Updated periodically	Q_target ← Q (hard copy)
or continuously	Q_target ← τQ + (1-τ)Q_target (soft update, τ=0.001)

DQN Algorithm

Initialize

Q-network with random weights θ

Target network with weights θ⁻ = θ

Replay buffer D (capacity N)

Exploration rate ε = 1.0

For each episode

s = environment.reset()

while not done:

// ε-greedy action selection

if random() < ε:

a = random_action()

else:

a = argmax_a Q(s, a; θ)

// Execute action

s', r, done = environment.step(a)

// Store transition

D.add(s, a, r, s', done)

// Learn from random batch

batch = D.sample(32)

for (sⱼ, aⱼ, rⱼ, s'ⱼ, doneⱼ) in batch:

if doneⱼ:

target = rⱼ

else:

target = rⱼ + γ × max_a' Q(s'ⱼ, a'; θ⁻)

loss += (Q(sⱼ, aⱼ; θ) - target)²

// Gradient descent on loss

θ ← θ - α × ∇_θ loss

// Update target network periodically

if steps % C == 0:

θ⁻ ← θ

// Decay exploration

ε = max(0.01, ε × 0.999)

s = s'

DQN Improvements

Double DQN

Problem	Q-learning overestimates values (max operator bias)
Solution	Decouple action selection from evaluation
Action	a* = argmax_a Q(s', a; θ) ← online network selects
Value	target = r + γ × Q(s', a*; θ⁻) ← target network evaluates

Dueling DQN

Separate value and advantage streams

Shared CNN features

/ \

V(s) stream A(s,a) stream

\ /

Q(s,a) = V(s) + A(s,a) - mean(A)

Why? Many states have similar value regardless of action

(if you're about to die, no action helps much)

V(s) captures state quality

A(s,a) captures action-specific advantage

The mean subtraction ensures identifiability

Q(s,a) = V(s) + (A(s,a) - max_a' A(s,a'))

Prioritized Experience Replay

Problem	Uniform sampling wastes time on "easy" transitions
Solution	Sample proportional to TD error
Priority	p_i = \|δ_i\| + ε
Sampling probability	P(i) = p_i^α / Σ p_j^α
High TD error	surprising transition → learn more from it

Rainbow DQN

Combines all improvements:

Example

Rainbow = DQN + Double DQN + Prioritized Replay + Dueling + Multi-step returns + Distributional RL + Noisy Nets Each improvement contributes independently Together: ~2× performance of any single improvement

Limitations of DQN

1. Discrete actions only	Can't handle continuous control
Solutions	DDPG, SAC (actor-critic for continuous actions)
2. Sample inefficient	Millions of frames needed
Atari	50M frames (200 hours of gameplay)
3. Overestimation	Despite Double DQN, still a concern
4. Exploration	ε-greedy is naive
Better	curiosity-driven, count-based exploration
5. Partial observability	Frame stacking helps but doesn't solve
Better	recurrent architectures (DRQN)

Results and Impact

Examples	Breakout (400% of human), Space Invaders (180%)
Struggled	Montezuma's Revenge (requires exploration)
Key insight	Same architecture, same hyperparameters for ALL games
Impact	Sparked modern deep RL revolution
Led to	AlphaGo, AlphaFold, robot manipulation, game AI

Interview Questions

Q: Why is experience replay necessary for DQN? A: Neural networks require i.i.d. (independent, identically distributed) training data. RL generates sequential, correlated data (state_t is similar to state_{t+1}). Without replay, the network overfits to recent experiences and catastrophically forgets old knowledge. Replay breaks correlation and enables reuse of rare experiences.

Q: What problem does the target network solve? A: Without a target network, updating Q(s,a) immediately changes the targets for all other states (since the same network computes both). This creates a "moving target" problem—like a dog chasing its own tail. The frozen target network provides stable targets for hundreds of updates before being refreshed.

Q: Why can't DQN handle continuous actions directly? A: DQN requires computing argmax_a Q(s,a)—finding the action with highest Q-value. With discrete actions, this is trivial (compare N values). With continuous actions, it requires solving an optimization problem at every step, which is expensive and potentially inaccurate. Actor-critic methods bypass this by learning an explicit policy network.

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Deep Q-Networks - DQN.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Artificial Intelligence topic.

Search Terms

artificial-intelligence, artificial intelligence, artificial, intelligence, reinforcement, learning, deep, networks

Continue learning this concept

Reinforcement LearningQ-Learning - Value-Based RLComplete Q-learning guide. Q-values, temporal difference. RL 2024.Reinforcement LearningActor-Critic Methods - Hybrid RLLearn actor-critic. Policy gradient and value estimation. RL advanced 2024.Reinforcement LearningMDP - RL FoundationMaster MDP for RL. States, actions, Bellman equation. RL fundamentals 2024.Reinforcement LearningPolicy Gradient Methods — Artificial IntelligenceMaster policy gradient. REINFORCE, actor-critic. RL advanced 2024.Reinforcement LearningReinforcement Learning Fundamentals — Artificial IntelligenceMaster RL basics. Agents, environments, rewards, Markov. RL intro 2024.Reinforcement LearningRL Projects - Game AI to RoboticsImplement RL projects. Game AI, robot control. Hands-on 2024.

PreviousActor-Critic Methods - Hybrid RL NextMDP - RL Foundation

AI Notes

Deep Q-Networks - DQN

Master DQN for RL. Experience replay, double DQN. RL 2024.

From Q-Learning to Deep Q-Networks

DQN Architecture

Input	State (e.g., 84×84×4 stacked grayscale frames)
Conv Layer 1	32 filters, 8×8, stride 4, ReLU
Conv Layer 2	64 filters, 4×4, stride 2, ReLU
Conv Layer 3	64 filters, 3×3, stride 1, ReLU
Flatten	Dense Layer: 512 units, ReLU
Output	Q-values for each action [Q(s,left), Q(s,right), Q(s,fire), ...]
4 stacked frames	captures motion/velocity information

Key Innovations

1. Experience Replay

Problem	Sequential RL data is highly correlated
Solution	Store transitions in a replay buffer, sample randomly
Buffer stores	(state, action, reward, next_state, done)
Capacity	1,000,000 transitions
Training	Sample random mini-batch of 32 transitions

2. Target Network

Problem	Q-network updates change the target Q-values simultaneously
"Chasing a moving target"	instability, divergence
Solution	Maintain a separate target network Q_target
Q_target is frozen	stable targets → stable learning
Updated periodically	Q_target ← Q (hard copy)
or continuously	Q_target ← τQ + (1-τ)Q_target (soft update, τ=0.001)

DQN Algorithm

Initialize

Q-network with random weights θ

Target network with weights θ⁻ = θ

Replay buffer D (capacity N)

Exploration rate ε = 1.0

For each episode

s = environment.reset()

while not done:

// ε-greedy action selection

if random() < ε:

a = random_action()

else:

a = argmax_a Q(s, a; θ)

// Execute action

s', r, done = environment.step(a)

// Store transition

D.add(s, a, r, s', done)

// Learn from random batch

batch = D.sample(32)

for (sⱼ, aⱼ, rⱼ, s'ⱼ, doneⱼ) in batch:

if doneⱼ:

target = rⱼ

else:

target = rⱼ + γ × max_a' Q(s'ⱼ, a'; θ⁻)

loss += (Q(sⱼ, aⱼ; θ) - target)²

// Gradient descent on loss

θ ← θ - α × ∇_θ loss

// Update target network periodically

if steps % C == 0:

θ⁻ ← θ

// Decay exploration

ε = max(0.01, ε × 0.999)

s = s'

DQN Improvements

Double DQN

Problem	Q-learning overestimates values (max operator bias)
Solution	Decouple action selection from evaluation
Action	a* = argmax_a Q(s', a; θ) ← online network selects
Value	target = r + γ × Q(s', a*; θ⁻) ← target network evaluates

Dueling DQN

Separate value and advantage streams

Shared CNN features

/ \

V(s) stream A(s,a) stream

\ /

Q(s,a) = V(s) + A(s,a) - mean(A)

Why? Many states have similar value regardless of action

(if you're about to die, no action helps much)

V(s) captures state quality

A(s,a) captures action-specific advantage

The mean subtraction ensures identifiability

Q(s,a) = V(s) + (A(s,a) - max_a' A(s,a'))

Prioritized Experience Replay

Problem	Uniform sampling wastes time on "easy" transitions
Solution	Sample proportional to TD error
Priority	p_i = \|δ_i\| + ε
Sampling probability	P(i) = p_i^α / Σ p_j^α
High TD error	surprising transition → learn more from it

Rainbow DQN

Combines all improvements:

Example

Limitations of DQN

1. Discrete actions only	Can't handle continuous control
Solutions	DDPG, SAC (actor-critic for continuous actions)
2. Sample inefficient	Millions of frames needed
Atari	50M frames (200 hours of gameplay)
3. Overestimation	Despite Double DQN, still a concern
4. Exploration	ε-greedy is naive
Better	curiosity-driven, count-based exploration
5. Partial observability	Frame stacking helps but doesn't solve
Better	recurrent architectures (DRQN)

Results and Impact

Examples	Breakout (400% of human), Space Invaders (180%)
Struggled	Montezuma's Revenge (requires exploration)
Key insight	Same architecture, same hyperparameters for ALL games
Impact	Sparked modern deep RL revolution
Led to	AlphaGo, AlphaFold, robot manipulation, game AI

Interview Questions

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Deep Q-Networks - DQN.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Artificial Intelligence topic.

Search Terms

artificial-intelligence, artificial intelligence, artificial, intelligence, reinforcement, learning, deep, networks

Continue learning this concept

PreviousActor-Critic Methods - Hybrid RL NextMDP - RL Foundation