ML Notes
HubAI vs ML vs Deep Learning — Machine LearningApplications of Machine LearningHistory of Machine LearningMachine Learning RoadmapWhat is Machine Learning?Calculus for Machine LearningData Preprocessing BasicsLinear Algebra for Machine LearningProbability Basics for Machine LearningPython for Machine LearningStatistics Basics for Machine LearningBias vs Variance TradeoffModel Evaluation OverviewOverfitting and UnderfittingReinforcement Learning Fundamentals — Machine LearningSupervised LearningTraining vs Testing DataTypes of Machine LearningUnsupervised LearningData Cleaning — Machine LearningData Collection for Machine LearningData Visualization for MLEncoding Categorical DataFeature EngineeringFeature ScalingHandling Missing ValuesTrain-Test SplitElastic Net RegressionLasso Regression (L1 Regularization)Multiple Linear RegressionPolynomial RegressionIntroduction to RegressionRegression ProjectsRidge Regression (L2 Regularization)Simple Linear RegressionIntroduction to ClassificationClassification ProjectsDecision TreesK-Nearest Neighbors (KNN)Logistic RegressionNaive Bayes ClassifierRandom Forest ClassificationSupport Vector Machine (SVM)XGBoost for ClassificationClustering in Machine Learning: Complete Guide to Unsupervised LearningClustering Projects: Real-World Applications and Implementation GuidesDBSCAN ClusteringGaussian Mixture Models (GMM): Probabilistic Clustering for Advanced Data AnalysisHierarchical ClusteringK-Means ClusteringDimensionality Reduction: Comprehensive Guide to Feature Reduction TechniquesFeature Selection Techniques: Choosing Informative Variables for ML ModelsLinear Discriminant Analysis (LDA): Supervised Dimensionality ReductionPrincipal Component Analysis (PCA)t-SNE: Non-Linear Dimensionality Reduction for Data VisualizationUMAP: Uniform Manifold Approximation and Projection for Modern Data VisualizationAdaBoost: Adaptive Boosting Algorithm Explained with ExamplesBagging: Bootstrap Aggregating for Reducing Model VarianceBoosting: Sequential Ensemble Learning for Bias and Variance ReductionEnsemble Learning: Complete Guide to Combining ML Models for Better PredictionsGradient Boosting: Building Powerful ML Models Through Sequential OptimizationRandom Forest: Complete Guide to Tree Ensembles and Feature ImportanceStacking: Advanced Ensemble Method Using Meta-Learners for Superior ML PredictionsActivation Functions in Neural NetworksBackpropagation: Complete Algorithm for Training Neural NetworksIntroduction to Deep Learning: Neural Networks, Architectures, and ApplicationsDeep Learning Projects: Hands-On Applications and Case StudiesNeural Networks Basics: Perceptron, Architecture, and TrainingPerceptron: Foundational Single-Neuron Classifier for Binary ClassificationPyTorch Basics: Building and Training Neural NetworksTensorFlow Basics: Building and Training Neural NetworksIntroduction to Natural Language Processing: Fundamentals and ApplicationsNLP Projects: Practical Applications from Sentiment Analysis to Machine TranslationStemming and Lemmatization: Reducing Words to Root Forms in NLPText Preprocessing: Essential Steps for NLP ApplicationsTF-IDF: Term Frequency-Inverse Document Frequency for Text Feature ExtractionTokenization in NLP: Breaking Text into Meaningful UnitsTransformers: The Architecture Behind Modern NLP ModelsWord Embeddings: Representing Words as Vectors for NLPIntroduction to Computer VisionComputer Vision ProjectsImage Classification: Building and Deploying CNN ModelsImage Processing BasicsImage Segmentation — Machine LearningObject DetectionOpenCV BasicsAgents and Environments in RLDeep Q-Networks (DQN)Policy Gradient Methods — Machine LearningQ-Learning AlgorithmReinforcement Learning: IntroductionReinforcement Learning ProjectsAccuracy, Precision, Recall: Classification Metrics Explained for ML InterviewConfusion Matrix: Complete Guide to Classification EvaluationCross-Validation: Comprehensive Guide to Robust Model EvaluationF1 Score: Balanced Classification Metric for Imbalanced DataGrid Search: Complete Guide to Hyperparameter Tuning in Machine LearningHyperparameter Tuning: Complete Guide to Optimizing Machine Learning ModelsModel Selection: Choosing the Best Algorithm for Your ML ProblemROC and AUC: Complete Guide to Classification Threshold EvaluationBuild an AI Assistant with LLMsBuild a Conversational ChatbotCustomer Churn PredictionFace Recognition System — Machine LearningHouse Price PredictionRecommendation System ProjectSentiment Analysis Project — Machine LearningSpam Email ClassifierML Case Study Questions for InterviewsML Coding Questions for Technical InterviewsMachine Learning Interview CheatsheetEssential Machine Learning Interview QuestionsStatistics for Machine Learning InterviewsDatasets for ML PracticeImportant ML FormulasMachine Learning GlossaryRecommended ML Books and CoursesCI/CD for Machine LearningDocker for Machine LearningFastAPI for Machine Learning APIsFlask API for Machine LearningMLOps: IntroductionModel Deployment StrategiesML Model Monitoring & Observability
Understanding Deep Q-Networks: combining Q-Learning with neural networks for high-dimensional state spaces.
DQN combines Q-Learning with deep neural networks to handle high-dimensional state spaces like images.
Problem with Tabular Q-Learning
Curse of Dimensionality:
- Image state: 84x84 RGB = 253,952 dimensions
- Q-table size: exponential in dimensions
- Infeasible for large state spaces
Solution: Use neural network to approximate Q-values
Example
Q(s,a) ≈ network(s)[a]
DQN Architecture
| Input | State (84x84x4 image) |
| Conv Layer 1 | 32 filters, 8x8, stride 4 |
| Conv Layer 2 | 64 filters, 4x4, stride 2 |
| Conv Layer 3 | 64 filters, 3x3, stride 1 |
| Dense | 512 units, ReLU |
| Output | Q-values for each action (18 outputs for Atari) |
Key Innovations
1. Experience Replay
Store transitions and train on mini-batches:
python example
replay_buffer = [] # (s, a, r, s', done)
# Store transition
replay_buffer.append((state, action, reward, next_state, done))
# Sample mini-batch for training
mini_batch = random.sample(replay_buffer, batch_size=32)Benefits:
- Breaks correlation between samples
- More stable training
- Better sample efficiency
2. Target Network
Use separate network for stability:
Example
Q-network: updated every step (fast)
Target network: updated every N steps (stable)
Loss = (r + γ*max Q_target(s', a') - Q(s,a))²
python example
# Every N steps
target_network.set_weights(main_network.get_weights())3. Reward Clipping
Normalize rewards to [-1, +1]:
Example
clipped_reward = max(-1, min(1, reward))
DQN Algorithm
Example
def dqn(env, episodes=10000):
# Create networks
Q = build_network() # Main network
Q_target = build_network() # Target network
Q_target.set_weights(Q.get_weights())
replay_buffer = []
epsilon = 1.0
for episode in range(episodes):
state = env.reset()
done = False
while not done:
# ε-greedy action selection
if random() < epsilon:
action = env.action_space.sample()
else:
Q_values = Q.predict(state[np.newaxis])[0]
action = np.argmax(Q_values)
# Take action
next_state, reward, done, _ = env.step(action)
# Store in replay buffer
replay_buffer.append((state, action, reward, next_state, done))
if len(replay_buffer) > 100000:
replay_buffer.pop(0)
# Train on mini-batch
if len(replay_buffer) > 32:
mini_batch = random.sample(replay_buffer, 32)
train_step(Q, Q_target, mini_batch)
state = next_state
# Update target network
if episode % 10000 == 0:
Q_target.set_weights(Q.get_weights())
# Decay epsilon
epsilon = max(0.01, epsilon * 0.995)
return Q
def train_step(Q, Q_target, mini_batch, learning_rate=0.00025):
states, actions, rewards, next_states, dones = zip(*mini_batch)
# Current Q-values
Q_values = Q.predict_on_batch(np.array(states))
# Next Q-values from target network
next_Q_values = Q_target.predict_on_batch(np.array(next_states))
# Bellman update
for i in range(len(mini_batch)):
if dones[i]:
Q_values[i, actions[i]] = rewards[i]
else:
Q_values[i, actions[i]] = rewards[i] + 0.99 * np.max(next_Q_values[i])
# Train
Q.fit(np.array(states), Q_values, verbose=0)
Implementation with TensorFlow
Example
import tensorflow as tf
import numpy as np
class DQN:
def __init__(self, state_size, action_size):
self.state_size = state_size
self.action_size = action_size
self.memory = []
self.epsilon = 1.0
self.epsilon_min = 0.01
self.epsilon_decay = 0.995
self.learning_rate = 0.00025
self.gamma = 0.99
self.model = self.build_model()
self.target_model = self.build_model()
self.update_target_model()
def build_model(self):
"""Build neural network"""
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(32, 8, strides=4, activation='relu',
input_shape=self.state_size),
tf.keras.layers.Conv2D(64, 4, strides=2, activation='relu'),
tf.keras.layers.Conv2D(64, 3, strides=1, activation='relu'),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(512, activation='relu'),
tf.keras.layers.Dense(self.action_size, activation='linear')
])
model.compile(optimizer=tf.keras.optimizers.Adam(self.learning_rate),
loss='mse')
return model
def update_target_model(self):
"""Copy weights from model to target_model"""
self.target_model.set_weights(self.model.get_weights())
def remember(self, state, action, reward, next_state, done):
"""Store experience in memory"""
self.memory.append((state, action, reward, next_state, done))
def act(self, state):
"""Choose action (ε-greedy)"""
if np.random.random() <= self.epsilon:
return np.random.randint(0, self.action_size)
Q_values = self.model.predict(state, verbose=0)
return np.argmax(Q_values[0])
def replay(self, batch_size):
"""Train on mini-batch"""
if len(self.memory) < batch_size:
return
mini_batch = np.random.choice(len(self.memory), batch_size, replace=False)
states = np.array([self.memory[i][0] for i in mini_batch])
actions = np.array([self.memory[i][1] for i in mini_batch])
rewards = np.array([self.memory[i][2] for i in mini_batch])
next_states = np.array([self.memory[i][3] for i in mini_batch])
dones = np.array([self.memory[i][4] for i in mini_batch])
# Current Q-values
target_Q_values = self.model.predict(states, verbose=0)
# Next Q-values
next_Q_values = self.target_model.predict(next_states, verbose=0)
# Bellman equation
for i in range(batch_size):
if dones[i]:
target_Q_values[i, actions[i]] = rewards[i]
else:
target_Q_values[i, actions[i]] = (
rewards[i] + self.gamma * np.max(next_Q_values[i])
)
# Train
self.model.fit(states, target_Q_values, epochs=1, verbose=0)
# Decay epsilon
if self.epsilon > self.epsilon_min:
self.epsilon *= self.epsilon_decay
Performance
DQN achieves superhuman performance on Atari games:
- Breakout: 30x human
- Pong: Perfect play
- Space Invaders: 6x human
- Many games: within human range
Summary
DQN:
- Approximates Q-values with neural network
- Handles high-dimensional state spaces
- Uses experience replay for stability
- Uses target network for stability
- Achieves superhuman performance
Continue the lesson path
Study next in this module
01Reinforcement LearningAgents and Environments in RLUnderstanding the agent-environment interaction model, state representations, and environment design in reinforcement learning.02Reinforcement LearningPolicy Gradient Methods — Machine LearningLearning optimal policies directly using gradient ascent on policy parameters.03Reinforcement LearningQ-Learning AlgorithmDeep dive into Q-Learning: off-policy learning, Bellman equation, and practical implementation.04Reinforcement LearningReinforcement Learning: IntroductionIntroduction to reinforcement learning: agents, environments, rewards, and the fundamentals of learning through interaction.