ML Notes
HubAI vs ML vs Deep Learning — Machine LearningApplications of Machine LearningHistory of Machine LearningMachine Learning RoadmapWhat is Machine Learning?Calculus for Machine LearningData Preprocessing BasicsLinear Algebra for Machine LearningProbability Basics for Machine LearningPython for Machine LearningStatistics Basics for Machine LearningBias vs Variance TradeoffModel Evaluation OverviewOverfitting and UnderfittingReinforcement Learning Fundamentals — Machine LearningSupervised LearningTraining vs Testing DataTypes of Machine LearningUnsupervised LearningData Cleaning — Machine LearningData Collection for Machine LearningData Visualization for MLEncoding Categorical DataFeature EngineeringFeature ScalingHandling Missing ValuesTrain-Test SplitElastic Net RegressionLasso Regression (L1 Regularization)Multiple Linear RegressionPolynomial RegressionIntroduction to RegressionRegression ProjectsRidge Regression (L2 Regularization)Simple Linear RegressionIntroduction to ClassificationClassification ProjectsDecision TreesK-Nearest Neighbors (KNN)Logistic RegressionNaive Bayes ClassifierRandom Forest ClassificationSupport Vector Machine (SVM)XGBoost for ClassificationClustering in Machine Learning: Complete Guide to Unsupervised LearningClustering Projects: Real-World Applications and Implementation GuidesDBSCAN ClusteringGaussian Mixture Models (GMM): Probabilistic Clustering for Advanced Data AnalysisHierarchical ClusteringK-Means ClusteringDimensionality Reduction: Comprehensive Guide to Feature Reduction TechniquesFeature Selection Techniques: Choosing Informative Variables for ML ModelsLinear Discriminant Analysis (LDA): Supervised Dimensionality ReductionPrincipal Component Analysis (PCA)t-SNE: Non-Linear Dimensionality Reduction for Data VisualizationUMAP: Uniform Manifold Approximation and Projection for Modern Data VisualizationAdaBoost: Adaptive Boosting Algorithm Explained with ExamplesBagging: Bootstrap Aggregating for Reducing Model VarianceBoosting: Sequential Ensemble Learning for Bias and Variance ReductionEnsemble Learning: Complete Guide to Combining ML Models for Better PredictionsGradient Boosting: Building Powerful ML Models Through Sequential OptimizationRandom Forest: Complete Guide to Tree Ensembles and Feature ImportanceStacking: Advanced Ensemble Method Using Meta-Learners for Superior ML PredictionsActivation Functions in Neural NetworksBackpropagation: Complete Algorithm for Training Neural NetworksIntroduction to Deep Learning: Neural Networks, Architectures, and ApplicationsDeep Learning Projects: Hands-On Applications and Case StudiesNeural Networks Basics: Perceptron, Architecture, and TrainingPerceptron: Foundational Single-Neuron Classifier for Binary ClassificationPyTorch Basics: Building and Training Neural NetworksTensorFlow Basics: Building and Training Neural NetworksIntroduction to Natural Language Processing: Fundamentals and ApplicationsNLP Projects: Practical Applications from Sentiment Analysis to Machine TranslationStemming and Lemmatization: Reducing Words to Root Forms in NLPText Preprocessing: Essential Steps for NLP ApplicationsTF-IDF: Term Frequency-Inverse Document Frequency for Text Feature ExtractionTokenization in NLP: Breaking Text into Meaningful UnitsTransformers: The Architecture Behind Modern NLP ModelsWord Embeddings: Representing Words as Vectors for NLPIntroduction to Computer VisionComputer Vision ProjectsImage Classification: Building and Deploying CNN ModelsImage Processing BasicsImage Segmentation — Machine LearningObject DetectionOpenCV BasicsAgents and Environments in RLDeep Q-Networks (DQN)Policy Gradient Methods — Machine LearningQ-Learning AlgorithmReinforcement Learning: IntroductionReinforcement Learning ProjectsAccuracy, Precision, Recall: Classification Metrics Explained for ML InterviewConfusion Matrix: Complete Guide to Classification EvaluationCross-Validation: Comprehensive Guide to Robust Model EvaluationF1 Score: Balanced Classification Metric for Imbalanced DataGrid Search: Complete Guide to Hyperparameter Tuning in Machine LearningHyperparameter Tuning: Complete Guide to Optimizing Machine Learning ModelsModel Selection: Choosing the Best Algorithm for Your ML ProblemROC and AUC: Complete Guide to Classification Threshold EvaluationBuild an AI Assistant with LLMsBuild a Conversational ChatbotCustomer Churn PredictionFace Recognition System — Machine LearningHouse Price PredictionRecommendation System ProjectSentiment Analysis Project — Machine LearningSpam Email ClassifierML Case Study Questions for InterviewsML Coding Questions for Technical InterviewsMachine Learning Interview CheatsheetEssential Machine Learning Interview QuestionsStatistics for Machine Learning InterviewsDatasets for ML PracticeImportant ML FormulasMachine Learning GlossaryRecommended ML Books and CoursesCI/CD for Machine LearningDocker for Machine LearningFastAPI for Machine Learning APIsFlask API for Machine LearningMLOps: IntroductionModel Deployment StrategiesML Model Monitoring & Observability
Understanding the agent-environment interaction model, state representations, and environment design in reinforcement learning.
The fundamental loop in RL is the continuous interaction between an agent and its environment.
Agent-Environment Loop
┌─────────────────────────────┐
│ Agent │
│ (Decision Maker) │
└────────────┬────────────────┘
│ action a_t
▼
┌─────────────────────────────┐
│ Environment │
│ (Dynamics, Rewards) │
└────────────┬────────────────┘
│ state s_{t+1}, reward r_t
▼
┌─────────────────────────────┐
│ Agent │
│ (Update Policy) │
└─────────────────────────────┘
State Representation
State: Complete description of environment configuration
Types of States
Fully Observable: Agent sees complete state
Example
state = [position, velocity, angle, angular_velocity]
Partially Observable: Agent sees partial state
Example
# Poker: only own hand visible
state = [own_cards, community_cards_public, betting_history]
Latent: Hidden features must be learned
python example
# Image: raw pixels, must learn features
state = image_array # shape (84, 84, 3)Action Space
Discrete: Finite set of actions
Example
actions = ['forward', 'backward', 'left', 'right']
n_actions = 4
Continuous: Infinite actions in range
Example
# Robotic control
action = [joint_1_angle, joint_2_angle, joint_3_angle]
# Each in [-1.0, 1.0]
Hybrid: Mix of discrete and continuous
Example
action = {
'direction': discrete, # [0, 1, 2] for left/forward/right
'speed': continuous # [0.0, 1.0]
}
Reward Structure
Single Reward
python example
reward = {
+1: goal reached,
-1: collision,
0: otherwise
}Shaped Reward
python example
reward = {
+100: goal,
-50: collision,
-0.1: per step (encourage efficiency),
+5: moved towards goal
}Multi-Objective
python example
rewards = {
'main_goal': 100,
'efficiency': -0.1, # per time step
'safety': -50 # for collision
}
# Combine with weights
total_reward = 1.0*main + 0.5*efficiency + 1.0*safetyAgent Architecture
┌──────────────────────────────┐
│ Observation/State │
│ (from environment) │
└────────────┬─────────────────┘
│
▼
┌──────────────────────────────┐
│ Feature Extraction │
│ (if needed) │
└────────────┬─────────────────┘
│
▼
┌──────────────────────────────┐
│ Policy Network │
│ (or Value Network) │
└────────────┬─────────────────┘
│
▼
┌──────────────────────────────┐
│ Action Selection │
│ (e.g., ε-greedy) │
└────────────┬─────────────────┘
│
▼
┌──────────────────────────────┐
│ Action to Environment │
└──────────────────────────────┘
Environment Types
Deterministic
Same action always produces same outcome:
python example
next_state, reward = env.step(action)
# DeterministicStochastic
Same action can have different outcomes:
python example
# Wind affects ball trajectory
next_state = stochastic_step(state, action)Episodic
Task has natural ending:
python example
# Game: start to win/lose
episode = []
done = False
while not done:
action = agent.get_action(state)
state, reward, done, _ = env.step(action)
episode.append((state, action, reward))Continuing
No natural ending:
python example
# Robot maintaining balance
for t in range(infinite):
action = agent.get_action(state)
state, reward, done, _ = env.step(action)Implementation Example
Example
import gym
import numpy as np
class RLAgent:
def __init__(self, env):
self.env = env
self.state_space = env.observation_space
self.action_space = env.action_space
self.learning_rate = 0.1
self.gamma = 0.99
self.epsilon = 0.1
def get_action(self, state):
"""Choose action (ε-greedy)"""
if np.random.random() < self.epsilon:
return self.action_space.sample()
else:
return self.get_best_action(state)
def run_episode(self):
"""Run one episode"""
state = self.env.reset()
episode_data = []
done = False
while not done:
# Choose action
action = self.get_action(state)
# Step environment
next_state, reward, done, info = self.env.step(action)
# Store transition
episode_data.append({
'state': state,
'action': action,
'reward': reward,
'next_state': next_state,
'done': done
})
# Update
self.update(state, action, reward, next_state, done)
state = next_state
return episode_data
def update(self, state, action, reward, next_state, done):
"""Update policy (Q-Learning)"""
pass # Implement algorithm-specific update
# Usage
env = gym.make('CartPole-v1')
agent = RLAgent(env)
for episode in range(100):
episode_data = agent.run_episode()
total_reward = sum(d['reward'] for d in episode_data)
print(f"Episode {episode}: {total_reward}")
State Space Representations
Tabular
Discrete states indexed in table:
Example
Q_table = {}
for state in all_states:
for action in all_actions:
Q_table[(state, action)] = 0.0
Linear Features
Hand-crafted features:
Example
features = np.array([
position,
velocity,
angle,
angular_velocity
])
Q_value = theta @ features
Neural Network
Learning features:
Example
import tensorflow as tf
model = tf.keras.Sequential([
tf.keras.layers.Dense(128, activation='relu', input_shape=(state_dim,)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dense(n_actions)
])
Summary
Key concepts:
- Agent observes state
- Takes action
- Receives reward and next state
- Updates policy
- Repeat until convergence
Continue the lesson path
Study next in this module
01Reinforcement LearningDeep Q-Networks (DQN)Understanding Deep Q-Networks: combining Q-Learning with neural networks for high-dimensional state spaces.02Reinforcement LearningPolicy Gradient Methods — Machine LearningLearning optimal policies directly using gradient ascent on policy parameters.03Reinforcement LearningQ-Learning AlgorithmDeep dive into Q-Learning: off-policy learning, Bellman equation, and practical implementation.04Reinforcement LearningReinforcement Learning: IntroductionIntroduction to reinforcement learning: agents, environments, rewards, and the fundamentals of learning through interaction.