ML Notes
HubAI vs ML vs Deep Learning — Machine LearningApplications of Machine LearningHistory of Machine LearningMachine Learning RoadmapWhat is Machine Learning?Calculus for Machine LearningData Preprocessing BasicsLinear Algebra for Machine LearningProbability Basics for Machine LearningPython for Machine LearningStatistics Basics for Machine LearningBias vs Variance TradeoffModel Evaluation OverviewOverfitting and UnderfittingReinforcement Learning Fundamentals — Machine LearningSupervised LearningTraining vs Testing DataTypes of Machine LearningUnsupervised LearningData Cleaning — Machine LearningData Collection for Machine LearningData Visualization for MLEncoding Categorical DataFeature EngineeringFeature ScalingHandling Missing ValuesTrain-Test SplitElastic Net RegressionLasso Regression (L1 Regularization)Multiple Linear RegressionPolynomial RegressionIntroduction to RegressionRegression ProjectsRidge Regression (L2 Regularization)Simple Linear RegressionIntroduction to ClassificationClassification ProjectsDecision TreesK-Nearest Neighbors (KNN)Logistic RegressionNaive Bayes ClassifierRandom Forest ClassificationSupport Vector Machine (SVM)XGBoost for ClassificationClustering in Machine Learning: Complete Guide to Unsupervised LearningClustering Projects: Real-World Applications and Implementation GuidesDBSCAN ClusteringGaussian Mixture Models (GMM): Probabilistic Clustering for Advanced Data AnalysisHierarchical ClusteringK-Means ClusteringDimensionality Reduction: Comprehensive Guide to Feature Reduction TechniquesFeature Selection Techniques: Choosing Informative Variables for ML ModelsLinear Discriminant Analysis (LDA): Supervised Dimensionality ReductionPrincipal Component Analysis (PCA)t-SNE: Non-Linear Dimensionality Reduction for Data VisualizationUMAP: Uniform Manifold Approximation and Projection for Modern Data VisualizationAdaBoost: Adaptive Boosting Algorithm Explained with ExamplesBagging: Bootstrap Aggregating for Reducing Model VarianceBoosting: Sequential Ensemble Learning for Bias and Variance ReductionEnsemble Learning: Complete Guide to Combining ML Models for Better PredictionsGradient Boosting: Building Powerful ML Models Through Sequential OptimizationRandom Forest: Complete Guide to Tree Ensembles and Feature ImportanceStacking: Advanced Ensemble Method Using Meta-Learners for Superior ML PredictionsActivation Functions in Neural NetworksBackpropagation: Complete Algorithm for Training Neural NetworksIntroduction to Deep Learning: Neural Networks, Architectures, and ApplicationsDeep Learning Projects: Hands-On Applications and Case StudiesNeural Networks Basics: Perceptron, Architecture, and TrainingPerceptron: Foundational Single-Neuron Classifier for Binary ClassificationPyTorch Basics: Building and Training Neural NetworksTensorFlow Basics: Building and Training Neural NetworksIntroduction to Natural Language Processing: Fundamentals and ApplicationsNLP Projects: Practical Applications from Sentiment Analysis to Machine TranslationStemming and Lemmatization: Reducing Words to Root Forms in NLPText Preprocessing: Essential Steps for NLP ApplicationsTF-IDF: Term Frequency-Inverse Document Frequency for Text Feature ExtractionTokenization in NLP: Breaking Text into Meaningful UnitsTransformers: The Architecture Behind Modern NLP ModelsWord Embeddings: Representing Words as Vectors for NLPIntroduction to Computer VisionComputer Vision ProjectsImage Classification: Building and Deploying CNN ModelsImage Processing BasicsImage Segmentation — Machine LearningObject DetectionOpenCV BasicsAgents and Environments in RLDeep Q-Networks (DQN)Policy Gradient Methods — Machine LearningQ-Learning AlgorithmReinforcement Learning: IntroductionReinforcement Learning ProjectsAccuracy, Precision, Recall: Classification Metrics Explained for ML InterviewConfusion Matrix: Complete Guide to Classification EvaluationCross-Validation: Comprehensive Guide to Robust Model EvaluationF1 Score: Balanced Classification Metric for Imbalanced DataGrid Search: Complete Guide to Hyperparameter Tuning in Machine LearningHyperparameter Tuning: Complete Guide to Optimizing Machine Learning ModelsModel Selection: Choosing the Best Algorithm for Your ML ProblemROC and AUC: Complete Guide to Classification Threshold EvaluationBuild an AI Assistant with LLMsBuild a Conversational ChatbotCustomer Churn PredictionFace Recognition System — Machine LearningHouse Price PredictionRecommendation System ProjectSentiment Analysis Project — Machine LearningSpam Email ClassifierML Case Study Questions for InterviewsML Coding Questions for Technical InterviewsMachine Learning Interview CheatsheetEssential Machine Learning Interview QuestionsStatistics for Machine Learning InterviewsDatasets for ML PracticeImportant ML FormulasMachine Learning GlossaryRecommended ML Books and CoursesCI/CD for Machine LearningDocker for Machine LearningFastAPI for Machine Learning APIsFlask API for Machine LearningMLOps: IntroductionModel Deployment StrategiesML Model Monitoring & Observability
Learning optimal policies directly using gradient ascent on policy parameters.
Policy gradient methods directly optimize the policy instead of learning value functions.
Policy Representation
Parameterized Policy:
π_θ(a|s) = P(action=a | state=s; parameters θ)
Typically neural network outputs probabilities
Policy Gradient Theorem
∇_θ J(θ) = E[∇_θ log π_θ(a|s) * Q^π(s,a)]
Gradient of objective = expected policy gradient weighted by action value
REINFORCE Algorithm
Simplest policy gradient method:
Example
def reinforce(env, learning_rate=0.001, episodes=1000):
# Build policy network
policy = build_policy_network()
optimizer = Adam(learning_rate)
for episode in range(episodes):
# Collect trajectory
states, actions, rewards = collect_trajectory(env, policy)
# Compute returns
returns = []
cumulative = 0
for r in reversed(rewards):
cumulative = r + 0.99 * cumulative
returns.insert(0, cumulative)
# Normalize returns
returns = (returns - np.mean(returns)) / (np.std(returns) + 1e-8)
# Update policy
with tf.GradientTape() as tape:
for s, a, g in zip(states, actions, returns):
log_probs = policy(s)
log_prob_a = log_probs[a]
loss = -log_prob_a * g # Negative for gradient ascent
loss = tf.reduce_mean(loss)
gradients = tape.gradient(loss, policy.trainable_variables)
optimizer.apply_gradients(zip(gradients, policy.trainable_variables))
return policy
Actor-Critic Methods
Combine policy gradient (actor) with value function (critic):
Actor: Outputs actions π_θ(a|s)
Critic: Estimates value V_φ(s)
Actor improved by critic feedback
Critic improved by TD error
A2C (Advantage Actor-Critic)
Actor loss: -log π_θ(a|s) * A(s,a)
A(s,a) = r + γV(s') - V(s) [Advantage = Q - V]
Example
class A2C:
def __init__(self, state_size, action_size):
self.actor = build_actor_network(action_size)
self.critic = build_critic_network()
self.actor_optimizer = Adam(0.001)
self.critic_optimizer = Adam(0.001)
def train_step(self, state, action, reward, next_state, done):
# Critic: estimate values
value = self.critic(state)
next_value = self.critic(next_state)
# TD error
td_error = reward + 0.99 * next_value * (1 - done) - value
# Update critic
critic_loss = td_error ** 2
critic_grad = tf.gradient(critic_loss, self.critic.trainable_variables)
self.critic_optimizer.apply_gradients(
zip(critic_grad, self.critic.trainable_variables)
)
# Update actor
log_prob = tf.math.log(self.actor(state)[action])
actor_loss = -log_prob * tf.stop_gradient(td_error)
actor_grad = tf.gradient(actor_loss, self.actor.trainable_variables)
self.actor_optimizer.apply_gradients(
zip(actor_grad, self.actor.trainable_variables)
)
PPO (Proximal Policy Optimization)
More stable than REINFORCE, better than A2C:
Example
clip_ratio = 0.2
clipped_surrogate = min(
r_t(θ) * A_t,
clip(r_t(θ), 1-clip_ratio, 1+clip_ratio) * A_t
)
loss = -E[clipped_surrogate]
Example
def ppo_update(policy, old_policy, advantages, states, actions, learning_rate=0.001):
"""Single PPO update step"""
for _ in range(num_epochs):
# Compute new policy probabilities
new_probs = policy(states)
new_log_probs = tf.math.log(new_probs + 1e-10)[actions]
# Old policy probabilities
old_probs = old_policy(states)
old_log_probs = tf.math.log(old_probs + 1e-10)[actions]
# Probability ratio
ratio = tf.exp(new_log_probs - old_log_probs)
# Clipped objective
clip_ratio = 0.2
clipped_ratio = tf.clip_by_value(ratio, 1 - clip_ratio, 1 + clip_ratio)
surr1 = ratio * advantages
surr2 = clipped_ratio * advantages
# Loss (minimize = maximize reward)
policy_loss = -tf.reduce_mean(tf.minimum(surr1, surr2))
# Update policy
# gradient descent...
Continuous Control
Policy outputs action distribution for continuous actions:
python example
# Gaussian policy for continuous actions
mean = network_mean(state)
log_std = network_log_std(state) # Learned parameter
std = exp(log_std)
# Sample action from distribution
action = mean + std * random_normal()
# Log probability
log_prob = -0.5 * ((action - mean) / std) ** 2 - log_stdAdvantages of Policy Gradient
Advantages:
- Direct optimization of policy
- Works with continuous action spaces
- Theoretically grounded
- No need for max operation (stable)
- Better convergence properties than Q-learning
Disadvantages:
- High variance (need many samples)
- Slower convergence than value-based
- Often requires large batches
Comparison
| Method | Type | Best For | Convergence |
|---|---|---|---|
| Q-Learning | Value | Discrete actions | Fast but unstable |
| DQN | Value | High-dim discrete | Stable, slow |
| REINFORCE | Policy | Simple baseline | Very slow |
| A2C | Actor-Critic | Small/medium | Medium |
| PPO | Policy | General use | Fast, stable |
Summary
Policy gradient:
- Optimize policy directly
- Use gradient ascent on expected return
- Reduces variance with baseline (critic)
- Works for continuous actions
- PPO is practical choice
Practical Implementation Tips
When implementing policy gradient methods in practice, several details make the difference between working and failing algorithms:
Reward Scaling: Raw rewards can vary enormously across environments. Normalize rewards by subtracting the mean and dividing by the standard deviation of a rolling window. This stabilizes gradient magnitudes and prevents the policy from oscillating.
Entropy Regularization: Add an entropy bonus to the loss function to encourage exploration. Without this, the policy can collapse to a deterministic strategy too early, getting stuck in local optima. The entropy coefficient is typically annealed from a high value (more exploration) to a low value (more exploitation) during training.
Gradient Clipping: Policy gradient updates can occasionally be very large, destabilizing training. Clip gradients by global norm (typically 0.5-1.0) to prevent catastrophic parameter updates while still allowing learning from strong signals.
Learning Rate Scheduling: Start with a moderate learning rate (3e-4 is a common default for Adam optimizer with policy gradients) and reduce it as training progresses. Too high a learning rate causes policy oscillation; too low means painfully slow learning.
When to Use Which Algorithm
Choosing the right policy gradient variant depends on your specific problem:
- REINFORCE: Educational purposes or extremely simple environments. Too high variance for practical use without significant modifications.
- A2C (Advantage Actor-Critic): Good for environments where you can run many parallel instances. Synchronous updates make debugging easier.
- A3C: Like A2C but asynchronous. Historical importance but largely superseded by PPO.
- PPO (Proximal Policy Optimization): The go-to choice for most practical applications. Robust, relatively hyperparameter-insensitive, and works across diverse environments. OpenAI's default for most tasks.
- SAC (Soft Actor-Critic): Best for continuous control tasks where sample efficiency matters. Uses entropy maximization for robust exploration.
- TRPO (Trust Region Policy Optimization): Theoretically elegant with guaranteed monotonic improvement, but complex to implement. PPO achieves similar results more simply.
Debugging Policy Gradient Training
Policy gradient methods are notoriously difficult to debug. Key diagnostics to monitor:
- Policy entropy: Should decrease gradually. Sudden collapse indicates premature convergence.
- Value function loss: Should decrease over training. If not, the critic isn't learning properly.
- KL divergence between old and new policy: Should stay small (PPO clips this by design).
- Episode reward trend: Plot rolling average. Expect noisy but upward-trending curves.
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Policy Gradient Methods — Machine Learning.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Machine Learning topic.
Search Terms
machine-learning, machine learning, machine, learning, reinforcement, policy, gradient, methods
Related Machine Learning Topics
Continue learning this concept
Reinforcement LearningAgents and Environments in RLUnderstanding the agent-environment interaction model, state representations, and environment design in reinforcement learning.Reinforcement LearningDeep Q-Networks (DQN)Understanding Deep Q-Networks: combining Q-Learning with neural networks for high-dimensional state spaces.Reinforcement LearningQ-Learning AlgorithmDeep dive into Q-Learning: off-policy learning, Bellman equation, and practical implementation.Reinforcement LearningReinforcement Learning: IntroductionIntroduction to reinforcement learning: agents, environments, rewards, and the fundamentals of learning through interaction.Reinforcement LearningReinforcement Learning ProjectsHands-on RL projects: game playing, robot control, and real-world applications.Ensemble LearningGradient Boosting: Building Powerful ML Models Through Sequential OptimizationMaster Gradient Boosting: residual fitting, loss optimization, learning rate, tree depth. Learn XGBoost alternatives, regularization, and machine learning interview questions.