AI Notes
Master policy gradient. REINFORCE, actor-critic. RL advanced 2024.
Introduction
Policy gradient methods are a family of reinforcement learning algorithms that directly optimize the policy function, which maps states to actions. Unlike value-based methods (like Q-learning) that learn a value function and derive a policy from it, policy gradient methods parameterize the policy itself and use gradient ascent to maximize expected cumulative reward. This direct approach offers several advantages, particularly for continuous action spaces and stochastic policies.
Why Policy Gradients?
Consider a robot learning to control its arm. The action space is continuous: joint torques can take any real value. Q-learning requires discretizing this space (losing precision) or using function approximation for the Q-function (which is complex for continuous actions). Policy gradients naturally handle continuous actions by outputting distribution parameters directly.
Value-based approach
Learn Q(s, a) for all state-action pairs
Policy: π(s) = argmax_a Q(s, a) ← requires solving optimization at each step
Problem: What if action space is continuous or very large?
Policy gradient approach
Directly learn π_θ(a|s) = probability of action a in state s
For continuous: output mean and variance of Gaussian distribution
For discrete: output probabilities over actions via softmax
The REINFORCE Algorithm
The simplest policy gradient algorithm, also called Monte Carlo Policy Gradient:
| Algorithm | REINFORCE |
| - Compute return | G_t = r_t + γr_{t+1} + γ²r_{t+2} + ... + γ^(T-t)r_T |
| - Compute policy gradient | ∇_θ log π_θ(a_t|s_t) × G_t |
| c. Update | θ = θ + α × Σ_t ∇_θ log π_θ(a_t|s_t) × G_t |
| Key insight | The Policy Gradient Theorem states: |
Worked Example: CartPole with REINFORCE
| Environment | Balance a pole on a cart |
| State | [cart_position, cart_velocity, pole_angle, pole_angular_velocity] |
| Actions | Push left (0) or push right (1) |
| Reward | +1 for each timestep pole stays upright |
| Policy network: State(4) | Dense(32, ReLU) → Dense(2, Softmax) → P(left), P(right) |
| For step 0 | ∇_θ log π(right|s0) × 13.1 |
| Meaning | "Taking 'right' in s0 led to decent return, increase its probability" |
| G_0 = much larger | stronger gradient signal |
| Network has learned | when pole tilts right, push right to correct |
| After 500 episodes | Policy achieves maximum score of 200 |
Reducing Variance with Baselines
REINFORCE has high variance because returns vary greatly between episodes. Subtracting a baseline reduces variance without introducing bias:
Gradient with baseline
∇_θ J(θ) = E[∇_θ log π_θ(a_t|s_t) × (G_t - b(s_t))]
Common baselines
- Average return: b = mean(G) across episodes
- Value function: b(s_t) = V(s_t) (learned estimate of expected return)
Advantage function
A(s_t, a_t) = G_t - V(s_t)
"How much better was this action compared to average?"
If A > 0: Action was better than expected → increase probability
If A < 0: Action was worse than expected → decrease probability
Proximal Policy Optimization (PPO)
PPO is the most widely used policy gradient algorithm in practice, combining simplicity with stability:
PPO Objective
L(θ) = E[min(r_t(θ) × A_t, clip(r_t(θ), 1-ε, 1+ε) × A_t)]
Where r_t(θ) = π_θ(a_t|s_t) / π_θ_old(a_t|s_t) (probability ratio)
ε = 0.2 (clipping range)
The clipping prevents too-large policy updates
If advantage A > 0 (good action):
Increase probability, but cap the ratio at 1+ε = 1.2
If advantage A < 0 (bad action):
Decrease probability, but cap the ratio at 1-ε = 0.8
This ensures stable training without the complexity of
trust region methods (TRPO).
Continuous Action Spaces
For continuous actions (e.g., robot joint torques):
Policy outputs Gaussian parameters:
π_θ(a|s) = N(μ_θ(s), σ_θ(s))
Network: State → Dense layers → (μ, log σ)
Action: a = μ + σ × ε, where ε ~ N(0,1) (reparameterization)
log π_θ(a|s) = -0.5 × ((a-μ)/σ)² - log(σ) - 0.5×log(2π)
Gradient pushes μ toward actions that got high advantage,
and adjusts σ (exploration) based on uncertainty.
Comparison with Value-Based Methods
| Aspect | Policy Gradient | Value-Based (Q-learning) |
|---|---|---|
| Action space | Continuous natural | Discrete (or discretized) |
| Policy type | Stochastic | Deterministic (ε-greedy) |
| Convergence | Local optimum | Global (tabular) |
| Variance | High | Low |
| Bias | Low (unbiased) | Can be biased |
| Sample efficiency | Lower | Higher |
| Stability | PPO is stable | DQN needs tricks |
Applications
Policy gradients power many state-of-the-art RL applications: robotic manipulation (continuous joint control), game playing (AlphaGo uses policy gradients for move selection), natural language generation (RLHF uses PPO to align LLMs), and autonomous driving (continuous steering and throttle control).
Summary
Policy gradient methods offer a principled approach to reinforcement learning that directly optimizes the policy. From the foundational REINFORCE algorithm to modern PPO, these methods handle continuous actions, stochastic policies, and large-scale problems effectively. Understanding the policy gradient theorem, variance reduction through baselines, and the advantage function provides the theoretical foundation for applying these powerful algorithms to real-world control and decision-making problems.
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Policy Gradient Methods — Artificial Intelligence.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Artificial Intelligence topic.
Search Terms
artificial-intelligence, artificial intelligence, artificial, intelligence, reinforcement, learning, policy, gradient
Related Artificial Intelligence Topics