Actor-Critic Methods - Hybrid RL

The Actor-Critic Idea

Actor-critic methods combine the best of policy gradient methods (the "actor") with value function estimation (the "critic"). The actor learns a policy π(a|s) that decides which actions to take, while the critic learns a value function V(s) or Q(s,a) that evaluates how good those actions are. This dual architecture addresses fundamental limitations of using either approach alone.

Pure policy gradient methods (REINFORCE) have high variance because they estimate returns from complete episode trajectories—a single lucky or unlucky episode can send gradients in misleading directions. Pure value-based methods (DQN) can't easily handle continuous action spaces and may converge to suboptimal policies. Actor-critic methods reduce variance through the critic's baseline while maintaining the actor's ability to handle any action space.

Architecture

ACTOR		CRITIC
π(a\|s;θ)		V(s;w)
(policy)		(value)

The Advantage Function

The key insight: instead of using raw returns, use the advantage—how much better an action is compared to the average:

Advantage	A(s,a) = Q(s,a) - V(s)
If A > 0: action is better than average	increase probability
If A < 0: action is worse than average	decrease probability

Algorithm: One-Step Actor-Critic

Initialize: policy parameters θ, value parameters w

for each episode:

s ← initial state

while not terminal:

a ~ π(a|s; θ) // sample action from policy

s', r ← environment(s, a) // take action, observe

// Critic update (TD learning)

δ = r + γ·V(s'; w) - V(s; w) // TD error

w ← w + α_w · δ · ∇_w V(s; w) // update value

// Actor update (policy gradient with baseline)

θ ← θ + α_θ · δ · ∇_θ log π(a|s; θ) // update policy

s ← s'

Why Actor-Critic Reduces Variance

REINFORCE (pure policy gradient)

∇J ∝ Σ_t (G_t) · ∇log π(a_t|s_t)

G_t = r_t + γr_{t+1} + γ²r_{t+2} + ... (full return)

High variance: G_t depends on entire future trajectory

Actor-Critic

∇J ∝ Σ_t (δ_t) · ∇log π(a_t|s_t)

δ_t = r_t + γV(s_{t+1}) - V(s_t) (one-step TD error)

Lower variance: only depends on one transition + learned V

Trade-off: introduces bias (V is approximate)

But variance reduction is usually worth the small bias

A2C: Advantage Actor-Critic

Synchronous version with multiple parallel environments:

A2C Algorithm:

1. Run N environments in parallel for T steps

2. Compute advantages for all transitions:

A_t = r_t + γV(s_{t+1}) - V(s_t) (or n-step returns)

3. Compute losses:

Policy loss = -Σ log π(a_t|s_t) · A_t

Value loss = Σ (V(s_t) - target_t)²

Entropy bonus = -Σ π·log(π) (encourages exploration)

Total loss = policy_loss + c₁·value_loss - c₂·entropy

4. Update parameters with gradient descent

A3C: Asynchronous Advantage Actor-Critic

- Decorrelation	workers see different experiences
- Faster	parallel data collection
- Diverse exploration	different workers explore different regions

PPO: Proximal Policy Optimization

The most popular modern actor-critic algorithm:

Key idea	Limit how much the policy changes per update
ε = 0.2 (typical)	policy can change by at most 20% per update
If A_t > 0 (good action)	maximize r_t, but cap at 1+ε
If A_t < 0 (bad action)	minimize r_t, but cap at 1-ε

SAC: Soft Actor-Critic

Maximum entropy RL—adds entropy bonus to encourage exploration:

Objective	maximize Σ E[r_t + α·H(π(·\|s_t))]
- Actor	π(a\|s; θ)
- Two critics	Q₁(s,a; w₁), Q₂(s,a; w₂) (twin critics reduce overestimation)
Key benefit	Automatically balances exploration and exploitation

Practical Comparison

Algorithm	On/Off Policy	Continuous Actions	Sample Efficiency	Stability
A2C	On-policy	Yes	Low	High
PPO	On-policy	Yes	Medium	Very high
SAC	Off-policy	Yes	High	High
TD3	Off-policy	Yes	High	Medium
DDPG	Off-policy	Yes	High	Low

Interview Questions

Q: Why use an actor AND a critic instead of just one? A: The critic alone (like DQN) struggles with continuous actions (can't enumerate all actions to find the max). The actor alone (REINFORCE) has high variance updates. Together: the critic reduces variance by providing a learned baseline, and the actor can handle any action space naturally.

Q: What is the advantage function and why is it better than raw returns? A: Advantage A(s,a) = Q(s,a) - V(s) measures how much better action a is compared to average. Using advantages instead of raw returns subtracts the state-dependent baseline V(s), dramatically reducing variance without introducing bias. All actions in a high-reward state get credit under raw returns; advantages correctly credit only the action's contribution.

Q: Why is PPO so popular in practice? A: PPO is simple to implement (no complex trust region computation), stable (clipping prevents catastrophic updates), and works well across diverse tasks (games, robotics, language models). It's the default algorithm for RLHF in language models (ChatGPT training) because of its reliability and relatively low hyperparameter sensitivity.

The Actor-Critic Idea

Architecture

ACTOR		CRITIC
π(a\|s;θ)		V(s;w)
(policy)		(value)

The Advantage Function

The key insight: instead of using raw returns, use the advantage—how much better an action is compared to the average:

Advantage	A(s,a) = Q(s,a) - V(s)
If A > 0: action is better than average	increase probability
If A < 0: action is worse than average	decrease probability

Algorithm: One-Step Actor-Critic

Initialize: policy parameters θ, value parameters w

for each episode:

s ← initial state

while not terminal:

a ~ π(a|s; θ) // sample action from policy

s', r ← environment(s, a) // take action, observe

// Critic update (TD learning)

δ = r + γ·V(s'; w) - V(s; w) // TD error

w ← w + α_w · δ · ∇_w V(s; w) // update value

// Actor update (policy gradient with baseline)

θ ← θ + α_θ · δ · ∇_θ log π(a|s; θ) // update policy

s ← s'

Why Actor-Critic Reduces Variance

REINFORCE (pure policy gradient)

∇J ∝ Σ_t (G_t) · ∇log π(a_t|s_t)

G_t = r_t + γr_{t+1} + γ²r_{t+2} + ... (full return)

High variance: G_t depends on entire future trajectory

Actor-Critic

∇J ∝ Σ_t (δ_t) · ∇log π(a_t|s_t)

δ_t = r_t + γV(s_{t+1}) - V(s_t) (one-step TD error)

Lower variance: only depends on one transition + learned V

Trade-off: introduces bias (V is approximate)

But variance reduction is usually worth the small bias

A2C: Advantage Actor-Critic

Synchronous version with multiple parallel environments:

A2C Algorithm:

1. Run N environments in parallel for T steps

2. Compute advantages for all transitions:

A_t = r_t + γV(s_{t+1}) - V(s_t) (or n-step returns)

3. Compute losses:

Policy loss = -Σ log π(a_t|s_t) · A_t

Value loss = Σ (V(s_t) - target_t)²

Entropy bonus = -Σ π·log(π) (encourages exploration)

Total loss = policy_loss + c₁·value_loss - c₂·entropy

4. Update parameters with gradient descent

A3C: Asynchronous Advantage Actor-Critic

- Decorrelation	workers see different experiences
- Faster	parallel data collection
- Diverse exploration	different workers explore different regions

PPO: Proximal Policy Optimization

The most popular modern actor-critic algorithm:

Key idea	Limit how much the policy changes per update
ε = 0.2 (typical)	policy can change by at most 20% per update
If A_t > 0 (good action)	maximize r_t, but cap at 1+ε
If A_t < 0 (bad action)	minimize r_t, but cap at 1-ε

SAC: Soft Actor-Critic

Maximum entropy RL—adds entropy bonus to encourage exploration:

Objective	maximize Σ E[r_t + α·H(π(·\|s_t))]
- Actor	π(a\|s; θ)
- Two critics	Q₁(s,a; w₁), Q₂(s,a; w₂) (twin critics reduce overestimation)
Key benefit	Automatically balances exploration and exploitation

Practical Comparison

Algorithm	On/Off Policy	Continuous Actions	Sample Efficiency	Stability
A2C	On-policy	Yes	Low	High
PPO	On-policy	Yes	Medium	Very high
SAC	Off-policy	Yes	High	High
TD3	Off-policy	Yes	High	Medium
DDPG	Off-policy	Yes	High	Low

Actor-Critic Methods - Hybrid RL

The Actor-Critic Idea

Architecture

The Advantage Function

Algorithm: One-Step Actor-Critic

Why Actor-Critic Reduces Variance

REINFORCE (pure policy gradient)

Actor-Critic

A2C: Advantage Actor-Critic

A3C: Asynchronous Advantage Actor-Critic

PPO: Proximal Policy Optimization

SAC: Soft Actor-Critic

Practical Comparison

Interview Questions

Continue learning this concept

Actor-Critic Methods - Hybrid RL

The Actor-Critic Idea

Architecture

The Advantage Function

Algorithm: One-Step Actor-Critic

Why Actor-Critic Reduces Variance

REINFORCE (pure policy gradient)

Actor-Critic

A2C: Advantage Actor-Critic

A3C: Asynchronous Advantage Actor-Critic

PPO: Proximal Policy Optimization

SAC: Soft Actor-Critic

Practical Comparison

Interview Questions

Continue learning this concept