Reinforcement Learning: Introduction

Introduction to reinforcement learning: agents, environments, rewards, and the fundamentals of learning through interaction.

Reinforcement Learning (RL) is a paradigm where agents learn to make decisions by interacting with an environment, receiving rewards for good actions and penalties for bad ones.

Key Concepts

Agent & Environment

Example

Agent ↓ (action) Environment ↓ (state, reward) Agent

Agent: Entity making decisions (AI system) Environment: World the agent interacts with Action: Decision the agent makes State: Current situation in environment Reward: Signal of success (+) or failure (-)

Markov Decision Process (MDP)

Formal model for RL:

Example

(S, A, P, R, γ) S = state space (all possible states) A = action space (all possible actions) P(s'|s,a) = transition probability R(s,a) = reward for action in state γ = discount factor (0-1, importance of future)

Agents vs Supervised Learning

Aspect	Supervised	RL
Data	Labeled examples	Rewards from environment
Learning	From static data	From dynamic interaction
Feedback	Immediate (label)	Delayed (reward)
Goal	Minimize error	Maximize cumulative reward
Environment	None	Interactive
Exploration	No	Yes (trade-off)

RL Applications

Games: Chess, Go, Atari, StarCraft Robotics: Robot control, navigation, manipulation Trading: Financial market trading Recommendation: Dynamic recommendation systems Control: Manufacturing, resource allocation Healthcare: Treatment optimization

Core Components

Value Function

Value of being in state s:

Example

V(s) = expected cumulative reward from state s V(s) = E[Σ γ^t * R_t | S_0 = s]

Policy

Rule for choosing actions:

π(a|s) = probability of taking action a in state s

Deterministic: a = π(s)

Stochastic: P(a|s) = π(a|s)

Reward Signal

Immediate feedback:

Example

R_t = reward at time t Goal: maximize Σ γ^t * R_t (discounted sum)

Model

Optional: understanding environment dynamics:

Example

Model-based: Learn P(s'|s,a) and R(s,a) Model-free: Learn directly without model

Exploration vs Exploitation

Exploration: Try new actions to discover better strategies Exploitation: Use known best actions

Dilemma: Balance finding optimal policy vs maximizing current reward

Exploration Strategies

ε-Greedy: Choose random action with probability ε, otherwise best known

python example

if random() < epsilon:
    action = random_action()
else:
    action = best_known_action()

Softmax: Choose probabilistically based on value estimates

python example

p(a) = exp(Q(a)/T) / Σ exp(Q(a)/T)
T = temperature (higher = more exploration)

UCB (Upper Confidence Bound): Optimistic exploration

python example

action = argmax(Q(a) + c*sqrt(ln(N)/n(a)))

Types of RL Agents

1. Value-Based

Learn value function, derive policy.

Approach: Estimate V(s) or Q(s,a) Example: Q-Learning, DQN Best for: Discrete action spaces

2. Policy-Based

Learn policy directly.

Approach: Optimize π(a|s) Example: REINFORCE, A3C Best for: Continuous action spaces

3. Actor-Critic

Combine both value and policy.

Approach: Actor (policy) + Critic (value) Example: A3C, PPO Best for: Complex environments

Simple Example: CartPole

python example

import gym
import numpy as np

# Create environment
env = gym.make('CartPole-v1')
state = env.reset()

# Simple random policy
for episode in range(10):
    state = env.reset()
    total_reward = 0
    
    for step in range(500):
        # Random action
        action = env.action_space.sample()
        
        # Step environment
        next_state, reward, done, _ = env.step(action)
        
        total_reward += reward
        
        if done:
            break
        
        state = next_state
    
    print(f"Episode {episode}: Total Reward = {total_reward}")

env.close()

Learning Paradigms

Online Learning

Learn while interacting (most RL).

Pros: Uses real experience Cons: May not be sample efficient

Offline Learning

Learn from pre-recorded data.

Pros: Can learn from expert demonstrations Cons: Can't explore new strategies

Batch Learning

Learn from batches of experience.

Pros: Can use GPU parallelization Cons: More complex implementation

Challenges in RL

Sample Efficiency: Need many interactions to learn
Exploration: Balance exploring vs exploiting
Reward Shaping: Designing appropriate rewards
Non-Stationarity: Environment or agent changes
Credit Assignment: Which actions led to reward?
Scalability: Large state/action spaces

Summary

RL fundamentals:

Agent learns by interacting with environment
Receives rewards for actions taken
Goal: maximize cumulative reward
Requires balancing exploration/exploitation
Enables autonomous learning systems

Reinforcement Learning (RL) is a paradigm where agents learn to make decisions by interacting with an environment, receiving rewards for good actions and penalties for bad ones.

Key Concepts

Agent & Environment

Example

Agent ↓ (action) Environment ↓ (state, reward) Agent

Markov Decision Process (MDP)

Formal model for RL:

Example

Agents vs Supervised Learning

Aspect	Supervised	RL
Data	Labeled examples	Rewards from environment
Learning	From static data	From dynamic interaction
Feedback	Immediate (label)	Delayed (reward)
Goal	Minimize error	Maximize cumulative reward
Environment	None	Interactive
Exploration	No	Yes (trade-off)

RL Applications

Core Components

Value Function

Value of being in state s:

Example

V(s) = expected cumulative reward from state s V(s) = E[Σ γ^t * R_t | S_0 = s]

Policy

Rule for choosing actions:

π(a|s) = probability of taking action a in state s

Deterministic: a = π(s)

Stochastic: P(a|s) = π(a|s)

Reward Signal

Immediate feedback:

Example

R_t = reward at time t Goal: maximize Σ γ^t * R_t (discounted sum)

Model

Optional: understanding environment dynamics:

Example

Model-based: Learn P(s'|s,a) and R(s,a) Model-free: Learn directly without model

Exploration vs Exploitation

Exploration: Try new actions to discover better strategies Exploitation: Use known best actions

Dilemma: Balance finding optimal policy vs maximizing current reward

Exploration Strategies

ε-Greedy: Choose random action with probability ε, otherwise best known

python example

if random() < epsilon:
    action = random_action()
else:
    action = best_known_action()

Softmax: Choose probabilistically based on value estimates

python example

p(a) = exp(Q(a)/T) / Σ exp(Q(a)/T)
T = temperature (higher = more exploration)

UCB (Upper Confidence Bound): Optimistic exploration

python example

action = argmax(Q(a) + c*sqrt(ln(N)/n(a)))

Types of RL Agents

1. Value-Based

Learn value function, derive policy.

Approach: Estimate V(s) or Q(s,a) Example: Q-Learning, DQN Best for: Discrete action spaces

2. Policy-Based

Learn policy directly.

Approach: Optimize π(a|s) Example: REINFORCE, A3C Best for: Continuous action spaces

3. Actor-Critic

Combine both value and policy.

Approach: Actor (policy) + Critic (value) Example: A3C, PPO Best for: Complex environments

Simple Example: CartPole

python example

import gym
import numpy as np

# Create environment
env = gym.make('CartPole-v1')
state = env.reset()

# Simple random policy
for episode in range(10):
    state = env.reset()
    total_reward = 0
    
    for step in range(500):
        # Random action
        action = env.action_space.sample()
        
        # Step environment
        next_state, reward, done, _ = env.step(action)
        
        total_reward += reward
        
        if done:
            break
        
        state = next_state
    
    print(f"Episode {episode}: Total Reward = {total_reward}")

env.close()

Learning Paradigms

Online Learning

Learn while interacting (most RL).

Pros: Uses real experience Cons: May not be sample efficient

Offline Learning

Learn from pre-recorded data.

Pros: Can learn from expert demonstrations Cons: Can't explore new strategies

Batch Learning

Learn from batches of experience.

Pros: Can use GPU parallelization Cons: More complex implementation

Challenges in RL

Sample Efficiency: Need many interactions to learn
Exploration: Balance exploring vs exploiting
Reward Shaping: Designing appropriate rewards
Non-Stationarity: Environment or agent changes
Credit Assignment: Which actions led to reward?
Scalability: Large state/action spaces

Summary

RL fundamentals:

Agent learns by interacting with environment
Receives rewards for actions taken
Goal: maximize cumulative reward
Requires balancing exploration/exploitation
Enables autonomous learning systems

Key Concepts

Agent & Environment

Markov Decision Process (MDP)

Agents vs Supervised Learning

RL Applications

Core Components

Value Function

Policy

Reward Signal

Model

Exploration vs Exploitation

Exploration Strategies

Types of RL Agents

1. Value-Based

2. Policy-Based

3. Actor-Critic

Simple Example: CartPole

Learning Paradigms

Online Learning

Offline Learning

Batch Learning

Challenges in RL

Summary

Continue learning this concept

Key Concepts

Agent & Environment

Markov Decision Process (MDP)

Agents vs Supervised Learning

RL Applications

Core Components

Value Function

Policy

Reward Signal

Model

Exploration vs Exploitation

Exploration Strategies

Types of RL Agents

1. Value-Based

2. Policy-Based

3. Actor-Critic

Simple Example: CartPole

Learning Paradigms

Online Learning

Offline Learning

Batch Learning

Challenges in RL

Summary

Continue learning this concept