Reinforcement Learning Fundamentals — Artificial Intelligence

Introduction

Reinforcement Learning (RL) is a paradigm of machine learning where an agent learns to make decisions by interacting with an environment. Unlike supervised learning, where correct answers are provided, an RL agent discovers optimal behavior through trial and error, receiving rewards or penalties for its actions. This mirrors how humans and animals learn many skills: a child learns to walk not from labeled examples of walking, but from repeatedly trying, falling, and gradually improving based on the feedback of staying upright.

The RL Framework

	Agent	─action──▶	Environment
		◀─state───
		◀─reward──

Key Concepts

States, Actions, and Rewards

Example	Robot learning to navigate a maze
States (S)	All possible positions in the maze
Actions (A)	Available moves from current state
Rewards (R)	Feedback signal
Reaching goal	+100
Hitting wall	-1 (small penalty)
Each step	-0.1 (encourages finding shortest path)
The agent's objective	Maximize cumulative future reward
γ = 0.9	reward 10 steps away worth 0.9^10 = 0.35 of immediate

Policy

Policy π	The agent's strategy — maps states to actions
Deterministic policy	π(s) = a
Example	"At intersection, always turn right"
Stochastic policy	π(a\|s) = P(taking a in state s)
Example	"At intersection, turn right 70%, go straight 30%"
Optimal policy π*	Achieves maximum expected cumulative reward

Value Functions

State-Value Function V^π(s):

"How good is it to be in state s, following policy π?"

V^π(s) = E[G_t | s_t = s, following π]

Action-Value Function Q^π(s, a):

"How good is it to take action a in state s, then follow π?"

Q^π(s, a) = E[G_t | s_t = s, a_t = a, then follow π]

Relationship:

V^π(s) = Σ_a π(a|s) × Q^π(s, a)

(Value of state = expected Q over action distribution)

Optimal value functions:

V*(s) = max_π V^π(s) (best possible value for state s)

Q*(s,a) = max_π Q^π(s,a) (best possible value for state-action pair)

Worked Example: Grid World

Reward	+1 at goal, -0.04 per step (time penalty)
[	→ → ↓]
[	→ → ↓]
[	→ → ↓]
[	→ → ★] ← goal
Episode 1: Random walk, takes 50 steps	low cumulative reward
Episode 100: Mostly goes right/down	moderate reward
Episode 1000: Nearly optimal path every time	high reward

Types of RL Problems

Model-Based vs Model-Free

Model-based: Agent learns how environment works (transition function)

P(s'|s,a) — "if I'm here and go right, where do I end up?"

Can plan ahead mentally without taking real actions

Model-free: Agent learns directly from experience

No model of environment — just learns values/policy from interaction

Simpler but may need more experience

On-Policy vs Off-Policy

On-policy: Learn about the policy currently being used

SARSA: Update Q(s,a) based on action actually taken

Off-policy: Learn about optimal policy while behaving differently

Q-Learning: Update Q(s,a) based on best possible next action

Can learn from old data or other agents' experiences

Episodic vs Continuing

Episodic: Clear start and end (games, maze navigation)

Continuing: Goes on forever (robot maintenance, stock trading)

Exploration vs Exploitation

The fundamental dilemma

Exploitation: Use current knowledge to maximize reward (go to known good restaurant)

Exploration: Try new things to discover potentially better options (try new restaurant)

Strategies

ε-greedy: With probability ε take random action, else take best known

ε = 0.1 means 10% exploration, 90% exploitation

Decay ε over time: explore more early, exploit more later

Softmax (Boltzmann): Choose actions proportional to their estimated value

P(a) = e^(Q(a)/τ) / Σ e^(Q(a')/τ)

τ (temperature): high = more random, low = more greedy

UCB (Upper Confidence Bound): Prefer uncertain actions

Choose a = argmax[Q(a) + c√(ln(t)/N(a))]

Untried actions have infinite UCB → always tried at least once

Applications of RL

Game Playing: AlphaGo combined deep RL with Monte Carlo tree search to defeat world champion in Go. Atari games solved from raw pixels using Deep Q-Networks.

Robotics: Learning locomotion (walking, running), manipulation (grasping objects), and navigation without manual programming of each behavior.

Resource Management: Data center cooling (Google reduced energy by 40%), network routing, cloud computing resource allocation.

Language Models: RLHF (Reinforcement Learning from Human Feedback) aligns GPT/ChatGPT responses with human preferences using PPO.

Summary

Reinforcement learning provides a framework for sequential decision-making under uncertainty. Through the interplay of states, actions, rewards, and policies, agents discover optimal behavior by interacting with their environment. Understanding the core concepts of value functions, the exploration-exploitation tradeoff, and the distinction between model-based and model-free methods provides the foundation for all advanced RL algorithms including Q-learning, policy gradients, and actor-critic methods.

Introduction

The RL Framework

	Agent	─action──▶	Environment
		◀─state───
		◀─reward──

Key Concepts

States, Actions, and Rewards

Example	Robot learning to navigate a maze
States (S)	All possible positions in the maze
Actions (A)	Available moves from current state
Rewards (R)	Feedback signal
Reaching goal	+100
Hitting wall	-1 (small penalty)
Each step	-0.1 (encourages finding shortest path)
The agent's objective	Maximize cumulative future reward
γ = 0.9	reward 10 steps away worth 0.9^10 = 0.35 of immediate

Policy

Policy π	The agent's strategy — maps states to actions
Deterministic policy	π(s) = a
Example	"At intersection, always turn right"
Stochastic policy	π(a\|s) = P(taking a in state s)
Example	"At intersection, turn right 70%, go straight 30%"
Optimal policy π*	Achieves maximum expected cumulative reward

Value Functions

State-Value Function V^π(s):

"How good is it to be in state s, following policy π?"

V^π(s) = E[G_t | s_t = s, following π]

Action-Value Function Q^π(s, a):

"How good is it to take action a in state s, then follow π?"

Q^π(s, a) = E[G_t | s_t = s, a_t = a, then follow π]

Relationship:

V^π(s) = Σ_a π(a|s) × Q^π(s, a)

(Value of state = expected Q over action distribution)

Optimal value functions:

V*(s) = max_π V^π(s) (best possible value for state s)

Q*(s,a) = max_π Q^π(s,a) (best possible value for state-action pair)

Worked Example: Grid World

Reward	+1 at goal, -0.04 per step (time penalty)
[	→ → ↓]
[	→ → ↓]
[	→ → ↓]
[	→ → ★] ← goal
Episode 1: Random walk, takes 50 steps	low cumulative reward
Episode 100: Mostly goes right/down	moderate reward
Episode 1000: Nearly optimal path every time	high reward

Types of RL Problems

Model-Based vs Model-Free

Model-based: Agent learns how environment works (transition function)

P(s'|s,a) — "if I'm here and go right, where do I end up?"

Can plan ahead mentally without taking real actions

Model-free: Agent learns directly from experience

No model of environment — just learns values/policy from interaction

Simpler but may need more experience

On-Policy vs Off-Policy

On-policy: Learn about the policy currently being used

SARSA: Update Q(s,a) based on action actually taken

Off-policy: Learn about optimal policy while behaving differently

Q-Learning: Update Q(s,a) based on best possible next action

Can learn from old data or other agents' experiences

Episodic vs Continuing

Episodic: Clear start and end (games, maze navigation)

Continuing: Goes on forever (robot maintenance, stock trading)

Exploration vs Exploitation

The fundamental dilemma

Exploitation: Use current knowledge to maximize reward (go to known good restaurant)

Exploration: Try new things to discover potentially better options (try new restaurant)

Strategies

ε-greedy: With probability ε take random action, else take best known

ε = 0.1 means 10% exploration, 90% exploitation

Decay ε over time: explore more early, exploit more later

Softmax (Boltzmann): Choose actions proportional to their estimated value

P(a) = e^(Q(a)/τ) / Σ e^(Q(a')/τ)

τ (temperature): high = more random, low = more greedy

UCB (Upper Confidence Bound): Prefer uncertain actions

Choose a = argmax[Q(a) + c√(ln(t)/N(a))]

Untried actions have infinite UCB → always tried at least once

Applications of RL

Game Playing: AlphaGo combined deep RL with Monte Carlo tree search to defeat world champion in Go. Atari games solved from raw pixels using Deep Q-Networks.

Robotics: Learning locomotion (walking, running), manipulation (grasping objects), and navigation without manual programming of each behavior.

Resource Management: Data center cooling (Google reduced energy by 40%), network routing, cloud computing resource allocation.

Language Models: RLHF (Reinforcement Learning from Human Feedback) aligns GPT/ChatGPT responses with human preferences using PPO.

Reinforcement Learning Fundamentals — Artificial Intelligence

Introduction

The RL Framework

Key Concepts

States, Actions, and Rewards

Policy

Value Functions

Worked Example: Grid World

Types of RL Problems

Model-Based vs Model-Free

On-Policy vs Off-Policy

Episodic vs Continuing

Exploration vs Exploitation

The fundamental dilemma

Strategies

Applications of RL

Summary

Continue learning this concept

Reinforcement Learning Fundamentals — Artificial Intelligence

Introduction

The RL Framework

Key Concepts

States, Actions, and Rewards

Policy

Value Functions

Worked Example: Grid World

Types of RL Problems

Model-Based vs Model-Free

On-Policy vs Off-Policy

Episodic vs Continuing

Exploration vs Exploitation

The fundamental dilemma

Strategies

Applications of RL

Summary

Continue learning this concept