

Reinforcement Learning · Lecture 2
The RL problem and Finite MDP

Rafael Fernandes Cunha
Today’s topics
1.Recap on exploration methods
2.The Reinforcement Learning (RL) framework
3.The Markov Decision Process (MDP)
4.Returns
5.Policies
6.Value functions
7.Bellman equations
2
1.Describe the RL framework.
2.Create an environment as an MDP.
3.Describe the goal of the RL agent.
4.Describe why value functions can be used to
evaluate a policy.
5.Explain the recursive nature of Bellman equations.
Learning outcomes
Recap on Exploration Methods
Formalizing the Multi-Armed Bandit problem
➢Multi-armed bandit: Set of distributions
➢At each step , the agent selects an action
➢The environment generates a reward



To achieve the goal, we learn a policy : a distribution on


Goal: Maximize cumulative reward
Update rule
NewEstimate
OldEstimate
StepSize
InformativeValue
Target – OldEstimate
Error
Gradient
Learning Rate
Fixed or variable
Vector
Table
Weights of a NN


Action-value update rule
Sample-average method
Weight-average method
![\begin{align*}
Q_{n+1}(a) = Q_n(a) + \alpha_n(a) \Big[ R_n - Q_n(a) \Big]
\end{align*}](media/870706deab.png)
From stochastic
approximation theory
Conditions required to
assure convergence
with probability 1
Steps are large enough to overcome any
initial conditions or random fluctuations
Steps become small enough to
assure convergence
Sequences that meet these conditions are usually:
➢Slow to converge.
➢Need considerable tunning for a satisfactory convergence rate.
➢Often used in theoretical work.
➢Not very used in applications and empirical research.
Sample-average method
(more suitable for stationary bandit)
Action-value function update equation
Weight-average method
(suitable for stationary and nonstationary bandit)

![\begin{align*}
Q_{n+1}(a) = Q_n(a) + \alpha_n(a) \Big[ R_n - Q_n(a) \Big]
\end{align*}](media/870706deab.png)

Algorithms: Action-value methods
Greedy
UCB
Softmax
-greedy
Action-value update rule
Policies
![\begin{align*}
Q_{n+1}(a) = Q_n(a) + \alpha_n(a) \Big[ R_n - Q_n(a) \Big]
\end{align*}](media/870706deab.png)
Method | Stochasticity | Exploration Strategy | Ability to Escape Local Optima |
Greedy | |||
𝜖-greedy | |||
UCB | |||
SoftMax |
Algorithms: Summary
Method | Stochasticity | Exploration Strategy | Ability to Escape Local Optima |
Greedy | Deterministic | ||
𝜖-greedy | Stochastic | ||
UCB | Deterministic | ||
SoftMax | Stochastic |
Algorithms: Summary
Method | Stochasticity | Exploration Strategy | Ability to Escape Local Optima |
Greedy | Deterministic | No exploration | |
𝜖-greedy | Stochastic | Random actions with a small probability | |
UCB | Deterministic | Guided by uncertainty of action-value estimates | |
SoftMax | Stochastic | Probabilistic selection based on action-value |
Algorithms: Summary
Method | Stochasticity | Exploration Strategy | Ability to Escape Local Optima |
Greedy | Deterministic | No exploration | No (can get trapped in a suboptimal action) |
𝜖-greedy | Stochastic | Random actions with a small probability | Yes (due to random exploration) |
UCB | Deterministic | Guided by uncertainty of action-value estimates | Yes (due to uncertainty-based exploration) |
SoftMax | Stochastic | Probabilistic selection based on action-value | Yes (assigns non-zero probability to all actions) |
Algorithms: Summary
The multi-armed bandit problem
•You are in a casino with many slot machines.
•You will play 1000 times.
•Each time, you choose one machine.
•Your goal is to win as much money as possible.

How do you choose which slot
machine to play each time?
Recap

Parameter study of various bandit algorithms. Each point is the average reward obtained
over 1000 steps with a particular algorithm at a particular setting of its parameter.
See book’s page 42 for details.
Frozen Lake game
How is this different from the bandit problem?

You will play this game 1000 times.
Your goal is to get the most points out of this experience.
You only know your position, the actions you can take, and the points
that you are receiving!
How do you choose your actions?
Frozen Lake game
At each time-step , the agent is in state , takes an action,
receives a reward , and transits to next-state
At each episode, we have a trajectory:

where T is the time at the final state.
Is it possible to learn which sequences of actions will maximize
our total points after playing the game 1000 times by only
having access to the data generated from the trajectories?
The reward is the only information that tells us if our
actions were good or not! Is this enough?
Yes! According to the reward hypothesis!

Frozen Lake Visualization.
Reward hypothesis
Any goal can be formalized as the outcome of maximizing a cumulative reward.
Reinforcement learning is based on the reward hypothesis:
Agent and Environment



Decision Maker
Frozen Lake game
Let’s understand and model the game.
What are the locations where the doll can be?
Which actions can the doll take?
After taking an action, how does the doll change its location?

Frozen Lake game
What are the locations where the doll can be?
Which actions can the doll take?
After taking an action, how does the doll change its location?
State Space =
Action Space = {up, down, left, right}
Transition function: Given state and action, provides the new state.

Frozen Lake game
Transition function: Given state and action, provides the new state.

The player may not always move in
the intended direction due to the
slippery nature of the frozen lake!
How can I represent this transitions mathematically?
State:
New State:
Action: right
State:
New State:
Action: right
Frozen Lake game

How can I represent this transitions mathematically?
State space


State random
variable at time t
Action space

up
down
left
right

Action random
variable at time t









Transition
function

Frozen Lake game
How can I represent this transitions mathematically?

Transition
function
State space

Action space

![\begin{align*}
p : \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow [0, 1]
\end{align*}](media/fc1f5cf58b.png)
What about the rewards?




Random variable
Expected reward,
achieved on a transition
starting in (s, a)
![\begin{align*}
r(s,a) = \mathbb{E}[R_t|S_{t-1} = s, A_{t-1} = a]
\end{align*}](media/ed3798409f.png)

A Markov Decision Process (MDP) is a tuple
Markov Decision Process

Set of all
possible states
Set of all
possible actions

![\begin{align*}
p : \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow [0, 1]
\end{align*}](media/fc1f5cf58b.png)
Probability of transitioning to , given a state and action .
Dynamics of the system.
Transition Function
Discount factor
![\begin{align*}
\gamma \in [0, 1]
\end{align*}](media/110f333da6.png)
Expected reward, achieved on a
transition starting in (s, a)


Markov Decision Process
A Markov Decision Process (MDP) is a tuple

Environment

Algorithm
MDP
A Markov Decision Process (MDP) is a tuple

Markov Decision Process


If we define
then
![\begin{align*}
\mathbb{E}[R_t|S_{t-1} = s, A_{t-1} = a] = \sum_r r \sum_{s'} p(s', r|s, a)
\end{align*}](media/b33a4fe5ab.png)
Markov Property
The future is independent of the past given the present.
Define history as all previous
states, actions, and rewards.

In a Markov Decision Process all states are assumed to have the Markov property.

Markov Property
➢The state captures all relevant information from the history.
➢Once the state is known, the history may be thrown away.
➢The state is a sufficient statistic of the past.




Markov Property
The future is independent of the past given the present.

Mountain car
Observations
Position of the car
Markov Property
Screen Image

Car Racing
Observations
Markov Property
Sequence of 4 frames
of the screen Image

Frozen Lake
Observations
Markov
Property
Position of the doll

Position, velocity, and
acceleration of the car
Markov Property
The future is independent of the past given the present.

Which of the following statements are true in a MDP?








Partially Observable Environments
Partial observability: The observations are not Markovian
➢A robot with camera vision isn’t told its absolute location
➢A poker playing agent only observes public cards
Using the observation as state would not be Markovian
This is called a partially observable Markov decision process (POMDP)
The environment state can still be Markov, but the agent does not know it
We might still be able to construct an “artificial” MDP from a POMDP. This will not be
covered in this course.
Example: cleaning robot

s | a | s‘ | p(s’ | s, a) | r(s, a, s’ ) |
H | W | H | 1 | |
H | W | L | 0 | - |
H | S | H | ||
H | S | L | ||
L | W | H | 0 | - |
L | W | L | 1 | |
L | S | H | -3 | |
L | S | L | ||
L | re | H | 1 | 0 |
L | re | L | 0 | - |

Example of a transition function



Example: cleaning robot

s | a | s‘ | p(s’ | s, a) | r(s, a, s’ ) |
H | W | H | 1 | |
H | W | L | 0 | - |
H | S | H | ||
H | S | L | ||
L | W | H | 0 | - |
L | W | L | 1 | |
L | S | H | -3 | |
L | S | L | ||
L | re | H | 1 | 0 |
L | re | L | 0 | - |

Example of a transition function




Policy

![\begin{align*}
\pi : \mathcal{S} \times \mathcal{A} \rightarrow [0, 1] \;\text{ for all }\; s \in \mathcal{S}, a \in \mathcal{A}(s)
\end{align*}](media/9976509826.png)
Policy is a mapping from states to probabilities of selecting each
possible action.
If a policy is deterministic, its representation can be simplified to:



Returns
Is return a random variable?
Undiscounted return (episodic / finite horizon problems)
Return depends on rewards.
Are rewards random variables?

Each episode ends in a terminal state.
Final time step.
Return is a random variable that
depends on MDP and policy!

Returns
Is return a random variable?
Undiscounted return (episodic / finite horizon problems)
Return depends on rewards.
Are rewards random variables?


Each episode ends in a terminal state.
Final time step.


Goal: Maximize what?
Expected return!
?
Goal of an RL agent
To find a target policy that maximizes the expected
return
Returns
Undiscounted return (episodic / finite horizon problems)

Discounted return (finite or infinite horizon problems)

![\begin{align*}
\gamma \in [0, 1]
\end{align*}](media/110f333da6.png)
➢For , immediate rewards are more important than delayed rewards
➢ close to 0 leads to “myopic” evaluation
➢ close to 1 leads to “far-sighted” evaluation
Why discount?
Most Markov decision processes are discounted.
➢Problem specification:
➢Immediate rewards may actually be more valuable (e.g., earning interest)
➢Animal/human behavior shows preference for immediate reward
➢Solution side:
➢Mathematically convenient to discount rewards
➢Avoids infinite returns in cyclic Markov processes
Reward and discount together determine the goal!
Maximize expected return
Returns
Unified Notation for Episodic and Continuing Tasks

Episodic

Continuing

Absorbing state


Unified Notation
Continuing
Episodic

Returns
Returns at successive time steps are related to each other



Returns: Gridworld Example
Gridworld with as terminal state.
State Space:
Action Space:
Transitions: Deterministic
Which policy is better?
Return with starting at:
Trajectory:
:
Returns: Gridworld Example
Gridworld with as terminal state.
State Space:
Action Space:
Transitions: Deterministic
Which policy is better?
Return with starting at:
:
Return with starting at:
:
Returns: Gridworld Example
Gridworld with as terminal state.
State Space:
Action Space:
Transitions: Deterministic
Which policy is better?
Return with starting at:
:
Return with starting at:
:
9 | 10 |
8 | 0 |
9 | 10 |
10 | 0 |
State-value function:
![\begin{align*}
v_\pi(s) \doteq \mathbb{E}_\pi[G_t|S_t = s]
\end{align*}](media/db589aa488.png)
Comparing policies
Goal of RL: To find a target policy that maximizes the expected return
![\begin{align*}
v_\pi(s) \doteq \mathbb{E}_\pi[G_t|S_t = s]
\end{align*}](media/db589aa488.png)
The state-value function gives the expected return when starting in state s, and
following the policy thereafter.

Value functions can be used to define a partial ordering over policies!
How can we evaluate how good the found policy is?
Comparing Policies: Ex1
Which policy is better?







Comparing Policies: Ex2
Which policy is better?

Is
?
It is NOT defined
which policy is better.





Recursive relationships for the value functions
Write as a function of the value function in the next state
Transition:
Recursive relationships for the value functions
![\begin{align*}
v_\pi(s) &\doteq \mathbb{E}_\pi[G_t|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1} + \gamma G_{t+1}|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1}|S_t = s] + \mathbb{E}_\pi[\gamma G_{t+1}|S_t = s]
\end{align*}](media/f45cfd4220.png)
![\begin{align*}
v_\pi(s) &\doteq \mathbb{E}_\pi[G_t|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1} + \gamma G_{t+1}|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1}|S_t = s] + \mathbb{E}_\pi[\gamma G_{t+1}|S_t = s]
\end{align*}](media/f45cfd4220.png)
![\begin{align*}
v_\pi(s) &\doteq \mathbb{E}_\pi[G_t|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1} + \gamma G_{t+1}|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1}|S_t = s] + \mathbb{E}_\pi[\gamma G_{t+1}|S_t = s]
\end{align*}](media/f45cfd4220.png)

![\begin{align*}
r(s,a) = \mathbb{E}[R_t|S_{t-1} = s, A_{t-1} = a]
\end{align*}](media/92f91fa46d.png)
![\begin{align*}
\mathbb{E}_\pi[R_{t+1}|S_t = s] &= \sum_a \pi(a|s) \mathbb{E}[R_{t+1}|S_t = s, A_t = a] \\ &= \sum_a \pi(a|s) r(s,a)
\end{align*}](media/76bcf8e71e.png)
![\begin{align*}
\mathbb{E}_\pi[R_{t+1}|S_t = s] &= \sum_a \pi(a|s) \mathbb{E}[R_{t+1}|S_t = s, A_t = a] \\ &= \sum_a \pi(a|s) r(s,a)
\end{align*}](media/76bcf8e71e.png)
![\begin{align*}
\mathbb{E}_\pi[R_{t+1}|S_t = s] &= \sum_a \pi(a|s) \mathbb{E}[R_{t+1}|S_t = s, A_t = a] \\ &= \sum_a \pi(a|s) r(s,a)
\end{align*}](media/76bcf8e71e.png)
Recursive relationships for the value functions
![\begin{align*}
v_\pi(s) &\doteq \mathbb{E}_\pi[G_t|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1} + \gamma G_{t+1}|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1}|S_t = s] + \mathbb{E}_\pi[\gamma G_{t+1}|S_t = s]
\end{align*}](media/f45cfd4220.png)
![\begin{align*}
v_\pi(s) &\doteq \mathbb{E}_\pi[G_t|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1} + \gamma G_{t+1}|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1}|S_t = s] + \mathbb{E}_\pi[\gamma G_{t+1}|S_t = s]
\end{align*}](media/f45cfd4220.png)
![\begin{align*}
v_\pi(s) &\doteq \mathbb{E}_\pi[G_t|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1} + \gamma G_{t+1}|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1}|S_t = s] + \mathbb{E}_\pi[\gamma G_{t+1}|S_t = s]
\end{align*}](media/f45cfd4220.png)

![\begin{align*}
r(s,a) = \mathbb{E}[R_t|S_{t-1} = s, A_{t-1} = a]
\end{align*}](media/92f91fa46d.png)
![\begin{align*}
\mathbb{E}_\pi[R_{t+1}|S_t = s] &= \sum_a \pi(a|s) \mathbb{E}[R_{t+1}|S_t = s, A_t = a] \\ &= \sum_a \pi(a|s) r(s,a)
\end{align*}](media/76bcf8e71e.png)
![\begin{align*}
\mathbb{E}_\pi[R_{t+1}|S_t = s] &= \sum_a \pi(a|s) \mathbb{E}[R_{t+1}|S_t = s, A_t = a] \\ &= \sum_a \pi(a|s) r(s,a)
\end{align*}](media/76bcf8e71e.png)
Recursive relationships for the value functions
![\begin{align*}
v_\pi(s) &\doteq \mathbb{E}_\pi[G_t|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1} + \gamma G_{t+1}|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1}|S_t = s] + \mathbb{E}_\pi[\gamma G_{t+1}|S_t = s]
\end{align*}](media/f45cfd4220.png)
![\begin{align*}
v_\pi(s) &\doteq \mathbb{E}_\pi[G_t|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1} + \gamma G_{t+1}|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1}|S_t = s] + \mathbb{E}_\pi[\gamma G_{t+1}|S_t = s]
\end{align*}](media/f45cfd4220.png)

![\begin{align*}
r(s,a) = \mathbb{E}[R_t|S_{t-1} = s, A_{t-1} = a]
\end{align*}](media/92f91fa46d.png)
![\begin{align*}
v_\pi(s) &\doteq \mathbb{E}_\pi[G_t|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1} + \gamma G_{t+1}|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1}|S_t = s] + \mathbb{E}_\pi[\gamma G_{t+1}|S_t = s]
\end{align*}](media/f45cfd4220.png)
![\begin{align*}
\mathbb{E}_\pi[\gamma G_{t+1}|S_t = s] &= \gamma\sum_{a}\pi(a|s)\mathbb{E}_\pi[G_{t+1}|S_t = s, A_t = a] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)\mathbb{E}_\pi[G_{t+1}|S_t = s, A_t = a, S_{t+1} = s'] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)\mathbb{E}_\pi[G_{t+1}|S_{t+1} = s'] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)v_\pi(s')
\end{align*}](media/477674e2b0.png)
![\begin{align*}
\mathbb{E}_\pi[\gamma G_{t+1}|S_t = s] &= \gamma\sum_{a}\pi(a|s)\mathbb{E}_\pi[G_{t+1}|S_t = s, A_t = a] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)\mathbb{E}_\pi[G_{t+1}|S_t = s, A_t = a, S_{t+1} = s'] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)\mathbb{E}_\pi[G_{t+1}|S_{t+1} = s'] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)v_\pi(s')
\end{align*}](media/477674e2b0.png)
![\begin{align*}
\mathbb{E}_\pi[\gamma G_{t+1}|S_t = s] &= \gamma\sum_{a}\pi(a|s)\mathbb{E}_\pi[G_{t+1}|S_t = s, A_t = a] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)\mathbb{E}_\pi[G_{t+1}|S_t = s, A_t = a, S_{t+1} = s'] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)\mathbb{E}_\pi[G_{t+1}|S_{t+1} = s'] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)v_\pi(s')
\end{align*}](media/477674e2b0.png)
![\begin{align*}
\mathbb{E}_\pi[\gamma G_{t+1}|S_t = s] &= \gamma\sum_{a}\pi(a|s)\mathbb{E}_\pi[G_{t+1}|S_t = s, A_t = a] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)\mathbb{E}_\pi[G_{t+1}|S_t = s, A_t = a, S_{t+1} = s'] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)\mathbb{E}_\pi[G_{t+1}|S_{t+1} = s'] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)v_\pi(s')
\end{align*}](media/477674e2b0.png)
![\begin{align*}
\mathbb{E}_\pi[\gamma G_{t+1}|S_t = s] &= \gamma\sum_{a}\pi(a|s)\mathbb{E}_\pi[G_{t+1}|S_t = s, A_t = a] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)\mathbb{E}_\pi[G_{t+1}|S_t = s, A_t = a, S_{t+1} = s'] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)\mathbb{E}_\pi[G_{t+1}|S_{t+1} = s'] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)v_\pi(s')
\end{align*}](media/477674e2b0.png)
![\begin{align*}
\mathbb{E}_\pi[R_{t+1}|S_t = s] &= \sum_a \pi(a|s) \mathbb{E}[R_{t+1}|S_t = s, A_t = a] \\ &= \sum_a \pi(a|s) r(s,a)
\end{align*}](media/76bcf8e71e.png)
![\begin{align*}
\mathbb{E}_\pi[R_{t+1}|S_t = s] &= \sum_a \pi(a|s) \mathbb{E}[R_{t+1}|S_t = s, A_t = a] \\ &= \sum_a \pi(a|s) r(s,a)
\end{align*}](media/76bcf8e71e.png)
![\begin{align*}
v_\pi(s') = \mathbb{E}_\pi[G_{t+1}|S_{t+1} = s']
\end{align*}](media/6af3d1125e.png)
Markov Property
Recursive relationships for the value functions
![\begin{align*}
v_\pi(s) &\doteq \mathbb{E}_\pi[G_t|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1} + \gamma G_{t+1}|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1}|S_t = s] + \mathbb{E}_\pi[\gamma G_{t+1}|S_t = s]
\end{align*}](media/f45cfd4220.png)
![\begin{align*}
v_\pi(s) &\doteq \mathbb{E}_\pi[G_t|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1} + \gamma G_{t+1}|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1}|S_t = s] + \mathbb{E}_\pi[\gamma G_{t+1}|S_t = s]
\end{align*}](media/f45cfd4220.png)

![\begin{align*}
r(s,a) = \mathbb{E}[R_t|S_{t-1} = s, A_{t-1} = a]
\end{align*}](media/92f91fa46d.png)
![\begin{align*}
v_\pi(s) &\doteq \mathbb{E}_\pi[G_t|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1} + \gamma G_{t+1}|S_t = s] \\ &= \mathbb{E}_\pi[R_{t+1}|S_t = s] + \mathbb{E}_\pi[\gamma G_{t+1}|S_t = s]
\end{align*}](media/f45cfd4220.png)
![\begin{align*}
\mathbb{E}_\pi[\gamma G_{t+1}|S_t = s] &= \gamma\sum_{a}\pi(a|s)\mathbb{E}_\pi[G_{t+1}|S_t = s, A_t = a] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)\mathbb{E}_\pi[G_{t+1}|S_t = s, A_t = a, S_{t+1} = s'] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)\mathbb{E}_\pi[G_{t+1}|S_{t+1} = s'] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)v_\pi(s')
\end{align*}](media/477674e2b0.png)
![\begin{align*}
\mathbb{E}_\pi[\gamma G_{t+1}|S_t = s] &= \gamma\sum_{a}\pi(a|s)\mathbb{E}_\pi[G_{t+1}|S_t = s, A_t = a] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)\mathbb{E}_\pi[G_{t+1}|S_t = s, A_t = a, S_{t+1} = s'] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)\mathbb{E}_\pi[G_{t+1}|S_{t+1} = s'] \\ &= \gamma\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)v_\pi(s')
\end{align*}](media/477674e2b0.png)
![\begin{align*}
\mathbb{E}_\pi[R_{t+1}|S_t = s] &= \sum_a \pi(a|s) \mathbb{E}[R_{t+1}|S_t = s, A_t = a] \\ &= \sum_a \pi(a|s) r(s,a)
\end{align*}](media/76bcf8e71e.png)
![\begin{align*}
\mathbb{E}_\pi[R_{t+1}|S_t = s] &= \sum_a \pi(a|s) \mathbb{E}[R_{t+1}|S_t = s, A_t = a] \\ &= \sum_a \pi(a|s) r(s,a)
\end{align*}](media/76bcf8e71e.png)

Bellman expectation equation for the state-value function
Recursive relationships for the value functions

Bellman Expectation equation for the state-value function
Alternatively



Bellman Expectation equation

Bellman Expectation equation for the state-value function
➢ and are state values to be calculated. Bootstrapping!
➢ is a given policy. Solving the equation is called policy evaluation.
➢ represents the dynamics of the MDP.
Backup diagram


Returns: Gridworld Example
Gridworld with as terminal state.
State Space:
Action Space:
Transitions: Deterministic
9 | 10 |
8 | 0 |
![\begin{align*}
q_\pi(s,a) \doteq \mathbb{E}_\pi[G_t|S_t = s, A_t = a]
\end{align*}](media/4d6ed2657a.png)
(state-)action value function
Returns: Gridworld Example
Gridworld with as terminal state.
State Space:
Action Space:
Transitions: Deterministic
9 | 10 |
8 | 0 |
![\begin{align*}
q_\pi(s,a) \doteq \mathbb{E}_\pi[G_t|S_t = s, A_t = a]
\end{align*}](media/4d6ed2657a.png)
(state-)action value function
9
10
8
0
0
0
0
Returns: Gridworld Example
Gridworld with as terminal state.
State Space:
Action Space:
Transitions: Deterministic
9 | 10 |
8 | 0 |
![\begin{align*}
q_\pi(s,a) \doteq \mathbb{E}_\pi[G_t|S_t = s, A_t = a]
\end{align*}](media/4d6ed2657a.png)
(state-)action value function
9
10
8
0
0
0
0
7
Value Functions
![\begin{align*}
v_\pi(s) \doteq \mathbb{E}_\pi[G_t|S_t = s]
\end{align*}](media/db589aa488.png)
The state-value function gives the long-term value of state s
![\begin{align*}
q_\pi(s,a) \doteq \mathbb{E}_\pi[G_t|S_t = s, A_t = a]
\end{align*}](media/4d6ed2657a.png)
We can define the (state-)action value function
How are they connected?
Value Functions
![\begin{align*}
v_\pi(s) \doteq \mathbb{E}_\pi[G_t|S_t = s]
\end{align*}](media/db589aa488.png)
![\begin{align*}
q_\pi(s,a) \doteq \mathbb{E}_\pi[G_t|S_t = s, A_t = a]
\end{align*}](media/4d6ed2657a.png)
We can define the (state-)action value function
![\begin{align*}
v_\pi(s) &\doteq \mathbb{E}_\pi[G_t|S_t = s] \\ &= \sum_{a}\pi(a|s)\mathbb{E}_\pi[G_t|S_t = s, A_t = a] \\ &= \sum_{a}\pi(a|s)q_\pi(s,a)
\end{align*}](media/0b80d83283.png)
Note that:
![\begin{align*}
v_\pi(s) &\doteq \mathbb{E}_\pi[G_t|S_t = s] \\ &= \sum_{a}\pi(a|s)\mathbb{E}_\pi[G_t|S_t = s, A_t = a] \\ &= \sum_{a}\pi(a|s)q_\pi(s,a)
\end{align*}](media/0b80d83283.png)
![\begin{align*}
v_\pi(s) &\doteq \mathbb{E}_\pi[G_t|S_t = s] \\ &= \sum_{a}\pi(a|s)\mathbb{E}_\pi[G_t|S_t = s, A_t = a] \\ &= \sum_{a}\pi(a|s)q_\pi(s,a)
\end{align*}](media/0b80d83283.png)
![\begin{align*}
= \mathbb{E}[q_\pi(S_t, A_t)|S_t = s, A_t \sim \pi]
\end{align*}](media/eae383a367.png)
The state-value function gives the long-term value of state s

Backup diagram

Try to prove it at home!

Given an MDP, for any policy , the value functions
obey the following expectations equations:
Bellman Expectation Equations
Theorem




Optimal action-value function
is the maximum action-value function over all policies
Optimal state-value function
is the maximum state-value function over all policies
Optimal Value Function
Definition (Optimal value functions)
The optimal value function specifies the best possible performance in the MDP
An MDP is “solved” when we know the optimal value function
Optimal Policy
Define a partial ordering over policies

Theorem (Optimal Policies)
For any Markov decision process
There exist an optimal policy that is better than or equal to all other policies

(There can be more than one such optimal policy.)
All optimal policies achieve the optimal state-value function

All optimal policies achieve the optimal action-value function

Finding an Optimal Policy
An optimal policy can be found by maximizing (greedy) over

➢There is always a deterministic optimal policy for any MDP
➢If we know , we immediately have the optimal policy
➢There can be multiple optimal policies
➢If multiple actions maximize , we can just pick any of these (including stochastically)
Bellman Optimality equations

![\begin{align*}
q_*(s,a) = \mathbb{E}[R_{t+1} + \gamma v_*(S_{t+1})|S_t = s, A_t = a]
\end{align*}](media/791f748c24.png)

Bellman Optimality equations

![\begin{align*}
v_*(s) &= \max_{a\in\mathcal{A}(s)}\, q_{\pi_*}(s,a) \\ &= \max_{a}\mathbb{E}_{\pi_*}[G_t|S_t = s, A_t = a] \\ &= \max_{a}\mathbb{E}_{\pi_*}[R_{t+1} + \gamma G_{t+1}|S_t = s, A_t = a] \\ &= \max_{a}\mathbb{E}_{\pi_*}[R_{t+1} + \gamma v_*(S_{t+1})|S_t = s, A_t = a]
\end{align*}](media/b9652183f3.png)

![\begin{align*}
v_*(s) &= \max_{a\in\mathcal{A}(s)}\, q_{\pi_*}(s,a) \\ &= \max_{a}\mathbb{E}_{\pi_*}[G_t|S_t = s, A_t = a] \\ &= \max_{a}\mathbb{E}_{\pi_*}[R_{t+1} + \gamma G_{t+1}|S_t = s, A_t = a] \\ &= \max_{a}\mathbb{E}_{\pi_*}[R_{t+1} + \gamma v_*(S_{t+1})|S_t = s, A_t = a]
\end{align*}](media/b9652183f3.png)
![\begin{align*}
v_*(s) &= \max_{a\in\mathcal{A}(s)}\, q_{\pi_*}(s,a) \\ &= \max_{a}\mathbb{E}_{\pi_*}[G_t|S_t = s, A_t = a] \\ &= \max_{a}\mathbb{E}_{\pi_*}[R_{t+1} + \gamma G_{t+1}|S_t = s, A_t = a] \\ &= \max_{a}\mathbb{E}_{\pi_*}[R_{t+1} + \gamma v_*(S_{t+1})|S_t = s, A_t = a]
\end{align*}](media/b9652183f3.png)
Bellman Optimality equations
![\begin{align*}
q_*(s,a) &= \mathbb{E}[R_{t+1} + \gamma v_*(S_{t+1})|S_t = s, A_t = a] \\ &= \mathbb{E}[R_{t+1} + \gamma\max_{a'}q_*(S_{t+1},a')|S_t = s, A_t = a] \\ &= \sum_{s',r}p(s',r|s,a)\Big[r + \gamma\max_{a'}q_*(s',a')\Big]
\end{align*}](media/469f6c1027.png)


![\begin{align*}
q_*(s,a) &= \mathbb{E}[R_{t+1} + \gamma v_*(S_{t+1})|S_t = s, A_t = a] \\ &= \mathbb{E}[R_{t+1} + \gamma\max_{a'}q_*(S_{t+1},a')|S_t = s, A_t = a] \\ &= \sum_{s',r}p(s',r|s,a)\Big[r + \gamma\max_{a'}q_*(s',a')\Big]
\end{align*}](media/469f6c1027.png)
Bellman Equations
(Reminder) Greedy on = Optimal Policy
An optimal policy can be found
by maximizing (greedy) over

Applying the Bellman equation:



Given an MDP, the optimal value functions obey the
following expectations equations:
Bellman Optimality Equations
Theorem



There can be no policy with a higher value than

Problems in RL
Estimating or
Policy evaluation / Prediction
Given a policy, what is my expected return under that behavior?
Given this treatment protocol/trading strategy, what is my expected return?
Estimating or
Control, because can be used for policy optimization
What is the optimal way of behaving? What is the optimal value function?
What is the optimal treatment? What is the optimal control policy to minimize time, fuel consumption, etc?
Exercise
Consider the following MDP:
➢ for all transitions that end up in , and for all other transitions.
What is ?

➢Actions have a 0.9 probability of success and with 0.1 probability we remain in the same state.
➢Consider a discount of .



➢ for all transitions that end up in , and for all other transitions.
What is ?
➢Actions have a 0.9 probability of success and with 0.1 probability we remain in the same state.

![\begin{align*}
r_0^\pi =\pi\left(a_1 \mid s_0\right)\left[p\left(s_2,-1 \mid s_0, a_1\right)(-1)+p\left(s_1,r \mid s_0, a_1\right)r+p\left(s_0, 0 \mid s_0, a_1\right) \cdot 0\right]
\end{align*}](media/89013de1ae.png)

![\begin{align*}
P_0^\pi \mathbf{V}&= \pi\left(a_1 \mid s_0\right)\left[p\left(s_0 \mid s_0 , a_1\right) v_\pi\left(s_0\right)+p\left(s_1 \mid s_0 , a_1\right) v_\pi\left(s_1\right)+p\left(s_2 \mid s_0, a_1\right) v_\pi\left(s_2\right)\right] \\
&=\begin{bmatrix}
\pi\left(a_1 \mid s_0\right) p\left(s_0 \mid s_0 , a_1\right) & \pi\left(a_1 \mid s_0\right) p\left(s_1 \mid s_0 , a_1\right) & \pi\left(a_1 \mid s_0\right) p\left(s_2 \mid s_0 , a_1\right)
\end{bmatrix}
\mathbf{V}
\end{align*}](media/c97c9ab2d7.png)
![\begin{align*}
P_0^\pi \mathbf{V}&= \pi\left(a_1 \mid s_0\right)\left[p\left(s_0 \mid s_0 , a_1\right) v_\pi\left(s_0\right)+p\left(s_1 \mid s_0 , a_1\right) v_\pi\left(s_1\right)+p\left(s_2 \mid s_0, a_1\right) v_\pi\left(s_2\right)\right] \\
&=\begin{bmatrix}
\pi\left(a_1 \mid s_0\right) p\left(s_0 \mid s_0 , a_1\right) & \pi\left(a_1 \mid s_0\right) p\left(s_1 \mid s_0 , a_1\right) & \pi\left(a_1 \mid s_0\right) p\left(s_2 \mid s_0 , a_1\right)
\end{bmatrix}
\mathbf{V}
\end{align*}](media/c97c9ab2d7.png)
Bellman Equation in Matrix Form
The Bellman value equation, for a given , can be expressed using matrices,


![\begin{align*}
r_i^\pi = \mathbb{E}[R_{t+1}|S_t = s_i, A_t \sim \pi]
\end{align*}](media/313cd55724.png)


Bellman Equation in Matrix Form
The Bellman value equation, for a given , can be expressed using matrices,
![\[ \mathbf{v} = \mathbf{r}^\pi + \gamma\mathbf{P}^\pi\mathbf{v} \]](media/8a0cc13653.png)
This is a linear equation that can be solved directly:

Computational complexity is

only possible for small problems
Iterative methods for larger problems
➢Dynamic programming
➢Monte-Carlo evaluation
➢Temporal-Difference learning
Solving the Bellman Optimality Equation
➢The Bellman optimality equation is non-linear
➢Cannot use the same direct matrix solution as for policy optimization (in general)
➢Many iterative solution methods:
➢Using models / dynamic programming
➢Value iteration
➢Policy iteration
➢Using samples
➢Monte Carlo
➢Q-learning
➢SARSA
Take home message
•Summary
•RL algorithms can be applied to environments that can be described as an MDP
•The goal of the RL agent is to learn a policy that maximizes the expected return
•Policies can be evaluated using value functions
•Bellman equations are recursive formulas for the value functions
•RL algorithms are heavily based on Bellman equations
Take home message
•Further studying:
•Chapter 3 – Reinforcement Learning: An introduction
•Short notes :
•Video:
https://www.youtube.com/watch?v=zSOMeug_i_M&list=PLqYmG7hTraZDVH599EItlEWsUOsJbAod
m&index=4&t=1s
•Next lecture:
•Dynamic Programming