

Reinforcement Learning · Lecture 3
Dynamic Programming

Rafael Fernandes Cunha
Today’s topics
1.Recap on MDP and Bellman Equations
2.Policy Evaluation
3.Policy Improvement
4.Policy Iteration
5.Value Iteration
6.Generalized Policy Iteration
2
1.Discuss the Policy Iteration algorithm.
2.Explain the foundations of the policy improvement
theorem.
3.Compare the difference between the policy iteration and
value iteration methods.
4.Explain why the value iteration method converges.
Learning outcomes
Recap on MDP and Bellman Equations

Algorithm
MDP
Reward
Returns
Returns from each state
State-Value function
Bellman Equations
Possible to
compare
Policies
Returns from each
state, given an action
Action-value function
Recursive
property of value
functions
Used the
Markov
Property!


Policy

A Markov Decision Process (MDP) is a tuple
Markov Decision Process
Set of all
possible states
Set of all
possible actions

Expected reward, achieved on a
transition starting in (s, a)
![\begin{align*}
r(s,a) \doteq \mathbb{E}[R_t|S_{t-1} = s, A_{t-1}=a]
\end{align*}](media/d5e62aeddc.png)

Discount factor
![\begin{align*}
\gamma \in [0,1]
\end{align*}](media/d0161b2d2c.png)
Probability of transitioning to , given a state and action .
Dynamics of the system.
Transition Function

![\begin{align*}
p: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow [0, 1]
\end{align*}](media/186a8ef309.png)
A Markov Decision Process (MDP) is a tuple
Markov Decision Process
Markov Property
Transition Function
Reward Function



![\begin{align*}
\mathbb{E}[R_t|S_{t-1}=s,A_{t-1}=a] = \mathbb{E}[R_t|h_{t-1}, S_{t-1}=s,A_{t-1}=a]
\end{align*}](media/871a53ea65.png)
Identities
![\begin{align*}
\mathbb{E}[R_t|S_{t-1}=s,A_{t-1}=a] = \sum_r r \sum_{s'} p(s', r|s, a)
\end{align*}](media/0048247d81.png)

Agent
Policy

Return


Value Functions
![\begin{align*}
v_{\pi}(s) \doteq \mathbb{E}_{\pi}[G_t | S_t = s]
\end{align*}](media/a1edbcb34b.png)
![\begin{align*}
q_{\pi}(s, a) \doteq \mathbb{E}_{\pi}[G_t | S_t = s, A_t = a]
\end{align*}](media/0d4081f136.png)

Optimal Value Functions


Value functions can be used to define a partial ordering over policies!

Optimal Policy
Define a partial ordering over policies

Theorem (Optimal Policies)
For any Markov decision process
There exist an optimal policy that is better than or equal to all other policies

(There can be more than one such optimal policy.)
All optimal policies achieve the optimal state-value function

All optimal policies achieve the optimal action-value function

Bellman Equations
Expectation



Bellman Equations
Expectation
Optimality








Solving the Bellman Optimality Equation
➢Many iterative solution methods:
➢Using models / dynamic programming
➢Value iteration
➢Policy iteration
➢Using samples
➢Monte Carlo
➢Q-learning
➢SARSA
Today’s Lecture
Dynamic programing
Dynamic programming refers to a collection of algorithms that can be used
to compute optimal policies given a perfect model of the environment as a
Markov decision process (MDP).
Sutton & Barto 2018
Methods consist of two important parts:
Policy evaluation
Policy improvement
Policy evaluation: Prediction problem


Does this algorithm always
converge?
Yes, under appropriate
conditions. More about this
later.
Turn this equality into an update rule


Bellman expectation equation
Example: 4x4 gridworld

What is the MDP in this example?
Suppose the agent follows the
equiprobable random policy.
Let’s use the policy evaluation
algorithm.
Which information is still missing?


Example: 4x4 gridworld
Suppose the agent follows the
equiprobable random policy.



for the random policy
Example: 4x4 gridworld
Equiprobable
random policy.




Example: 4x4 gridworld
Equiprobable
random policy.




Example: 4x4 gridworld
Equiprobable
random policy.




Example: 4x4 gridworld

Example: 4x4 gridworld

Policy Improvement

We can improve our policy by acting greedy with respect to the evaluation that we have.
Being greedy with respect to the values of the random policy, in the previous example, gives us a better policy.

![\begin{align*}
\pi'(s) &\doteq \operatorname*{argmax}_a q_\pi(s, a) \\ &= \operatorname*{argmax}_a \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_t = s, A_t = a] \\ &= \operatorname*{argmax}_a \sum_{s', r} p(s', r | s, a) \Big[ r + \gamma v_\pi(s') \Big],
\end{align*}](media/f301556309.png)
![\begin{align*}
v_{\pi'}(s) &= \max_a \mathbb{E}[R_{t+1} + \gamma v_{\pi'}(S_{t+1}) \mid S_t = s, A_t = a] \\ &= \max_a \sum_{s', r} p(s', r | s, a) \Big[ r + \gamma v_{\pi'}(s') \Big].
\end{align*}](media/8f3abddbe2.png)
Policy Improvement



What can one say
about these policies?

If Deterministic Policy


Policy Improvement


What can one say
about these policies?


Policy Improvement


What can one say
about these policies?





Choose a different
action only on state s

Same as the previous policy, differing
only at state s.




Policy Improvement


What can one say
about these policies?


Choose a different
action only on state s

Same as the previous policy, differing
only at state s.



Policy Improvement: Comparing Policies





Different action chosen at leads to a higher expected return at state !
Policy Improvement


What can one say
about these policies?





What can one say
about these policies?

Same as the previous policy, differing
only at state s.
Policy Improvement




Different Deterministic policies
Same policies


Policy Improvement



Different Deterministic policies




Same as the previous policy, differing
only at state s.
Policy Improvement Theorem



Let and ’ be any pair of deterministic policies such that,
Then the policy must be as good as, or better than .
That is, it must obtain grater or equal expected return from all states





![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
Deterministic policy
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
Theorem Hypothesis
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
Selecting in s, and thereafter
following the existing policy




![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
Deterministic policy
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
Theorem Hypothesis
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
Selecting in s, and thereafter
following the existing policy
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)




![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
Deterministic policy
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
Theorem Hypothesis
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
Selecting in s, and thereafter
following the existing policy
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
Theorem Hypothesis is for
all states

![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)




![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
Selecting in s’, and
thereafter following the existing
policy
Selecting in s, and thereafter
following the existing policy




![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)




![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)




![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)
![\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}](media/148268a0d9.png)

Policy Iteration
Policy Evaluation
Policy Improvement
Sequence of monotonically improving policies and value functions.
A finite MDP has only a finite
number of deterministic policies
Process must converge to an optimal policy and the optimal
value function in a finite number of iterations.
Each policy is guaranteed to be a strict improvement over the previous one (unless it is already optimal).
Policy Iteration

Policy Iteration
Must policy evaluation converge to before applying policy improvement?
Can we stop before that?
➢Threshold on the change of the values.
➢Stopping at some k iterations of iterative policy evaluation.
➢In the small gridworld, k=3 was sufficient to achieve optimal policy.
Extreme: Update policy every
iteration, that is, use .
Value Iteration
Value Iteration


Bellman optimality equation
Bellman expectation equation
Policy Evaluation
Value Iteration

Turn into an
update rule

Turn into an
update rule
Equivalent to policy iteration, with step of policy
evaluation between each two (greedy) policy
improvement steps.
Value Iteration

Value Iteration



Value Iteration


Example: Shortest Path

Generalized Policy Iteration (GPI)



Alternating with some chosen iterations.
If both process converge,
the value function and
policy must be optimal.
Almost all RL methods are
well described as GPI
All have identifiable policies
and value functions
Policy being improved with
respect to the value function
Value function being driven toward
the value function for the policy.
Why Value Iteration converges?

Generating a sequence , , , …, using the Bellman optimality equation will always converge
to the optimal value function when ?
We need to check if “approaches” .
We are talking about distances between two functions.
How to measure distance between functions?
Some recap on functional analysis
First, we need to understand the concepts of:
➢ Operator
➢Fixed Point
➢-contraction mapping
Some recap on functional analysis
First, we need to understand the concepts of:
➢ Operator
➢Fixed Point
➢-contraction mapping
Policy Evaluation
Bellman Operators for the state-value function
Deterministic policy
Transitions are deterministic
Simplification
(abuse of notation)
Operator notation
Bellman Operators for the state-value function
Deterministic policy
Transitions are deterministic
Simplification
(abuse of notation)
Operator notation
First application of the operator
Second application of the operator
Bellman Operators for the state-value function

Bellman Expectation equation



Bellman Expectation Operator (Point-wise)
Mapping

Bellman Optimality equation



Mapping
Bellman Optimality Operator (Point-wise)
Space of bounded real-valued
functions over the state space.
Bellman Operators for the state-value function
As a common convention, we drop the index




Bellman Expectation equation



Bellman Expectation Operator (Point-wise)
Mapping

Bellman Optimality equation



Mapping
Bellman Optimality Operator (Point-wise)
Some recap on functional analysis
First, we need to understand the concepts of:
➢ Operator
➢Fixed Point
➢-contraction mapping
Fixed Point: Example
Using operator notation
Initial function
Third iteration
General iteration
Fixed point
First iteration
Second iteration
Fixed Point
Applying the operator leaves you at the same point!
Fixed point of the operator


Where be a vector space, equipped with a norm
Some recap on functional analysis
First, we need to understand the concepts of:
➢ Operator
➢Fixed Point
➢-contraction mapping
-contraction
An mapping is a -contraction mapping if for any , s.t.

Let be a vector space, equipped with a norm
Example
-contraction
An mapping is a -contraction mapping if for any , s.t.

Let be a vector space, equipped with a norm
Every contraction is also continuous. This means that


If the sequence converges in norm to x, the operator over the sequence converges in norm to the
operator over x.
Banach Fixed Point Theorem
has a unique fixed point , such that
Idependent of the starting point, the sequence converges
to in a geometric fashion:
Let be a vector space, equipped with a norm
Let be a -contraction mapping


Banach Fixed Point Theorem
Banach Fixed Point Theorem
has a unique fixed point , such that
Idependent of the starting point, the sequence converges
to in a geometric fashion:
Let be a vector space, equipped with a norm
Let be a -contraction mapping
Banach Fixed Point Theorem
How to prove that value
iteration converges using the
Banach Fixed Point Theorem?
Show that the Bellman operator is
a -contraction mapping, has a
fixed point, and is monotonic!
Properties of the Bellman Operator T*



Mapping
Bellman Optimality Operator (Point-wise)
1) has one unique fixed point v*

2) is a -contraction wrt. to

3) is monotonic


component-wise
Proposition: T* is a -contraction wrt.




Proposition: T* is a -contraction wrt.





Thus we get:

Proposition: T* is monotonic (Try at home)




Thus

Value Iteration through the lens of the Bellman Operator


Proof: Direct application of the Banach Fixed Point Theorem.

Update Rule
Value Iteration through the lens of the Bellman Operator

Proof: Direct application of the Banach Fixed Point Theorem.





Fixed Point property
Contraction property
Iterative Application

Update rule
Synchronous Dynamic Programming Algorithms
Problem | Bellman Equation | Algorithm |
Prediction | Bellman Expectation Equation | Iterative Policy Evaluation |
Synchronous Dynamic Programming Algorithms
Problem | Bellman Equation | Algorithm |
Prediction | Bellman Expectation Equation | Iterative Policy Evaluation |
Control | Bellman Expectation Equation + (Greedy) Policy Improvement | Policy Iteration |
Synchronous Dynamic Programming Algorithms
Problem | Bellman Equation | Algorithm |
Prediction | Bellman Expectation Equation | Iterative Policy Evaluation |
Control | Bellman Expectation Equation + (Greedy) Policy Improvement | Policy Iteration |
Control | Bellman Optimality Equation | Value Iteration |
Algorithms are based on:


State-value function
Action-value function


Complexity per iteration
Complexity per iteration
Asynchronous Dynamic Programming
DP methods described so far used synchronous updates (all states in parallel)
Asynchronous DP
➢backs up states individually, in any order
➢can significantly reduce computation
➢guaranteed to converge if all states continue to be selected
Asynchronous Dynamic Programming
Three simple ideas for asynchronous dynamic programming:
➢ In-place dynamic programming
➢ Prioritized sweeping
➢ Real-time dynamic programming
In-Place Dynamic Programming
Synchronous value iteration stores two copies of the value function


For all in :

In-place value iteration only stores one copy of the value function
For all in :
Prioritized Sweeping
➢Backup the state with the largest Bellman error
➢After each backup, update errors of affected predecessor states
➢Requires knowledge of reverse dynamics (predecessors)
➢Efficient via priority queue maintenance
Compute Bellman errors and store in a priority queue.

📌 Focus updates where they matter most → faster convergence
Real-Time Dynamic Programming
Idea: only update states that are relevant to the agent
Agent is in state
Update state value, or states that it
expects to be in soon.
Full-Width Backups

➢Standard DP uses full-width backups
➢For each backup (sync or async)
➢Every successor state and action is considered
➢Using true model of transitions and reward function
➢DP is effective for medium-sized problems (millions of states)
➢For large problems DP suffers from curse of dimensionality
➢Number of states grows exponentially with number of state variables
➢Even one full backup can be too expensive
Sample Backups

➢In subsequent lectures we will consider sample backups
➢Using sample rewards and sample transitions (s, a, r, s’)
(Instead of reward function r and transition dynamics p)
➢Advantages
➢Model-free: no prior knowledge of MDP required
➢Breaks the curse of dimensionality through sampling
➢Cost of backup is constant, independent of
Take home message
•The Bellman equations serves as the base to iterative algorithms.
•Algorithms are based on Policy Evaluation and Policy Improvement.
•Policy Iteration must converge to an optimal policy and the optimal value function
in a finite number of iterations.
•Value Iteration is a Policy iteration switching after each step.
•The Bellman operator has properties that assures the convergence of value
iteration.
•DP can be synchronous or asynchronous.
•DP requires the knowledge of the reward function r and transition function p.
•DP is computationally expensive.
Take home message
•Further studying:
•Chapter 4 – Reinforcement Learning: An introduction
•Short notes: https://people.stfx.ca/jdelamer/courses/csci-531/lectures/mdp/dynamic-
programming/dynamic-programming.html
•Video (UCL x DeepMind)
•Lectures 3 (after min 50) :
https://www.youtube.com/watch?v=zSOMeug_i_M&list=PLqYmG7hTraZCRwoyGxvQkqVrZgDQi4m-
5&index=3&t=2917s
•Lecture 4:
https://www.youtube.com/watch?v=XpbLq7rIJAA&list=PLqYmG7hTraZCRwoyGxvQkqVrZgDQi4m
-5&index=4
•Next lecture:
•Monte Carlo Methods and TD-learning