Reinforcement Learning · Lecture 3

Dynamic Programming

Rafael Fernandes Cunha

rafaelcunha2013.github.io

Today’s topics

1.Recap on MDP and Bellman Equations

2.Policy Evaluation

3.Policy Improvement

4.Policy Iteration

5.Value Iteration

6.Generalized Policy Iteration

2

1.Discuss the Policy Iteration algorithm.

2.Explain the foundations of the policy improvement
theorem.

3.Compare the difference between the policy iteration and
value iteration methods.

4.Explain why the value iteration method converges.

Learning outcomes

Recap on MDP and Bellman Equations

Algorithm

MDP

Reward

Returns

Returns from each state

State-Value function

Bellman Equations

Possible to
compare
Policies

Returns from each
state, given an action

Action-value function

Recursive
property of value
functions

Used the
Markov
Property!

\begin{align*}
 ( \mathcal{S}, \mathcal{A}, p, r, \gamma )
\end{align*}
\begin{align*}
Pr\{A_t=a|S_t = s\}
\end{align*}

Policy

\begin{align*}
\pi(a|s)
\end{align*}

A Markov Decision Process (MDP) is a tuple

Markov Decision Process

Set of all
possible states

Set of all
possible actions

\begin{align*}
 ( \mathcal{S}, \mathcal{A}, p, r, \gamma )
\end{align*}

Expected reward, achieved on a
transition starting in (s, a)

\begin{align*}
r(s,a) \doteq \mathbb{E}[R_t|S_{t-1} = s, A_{t-1}=a]
\end{align*}
\begin{align*}
r: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}
\end{align*}

Discount factor

\begin{align*}
\gamma \in [0,1] 
\end{align*}

Probability of transitioning to , given a state and action .

Dynamics of the system.

Transition Function

\begin{align*}
  p(s'|s,a) \doteq Pr\{S_t = s'| S_{t-1} = s, A_{t-1}=a\}
\end{align*}
\begin{align*}
  p: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow [0, 1]
\end{align*}

A Markov Decision Process (MDP) is a tuple

Markov Decision Process

Markov Property

Transition Function

Reward Function

\begin{align*}
 ( \mathcal{S}, \mathcal{A}, p, r, \gamma )
\end{align*}
\begin{align*}
h_{t-1} = \{S_1, \ldots, S_{t-1}, A_1, \ldots, A_{t-1}, R_1, \ldots, R_{t-1} \}
\end{align*}
\begin{align*}
p(S_{t+1} = s' | S_t = s) = p(S_{t+1} = s' | h_{t-1}, S_t=s)
\end{align*}
\begin{align*}
\mathbb{E}[R_t|S_{t-1}=s,A_{t-1}=a] = \mathbb{E}[R_t|h_{t-1}, S_{t-1}=s,A_{t-1}=a]
\end{align*}

Identities

\begin{align*}
\mathbb{E}[R_t|S_{t-1}=s,A_{t-1}=a] = \sum_r r \sum_{s'} p(s', r|s, a)
\end{align*}
\begin{align*}
p(s' |s,a) = \sum_r p(s', r| s,a)
\end{align*}

Agent

Policy

\begin{align*}
\pi(a|s) \doteq Pr\{A_t = a|S_t = s\}
\end{align*}

Return

\begin{align*}
G_t   \doteq \sum_{k=0}^{T-t-1} \gamma^k R_{t+k+1}
\end{align*}
\begin{align*}
G_t =R_{t+1} + \gamma G_{t+1}
\end{align*}

Value Functions

\begin{align*}
v_{\pi}(s) \doteq \mathbb{E}_{\pi}[G_t | S_t = s] 
\end{align*}
\begin{align*}
q_{\pi}(s, a) \doteq \mathbb{E}_{\pi}[G_t | S_t = s, A_t = a]
\end{align*}
\begin{align*}
v_{\pi}(s)  = \sum_a \pi(a|s) q_{\pi}(s, a)
\end{align*}

Optimal Value Functions

\begin{align*}
v_*(s) \doteq \max_{\pi} v_{\pi}(s)
\end{align*}
\begin{align*}
q_*(s, a) \doteq \max_{\pi} q_{\pi}(s, a)
\end{align*}

Value functions can be used to define a partial ordering over policies!

\begin{align*}
\pi \geq \pi'\iff v_{\pi}(s) \geq v_{\pi'}(s) \textrm{ , for all } s \in \mathcal{S}
\end{align*}

Optimal Policy

Define a partial ordering over policies

\begin{align*}
\pi \geq \pi' \iff v_\pi(s) \geq v_{\pi'}(s) \ , \text{ for all } s \in \mathcal{S}
\end{align*}

Theorem (Optimal Policies)

For any Markov decision process

There exist an optimal policy that is better than or equal to all other policies

\begin{align*}
\pi' \geq \pi, \forall \pi
\end{align*}

(There can be more than one such optimal policy.)

All optimal policies achieve the optimal state-value function

\begin{align*}
v_{\pi_*}(s) = v_*(s)
\end{align*}

All optimal policies achieve the optimal action-value function

\begin{align*}
q_{\pi_*}(s, a) = q_*(s, a)
\end{align*}

Bellman Equations

Expectation

\begin{align*}
v_\pi(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_\pi(s')\Big)
\end{align*}

Bellman Equations

Expectation

Optimality

\begin{align*}
v_\pi(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_\pi(s')\Big)
\end{align*}
\begin{align*}
q_\pi(s, a) = \sum_{s', r} p(s', r | s, a) \Big( r + \gamma \sum_{a'} \pi(a' | s') q_\pi(s', a') \Big)
\end{align*}
\begin{align*}
v_*(s) = \max_a \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_*(s')\Big)
\end{align*}
\begin{align*}
q_*(s, a) = \sum_{s', r} p(s', r | s, a) \Big( r + \gamma \max_{a'} q_*(s', a') \Big)
\end{align*}

Solving the Bellman Optimality Equation

​

​

➢Many iterative solution methods:

​

➢Using models / dynamic programming

➢Value iteration

➢Policy iteration

​

➢Using samples

➢Monte Carlo

➢Q-learning

➢SARSA

Today’s Lecture

Dynamic programing

Dynamic programming refers to a collection of algorithms that can be used

to compute optimal policies given a perfect model of the environment as a

Markov decision process (MDP).

Sutton & Barto 2018

Methods consist of two important parts:

Policy evaluation

Policy improvement

Policy evaluation: Prediction problem

\begin{align*}
v_\pi(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_\pi(s')\Big)
\end{align*}

Does this algorithm always
converge?

Yes, under appropriate
conditions. More about this
later.

Turn this equality into an update rule

\begin{align*}
v_{k+1}(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_k(s')\Big)
\end{align*}
\begin{align*}
v_0, v_1, v_2, \ldots
\end{align*}

Bellman expectation equation

Example: 4x4 gridworld

What is the MDP in this example?

Suppose the agent follows the
equiprobable random
policy.

Let’s use the policy evaluation
algorithm.

Which information is still missing?

Example: 4x4 gridworld

Suppose the agent follows the
equiprobable random policy.

for the random policy

Example: 4x4 gridworld

Equiprobable
random policy.

\begin{align*}
v_{k+1}(s) = \sum_{a} \pi(a | s) \sum_{s', r} p(s', r | s, a) \Big( r + \gamma v_k(s') \Big)
\end{align*}
\begin{align*}
    v_{3}(s_2) &= \sum_a \pi(a|s_2) \sum_{s',r}p(s',r|s_2,a)\Big(r + \gamma  v_{2}(s')\Big) \\
&= \pi(a=\uparrow|s_2) \sum_{s',r}p(s',r|s_2,a=\uparrow) \Big(r + \gamma  v_{2}(s')\Big)  \\
&+ \pi(a=\rightarrow|s_2) \sum_{s',r}p(s',r|s_2,a=\rightarrow) \Big(r + \gamma  v_{2}(s')\Big)  \\
&+ \pi(a=\downarrow|s_2) \sum_{s',r}p(s',r|s_2,a=\downarrow) \Big(r + \gamma  v_{2}(s')\Big)  \\
&+ \pi(a=\leftarrow|s_2) \sum_{s',r}p(s',r|s_2,a=\leftarrow) \Big(r + \gamma  v_{2}(s')\Big)  
\end{align*}
\begin{align*}
    v_{3}(s_2) &= \sum_a \pi(a|s_2) \sum_{s',r}p(s',r|s_2,a)\Big(r + \gamma  v_{2}(s')\Big) \\
&= \pi(a=\uparrow|s_2) \sum_{s',r}p(s',r|s_2,a=\uparrow) \Big(r + \gamma  v_{2}(s')\Big)  \\
&+ \pi(a=\rightarrow|s_2) \sum_{s',r}p(s',r|s_2,a=\rightarrow) \Big(r + \gamma  v_{2}(s')\Big)  \\
&+ \pi(a=\downarrow|s_2) \sum_{s',r}p(s',r|s_2,a=\downarrow) \Big(r + \gamma  v_{2}(s')\Big)  \\
&+ \pi(a=\leftarrow|s_2) \sum_{s',r}p(s',r|s_2,a=\leftarrow) \Big(r + \gamma  v_{2}(s')\Big)  
\end{align*}

Example: 4x4 gridworld

Equiprobable
random policy.

\begin{align*}
v_{k+1}(s) = \sum_{a} \pi(a | s) \sum_{s', r} p(s', r | s, a) \Big( r + \gamma v_k(s') \Big)
\end{align*}
\begin{align*}
    v_{3}(s_2) 
&= \pi(a=\uparrow|s_2)  \Big(r + \gamma  v_{2}(s_2)\Big)  \\
&+ \pi(a=\rightarrow|s_2) \Big(r + \gamma  v_{2}(s_3)\Big)  \\
&+ \pi(a=\downarrow|s_2) \Big(r + \gamma  v_{2}(s_6)\Big)  \\
&+ \pi(a=\leftarrow|s_2)  \Big(r + \gamma  v_{2}(s_1)\Big)  
\end{align*}

Example: 4x4 gridworld

Equiprobable
random policy.

\begin{align*}
v_{k+1}(s) = \sum_{a} \pi(a | s) \sum_{s', r} p(s', r | s, a) \Big( r + \gamma v_k(s') \Big)
\end{align*}
\begin{align*}
    v_{3}(s_2) 
&= 0.25  \Big(-1 +   (-2.0)\Big)  \\
&+ 0.25   \Big(-1 +   (-2.0)\Big)  \\
&+ 0.25   \Big(-1 +   (-2.0)\Big)  \\
&+ 0.25   \Big(-1 +  (-1.7)\Big)  
\end{align*}
\begin{align*}
    v_3(s_2) = -2.9 
\end{align*}

Example: 4x4 gridworld

Example: 4x4 gridworld

Policy Improvement

We can improve our policy by acting greedy with respect to the evaluation that we have.

Being greedy with respect to the values of the random policy, in the previous example, gives us a better policy.

\begin{align*}
\pi'(s) &\doteq \operatorname*{argmax}_a q_\pi(s, a) \\ &= \operatorname*{argmax}_a \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_t = s, A_t = a] \\ &= \operatorname*{argmax}_a \sum_{s', r} p(s', r | s, a) \Big[ r + \gamma v_\pi(s') \Big],
\end{align*}
\begin{align*}
v_{\pi'}(s) &= \max_a \mathbb{E}[R_{t+1} + \gamma v_{\pi'}(S_{t+1}) \mid S_t = s, A_t = a] \\ &= \max_a \sum_{s', r} p(s', r | s, a) \Big[ r + \gamma v_{\pi'}(s') \Big].
\end{align*}

Policy Improvement

\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_\pi(s) = \sum_{a} \pi(a | s) q_\pi(s, a)
\end{align*}
\begin{align*}
v_{\pi_{\text{new}}}(s) \geq v_\pi(s)
\end{align*}

What can one say
about these policies?

\begin{align*}
\pi_{\text{new}} \geq \pi
\end{align*}

If Deterministic Policy

\begin{align*}
a = \pi(s)
\end{align*}
\begin{align*}
v_\pi(s) = q_\pi(s, \pi(s))
\end{align*}

Policy Improvement

\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_{\pi_{\text{new}}}(s) \geq v_\pi(s)
\end{align*}

What can one say
about these policies?

\begin{align*}
\pi_{\text{new}} \geq \pi
\end{align*}
\begin{align*}
v_\pi(s) = q_\pi(s, \pi(s))
\end{align*}

Policy Improvement

\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_{\pi_{\text{new}}}(s) \geq v_\pi(s)
\end{align*}

What can one say
about these policies?

\begin{align*}
\pi_{\text{new}} \geq \pi
\end{align*}
\begin{align*}
v_\pi(s) = q_\pi(s, \pi(s))
\end{align*}
\begin{align*}
q_\pi(s, a)
\end{align*}
\begin{align*}
a \neq \pi(s)
\end{align*}
\begin{align*}
a = \pi_{\text{new}}(s)
\end{align*}

Choose a different
action only on state s

\begin{align*}
\pi_{\text{new}}(s)
\end{align*}

Same as the previous policy, differing
only at state
s.

\begin{align*}
\pi_{\text{new}}(s') = \begin{cases} \arg\max_{a \in A} q_\pi(s', a) & \text{if } s' = s \\ \pi(s') & \text{if } s' \neq s \end{cases}
\end{align*}
\begin{align*}
q_\pi(s, \pi(s))
\end{align*}
\begin{align*}
\geq
\end{align*}
\begin{align*}
q_\pi(s, \pi_{\text{new}}(s))
\end{align*}

Policy Improvement

\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_{\pi_{\text{new}}}(s) \geq v_\pi(s)
\end{align*}

What can one say
about these policies?

\begin{align*}
\pi_{\text{new}} \geq \pi
\end{align*}
\begin{align*}
v_\pi(s) = q_\pi(s, \pi(s))
\end{align*}

Choose a different
action only on state s

\begin{align*}
\pi_{\text{new}}(s)
\end{align*}

Same as the previous policy, differing
only at state
s.

\begin{align*}
\pi_{\text{new}}(s') = \begin{cases} \arg\max_{a \in A} q_\pi(s', a) & \text{if } s' = s \\ \pi(s') & \text{if } s' \neq s \end{cases}
\end{align*}
\begin{align*}
a \neq \pi(s)
\end{align*}
\begin{align*}
a = \pi_{\text{new}}(s)
\end{align*}

Policy Improvement: Comparing Policies

\begin{align*}
\mathcal{S} = \{s_1, s_2, s_3, s_4\}
\end{align*}
\begin{align*}
v_\pi(s)
\end{align*}
\begin{align*}
s_1 \qquad s_2 \qquad s_3 \qquad s_4
\end{align*}
\begin{align*}
v_{\pi_{\text{new}}}(s)
\end{align*}
\begin{align*}
s_1 \qquad s_2 \qquad s_3 \qquad s_4
\end{align*}

Different action chosen at leads to a higher expected return at state !

Policy Improvement

\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_{\pi_{\text{new}}}(s) \geq v_\pi(s)
\end{align*}

What can one say
about these policies?

\begin{align*}
\pi_{\text{new}} \geq \pi
\end{align*}
\begin{align*}
v_\pi(s) = q_\pi(s, \pi(s))
\end{align*}
\begin{align*}
q_\pi(s, \pi(s))
\end{align*}
\begin{align*}
\geq
\end{align*}
\begin{align*}
q_\pi(s, \pi_{\text{new}}(s))
\end{align*}

What can one say
about these policies?

\begin{align*}
\pi_{\text{new}}(s)
\end{align*}

Same as the previous policy, differing
only at state
s.

Policy Improvement

\begin{align*}
q_\pi(s, \pi(s))
\end{align*}
\begin{align*}
\geq
\end{align*}
\begin{align*}
q_\pi(s, \pi_{\text{new}}(s))
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}

Different Deterministic policies

Same policies

\begin{align*}
v_\pi(s) = q_\pi(s, \pi(s))
\end{align*}
\begin{align*}
q_\pi(s, \pi(s))
\end{align*}

Policy Improvement

\begin{align*}
\geq
\end{align*}
\begin{align*}
q_\pi(s, \pi_{\text{new}}(s))
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}

Different Deterministic policies

\begin{align*}
v_\pi(s) = q_\pi(s, \pi(s))
\end{align*}
\begin{align*}
v_{\pi_{\text{new}}}(s) \geq v_\pi(s)
\end{align*}
\begin{align*}
\pi_{\text{new}} \geq \pi
\end{align*}
\begin{align*}
\pi_{\text{new}}(s)
\end{align*}

Same as the previous policy, differing
only at state
s.

Policy Improvement Theorem

\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_\pi(s) \leq q_\pi(s, \pi'(s))
\end{align*}
\begin{align*}
v_\pi(s) \leq v_{\pi'}(s)
\end{align*}

Let and ’ be any pair of deterministic policies such that,

Then the policy must be as good as, or better than .

That is, it must obtain grater or equal expected return from all states

\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_\pi(s) \leq q_\pi(s, \pi'(s))
\end{align*}
\begin{align*}
v_\pi(s) \leq v_{\pi'}(s)
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}

Deterministic policy

\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}

Theorem Hypothesis

\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}

Selecting in s, and thereafter
following the existing policy

\begin{align*}
v_\pi(s) \leq q_\pi(s, \pi'(s))
\end{align*}
\begin{align*}
v_\pi(s) \leq v_{\pi'}(s)
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}

Deterministic policy

\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}

Theorem Hypothesis

\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}

Selecting in s, and thereafter
following the existing policy

\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) \leq q_\pi(s, \pi'(s))
\end{align*}
\begin{align*}
v_\pi(s) \leq v_{\pi'}(s)
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}

Deterministic policy

\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}

Theorem Hypothesis

\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}

Selecting in s, and thereafter
following the existing policy

\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}

Theorem Hypothesis is for
all states

\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) \leq q_\pi(s, \pi'(s))
\end{align*}
\begin{align*}
v_\pi(s) \leq v_{\pi'}(s)
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}

Selecting in s’, and
thereafter following the existing
policy

Selecting in s, and thereafter
following the existing policy

\begin{align*}
v_\pi(s) \leq q_\pi(s, \pi'(s))
\end{align*}
\begin{align*}
v_\pi(s) \leq v_{\pi'}(s)
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) \leq q_\pi(s, \pi'(s))
\end{align*}
\begin{align*}
v_\pi(s) \leq v_{\pi'}(s)
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) \leq q_\pi(s, \pi'(s))
\end{align*}
\begin{align*}
v_\pi(s) \leq v_{\pi'}(s)
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
v_\pi(s) &= \sum_{a}\pi(a|s)q_\pi(s,a) \\ &= q_\pi(s,\pi(s)) \\ &\leq q_\pi(s,\pi'(s)) \\ &= \mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s, A_t = \pi'(s)] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma q_\pi(S_{t+1},\pi'(S_{t+1}))|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma\mathbb{E}[R_{t+2} + \gamma v_\pi(S_{t+2})|S_{t+1}, A_{t+1} = \pi'(S_{t+1})]|S_t = s] \\ &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 v_\pi(S_{t+2})|S_t = s] \\ &\leq \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 v_\pi(S_{t+3})|S_t = s] \\ &\ldots \\[1ex] &= \mathbb{E}_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \cdots|S_t = s] \\ &= v_{\pi'}(s)
\end{align*}
\begin{align*}
\pi_0 \xrightarrow{\;\;\text{E}\;\;} v_{\pi_0} \xrightarrow{\;\;\text{I}\;\;} \pi_1 \xrightarrow{\;\;\text{E}\;\;} v_{\pi_1} \xrightarrow{\;\;\text{I}\;\;} \pi_2 \xrightarrow{\;\;\text{E}\;\;} \cdots \xrightarrow{\;\;\text{I}\;\;} \pi_* \xrightarrow{\;\;\text{E}\;\;} v_*
\end{align*}

Policy Iteration

Policy Evaluation

Policy Improvement

Sequence of monotonically improving policies and value functions.

A finite MDP has only a finite
number of deterministic policies

Process must converge to an optimal policy and the optimal
value function
in a finite number of iterations.

Each policy is guaranteed to be a strict improvement over the previous one (unless it is already optimal).

Policy Iteration

Policy Iteration

Must policy evaluation converge to before applying policy improvement?

Can we stop before that?

➢Threshold on the change of the values.

➢Stopping at some k iterations of iterative policy evaluation.

➢In the small gridworld, k=3 was sufficient to achieve optimal policy.

Extreme: Update policy every
iteration, that is, use
.

Value Iteration

Value Iteration

\begin{align*}
v_\pi(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_\pi(s')\Big)
\end{align*}
\begin{align*}
v_*(s) = \max_a \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_*(s')\Big)
\end{align*}

Bellman optimality equation

Bellman expectation equation

Policy Evaluation

Value Iteration

\begin{align*}
v_{k+1}(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_k(s')\Big)
\end{align*}

Turn into an
update rule

\begin{align*}
v_{k+1}(s) = \max_a \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_k(s')\Big)
\end{align*}

Turn into an
update rule

Equivalent to policy iteration, with step of policy
evaluation
between each two (greedy) policy
improvement steps.

Value Iteration

Value Iteration

Value Iteration

Example: Shortest Path

Generalized Policy Iteration (GPI)

Alternating with some chosen iterations.

If both process converge,
the value function and
policy must be optimal.

Almost all RL methods are
well described as GPI

All have identifiable policies
and value functions

Policy being improved with
respect to the value function

Value function being driven toward
the value function for the policy.

Why Value Iteration converges?

\begin{align*}
v_{k+1}(s) = \max_a \sum_{s', r} p(s', r | s, a) \Big( r + \gamma v_k(s') \Big)
\end{align*}

Generating a sequence , , , …, using the Bellman optimality equation will always converge
to the optimal value function
when ?

We need to check if “approaches” .

We are talking about distances between two functions.

How to measure distance between functions?

Some recap on functional analysis

First, we need to understand the concepts of:

​

➢ Operator

​

➢Fixed Point

​

➢-contraction mapping

​

Some recap on functional analysis

First, we need to understand the concepts of:

​

➢ Operator

​

➢Fixed Point

​

➢-contraction mapping

​

\begin{align*}
    v_{\pi}(s) = \sum_a \pi(a|s) \sum_{s',r} p(s', r|s,a) \Big(r + \gamma v_{\pi}(s')\Big) 
\end{align*}

Policy Evaluation

Bellman Operators for the state-value function

\begin{align*}
    r(s_1, a_1) &= 1 \\
 r(s_2, a_2) &= 0 \\
r(s_3, a_3) &= -1 \\
\end{align*}
\begin{align*}
    \pi(a_1|s_1) = 1 \\
 \pi(a_2|s_2) = 1 \\
 \pi(a_3|s_3) = 1 
\end{align*}
\begin{align*}
    s_1 
\end{align*}
\begin{align*}
    s_2 
\end{align*}
\begin{align*}
    s_3 
\end{align*}
\begin{align*}
    a_1 
\end{align*}
\begin{align*}
    a_2 
\end{align*}
\begin{align*}
    a_3 
\end{align*}

Deterministic policy

Transitions are deterministic

\begin{align*}
    \gamma = 0.9 
\end{align*}
\begin{align*}
    \pi(s_1) = a_1 \\
 \pi(s_2) = a_2 \\
 \pi(s_3) = a_3 
\end{align*}

Simplification
(abuse of notation)

\begin{align*}
    v_{\pi}(s) = r + \gamma v_{\pi}(s') 
\end{align*}
\begin{align*}
    v_{\pi}(s_1) &= 1 + \gamma v_{\pi}(s_2) \\ 
 v_{\pi}(s_2) &= 0 + \gamma v_{\pi}(s_3) \\ 
 v_{\pi}(s_3) &= -1 + \gamma v_{\pi}(s_1)
\end{align*}
\begin{align*}
    v_{k+1}(s) = r + \gamma v_k(s') 
\end{align*}
\begin{align*}
    (T^{\pi}_{\mathcal{V}})f(s) = r + \gamma f(s') 
\end{align*}
\begin{align*}
    v_{k+1}(s) = T^{\pi}v_k(s) 
\end{align*}

Operator notation

Bellman Operators for the state-value function

\begin{align*}
    r(s_1, a_1) &= 1 \\
 r(s_2, a_2) &= 0 \\
r(s_3, a_3) &= -1 \\
\end{align*}
\begin{align*}
    \pi(a_1|s_1) = 1 \\
 \pi(a_2|s_2) = 1 \\
 \pi(a_3|s_3) = 1 
\end{align*}
\begin{align*}
    \pi(s_1) = a_1 \\
 \pi(s_2) = a_2 \\
 \pi(s_3) = a_3 
\end{align*}
\begin{align*}
    s_1 
\end{align*}
\begin{align*}
    s_2 
\end{align*}
\begin{align*}
    s_3 
\end{align*}
\begin{align*}
    a_1 
\end{align*}
\begin{align*}
    a_2 
\end{align*}
\begin{align*}
    a_3 
\end{align*}

Deterministic policy

Transitions are deterministic

\begin{align*}
    \gamma = 0.9 
\end{align*}

Simplification
(abuse of notation)

\begin{align*}
    v_{k+1}(s) = r + \gamma v_k(s') 
\end{align*}
\begin{align*}
    (T^{\pi}_{\mathcal{V}})f(s) = r + \gamma f(s') 
\end{align*}
\begin{align*}
    v_{k+1}(s) = T^{\pi}v_k(s) 
\end{align*}

Operator notation

\begin{align*}
    v_1(s_1) &= T^{\pi}v_0(s_1) = 1 + \gamma v_0(s_2) = 1 \\
  v_1(s_2) &= T^{\pi}v_0(s_2) = 0 + \gamma v_0(s_3) = 0 \\
 v_1(s_3) &= T^{\pi}v_0(s_3) = -1 + \gamma v_0(s_0) = -1 
\end{align*}

First application of the operator

Second application of the operator

\begin{align*}
    v_2(s_1) &= (T^{\pi})^2 (v_0(s_1))= T^{\pi}v_1(s_1) = 1 + \gamma v_1(s_2) = 1 \\
  v_2(s_2) &= (T^{\pi})^2 (v_0(s_2))= T^{\pi}v_1(s_2) = 0 + \gamma v_1(s_3) = -0.9 \\
 v_2(s_3) &= (T^{\pi})^2 (v_0(s_3))= T^{\pi}v_1(s_3) = -1 + \gamma v_1(s_0) = -0.1 
\end{align*}

Bellman Operators for the state-value function

\begin{align*}
v_\pi(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_\pi(s')\Big)
\end{align*}

Bellman Expectation equation

\begin{align*}
(T^\pi_{\mathcal{V}}) f(s) = \sum_{a} \pi(a | s) \sum_{s', r} p(s', r | s, a) \Big( r + \gamma f(s') \Big)
\end{align*}
\begin{align*}
\forall f \in \mathcal{V}
\end{align*}
\begin{align*}
T^\pi_{\mathcal{V}} : \mathcal{V} \to \mathcal{V}
\end{align*}

Bellman Expectation Operator (Point-wise)

Mapping

\begin{align*}
v_*(s) = \max_a \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_*(s')\Big)
\end{align*}

Bellman Optimality equation

\begin{align*}
(T^*_{\mathcal{V}}) f(s) = \max_a \sum_{s', r} p(s', r | s, a) \Big( r + \gamma f(s') \Big)
\end{align*}
\begin{align*}
T^*_{\mathcal{V}} : \mathcal{V} \to \mathcal{V}
\end{align*}
\begin{align*}
\forall f \in \mathcal{V}
\end{align*}

Mapping

Bellman Optimality Operator (Point-wise)

Space of bounded real-valued
functions over the state space.

Bellman Operators for the state-value function

As a common convention, we drop the index

\begin{align*}
\mathcal{V}
\end{align*}
\begin{align*}
T^* = T^*_{\mathcal{V}}
\end{align*}
\begin{align*}
T^\pi = T^\pi_{\mathcal{V}}
\end{align*}
\begin{align*}
v_\pi(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_\pi(s')\Big)
\end{align*}

Bellman Expectation equation

\begin{align*}
(T^\pi_{\mathcal{V}}) f(s) = \sum_{a} \pi(a | s) \sum_{s', r} p(s', r | s, a) \Big( r + \gamma f(s') \Big)
\end{align*}
\begin{align*}
\forall f \in \mathcal{V}
\end{align*}
\begin{align*}
T^\pi_{\mathcal{V}} : \mathcal{V} \to \mathcal{V}
\end{align*}

Bellman Expectation Operator (Point-wise)

Mapping

\begin{align*}
v_*(s) = \max_a \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_*(s')\Big)
\end{align*}

Bellman Optimality equation

\begin{align*}
(T^*_{\mathcal{V}}) f(s) = \max_a \sum_{s', r} p(s', r | s, a) \Big( r + \gamma f(s') \Big)
\end{align*}
\begin{align*}
T^*_{\mathcal{V}} : \mathcal{V} \to \mathcal{V}
\end{align*}
\begin{align*}
\forall f \in \mathcal{V}
\end{align*}

Mapping

Bellman Optimality Operator (Point-wise)

Some recap on functional analysis

First, we need to understand the concepts of:

​

➢ Operator

​

➢Fixed Point

​

➢-contraction mapping

​

Fixed Point: Example

\begin{align*}
    f(x) = 0.5 x 
\end{align*}

Using operator notation

\begin{align*}
    \mathcal{T} f(x) = 0.5 f(x) 
\end{align*}
\begin{align*}
    f_0(x) = x 
\end{align*}

Initial function

\begin{align*}
   f_3(x) = \mathcal{T}  f_2(x)  = 0.5 f_2(x) = 0.5 \times 0.25 x = 0.125x 
\end{align*}

Third iteration

\begin{align*}
    f_k(x) = 0.5^k x 
\end{align*}

General iteration

\begin{align*}
    \mathcal{T}f^*(x) = f^*(x) 
\end{align*}

Fixed point

\begin{align*}
    f^*(x) = 0 
\end{align*}
\begin{align*}
    f_1(x) = \mathcal{T} f_0(x) = 0.5f_0(x) = 0.5x 
\end{align*}

First iteration

\begin{align*}
    f_2(x) = \mathcal{T} f_1(x) = 0.5 f_1(x) = 0.5 \times 0.5 x = 0.25x 
\end{align*}

Second iteration

Fixed Point

Applying the operator leaves you at the same point!

Fixed point of the operator

\begin{align*}
\mathcal{T} x = x
\end{align*}
\begin{align*}
x \in \boldsymbol{\mathcal{X}}
\end{align*}

Where be a vector space, equipped with a norm

Some recap on functional analysis

First, we need to understand the concepts of:

​

➢ Operator

​

➢Fixed Point

​

➢-contraction mapping

​

-contraction

An mapping is a -contraction mapping if for any , s.t.

\begin{align*}
||\mathcal{T} x_1 - \mathcal{T} x_2|| \leq \gamma ||x_1 - x_2||
\end{align*}

Let be a vector space, equipped with a norm

\begin{align*}
    ||0.5 x_1 - 0.5 x_2|| = 0.5 ||x_1 - x_2|| \leq \gamma ||x_1 - x_2|| \textrm{ for any } \gamma \in [0.5, 1) 
\end{align*}
\begin{align*}
    f(x) = 0.5 x 
\end{align*}

Example

-contraction

An mapping is a -contraction mapping if for any , s.t.

\begin{align*}
||\mathcal{T} x_1 - \mathcal{T} x_2|| \leq \gamma ||x_1 - x_2||
\end{align*}

Let be a vector space, equipped with a norm

Every contraction is also continuous. This means that

\begin{align*}
x_n \to_{||\cdot||} x
\end{align*}
\begin{align*}
\mathcal{T} x_n \to_{||\cdot||} \mathcal{T} x
\end{align*}

If the sequence converges in norm to x, the operator over the sequence converges in norm to the
operator over x.

Banach Fixed Point Theorem

has a unique fixed point , such that

Idependent of the starting point, the sequence converges
to
in a geometric fashion:

Let be a vector space, equipped with a norm

Let be a -contraction mapping

\begin{align*}
||x_n - x^*|| \leq \gamma^n ||x_0 - x^*||
\end{align*}
\begin{align*}
\lim_{n \to \infty} ||x_n - x^*|| \leq \lim_{n \to \infty} (\gamma^n ||x_0 - x^*||) = 0
\end{align*}

Banach Fixed Point Theorem

Banach Fixed Point Theorem

has a unique fixed point , such that

Idependent of the starting point, the sequence converges
to
in a geometric fashion:

Let be a vector space, equipped with a norm

Let be a -contraction mapping

Banach Fixed Point Theorem

How to prove that value
iteration converges using the
Banach Fixed Point Theorem?

Show that the Bellman operator is
a
-contraction mapping, has a
fixed point, and is monotonic!

Properties of the Bellman Operator T*

\begin{align*}
(T^*_{\mathcal{V}}) f(s) = \max_a \sum_{s', r} p(s', r | s, a) \Big( r + \gamma f(s') \Big)
\end{align*}
\begin{align*}
T^*_{\mathcal{V}} : \mathcal{V} \to \mathcal{V}
\end{align*}
\begin{align*}
\forall f \in \mathcal{V}
\end{align*}

Mapping

Bellman Optimality Operator (Point-wise)

1) has one unique fixed point v*

\begin{align*}
T^* v_* = v_*
\end{align*}

2) is a -contraction wrt. to

\begin{align*}
||T^* v - T^* u||_\infty \leq \gamma ||v - u||_\infty, \quad \forall u, v \in \mathcal{V}
\end{align*}

3) is monotonic

\begin{align*}
\forall u, v \in \mathcal{V} \text{ s.t. } u \leq v
\end{align*}
\begin{align*}
T^* u \leq T^* v
\end{align*}

component-wise

Proposition: T* is a -contraction wrt.

\begin{align*}
|T^* v(s) - T^* u(s)| = |\max_a \sum_{s', r} p(s', r | s, a)(r + \gamma v(s')) - \max_b \sum_{s'', r} p(s'', r | s, b)(r + \gamma u(s''))|
\end{align*}
\begin{align*}
(T^*_{\mathcal{V}}) f(s) = \max_a \sum_{s', r} p(s', r | s, a) \Big( r + \gamma f(s') \Big)
\end{align*}
\begin{align*}
\leq \max_a |\sum_{s', r} p(s', r | s, a)(r + \gamma v(s')) - \sum_{s', r} p(s', r | s, a)(r + \gamma u(s'))|
\end{align*}
\begin{align*}
|\max_a f(a) - \max_b g(b)| \leq \max_a |f(a) - g(a)|
\end{align*}

Proposition: T* is a -contraction wrt.

\begin{align*}
|T^* v(s) - T^* u(s)| = |\max_a \sum_{s', r} p(s', r | s, a)(r + \gamma v(s')) - \max_b \sum_{s'', r} p(s'', r | s, b)(r + \gamma u(s''))|
\end{align*}
\begin{align*}
(T^*_{\mathcal{V}}) f(s) = \max_a \sum_{s', r} p(s', r | s, a) \Big( r + \gamma f(s') \Big)
\end{align*}
\begin{align*}
\leq \max_a |\sum_{s', r} p(s', r | s, a)(r + \gamma v(s')) - \sum_{s', r} p(s', r | s, a)(r + \gamma u(s'))|
\end{align*}
\begin{align*}
= \gamma \max_a |\sum_{s', r} p(s', r | s, a)(v(s') - u(s'))|
\end{align*}
\begin{align*}
\leq \gamma \max_{s'} |(v(s') - u(s'))|
\end{align*}

Thus we get:

\begin{align*}
||T^* v - T^* u||_\infty \leq \gamma ||(v - u)||_\infty, \quad \forall u, v \in \mathcal{V}
\end{align*}

Proposition: T* is monotonic (Try at home)

\begin{align*}
v(s) \leq u(s), \forall s \to \sum_{a} \pi(a | s) \sum_{s', r} p(s', r | s, a)(r + \gamma v(s')) \leq \sum_{a} \pi(a | s) \sum_{s', r} p(s', r | s, a)(r + \gamma u(s'))
\end{align*}
\begin{align*}
T^* v(s) - T^* u(s) = \max_a \sum_{s', r} p(s', r | s, a)(r + \gamma v(s')) - \max_b \sum_{s'', r'} p(s'', r' | s, b)(r' + \gamma u(s''))
\end{align*}
\begin{align*}
\leq \max_a (\sum_{s', r} p(s', r | s, a)(r + \gamma v(s')) - \sum_{s', r} p(s', r | s, a)(r + \gamma u(s')))
\end{align*}
\begin{align*}
\leq 0, \forall s.
\end{align*}

Thus

\begin{align*}
T^* v(s) \leq T^* u(s), \forall s \in \mathcal{S}
\end{align*}

Value Iteration through the lens of the Bellman Operator

\begin{align*}
\text{As } k \to \infty, \quad v_k \to_{||\cdot||_\infty} v_*
\end{align*}

Proof: Direct application of the Banach Fixed Point Theorem.

\begin{align*}
||v_k - v_*||_\infty &= ||T^* v_{k-1} - v_*||_\infty \\ &= ||T^* v_{k-1} - T^* v_*||_\infty \\ &\leq \gamma||v_{k-1} - v_*||_\infty \\ &\leq \gamma^k||v_0 - v_*||_\infty
\end{align*}

Update Rule

Value Iteration through the lens of the Bellman Operator

\begin{align*}
\text{As } k \to \infty, \quad v_k \to_{||\cdot||_\infty} v_*
\end{align*}

Proof: Direct application of the Banach Fixed Point Theorem.

\begin{align*}
||v_k - v_*||_\infty &= ||T^* v_{k-1} - v_*||_\infty \\ &= ||T^* v_{k-1} - T^* v_*||_\infty \\ &\leq \gamma||v_{k-1} - v_*||_\infty \\ &\leq \gamma^k||v_0 - v_*||_\infty
\end{align*}
\begin{align*}
||v_k - v_*||_\infty &= ||T^* v_{k-1} - v_*||_\infty \\ &= ||T^* v_{k-1} - T^* v_*||_\infty \\ &\leq \gamma||v_{k-1} - v_*||_\infty \\ &\leq \gamma^k||v_0 - v_*||_\infty
\end{align*}
\begin{align*}
||v_k - v_*||_\infty &= ||T^* v_{k-1} - v_*||_\infty \\ &= ||T^* v_{k-1} - T^* v_*||_\infty \\ &\leq \gamma||v_{k-1} - v_*||_\infty \\ &\leq \gamma^k||v_0 - v_*||_\infty
\end{align*}
\begin{align*}
||v_k - v_*||_\infty &= ||T^* v_{k-1} - v_*||_\infty \\ &= ||T^* v_{k-1} - T^* v_*||_\infty \\ &\leq \gamma||v_{k-1} - v_*||_\infty \\ &\leq \gamma^k||v_0 - v_*||_\infty
\end{align*}

Fixed Point property

Contraction property

Iterative Application

\begin{align*}
= \gamma ||T^* v_{k-2} - v_*||_\infty
\end{align*}

Update rule

Synchronous Dynamic Programming Algorithms

Problem

Bellman Equation

Algorithm

Prediction

Bellman Expectation Equation

Iterative Policy Evaluation

Synchronous Dynamic Programming Algorithms

Problem

Bellman Equation

Algorithm

Prediction

Bellman Expectation Equation

Iterative Policy Evaluation

Control

Bellman Expectation Equation + (Greedy) Policy Improvement

Policy Iteration

Synchronous Dynamic Programming Algorithms

Problem

Bellman Equation

Algorithm

Prediction

Bellman Expectation Equation

Iterative Policy Evaluation

Control

Bellman Expectation Equation + (Greedy) Policy Improvement

Policy Iteration

Control

Bellman Optimality Equation

Value Iteration

Algorithms are based on:

\begin{align*}
v_\pi(s)
\end{align*}
\begin{align*}
v_*(s)
\end{align*}

State-value function

Action-value function

\begin{align*}
q_\pi(s, a)
\end{align*}
\begin{align*}
q_*(s, a)
\end{align*}

Complexity per iteration

Complexity per iteration

Asynchronous Dynamic Programming

DP methods described so far used synchronous updates (all states in parallel)

Asynchronous DP

➢backs up states individually, in any order

​

➢can significantly reduce computation

​

➢guaranteed to converge if all states continue to be selected

​

Asynchronous Dynamic Programming

Three simple ideas for asynchronous dynamic programming:

​

➢ In-place dynamic programming

​

➢ Prioritized sweeping

​

➢ Real-time dynamic programming

In-Place Dynamic Programming

Synchronous value iteration stores two copies of the value function

\begin{align*}
\color{red}{v_{\text{new}}}\color{black}(s) \leftarrow \max_a \sum_{s',r} p(s',r|s,a)(r + \gamma \color{red}{v_{\text{old}}}\color{black}(s'))
\end{align*}

For all in :

\begin{align*}
\color{red}{v}\color{black}(s) \leftarrow \max_a \sum_{s',r} p(s',r|s,a)(r + \gamma \color{red}{v}\color{black}(s'))
\end{align*}

In-place value iteration only stores one copy of the value function

For all in :

Prioritized Sweeping

➢Backup the state with the largest Bellman error

​

➢After each backup, update errors of affected predecessor states

​

➢Requires knowledge of reverse dynamics (predecessors)

​

➢Efficient via priority queue maintenance

Compute Bellman errors and store in a priority queue.

\begin{align*}
\Big| \max_a \sum_{s', r} p(s', r | s, a)(r + \gamma v(s')) - v(s) \Big|
\end{align*}

📌 Focus updates where they matter most → faster convergence

Real-Time Dynamic Programming

Idea: only update states that are relevant to the agent

Agent is in state

Update state value, or states that it
expects to be in soon.

Full-Width Backups

➢Standard DP uses full-width backups

​

➢For each backup (sync or async)

➢Every successor state and action is considered

➢Using true model of transitions and reward function

​

➢DP is effective for medium-sized problems (millions of states)

​

➢For large problems DP suffers from curse of dimensionality

➢Number of states grows exponentially with number of state variables

​

➢Even one full backup can be too expensive

Sample Backups

➢In subsequent lectures we will consider sample backups

​

➢Using sample rewards and sample transitions (s, a, r, s’)

(Instead of reward function r and transition dynamics p)

​

➢Advantages

​

➢Model-free: no prior knowledge of MDP required

​

➢Breaks the curse of dimensionality through sampling

​

➢Cost of backup is constant, independent of

Take home message

•The Bellman equations serves as the base to iterative algorithms.

•Algorithms are based on Policy Evaluation and Policy Improvement.

•Policy Iteration must converge to an optimal policy and the optimal value function
in a finite number of iterations.

•Value Iteration is a Policy iteration switching after each step.

•The Bellman operator has properties that assures the convergence of value
iteration.

•DP can be synchronous or asynchronous.

•DP requires the knowledge of the reward function r and transition function p.

•DP is computationally expensive.

​

Take home message

•Further studying:

•Chapter 4 – Reinforcement Learning: An introduction

•Short notes: https://people.stfx.ca/jdelamer/courses/csci-531/lectures/mdp/dynamic-
programming/dynamic-programming.html

•Video (UCL x DeepMind)

•Lectures 3 (after min 50) :
https://www.youtube.com/watch?v=zSOMeug_i_M&list=PLqYmG7hTraZCRwoyGxvQkqVrZgDQi4m-
5&index=3&t=2917s

•Lecture 4:
https://www.youtube.com/watch?v=XpbLq7rIJAA&list=PLqYmG7hTraZCRwoyGxvQkqVrZgDQi4m
-5&index=4

​

•Next lecture:

•Monte Carlo Methods and TD-learning

​