

Reinforcement Learning · Lecture 5
Model-Free Control

Rafael Fernandes Cunha
Today’s topics
1.Recap Monte Carlo and TD-learning
2.Model-Free Control
1.Monte Carlo
2.SARSA
3.Q-learning
4.Expected SARSA
5.Double Q-learning
6.DQN
3.Importance Sampling
2
1.Explain the SARSA algorithm
2.Explain Q-learning
3.Compare On-policy x Off-policy algorithms
4.Illustrate the difference between Q-learning
and Double Q-learning
5.Explain the expected SARSA algorithm
6.Explain the intuition behind DQN
7.Summarize the importance Sampling
methodology.
Learning outcomes
Dynamic Programming Backup

![\begin{align*}
v(S_t) \leftarrow \mathbb{E}[R_{t+1} + \gamma v(S_{t+1} | A_t \sim \pi(S_t))]
\end{align*}](media/a071651fb0.png)

Expectation notation
Explicit probabilities
Monte-Carlo Backup



Temporal-Difference Learning Backup



Sampled Returns (MC) vs Bootstrapping (TD)
MC target (Sampled Returns):
TD target (Bootstrapping):
Biased. Uses next reward plus estimated value (bootstrapping).
High Variance - Return depends on many random actions, transitions, rewards.
Low Variance: TD target depends on one random action, transition, reward.
Variance:
Variance:
Bias:


Update:
Unbiased. Uses actual returns.
Bias:
Update:
Offline – It needs to wait until the end of the episode to compute the return.
Online – It can update after each step.
Between MC and TD: Multi-Step TD
Unified View of
Reinforcement Learning


Multi-Step Prediction
Let the TD target look
steps into the future



-step returns



n-step TD

Mixed Multi-Step Returns

TD(0)

MC

Mixed Multi-Step Returns

Incremental Update rule
NewEstimate
OldEstimate
StepSize
InformativeValue
Estimate
Target
OldEstimate
-
Prediction:
Each algorithm changes only the Target value!
Algorithms: Model-Free Prediction
Monte Carlo
TD(0)
n-step TD
TD(
Mixed Multi-Step Returns

-step returns

Sampled Returns
Bootstrapping
Model-Free Control
➢Model-free prediction
➢Estimate the value function of an unknown MDP
➢Monte Carlo first/every visit
➢TD-learning (TD(0), n-step TD, TD())
➢Model-free control
➢Optimize the value function of an unknown MDP
➢Monte Carlo
➢SARSA
➢Q-learning
➢Expected SARSA
➢Double Q-learning
➢DQN
➢Importance Sampling
Today’s Lecture
Algorithms: Model-Free Control
Monte Carlo
SARSA
Q-learning
Expected SARSA
Double Q-learning
DQN
Generalized Policy Iteration (GPI)



Alternating with some chosen iterations.
Greedy policy improvement over
requires the model of the MDP

Greedy policy improvement over
is model-free



Generalized Policy Iteration (GPI)



Alternating with some chosen iterations.
Policy evaluation: Monte-Carlo
Policy improvement: Greedy Policy?
What is the problem here?
Can’t sample all s, a, when
learning by interaction!



Generalized Policy Iteration (GPI)



Alternating with some chosen iterations.
Policy evaluation: Monte-Carlo
Policy improvement: -greedy Policy!
Keep exploring all s, a, when
learning by interaction!
-greedy


Model-free control (MC)
Sample episode using , , …,
Generalizes the -greedy
bandit algorithm
Repeat
For each state and action in the episode,



Improve policy based on new action-value function


Algorithms: Model-Free Control
Monte Carlo
SARSA
Q-learning
Expected SARSA
Double Q-learning
DQN

GLIE
Greedy in the Limit with Infinite Exploration (GLIE)
All state-action pairs are explored infinitely many times
The policy converges to a greedy policy
Example
-greedy with
Theorem: GLIE Model-free control converges to the optimal action-value function,


Approach to ensure all actions are selected infinitely often
On-policy: Learn about target policy from experience sampled from behaviour policy
Off-policy: Learn about target policy from experience sampled from behaviour policy
➢Learn “counterfactually” about other things you could do: “what if …?”
➢“What if I would turn left” => new observations, rewards?
➢“What if I would play more defensively?” => different win probability?
➢“What if I would continue to go forward?” => how long until I bump into a wall?
➢“What if I would act greedy?” even if my behaviour policy is -greedy?
MC vs. TD Control
Temporal-difference (TD) learning has several advantages over Monte-Carlo (MC)
➢Lower variance
➢Online
➢Can learn from incomplete sequences
Let’s use TD instead of MC for control
➢Apply TD to
➢Use, e.g., -greedy policy improvement
➢Update every time-step
SARSA: On-policy TD Control
Policy evaluation: SARSA
Policy improvement: -greedy
Can I have another policy here?
Which condition should it obey?
Tabular SARSA converges to the optimal action-
value function,
Behaviour Policy is GLIE
Theorem



Example: Windy Gridworld
Policy: -greedy ()
Algorithm: SARSA ()
initial is 0 for all s, a

Why Monte-Carlo methods
cannot easily be used here?
Because termination is not
guaranteed for all policies.
Online learning methods
quickly learn a better policy
during the episode.
Algorithms: Model-Free Control
Monte Carlo
SARSA
Q-learning
Expected SARSA
Double Q-learning
DQN


Q-learning (Off-policy Learning)
Evaluate target policy
to compute or
Use behaviour policy to
generate actions
-greedy
greedy

Greedy policy does
not explore
➢Learn from observing humans or other agents (e.g., from logged data)
➢Re-use experience from old policies (e.g., from your own past experience)
➢Learn about multiple policies while following one policy
➢Learn about greedy policy while following exploratory policy
Q-learning Control Algorithm
Q-learning control converges to the optimal
action-value function,
Take each action in each state infinitely often
Theorem
From stochastic
approximation theory
Conditions required to
assure convergence
with probability 1
Note: no need for greedy behaviour!
Works for any policy that eventually
selects all actions sufficiently often!
Recall the conditions over step-size of update rules
Steps are large enough to overcome any
initial conditions or random fluctuations

Steps become small enough to
assure convergence





Example: Cliff Walking
SARSA (On-policy)
Q-learning (Off-policy)
Behaviour policy: -greedy
Target policy: -greedy
Behaviour policy: -greedy
Target policy: greedy


Example: Cliff Walking
SARSA (On-policy)
Q-learning (Off-policy)
Behaviour policy: -greedy
Target policy: -greedy
Behaviour policy: -greedy
Target policy: greedy
If , both methods would asymptotically converge to the optimal policy.
Which policy to evaluate when
running the experiment?


Algorithms: Model-Free Control
Monte Carlo
SARSA
Q-learning
Expected SARSA
Double Q-learning
DQN



Expected SARSA
Expected SARSA (sometimes called “General Q-learning” )
Q-learning is a special case with greedy target policy



Consider probabilities
under
Update towards
value of alternative action
Action may be chosen using
behaviour policy .
(off-policy learning)
Algorithms: Model-Free Control
Monte Carlo
SARSA
Q-learning
Expected SARSA
Double Q-learning
DQN




Example: Maximization Bias on Simple MDP

Policy: -greedy, with







% left actions from A
What is the optimal % of left actions
from A for this -greedy policy ?
Example: Maximization Bias on Simple MDP

Policy: -greedy, with







% left actions from A
What is the optimal % of left actions
from A for this -greedy policy ?
Example: Maximization Bias on Simple MDP

Policy: -greedy, with







% left actions from A
What is the optimal % of left actions
from A for this -greedy policy ?
Why does q-learning initially selects
left actions from A with a high %?


Q-learning overestimation
Selecting the action
Evaluating the action
Classical Q-learning has potential issues
But values are approximate
More likely to select overestimated values
Less likely to select underestimated values
This causes upward bias!
Same action-value function!

Q-learning overestimation
Selecting the action
Evaluating the action
Q-learning overestimates because:
1) It uses the same action-value function to select and to evaluate the action!
2) It has the maximization term.
➢Q-learning will update if the state actually has high value
Decouple selection from evaluation
Solution
Double Q-learning
➢Store two action-value functions: and
➢Each , pick or (e.g., randomly) and update it.
➢Can use both to act (e.g., use policy based on )
Double Q-learning also converges to the optimal policy under the same conditions as Q-learning
![\begin{align*}
\textcolor{blue}{q_2}(S_t, A_t) \leftarrow \textcolor{blue}{q_2}(S_t, A_t) + \alpha \big[R_{t+1} + \gamma \textcolor{red}{q_1}\Big(S_{t+1}, \argmax_a \textcolor{blue}{q_2}(S_{t+1}, a)\Big) - \textcolor{blue}{q_2}(S_t, A_t)\Big]
\end{align*}](media/a89a6088ec.png)
![\begin{align*}
\textcolor{red}{q_1}(S_t, A_t) \leftarrow \textcolor{red}{q_1}(S_t, A_t) + \alpha \big[R_{t+1} + \gamma \textcolor{blue}{q_2}\Big(S_{t+1}, \argmax_a \textcolor{red}{q_1}(S_{t+1}, a)\Big) - \textcolor{red}{q_1}(S_t, A_t)\Big]
\end{align*}](media/b1db6d6709.png)

Example: Maximization Bias on Simple MDP

Double learning
The idea of double Q-learning can be generalized to other updates
Using -greedy in SARSA
can also overestimate
Solution
Double SARSA!
Algorithms: Model-Free Control
Monte Carlo
SARSA
Q-learning
Expected SARSA
Double Q-learning
DQN




![\begin{align*}
\textcolor{blue}{q_2}(S_t, A_t) \leftarrow \textcolor{blue}{q_2}(S_t, A_t) + \alpha \big[R_{t+1} + \gamma \textcolor{red}{q_1}\Big(S_{t+1}, \argmax_a \textcolor{blue}{q_2}(S_{t+1}, a)\Big) - \textcolor{blue}{q_2}(S_t, A_t)\Big]
\end{align*}](media/a89a6088ec.png)
![\begin{align*}
\textcolor{red}{q_1}(S_t, A_t) \leftarrow \textcolor{red}{q_1}(S_t, A_t) + \alpha \big[R_{t+1} + \gamma \textcolor{blue}{q_2}\Big(S_{t+1}, \argmax_a \textcolor{red}{q_1}(S_{t+1}, a)\Big) - \textcolor{red}{q_1}(S_t, A_t)\Big]
\end{align*}](media/b1db6d6709.png)
Deep Reinforcement Learning
q-learning
Very big
state space
Impossible to
make a q-table
Approximate q-value
function by a neural network


Weight update

Deep Reinforcement Learning
q-learning
Very big
state space
Impossible to
make a q-table
Approximate q-value
function by a neural network
Simple Idea? Why did it take
so long to be invented?

Loss function
![\begin{align*}
L(\mathbf{w}) = \frac{1}{2} \Big(R_{t+1} + \gamma \Big[\!\!\Big[ \max_{a'} q_{\mathbf{w}}(S_{t+1}, a') \Big]\!\!\Big] - q_{\mathbf{w}}(S_t, A_t) \Big)^2
\end{align*}](media/5138ddd7e2.png)
The deadly triad
•Bootstrapping
•Off-policy learning
•Function approximation
Make the algorithm diverge!
Learning data is not Independent and Identically distributed (iid)
Main difference when comparing with supervised learning datasets!
Experience Replay Buffer
Experience Replay Buffer store experiences (Transitions)
➢Sampling from this buffer helps get less correlated data.

Target Network


Freeze Target network
Avoid chasing a moving target
Algorithms: Model-Free Control
Monte Carlo
SARSA
Q-learning
Expected SARSA
Double Q-learning
DQN




![\begin{align*}
\textcolor{blue}{q_2}(S_t, A_t) \leftarrow \textcolor{blue}{q_2}(S_t, A_t) + \alpha \big[R_{t+1} + \gamma \textcolor{red}{q_1}\Big(S_{t+1}, \argmax_a \textcolor{blue}{q_2}(S_{t+1}, a)\Big) - \textcolor{blue}{q_2}(S_t, A_t)\Big]
\end{align*}](media/a89a6088ec.png)
![\begin{align*}
\textcolor{red}{q_1}(S_t, A_t) \leftarrow \textcolor{red}{q_1}(S_t, A_t) + \alpha \big[R_{t+1} + \gamma \textcolor{blue}{q_2}\Big(S_{t+1}, \argmax_a \textcolor{red}{q_1}(S_{t+1}, a)\Big) - \textcolor{red}{q_1}(S_t, A_t)\Big]
\end{align*}](media/b1db6d6709.png)

![\begin{align*}
L(\mathbf{w}) = \frac{1}{2} \Big(R_{t+1} + \gamma \Big[\!\!\Big[ \max_{a'} q_{\mathbf{w}}(S_{t+1}, a') \Big]\!\!\Big] - q_{\mathbf{w}}(S_t, A_t) \Big)^2
\end{align*}](media/5138ddd7e2.png)
Off-Policy Learning: Importance Sampling Corrections
Off-policy: Learn about target policy from experience sampled from behaviour policy
Evaluate target policy
to compute or
Use behaviour policy
to generate actions
➢Q-learning is one example, but there are other options.
➢Note that you are sampling from one probability distribution to learn about a different one.
➢There are general tools to help with this.
➢Caveat: you can’t expect to learn about things you never do!
This may create problems!
Importance Sampling corrections
Given with random inputs ,
and distribution
Estimate the expectation of under
a different (target) distribution
Problem
Solution
Weigh the data by the ration
Intuition:
➢Scale up events that are rare under , but common under
➢Scale down events that are common under , but rare under
![\begin{align*}
\mathbb{E}_{x \sim d}[f(x)] =
\end{align*}](media/3041b02a54.png)


![\begin{align*}
=\mathbb{E}_{x \sim d' } \Big[ \frac{d(x)}{d' (x)} f(x) \Big]
\end{align*}](media/196f4e1011.png)
Importance sampling corrections
➢Example: estimate one-step reward
➢Behaviour is


![\begin{align*}
= \mathbb{E} \Big[\frac{\pi(A_t|S_t)}{\mu(A_t|S_t)}R_{t+1} | S_t = s, A_t \sim \mu \Big]
\end{align*}](media/8fad1d714d.png)
![\begin{align*}
\mathbb{E}[R_{t+1}|S_t = s, A_t \sim \pi]
\end{align*}](media/c32a33fb6b.png)
Importance sampling corrections
➢Example: estimate one-step reward
➢Behaviour is
![\begin{align*}
\mathbb{E}[R_{t+1}|S_t = s, A_t \sim \pi]
\end{align*}](media/c32a33fb6b.png)
![\begin{align*}
= \mathbb{E} \Big[\frac{\pi(A_t|S_t)}{\mu(A_t|S_t)}R_{t+1} | S_t = s, A_t \sim \mu \Big]
\end{align*}](media/8fad1d714d.png)
![\begin{align*}
= \mathbb{E} \Big[\frac{\pi(A_t|S_t)}{\mu(A_t|S_t)}R_{t+1} | S_t = s, A_t \sim \mu \Big]
\end{align*}](media/8fad1d714d.png)
Importance sampling corrections
➢Example: estimate one-step reward
➢Behaviour is
When following policy , use as unbiased sample!
![\begin{align*}
\mathbb{E}[R_{t+1}|S_t = s, A_t \sim \pi]
\end{align*}](media/c32a33fb6b.png)
![\begin{align*}
p_\pi(\tau) &= p_\pi(s_0, a_0, r_1, s_1, a_1, r_2, s_2) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_0, a_0, r_1, s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) p_\pi(a_1 | s_0, a_0, r_1, s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0) \pi(a_0 | s_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p(s_0) \prod_{t=0}^{T-1} \pi(a_t | s_t) p(s_{t+1}, r_{t+1} | s_t, a_t)
\end{align*}](media/d946e2cfd9.png)
![\begin{align*}
p_\pi(\tau) &= p_\pi(s_0, a_0, r_1, s_1, a_1, r_2, s_2) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_0, a_0, r_1, s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) p_\pi(a_1 | s_0, a_0, r_1, s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0) \pi(a_0 | s_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p(s_0) \prod_{t=0}^{T-1} \pi(a_t | s_t) p(s_{t+1}, r_{t+1} | s_t, a_t)
\end{align*}](media/d946e2cfd9.png)
![\begin{align*}
p_\pi(\tau) &= p_\pi(s_0, a_0, r_1, s_1, a_1, r_2, s_2) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_0, a_0, r_1, s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) p_\pi(a_1 | s_0, a_0, r_1, s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0) \pi(a_0 | s_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p(s_0) \prod_{t=0}^{T-1} \pi(a_t | s_t) p(s_{t+1}, r_{t+1} | s_t, a_t)
\end{align*}](media/d946e2cfd9.png)
Importance Sampling for Off-Policy Monte-Carlo
➢Goal: estimate
➢Data: trajectory generated with
➢First-step: Compute the probability of a trajectory generated following policy
Definition of conditional probability on

Markov Property
Def. of conditional probability on
![\begin{align*}
p_\pi(\tau) &= p_\pi(s_0, a_0, r_1, s_1, a_1, r_2, s_2) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_0, a_0, r_1, s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) p_\pi(a_1 | s_0, a_0, r_1, s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0) \pi(a_0 | s_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p(s_0) \prod_{t=0}^{T-1} \pi(a_t | s_t) p(s_{t+1}, r_{t+1} | s_t, a_t)
\end{align*}](media/d946e2cfd9.png)
![\begin{align*}
p_\pi(\tau) &= p_\pi(s_0, a_0, r_1, s_1, a_1, r_2, s_2) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_0, a_0, r_1, s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) p_\pi(a_1 | s_0, a_0, r_1, s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0) \pi(a_0 | s_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p(s_0) \prod_{t=0}^{T-1} \pi(a_t | s_t) p(s_{t+1}, r_{t+1} | s_t, a_t)
\end{align*}](media/d946e2cfd9.png)
![\begin{align*}
p_\pi(\tau) &= p_\pi(s_0, a_0, r_1, s_1, a_1, r_2, s_2) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_0, a_0, r_1, s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) p_\pi(a_1 | s_0, a_0, r_1, s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0) \pi(a_0 | s_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p(s_0) \prod_{t=0}^{T-1} \pi(a_t | s_t) p(s_{t+1}, r_{t+1} | s_t, a_t)
\end{align*}](media/d946e2cfd9.png)
![\begin{align*}
p_\pi(\tau) &= p_\pi(s_0, a_0, r_1, s_1, a_1, r_2, s_2) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_0, a_0, r_1, s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) p_\pi(a_1 | s_0, a_0, r_1, s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0) \pi(a_0 | s_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p(s_0) \prod_{t=0}^{T-1} \pi(a_t | s_t) p(s_{t+1}, r_{t+1} | s_t, a_t)
\end{align*}](media/d946e2cfd9.png)
![\begin{align*}
p_\pi(\tau) &= p_\pi(s_0, a_0, r_1, s_1, a_1, r_2, s_2) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_0, a_0, r_1, s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) p_\pi(a_1 | s_0, a_0, r_1, s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0) \pi(a_0 | s_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p(s_0) \prod_{t=0}^{T-1} \pi(a_t | s_t) p(s_{t+1}, r_{t+1} | s_t, a_t)
\end{align*}](media/d946e2cfd9.png)
![\begin{align*}
p_\pi(\tau) &= p_\pi(s_0, a_0, r_1, s_1, a_1, r_2, s_2) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_0, a_0, r_1, s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) p_\pi(a_1 | s_0, a_0, r_1, s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0) \pi(a_0 | s_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p(s_0) \prod_{t=0}^{T-1} \pi(a_t | s_t) p(s_{t+1}, r_{t+1} | s_t, a_t)
\end{align*}](media/d946e2cfd9.png)
![\begin{align*}
p_\pi(\tau) &= p_\pi(s_0, a_0, r_1, s_1, a_1, r_2, s_2) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_0, a_0, r_1, s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) p_\pi(a_1 | s_0, a_0, r_1, s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0) \pi(a_0 | s_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p(s_0) \prod_{t=0}^{T-1} \pi(a_t | s_t) p(s_{t+1}, r_{t+1} | s_t, a_t)
\end{align*}](media/d946e2cfd9.png)
![\begin{align*}
p_\pi(\tau) &= p_\pi(s_0, a_0, r_1, s_1, a_1, r_2, s_2) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_0, a_0, r_1, s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) p_\pi(a_1 | s_0, a_0, r_1, s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0) \pi(a_0 | s_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p(s_0) \prod_{t=0}^{T-1} \pi(a_t | s_t) p(s_{t+1}, r_{t+1} | s_t, a_t)
\end{align*}](media/d946e2cfd9.png)
Importance Sampling for Off-Policy Monte-Carlo
➢Goal: estimate
➢Data: trajectory generated with
➢First-step: Compute the probability of a trajectory generated following policy
Definition of conditional probability on

Markov Property
Def. of conditional probability on
Definition of policy (“It is Markov”)
Conditional probability on
Definition of policy on
General formulation for trajectories until Terminal state.
![\begin{align*}
p_\pi(\tau) &= p_\pi(s_0, a_0, r_1, s_1, a_1, r_2, s_2) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_0, a_0, r_1, s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) p_\pi(a_1 | s_0, a_0, r_1, s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0) \pi(a_0 | s_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p(s_0) \prod_{t=0}^{T-1} \pi(a_t | s_t) p(s_{t+1}, r_{t+1} | s_t, a_t)
\end{align*}](media/d946e2cfd9.png)
Importance Sampling for Off-Policy Monte-Carlo
➢Goal: estimate
➢Data: trajectory generated with
➢First-step: Compute the probability of a trajectory generated following policy
![\begin{align*}
p_\pi(\tau) &= p_\pi(s_0, a_0, r_1, s_1, a_1, r_2, s_2) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_0, a_0, r_1, s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) p_\pi(a_1 | s_0, a_0, r_1, s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0) \pi(a_0 | s_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p(s_0) \prod_{t=0}^{T-1} \pi(a_t | s_t) p(s_{t+1}, r_{t+1} | s_t, a_t)
\end{align*}](media/d946e2cfd9.png)
Importance Sampling for Off-Policy Monte-Carlo
➢Goal: estimate
➢Data: trajectory generated with
➢First-step: Compute the probability of a trajectory generated following policy

➢Solution: use return , and correct:

![\begin{align*}
p_\pi(\tau) &= p_\pi(s_0, a_0, r_1, s_1, a_1, r_2, s_2) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_0, a_0, r_1, s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) p_\pi(a_1 | s_0, a_0, r_1, s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0) \pi(a_0 | s_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p(s_0) \prod_{t=0}^{T-1} \pi(a_t | s_t) p(s_{t+1}, r_{t+1} | s_t, a_t)
\end{align*}](media/d946e2cfd9.png)
![\begin{align*}
p_\pi(\tau) &= p_\pi(s_0, a_0, r_1, s_1, a_1, r_2, s_2) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_0, a_0, r_1, s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1, a_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) p_\pi(a_1 | s_0, a_0, r_1, s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0, r_1, s_1) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0, a_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p_\pi(s_0) \pi(a_0 | s_0) p(s_1, r_1 | s_0, a_0) \pi(a_1 | s_1) p(s_2, r_2 | s_1, a_1) \\[1em] &= p(s_0) \prod_{t=0}^{T-1} \pi(a_t | s_t) p(s_{t+1}, r_{t+1} | s_t, a_t)
\end{align*}](media/d946e2cfd9.png)
Importance Sampling for Off-Policy TD Updates
➢Use TD targets generated from to evaluate
➢Weigh TD target by importance sampling
➢Only need a single importance sampling correction
➢Much lower variance than Monte-Carlo importance sampling
➢Policies only need to be similar over a single step

Importance Sampling for Off-Policy TD Updates (Proof)
![\begin{align*}
\mathbb{E}_{\mu} \Big[\frac{\pi(A_t|S_t)}{\mu(A_t|S_t)}(R_{t+1} + \gamma v(S_{t+1})) - v(S_t) \Big| S_t = s \Big]
\end{align*}](media/110e3072ca.png)
![\begin{align*}
= \sum_a \mu(a|s) \Big(\frac{\pi(a|s)}{\mu(a|s)} \mathbb{E}[R_{t+1} + \gamma v(S_{t+1})|S_t = s, A_t = a] - v(s) \Big)
\end{align*}](media/cf93ae7da5.png)
![\begin{align*}
= \sum_a \pi(a|s) \mathbb{E}[R_{t+1} + \gamma v(S_{t+1})|S_t = s, A_t = a] - \sum_a \mu(a|s) v(s)
\end{align*}](media/974ecf569d.png)
![\begin{align*}
= \sum_a \pi(a|s) \mathbb{E}[R_{t+1} + \gamma v(S_{t+1})|S_t = s, A_t = a] - \sum_a \pi(a|s) v(s)
\end{align*}](media/18b3403518.png)
![\begin{align*}
= \sum_a \pi(a|s) \Big(\mathbb{E}[R_{t+1} + \gamma v(S_{t+1})|S_t = s, A_t = a] - v(s) \Big)
\end{align*}](media/d70bdd505b.png)
![\begin{align*}
= \mathbb{E}_{\pi}\Big[R_{t+1} + \gamma v(S_{t+1}) - v(s)|S_t = s\Big]
\end{align*}](media/cf7c5af122.png)
Take home message
•Q-learning uses a greedy target policy
•SARSA uses a stochastic sample from the behaviour as target policy
•Expected SARSA uses any target policy
•Double learning uses a separate value function to evaluate the policy
•Double learning is not necessary if there is no correlation between target policy
and value function.
•When using a greedy policy (Q-learning), there are strong correlations. Then
double learning (Double Q-learning) can be useful.
•DQN is q-learning with function approximation on the state space plus replay
buffer and target network to help stabilize training.
Take home message
•Further studying:
•Chapter 6– Reinforcement Learning: An introduction
•Short notes:
•https://people.stfx.ca/jdelamer/courses/csci-531/lectures/rl/monte-carlo/monte-carlo-methods.html
•https://people.stfx.ca/jdelamer/courses/csci-531/lectures/drl/deep-q-learning.html
•Video (UCL x DeepMind)
•Lectures 6:
https://www.youtube.com/watch?v=t9uf9cuogBo&list=PLqYmG7hTraZCRwoyGxvQkqVrZgDQi4m-
5&index=6
•Next lecture:
•Function Approximators