Reinforcement Learning · Lecture 4

Monte Carlo Methods and TD Learning

Rafael Fernandes Cunha

rafaelcunha2013.github.io

Today’s topics

1.Recap on Dynamic Programming

2.Monte Carlo algorithm

3.Temporal-Difference (TD) algorithms

1.TD(0)

2.n-steps TD

3.TD()

2

1.Explain the Monte Carlo algorithm

2.Explain TD algorithms

3.Analyze the differences between Monte Carlo
and TD methods

Learning outcomes

Policy Improvement Theorem

Let and ’ be any pair of deterministic policies such that,

Then the policy must be as good as, or better than .

That is, it must obtain greater or equal expected return from all states

\begin{align*}
s \in \mathcal{S}
\end{align*}
\begin{align*}
\forall s \in \mathcal{S}
\end{align*}
\begin{align*}
v_{\pi}(s) \leq q_{\pi}(s, \pi'(s))
\end{align*}
\begin{align*}
v_{\pi}(s) \leq v_{\pi'}(s)
\end{align*}

Value Iteration

\begin{align*}
v_\pi(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_\pi(s')\Big)
\end{align*}
\begin{align*}
v_*(s) = \max_a \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_*(s')\Big)
\end{align*}

Bellman optimality equation

Bellman expectation equation

Policy Evaluation

Value Iteration

\begin{align*}
v_{k+1}(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_k(s')\Big)
\end{align*}

Turn into an
update rule

\begin{align*}
v_{k+1}(s) = \max_a \sum_{s',r} p(s',r|s,a)\Big(r + \gamma v_k(s')\Big)
\end{align*}

Turn into an
update rule

Equivalent to policy iteration, with step of policy
evaluation
between each two (greedy) policy
improvement steps.

Generalized Policy Iteration (GPI)

Alternating with some chosen iterations.

If both process converge,
the value function and
policy must be optimal.

Almost all RL methods are
well described as GPI

All have identifiable policies
and value functions

Policy being improved with
respect to the value function

Value function been driven toward
the value function for the policy.

Today’s Lecture

Solving the Bellman Optimality Equation

​

​

➢Many iterative solution methods:

​

➢Using models / dynamic programming

➢Value iteration

➢Policy iteration

​

➢Using samples (Model-free)

​

➢Prediction: Estimate the value function of an unknown MDP

➢Monte Carlo first/every visit

➢TD-learning (TD(0), n-step TD, TD())

​

➢Control: Optimize the value function of an unknown MDP

➢Monte Carlo exploring starts

➢SARSA, Q-learning, etc

➢DQN

​

Sampling vs Expectation

Consider a random variable X that can take values from a finite set of real numbers denoted by

Goal: Calculate the expected value of X.

Recall the Multi-Armed Bandit problem

Model-based (Expected Value)

\begin{align*}
\mathbb{E}[X] = \sum_{x \in \mathcal{X}} p(x) x
\end{align*}

Model-free (Sample Mean)

We sample from X:

\begin{align*}
\{x_1, x_2, \ldots, x_n\}
\end{align*}
\begin{align*}
\mathbb{E}[X] \approx \bar{x} = \frac{1}{n} \sum_{j=1}^{n} x_j
\end{align*}

Monte Carlo Algorithm

➢We can use experience samples to learn without a model

​

➢We call direct sampling of episodes Monte Carlo

​

➢MC is model-free: no knowledge of MDP required, only samples

What does it mean “without
a model” here?

MDP

\begin{align*}
(\mathcal{S}, \mathcal{A}, p, r, \gamma)
\end{align*}

Relationship between Monte Carlo and Bandits

\begin{align*}
(\mathcal{A}, r)
\end{align*}
\begin{align*}
(\mathcal{S}, \mathcal{A}, p, r, \gamma)
\end{align*}

MDP

Bandit

What does it mean
“without a model”
here?

How do we estimate the action
function in the bandit problem?

\begin{align*}
q_*(a) = \mathbb{E}[R_t | A_t = a]
\end{align*}

Action Value

We sample rewards from a reward
distribution, and use an update rule.

\begin{align*}
v_\pi(s) \doteq \mathbb{E}_\pi[G_t | S_t = s]
\end{align*}
\begin{align*}
G_t \doteq R_{t+1} + R_{t+2} + \ldots + R_T
\end{align*}

How do we estimate the state-value function in the MDP
problem?

We sample trajectories from an MDP, and use an update rule!

Incremental Implementation

\begin{align*}
    Q_{n+1}(a) = Q_n(a) + \alpha_n(a) \Big[R_n - Q_n(a)\Big] 
\end{align*}
\begin{align*}
S_0, A_0, R_1, S_1, A_1, R_2, \ldots, S_{T-1}, A_{T-1}, R_T, S_T \sim \pi
\end{align*}

Relationship between Monte Carlo and Bandits

\begin{align*}
(\mathcal{A}, r)
\end{align*}
\begin{align*}
(\mathcal{S}, \mathcal{A}, p, r, \gamma)
\end{align*}

MDP

Bandit

What does it mean
“without a model”
here?

How do we estimate the action
function in the bandit problem?

\begin{align*}
q_*(a) = \mathbb{E}[R_t | A_t = a]
\end{align*}

Action Value

We sample rewards from a reward
distribution, and use an update rule.

\begin{align*}
v_\pi(s) \doteq \mathbb{E}_\pi[G_t | S_t = s]
\end{align*}
\begin{align*}
G_t \doteq R_{t+1} + R_{t+2} + \ldots + R_T
\end{align*}

How do we estimate the state-value function in the MDP
problem?

We sample trajectories from an MDP, and use an update rule!

\begin{align*}
v_{n+1}(S_t) = v_n(S_t) + \alpha\Big(G_t - v_n(S_t)\Big)
\end{align*}

Incremental Implementation

\begin{align*}
    Q_{n+1}(a) = Q_n(a) + \alpha_n(a) \Big[R_n - Q_n(a)\Big] 
\end{align*}

NewEstimate

OldEstimate

StepSize

Informative
Value

\begin{align*}
v_{n+1}(S_t) = v_n(S_t) + \alpha\Big(G_t - v_n(S_t)\Big)
\end{align*}
\begin{align*}
v_{n+1}(S_t) = v_n(S_t) + \alpha\Big(G_t - v_n(S_t)\Big)
\end{align*}
\begin{align*}
v_{n+1}(S_t) = v_n(S_t) + \alpha\Big(G_t - v_n(S_t)\Big)
\end{align*}
\begin{align*}
v_{n+1}(S_t) = v_n(S_t) + \alpha\Big(G_t - v_n(S_t)\Big)
\end{align*}

Error

Error

Model-Free Prediction: Monte Carlo Algorithms

Monte-Carlo Policy Evaluation

➢With MDP’s, we are considering sequential decision problems.

➢Goal: Learn or from episodes of experience under policy .

\begin{align*}
S_0, A_0, R_1, S_1, A_1, R_2, \ldots, S_{T-1}, A_{T-1}, R_T, S_T \sim \pi
\end{align*}

➢The value function is the expected return from each state:

\begin{align*}
v_\pi(s) \doteq \mathbb{E}_\pi[G_t | S_t = s]
\end{align*}

➢We can just use the sample mean return instead of the expected return.

​

➢We call this Monte Carlo policy evaluation.

Monte-Carlo prediction (first-visit)

MC First-visit vs Every visit

\begin{align*}
s_1 \xrightarrow{a_2} s_2 \xrightarrow{a_4} s_1 \xrightarrow{a_2} s_2 \xrightarrow{a_3} s_5 \xrightarrow{a_1} \ldots &\quad [\text{original episode}] \\ s_2 \xrightarrow{a_4} s_1 \xrightarrow{a_2} s_2 \xrightarrow{a_3} s_5 \xrightarrow{a_1} \ldots &\quad [\text{subepisode starting from } (s_2, a_4)] \\ s_1 \xrightarrow{a_2} s_2 \xrightarrow{a_3} s_5 \xrightarrow{a_1} \ldots &\quad [\text{subepisode starting from } (s_1, a_2)] \\ s_2 \xrightarrow{a_3} s_5 \xrightarrow{a_1} \ldots &\quad [\text{subepisode starting from } (s_2, a_3)] \\ s_5 \xrightarrow{a_1} \ldots &\quad [\text{subepisode starting from } (s_5, a_1)]
\end{align*}

Remove the “Unless appears in the previous states” condition.

Monte-Carlo prediction (first-visit)

Every-visit

Example: Blackjack

➢State Space:

➢Current sum (12-21)

➢Dealer’s showing card (ace-10)

➢Do I have a “usable” ace? (yes-no)

​

➢Action Space = {stick, draw}

​

➢Reward function:

➢for stick:

➢+1 if sum of cards > sum of dealer cards

➢0 if sum of cards = sum of dealer cards

➢-1 if sum of cards < sum of dealer cards

​

➢for draw:

➢-1 if sum of cards > 21 (and terminate)

➢0 otherwise

​

➢Transition function:

➢Automatically draw if sum of cards < 12

➢Next card randomly sampled from a deck.

Stop receiving cards (and terminate)

Take another card (random, no replacement)

Example: Blackjack

Disadvantage of Monte-Carlo Learning

➢We have seen that MC algorithms can be used to learn value predictions

​

➢But when episodes are long, learning can be slow

➢...we have to wait until an episode ends before we can learn

➢...return can have high variance

Are there alternatives?

Temporal Difference Learning by Sampling Bellman Equations

Bellman Equations

\begin{align*}
v_\pi(s) = \mathbb{E}_\pi[R_{t+1} + \gamma v_\pi(S_{t+1}) | S_t = s]
\end{align*}

Policy Evaluation (Iterative)

\begin{align*}
v_{k+1}(s) = \mathbb{E}_\pi[R_{t+1} + \gamma v_k(S_{t+1}) | S_t = s]
\end{align*}

Sample this equation!

\begin{align*}
v_{t+1}(S_t) = R_{t+1} + \gamma v_t(S_{t+1})
\end{align*}

What are possible problems with this approach?

Changes are very abrupt! Use update rule with a small step-size.

\begin{align*}
    \text{NewEstimate} \leftarrow \text{OldEstimate} + \text{StepSize} \Big [ \text{Target} - \text{OldEstimate} \Big ] 
\end{align*}
\begin{align*}
    \text{NewEstimate} \leftarrow \text{OldEstimate} + \text{StepSize} \Big [ \text{Target} - \text{OldEstimate} \Big ] 
\end{align*}
\begin{align*}
v_{t+1}(S_t) \leftarrow v_t(S_t) + \alpha\Big(R_{t+1} + \gamma v_t(S_{t+1}) - v_t(S_t)\Big)
\end{align*}

Temporal Difference Learning by Sampling Bellman Equations

\begin{align*}
v_{t+1}(S_t) = R_{t+1} + \gamma v_t(S_{t+1})
\end{align*}

Target

Time-difference error

Error in the estimate!

Reduced by taking a step toward the “target”

Temporal difference learning

Prediction setting: learn online from experience under policy

\begin{align*}
v_{n+1}(S_t) \leftarrow v_n(S_t) + \alpha\Big(G_t - v_n(S_t)\Big)
\end{align*}

Monte-Carlo

Update value towards sampled returns

\begin{align*}
v_{t+1}(S_t) \leftarrow v_t(S_t) + \alpha\Big(R_{t+1} + \gamma v_t(S_{t+1}) - v_t(S_t)\Big)
\end{align*}

Temporal-difference learning:

Update value towards estimated return

Target

Target

\begin{align*}
\delta_t
\end{align*}
\begin{align*}
v_{n+1}(S_t) \leftarrow v_n(S_t) + \alpha\Big(G_t - v_n(S_t)\Big)
\end{align*}
\begin{align*}
v_{t+1}(S_t) \leftarrow v_t(S_t) + \alpha\Big(R_{t+1} + \gamma v_t(S_{t+1}) - v_t(S_t)\Big)
\end{align*}

Temporal difference learning

Prediction setting: learn online from experience under policy

Monte-Carlo

Update value towards sampled returns

Temporal-difference learning:

Update value towards estimated return

TD error

TD(0) algorithm

\renewcommand{\thealgorithm}{}

\begin{algorithm}
\caption{Tabular TD($0$) for estimating \(V \approx v_\pi\).}
\begin{algorithmic}[1]
\State Input: the policy $\pi$ to be evaluated
\State Algorithm parameters: step size \(\alpha \in (0, 1]\)
\State Initialize \(V(s)\), for all \(s \in \mathcal{S}^+\), arbitrarily except that \(V(\text{terminal}) = 0\) \\

\For{each episode}
\State Initialize \(S\)

\For{\(t=1, ..., T\) (until terminal state)}
\State $A \leftarrow$ action given by $\pi$ for $S$
\State Take action \(A\), observe \(R\), \(S'\)
\State \(\delta = R + \gamma V(S') - V(S)\)
\State \(V(s) \leftarrow V(s) + \alpha \delta \)
\State \(S \leftarrow S'\)
\EndFor
\EndFor
\label{tabular TD(lambda)}
\end{algorithmic}
\end{algorithm}

Dynamic Programming Backup

\begin{align*}
    v(s) \leftarrow \sum_a \pi(a|s)\sum_{s', r} p(s',r|s,a)(r+ \gamma v(s'))
\end{align*}

Monte-Carlo Backup

Temporal-Difference Backup

Sampling: update samples an expectation

Monte-Carlo

Dynamic Programming

Temporal-Difference learning

Bootstrapping: update involves an estimate

Monte-Carlo

Dynamic Programming

Temporal-Difference learning

Bootstrapping and Sampling

\begin{align*}
    v(S_t) \leftarrow v(S_t) + \alpha \Big(G_t - v(S_t)\Big) 
\end{align*}
\begin{align*}
    v(S_t) \leftarrow v(S_t) + \alpha \Big(R_{t+1} + v(S_{t+1}) - v(S_t)\Big) 
\end{align*}
\begin{align*}
    v(S_t) \leftarrow \mathbb{E}_{\pi}[R_{t+1} + \gamma v(S_{t+1})| S_t = s]
\end{align*}
\begin{align*}
    v(s) \leftarrow \sum_a \pi(a|s)\sum_{s', r} p(s',r|s,a)(r+ \gamma v(s'))
\end{align*}

Temporal difference learning

➢TD is model-free (no knowledge of MDP) and learns directly from experience

​

➢TD can learn from incomplete episodes, by bootstrapping

​

➢TD can learn during each episode

Example: Driving Home

State

Elapsed Time
(minutes)

Predicted Time to
Go

Predicted Total
Time

Leaving office

0

30

30

reach car

5

35

40

Exiting highway

20

15

35

2ndary road

30

10

40

home street

40

3

43

arrive home

43

0

43

Example: Driving Home (MC vs. TD)

Changes recommended by Monte Carlo methods ()

Changes recommended by TD methods ()

Advantages and Disadvantages of MC vs. TD

➢TD can learn before knowing the final outcome

➢TD can learn online after every step

➢MC must wait until end of episode before return is known (offline)

​

➢TD can learn without the final outcome

➢TD can learn from incomplete sequences

➢MC can only learn from complete sequences

➢TD works in continuing (non-terminating) environments

➢MC only works for episodic (terminating) environments

​

➢TD is independent of the temporal span of the prediction

➢TD can learn from single transitions

➢MC must store all predictions (or states) to update at the end of an episode

​

➢TD needs reasonable value estimates

Bias/Variance Trade-Off

MC return

\begin{align*}
G_t = R_{t+1} + \gamma R_{t+2} + \ldots
\end{align*}

Unbiased estimate of

Why?

TD target

Biased estimate of

Why?

\begin{align*}
R_{t+1} + \gamma v_t(S_{t+1})
\end{align*}

Higher Variance: Return depends on many random actions, transitions, rewards.

Lower Variance: TD target depends on one random action, transition, reward.

Variance:

Bias:

Variance:

Bias:

Bias/Variance Trade-Off

➢In some cases, TD can have irreducible bias

​

➢The world may be partially observable

➢MC would implicitly account for all the latent variables

​

➢The function to approximate the values may fit poorly

​

➢In the tabular case, both MC and TD will converge:

Example: Random Walk

➢Uniform random transitions (50% left, 50% right)

​

➢Initial values are , for all states

​

Example: Random Walk

True values

\begin{align*}
v(A) = \frac{1}{6}
\end{align*}
\begin{align*}
v(B) = \frac{2}{6}
\end{align*}
\begin{align*}
v(D) = \frac{4}{6}
\end{align*}
\begin{align*}
v(E) = \frac{5}{6}
\end{align*}
\begin{align*}
v(C) = \frac{3}{6}
\end{align*}

Example: Random Walk (MC vs. TD)

Which method performed
better on this task?

Batch MC and TD

➢Tabular MC and TD converge: as experience and

But what about finite experience?

Repeatedly sample each episode and apply MC or TD(0)

Sampling from an
empirical model

Consider a fixed batch of experiences:

Episode 1:

Episode k:

\begin{align*}
S^1_1, A^1_1, R^1_2, \ldots, S^1_T
\end{align*}
\begin{align*}
S^k_1, A^k_1, R^k_2, \ldots, S^k_T
\end{align*}

Example: Batch Learning in Two States

➢Two states A, B

➢No discounting

➢8 episodes

What is v(A), v(B)?

1)A, 0, B, 0

2)B, 1

3)B, 1

4)B, 1

5)B, 1

6)B, 1

7)B, 1

8)B, 0

Batch TD(0)

Batch MC

Differences in batch solutions

MC converges to the best mean-squared fit for the observed returns

\begin{align*}
\sum_{k=1}^{K} \sum_{t=1}^{T_k} \Big(G^k_t - v(S^k_t)\Big)^2
\end{align*}

TD converges to the solution of the max likelihood Markov model, given the data

​

➢Solution to the empirical MDP that best fits the data

​

➢In the AB example:

\begin{align*}
\hat{p}(S_{t+1} = B | S_t = A) = 1
\end{align*}
\begin{align*}
v(A) = v(B) = 0.75
\end{align*}
\begin{align*}
v(A) = 0
\end{align*}
\begin{align*}
v(A) = v(B) = 0.75
\end{align*}

Advantages and Disadvantages of MC vs. TD

➢TD exploits the Markov property

➢Can help in fully-observable environments

​

➢MC does not exploit the Markov property

➢Can help in partially-observable environments

​

➢With finite data, or with function approximation, the solutions may differ

Between MC and TD: Multi-Step TD

Unified View of
Reinforcement Learning

Multi-Step Updates

➢TD uses value estimates, which might be inaccurate.

​

➢In addition, information can propagate back quite slowly.

​

➢In MC, information propagates faster, but the updates are noisier.

​

➢We can go somewhere in between TD and MC

Multi-Step Prediction

Let the TD target look
steps into the future

Multi-Step Prediction

Let the TD target look
steps into the future

\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}

Multi-Step Prediction

Let the TD target look
steps into the future

\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}

-step returns

\begin{align*}
G^{(n)}_t \doteq R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{n-1} R_{t+n} + \gamma^n v(S_{t+n})
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}

n-step TD

\begin{align*}
v_{k+1}(S_t) \leftarrow v_k(S_t) + \alpha\Big(G^{(n)}_t - v_k(S_t)\Big)
\end{align*}

Example: Large Random Walk

Imagine a LARGER random walk.

19 states instead of only 5

Average RMS error
over 19 states and first
10 episodes

Performance of the -step TD method as a function of

Mixed Multi-Step Returns

\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}

n-step TD

\begin{align*}
v_{k+1}(S_t) \leftarrow v_k(S_t) + \alpha\Big(G^{(n)}_t - v_k(S_t)\Big)
\end{align*}

Mixed Multi-Step Returns

\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}

Which should we choose?

Why not select all n-step returns?

Let’s consider a weighted sum with .

n-step TD

\begin{align*}
v_{k+1}(S_t) \leftarrow v_k(S_t) + \alpha\Big(G^{(n)}_t - v_k(S_t)\Big)
\end{align*}

Mixed Multi-Step Returns

\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}

Which should we choose?

Why not select all n-step returns?

Let’s consider a weighted sum with .

n-step TD

\begin{align*}
v_{k+1}(S_t) \leftarrow v_k(S_t) + \alpha\Big(G^{(n)}_t - v_k(S_t)\Big)
\end{align*}

Mixed Multi-Step Returns

\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}

Note that:

To normalize this sum, multiply it by

Let’s use these weights to
average the n-step returns!

n-step TD

\begin{align*}
v_{k+1}(S_t) \leftarrow v_k(S_t) + \alpha\Big(G^{(n)}_t - v_k(S_t)\Big)
\end{align*}

Mixed Multi-Step Returns

\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}
\begin{align*}
n &= 1 \quad (TD) & G_t^{(1)} &= R_{t+1} + \gamma v(S_{t+1}) \\ n &= 2 & G_t^{(2)} &= R_{t+1} + \gamma R_{t+2} + \gamma^2 v(S_{t+2}) \\ &\ldots & \ldots\; & \\ n &= \infty \quad (MC) & G_t^{(\infty)} &= R_{t+1} + \gamma R_{t+2} + \ldots + \gamma^{T-t-1} R_T
\end{align*}

Note that:

To normalize this sum, multiply it by

Let’s use these weights to
average the n-step returns!

Mixed Multi-Step Returns

update equation

\begin{align*}
v_{k+1}(S_t) \leftarrow v_k(S_t) + \alpha\bigg(G^{(\lambda)}_t - v_k(S_t)\bigg)
\end{align*}

n-step TD

\begin{align*}
v_{k+1}(S_t) \leftarrow v_k(S_t) + \alpha\Big(G^{(n)}_t - v_k(S_t)\Big)
\end{align*}

How to compute TD() online?

Eligibility Traces

Mixed Multi-Step Returns

update equation

\begin{align*}
G^{(\lambda)}_t = R_{t+1} + \gamma\Big((1 - \lambda) v(S_{t+1}) + \lambda G^{(\lambda)}_{t+1}\Big)
\end{align*}

Try to derive
it at home.

\begin{align*}
G^{\lambda=0}_t = R_{t+1} + \gamma v(S_{t+1})
\end{align*}

TD(0)

\begin{align*}
G^{\lambda=1}_t = R_{t+1} + \gamma G_{t+1}
\end{align*}

MC

How to obtain TD(0) and MC from the equation above?

Eligibility Traces

Mixed Multi-Step Returns

update equation

\begin{align*}
G^{(\lambda)}_t = R_{t+1} + \gamma\Big((1 - \lambda) v(S_{t+1}) + \lambda G^{(\lambda)}_{t+1}\Big)
\end{align*}

Try to derive
it at home.

\begin{align*}
\Delta v(s) &= \alpha \sum_{t=0}^{T-1} \Big( G_t^{(\lambda)} - v_k(S_t) \Big)\, \mathbb{I}(S_t = s)
\end{align*}

Total change of
over an episode.

Eligibility Traces

Mixed Multi-Step Returns

update equation

\begin{align*}
G^{(\lambda)}_t = R_{t+1} + \gamma\Big((1 - \lambda) v(S_{t+1}) + \lambda G^{(\lambda)}_{t+1}\Big)
\end{align*}

Try to derive
it at home.

\begin{align*}
\Delta v(s) &= \alpha \sum_{t=0}^{T-1} \Big( G_t^{(\lambda)} - v_k(S_t) \Big)\, \mathbb{I}(S_t = s)
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}

Total change of
over an episode.

\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}

Eligibility Traces

Mixed Multi-Step Returns

update equation

\begin{align*}
G^{(\lambda)}_t = R_{t+1} + \gamma\Big((1 - \lambda) v(S_{t+1}) + \lambda G^{(\lambda)}_{t+1}\Big)
\end{align*}

Try to derive
it at home.

\begin{align*}
\Delta v(s) &= \alpha \sum_{t=0}^{T-1} \Big( G_t^{(\lambda)} - v_k(S_t) \Big)\, \mathbb{I}(S_t = s)
\end{align*}

Expand the inner term

Identify the one-step TD error

What do the shaded terms represent?

Total change of
over an episode.

\begin{align*}
\delta_j \equiv R_{j+1} + \gamma v_k(S_{j+1}) - v_k(S_j)
\end{align*}

One-step TD error

\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}

Eligibility Traces

Mixed Multi-Step Returns

update equation

\begin{align*}
G^{(\lambda)}_t = R_{t+1} + \gamma\Big((1 - \lambda) v(S_{t+1}) + \lambda G^{(\lambda)}_{t+1}\Big)
\end{align*}

Try to derive
it at home.

\begin{align*}
\Delta v(s) &= \alpha \sum_{t=0}^{T-1} \Big( G_t^{(\lambda)} - v_k(S_t) \Big)\, \mathbb{I}(S_t = s)
\end{align*}

Expand the inner term

Identify the one-step TD error

Substitute the one-step TD error

Recursive relation for the error term

Total change of
over an episode.

\begin{align*}
\delta_j \equiv R_{j+1} + \gamma v_k(S_{j+1}) - v_k(S_j)
\end{align*}

One-step TD error

\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}

Eligibility Traces

Mixed Multi-Step Returns

update equation

\begin{align*}
G^{(\lambda)}_t = R_{t+1} + \gamma\Big((1 - \lambda) v(S_{t+1}) + \lambda G^{(\lambda)}_{t+1}\Big)
\end{align*}

Try to derive
it at home.

\begin{align*}
\Delta v(s) &= \alpha \sum_{t=0}^{T-1} \Big( G_t^{(\lambda)} - v_k(S_t) \Big)\, \mathbb{I}(S_t = s)
\end{align*}

Expand the inner term

Identify the one-step TD error

Substitute the one-step TD error

Recursive relation for the error term

Expand up to the terminal step T-1.

\begin{align*}
G^{(\lambda)}_T - v_k(S_T) = 0 \text{ assuming } v_k(S_T) = 0
\end{align*}

Total change of
over an episode.

\begin{align*}
\delta_j \equiv R_{j+1} + \gamma v_k(S_{j+1}) - v_k(S_j)
\end{align*}

One-step TD error

\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}

Eligibility Traces

Mixed Multi-Step Returns

update equation

\begin{align*}
G^{(\lambda)}_t = R_{t+1} + \gamma\Big((1 - \lambda) v(S_{t+1}) + \lambda G^{(\lambda)}_{t+1}\Big)
\end{align*}

Try to derive
it at home.

\begin{align*}
\Delta v(s) &= \alpha \sum_{t=0}^{T-1} \Big( G_t^{(\lambda)} - v_k(S_t) \Big)\, \mathbb{I}(S_t = s)
\end{align*}
\begin{align*}
&= \alpha \sum_{t=0}^{T-1} \Bigg[ \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j \Bigg]\, \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \delta_j \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j \Bigg( \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \Bigg) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j E_j(s)
\end{align*}

Change the order of summation

\begin{align*}
\delta_j \equiv R_{j+1} + \gamma v_k(S_{j+1}) - v_k(S_j)
\end{align*}

One-step TD error

Eligibility Traces

Mixed Multi-Step Returns

update equation

\begin{align*}
G^{(\lambda)}_t = R_{t+1} + \gamma\Big((1 - \lambda) v(S_{t+1}) + \lambda G^{(\lambda)}_{t+1}\Big)
\end{align*}

Try to derive
it at home.

\begin{align*}
\Delta v(s) &= \alpha \sum_{t=0}^{T-1} \Big( G_t^{(\lambda)} - v_k(S_t) \Big)\, \mathbb{I}(S_t = s)
\end{align*}
\begin{align*}
&= \alpha \sum_{t=0}^{T-1} \Bigg[ \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j \Bigg]\, \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \delta_j \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j \Bigg( \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \Bigg) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j E_j(s)
\end{align*}
\begin{align*}
&= \alpha \sum_{t=0}^{T-1} \Bigg[ \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j \Bigg]\, \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \delta_j \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j \Bigg( \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \Bigg) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j E_j(s)
\end{align*}
\begin{align*}
&= \alpha \sum_{t=0}^{T-1} \Bigg[ \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j \Bigg]\, \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \delta_j \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j \Bigg( \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \Bigg) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j E_j(s)
\end{align*}

Change the order of summation

\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
\delta_j \equiv R_{j+1} + \gamma v_k(S_{j+1}) - v_k(S_j)
\end{align*}

One-step TD error

Eligibility Traces

Mixed Multi-Step Returns

update equation

\begin{align*}
G^{(\lambda)}_t = R_{t+1} + \gamma\Big((1 - \lambda) v(S_{t+1}) + \lambda G^{(\lambda)}_{t+1}\Big)
\end{align*}

Try to derive
it at home.

\begin{align*}
\Delta v(s) &= \alpha \sum_{t=0}^{T-1} \Big( G_t^{(\lambda)} - v_k(S_t) \Big)\, \mathbb{I}(S_t = s)
\end{align*}
\begin{align*}
&= \alpha \sum_{t=0}^{T-1} \Bigg[ \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j \Bigg]\, \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \delta_j \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j \Bigg( \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \Bigg) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j E_j(s)
\end{align*}
\begin{align*}
&= \alpha \sum_{t=0}^{T-1} \Bigg[ \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j \Bigg]\, \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \delta_j \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j \Bigg( \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \Bigg) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j E_j(s)
\end{align*}
\begin{align*}
&= \alpha \sum_{t=0}^{T-1} \Bigg[ \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j \Bigg]\, \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \delta_j \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j \Bigg( \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \Bigg) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j E_j(s)
\end{align*}
\begin{align*}
&= \alpha \sum_{t=0}^{T-1} \Bigg[ \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j \Bigg]\, \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \delta_j \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j \Bigg( \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \Bigg) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j E_j(s)
\end{align*}

Change the order of summation

Defined as Eligibility Traces:

\begin{align*}
E_j(s) &\equiv \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= (\gamma\lambda)^{j-j} \mathbb{I}(S_j = s) + \sum_{t=0}^{j-1} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda \sum_{t=0}^{j-1} (\gamma\lambda)^{(j-1)-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda E_{j-1}(s)
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
\delta_j \equiv R_{j+1} + \gamma v_k(S_{j+1}) - v_k(S_j)
\end{align*}

One-step TD error

Eligibility Traces

Mixed Multi-Step Returns

update equation

\begin{align*}
G^{(\lambda)}_t = R_{t+1} + \gamma\Big((1 - \lambda) v(S_{t+1}) + \lambda G^{(\lambda)}_{t+1}\Big)
\end{align*}

Try to derive
it at home.

\begin{align*}
\Delta v(s) &= \alpha \sum_{t=0}^{T-1} \Big( G_t^{(\lambda)} - v_k(S_t) \Big)\, \mathbb{I}(S_t = s)
\end{align*}
\begin{align*}
&= \alpha \sum_{t=0}^{T-1} \Bigg[ \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j \Bigg]\, \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \delta_j \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j \Bigg( \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \Bigg) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j E_j(s)
\end{align*}
\begin{align*}
E_j(s) &\equiv \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= (\gamma\lambda)^{j-j} \mathbb{I}(S_j = s) + \sum_{t=0}^{j-1} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda \sum_{t=0}^{j-1} (\gamma\lambda)^{(j-1)-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda E_{j-1}(s)
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
\delta_j \equiv R_{j+1} + \gamma v_k(S_{j+1}) - v_k(S_j)
\end{align*}

One-step TD error

Eligibility Traces

Recurse property of Eligibility Traces

\begin{align*}
E_j(s) &\equiv \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= (\gamma\lambda)^{j-j} \mathbb{I}(S_j = s) + \sum_{t=0}^{j-1} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda \sum_{t=0}^{j-1} (\gamma\lambda)^{(j-1)-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda E_{j-1}(s)
\end{align*}
\begin{align*}
E_j(s) &\equiv \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= (\gamma\lambda)^{j-j} \mathbb{I}(S_j = s) + \sum_{t=0}^{j-1} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda \sum_{t=0}^{j-1} (\gamma\lambda)^{(j-1)-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda E_{j-1}(s)
\end{align*}
\begin{align*}
E_j(s) &\equiv \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= (\gamma\lambda)^{j-j} \mathbb{I}(S_j = s) + \sum_{t=0}^{j-1} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda \sum_{t=0}^{j-1} (\gamma\lambda)^{(j-1)-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda E_{j-1}(s)
\end{align*}
\begin{align*}
E_j(s) &\equiv \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= (\gamma\lambda)^{j-j} \mathbb{I}(S_j = s) + \sum_{t=0}^{j-1} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda \sum_{t=0}^{j-1} (\gamma\lambda)^{(j-1)-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda E_{j-1}(s)
\end{align*}
\begin{align*}
E_j(s) &\equiv \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= (\gamma\lambda)^{j-j} \mathbb{I}(S_j = s) + \sum_{t=0}^{j-1} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda \sum_{t=0}^{j-1} (\gamma\lambda)^{(j-1)-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda E_{j-1}(s)
\end{align*}

Eligibility Traces

Mixed Multi-Step Returns

update equation

\begin{align*}
G^{(\lambda)}_t = R_{t+1} + \gamma\Big((1 - \lambda) v(S_{t+1}) + \lambda G^{(\lambda)}_{t+1}\Big)
\end{align*}

Try to derive
it at home.

\begin{align*}
\Delta v(s) &= \alpha \sum_{t=0}^{T-1} \Big( G_t^{(\lambda)} - v_k(S_t) \Big)\, \mathbb{I}(S_t = s)
\end{align*}
\begin{align*}
&= \alpha \sum_{t=0}^{T-1} \Bigg[ \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j \Bigg]\, \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \delta_j \mathbb{I}(S_t = s) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j \Bigg( \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \Bigg) \\ &= \alpha \sum_{j=0}^{T-1} \delta_j E_j(s)
\end{align*}
\begin{align*}
E_j(s) &\equiv \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= (\gamma\lambda)^{j-j} \mathbb{I}(S_j = s) + \sum_{t=0}^{j-1} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda \sum_{t=0}^{j-1} (\gamma\lambda)^{(j-1)-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda E_{j-1}(s)
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
G_t^{(\lambda)} - v_k(S_t) &= R_{t+1} + \gamma \Big[ (1-\lambda) v_k(S_{t+1}) + \lambda G_{t+1}^{(\lambda)} \Big] - v_k(S_t) \\ &= R_{t+1} + \gamma v_k(S_{t+1}) - \gamma\lambda v_k(S_{t+1}) + \gamma\lambda G_{t+1}^{(\lambda)} - v_k(S_t) \\ &= (R_{t+1} + \gamma v_k(S_{t+1}) - v_k(S_t)) + \gamma\lambda G_{t+1}^{(\lambda)} - \gamma\lambda v_k(S_{t+1}) \\ &= \delta_t + \gamma\lambda \Big( G_{t+1}^{(\lambda)} - v_k(S_{t+1}) \Big) \\ &= \delta_t + \gamma\lambda \Big[ \delta_{t+1} + \gamma\lambda \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \Big] \\ &= \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2 \Big( G_{t+2}^{(\lambda)} - v_k(S_{t+2}) \Big) \\ &= \sum_{j=t}^{T-1} (\gamma\lambda)^{j-t} \delta_j
\end{align*}
\begin{align*}
\delta_j \equiv R_{j+1} + \gamma v_k(S_{j+1}) - v_k(S_j)
\end{align*}

One-step TD error

Eligibility Traces

Recurse property of Eligibility Traces

\begin{align*}
E_j(s) &\equiv \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= (\gamma\lambda)^{j-j} \mathbb{I}(S_j = s) + \sum_{t=0}^{j-1} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda \sum_{t=0}^{j-1} (\gamma\lambda)^{(j-1)-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda E_{j-1}(s)
\end{align*}
\renewcommand{\thealgorithm}{}
\begin{algorithm}
\caption{Tabular TD(\(\lambda\)) for estimating \(V \approx v_\pi\).}
\begin{algorithmic}[1]
\State Input: the policy $\pi$ to be evaluated
\State Algorithm parameters: step size \(\alpha \in (0, 1]\), \textcolor{red}{trace decay parameter \(\lambda \in [0,1]\)}
\State Initialize \(V(s)\), for all \(s \in \mathcal{S}^+\), arbitrarily except that \(V(\text{terminal}) = 0\) \\

\For{each episode}
\State Initialize \(S\)
\State \textcolor{red}{Initialize eligibility traces \(e(s) = 0\) for all states \(s \in \mathcal{S}\)}
\For{\(t=1, ..., T\) (until terminal state)}
\State $A \leftarrow$ action given by $\pi$ for $S$
\State Take action \(A\), observe \(R\), \(S'\)
\State \(\delta = R + \gamma V(S') - V(S)\)
\State \textcolor{red}{\(e(S) \leftarrow e(S) + 1\) }
\For{all states \(s \in \mathcal{S}\)}
\State \(V(s) \leftarrow V(s) + \alpha \delta \textcolor{red}{e(s)}\)
\State \textcolor{red}{\(e(s) \leftarrow \gamma \lambda e(s)\)}
\EndFor
\State \(S \leftarrow S'\)
\EndFor
\EndFor
\label{tabular TD(lambda)}
\end{algorithmic}
\end{algorithm}
\begin{align*}
E_j(s) &\equiv \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= (\gamma\lambda)^{j-j} \mathbb{I}(S_j = s) + \sum_{t=0}^{j-1} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda \sum_{t=0}^{j-1} (\gamma\lambda)^{(j-1)-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda E_{j-1}(s)
\end{align*}

Recurse property of Eligibility Traces

\begin{align*}
E_j(s) &\equiv \sum_{t=0}^{j} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= (\gamma\lambda)^{j-j} \mathbb{I}(S_j = s) + \sum_{t=0}^{j-1} (\gamma\lambda)^{j-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda \sum_{t=0}^{j-1} (\gamma\lambda)^{(j-1)-t} \mathbb{I}(S_t = s) \\ &= \mathbb{I}(S_j = s) + \gamma\lambda E_{j-1}(s)
\end{align*}

(Online algorithm!)

\renewcommand{\thealgorithm}{}
\begin{algorithm}
\caption{Tabular TD(\(\lambda\)) for estimating \(V \approx v_\pi\).}
\begin{algorithmic}[1]
\State Input: the policy $\pi$ to be evaluated
\State Algorithm parameters: step size \(\alpha \in (0, 1]\), \textcolor{red}{trace decay parameter \(\lambda \in [0,1]\)}
\State Initialize \(V(s)\), for all \(s \in \mathcal{S}^+\), arbitrarily except that \(V(\text{terminal}) = 0\) \\

\For{each episode}
\State Initialize \(S\)
\State \textcolor{red}{Initialize eligibility traces \(e(s) = 0\) for all states \(s \in \mathcal{S}\)}
\For{\(t=1, ..., T\) (until terminal state)}
\State $A \leftarrow$ action given by $\pi$ for $S$
\State Take action \(A\), observe \(R\), \(S'\)
\State \(\delta = R + \gamma V(S') - V(S)\)
\State \textcolor{red}{\(e(S) \leftarrow e(S) + 1\) }
\For{all states \(s \in \mathcal{S}\)}
\State \(V(s) \leftarrow V(s) + \alpha \delta \textcolor{red}{e(s)}\)
\State \textcolor{red}{\(e(s) \leftarrow \gamma \lambda e(s)\)}
\EndFor
\State \(S \leftarrow S'\)
\EndFor
\EndFor
\label{tabular TD(lambda)}
\end{algorithmic}
\end{algorithm}
\renewcommand{\thealgorithm}{}

\begin{algorithm}
\caption{Tabular TD($0$) for estimating \(V \approx v_\pi\).}
\begin{algorithmic}[1]
\State Input: the policy $\pi$ to be evaluated
\State Algorithm parameters: step size \(\alpha \in (0, 1]\)
\State Initialize \(V(s)\), for all \(s \in \mathcal{S}^+\), arbitrarily except that \(V(\text{terminal}) = 0\) \\

\For{each episode}
\State Initialize \(S\)

\For{\(t=1, ..., T\) (until terminal state)}
\State $A \leftarrow$ action given by $\pi$ for $S$
\State Take action \(A\), observe \(R\), \(S'\)
\State \(\delta = R + \gamma V(S') - V(S)\)
\State \(V(s) \leftarrow V(s) + \alpha \delta \)
\State \(S \leftarrow S'\)
\EndFor
\EndFor
\label{tabular TD(lambda)}
\end{algorithmic}
\end{algorithm}

(Online algorithm!)

Benefits of mixed multi-step returns

➢Mixed multi-step returns have benefits from both TD and MC

​

➢Bootstrapping can have issues with bias

​

➢Monte Carlo can have issues with variance

​

➢Typically, intermediate values of or are good (e.g., )

Mixed Multi-Step Returns

RMS error at the
end of the
episode over the
first 10 episodes

Gridworld visualization for TD(0) vs TD()

Take home message

•It is possible to evaluate a policy by only using sampled trajectories

•Monte Carlo algorithms sample returns.

•Unbiased estimates

•Higher Variance

•Time-Difference methods bootstrap

•Biased estimates

•Lower Variance

•Intermediate solutions try to get the best of both worlds

•-step TD

•TD()

​

Take home message

•Further studying:

•Sections 5.1, 5.2, 6.1, 6.2, 6.3, 7.1, 7.2, 12.1, 12.2– Reinforcement Learning: An introduction

•Short notes:

•Monte Carlo: https://people.stfx.ca/jdelamer/courses/csci-531/lectures/rl/monte-carlo/monte-carlo-
methods.html

•TD-learning: https://people.stfx.ca/jdelamer/courses/csci-531/lectures/rl/Temporal-
difference%20learning/td-learning.html

•Video (UCL x DeepMind)

•Lectures 5:
https://www.youtube.com/watch?v=eaWfWoVUTEw&list=PLqYmG7hTraZCRwoyGxvQkqVrZgDQi4m-
5&index=5&t=13s

•Next lecture:

•MC control, SARSA, Q-learning, Double Q-learning, DQN

​