Exercise 4.3.2

Batch Monte Carlo versus batch TD(0): value estimates from eight episodes

Reinforcement Learning › Exercises › Lecture 4 · Monte Carlo Methods and TD Learning › 4.3 MC vs TD: bias and variance, batch updates
EasyWritten

Place yourself now in the role of the predictor of returns for an unknown Markov reward process. Suppose you observe the following eight episodes:

Table 1: Batch of Real Experience
Episode 1 Episode 2 Episode 3 Episode 4 Episode 5 Episode 6 Episode 7 Episode 8
A, 1, B, 0 B, 1 B, 1 A, 0, B, 1 B, 1 B, 0 B, 1 B, 0

We call this batch a result of “real experiences". This means that the first episode started in state A, and transitioned to B with a reward of \(1\), and then transitioned from B to the terminal state with a reward of \(0\).

  • a) Draw the Markov process that can be inferred from this data.

  • b) What is the state-value estimate of \(V(A)\) and \(V(B)\) when using: (a)Batch Monte Carlo, (b)Batch TD(0)? Consider \(\gamma = 1\).

Adapted from Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Example 6.4. Licensed CC BY-NC-ND 2.0.