Exercise 4.3.2
Batch Monte Carlo versus batch TD(0): value estimates from eight episodes
Place yourself now in the role of the predictor of returns for an unknown Markov reward process. Suppose you observe the following eight episodes:
| Episode 1 | Episode 2 | Episode 3 | Episode 4 | Episode 5 | Episode 6 | Episode 7 | Episode 8 |
|---|---|---|---|---|---|---|---|
| A, 1, B, 0 | B, 1 | B, 1 | A, 0, B, 1 | B, 1 | B, 0 | B, 1 | B, 0 |
We call this batch a result of “real experiences". This means that the first episode started in state A, and transitioned to B with a reward of \(1\), and then transitioned from B to the terminal state with a reward of \(0\).
-
a) Draw the Markov process that can be inferred from this data.
-
b) What is the state-value estimate of \(V(A)\) and \(V(B)\) when using: (a)Batch Monte Carlo, (b)Batch TD(0)? Consider \(\gamma = 1\).
Adapted from Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Example 6.4. Licensed CC BY-NC-ND 2.0.