Exercise 4.3.3

MC vs TD: when a TD update beats a Monte Carlo update (driving home)

Reinforcement Learning › Exercises › Lecture 4 · Monte Carlo Methods and TD Learning › 4.3 MC vs TD: bias and variance, batch updates
MediumWritten

This is an exercise to help develop your intuition about why TD methods are often more efficient than Monte Carlo methods. Consider the driving home example (example 6.1 from the book) and how it is addressed by TD and Monte Carlo methods.

  • (a) Can you imagine a scenario in which a TD update would be better on average than a Monte Carlo update? Give an example scenario—a description of past experience and a current state—in which you would expect the TD update to be better. Here’s a hint: Suppose you have lots of experience driving home from work. Then you move to a new building and a new parking lot (but you still enter the highway at the same place). Now you are starting to learn predictions for the new building.

  • (b) Can you see why TD updates are likely to be much better, at least initially, in this case? Might the same sort of thing happen in the original scenario?

Source: Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 6.2; Example 6.1. Licensed CC BY-NC-ND 2.0.