Exercise 2.5.1

Returns and discounting: optimal policy of a two-action MDP for γ = 0, 0.5, 0.9

MediumWritten

Consider the continuing MDP shown in Figure 1. The only decision to be made is that in the top state, where two actions are available, left and right. The numbers show the rewards that are received deterministically after each action. There are exactly two deterministic policies, \(\pi_{\text{left}}\) and \(\pi_{\text{right}}\). What policy is optimal if \(\gamma = 0\)? If \(\gamma = 0.9\)? If \(\gamma = 0.5\)?

figex3.22.png
Figure 1: Single state MDP with two actions.

Source: Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 3.22. The figure of Exercise 3.22 is redrawn from the book. Licensed CC BY-NC-ND 2.0.