Exercise 2.3.4

Returns and discounting: pole balancing, episodic vs continuing return

MediumWritten

The pole-balancing problem (also called inverted pendulum) is a classic reinforcement learning task in which an agent must apply forces to a cart so that a pole hinged to it stays upright. The task ends (failure) if the pole falls too far or the cart moves out of bounds. Suppose you treated pole-balancing as an episodic task but also used discounting, with all rewards zero except for \(-1\) upon failure. What then would the return be at each time? How does this return differ from that in the discounted, continuing formulation of this task?

Source: Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 3.6. Licensed CC BY-NC-ND 2.0.