Exercise 2.3.3

Returns and discounting: maze, reward design, discounted and undiscounted

MediumWritten

Imagine that you are designing a robot to run a maze. The task seems to break down naturally into episodes – the successive runs through the maze. You decide to treat it as an episodic task. Now, consider the following scenarios:

  • (a) The robot receives a reward of 0 at all times unless it escapes the maze when it receives a reward of 1. You try to maximize the undiscounted return. After running the learning agent for a while, you find that it is showing no improvement in escaping from the maze. What is going wrong? Have you effectively communicated to the agent what you want it to achieve?

  • (b) Now, consider the discounted return with \(0<\gamma<1\). Will the agent learn to escape the maze as soon as possible? Give an expression for the return at time \(t\) if the agent completes the maze at time \(T\).

  • (c) You decide to take a different route – you will design the reward function yourself. What is a possible reward function so that the agent will successfully complete the maze in the undiscounted, episodic case?

Adapted from Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 3.7. Licensed CC BY-NC-ND 2.0.