Exercise 2.2.2

Markov decision process: modelling a warehouse robot, Markov property

EasyWritten

Consider a scenario in which an autonomous robot navigates through a warehouse to fetch items and deliver them to designated drop-off zones. The warehouse is divided into a grid where each cell can either be empty, contain an item, or be a drop-off zone. Some areas of the warehouse might have slippery floors, causing the robot to occasionally slide into unintended adjacent cells.

  • (a) Explain the difference between stochastic and deterministic transition functions. Considering the scenario, which would be more appropriate and why?

  • (b) Explain what the Markov property is and why it is theoretically required when using Reinforcement Learning algorithms to solve this problem.

  • (c) Describe how you would model this problem as a Markov Decision Process. Include the definition of states, actions, transition function, and reward function.

  • (d) Provide an example from the scenario that illustrates the memorylessness feature of the Markov property.