Exercise 2.2.2
Markov decision process: modelling a warehouse robot, Markov property
Consider a scenario in which an autonomous robot navigates through a warehouse to fetch items and deliver them to designated drop-off zones. The warehouse is divided into a grid where each cell can either be empty, contain an item, or be a drop-off zone. Some areas of the warehouse might have slippery floors, causing the robot to occasionally slide into unintended adjacent cells.
-
(a) Explain the difference between stochastic and deterministic transition functions. Considering the scenario, which would be more appropriate and why?
-
(b) Explain what the Markov property is and why it is theoretically required when using Reinforcement Learning algorithms to solve this problem.
-
(c) Describe how you would model this problem as a Markov Decision Process. Include the definition of states, actions, transition function, and reward function.
-
(d) Provide an example from the scenario that illustrates the memorylessness feature of the Markov property.