Exercise 7.1.6
Stochastic policies: perceptual aliasing, policy gradient vs Q-learning
Policy gradient methods are often the preferred choice for certain types of reinforcement learning problems where value-based methods like Q-learning are difficult to apply.
Consider a problem with perceptual aliasing, where multiple distinct underlying states produce the same observation for an agent. If we constrain our agent to use a reactive policy that maps the current observation directly to an action, the optimal policy within this class may need to be stochastic.
Analyze how both a policy gradient method and a value-based method (like Q-learning) would attempt to find an optimal policy in this scenario. In your analysis, focus on what each method learns and how it derives a policy from that learned knowledge. Based on this, which approach is more suitable, and why?