Exercise 7.1.6

Stochastic policies: perceptual aliasing, policy gradient vs Q-learning

Reinforcement Learning › Exercises › Lecture 7 · Policy Gradients and Actor-Critic › 7.1 Policy-based RL: parameterized policies
EasyWritten

Policy gradient methods are often the preferred choice for certain types of reinforcement learning problems where value-based methods like Q-learning are difficult to apply.

Consider a problem with perceptual aliasing, where multiple distinct underlying states produce the same observation for an agent. If we constrain our agent to use a reactive policy that maps the current observation directly to an action, the optimal policy within this class may need to be stochastic.

Analyze how both a policy gradient method and a value-based method (like Q-learning) would attempt to find an optimal policy in this scenario. In your analysis, focus on what each method learns and how it derives a policy from that learned knowledge. Based on this, which approach is more suitable, and why?