Exercise 7.2.1

Policy-gradient theorem: on-policy requirement and off-policy mismatch

MediumWritten

Policy gradient methods are often the preferred choice for certain types of reinforcement learning problems where value-based methods like Q-learning are difficult to apply.

  • a) By referencing the mathematical formulation of the policy gradient, \(\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[...]\), explain precisely why these methods require on-policy data for their updates.

  • b) What is the theoretical problem with trying to use off-policy data (e.g., data from an older policy \(\pi_{\theta_{old}}\)) to compute the gradient for the current policy \(\pi_\theta\) without any correction? (You do not need to derive Importance Sampling, but explain the source of the issue).