Exercise 7.2.1
Policy-gradient theorem: on-policy requirement and off-policy mismatch
Policy gradient methods are often the preferred choice for certain types of reinforcement learning problems where value-based methods like Q-learning are difficult to apply.
-
a) By referencing the mathematical formulation of the policy gradient, \(\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[...]\), explain precisely why these methods require on-policy data for their updates.
-
b) What is the theoretical problem with trying to use off-policy data (e.g., data from an older policy \(\pi_{\theta_{old}}\)) to compute the gradient for the current policy \(\pi_\theta\) without any correction? (You do not need to derive Importance Sampling, but explain the source of the issue).