Exercise 7.4.3
Policy-gradient theorem: unbiasedness of the baseline estimate
Unbiasedness of the Policy Gradient with a Baseline
Let the policy be denoted by \(\pi_\theta(a|s)\), where \(\theta\) are the parameters. The performance objective is the expected total return, \(J(\theta)\). The Policy Gradient Theorem states that the true gradient of this objective is given by: \[\nabla_\theta J(\theta) = \mathbb{E}_{s \sim d^{\pi_\theta}, a \sim \pi_\theta(\cdot|s)} [Q^{\pi_\theta}(s, a) \nabla_\theta \log \pi_\theta(a|s)]\] Here, \(d^{\pi_\theta}(s)\) is the on-policy state distribution, and \(Q^{\pi_\theta}(s, a)\) is the action-value function.
Now, consider a single-sample stochastic estimate of the gradient, \(\hat{g}\), which is computed using a state \(s\) and action \(a\) sampled from the environment, and which includes a state-dependent baseline \(b(s)\): \[\hat{g}(s, a, \theta) = (Q^{\pi_\theta}(s, a) - b(s)) \nabla_\theta \log \pi_\theta(a|s)\] Prove that this stochastic estimate is unbiased. That is, prove that its expected value is equal to the true gradient: \[\mathbb{E}_{s \sim d^{\pi_\theta}, a \sim \pi_\theta(\cdot|s)}[\hat{g}(s, a, \theta)] = \nabla_\theta J(\theta)\] (For this proof, you may assume that the state and action spaces are discrete.)
(Hint: The proof requires showing that the expected value of the term involving the baseline is equal to zero.)