Exercise 7.4.3

Policy-gradient theorem: unbiasedness of the baseline estimate

Reinforcement Learning › Exercises › Lecture 7 · Policy Gradients and Actor-Critic › 7.4 Reducing the variance: causality and baseline
MediumWritten

Unbiasedness of the Policy Gradient with a Baseline

Let the policy be denoted by \(\pi_\theta(a|s)\), where \(\theta\) are the parameters. The performance objective is the expected total return, \(J(\theta)\). The Policy Gradient Theorem states that the true gradient of this objective is given by: \[\nabla_\theta J(\theta) = \mathbb{E}_{s \sim d^{\pi_\theta}, a \sim \pi_\theta(\cdot|s)} [Q^{\pi_\theta}(s, a) \nabla_\theta \log \pi_\theta(a|s)]\] Here, \(d^{\pi_\theta}(s)\) is the on-policy state distribution, and \(Q^{\pi_\theta}(s, a)\) is the action-value function.

Now, consider a single-sample stochastic estimate of the gradient, \(\hat{g}\), which is computed using a state \(s\) and action \(a\) sampled from the environment, and which includes a state-dependent baseline \(b(s)\): \[\hat{g}(s, a, \theta) = (Q^{\pi_\theta}(s, a) - b(s)) \nabla_\theta \log \pi_\theta(a|s)\] Prove that this stochastic estimate is unbiased. That is, prove that its expected value is equal to the true gradient: \[\mathbb{E}_{s \sim d^{\pi_\theta}, a \sim \pi_\theta(\cdot|s)}[\hat{g}(s, a, \theta)] = \nabla_\theta J(\theta)\] (For this proof, you may assume that the state and action spaces are discrete.)

(Hint: The proof requires showing that the expected value of the term involving the baseline is equal to zero.)