Exercise 7.5.2
Baseline and advantage function: TD error as advantage estimate
A general form of the policy gradient update is derived from the action-value function, \(Q^{\pi_\theta}(s,a)\): \[\theta \leftarrow \theta + \alpha Q^{\pi_\theta}(s,a) \nabla_\theta \log \pi_\theta(a|s)\] While this is theoretically sound, using estimates for \(Q^{\pi_\theta}(s,a)\) can lead to high variance.
-
(a) What is the general technique for modifying the learning signal, \(Q^{\pi_\theta}(s,a)\), in order to reduce its variance without introducing any bias into the policy gradient estimate?
-
(b) A powerful and common choice for the baseline is the formal definition of the average value of a state under the current policy. What is this quantity called in reinforcement learning? Using this quantity as the baseline, define the resulting learning signal (this is known as the Advantage Function).
-
(c) Implementing the learning signal you defined in (b) appears to require learning two separate function approximators, which is inefficient. Describe how to construct a practical, single-sample estimate of this signal using only a single learned state-value function, \(\hat{v}(s,w)\). Write down the resulting formula and explain how its components serve as estimates for the terms in the original signal’s definition.