Exercise 7.4.1

Baseline in REINFORCE: motivation and required property

Reinforcement Learning › Exercises › Lecture 7 · Policy Gradients and Actor-Critic › 7.4 Reducing the variance: causality and baseline
EasyWritten

The Variance Problem and the Role of Baselines

The standard REINFORCE algorithm is known to be an unbiased estimator of the policy gradient, but it often suffers from high variance in practice, which can make learning slow and unstable.

  • (a) Explain the motivation for introducing a state-dependent baseline, \(b(s)\), into the policy gradient update. What specific problem is the baseline designed to mitigate, and how does it achieve this?

  • (b) What is the key theoretical property that any valid baseline must satisfy to ensure that the policy gradient estimate remains correct? Briefly explain the mathematical reason why this property is essential.