Exercise 7.4.4

REINFORCE with baseline: unbiasedness, variance, exploration, principles

Reinforcement Learning › Exercises › Lecture 7 · Policy Gradients and Actor-Critic › 7.4 Reducing the variance: causality and baseline
MediumWritten

The landscape of reinforcement learning (RL) offers a variety of methodologies for tackling a wide range of decision-making problems. Among these, policy-gradient methods and value-based methods are particularly noteworthy due to their unique approaches and areas of applicability. One of the key policy-gradient methods is the REINFORCE with baseline algorithm, which updates policies directly in a gradient ascent manner to maximize the expected return. The gradient formula for the REINFORCE with baseline algorithm is given by: \[\begin{equation*} \nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta}} \left[\sum_{t=0}^{T-1} (G_t - b(s))\nabla_\theta \log \pi_\theta(a_t|s_t) \right] \end{equation*}\] where \(J(\theta)\) is the objective function representing the expected return, \(\pi_\theta(a_t|s_t)\) is the policy (a distribution over actions given the current state and parameterized by \(\theta\)), \(G_t\) is the return from time \(t\) onwards, and \(\tau\) is the trajectory.

  • a) Prove mathematically that \[\begin{equation*} \mathbb{E}_{\tau \sim \pi_{\theta}} \left[\sum_{t=0}^{T-1} (G_t - b(s))\nabla_\theta \log \pi_\theta(a_t|s_t) \right] = \mathbb{E}_{\tau \sim \pi_{\theta}} \left[\sum_{t=0}^{T-1} (G_t )\nabla_\theta \log \pi_\theta(a_t|s_t) \right] \end{equation*}\]

  • b) Why does subtracting a baseline \(b(s)\) from the return \(G_t\) improve the performance of the REINFORCE algorithm?

  • c) Explain how exploration is driven in policy-gradient methods like REINFORCE.

  • d) Outline two core principles that differentiate policy-gradient methods from value-based methods.