Exercise 7.4.2

REINFORCE vs value-based: action spaces, exploration, variance (mock)

Reinforcement Learning › Exercises › Lecture 7 · Policy Gradients and Actor-Critic › 7.4 Reducing the variance: causality and baseline
EasyWritten

The landscape of reinforcement learning (RL) offers a variety of methodologies for tackling a wide range of decision-making problems. Among these, policy-gradient methods and value-based methods are particularly noteworthy due to their unique approaches and areas of applicability. One of the key policy-gradient methods is the REINFORCE algorithm, which updates policies directly in a gradient ascent manner to maximize the expected return. The gradient formula for the REINFORCE algorithm is given by: \[\begin{equation*} \nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta}} \left[\sum_{t=0}^{T-1} G_t\nabla_\theta \log \pi_\theta(a_t|s_t) \right] \end{equation*}\] where \(J(\theta)\) is the objective function representing the expected return, \(\pi_\theta(a_t|s_t)\) is the policy (a distribution over actions given the current state and parameterized by \(\theta\)), \(G_t\) is the return from time \(t\) onwards, and \(\tau\) is the trajectory.

  • 1. Applicability to Complex Action Spaces: Explain why policy-gradient methods are better suited than value-based methods for dealing with high-dimensional and continuous action spaces. Provide one example where this advantage is particularly significant.

  • 2. Conceptual Foundations: Outline two core principles that differentiate policy-gradient methods from value-based methods.

  • 3. Exploration: Explain how exploration is driven in policy-gradient methods like REINFORCE.

  • 4. Variance: Explain what is the variance problem in the REINFORCE algorithm and provide a possible mitigation to it.