Exercise 7.3.1
REINFORCE derivation: model-dependent terms and model-free property
The Derivation and Implications of the REINFORCE Update
The objective in policy gradient methods is to maximize the expected return, \(J(\theta) = \mathbb{E}_{\tau \sim p_\theta(\tau)}[R(\tau)]\). This expectation is taken over trajectories, \(\tau\), whose probability is given by: \[p_\theta(\tau) = p(s_0) \prod_{t=0}^{T-1} \pi_\theta(a_t|s_t) p(s_{t+1}|s_t, a_t)\] where \(p(s_0)\) is the initial state distribution, \(\pi_\theta(a_t|s_t)\) is a conditional probability distribution over actions given the current state (parameterized by \(\theta\)), and \(p(s_{t+1}|s_t, a_t)\) is the conditional probability of the next state given the current state and action.
Through a key derivation, this objective leads to the practical REINFORCE update rule: \[\theta \leftarrow \theta + \alpha G_t \nabla_\theta \log \pi_\theta(a_t|s_t)\] where \(\alpha\) is the learning rate and \(G_t\) is the return from time step \(t\).
-
(a) Examine the terms in the formula for \(p_\theta(\tau)\). Identify the term that makes this probability distribution dependent on the environment’s model.
-
(b) The policy gradient derivation requires computing the gradient of the log-probability of a trajectory, \(\nabla_\theta \log p_\theta(\tau)\). Derive a simplified expression for this term.
-
(c) Using your findings from (a) and (b), explain how the final REINFORCE update rule can be implemented without knowledge of the term you identified as being model-dependent. What is this property of an algorithm called, and why is it so crucial for applying reinforcement learning in practice?