Exercise 7.5.3

TD target as estimate of V or Q; TD error as advantage

Reinforcement Learning › Exercises › Lecture 7 · Policy Gradients and Actor-Critic › 7.5 Actor-critic, advantage function, A2C
MediumWritten

The quantity \(R_{t+1} + \gamma V^{\pi}(S_{t+1})\), known as the TD target, is a cornerstone of reinforcement learning. Its interpretation depends critically on the context of the estimation problem. This question explores this dual role and its application in Actor-Critic algorithms.

Analyze the following two scenarios:

  • (a) Scenario A: An agent is in state \(S_t\) and is about to take an action by sampling from its policy, \(A_t \sim \pi(\cdot|S_t)\). In this context, what fundamental quantity is the TD target a single-sample estimate of?

  • (b) Scenario B:An agent was in state \(S_t\), took a specific action \(A_t\), and then observed the outcome \((R_{t+1}, S_{t+1})\). In this new context, what fundamental quantity is the TD target a single-sample estimate of?

  • (c) In an Actor-Critic algorithm, the actor’s update should be guided by how much better or worse the chosen action was compared to the average value of the state. Demonstrate how the TD error, \(\delta_t = R_{t+1} + \gamma V^{\pi}(S_{t+1}) - V^{\pi}(S_t)\), provides exactly this measurement. Your demonstration must explicitly use your conclusions from (a) and (b) to interpret the components of the TD error formula. Conclude by stating the formal name for the quantity that the TD error is estimating.