Exercise 4.2.1
TD error for action values: identity G_t - Q(S_t,A_t) as a sum of TD errors
Reinforcement Learning › Exercises › Lecture 4 · Monte Carlo Methods and TD Learning › 4.2 Temporal-difference learning: TD(0)
Show that an action-value version of \[G_t - V(S_t) = \sum^{T-1}_{k=t} \gamma^{k-t} \delta_k, \quad (I)\] holds for the action-value form of the TD error \(\delta_t = R_{t+1} + \gamma Q(S_{t+1}, A_{t+1}) - Q(S_t, A_t)\), assuming that the values don’t change from step to step.
Source: Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 6.8. Licensed CC BY-NC-ND 2.0.