Exercise 2.4.7
Bellman expectation equations: q_π in terms of v_π and four-argument p
Reinforcement Learning › Exercises › Lecture 2 · The RL Problem and Finite MDPs › 2.4 Bellman expectation equations
Give an equation for \(q_{\pi}\) in terms of \(v_{\pi}\) and the four-argument \(p\). Hint: start from the definition of \(q_{\pi}\) and go step by step until you get the result in terms of \(v_{\pi}\) and \(p\). Giving only the final expression is not enough: show the derivation.
Source: Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 3.13. Licensed CC BY-NC-ND 2.0.