Exercise 2.4.7

Bellman expectation equations: q_π in terms of v_π and four-argument p

MediumWritten

Give an equation for \(q_{\pi}\) in terms of \(v_{\pi}\) and the four-argument \(p\). Hint: start from the definition of \(q_{\pi}\) and go step by step until you get the result in terms of \(v_{\pi}\) and \(p\). Giving only the final expression is not enough: show the derivation.

Source: Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 3.13. Licensed CC BY-NC-ND 2.0.