Exercise 2.4.5
Bellman expectation equations: prove q_π from v_π and p
Reinforcement Learning › Exercises › Lecture 2 · The RL Problem and Finite MDPs › 2.4 Bellman expectation equations
Using the properties of expected value, the Markov property, and the previous subquestion prove that \[\begin{align*} q_{\pi}(s,a)=\sum_{s', r}p(s', r|s,a)(r+\gamma v_{\pi}(s')). \end{align*}\]
Adapted from Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 3.13. Licensed CC BY-NC-ND 2.0.