Exercise 2.4.6

Bellman expectation equations: prove q_π from v_π and p

MediumWritten

Using the properties of expected value, the Markov property, and the previous subquestion prove that \[\begin{align*} q_{\pi}(s,a)=\sum_{s', r}p(s', r|s,a)(r+\gamma v_{\pi}(s')). \end{align*}\]

Adapted from Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 3.13. Licensed CC BY-NC-ND 2.0.