Exercise 2.4.4
Bellman expectation equations: derive v_π(s) = Σ π(a|s) q_π(s,a)
Reinforcement Learning › Exercises › Lecture 2 · The RL Problem and Finite MDPs › 2.4 Bellman expectation equations
Derive the relation between \(q_{\pi}(s,a)\) and \(v_{\pi}(s)\), namely that \[\begin{align*} v_{\pi}(s) = \sum_a \pi(a|s) q_\pi(s,a). \end{align*}\] It might be helpful to start from \(\mathbb{E}(X|Y=y)=\sum_x x \times \mathbb{P}(X=x|Y=y)\) and \(\pi(a|s)=\mathbb{P}(a|s)\), then apply the definition of expectation and conditional probability.
Adapted from Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 3.12. Licensed CC BY-NC-ND 2.0.