Exercise 2.4.4

Bellman expectation equations: derive v_π(s) = Σ π(a|s) q_π(s,a)

MediumWritten

Derive the relation between \(q_{\pi}(s,a)\) and \(v_{\pi}(s)\), namely that \[\begin{align*} v_{\pi}(s) = \sum_a \pi(a|s) q_\pi(s,a). \end{align*}\] It might be helpful to start from \(\mathbb{E}(X|Y=y)=\sum_x x \times \mathbb{P}(X=x|Y=y)\) and \(\pi(a|s)=\mathbb{P}(a|s)\), then apply the definition of expectation and conditional probability.

Adapted from Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 3.12. Licensed CC BY-NC-ND 2.0.