Exercise 3.4.1
Value iteration: action-value analogue q_{k+1}(s,a)
From b) and c), by substitution one gets a recursive relation for \(q_{\pi}\): \[\begin{align*} q_{\pi}(s,a)=\sum_{s', r}p(s', r|s,a)(r+\gamma \sum_{a'} \pi(a'|s') q_{\pi}(s',a')) \end{align*}\]
Consider the value iteration update: \[\begin{align*} v_{k+1}(s) = \max_a \mathbb{E}(R_{t+1} + \gamma v_k(S_{t+1})|S_t=s, A_t=a)\\ = \max_a \sum_{s',r}p(s',r|s,a) ( r + \gamma v_k(s'))\\ = \max_a (r(s,a) + \gamma \sum_{s'} p(s'|s,a)v_k(s')) \end{align*}\] It takes the next action \(a\) that maximizes \(v_{k+1}\). What is the analog of the value iteration update for action values, \(q_{k+1}(s,a)\)?
Source: Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 4.10. Licensed CC BY-NC-ND 2.0.