Exercise 3.3.1
Policy iteration: stochastic policies, 3-argument p, and the action-value version
Consider the policy iteration algorithm for estimating \(\pi \approx \pi_*\).
-
(a) Modify the given algorithm to work with stochastic policies \(\pi(a|s)\), the three-argument transition function \(p(s'|s,a)\) and two argument reward function \(r(s,a)\) instead of the four-argument transition function \(p(s',r|s,a)\) and triple argument reward function \(r(s,a,s')\). The given algorithm assumes the latter ((\(r\) is shorthand for triple argument reward function).
-
(b) How would policy iteration be defined for action values? Give a complete algorithm for computing \(q_*\) analogous to the one below for computing \(v_*\). Assume a deterministic policy.
Adapted from Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 4.5. Licensed CC BY-NC-ND 2.0.