Exercise 3.3.1

Policy iteration: stochastic policies, 3-argument p, and the action-value version

MediumWritten

Consider the policy iteration algorithm for estimating \(\pi \approx \pi_*\).

boxed text
Policy iteration pseudocode from Sutton & Barto (2018), Section 4.3.
  • (a) Modify the given algorithm to work with stochastic policies \(\pi(a|s)\), the three-argument transition function \(p(s'|s,a)\) and two argument reward function \(r(s,a)\) instead of the four-argument transition function \(p(s',r|s,a)\) and triple argument reward function \(r(s,a,s')\). The given algorithm assumes the latter ((\(r\) is shorthand for triple argument reward function).

  • (b) How would policy iteration be defined for action values? Give a complete algorithm for computing \(q_*\) analogous to the one below for computing \(v_*\). Assume a deterministic policy.

Adapted from Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 4.5. Licensed CC BY-NC-ND 2.0.