Exercise 5.7.2
Importance sampling: off-policy TD(0) update (exercise 6.7)
Design an off-policy version of the TD(0) update that can be used with arbitrary target policy \(\pi\) and covering behavior policy \(b\), using at each step \(t\) the importance sampling ratio \(\rho_{t:t}\), where \[\begin{align*} \rho_{t:T-1} \doteq \frac{\prod_{k=t}^{T-1} \pi(A_k|S_k)p(S_{k+1}|S_k, A_k)}{\prod_{k=t}^{T-1} b(A_k|S_k)p(S_{k+1}|S_k, A_k)} = \prod_{k=t}^{T-1} \frac{\pi(A_k|S_k)}{b(A_k|S_k)} \end{align*}\]
Source: Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 6.7. Licensed CC BY-NC-ND 2.0.