Exercise 8.1.3
Dyna-Q: stochastic and changing environments
Reinforcement Learning › Exercises › Lecture 8 · Model-Based Reinforcement Learning › 8.1 Model-based RL and Dyna-Q
Tabular Dyna-Q
Initialize \(Q(s,a)\) and \(\mathit{Model}(s,a)\) for all \(s \in \mathcal{S}\) and \(a \in \mathcal{A}(s)\)
Loop forever:
- \(S \leftarrow\) current (nonterminal) state
- \(A \leftarrow \varepsilon\text{-greedy}(S, Q)\)
- Take action \(A\); observe resultant reward, \(R\), and state, \(S'\)
- \(Q(S,A) \leftarrow Q(S,A) + \alpha\left[R + \gamma \max_a Q(S',a) - Q(S,A)\right]\)
- \(\mathit{Model}(S,A) \leftarrow R, S'\) (assuming deterministic environment)
- Loop repeat \(n\) times:\(S \leftarrow\) random previously observed state
\(A \leftarrow\) random action previously taken in \(S\)
\(R, S' \leftarrow \mathit{Model}(S,A)\)
\(Q(S,A) \leftarrow Q(S,A) + \alpha\left[R + \gamma \max_a Q(S',a) - Q(S,A)\right]\)
How might the tabular Dyna-Q algorithm above be modified to handle stochastic environments? How might this modification perform poorly on changing environments? How could the algorithm be modified to handle stochastic environments and changing environments?
Source: Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 8.5. The Tabular Dyna-Q algorithm of Section 8.2 is reproduced from the book. Licensed CC BY-NC-ND 2.0.